Introducing the Google Custom Search Engine

As we saw in Chapter 3, The Collector Service - Using Controllers to Collect Data, not every website is open to HTTP requests coming from unknown sources. We experienced cross-origin resource sharing (CORS) issues before and we saw how to use Yahoo APIs to work around it. So what we need here is a list of links related to searched keywords. Assembling a URL which uses Google to search for keywords is easy. What is not easy is passing that search result to the Yahoo API for extracting the contents. If you do that, you will get an "access denied" response from Google.

That makes sense because if Google people don't set any control over the source and number of queries that hit their servers, they will be vulnerable and a few lines of code can put a lot of pressure on, or even crash, their servers. Basically, what they say is: you are welcome to use our search engine in browser/mobile applications but if you are going to make requests via code or command...