In this chapter, we saw how we can write a custom crawler from scratch. This task is made easier using Python's modules, such as requests, BeautifulSoup, and so on. Feel free to download the whole code base and test the crawler with various other websites in order to examine its coverage. There may be occasions in which the crawler does not give 100% coverage. Take a look and see for yourself the limitations of the crawler and how it can be improved.
United States
United Kingdom
India
Germany
France
Canada
Russia
Spain
Brazil
Australia
Argentina
Austria
Belgium
Bulgaria
Chile
Colombia
Cyprus
Czechia
Denmark
Ecuador
Egypt
Estonia
Finland
Greece
Hungary
Indonesia
Ireland
Italy
Japan
Latvia
Lithuania
Luxembourg
Malaysia
Malta
Mexico
Netherlands
New Zealand
Norway
Philippines
Poland
Portugal
Romania
Singapore
Slovakia
Slovenia
South Africa
South Korea
Sweden
Switzerland
Taiwan
Thailand
Turkey
Ukraine