Search icon Close icon
Search icon CANCEL
Subscription
0
Cart icon
Your Cart (0 item)
Close icon
You have no products in your basket yet
Save more on your purchases now! discount-offer-chevron-icon
Savings automatically calculated. No voucher code required.
Checkout

Change country

Modal Close icon
Country selected Country selected
country flag Germany
Country selected
country flag Slovakia
Country selected
country flag Canada
Country selected
country flag Brazil
Country selected
country flag Singapore
Country selected
country flag Hungary
Country selected
country flag Philippines
Country selected
country flag Mexico
Country selected
country flag Thailand
Country selected
country flag Ukraine
Country selected
country flag Luxembourg
Country selected
country flag Estonia
Country selected
country flag Lithuania
Country selected
country flag Norway
Country selected
country flag Chile
Country selected
country flag United States
Country selected
country flag Great Britain
Country selected
country flag India
Country selected
country flag Spain
Country selected
country flag South Korea
Country selected
country flag Ecuador
Country selected
country flag Colombia
Country selected
country flag Taiwan
Country selected
country flag Switzerland
Country selected
country flag Indonesia
Country selected
country flag Cyprus
Country selected
country flag Denmark
Country selected
country flag Finland
Country selected
country flag Poland
Country selected
country flag Malta
Country selected
country flag Czechia
Country selected
country flag New Zealand
Country selected
country flag Austria
Country selected
country flag Turkey
Country selected
country flag France
Country selected
country flag Sweden
Country selected
country flag Italy
Country selected
country flag Egypt
Country selected
country flag Belgium
Country selected
country flag Portugal
Country selected
country flag Slovenia
Country selected
country flag Ireland
Country selected
country flag Romania
Country selected
country flag Greece
Country selected
country flag Argentina
Country selected
country flag Malaysia
Country selected
country flag South Africa
Country selected
country flag Netherlands
Country selected
country flag Bulgaria
Country selected
country flag Latvia
Country selected
country flag Australia
Country selected
country flag Japan
Country selected
country flag Russia
Country selected
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Videos
Audiobooks
Learning Hub
Conferences
Free Learning
Arrow right icon
Arrow up icon
GO TO TOP
Hands-On Web Scraping with Python

You're reading from   Hands-On Web Scraping with Python Extract quality data from the web using effective Python techniques

Arrow left icon
Product type Paperback
Published in Oct 2023
Publisher Packt
ISBN-13 9781837636211
Length 324 pages
Edition 2nd Edition
Languages
Tools
Concepts
Arrow right icon
Author (1):
Arrow left icon
Anish Chapagain Anish Chapagain
Author Profile Icon Anish Chapagain
Anish Chapagain
LinkedIn
Anish Chapagain is a software engineer with a passion for data science, its processes, and Python programming, which began around 2007. He has been working with web scraping and analysis-related tasks for more than 5 years, and is currently pursuing freelance projects in the web scraping domain. Anish previously worked as a trainer, web/software developer, and as a banker, where he was exposed to data and gained further insights into topics including data analysis, visualization, data mining, information processing, and knowledge discovery. He has an MSc in computer systems from Bangor University (University of Wales), United Kingdom, and an Executive MBA from Himalayan Whitehouse International College, Kathmandu, Nepal.
Read more
See other products by Anish Chapagain
Arrow right icon
View More author details
Toc

Table of Contents (20) Chapters Close

Preface 1. Part 1:Python and Web Scraping
2. Chapter 1: Web Scraping Fundamentals FREE CHAPTER 3. Chapter 2: Python Programming for Data and Web 4. Part 2:Beginning Web Scraping
5. Chapter 3: Searching and Processing Web Documents 6. Chapter 4: Scraping Using PyQuery, a jQuery-Like Library for Python 7. Chapter 5: Scraping the Web with Scrapy and Beautiful Soup 8. Part 3:Advanced Scraping Concepts
9. Chapter 6: Working with the Secure Web 10. Chapter 7: Data Extraction Using Web APIs 11. Chapter 8: Using Selenium to Scrape the Web 12. Chapter 9: Using Regular Expressions and PDFs 13. Part 4:Advanced Data-Related Concepts
14. Chapter 10: Data Mining, Analysis, and Visualization 15. Chapter 11: Machine Learning and Web Scraping 16. Part 5:Conclusion
17. Chapter 12: After Scraping – Next Steps and Data Analysis 18. Index 19. Other Books You May Enjoy

Parsing robots.txt and sitemap.xml

In this section, we will introduce robots.txt- and sitemap.xml-related information and follow the instructions or resources available in those two files. We mentioned them in the Data-finding techniques used in web pages section of Chapter 1. In general, we can dive deep into the pages, or the directory with pages, of websites and find data or manage missing or hidden links using the robots.txt and sitemap.xml files.

The robots.txt file

The robots.txt file, or the Robots Exclusion Protocol, is a web-based standard or protocol used by websites to exchange information with automated scripts. robots.txt carries instructions regarding site-based links or resources to web robots (crawlers, spiders, web wanderers, or web bots), and uses directives such as Allow, Disallow, SiteMap, Crawl-delay, and User-agent to direct robots’ behavior.

We can find robots.txt by adding robots.txt to the main URL. For example, robots.txt for https://www.python...

lock icon The rest of the chapter is locked
arrow left Previous Section
Section 6 of 8
Next Section arrow right
Register for a free Packt account to unlock a world of extra content!
A free Packt account unlocks extra newsletters, articles, discounted offers, and much more. Start advancing your knowledge today.
Sign up now
Unlock this book and the full library FREE for 7 days
Get unlimited access to 7000+ expert-authored eBooks and videos courses covering every tech area you can think of
Start free trial
Renews at $19.99/month. Cancel anytime

Authors (1)

Profile icon Anish Chapagain
Anish Chapagain
LinkedIn icon Github icon
Anish Chapagain is a software engineer with a passion for data science, and artificial intelligence, its processes and Python programming, which began around 2007. He has been working with web scraping, data analysis, visualization and reporting-related tasks, projects for more than 10 years, and is also working as freelancer. Anish previously worked as a trainer, web/software developer, team leader, and as a banker, where he was exposed to data and gained further insights into topics like data mining, data analysis, reporting, information processing and knowledge discovery. He has an MSc in computer systems from Bangor University (United Kingdom), and an Executive MBA from Himalayan Whitehouse International College, Kathmandu, Nepal.
Read more
See other products by Anish Chapagain

Personalised recommendations for you

Based on your interests and search pattern
Left arrow icon
Modern Full-Stack React Projects
Modern Full-Stack React Projects
Read more
Full-Stack React Projects is a complete guide to learning full-stack web development, understanding the creation and integration of backend systems, and advancing your career as a frontend developer.
Read more
Jun 2024 16h 52m
Modern Full-Stack React Projects
Modern Full-Stack React Projects
Read more
Full-Stack React Projects is a complete guide to learning full-stack web development, understanding the creation and integration of backend systems, and advancing your career as a frontend developer.
Read more
Jun 2024 16h 52m
Modern Full-Stack React Projects
Modern Full-Stack React Projects
Read more
Full-Stack React Projects is a complete guide to learning full-stack web development, understanding the creation and integration of backend systems, and advancing your career as a frontend developer.
Read more
Jun 2024 16h 52m
Modern Full-Stack React Projects
Modern Full-Stack React Projects
Read more
Full-Stack React Projects is a complete guide to learning full-stack web development, understanding the creation and integration of backend systems, and advancing your career as a frontend developer.
Read more
Jun 2024 16h 52m
Modern Full-Stack React Projects
Modern Full-Stack React Projects
Read more
Full-Stack React Projects is a complete guide to learning full-stack web development, understanding the creation and integration of backend systems, and advancing your career as a frontend developer.
Read more
Jun 2024 16h 52m
Modern Full-Stack React Projects
Modern Full-Stack React Projects
Read more
Full-Stack React Projects is a complete guide to learning full-stack web development, understanding the creation and integration of backend systems, and advancing your career as a frontend developer.
Read more
Jun 2024 16h 52m
Modern Full-Stack React Projects
Modern Full-Stack React Projects
Read more
Full-Stack React Projects is a complete guide to learning full-stack web development, understanding the creation and integration of backend systems, and advancing your career as a frontend developer.
Read more
Jun 2024 16h 52m
Modern Full-Stack React Projects
Modern Full-Stack React Projects
Read more
Full-Stack React Projects is a complete guide to learning full-stack web development, understanding the creation and integration of backend systems, and advancing your career as a frontend developer.
Read more
Jun 2024 16h 52m
Mastering Node.js Web Development
Mastering Node.js Web Development
Read more
Explore Node.js with practical examples that will teach you how to utilize open-source packages for real-world solutions. Gain the skills to develop and deploy server-side applications that enhance your client-side projects.
Read more
Jun 2024 25h 56m
Mastering Node.js Web Development
Mastering Node.js Web Development
Read more
Explore Node.js with practical examples that will teach you how to utilize open-source packages for real-world solutions. Gain the skills to develop and deploy server-side applications that enhance your client-side projects.
Read more
Jun 2024 25h 56m
Mastering Node.js Web Development
Mastering Node.js Web Development
Read more
Explore Node.js with practical examples that will teach you how to utilize open-source packages for real-world solutions. Gain the skills to develop and deploy server-side applications that enhance your client-side projects.
Read more
Jun 2024 25h 56m
Mastering Node.js Web Development
Mastering Node.js Web Development
Read more
Explore Node.js with practical examples that will teach you how to utilize open-source packages for real-world solutions. Gain the skills to develop and deploy server-side applications that enhance your client-side projects.
Read more
Jun 2024 25h 56m
Right arrow icon