Search icon CANCEL
Arrow left icon
Explore Products
Best Sellers
New Releases
Books
Videos
Audiobooks
Learning Hub
Conferences
Free Learning
Arrow right icon
Arrow up icon
GO TO TOP
WordPress Search Engine Optimization- Second Edition

You're reading from   WordPress Search Engine Optimization- Second Edition A complete guide to dominating search engines with your WordPress site

Arrow left icon
Product type Paperback
Published in Oct 2015
Publisher
ISBN-13 9781785887642
Length 318 pages
Edition 1st Edition
Concepts
Arrow right icon
Toc

Table of Contents (14) Chapters Close

Preface 1. Getting Started – SEO Basics FREE CHAPTER 2. Optimizing WordPress for SEO 3. Researching and Working with Keywords 4. Understanding Technical Optimization 5. Creating Optimized and Engaging Content 6. Link Building 7. Using Social Media 8. Avoiding Black Hat Techniques 9. Avoiding SEO Mistakes 10. Testing Your Site and Monitoring Your Progress A. WordPress SEO Plugins B. Other SEO Resources Index

The ultimate WordPress robots.txt file


We learned in Chapter 2, Customizing WordPress for SEO, that WordPress generates archive, tag, comment, and category pages that raise duplicate content issues. We can signal to search engines to ignore these duplicate content pages with a robots.txt file. In this section, we'll kill a few birds with one ultimate robots.txt file. We'll tell search engines to ignore our duplicated pages. We'll go further: we'll instruct search engines not to index our admin area and not to index non-essential folders on our server. As an option, we can also ask bad bots not to index any pages on our site, though they tend to usually do as they wish.

Before we customize our robots.txt file, bear in mind that the WordPress SEO by Yoast plugin can help us accomplish the same goal, but not as reliably. Furthermore, robots.txt is very handy for all sorts of purposes, so it's a great idea to know how to leverage it.

You can create a robots.txt file in any text editor. Place the...

lock icon The rest of the chapter is locked
Register for a free Packt account to unlock a world of extra content!
A free Packt account unlocks extra newsletters, articles, discounted offers, and much more. Start advancing your knowledge today.
Unlock this book and the full library FREE for 7 days
Get unlimited access to 7000+ expert-authored eBooks and videos courses covering every tech area you can think of
Renews at €18.99/month. Cancel anytime