Surveying the webscape

Since web vulnerabilities are so tied to the site code and its relative security, we are going to start with surveying the landscape of web insecurity and the three top exploit classes. Classes of attacks include many specific exploits and, generally, cannot be completely solved by changing the .htaccess file.

Concept of Robots.txt

You can use the .htaccess file to block access to some of the site directories, in a similar way to how you can use the robots.txt file to request that robots ignore or do not index some directories. We use wget robots.txt htaccess at the very beginning to see what the site owners are hiding from searchengine spiders and to find out where the rewrites are going. If we know there is a wp-admin folder, we can know to dig in there immediately. We can also look for the paid content stored directly on the server. In the following robots.txt file, the unixtux folder might hold paid content that an evil hacker could sell. The following is the content...