Learn how Google Sees Your Site - Spider it Yourself
If your are just getting into SEO, or are wondering why your site isn’t showing up in Google results the way you want it, start by trying to understand how Googlebot sees your site. An excellent way to do this is by using your own robot, or a link-checker, to spider your site. I’ve always used Xenu’s Link Sleuth to spider my site. Not only can you find broken or dead links, but you can get some insight into how search engines crawl your site.
This is especially important for dynamic sites. Sites that append session IDs or other session or user-specific data to URLs, or depend on cookies or (ack) javascript for navigation will either not spider at all, or cause duplicate page entries for the same content. Recently, Google has gotten better at handling PHPSESSID, but it’s a good idea to try to remove any extraneous session information until a session is necessary.
Spidering your site will give you a lot of information about how search engines see your site. It’s up to you to interpret those results and work to make sure that what they see reflects the true architecture of your site.
