Google, MSN, & Yahoo spiders, or crawlers, follow links on web pages.
If there are no links to a page, it should not be crawled by a search engine. Spiders generally don't look at a url & guess what another link should be. That is usually indicative of something a little more nefarious.
You may have links that used to exist, but do not now. These could be from removed the content; or from dynamically generated your urls & the information used to generate those pages no longer exists.In this case, the links will still be in the search engines index & polled every so often. This is not a bad thing, as the link will eventually fall out of the search engines index. You can request removal through the webmaster admin tools if this bugs you.
Yahoo's spider deliberately generates a dummy url to generate a "404 - page not found" error to understand how your site handles this. Don't be too worried that Yahoo is spidering weird links for you site.
If you feel you need to, you can trap these requests in your "page not found" web site code.
user_agent = request.user_agent.downcase
if [ 'msnbot', 'yahoo! slurp','googlebot' ].detect { |b| user_agent.include? b }
# This is a request from a search engine ..do something here..
end
Trackback URL