Uncrawled URLs in search results

Get the latest Flash Player to see this video.

* HD format might not be available

Play in: Low Quality | High Quality | HD Quality *

Local Comments (0)

Nothing Found!

Youtube Comments (16)

Mar 20, 2012
SNiiP35x323 Says:
ahh fuck you

Jan 1, 2012
oceanofweb Says:
BUt what if we remove from URL removal tool and then since people will be linking to those specific urls ?

Nov 11, 2011
agapitoflores001 Says:
This one is very informative. I am learning a lot.

Oct 9, 2011
lemannequin Says:
@RickettsFish: Combining crawling with indexing / serving directives Robots meta tags and X-Robots-Tag headers are discovered when a URL is crawled. If a page is disallowed from crawling through the robots.txt file, then any information about indexing or serving directives will not be found and will therefore be ignored. If indexing or serving directives must be followed, the URLs containing those directives cannot be disallowed from crawling.

Oct 8, 2011
lemannequin Says:
@RickettsFish Then, this is my guess: if it's the case that many other websites link to your blocked website, Google may have to do some very basic crawling of your website, just to decide (via presence of "noindex" meta) if it should be listed or not on SERPs. But then, I'm just guessing.

Oct 8, 2011
lemannequin Says:
Good question, seems like a catch-22 situation. As I understand it, if you use robots.txt to block crawlers, your website won't get crawled. And it *may* not get indexed at all, unless other websites link to it. In that case, Google will show it in its index. So far, that's what Matt clearly explained in this video.

Aug 15, 2011
RickettsFish Says:
This was very helpful, but I have a question. If we use noindex, you made it sound like we need to allow Google to crawl the page by not blocking it via robots.txt. Otherwise, if we blocked it, Googlebot wouldn't read the noindex tag. Is that right?

Feb 15, 2011
mutchy126 Says:
Also robots.txt Sites dont have a Cached Version

Dec 27, 2010
chiangmaihotel Says:
Thanks for this tip.It still avaiable today?

Oct 31, 2009
mkarakas0690 Says:
that tip is so good for me.. thanx matt!.. :)

Oct 6, 2009
MichaelDadona Says:
I think better maintain that way, Matt. As many bloggers can 'ferry' that anchor texts like what you said. Especially, your example for NISSAN. Many small entrepreneurs (vendors) related to the industry able to get benefits from it. Like mine, Nissan Impul.

Oct 5, 2009
kevinargh Says:
Just make sure you remove the rule from robots.txt first or Google will never see the noindex meta tag on the page.

Oct 5, 2009
allison30dc Says:
Cool, that explains a lot, including what happened with those Google local listings that appeared to have been crawled that were robots.txt'd

Oct 5, 2009
danielgayle Says:
This was actually pretty interesting. I didn't know that the meta tag "noindex" would actually totally dump it from the index. Very cool.