Can I use robots.txt to optimize Googlebot's crawl?

Get the latest Flash Player to see this video.

* HD format might not be available

Play in: Low Quality | High Quality | HD Quality *

Local Comments (0)

Nothing Found!

Youtube Comments (16)

Jan 11, 2012
nationaloutlookmag Says:
You can always remove the already indexed pages too.

Dec 12, 2011
hellmind Says:
Why you dont stop answering question from that 5 year old ?, It's your cousin or something?

Nov 11, 2011
agapitoflores001 Says:
He is funny. I find it educational and entertaining as well. He lifted me up from boredom.

Nov 9, 2011
jasonmorgan12390 Says:
nice video thanks for sharing very interesting .. god bless

Aug 18, 2010
bigal21110 Says:
good note but I think that is OK. I think he meant doing it on weekly basis, for long period of time, could confuse the google bot. But technically, you should go live if you aren't ready

Aug 18, 2010
macfilas Says:
KINDA NOOOOOOOOOO VIDEOS fan club!!!

Aug 17, 2010
chiasma72 Says:
There were NO ROBOTS txting, as suggested by the title. You're a nice guy Matt, but I'm done with you professionally. :-)

Aug 17, 2010
mhaidarhanif Says:
Google need the brand new Googlebot technology. Just like the new search algorithm.

Aug 17, 2010
mhaidarhanif Says:
Go to the webmaster forum. If you're lucky, Matt will answer your question in a video.

Aug 17, 2010
Sander33333 Says:
How can i ask a question for a video?

Aug 16, 2010
iLovePalestineDotCom Says:
Something that Matt did not touch on is the fact that when you present Google with a new robots.txt file restricting certain area of your site from being indexed although that is not what you really wanted to accomplish. You are in essence telling Google, if you have already indexed certain areas REMOVE THEM because NOW they are restricted. I was so hoping that Matt touches on that but he didn't. So we do not know if this act damage the status of already indexed pages or not.

Aug 16, 2010
Robbertbiz Says:
Good answer man!

Aug 16, 2010
blopaaugusto Says:
Thanks for the tips Matt!

Aug 16, 2010
prisontv Says:
Don't scare googlebot, come on man :(

Aug 16, 2010
seosemarpit Says:
Awesome Matt you Rockkk!

Video Details

Can I use robots.txt to optimize Googlebot's crawl? For example, can I disallow all but one section of a site (for a week) to ensure it is crawled, and then revert to a 'normal' robots.txt? Blind Five Year Old, SF, CA More