If I don't need to block crawlers, should I create a...

Get the latest Flash Player to see this video.

* HD format might not be available

Play in: Low Quality | High Quality | HD Quality *

Local Comments (0)

Nothing Found!

Youtube Comments (6)

Mar 8, 2012
adrianTNT Says:
One of the main reason for pagerank rare updates is that Google would not like you to instantly see what increases and decreases page rank, you would be able to manipulate rankings more easily if you instantly see the results of your seo tuning.

Aug 27, 2011
LinuxForHumans Says:
I have had blank robots.txt files for a while now, simply because my logs get super full of 404 not found errors for robots.txt when it doesn't exist. With this, though, I'll just add that extra bit of "confirmation" to make sure all my content gets crawled.

Aug 27, 2011
david88at Says:
When are you going to stop giving out this 1st grade advice? Why don't you come out and just tell it like it is? SEO is an art of mass manipulation and crap-link racketeering; and you are constantly shuffling things around to try and negate the barrage of garbage people throw your way. No one who is worth their spam-based pay checks pays any attention to Matt Cutts.

Aug 27, 2011
ireshsl Says:
You need a robots.txt file only if your site includes content that you don't want search engines to index. If you want search engines to index everything in your site, you don't need a robots.txt file (not even an empty one). Source: Google Webmaster Tools Help Help articles › Using Webmaster Tools › Site configuration › Block or remove pages using a robots.txt file I am confused now :)

Video Details

Is it better to have a blank robots.txt file, a robots.txt that contains User-agent: * Disallow:" or no robots.txt file at all?" Corey S, Pennsylvania Have a question? Ask it in our Webmaster Help Forum: http://www.google.com/support/forum/p/Webmast
ers?hl=en Want your question to be answere... More