Should I block duplicate pages using robots.txt?

Get the latest Flash Player to see this video.

* HD format might not be available

Play in: Low Quality | High Quality | HD Quality *

Local Comments (0)

Nothing Found!

Youtube Comments (22)

Nov 13, 2011
kristinedwards58 Says:
great clip keep it up =)

Nov 12, 2011
agapitoflores001 Says:
I think this has already been answered on previous videos. But anyway, like other videos, it helps a lot.

Feb 16, 2011
Webnauts Says:
Using 304 If-Modified-Since in combination of a meta robots directives "noindex,nosnippet,noarchive,follow" would be the best way to go. Everything else is simply BS.

Apr 3, 2010
krishthewiz Says:
It sounded like the answer at the end was that we should not block. I also think that depending on the case you have to do a combination of these techniques, meta-robots, robots.txt and canonical, especially if re-architect-ing the site is not an easy option (in some CMSes' its never an option). Tried the parameter filter that Google provided, and it doesn't work as fast as I wanted it to. leaving Google to identify the dupe increase the dupes cache count 10x and resulted in a rank fall.

Mar 18, 2010
ericofranco Says:
agreed. He should mention canonical as the best choice here.

Mar 13, 2010
palbertus Says:
"We can figure out the dups on our own". Looks like Google would prefer to crawl all your site and take the filtering job on their own !

Mar 11, 2010
dyoungprod Says:
Surprised canonical isn't mentioned as a solution here

Mar 11, 2010
iceveiled Says:
Actually web users have made google popular and the most used search engine, so if you want to point fingers, blame the collective world using the internet. I don't know about you but I don't want to go back to 1998 when search query results were filled with pages with ridiculous keyword spamming, hidden text, and the like. I'm not saying that people aren't gaming the google algorhytms as they are and forever will. But search has improved, thanks to google.

Mar 11, 2010
SEOMofo Says:
Matt, At the beginning of the video, it sounds like your answer is we SHOULD NOT block the URLs, because Google needs to crawl everything and figure out the duplicates for itself. But then at about 0:57 you seem to reverse your stance by saying we SHOULD block them. Can you please clarify? Thanks, SEOmofo

Mar 11, 2010
BIGELLOW Says:
You only have to jump through the hoops if you want Google to index your site and if you want to rank highly. If you aren't concerned about search engines or ranking of your site, then you can completely ignore the "hoops".

Mar 10, 2010
infiltrator7777 Says:
That still doesn't answer the question. There are no rules here. He's only making assumptions. Do you think you know?

Mar 10, 2010
bigal21110 Says:
so does it mean that if we have dups on our page, that doesnt hurt our ranking?

Mar 10, 2010
wicked4u2c Says:
They are not rules by "Google". They are standards for their Search Engine, just like Bing and others have theirs. At least Google has a great channel to help developers make sure their sites are optimized well for Google Search Engine. After all, Google is the most used Search Engine out there.

Mar 10, 2010
PropellerBusted Says:
Buddha knows many things about the Internet.

Mar 10, 2010
MsIleaneSpeaks Says:
I'm not a webmaster and after hearing this answer, I don't think most people that "think" they are webmaster really are either. LOL Thanks for answering a question that I'm sure many people have. @Ileane

Mar 10, 2010
jschroeffel Says:
No mention of canonical with this question? odd

Mar 10, 2010
infiltrator7777 Says:
Isn't it SAD how we have to jump through hoops now because of Google rules?? Who says Google makes the internet rules?

Mar 10, 2010
damianemanuel Says:
thx Matt, i love your videos.

Mar 10, 2010
carterblizz Says:
what about canonical link tag?

Mar 10, 2010
jazz0900 Says:
doesn't creating duplicate content in the first place cause you to get ranked down?

Mar 10, 2010
fferrero2008 Says:
..and leaves the train 4 hours... LOL Nice video, nice answer! Thanks!

Video Details

Halfdeck from Davis, CA asks: "If Google crawls 1,000 pages/day, Googlebot crawling many dupe content pages may slow down indexing of a large site. In that scenario, do you recommend blocking dupes using robots.txt or is using META ROBOTS NOINDEX,NOFOLLOW a better alternative?" Short answer: N... More