Where Should I Upload robots.txt file in CakePHP installation & Why I Thanked Google Webmaster Tool Once Again

For those rather newbies (ME too!), robots.txt in a text file, which restricts access to specific folders or files in your web page. As such you create some rules using that file. Well behaved robots (like GoogleBOT) will follow these instructions before accessing any location.

We upload that file usually under /public_html/ folder of our web hosting directory, technically called root of our web servers. CakePHP has its own folder architecture, and, if you follow the default CakePHP installation, you should upload your robots.txt file under:

http://your-lovely-domain-name.com/app/webroot/robots.txt 


As such, I have subscribed to free Google Webmaster Tool. I found a notice long ago indicating my robots.txt file was not well formatted. I simply ignored it. Because, I checked that file manually many times and found nothing wrong.

Recently, Webmaster tool was displaying bunch of urls restricted by robots.txt file. I examined it and found the fault was in upload location itself. Issue was resolved finally.

Lesson today?

ALWAYS DOUBLE CHECK EVERYTHING WHEN YOU ARE GOING WITH CAKEPHP

And a big thanks to Google Webmaster Tool for pointing out to the urls restricted by robots.txt and finally, helping me to get into the problem location.

Thanks.