Welcome to Hedir community. Hedir is a community helping to rank the sites on the basis of real customer feedback. Ranking is not based on automated checking of site properties but on the basis of experience with the site/business as a whole. Explore more about Hedir.
Posted: Fri Dec 02, 2005 5:48 am Post subject: robots.txt
Is it essential to have a robots.txt file for each and every site ?
If yes, then is it true that it is used only to prevent specific spiders not to crawl those pages which we want to hide and in absence of robots.txt file SE will fill all the error log files?
What harm it may bring incase of filling error log files?
Posted: Fri Dec 02, 2005 8:36 am Post subject: Robots.txt just to make order Latest Blog Post : http://acorus.hedir.com
It is recommended to use it. What is the meaning of word essential? Is validity or optimization essential. I guess it is not essential, just recommended.
As You say it is standard robots exclusion protocol, so it is used to exclude directories You do not want to index. IMO this is the way to make it easier for algo to index only Your content. I use it on every site, I treat it as friendly advice to robots. Not necessary but helpful. _________________ Feel invited to my blog http://www.acorus.hedir.com/
Today when I was searching for the robots.txt file for one reputed site I found initially it was showing file not found message but soon the page redirected to its home page. What kind of SEO techniques they have applied and why ?
Then even in absence of this file, search engine will still crawl site pages but may leave pages without indexing and behavior patterns for reading a particular site page is different for different spiders right ?
Posted: Sat Dec 03, 2005 6:33 pm Post subject: Latest Blog Post : http://norah.hedir.com
Quote:
in absence of robots.txt file SE will fill all the error log files?
Can you please explain it.
It is always good to have a robots.txt as some pages adds to duplicate content but are important from customer's point of view. SEs are not intelligent enough and robots.txt comes handy to pump intelligence.
Quote:
Today when I was searching for the robots.txt file for one reputed site I found initially it was showing file not found message but soon the page redirected to its home page.
This is no SEO trick, IMO they have designed their customized (404) error page which redirects to homepage. Try searching for thatwebsite.com/blahblah.html and it will follow the same path as of robots.txt.
in absence of robots.txt file SE will fill all the error log files?
Can you please explain it.
We know the following general explanation of error log files :
Whenever server encounters any errors during processing request it will store this information in a file which is error log file, the error log directive generates name and location for this file. Whenever server faces problem during its operation administrators are suppose to check this file first to know about the detail about the errors occurred initially and to decide accordingly....we know a typical error log file is written in error log on UNIX systems and in error.log on WINDOWS system.
Now my question is in absence of robots.txt file will these file be crawled by SE and will these error files be populated with the fresh error logs?
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You can vote in polls in this forum You cannot attach files in this forum You cannot download files in this forum
Search
Register
Login