Robots txt prepared a little careless poison like minefield deep a
for the most simple example, if you want your a file in the b.html crawl, so how do you write is
Robots.txt open questions written
in Robots.txt written rules, it does not execute tree structure analysis, in other words when writing should not be put on the parent folder, but according to a nearby principle, if we put the parent folder on the front, the spider will think this way is closed, can not access, and our ultimate goal is to visit. As a result, the target and actual is very different.
every day we write original content, and then update to your web site, you want to have what we have to do? Of course is to discuss the search engine, we all know that search engine value of original content, for the original.
Robots.txt prepared by the
Allow:/a/b.html Disallow:/a/ or Disallow:/a/ Allow:/a/b.html?
general, the website does not need to access file management background files, scripts, attachments, database files, etc..
many webmaster, especially beginners webmaster for Robots.txt understanding is too one-sided, they think that since Robots.txt may provide access to the spider, why should we not open, all files are arranged to visit, so the site included the amount immediately rose, in fact, we think the problems are far from simple, we all know fixed some files in Web site is not transmitted to the search engine access, if we take the website "full open", the consequences is to increase web server load, reduce the rate of slow access speed, crawling spiders, no use for the website included, so the fixed do not need access to files, we directly Disallow off you can.
everyone has their own privacy, the website also is such, the site also don’t want to be the spider to see things, such as membership password and so on, and to block out this part of privacy, general webmaster will think of Robots.txt in the first time. Yes, the use of Robots.txt file can indeed give spider way, tell it what the place to go, what place should not go, it is also for this reason, many owners are keen to use the Robots.txt, but you really will use Robots.txt? For the preparation of the rules and you know how much I don’t? Only, but for this after all a little experience, to share out here, so we have a reference, of course, the old bird can fly.
prepared by the repeated problems ?