![]() ![]() This just happens to be part of what my site is about. You can also look up the domain name and IP address on the web and get an opinion. There are more giveaways, but they require more work. ![]() speed = one access : 2sec.)ĭoes the bot access javascript? No = bad. ![]() Here is what to look for:ĭoes the agent name contain a referring URL? No = Bad.ĭoes the agent name change over time? Yes = bad.ĭoes the bot access many pages very quickly? Yes = bad. Many of them are data miners and content scrapers that will use your content in some form or other to monetize their own site. #Awstats osx how toEach valid spider/bot will have as part of their agent name a URL that refers to how to block using robots.txt.Īny bad bot will, however, try and just plain spider your site anyway. tells you how to use the robots.txt file. Obviously there is Google, Bing, Yandex, Baidu, and so on, but there are valid smaller ones too that you will want to decide whether to allow or not.Įach one will have a method of blocking their accesses using robots.txt. Generally speaking, begin by research what search engines there are and what agents, domain names, and IP address blocks they use so that you become familiar. But how to know which ones are bad? That is though. Yes there are a bunch of bots out there and it seem like it can be a full time job of controlling their behavior. It will help you to better know what is going on. I would suggest that there are better site performance products out there that are free and I suggest that you install one of them. This is right up my ally, however, I use Sawmill and other tools and do not use AWStats. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |