Many consider AI bots as very bad because they ignore established guidelines when crawling a given site. There is a well known and respected way sites can tell good bots how to behave, called robots.txt. Any website can create one and expect good bots (Google, Bing, Facebook, etc) to honor it. For example https://lemmy.world/robots.txt tells bots what paths it should NOT crawl, and the maximum speed at which it should crawl the site.
Many AI bots have been seen completely disregarding this established standard.
Does AI scrapers and agents count as bae, or good bots ?
Many consider AI bots as very bad because they ignore established guidelines when crawling a given site. There is a well known and respected way sites can tell good bots how to behave, called robots.txt. Any website can create one and expect good bots (Google, Bing, Facebook, etc) to honor it. For example https://lemmy.world/robots.txt tells bots what paths it should NOT crawl, and the maximum speed at which it should crawl the site.
Many AI bots have been seen completely disregarding this established standard.