AddonIndexBot
The crawler of the Addon Index. It looks for new information about add-on aircraft for Microsoft Flight Simulator on public websites and suggests changes to the directory. A human reviews every change before it appears on the site.
The bot is being set up and does not fetch any pages regularly yet.
Identification
User agent: AddonIndexBot/1.0 (+https://www.addonindex.com/bot; sievers@strkt.de)
What it reads
- forums.flightsimulator.com · Sitemap of new topics
- flightsim.to · Sitemap of new and updated entries
Further sites are only added after their robots.txt and terms of use have been checked.
Rules it follows
- Reads robots.txt before every request and obeys it, including Crawl-delay.
- At most one request per second per host, one request at a time, at most 200 per site and run.
- Mainly reads sitemaps and feeds. Individual pages only when the title looks like an aircraft add-on.
- Never downloads images, logos or files and never stores article text.
- Only facts such as status, date and price are stored, plus a short quote as evidence, the link and the date.
- Honors reservations against AI and text and data mining (Content-Signal, tdm-reservation). In that case only title, address and date are recorded.
- Never circumvents blocks: on 403, 429 or a Cloudflare challenge it stops, and after three failures in a row it leaves the site alone entirely.
- Never stores names of private individuals from forum posts.
Opting out
One entry in robots.txt is enough; it applies from the next run:
User-agent: AddonIndexBot Disallow: /
Contact
Questions, complaints or the wish not to be read anymore: sievers@strkt.de