hurtbybot-bot crawler

Hurt by Bot uses an automated crawler that identifies itself as:

hurtbybot-bot/1.0 (+https://hurtbybot.com/bot)

• It fetches news articles, official press releases and public court records to find and verify reports of AI-related harm.
• It respects robots.txt (including rules for the hurtbybot-bot user agent) and waits several seconds between requests to the same site.
• It does not log in, does not bypass paywalls and does not collect personal data.
• We link to the original article and summarise it in our own words; we do not republish articles or images.
• We never fetch pages from publishers whose terms of use forbid automated access.
• News discovery also uses open data from the GDELT Project.

To stop the crawler, add this to your robots.txt:

User-agent: hurtbybot-bot Disallow: /