Skip to content
kanven

For publishers

How Kanven reads the web, and how to say no.

Kanven reads public news, research and government documents so it can cite them in policy analysis. It identifies itself, follows your rules, and stops when you ask.

Who we are on your server

Every request carries this user agent:

Mozilla/5.0 (compatible; KanvenResearchBot/0.3; policy research)

To address Kanven by name in robots.txt, use KanvenResearchBot.

What we respect

  • robots.txt, read as the standard (RFC 9309) defines it, including wildcards. A group written for KanvenResearchBot replaces your general rules. If your robots.txt can’t be read because your server errors or doesn’t answer, we stay off the site until it can.
  • Crawl-delay: we wait at least as long as you ask between requests, and never less than a second.
  • “Don’t index” and “don’t mine” signals: an X-Robots-Tag header or a <meta name="robots"> tag containing noindex, none or noai; the tdm-reservation header or meta tag; and a TDM reservation in /.well-known/tdmrep.json (W3C TDMRep). Pages that carry any of these are not stored.
  • Public content only. We never log in, get around paywalls, solve captchas or disguise who we are. Sites that block automated clients stay blocked.

What we do with what we read

We keep the text so analyses can find and quote it, and we extract figures from it. Answers show short excerpts with your publication’s name and a link to the original. They never show full articles. We don’t use what we read to train AI models.

Government documents (laws, budgets, economic surveys, official reports) are read only from government websites, and are cited by page.

Opting out or asking for removal

To stop future visits, add a rule to robots.txt (for example User-agent: KanvenResearchBot followed by Disallow: /) or use any of the signals above. To also remove what we’ve already stored, email prashim.py@gmail.com with your domain. We delete it and add it to an exclusion list the crawler checks before every request.

Questions about how we cite or use sources: contact us.