Bots and crawling
How CivCodex crawls the web, and how crawlers may use CivCodex.
How we crawl
Our verification worker fetches records from Wikidata and other listed sources with an identifying user agent (CivCodex-Verify) at a low rate and honours robots.txt. It never crawls the open web at large.
Crawling CivCodex
Search engines are welcome. robots.txt and sitemap.xml describe what is public; account pages and filtered views are excluded. Please fetch at a polite rate.
Structured data and API
Pages carry JSON-LD. A read API exists for our own site; bulk or commercial access to the data is not offered yet — ask via the About page.
AI training
CivCodex text is AI-assisted and source-backed. If you use it to train models, keep the evidence levels and source attributions with the text.