Merge pull request #105 from jsheard/patch-1

Include "AI Agents" from Dark Visitors
2025-06-19 10:00:52 +00:00 · 2025-04-14 10:08:38 -07:00 · 2025-04-14 15:46:01 +01:00
1 changed files with 1 additions and 0 deletions
--- a/code/robots.py
+++ b/code/robots.py
@ -30,6 +30,7 @@ def updated_robots_json(soup):
    """Update AI scraper information with data from darkvisitors."""
    existing_content = load_robots_json()
    to_include = [
+        "AI Agents",
        "AI Assistants",
        "AI Data Scrapers",
        "AI Search Crawlers",
Author	SHA1	Message	Date
Cory Dransfeldt	156e6baa09	Merge pull request #105 from jsheard/patch-1 Some checks are pending / run-tests (push) Waiting to run Details Include "AI Agents" from Dark Visitors	2025-04-14 10:08:38 -07:00
Joshua Sheard	d9f882a9b2	Include "AI Agents" from Dark Visitors	2025-04-14 15:46:01 +01:00