160 commits

Author	SHA1	Message	Date
ai.robots.txt	5326c202b5	Merge pull request #154 from paulrudy/main ... re-add facebookexternalhit	2025-06-16 15:12:42 +00:00
ai.robots.txt	eb05f2f527	Merge pull request #153 from sergiospagnuolo/Poseidon ... Update robots.json with new crawler	2025-06-14 14:04:03 +00:00
ai.robots.txt	e53d81c66d	Merge pull request #152 from ai-robots-txt/MyCentralAIScraperBot ... chore(robots.json): adds MyCentralAIScraperBot	2025-06-13 09:28:41 +00:00
ai.robots.txt	842e2256e8	Merge pull request #150 from ai-robots-txt/semrush-bots ... chore(robots.json): adds additional SemrushBot user agents	2025-06-12 07:12:00 +00:00
ai.robots.txt	3759a6bf14	chore(robots.json): adds EchoboxBot (#148)	2025-06-09 15:44:36 +00:00
ai.robots.txt	ac7ed17e71	Merge pull request #145 from ai-robots-txt/aws-bedrockbot ... chore(robots.json): adds bedrockbot	2025-06-05 16:51:17 +00:00
ai.robots.txt	75ea75a95b	Merge pull request #143 from ai-robots-txt/panscient ... chore(robots.json): adds Panscient	2025-06-04 18:04:06 +00:00
ai.robots.txt	9c28c63a0c	Merge pull request #142 from ai-robots-txt/quillbot ... chore(robots.json): adds Quillbot	2025-06-04 17:54:57 +00:00
ai.robots.txt	3efabc603d	Merge pull request #141 from Ivan-Chupin/patch-1 ... Add SBIntuitionsBot	2025-06-03 23:28:48 +00:00
ai.robots.txt	080946c360	Merge pull request #140 from ai-robots-txt/yandex-bots ... chore(robots.json): adds YandexAdditional crawlers	2025-06-03 19:51:25 +00:00
ai.robots.txt	d239e7e5ad	Merge pull request #139 from ai-robots-txt/workflow-fix ... chore(ai_robots_update.yml): correct workflow by revising git flags + adding guard	2025-06-03 01:52:35 +00:00
ai.robots.txt	093ab81d78	Update from Dark Visitors	2025-05-23 00:58:57 +00:00
ai.robots.txt	fedb658cc0	Merge pull request #133 from ai-robots-txt/wpbot ... chore(robots.json): adds wpbot	2025-05-21 21:06:05 +00:00
ai.robots.txt	7c5389f4a0	Merge pull request #98 from kylebuckingham/main ... Updating Claude Bots	2025-05-21 19:00:23 +00:00
ai.robots.txt	1c2acd75b7	Merge pull request #126 from ai-robots-txt/mistral-bot ... chore(robots.json): adds MistralAI-User/1.0 crawler	2025-05-21 15:27:26 +00:00
ai.robots.txt	dd1ed174b7	Merge pull request #129 from ai-robots-txt/google-cloudvertexbot ... chore(robots.json): adds Google-CloudVertexBot	2025-05-16 11:35:15 +00:00
ai.robots.txt	1c470babbe	Merge pull request #123 from joehoyle/patch-1 ... Fix JSON syntax error	2025-05-15 16:12:30 +00:00
ai.robots.txt	c6c7f1748f	Update from Dark Visitors	2025-04-26 00:55:12 +00:00
ai.robots.txt	4654e14e9c	Merge pull request #112 from maiavixen/main ... Fixed meta-external* being titlecase, and removed period for consistency	2025-04-24 07:00:34 +00:00
ai.robots.txt	bbec639c14	Merge pull request #109 from dennislee1/patch-1 ... AI bots to consider adding	2025-04-22 14:50:26 +00:00
ai.robots.txt	e0cdb278fb	Update from Dark Visitors	2025-04-16 00:57:11 +00:00
ai.robots.txt	4a764bba18	Merge pull request #102 from ai-robots-txt/imgproxy-bot ... chore(robots.json): adds imgproxy crawler	2025-04-10 19:22:34 +00:00
Frederic Barthelemy	5f5a89c38c	Fix html-mangled hyphen in Perplexity-Users ... Fixes: #99	2025-04-04 17:34:14 -07:00
ai.robots.txt	5b8650b99b	Update from Dark Visitors	2025-03-29 00:54:10 +00:00
Cory Dransfeldt	ec18af7624	Revert "Merge pull request #91 from deyigifts/perplexity-user" ... This reverts commit 68d1d93714.	2025-03-27 12:51:22 -07:00
ai.robots.txt	68d1d93714	Merge pull request #91 from deyigifts/perplexity-user ... Update perplexity bots	2025-03-27 19:29:30 +00:00
Dennis Camera	0bd3fa63b8	table-of-bot-metrics.md: Escape robot names for Markdown table ... Some characters which could occur in a crawler's name have a special meaning in Markdown. They are escaped to prevent them from having unintended side effects. The escaping is only applied to the first (Name) column of the table. The rest of the columns is expected to already be Markdown encoded in robots.json.	2025-02-18 12:53:27 +01:00
ai.robots.txt	693289bb29	chore: add Brightbot 1.0	2025-02-16 21:37:52 +00:00
ai.robots.txt	89d4c6e5ca	Merge pull request #73 from nisbet-hubbard/patch-8 ... Actually block Semrush’s AI tools	2025-02-01 10:51:01 +00:00
ai.robots.txt	6c552a3daa	Merge pull request #71 from jsheard/patch-1 ... Add Crawlspace	2025-01-20 17:45:42 +00:00
ai.robots.txt	ec454b71d3	Merge pull request #67 from Nightfirecat/semrushbot ... Block SemrushBot	2025-01-06 20:51:56 +00:00
ai.robots.txt	ca8620e28b	Merge pull request #63 from glyn/push-paths ... Convert robots.json more frequently	2025-01-05 05:05:20 +00:00
dark-visitors	609ddca392	Updated from new robots.json	2024-11-24 00:57:06 +00:00
dark-visitors	58985737e7	Updated from new robots.json	2024-11-19 16:46:21 +00:00
dark-visitors	9e06cf3bc9	Updated from new robots.json	2024-10-29 00:52:12 +00:00
fabianegli	0c05461f84	simplify repo and added some tests	2024-10-19 13:06:34 +02:00
ai.robots.txt	6a359e7fd7	Fix typo and trigger rerun of main job	2024-10-19 03:43:00 +00:00
ai.robots.txt	a80bd18fb8	Dump out file contents in PHP script	2024-10-19 03:34:29 +00:00
ai.robots.txt	9be286626d	Merge pull request #43 from lxjv/main ... Update robots.json with Claude respect link	2024-10-08 02:30:17 +00:00
ai.robots.txt	6da804e826	chore: add ISSCyberRiskCrawler	2024-09-30 23:50:18 +00:00
ai.robots.txt	6d9ce1d62a	chore: add sidetrade bot	2024-09-28 20:58:18 +00:00
ai.robots.txt	632e9d6510	Daily update from Dark Visitors	2024-09-28 01:17:19 +00:00
ai.robots.txt	44d975c799	Merge pull request #42 from commoncrawl/main ... feat: make CCBot entry more accurate	2024-09-27 00:21:49 +00:00
ai.robots.txt	6b8d7f5890	Daily update from Dark Visitors	2024-09-09 01:16:21 +00:00
ai.robots.txt	1c1b423684	chore: add iaskspider/2.0	2024-09-07 02:05:43 +00:00
ai.robots.txt	fb5c995243	Daily update from Dark Visitors	2024-09-03 01:12:57 +00:00
ai.robots.txt	7151f6c569	Removing previously generated files	2024-09-03 01:12:56 +00:00
ai.robots.txt	c9325c9e18	Daily update from Dark Visitors	2024-09-02 01:15:07 +00:00
ai.robots.txt	567bd00aec	Removing previously generated files	2024-09-02 01:15:07 +00:00
ai.robots.txt	543e993b08	Daily update from Dark Visitors	2024-09-01 01:24:53 +00:00