cdransf/ai.robots.txt
cdransf/ai.robots.txt
1
1
Fork 0
mirror of https://github.com/ai-robots-txt/ai.robots.txt.git synced 2025-10-09 19:43:32 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions1
475 commits 3 branches 43 tags 2 MiB
Commit graph

475 commits

Author SHA1 Message Date
dark-visitors
2b68568ac2 Update from Dark Visitors 2025-06-14 00:58:11 +00:00
ai.robots.txt
e53d81c66d Merge pull request from ai-robots-txt/MyCentralAIScraperBot 
chore(robots.json): adds MyCentralAIScraperBot
 2025-06-13 09:28:41 +00:00
Glyn Normington
20e327e74e
 		Merge pull request from ai-robots-txt/MyCentralAIScraperBot 
chore(robots.json): adds MyCentralAIScraperBot
 2025-06-13 10:28:32 +01:00
Glyn Normington
8f17718e76
 		Fix typo 2025-06-13 10:28:12 +01:00
Cory Dransfeldt
d760f9216f
 		chore(robots.json): adds MyCentralAIScraperBot 2025-06-12 13:08:29 -07:00
ai.robots.txt
842e2256e8 Merge pull request from ai-robots-txt/semrush-bots 
chore(robots.json): adds additional SemrushBot user agents
 2025-06-12 07:12:00 +00:00
Glyn Normington
229ea20426
 		Merge pull request from ai-robots-txt/semrush-bots 
chore(robots.json): adds additional SemrushBot user agents
 2025-06-12 08:11:51 +01:00
Cory Dransfeldt
14d68f05ba
 		chore(robots.json): adds additional SemrushBot user agents 2025-06-11 13:50:53 -07:00
dark-visitors
cf598b6b71 Update from Dark Visitors 2025-06-10 01:00:37 +00:00
ai.robots.txt
3759a6bf14 chore(robots.json): adds EchoboxBot () 2025-06-09 15:44:36 +00:00
Cory Dransfeldt
7867c3e26c
 		chore(robots.json): adds EchoboxBot () 2025-06-09 16:44:25 +01:00
dark-visitors
e21f6ae1b6 Update from Dark Visitors 2025-06-06 00:59:25 +00:00
ai.robots.txt
ac7ed17e71 Merge pull request from ai-robots-txt/aws-bedrockbot 
chore(robots.json): adds bedrockbot
 2025-06-05 16:51:17 +00:00
Glyn Normington
81747e6772
 		Merge pull request from ai-robots-txt/aws-bedrockbot 
chore(robots.json): adds bedrockbot
 2025-06-05 17:51:03 +01:00
Cory Dransfeldt
528d77bf07
 		chore(robots.json): adds bedrockbot 2025-06-05 09:14:23 -07:00
dark-visitors
77393df5aa Update from Dark Visitors 2025-06-05 00:59:28 +00:00
ai.robots.txt
75ea75a95b Merge pull request from ai-robots-txt/panscient 
chore(robots.json): adds Panscient
 2025-06-04 18:04:06 +00:00
Glyn Normington
2fca1ddcf1
 		Merge pull request from ai-robots-txt/panscient 
chore(robots.json): adds Panscient
 2025-06-04 19:03:53 +01:00
ai.robots.txt
9c28c63a0c Merge pull request from ai-robots-txt/quillbot 
chore(robots.json): adds Quillbot
 2025-06-04 17:54:57 +00:00
Cory Dransfeldt
395c013eea
 		Merge pull request from ai-robots-txt/quillbot 
chore(robots.json): adds Quillbot
 2025-06-04 10:54:46 -07:00
Cory Dransfeldt
4568d69b0e
 		chore(robots.json): adds Panscient 2025-06-04 10:54:14 -07:00
Cory Dransfeldt
03831a7eb5
 		chore(robots.json): adds Quillbot 2025-06-04 10:46:58 -07:00
dark-visitors
2b5a59a303 Update from Dark Visitors 2025-06-04 01:00:07 +00:00
ai.robots.txt
3efabc603d Merge pull request from Ivan-Chupin/patch-1 
Add SBIntuitionsBot
 2025-06-03 23:28:48 +00:00
Cory Dransfeldt
b35f9a31d7
 		Merge pull request from Ivan-Chupin/patch-1 
Add SBIntuitionsBot
 2025-06-03 16:28:36 -07:00
Ivan Chupin
8f75f4a2f5
 		Add SBIntuitionsBot 2025-06-04 03:48:42 +05:00
ai.robots.txt
080946c360 Merge pull request from ai-robots-txt/yandex-bots 
chore(robots.json): adds YandexAdditional crawlers
 2025-06-03 19:51:25 +00:00
Glyn Normington
7eec033cad
 		Merge pull request from ai-robots-txt/yandex-bots 
chore(robots.json): adds YandexAdditional crawlers
 2025-06-03 20:51:14 +01:00
Cory Dransfeldt
3187fd8a32
 		chore(robots.json): adds YandexAdditional crawlers 2025-06-03 12:41:57 -07:00
ai.robots.txt
d239e7e5ad Merge pull request from ai-robots-txt/workflow-fix 
chore(ai_robots_update.yml): correct workflow by revising git flags + adding guard
 2025-06-03 01:52:35 +00:00
Glyn Normington
9dbf34010a
 		Merge pull request from ai-robots-txt/workflow-fix 
chore(ai_robots_update.yml): correct workflow by revising git flags + adding guard
 2025-06-03 02:52:23 +01:00
dark-visitors
87016d1504 Update from Dark Visitors 2025-06-03 01:00:29 +00:00
Cory Dransfeldt
899ce01c55
 		chore(ai_robots_update.yml): correct workflow by revising git flags + adding guard 2025-06-02 14:56:09 -07:00
Glyn Normington
4af776f0a0
 		Merge pull request from ai-robots-txt/imgproxy-revert 
chore(robots.json): revert "adds imgproxy crawler"
 2025-06-02 20:21:10 +01:00
Cory Dransfeldt
1dd66b6969
 		Revert "chore(robots.json): adds imgproxy crawler" 
This reverts commit b65f45e408.
 2025-06-02 11:53:06 -07:00
Cory Dransfeldt
814df6b9a0
 		Merge pull request from not-not-the-imp/patch-1 
Add AndiBot and PhindBot
 2025-05-31 16:03:16 -07:00
Cory Dransfeldt
268922f8f2
 		Update robots.json 2025-05-31 16:02:05 -07:00
Cory Dransfeldt
4259b25ccc
 		Update robots.json 2025-05-31 16:01:09 -07:00
Cory Dransfeldt
d22b9ec51a
 		Update robots.json 2025-05-31 16:00:13 -07:00
imp
3e8edd083e
 		Add AndiBot and PhindBot 
Fixes
2025-05-23 13:03:49 +01:00
ai.robots.txt
093ab81d78 Update from Dark Visitors 2025-05-23 00:58:57 +00:00
dark-visitors
7bf7f9164d Update from Dark Visitors 2025-05-22 00:58:45 +00:00
ai.robots.txt
fedb658cc0 Merge pull request from ai-robots-txt/wpbot 
chore(robots.json): adds wpbot
 2025-05-21 21:06:05 +00:00
Glyn Normington
851eabe059
 		Merge pull request from ai-robots-txt/wpbot 
chore(robots.json): adds wpbot
 2025-05-21 22:05:51 +01:00
ai.robots.txt
7c5389f4a0 Merge pull request from kylebuckingham/main 
Updating Claude Bots
 2025-05-21 19:00:23 +00:00
Cory Dransfeldt
af597586b6
 		Merge pull request from kylebuckingham/main 
Updating Claude Bots
 2025-05-21 12:00:11 -07:00
Cory Dransfeldt
b1d9a60a38
 		chore(robots.json): adds wpbot 2025-05-21 11:40:33 -07:00
ai.robots.txt
1c2acd75b7 Merge pull request from ai-robots-txt/mistral-bot 
chore(robots.json): adds MistralAI-User/1.0 crawler
 2025-05-21 15:27:26 +00:00
Glyn Normington
202d3c3b9a
 		Merge pull request from ai-robots-txt/mistral-bot 
chore(robots.json): adds MistralAI-User/1.0 crawler
 2025-05-21 16:27:14 +01:00
Glyn Normington
0a78fe1e76
 		Merge pull request from ai-robots-txt/crawler-policy-update 
chore(README): updates the opening line of our README to clarify the types of agents we block
 2025-05-21 15:13:35 +01:00