cdransf/ai.robots.txt
1
1
Fork 0
mirror of https://github.com/ai-robots-txt/ai.robots.txt.git synced 2025-06-19 18:10:52 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 1
480 commits 3 branches 38 tags 1.4 MiB
Commit graph

88 commits

Author SHA1 Message Date
paulrudy
7535893aec re-add facebookexternalhit 2025-06-15 16:49:07 -07:00
Cory Dransfeldt
26a46c409d
Merge pull request #153 from sergiospagnuolo/Poseidon 
Update robots.json with new crawler
 2025-06-14 07:03:52 -07:00
dark-visitors
2b68568ac2 Update from Dark Visitors
Some checks are pending
/ ai-robots-txt (push) Waiting to run
Details
/ run-tests (push) Waiting to run
Details
/ lint-json (push) Waiting to run
Details
2025-06-14 00:58:11 +00:00
Sérgio Spagnuolo
b05f2fee00
Update robots.json with new crawler 
Update with Poseidon Research Crawler as found in nytimes.com/robots.txt
 2025-06-13 17:15:13 -03:00
Glyn Normington
20e327e74e
Merge pull request #152 from ai-robots-txt/MyCentralAIScraperBot 
chore(robots.json): adds MyCentralAIScraperBot
 2025-06-13 10:28:32 +01:00
Glyn Normington
8f17718e76
Fix typo 2025-06-13 10:28:12 +01:00
Cory Dransfeldt
d760f9216f
chore(robots.json): adds MyCentralAIScraperBot 2025-06-12 13:08:29 -07:00
Cory Dransfeldt
14d68f05ba
chore(robots.json): adds additional SemrushBot user agents 2025-06-11 13:50:53 -07:00
dark-visitors
cf598b6b71 Update from Dark Visitors
Some checks failed
/ ai-robots-txt (push) Has been cancelled
Details
/ run-tests (push) Has been cancelled
Details
/ lint-json (push) Has been cancelled
Details
2025-06-10 01:00:37 +00:00
Cory Dransfeldt
7867c3e26c
chore(robots.json): adds EchoboxBot (#148) 2025-06-09 16:44:25 +01:00
dark-visitors
e21f6ae1b6 Update from Dark Visitors
Some checks failed
/ ai-robots-txt (push) Has been cancelled
Details
/ run-tests (push) Has been cancelled
Details
/ lint-json (push) Has been cancelled
Details
2025-06-06 00:59:25 +00:00
Glyn Normington
81747e6772
Merge pull request #145 from ai-robots-txt/aws-bedrockbot 
chore(robots.json): adds bedrockbot
 2025-06-05 17:51:03 +01:00
Cory Dransfeldt
528d77bf07
chore(robots.json): adds bedrockbot 2025-06-05 09:14:23 -07:00
dark-visitors
77393df5aa Update from Dark Visitors
Some checks are pending
/ ai-robots-txt (push) Waiting to run
Details
/ run-tests (push) Waiting to run
Details
/ lint-json (push) Waiting to run
Details
2025-06-05 00:59:28 +00:00
Glyn Normington
2fca1ddcf1
Merge pull request #143 from ai-robots-txt/panscient 
chore(robots.json): adds Panscient
 2025-06-04 19:03:53 +01:00
Cory Dransfeldt
4568d69b0e
chore(robots.json): adds Panscient 2025-06-04 10:54:14 -07:00
Cory Dransfeldt
03831a7eb5
chore(robots.json): adds Quillbot 2025-06-04 10:46:58 -07:00
dark-visitors
2b5a59a303 Update from Dark Visitors
Some checks are pending
/ ai-robots-txt (push) Waiting to run
Details
/ run-tests (push) Waiting to run
Details
/ lint-json (push) Waiting to run
Details
2025-06-04 01:00:07 +00:00
Ivan Chupin
8f75f4a2f5
Add SBIntuitionsBot 2025-06-04 03:48:42 +05:00
Cory Dransfeldt
3187fd8a32
chore(robots.json): adds YandexAdditional crawlers 2025-06-03 12:41:57 -07:00
dark-visitors
87016d1504 Update from Dark Visitors 2025-06-03 01:00:29 +00:00
Cory Dransfeldt
1dd66b6969
Revert "chore(robots.json): adds imgproxy crawler" 
This reverts commit b65f45e408.
 2025-06-02 11:53:06 -07:00
Cory Dransfeldt
268922f8f2
Update robots.json 2025-05-31 16:02:05 -07:00
Cory Dransfeldt
4259b25ccc
Update robots.json 2025-05-31 16:01:09 -07:00
Cory Dransfeldt
d22b9ec51a
Update robots.json 2025-05-31 16:00:13 -07:00
imp
3e8edd083e
Add AndiBot and PhindBot 
Fixes #75
 2025-05-23 13:03:49 +01:00
dark-visitors
7bf7f9164d Update from Dark Visitors
Some checks failed
/ run-tests (push) Waiting to run
Details
/ lint-json (push) Waiting to run
Details
/ ai-robots-txt (push) Has been cancelled
Details
2025-05-22 00:58:45 +00:00
Glyn Normington
851eabe059
Merge pull request #133 from ai-robots-txt/wpbot 
chore(robots.json): adds wpbot
 2025-05-21 22:05:51 +01:00
Cory Dransfeldt
af597586b6
Merge pull request #98 from kylebuckingham/main 
Updating Claude Bots
 2025-05-21 12:00:11 -07:00
Cory Dransfeldt
b1d9a60a38
chore(robots.json): adds wpbot 2025-05-21 11:40:33 -07:00
Glyn Normington
202d3c3b9a
Merge pull request #126 from ai-robots-txt/mistral-bot 
chore(robots.json): adds MistralAI-User/1.0 crawler
 2025-05-21 16:27:14 +01:00
dark-visitors
7a2e6cba52 Update from Dark Visitors
Some checks failed
/ ai-robots-txt (push) Has been cancelled
Details
/ run-tests (push) Has been cancelled
Details
/ lint-json (push) Has been cancelled
Details
2025-05-17 00:57:28 +00:00
Cory Dransfeldt
ca918a963f
chore(robots.json): adds Google-CloudVertexBot 2025-05-15 21:16:49 -07:00
Cory Dransfeldt
5fba0b746d
chore(robots.json): adds MistralAI-User/1.0 crawler 2025-05-15 20:45:20 -07:00
dark-visitors
16d1de7094 Update from Dark Visitors
Some checks are pending
/ ai-robots-txt (push) Waiting to run
Details
/ run-tests (push) Waiting to run
Details
/ lint-json (push) Waiting to run
Details
2025-05-16 00:59:08 +00:00
Joe Hoyle
0c56b96fd9
Fix JSON syntax error 2025-05-15 11:26:47 -04:00
Cory Dransfeldt
9539256cb3
chore(robots.json): adds QualifiedBot crawler 2025-05-15 07:16:07 -07:00
dark-visitors
934ac7b318 Update from Dark Visitors
Some checks failed
/ run-tests (push) Waiting to run
Details
/ ai-robots-txt (push) Has been cancelled
Details
2025-04-25 00:56:57 +00:00
maia
9d846ced45
Update robots.json 
Lowercase meta-external* as that was not technically the UA for the bots, also removed a period in the "respect" for consistency
 2025-04-24 04:08:20 +02:00
dark-visitors
8d25a424d9 Update from Dark Visitors
Some checks failed
/ ai-robots-txt (push) Has been cancelled
Details
/ run-tests (push) Has been cancelled
Details
2025-04-23 00:56:52 +00:00
Dennis Lee
33c5ce1326
Update robots.json 
Updated robots list with five new proposed AI bots:

aiHitBot
Cotoyogi
Factset_spyderbot
FirecrawlAgent
TikTokSpider
 2025-04-21 18:55:11 +01:00
Kyle Buckingham
fd41de8522
Update robots.json 
Co-authored-by: Glyn Normington <work@underlap.org>
 2025-04-16 16:43:03 -07:00
Kyle Buckingham
4a6f37d727
Update robots.json 
Co-authored-by: Glyn Normington <work@underlap.org>
 2025-04-16 16:42:58 -07:00
dark-visitors
a96e330989 Update from Dark Visitors
Some checks are pending
/ run-tests (push) Waiting to run
Details
2025-04-15 00:57:01 +00:00
dark-visitors
305188b2e7 Update from Dark Visitors
Some checks failed
/ run-tests (push) Has been cancelled
Details
2025-04-11 00:55:52 +00:00
Cory Dransfeldt
b65f45e408
chore(robots.json): adds imgproxy crawler 2025-04-10 10:12:51 -07:00
Frederic Barthelemy
5f5a89c38c
Fix html-mangled hyphen in Perplexity-Users 
Fixes: #99
 2025-04-04 17:34:14 -07:00
Kyle Buckingham
ae8f74c10c
Update robots.json 2025-04-01 15:22:04 -07:00
dark-visitors
c249de99a3 Update from Dark Visitors 2025-03-28 00:54:28 +00:00
deyigifts
6ecfcdfcbf
Update perplexity bot 
Update based on perplexity bot docs
 2025-03-24 14:16:57 +08:00