Glyn Normington
|
2fca1ddcf1
|
Merge pull request #143 from ai-robots-txt/panscient
chore(robots.json): adds Panscient
|
2025-06-04 19:03:53 +01:00 |
|
|
|
4568d69b0e
|
chore(robots.json): adds Panscient
|
2025-06-04 10:54:14 -07:00 |
|
|
|
03831a7eb5
|
chore(robots.json): adds Quillbot
|
2025-06-04 10:46:58 -07:00 |
|
dark-visitors
|
2b5a59a303
|
Update from Dark Visitors
/ ai-robots-txt (push) Waiting to run
/ run-tests (push) Waiting to run
/ lint-json (push) Waiting to run
|
2025-06-04 01:00:07 +00:00 |
|
Ivan Chupin
|
8f75f4a2f5
|
Add SBIntuitionsBot
|
2025-06-04 03:48:42 +05:00 |
|
|
|
3187fd8a32
|
chore(robots.json): adds YandexAdditional crawlers
|
2025-06-03 12:41:57 -07:00 |
|
dark-visitors
|
87016d1504
|
Update from Dark Visitors
|
2025-06-03 01:00:29 +00:00 |
|
|
|
1dd66b6969
|
Revert "chore(robots.json): adds imgproxy crawler"
This reverts commit b65f45e408.
|
2025-06-02 11:53:06 -07:00 |
|
|
|
268922f8f2
|
Update robots.json
|
2025-05-31 16:02:05 -07:00 |
|
|
|
4259b25ccc
|
Update robots.json
|
2025-05-31 16:01:09 -07:00 |
|
|
|
d22b9ec51a
|
Update robots.json
|
2025-05-31 16:00:13 -07:00 |
|
imp
|
3e8edd083e
|
Add AndiBot and PhindBot
Fixes #75
|
2025-05-23 13:03:49 +01:00 |
|
dark-visitors
|
7bf7f9164d
|
Update from Dark Visitors
/ run-tests (push) Waiting to run
/ lint-json (push) Waiting to run
/ ai-robots-txt (push) Has been cancelled
|
2025-05-22 00:58:45 +00:00 |
|
Glyn Normington
|
851eabe059
|
Merge pull request #133 from ai-robots-txt/wpbot
chore(robots.json): adds wpbot
|
2025-05-21 22:05:51 +01:00 |
|
|
|
af597586b6
|
Merge pull request #98 from kylebuckingham/main
Updating Claude Bots
|
2025-05-21 12:00:11 -07:00 |
|
|
|
b1d9a60a38
|
chore(robots.json): adds wpbot
|
2025-05-21 11:40:33 -07:00 |
|
Glyn Normington
|
202d3c3b9a
|
Merge pull request #126 from ai-robots-txt/mistral-bot
chore(robots.json): adds MistralAI-User/1.0 crawler
|
2025-05-21 16:27:14 +01:00 |
|
dark-visitors
|
7a2e6cba52
|
Update from Dark Visitors
/ ai-robots-txt (push) Has been cancelled
/ run-tests (push) Has been cancelled
/ lint-json (push) Has been cancelled
|
2025-05-17 00:57:28 +00:00 |
|
|
|
ca918a963f
|
chore(robots.json): adds Google-CloudVertexBot
|
2025-05-15 21:16:49 -07:00 |
|
|
|
5fba0b746d
|
chore(robots.json): adds MistralAI-User/1.0 crawler
|
2025-05-15 20:45:20 -07:00 |
|
dark-visitors
|
16d1de7094
|
Update from Dark Visitors
/ ai-robots-txt (push) Waiting to run
/ run-tests (push) Waiting to run
/ lint-json (push) Waiting to run
|
2025-05-16 00:59:08 +00:00 |
|
Joe Hoyle
|
0c56b96fd9
|
Fix JSON syntax error
|
2025-05-15 11:26:47 -04:00 |
|
|
|
9539256cb3
|
chore(robots.json): adds QualifiedBot crawler
|
2025-05-15 07:16:07 -07:00 |
|
dark-visitors
|
934ac7b318
|
Update from Dark Visitors
/ run-tests (push) Waiting to run
/ ai-robots-txt (push) Has been cancelled
|
2025-04-25 00:56:57 +00:00 |
|
maia
|
9d846ced45
|
Update robots.json
Lowercase meta-external* as that was not technically the UA for the bots, also removed a period in the "respect" for consistency
|
2025-04-24 04:08:20 +02:00 |
|
dark-visitors
|
8d25a424d9
|
Update from Dark Visitors
/ ai-robots-txt (push) Has been cancelled
/ run-tests (push) Has been cancelled
|
2025-04-23 00:56:52 +00:00 |
|
Dennis Lee
|
33c5ce1326
|
Update robots.json
Updated robots list with five new proposed AI bots:
aiHitBot
Cotoyogi
Factset_spyderbot
FirecrawlAgent
TikTokSpider
|
2025-04-21 18:55:11 +01:00 |
|
Kyle Buckingham
|
fd41de8522
|
Update robots.json
Co-authored-by: Glyn Normington <work@underlap.org>
|
2025-04-16 16:43:03 -07:00 |
|
Kyle Buckingham
|
4a6f37d727
|
Update robots.json
Co-authored-by: Glyn Normington <work@underlap.org>
|
2025-04-16 16:42:58 -07:00 |
|
dark-visitors
|
a96e330989
|
Update from Dark Visitors
/ run-tests (push) Waiting to run
|
2025-04-15 00:57:01 +00:00 |
|
dark-visitors
|
305188b2e7
|
Update from Dark Visitors
/ run-tests (push) Has been cancelled
|
2025-04-11 00:55:52 +00:00 |
|
|
|
b65f45e408
|
chore(robots.json): adds imgproxy crawler
|
2025-04-10 10:12:51 -07:00 |
|
Frederic Barthelemy
|
5f5a89c38c
|
Fix html-mangled hyphen in Perplexity-Users
Fixes: #99
|
2025-04-04 17:34:14 -07:00 |
|
Kyle Buckingham
|
ae8f74c10c
|
Update robots.json
|
2025-04-01 15:22:04 -07:00 |
|
dark-visitors
|
c249de99a3
|
Update from Dark Visitors
|
2025-03-28 00:54:28 +00:00 |
|
deyigifts
|
6ecfcdfcbf
|
Update perplexity bot
Update based on perplexity bot docs
|
2025-03-24 14:16:57 +08:00 |
|
dark-visitors
|
abfd6dfcd1
|
Update from Dark Visitors
|
2025-02-17 00:53:32 +00:00 |
|
|
|
a9ec4ffa6f
|
chore: add Brightbot 1.0
|
2025-02-16 13:36:39 -08:00 |
|
dark-visitors
|
bebffccc0c
|
Update from Dark Visitors
|
2025-02-02 00:52:50 +00:00 |
|
nisbet-hubbard
|
05b79b8a58
|
Update robots.json
|
2025-01-27 19:41:03 +08:00 |
|
dark-visitors
|
9c060dee1c
|
Update from Dark Visitors
|
2025-01-21 00:49:22 +00:00 |
|
Joshua Sheard
|
7427d96bac
|
Update robots.json
Co-authored-by: Glyn Normington <work@underlap.org>
|
2025-01-20 10:59:02 +00:00 |
|
Joshua Sheard
|
5aa08bc002
|
Add Crawlspace
|
2025-01-19 22:03:50 +00:00 |
|
Jordan Atwood
|
143f8f2285
|
Block SemrushBot
|
2025-01-06 12:34:38 -08:00 |
|
dark-visitors
|
2036a68c1f
|
Update from Dark Visitors
|
2024-12-04 00:55:50 +00:00 |
|
dark-visitors
|
37065f9118
|
Update from Dark Visitors
|
2024-11-24 00:57:05 +00:00 |
|
Glyn Normington
|
80002f5e17
|
Allow facebookexternalhit
At the time of writing, this crawler does not
appear to be for the purpose of AI.
See: https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/
(accessed on 19 November 2024).
Fixes https://github.com/ai-robots-txt/ai.robots.txt/issues/40
|
2024-11-19 03:33:45 +00:00 |
|
dark-visitors
|
bc0a0ad0e9
|
Update from Dark Visitors
|
2024-10-29 00:52:12 +00:00 |
|
dark-visitors
|
fe5f407673
|
Update from Dark Visitors
|
2024-10-27 00:54:47 +00:00 |
|
Glyn Normington
|
38a388097c
|
Fix typo and trigger rerun of main job
|
2024-10-19 04:42:27 +01:00 |
|
