Brendan Ngwa Nforbi
|
5c170bb6e3
|
Merge branch 'ai-robots-txt:main' into feature/ua-synonyms-support
|
2025-06-12 08:49:22 -07:00 |
|
brndnprog
|
3148669363
|
feat: add ua-synonyms support with tests, scoped to json_to_* functions
|
2025-06-12 08:31:54 -07:00 |
|
ai.robots.txt
|
842e2256e8
|
Merge pull request #150 from ai-robots-txt/semrush-bots
/ run-tests (push) Has been cancelled
/ lint-json (push) Has been cancelled
/ ai-robots-txt (push) Has been cancelled
chore(robots.json): adds additional SemrushBot user agents
|
2025-06-12 07:12:00 +00:00 |
|
Glyn Normington
|
229ea20426
|
Merge pull request #150 from ai-robots-txt/semrush-bots
chore(robots.json): adds additional SemrushBot user agents
|
2025-06-12 08:11:51 +01:00 |
|
brndnprog
|
f0606f0eac
|
feat: support ua-synonyms in robots.json to reduce duplication (#144)
|
2025-06-11 22:04:39 -07:00 |
|
|
|
14d68f05ba
|
chore(robots.json): adds additional SemrushBot user agents
|
2025-06-11 13:50:53 -07:00 |
|
dark-visitors
|
cf598b6b71
|
Update from Dark Visitors
/ ai-robots-txt (push) Has been cancelled
/ run-tests (push) Has been cancelled
/ lint-json (push) Has been cancelled
|
2025-06-10 01:00:37 +00:00 |
|
ai.robots.txt
|
3759a6bf14
|
chore(robots.json): adds EchoboxBot (#148)
/ ai-robots-txt (push) Waiting to run
/ run-tests (push) Waiting to run
/ lint-json (push) Waiting to run
|
2025-06-09 15:44:36 +00:00 |
|
|
|
7867c3e26c
|
chore(robots.json): adds EchoboxBot (#148)
|
2025-06-09 16:44:25 +01:00 |
|
dark-visitors
|
e21f6ae1b6
|
Update from Dark Visitors
/ ai-robots-txt (push) Has been cancelled
/ run-tests (push) Has been cancelled
/ lint-json (push) Has been cancelled
|
2025-06-06 00:59:25 +00:00 |
|
ai.robots.txt
|
ac7ed17e71
|
Merge pull request #145 from ai-robots-txt/aws-bedrockbot
/ ai-robots-txt (push) Waiting to run
/ run-tests (push) Waiting to run
/ lint-json (push) Waiting to run
chore(robots.json): adds bedrockbot
|
2025-06-05 16:51:17 +00:00 |
|
Glyn Normington
|
81747e6772
|
Merge pull request #145 from ai-robots-txt/aws-bedrockbot
chore(robots.json): adds bedrockbot
|
2025-06-05 17:51:03 +01:00 |
|
|
|
528d77bf07
|
chore(robots.json): adds bedrockbot
|
2025-06-05 09:14:23 -07:00 |
|
dark-visitors
|
77393df5aa
|
Update from Dark Visitors
/ ai-robots-txt (push) Waiting to run
/ run-tests (push) Waiting to run
/ lint-json (push) Waiting to run
|
2025-06-05 00:59:28 +00:00 |
|
ai.robots.txt
|
75ea75a95b
|
Merge pull request #143 from ai-robots-txt/panscient
/ ai-robots-txt (push) Waiting to run
/ run-tests (push) Waiting to run
/ lint-json (push) Waiting to run
chore(robots.json): adds Panscient
|
2025-06-04 18:04:06 +00:00 |
|
Glyn Normington
|
2fca1ddcf1
|
Merge pull request #143 from ai-robots-txt/panscient
chore(robots.json): adds Panscient
|
2025-06-04 19:03:53 +01:00 |
|
ai.robots.txt
|
9c28c63a0c
|
Merge pull request #142 from ai-robots-txt/quillbot
chore(robots.json): adds Quillbot
|
2025-06-04 17:54:57 +00:00 |
|
|
|
395c013eea
|
Merge pull request #142 from ai-robots-txt/quillbot
chore(robots.json): adds Quillbot
|
2025-06-04 10:54:46 -07:00 |
|
|
|
4568d69b0e
|
chore(robots.json): adds Panscient
|
2025-06-04 10:54:14 -07:00 |
|
|
|
03831a7eb5
|
chore(robots.json): adds Quillbot
|
2025-06-04 10:46:58 -07:00 |
|
dark-visitors
|
2b5a59a303
|
Update from Dark Visitors
/ ai-robots-txt (push) Waiting to run
/ run-tests (push) Waiting to run
/ lint-json (push) Waiting to run
|
2025-06-04 01:00:07 +00:00 |
|
ai.robots.txt
|
3efabc603d
|
Merge pull request #141 from Ivan-Chupin/patch-1
Add SBIntuitionsBot
|
2025-06-03 23:28:48 +00:00 |
|
|
|
b35f9a31d7
|
Merge pull request #141 from Ivan-Chupin/patch-1
Add SBIntuitionsBot
|
2025-06-03 16:28:36 -07:00 |
|
Ivan Chupin
|
8f75f4a2f5
|
Add SBIntuitionsBot
|
2025-06-04 03:48:42 +05:00 |
|
ai.robots.txt
|
080946c360
|
Merge pull request #140 from ai-robots-txt/yandex-bots
/ ai-robots-txt (push) Waiting to run
/ run-tests (push) Waiting to run
/ lint-json (push) Waiting to run
chore(robots.json): adds YandexAdditional crawlers
|
2025-06-03 19:51:25 +00:00 |
|
Glyn Normington
|
7eec033cad
|
Merge pull request #140 from ai-robots-txt/yandex-bots
chore(robots.json): adds YandexAdditional crawlers
|
2025-06-03 20:51:14 +01:00 |
|
|
|
3187fd8a32
|
chore(robots.json): adds YandexAdditional crawlers
|
2025-06-03 12:41:57 -07:00 |
|
ai.robots.txt
|
d239e7e5ad
|
Merge pull request #139 from ai-robots-txt/workflow-fix
/ ai-robots-txt (push) Waiting to run
/ run-tests (push) Waiting to run
/ lint-json (push) Waiting to run
chore(ai_robots_update.yml): correct workflow by revising git flags + adding guard
|
2025-06-03 01:52:35 +00:00 |
|
Glyn Normington
|
9dbf34010a
|
Merge pull request #139 from ai-robots-txt/workflow-fix
chore(ai_robots_update.yml): correct workflow by revising git flags + adding guard
|
2025-06-03 02:52:23 +01:00 |
|
dark-visitors
|
87016d1504
|
Update from Dark Visitors
|
2025-06-03 01:00:29 +00:00 |
|
|
|
899ce01c55
|
chore(ai_robots_update.yml): correct workflow by revising git flags + adding guard
|
2025-06-02 14:56:09 -07:00 |
|
Glyn Normington
|
4af776f0a0
|
Merge pull request #136 from ai-robots-txt/imgproxy-revert
/ ai-robots-txt (push) Waiting to run
/ run-tests (push) Waiting to run
/ lint-json (push) Waiting to run
chore(robots.json): revert "adds imgproxy crawler"
|
2025-06-02 20:21:10 +01:00 |
|
|
|
1dd66b6969
|
Revert "chore(robots.json): adds imgproxy crawler"
This reverts commit b65f45e408.
|
2025-06-02 11:53:06 -07:00 |
|
|
|
814df6b9a0
|
Merge pull request #134 from not-not-the-imp/patch-1
/ ai-robots-txt (push) Has been cancelled
/ run-tests (push) Has been cancelled
/ lint-json (push) Has been cancelled
Add AndiBot and PhindBot
|
2025-05-31 16:03:16 -07:00 |
|
|
|
268922f8f2
|
Update robots.json
|
2025-05-31 16:02:05 -07:00 |
|
|
|
4259b25ccc
|
Update robots.json
|
2025-05-31 16:01:09 -07:00 |
|
|
|
d22b9ec51a
|
Update robots.json
|
2025-05-31 16:00:13 -07:00 |
|
imp
|
3e8edd083e
|
Add AndiBot and PhindBot
Fixes #75
|
2025-05-23 13:03:49 +01:00 |
|
ai.robots.txt
|
093ab81d78
|
Update from Dark Visitors
/ run-tests (push) Has been cancelled
/ lint-json (push) Has been cancelled
|
2025-05-23 00:58:57 +00:00 |
|
dark-visitors
|
7bf7f9164d
|
Update from Dark Visitors
/ run-tests (push) Waiting to run
/ lint-json (push) Waiting to run
/ ai-robots-txt (push) Has been cancelled
|
2025-05-22 00:58:45 +00:00 |
|
ai.robots.txt
|
fedb658cc0
|
Merge pull request #133 from ai-robots-txt/wpbot
chore(robots.json): adds wpbot
|
2025-05-21 21:06:05 +00:00 |
|
Glyn Normington
|
851eabe059
|
Merge pull request #133 from ai-robots-txt/wpbot
chore(robots.json): adds wpbot
|
2025-05-21 22:05:51 +01:00 |
|
ai.robots.txt
|
7c5389f4a0
|
Merge pull request #98 from kylebuckingham/main
Updating Claude Bots
|
2025-05-21 19:00:23 +00:00 |
|
|
|
af597586b6
|
Merge pull request #98 from kylebuckingham/main
Updating Claude Bots
|
2025-05-21 12:00:11 -07:00 |
|
|
|
b1d9a60a38
|
chore(robots.json): adds wpbot
|
2025-05-21 11:40:33 -07:00 |
|
ai.robots.txt
|
1c2acd75b7
|
Merge pull request #126 from ai-robots-txt/mistral-bot
/ ai-robots-txt (push) Waiting to run
/ run-tests (push) Waiting to run
/ lint-json (push) Waiting to run
chore(robots.json): adds MistralAI-User/1.0 crawler
|
2025-05-21 15:27:26 +00:00 |
|
Glyn Normington
|
202d3c3b9a
|
Merge pull request #126 from ai-robots-txt/mistral-bot
chore(robots.json): adds MistralAI-User/1.0 crawler
|
2025-05-21 16:27:14 +01:00 |
|
Glyn Normington
|
0a78fe1e76
|
Merge pull request #132 from ai-robots-txt/crawler-policy-update
chore(README): updates the opening line of our README to clarify the types of agents we block
|
2025-05-21 15:13:35 +01:00 |
|
|
|
8b151b2cdc
|
Update README.md
Co-authored-by: Glyn Normington <glyn.normington@gmail.com>
|
2025-05-21 06:52:36 -07:00 |
|
|
|
8a8001cbec
|
chore(README): updates the opening line of our README to clarify the types of agents we block
|
2025-05-20 13:55:25 -07:00 |
|
