ai.robots.txt
|
093ab81d78
|
Update from Dark Visitors
/ run-tests (push) Has been cancelled
/ lint-json (push) Has been cancelled
|
2025-05-23 00:58:57 +00:00 |
|
dark-visitors
|
7bf7f9164d
|
Update from Dark Visitors
/ run-tests (push) Waiting to run
/ lint-json (push) Waiting to run
/ ai-robots-txt (push) Has been cancelled
|
2025-05-22 00:58:45 +00:00 |
|
ai.robots.txt
|
fedb658cc0
|
Merge pull request #133 from ai-robots-txt/wpbot
chore(robots.json): adds wpbot
|
2025-05-21 21:06:05 +00:00 |
|
Glyn Normington
|
851eabe059
|
Merge pull request #133 from ai-robots-txt/wpbot
chore(robots.json): adds wpbot
|
2025-05-21 22:05:51 +01:00 |
|
ai.robots.txt
|
7c5389f4a0
|
Merge pull request #98 from kylebuckingham/main
Updating Claude Bots
|
2025-05-21 19:00:23 +00:00 |
|
|
|
af597586b6
|
Merge pull request #98 from kylebuckingham/main
Updating Claude Bots
|
2025-05-21 12:00:11 -07:00 |
|
|
|
b1d9a60a38
|
chore(robots.json): adds wpbot
|
2025-05-21 11:40:33 -07:00 |
|
ai.robots.txt
|
1c2acd75b7
|
Merge pull request #126 from ai-robots-txt/mistral-bot
/ ai-robots-txt (push) Waiting to run
/ run-tests (push) Waiting to run
/ lint-json (push) Waiting to run
chore(robots.json): adds MistralAI-User/1.0 crawler
|
2025-05-21 15:27:26 +00:00 |
|
Glyn Normington
|
202d3c3b9a
|
Merge pull request #126 from ai-robots-txt/mistral-bot
chore(robots.json): adds MistralAI-User/1.0 crawler
|
2025-05-21 16:27:14 +01:00 |
|
Glyn Normington
|
0a78fe1e76
|
Merge pull request #132 from ai-robots-txt/crawler-policy-update
chore(README): updates the opening line of our README to clarify the types of agents we block
|
2025-05-21 15:13:35 +01:00 |
|
|
|
8b151b2cdc
|
Update README.md
Co-authored-by: Glyn Normington <glyn.normington@gmail.com>
|
2025-05-21 06:52:36 -07:00 |
|
|
|
8a8001cbec
|
chore(README): updates the opening line of our README to clarify the types of agents we block
|
2025-05-20 13:55:25 -07:00 |
|
Glyn Normington
|
fe1267e290
|
Merge pull request #131 from Mihitoko/mention-x-robots-tag-for-bing
/ run-tests (push) Has been cancelled
/ lint-json (push) Has been cancelled
Mention X-Robots-Tag header as alternative for bing
|
2025-05-20 07:52:32 +01:00 |
|
Mihitoko
|
9297c7dfa3
|
Mention X-Robots-Tag header as alternative for bing
|
2025-05-20 00:10:05 +02:00 |
|
dark-visitors
|
7a2e6cba52
|
Update from Dark Visitors
/ ai-robots-txt (push) Has been cancelled
/ run-tests (push) Has been cancelled
/ lint-json (push) Has been cancelled
|
2025-05-17 00:57:28 +00:00 |
|
ai.robots.txt
|
dd1ed174b7
|
Merge pull request #129 from ai-robots-txt/google-cloudvertexbot
/ ai-robots-txt (push) Waiting to run
/ run-tests (push) Waiting to run
/ lint-json (push) Waiting to run
chore(robots.json): adds Google-CloudVertexBot
|
2025-05-16 11:35:15 +00:00 |
|
Glyn Normington
|
89c0fbaf86
|
Merge pull request #129 from ai-robots-txt/google-cloudvertexbot
chore(robots.json): adds Google-CloudVertexBot
|
2025-05-16 12:35:04 +01:00 |
|
|
|
ca918a963f
|
chore(robots.json): adds Google-CloudVertexBot
|
2025-05-15 21:16:49 -07:00 |
|
|
|
5fba0b746d
|
chore(robots.json): adds MistralAI-User/1.0 crawler
|
2025-05-15 20:45:20 -07:00 |
|
dark-visitors
|
16d1de7094
|
Update from Dark Visitors
/ ai-robots-txt (push) Waiting to run
/ run-tests (push) Waiting to run
/ lint-json (push) Waiting to run
|
2025-05-16 00:59:08 +00:00 |
|
Glyn Normington
|
73f6f67adf
|
Merge pull request #125 from holysoles/lint_robots_json
/ ai-robots-txt (push) Waiting to run
/ run-tests (push) Waiting to run
/ lint-json (push) Waiting to run
lint robots.json during pull requests
|
2025-05-15 17:26:15 +01:00 |
|
Patrick Evans
|
498aa50760
|
lint robots.json during pull requests
|
2025-05-15 11:15:25 -05:00 |
|
ai.robots.txt
|
1c470babbe
|
Merge pull request #123 from joehoyle/patch-1
Fix JSON syntax error
|
2025-05-15 16:12:30 +00:00 |
|
Adam Newbold
|
84d63916d2
|
Merge pull request #123 from joehoyle/patch-1
Fix JSON syntax error
|
2025-05-15 12:12:21 -04:00 |
|
Joe Hoyle
|
0c56b96fd9
|
Fix JSON syntax error
|
2025-05-15 11:26:47 -04:00 |
|
|
|
28e69e631b
|
Merge pull request #122 from ai-robots-txt/qualified-bot
/ ai-robots-txt (push) Waiting to run
/ run-tests (push) Waiting to run
chore(robots.json): adds QualifiedBot crawler
|
2025-05-15 07:17:51 -07:00 |
|
|
|
9539256cb3
|
chore(robots.json): adds QualifiedBot crawler
|
2025-05-15 07:16:07 -07:00 |
|
|
|
9659c88b0c
|
Merge pull request #121 from solution-libre/add-traefik-plugin
/ run-tests (push) Waiting to run
Add Traefik plugin to the README.md file
|
2025-05-14 16:45:34 -07:00 |
|
Florent Poinsaut
|
c66d180295
|
Merge branch 'main' into add-traefik-plugin
|
2025-05-14 22:06:56 +02:00 |
|
Glyn Normington
|
9a9b1b41c0
|
Merge pull request #119 from ai-robots-txt/bing-ai-opt-out-instructions
/ run-tests (push) Waiting to run
Bing AI opt-out instructions
|
2025-05-14 19:18:20 +01:00 |
|
Florent Poinsaut
|
b4610a725c
|
Add Traefik plugin
|
2025-05-14 14:11:56 +02:00 |
|
|
|
36a52a88d8
|
Bing AI opt-out instructions
|
2025-05-12 20:20:18 -07:00 |
|
ai.robots.txt
|
678380727e
|
Merge pull request #115 from glyn/syntax
/ run-tests (push) Has been cancelled
/ ai-robots-txt (push) Has been cancelled
Fix Python syntax error
|
2025-05-01 10:29:06 +00:00 |
|
Glyn Normington
|
fb8188c49d
|
Merge pull request #115 from glyn/syntax
Fix Python syntax error
|
2025-05-01 11:28:54 +01:00 |
|
Glyn Normington
|
ec995cd686
|
Fix Python syntax error
|
2025-05-01 11:27:40 +01:00 |
|
Crazyroostereye
|
1310dbae46
|
Added a Caddyfile converter (#110)
Co-authored-by: Julian Beittel <julian@beittel.net>
Co-authored-by: Glyn Normington <work@underlap.org>
|
2025-05-01 11:21:32 +01:00 |
|
Glyn Normington
|
91a88e2fa8
|
Merge pull request #113 from rwijnen-um/feature/haproxy
/ ai-robots-txt (push) Has been cancelled
/ run-tests (push) Has been cancelled
HAProxy converter added.
|
2025-04-28 09:00:16 +01:00 |
|
Rik Wijnen
|
a4a9f2ac2b
|
Tests for HAProxy file added.
|
2025-04-28 09:30:26 +02:00 |
|
Rik Wijnen
|
66da70905f
|
Fixed incorrect English sentence.
|
2025-04-28 09:09:40 +02:00 |
|
Rik Wijnen
|
50e739dd73
|
HAProxy converter added.
|
2025-04-28 08:51:02 +02:00 |
|
ai.robots.txt
|
c6c7f1748f
|
Update from Dark Visitors
/ run-tests (push) Has been cancelled
|
2025-04-26 00:55:12 +00:00 |
|
dark-visitors
|
934ac7b318
|
Update from Dark Visitors
/ run-tests (push) Waiting to run
/ ai-robots-txt (push) Has been cancelled
|
2025-04-25 00:56:57 +00:00 |
|
ai.robots.txt
|
4654e14e9c
|
Merge pull request #112 from maiavixen/main
/ ai-robots-txt (push) Waiting to run
/ run-tests (push) Waiting to run
Fixed meta-external* being titlecase, and removed period for consistency
|
2025-04-24 07:00:34 +00:00 |
|
Glyn Normington
|
9bf31fbca8
|
Merge pull request #112 from maiavixen/main
Fixed meta-external* being titlecase, and removed period for consistency
|
2025-04-24 08:00:24 +01:00 |
|
maia
|
9d846ced45
|
Update robots.json
Lowercase meta-external* as that was not technically the UA for the bots, also removed a period in the "respect" for consistency
|
2025-04-24 04:08:20 +02:00 |
|
dark-visitors
|
8d25a424d9
|
Update from Dark Visitors
/ ai-robots-txt (push) Has been cancelled
/ run-tests (push) Has been cancelled
|
2025-04-23 00:56:52 +00:00 |
|
ai.robots.txt
|
bbec639c14
|
Merge pull request #109 from dennislee1/patch-1
/ ai-robots-txt (push) Waiting to run
/ run-tests (push) Waiting to run
AI bots to consider adding
|
2025-04-22 14:50:26 +00:00 |
|
|
|
422cf9e29b
|
Merge pull request #109 from dennislee1/patch-1
AI bots to consider adding
|
2025-04-22 07:50:14 -07:00 |
|
Dennis Lee
|
33c5ce1326
|
Update robots.json
Updated robots list with five new proposed AI bots:
aiHitBot
Cotoyogi
Factset_spyderbot
FirecrawlAgent
TikTokSpider
|
2025-04-21 18:55:11 +01:00 |
|
|
|
774b1ddf52
|
Merge pull request #107 from glyn/sponsorship
/ run-tests (push) Has been cancelled
Clarify our position on sponsorship
|
2025-04-18 11:40:06 -07:00 |
|
