ai.robots.txt
5326c202b5
Merge pull request #154 from paulrudy/main
...
/ ai-robots-txt (push) Waiting to run
/ run-tests (push) Waiting to run
/ lint-json (push) Waiting to run
re-add facebookexternalhit
2025-06-16 15:12:42 +00:00
ai.robots.txt
eb05f2f527
Merge pull request #153 from sergiospagnuolo/Poseidon
...
/ ai-robots-txt (push) Has been cancelled
/ run-tests (push) Has been cancelled
/ lint-json (push) Has been cancelled
Update robots.json with new crawler
2025-06-14 14:04:03 +00:00
ai.robots.txt
e53d81c66d
Merge pull request #152 from ai-robots-txt/MyCentralAIScraperBot
...
/ ai-robots-txt (push) Waiting to run
/ run-tests (push) Waiting to run
/ lint-json (push) Waiting to run
chore(robots.json): adds MyCentralAIScraperBot
2025-06-13 09:28:41 +00:00
ai.robots.txt
842e2256e8
Merge pull request #150 from ai-robots-txt/semrush-bots
...
/ run-tests (push) Has been cancelled
/ lint-json (push) Has been cancelled
/ ai-robots-txt (push) Has been cancelled
chore(robots.json): adds additional SemrushBot user agents
2025-06-12 07:12:00 +00:00
ai.robots.txt
3759a6bf14
chore(robots.json): adds EchoboxBot ( #148 )
/ ai-robots-txt (push) Waiting to run
/ run-tests (push) Waiting to run
/ lint-json (push) Waiting to run
2025-06-09 15:44:36 +00:00
ai.robots.txt
ac7ed17e71
Merge pull request #145 from ai-robots-txt/aws-bedrockbot
...
/ ai-robots-txt (push) Waiting to run
/ run-tests (push) Waiting to run
/ lint-json (push) Waiting to run
chore(robots.json): adds bedrockbot
2025-06-05 16:51:17 +00:00
ai.robots.txt
75ea75a95b
Merge pull request #143 from ai-robots-txt/panscient
...
/ ai-robots-txt (push) Waiting to run
/ run-tests (push) Waiting to run
/ lint-json (push) Waiting to run
chore(robots.json): adds Panscient
2025-06-04 18:04:06 +00:00
ai.robots.txt
9c28c63a0c
Merge pull request #142 from ai-robots-txt/quillbot
...
chore(robots.json): adds Quillbot
2025-06-04 17:54:57 +00:00
ai.robots.txt
3efabc603d
Merge pull request #141 from Ivan-Chupin/patch-1
...
Add SBIntuitionsBot
2025-06-03 23:28:48 +00:00
ai.robots.txt
080946c360
Merge pull request #140 from ai-robots-txt/yandex-bots
...
/ ai-robots-txt (push) Waiting to run
/ run-tests (push) Waiting to run
/ lint-json (push) Waiting to run
chore(robots.json): adds YandexAdditional crawlers
2025-06-03 19:51:25 +00:00
ai.robots.txt
d239e7e5ad
Merge pull request #139 from ai-robots-txt/workflow-fix
...
/ ai-robots-txt (push) Waiting to run
/ run-tests (push) Waiting to run
/ lint-json (push) Waiting to run
chore(ai_robots_update.yml): correct workflow by revising git flags + adding guard
2025-06-03 01:52:35 +00:00
ai.robots.txt
093ab81d78
Update from Dark Visitors
/ run-tests (push) Has been cancelled
/ lint-json (push) Has been cancelled
2025-05-23 00:58:57 +00:00
ai.robots.txt
fedb658cc0
Merge pull request #133 from ai-robots-txt/wpbot
...
chore(robots.json): adds wpbot
2025-05-21 21:06:05 +00:00
ai.robots.txt
7c5389f4a0
Merge pull request #98 from kylebuckingham/main
...
Updating Claude Bots
2025-05-21 19:00:23 +00:00
ai.robots.txt
1c2acd75b7
Merge pull request #126 from ai-robots-txt/mistral-bot
...
/ ai-robots-txt (push) Waiting to run
/ run-tests (push) Waiting to run
/ lint-json (push) Waiting to run
chore(robots.json): adds MistralAI-User/1.0 crawler
2025-05-21 15:27:26 +00:00
ai.robots.txt
dd1ed174b7
Merge pull request #129 from ai-robots-txt/google-cloudvertexbot
...
/ ai-robots-txt (push) Waiting to run
/ run-tests (push) Waiting to run
/ lint-json (push) Waiting to run
chore(robots.json): adds Google-CloudVertexBot
2025-05-16 11:35:15 +00:00
ai.robots.txt
1c470babbe
Merge pull request #123 from joehoyle/patch-1
...
Fix JSON syntax error
2025-05-15 16:12:30 +00:00
ai.robots.txt
c6c7f1748f
Update from Dark Visitors
/ run-tests (push) Has been cancelled
2025-04-26 00:55:12 +00:00
ai.robots.txt
4654e14e9c
Merge pull request #112 from maiavixen/main
...
/ ai-robots-txt (push) Waiting to run
/ run-tests (push) Waiting to run
Fixed meta-external* being titlecase, and removed period for consistency
2025-04-24 07:00:34 +00:00
ai.robots.txt
bbec639c14
Merge pull request #109 from dennislee1/patch-1
...
/ ai-robots-txt (push) Waiting to run
/ run-tests (push) Waiting to run
AI bots to consider adding
2025-04-22 14:50:26 +00:00
ai.robots.txt
e0cdb278fb
Update from Dark Visitors
/ run-tests (push) Has been cancelled
2025-04-16 00:57:11 +00:00
ai.robots.txt
4a764bba18
Merge pull request #102 from ai-robots-txt/imgproxy-bot
...
/ run-tests (push) Waiting to run
chore(robots.json): adds imgproxy crawler
2025-04-10 19:22:34 +00:00
Frederic Barthelemy
5f5a89c38c
Fix html-mangled hyphen in Perplexity-Users
...
Fixes : #99
2025-04-04 17:34:14 -07:00
ai.robots.txt
5b8650b99b
Update from Dark Visitors
/ run-tests (push) Has been cancelled
2025-03-29 00:54:10 +00:00
ec18af7624
Revert "Merge pull request #91 from deyigifts/perplexity-user"
...
This reverts commit 68d1d93714
2025-03-27 12:51:22 -07:00
ai.robots.txt
68d1d93714
Merge pull request #91 from deyigifts/perplexity-user
...
Update perplexity bots
2025-03-27 19:29:30 +00:00
Dennis Camera
a884a2afb9
.htaccess: Make regex in RewriteCond safe
...
Improve the regular expression by removing unneeded anchors and
escaping special characters (not just space) to prevent false positives
or a misbehaving rewrite rule.
2025-02-18 12:53:22 +01:00
Dennis Camera
c0d418cd87
.htaccess: Allow robots access to /robots.txt
2025-02-18 12:49:29 +01:00
ai.robots.txt
693289bb29
chore: add Brightbot 1.0
2025-02-16 21:37:52 +00:00
ai.robots.txt
89d4c6e5ca
Merge pull request #73 from nisbet-hubbard/patch-8
...
Actually block Semrush’s AI tools
2025-02-01 10:51:01 +00:00
ai.robots.txt
6c552a3daa
Merge pull request #71 from jsheard/patch-1
...
Add Crawlspace
2025-01-20 17:45:42 +00:00
Massimo Gismondi
933aa6159d
Implementing htaccess generation
2025-01-07 11:02:29 +01:00
