cdransf/ai.robots.txt
1
1
Fork 0
mirror of https://github.com/ai-robots-txt/ai.robots.txt.git synced 2025-06-19 02:00:51 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 1

Commit graph

  • 6baa7725b3
         Merge branch 'main' into main Crazyroostereye 2025-04-30 22:13:51 +02:00
  • 8164cec0c8
         Update bingbot-extrastep.md Dennis Lee 2025-04-29 17:56:14 +01:00
  • bd5fb8a37f
         Bing AI opt-out instructions Dennis Lee 2025-04-28 22:02:12 +01:00
  • 91a88e2fa8
         Merge pull request #113 from rwijnen-um/feature/haproxy Glyn Normington 2025-04-28 09:00:16 +01:00
  • a4a9f2ac2b Tests for HAProxy file added. Rik Wijnen 2025-04-28 09:30:26 +02:00
  • 66da70905f Fixed incorrect English sentence. Rik Wijnen 2025-04-28 09:09:40 +02:00
  • 50e739dd73 HAProxy converter added. Rik Wijnen 2025-04-28 08:42:52 +02:00
  • df6d7f9723 Accedently change to the wrong Commit Julian Beittel 2025-04-27 12:19:30 +02:00
  • daa0d80c95 Fixed Diff Julian Beittel 2025-04-27 12:16:14 +02:00
  • c6c7f1748f Update from Dark Visitors ai.robots.txt 2025-04-26 00:55:12 +00:00
  • 934ac7b318 Update from Dark Visitors dark-visitors 2025-04-25 00:56:57 +00:00
  • 4654e14e9c Merge pull request #112 from maiavixen/main ai.robots.txt 2025-04-24 07:00:34 +00:00
  • 9bf31fbca8
         Merge pull request #112 from maiavixen/main Glyn Normington 2025-04-24 08:00:24 +01:00
  • 9d846ced45
         Update robots.json maia 2025-04-24 04:08:20 +02:00
  • 1e92364c13 Added Test and Docu Julian Beittel 2025-04-23 09:41:18 +02:00
  • 5864d9db4e
         Merge d79ca19f38 into 8d25a424d9 Katrin Leinweber 2025-04-23 09:28:07 +02:00
  • 8d25a424d9 Update from Dark Visitors dark-visitors 2025-04-23 00:56:52 +00:00
  • e519f9f94c Added a Caddyfile converter Julian Beittel 2025-04-22 20:17:37 +02:00
  • bbec639c14 Merge pull request #109 from dennislee1/patch-1 v1.29 ai.robots.txt 2025-04-22 14:50:26 +00:00
  • 422cf9e29b
         Merge pull request #109 from dennislee1/patch-1 Cory Dransfeldt 2025-04-22 07:50:14 -07:00
  • 33c5ce1326
         Update robots.json Dennis Lee 2025-04-21 18:55:11 +01:00
  • 774b1ddf52
         Merge pull request #107 from glyn/sponsorship Cory Dransfeldt 2025-04-18 11:40:06 -07:00
  • b1856e6988 Donations Glyn Normington 2025-04-18 18:40:44 +01:00
  • d05ede8fe1 Clarify our position on sponsorship Glyn Normington 2025-04-18 17:46:56 +01:00
  • fd41de8522
         Update robots.json Kyle Buckingham 2025-04-16 16:43:03 -07:00
  • 4a6f37d727
         Update robots.json Kyle Buckingham 2025-04-16 16:42:58 -07:00
  • e0cdb278fb Update from Dark Visitors v1.28 ai.robots.txt 2025-04-16 00:57:11 +00:00
  • a96e330989 Update from Dark Visitors dark-visitors 2025-04-15 00:57:01 +00:00
  • 156e6baa09
         Merge pull request #105 from jsheard/patch-1 Cory Dransfeldt 2025-04-14 10:08:38 -07:00
  • d9f882a9b2
         Include "AI Agents" from Dark Visitors Joshua Sheard 2025-04-14 15:46:01 +01:00
  • 305188b2e7 Update from Dark Visitors dark-visitors 2025-04-11 00:55:52 +00:00
  • 4a764bba18 Merge pull request #102 from ai-robots-txt/imgproxy-bot v1.27 ai.robots.txt 2025-04-10 19:22:34 +00:00
  • a891ad7213
         Merge pull request #102 from ai-robots-txt/imgproxy-bot Cory Dransfeldt 2025-04-10 12:22:23 -07:00
  • b65f45e408
         chore(robots.json): adds imgproxy crawler Cory Dransfeldt 2025-04-10 10:12:51 -07:00
  • 49e58b1573
         Merge pull request #100 from fbartho/fb/fix-perplexity-users Glyn Normington 2025-04-05 17:32:19 +01:00
  • c6f308cbd0
         PR Feedback: log special-case, comment consistency Frederic Barthelemy 2025-04-05 09:01:52 -07:00
  • 5f5a89c38c
         Fix html-mangled hyphen in Perplexity-Users Frederic Barthelemy 2025-04-04 17:34:14 -07:00
  • 6b0349f37d
         fix python complaining about f-string syntax Frederic Barthelemy 2025-04-04 15:20:30 -07:00
  • 8dc36aa2e2
         Update robots.txt Kyle Buckingham 2025-04-01 15:23:28 -07:00
  • ae8f74c10c
         Update robots.json Kyle Buckingham 2025-04-01 15:22:04 -07:00
  • b3d057f93f
         Update claude bots Kyle Buckingham 2025-04-01 15:13:51 -07:00
  • 5b8650b99b Update from Dark Visitors ai.robots.txt 2025-03-29 00:54:10 +00:00
  • c249de99a3 Update from Dark Visitors dark-visitors 2025-03-28 00:54:28 +00:00
  • ec18af7624
         Revert "Merge pull request #91 from deyigifts/perplexity-user" Cory Dransfeldt 2025-03-27 12:51:22 -07:00
  • 6851413c52 Merge pull request #94 from ThomasLeister/feature/implement-nginx-configuration-snippet-export ai.robots.txt 2025-03-27 19:49:15 +00:00
  • dba03d809c
         Merge pull request #94 from ThomasLeister/feature/implement-nginx-configuration-snippet-export Glyn Normington 2025-03-27 19:49:05 +00:00
  • 68d1d93714 Merge pull request #91 from deyigifts/perplexity-user ai.robots.txt 2025-03-27 19:29:30 +00:00
  • 1183187be9
         Merge pull request #91 from deyigifts/perplexity-user Cory Dransfeldt 2025-03-27 12:29:21 -07:00
  • 7c3b5a2cb2
         Add tests for Nginx config generator Thomas Leister 2025-03-27 16:12:18 +01:00
  • d79ca19f38 Add Lightpanda due to its AI/LLM focus Katrin Leinweber 2025-03-27 17:59:09 +01:00
  • 4f3f4cd0dd
         Add assembled version of nginx-block-ai-bots.conf file Thomas Leister 2025-03-27 12:28:50 +01:00
  • 5a312c5f4d
         Mention Nginx config feature in README Thomas Leister 2025-03-27 12:28:11 +01:00
  • da85207314
         Implement new function "json_to_nginx" which outputs an Nginx configuration snippet Thomas Leister 2025-03-27 12:27:09 +01:00
  • 84f4ed85a6
         Update robots.json katagaki 2025-03-26 13:59:27 +09:00
  • 35a461ff18
         Update robots.json katagaki 2025-03-26 13:58:58 +09:00
  • 9f506ac0aa
         Add Kagibot to robots.txt katagaki 2025-03-26 13:52:10 +09:00
  • 6ecfcdfcbf
         Update perplexity bot deyigifts 2025-03-24 14:16:57 +08:00
  • e5bf972187
         Added AhrefsBot theRealProHacker 2025-02-28 13:40:37 +01:00
  • 3be8edc5e4
         Added bing bots (see https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0) theRealProHacker 2025-02-28 13:31:02 +01:00
  • 33288d3e23
         Added SemrushBot/7~bl theRealProHacker 2025-02-28 13:20:19 +01:00
  • 5e7c3c432f
         Merge pull request #83 from glyn/81-doc-testing Cory Dransfeldt 2025-02-19 09:19:44 -08:00
  • 9f41d4c11c
         Merge pull request #84 from sideeffect42/tests-workflow Glyn Normington 2025-02-18 19:42:55 +00:00
  • 8a74896333 Add workflow to run tests on pull request or push to main Dennis Camera 2025-02-18 10:23:40 +01:00
  • 1d55a205e4 Document testing in README Glyn Normington 2025-02-18 05:08:28 +00:00
  • 8494a7fcaa
         Merge pull request #80 from sideeffect42/htaccess-allow-robots_txt Glyn Normington 2025-02-18 16:42:36 +00:00
  • c7c1e7b96f robots.py: Make executable Dennis Camera 2025-02-18 10:15:10 +01:00
  • 17b826a6d3 Update tests and convert to stock unittest Dennis Camera 2025-02-18 10:13:27 +01:00
  • 0bd3fa63b8 table-of-bot-metrics.md: Escape robot names for Markdown table Dennis Camera 2025-02-18 10:12:04 +01:00
  • a884a2afb9 .htaccess: Make regex in RewriteCond safe Dennis Camera 2025-02-17 21:00:57 +01:00
  • c0d418cd87 .htaccess: Allow robots access to /robots.txt Dennis Camera 2025-02-17 21:00:57 +01:00
  • abfd6dfcd1 Update from Dark Visitors dark-visitors 2025-02-17 00:53:32 +00:00
  • 693289bb29 chore: add Brightbot 1.0 v1.26 ai.robots.txt 2025-02-16 21:37:52 +00:00
  • a9ec4ffa6f
         chore: add Brightbot 1.0 Cory Dransfeldt 2025-02-16 13:36:39 -08:00
  • 03aa829913
         Merge pull request #79 from always-be-testing/main Glyn Normington 2025-02-16 04:33:40 +00:00
  • 5b13c2e504
         add more concise message about verified bots always-be-testing 2025-02-15 11:22:10 -05:00
  • af87b85d7f include return after heading always-be-testing 2025-02-14 12:39:08 -05:00
  • f99339922f grammar update and include syntax for verified bot condition always-be-testing 2025-02-14 12:36:33 -05:00
  • e396a2ec78 forgot to include heading always-be-testing 2025-02-14 12:31:20 -05:00
  • 261a2b83b9 update README to inclide list of ai bots Cloudflare considers verified always-be-testing 2025-02-14 12:26:19 -05:00
  • bebffccc0c Update from Dark Visitors v1.25 dark-visitors 2025-02-02 00:52:50 +00:00
  • 89d4c6e5ca Merge pull request #73 from nisbet-hubbard/patch-8 ai.robots.txt 2025-02-01 10:51:01 +00:00
  • f9e2c5810b
         Merge pull request #73 from nisbet-hubbard/patch-8 Glyn Normington 2025-02-01 10:50:50 +00:00
  • 05b79b8a58
         Update robots.json nisbet-hubbard 2025-01-27 19:41:03 +08:00
  • 9c060dee1c Update from Dark Visitors v1.24 dark-visitors 2025-01-21 00:49:22 +00:00
  • 6c552a3daa Merge pull request #71 from jsheard/patch-1 ai.robots.txt 2025-01-20 17:45:42 +00:00
  • f621fb4852
         Merge pull request #71 from jsheard/patch-1 Glyn Normington 2025-01-20 17:45:29 +00:00
  • 7427d96bac
         Update robots.json Joshua Sheard 2025-01-20 10:59:02 +00:00
  • 81cc81b35e
         Merge pull request #68 from MassiminoilTrace/main Glyn Normington 2025-01-20 07:33:54 +00:00
  • 4f03818280 Removed if condition and added a little comments Massimo Gismondi 2025-01-20 06:51:06 +01:00
  • a9956f7825 Removed additional sections Massimo Gismondi 2025-01-20 06:50:48 +01:00
  • 33c38ee70b
         Update README.md Massimo Gismondi 2025-01-20 06:28:32 +01:00
  • 52241bdca6
         Update README.md Massimo Gismondi 2025-01-20 06:27:56 +01:00
  • 013b7abfa1
         Update README.md Massimo Gismondi 2025-01-20 06:27:02 +01:00
  • 70fd6c0fb1
         Add mention of htaccess in readme Massimo Gismondi 2025-01-20 06:25:07 +01:00
  • 5aa08bc002
         Add Crawlspace Joshua Sheard 2025-01-19 22:03:50 +00:00
  • d65128d10a
         Removed paragraph in favour of future FAQ.md Massimo Gismondi 2025-01-18 12:41:09 +01:00
  • 1cc4b59dfc
         Shortened htaccess instructions Massimo Gismondi 2025-01-18 12:40:03 +01:00
  • 8aee2f24bb
         Fixed space in comment Massimo Gismondi 2025-01-18 12:39:07 +01:00
  • b455af66e7 Adding clarification about performance and code comment Massimo Gismondi 2025-01-17 21:42:08 +01:00
  • 189e75bbfd Adding usage instructions Massimo Gismondi 2025-01-17 21:25:23 +01:00