Commit graph

Select branches

Hide pull requests

main

missing-dependency

semrush-bots

#1

#10

#100

#102

#105

#107

#109

#110

#112

#113

#114

#115

#116

#118

#119

#121

#122

#123

#125

#126

#128

#129

#130

#130

#131

#132

#133

#134

#136

#138

#139

#14

#140

#141

#142

#143

#145

#148

#15

#150

#151

#151

#152

#153

#154

#16

#17

#18

#2

#21

#22

#24

#25

#27

#28

#29

#31

#32

#33

#35

#36

#37

#38

#39

#41

#42

#43

#45

#46

#47

#49

#50

#51

#54

#55

#56

#57

#58

#6

#61

#62

#63

#64

#65

#66

#67

#68

#71

#73

#79

#8

#80

#83

#84

#86

#91

#92

#94

#95

#95

#97

#98

v1

v1.1

v1.10

v1.11

v1.12

v1.13

v1.14

v1.15

v1.16

v1.17

v1.18

v1.19

v1.2

v1.20

v1.21

v1.22

v1.23

v1.24

v1.25

v1.26

v1.27

v1.28

v1.29

v1.3

v1.30

v1.31

v1.32

v1.33

v1.34

v1.35

v1.36

v1.37

v1.4

v1.5

v1.6

v1.7

v1.8

v1.9

Mono Color

• 933aa6159d Implementing htaccess generation Massimo Gismondi 2025-01-07 11:02:29 +01:00
• b7f908e305
  Merge pull request #66 from fabianegli/patch-1 v1.23 Glyn Normington 2025-01-07 03:54:40 +00:00
• ec454b71d3 Merge pull request #67 from Nightfirecat/semrushbot ai.robots.txt 2025-01-06 20:51:56 +00:00
• 565dca3dc0
  Merge pull request #67 from Nightfirecat/semrushbot Cory Dransfeldt 2025-01-06 12:51:43 -08:00
• 143f8f2285
  Block SemrushBot Jordan Atwood 2025-01-06 12:34:38 -08:00
• 8e98cc6049
  Merge pull request #61 from glyn/improve-naming Cory Dransfeldt 2025-01-06 08:10:47 -08:00
• 30ee957011
  bail when NO changes are staged Fabian Egli 2025-01-06 12:05:42 +01:00
• 83cd546470
  allow Action to succeed even if no changes were made Fabian Egli 2025-01-06 11:39:41 +01:00
• 0777befd36
  Create .htaccess Erdem 2025-01-06 02:56:07 +03:00
• 15fc333a70
  Update robots.txt Erdem 2025-01-06 02:38:12 +03:00
• ca8620e28b Merge pull request #63 from glyn/push-paths v1.22 ai.robots.txt 2025-01-05 05:05:20 +00:00
• b9df958b39
  Merge pull request #63 from glyn/push-paths Glyn Normington 2025-01-05 05:05:01 +00:00
• c01a684036 Convert robots.json more frequently Glyn Normington 2025-01-05 05:03:50 +00:00
• d2be15447c
  Merge pull request #62 from ai-robots-txt/missing-dependency Glyn Normington 2025-01-05 01:46:27 +00:00
• 9e372d0696 Ensure dependency installed missing-dependency Glyn Normington 2025-01-05 01:45:33 +00:00
• 996b9c678c Improve job name Glyn Normington 2025-01-04 05:28:41 +00:00
• e4c12ee2f8 Rename in test code Glyn Normington 2025-01-04 05:03:48 +00:00
• 3a43714908 Rename Python code Glyn Normington 2025-01-04 04:55:34 +00:00
• 2036a68c1f Update from Dark Visitors v1.21 dark-visitors 2024-12-04 00:55:50 +00:00
• 24666e8b15
  Merge pull request #58 from fabianegli/fabianegli-restore-attribution v1.20 Glyn Normington 2024-11-29 09:05:16 +00:00
• eb8e1a49b5 Revert "specify file encodings in tests" fabianegli 2024-11-29 09:02:47 +01:00
• b64284d684 restore correct attribution logic to before PR #55 fabianegli 2024-11-26 09:41:46 +01:00
• bd38c30194 specify file encodings in tests fabianegli 2024-11-26 09:12:11 +01:00
• aff37825ce
  Create azure-functions-app-python.yml Tyler Hawthorne 2024-11-25 09:14:53 -05:00
• db9643b29b
  Create python-publish.yml Tyler Hawthorne 2024-11-25 09:14:18 -05:00
• 609ddca392 Updated from new robots.json dark-visitors 2024-11-24 00:57:06 +00:00
• 37065f9118 Update from Dark Visitors dark-visitors 2024-11-24 00:57:05 +00:00
• 58985737e7 Updated from new robots.json dark-visitors 2024-11-19 16:46:21 +00:00
• 584e66cb99
  Merge pull request #56 from glyn/40-exclude-facebookexternalhit Cory Dransfeldt 2024-11-19 08:46:05 -08:00
• 80002f5e17 Allow facebookexternalhit Glyn Normington 2024-11-19 03:33:45 +00:00
• 71db599b41
  Merge pull request #55 from norwd/feature/add-robots.txt-file-to-release Glyn Normington 2024-11-13 01:39:11 +00:00
• e8f0784a00
  Explicitly use release tag for checkout Y. Meyer-Norwood 2024-11-13 10:26:37 +13:00
• 94ceb3cffd
  Add authentication for gh command Y. Meyer-Norwood 2024-11-11 13:04:55 +13:00
• adfd4af872
  Create upload-robots-txt-file-to-release.yml Y. Meyer-Norwood 2024-11-11 12:58:40 +13:00
• d50615d394 Improve formatting Glyn Normington 2024-11-10 01:06:13 +00:00
• 2c88909be3 Fix formatting Glyn Normington 2024-11-10 01:02:18 +00:00
• 6f58ddc623
  Merge pull request #54 from glyn/rationale Glyn Normington 2024-11-10 00:58:29 +00:00
• 9295b6a963 Clarify our rationale Glyn Normington 2024-11-09 04:45:47 +00:00
• 9e06cf3bc9 Updated from new robots.json v1.19 dark-visitors 2024-10-29 00:52:12 +00:00
• bc0a0ad0e9 Update from Dark Visitors dark-visitors 2024-10-29 00:52:12 +00:00
• fe5f407673 Update from Dark Visitors dark-visitors 2024-10-27 00:54:47 +00:00
• a66b16827d
  Merge pull request #51 from fabianegli/php-to-python-plus-tests Adam Newbold 2024-10-22 21:32:58 -04:00
• 3ab22bc498 make conversions and updates separately triggerable fabianegli 2024-10-19 19:56:41 +02:00
• 6ab8fb2d37 no more failure when run without network fabianegli 2024-10-19 19:11:01 +02:00
• 7e2b3ab037 rename action fabianegli 2024-10-19 19:09:34 +02:00
• 0c05461f84 simplify repo and added some tests fabianegli 2024-10-19 13:06:34 +02:00
• 6bb598820e ignore venv fabianegli 2024-10-18 23:24:13 +02:00
• d62cab66c5
  Merge pull request #50 from glyn/fix-typo Glyn Normington 2024-10-19 04:43:09 +01:00
• 6a359e7fd7 Fix typo and trigger rerun of main job ai.robots.txt 2024-10-19 03:43:00 +00:00
• 38a388097c Fix typo and trigger rerun of main job Glyn Normington 2024-10-19 04:42:27 +01:00
• 83c8603071
  Merge pull request #49 from glyn/php-diagnostics Glyn Normington 2024-10-19 04:34:53 +01:00
• a80bd18fb8 Dump out file contents in PHP script ai.robots.txt 2024-10-19 03:34:29 +00:00
• bdf30be7dc Dump out file contents in PHP script Glyn Normington 2024-10-19 04:33:46 +01:00
• 4d47b17c45
  Merge pull request #47 from fabianegli/fabianegli-patch-1 Glyn Normington 2024-10-19 02:58:05 +01:00
• faf81efb12 Daily update from Dark Visitors dark-visitors 2024-10-19 01:17:15 +00:00
• 25adc6b802
  log git repository status Fabian Egli 2024-10-19 00:28:41 +02:00
• b584f613cd
  add some signposts to the log Fabian Egli 2024-10-19 00:13:09 +02:00
• b3068a8d90
  add some signposts Fabian Egli 2024-10-19 00:12:25 +02:00
• a46d06d436
  log changes made by the action in main.yml Fabian Egli 2024-10-19 00:04:15 +02:00
• cfaade6e2f
  log the diff in the update action daily_update.yml Fabian Egli 2024-10-19 00:01:15 +02:00
• 04f630f7f8
  Merge pull request #45 from glyn/faq-update Cory Dransfeldt 2024-10-18 06:35:47 -07:00
• 898c8ab82d
  Merge pull request #46 from isagalaev/case-insensitive-sorting Glyn Normington 2024-10-18 07:57:56 +01:00
• 7bb5efd462
  Sort the content case-insensitively before dumping to JSON Ivan Sagalaev 2024-10-17 21:08:43 -04:00
• e6bb7cae9e Augment the "why" FAQ Glyn Normington 2024-10-17 12:27:05 +01:00
• b229f5b936 Re-order the FAQ Glyn Normington 2024-10-17 12:25:54 +01:00
• b1491d2694 Daily update from Dark Visitors dark-visitors 2024-10-09 01:17:37 +00:00
• 9be286626d Merge pull request #43 from lxjv/main ai.robots.txt 2024-10-08 02:30:17 +00:00
• 01993b98c3
  Merge pull request #43 from lxjv/main Glyn Normington 2024-10-08 03:30:07 +01:00
• dc15afe847
  Update robots.json with Claude respect link Laker Turner 2024-10-07 17:38:01 +01:00
• 6da804e826 chore: add ISSCyberRiskCrawler v1.18 ai.robots.txt 2024-09-30 23:50:18 +00:00
• 9c2394f23b
  chore: add ISSCyberRiskCrawler Cory Dransfeldt 2024-09-30 16:25:20 -07:00
• 6d9ce1d62a chore: add sidetrade bot v1.17 ai.robots.txt 2024-09-28 20:58:18 +00:00
• 6a988be27f
  chore: add sidetrade bot Cory Dransfeldt 2024-09-28 13:58:00 -07:00
• 632e9d6510 Daily update from Dark Visitors ai.robots.txt 2024-09-28 01:17:19 +00:00
• 7851cea4fd Daily update from Dark Visitors dark-visitors 2024-09-27 01:18:04 +00:00
• 75343c790e
  Merge pull request #38 from urvish-p80/main Glyn Normington 2024-09-27 01:26:04 +01:00
• 44d975c799 Merge pull request #42 from commoncrawl/main ai.robots.txt 2024-09-27 00:21:49 +00:00
• 2f67e77ddb
  Merge pull request #42 from commoncrawl/main Glyn Normington 2024-09-27 01:21:37 +01:00
• a6de89e6bd feat: make CCBot entry more accurate Greg Lindahl 2024-09-26 21:41:28 +00:00
• 60bdfa7eb3
  Merge pull request #41 from cityrolr/patch-1 Cory Dransfeldt 2024-09-24 12:53:52 -07:00
• af05890b07
  Update README.md Julian Mair 2024-09-23 23:27:27 +02:00
• d2cd37442c
  Update table-of-bot-metrics.md Michael Davey 2024-09-23 22:26:05 +01:00
• c1e6265ef4
  Update robots.json Michael Davey 2024-09-23 18:03:19 +01:00
• 2c68b9a88b
  Update robots.txt Michael Davey 2024-09-23 17:59:44 +01:00
• 0106d4b15a
  Add additional resource - README.md Urvish Patel 2024-09-23 08:19:27 -04:00
• 1abf68b107
  Adding dataprovider.com's spider Gregory Hammond 2024-09-14 23:24:43 +00:00
• 6b8d7f5890 Daily update from Dark Visitors ai.robots.txt 2024-09-09 01:16:21 +00:00
• 5963cbf9f7 Daily update from Dark Visitors dark-visitors 2024-09-08 01:19:31 +00:00
• b15b8062ce
  Merge pull request #36 from cramforce/patch-1 Glyn Normington 2024-09-08 01:26:07 +01:00
• 809851ae88
  Add instructions for AI bot blocking on Vercel Malte Ubl 2024-09-07 15:59:25 -07:00
• 1c1b423684 chore: add iaskspider/2.0 v1.16 ai.robots.txt 2024-09-07 02:05:43 +00:00
• 8373294404
  chore: add iaskspider/2.0 Cory Dransfeldt 2024-09-06 19:05:26 -07:00
• b30ca5f193
  Merge pull request #35 from nisbet-hubbard/patch-7 Cory Dransfeldt 2024-09-02 18:40:57 -07:00
• fb5c995243 Daily update from Dark Visitors ai.robots.txt 2024-09-03 01:12:57 +00:00
• 7151f6c569 Removing previously generated files ai.robots.txt 2024-09-03 01:12:56 +00:00
• cc18b8617c
  Update main.yml nisbet-hubbard 2024-09-03 07:48:48 +08:00
• c9325c9e18 Daily update from Dark Visitors ai.robots.txt 2024-09-02 01:15:07 +00:00
• 567bd00aec Removing previously generated files ai.robots.txt 2024-09-02 01:15:07 +00:00
• 543e993b08 Daily update from Dark Visitors ai.robots.txt 2024-09-01 01:24:53 +00:00
• 01589718df Removing previously generated files ai.robots.txt 2024-09-01 01:24:52 +00:00