Commit graph

Select branches

Hide pull requests

main

missing-dependency

semrush-bots

#1

#10

#100

#102

#105

#107

#109

#110

#112

#113

#114

#115

#116

#118

#119

#121

#122

#123

#125

#126

#128

#129

#130

#130

#131

#132

#133

#134

#136

#138

#139

#14

#140

#141

#142

#143

#145

#148

#15

#150

#151

#151

#152

#153

#154

#16

#17

#18

#2

#21

#22

#24

#25

#27

#28

#29

#31

#32

#33

#35

#36

#37

#38

#39

#41

#42

#43

#45

#46

#47

#49

#50

#51

#54

#55

#56

#57

#58

#6

#61

#62

#63

#64

#65

#66

#67

#68

#71

#73

#79

#8

#80

#83

#84

#86

#91

#92

#94

#95

#95

#97

#98

v1

v1.1

v1.10

v1.11

v1.12

v1.13

v1.14

v1.15

v1.16

v1.17

v1.18

v1.19

v1.2

v1.20

v1.21

v1.22

v1.23

v1.24

v1.25

v1.26

v1.27

v1.28

v1.29

v1.3

v1.30

v1.31

v1.32

v1.33

v1.34

v1.35

v1.36

v1.37

v1.4

v1.5

v1.6

v1.7

v1.8

v1.9

Mono Color

• 0072b8f5f0 Removing previously generated files ai.robots.txt 2024-08-04 21:53:47 +00:00
• 9c7257e7cf
  Merge pull request #27 from jsheard/patch-1 Cory Dransfeldt 2024-08-04 14:53:36 -07:00
• 8dbbdbf44c
  Add Cloudflares first-party scraper blocking to FAQ Joshua Sheard 2024-08-04 21:38:02 +01:00
• 146fd4ffba
  Fix Imagesift user agent Joshua Sheard 2024-08-04 21:33:04 +01:00
• c7b781034e chore: restore FriendlyCrawler + ImageSift v1.12 ai.robots.txt 2024-08-04 19:29:01 +00:00
• 9a8fa66772 Removing previously generated files ai.robots.txt 2024-08-04 19:29:00 +00:00
• 1ca936ce11
  chore: restore FriendlyCrawler + ImageSift Cory Dransfeldt 2024-08-04 12:28:48 -07:00
• 8de5bc8e01 Merge pull request #25 from mirium999/add_icc_crawler ai.robots.txt 2024-08-04 01:21:56 +00:00
• 8c632e1ba4 Removing previously generated files ai.robots.txt 2024-08-04 01:21:55 +00:00
• 8d4d52cdab
  Merge pull request #25 from mirium999/add_icc_crawler Adam Newbold 2024-08-03 21:21:45 -04:00
• 5826c18909 Add ICC-Crawler Mirium999 2024-08-04 10:11:25 +09:00
• ffbad453f3 Merge pull request #24 from nisbet-hubbard/patch-5 ai.robots.txt 2024-08-03 14:27:47 +00:00
• b1907d86be Removing previously generated files ai.robots.txt 2024-08-03 14:27:46 +00:00
• 55c585e9e3
  Merge pull request #24 from nisbet-hubbard/patch-5 Cory Dransfeldt 2024-08-03 07:27:37 -07:00
• 2b56c72bac
  Update FAQ.md nisbet-hubbard 2024-08-03 14:27:25 +08:00
• b24e5cb3bb
  Update FAQ.md nisbet-hubbard 2024-08-03 14:12:50 +08:00
• 74b1502839
  Update FAQ.md nisbet-hubbard 2024-08-03 14:04:58 +08:00
• d8de1ebdd5 chore: contribution note ai.robots.txt 2024-08-02 16:32:00 +00:00
• 9d8d3de8ed Removing previously generated files ai.robots.txt 2024-08-02 16:31:59 +00:00
• 349c35eed6
  chore: contribution note Cory Dransfeldt 2024-08-02 09:31:48 -07:00
• b144225ece chore: drop in additional data ai.robots.txt 2024-08-01 22:33:23 +00:00
• 06b950bce9 Removing previously generated files ai.robots.txt 2024-08-01 22:33:23 +00:00
• b20dfec1e4
  chore: drop in additional data Cory Dransfeldt 2024-08-01 15:33:07 -07:00
• f18f0d99b9 chore: remove test data ai.robots.txt 2024-08-01 22:29:02 +00:00
• 747cc834c4 Removing previously generated files ai.robots.txt 2024-08-01 22:29:01 +00:00
• efabf3e721
  chore: remove test data Cory Dransfeldt 2024-08-01 15:25:55 -07:00
• 1fdc79dacb Adding GitHub Action Adam Newbold 2024-08-01 18:17:19 -04:00
• 17a84f2c2d
  chore: update robots table Cory Dransfeldt 2024-08-01 15:06:49 -07:00
• 6c596a50ea
  chore: move FAQ into repo v1.11 Cory Dransfeldt 2024-08-01 07:53:43 -07:00
• 6a8e7a8eb0
  Merge pull request #22 from nisbet-hubbard/patch-4 Cory Dransfeldt 2024-08-01 07:49:30 -07:00
• df89722038
  Add PetalBot (and facebookexternalhit?) nisbet-hubbard 2024-07-31 18:27:29 +08:00
• fa7b64ae4b
  chore: add Scrapy v1.10 Cory Dransfeldt 2024-07-30 10:28:46 -07:00
• 55b4505e30
  chore: add Timpibot v1.9 Cory Dransfeldt 2024-07-29 12:38:22 -07:00
• d49e860b74
  chore: add VelenPublicWebCrawler v1.8 Cory Dransfeldt 2024-07-29 12:12:42 -07:00
• 6e323554c6
  chore: add Meta-ExternalAgent v1.7 Cory Dransfeldt 2024-07-29 08:27:31 -07:00
• 1df6d3bec1
  Add meta-externalagent nisbet-hubbard 2024-07-29 20:04:59 +08:00
• 1c9810d8c5
  Add facebookexternalhit nisbet-hubbard 2024-07-29 19:37:58 +08:00
• 2972926532
  chore: add OAI-SearchBot v1.6 Cory Dransfeldt 2024-07-26 09:06:10 -07:00
• c17cae6e9d
  link to bot metrics table Glyn Normington 2024-07-17 02:28:32 +01:00
• 3692d66918
  chore: update bots table v1.5 Cory Dransfeldt 2024-07-16 12:07:01 -07:00
• af52578965
  chore: drop google adbot; add GoogleOther bots Cory Dransfeldt 2024-07-16 12:05:34 -07:00
• 570fd36ea2
  chore: update bots table Cory Dransfeldt 2024-07-10 19:47:23 -07:00
• 0ca6bce87e
  chore: add ImagesiftBot v1.4 Cory Dransfeldt 2024-07-09 17:41:32 -07:00
• 0971af19b6
  chore: peer39 unrelated to ai Cory Dransfeldt 2024-07-09 17:39:51 -07:00
• 74fa789985
  Merge pull request #18 from glyn/add-ref Cory Dransfeldt 2024-06-29 12:22:23 -07:00
• 4a8ce0b51d Add reference Glyn Normington 2024-06-29 12:20:38 +01:00
• 70e8985622
  chore: remove unused image v1.3 Cory Dransfeldt 2024-06-22 12:44:02 -07:00
• fd4ade555c
  Merge pull request #17 from corbindavenport/main Cory Dransfeldt 2024-06-22 12:42:52 -07:00
• fe98e41546
  Update Perplexity information Corbin Davenport 2024-06-21 22:17:45 -04:00
• 20dc4e16ad
  Update releases feed links Corbin Davenport 2024-06-21 22:02:54 -04:00
• 149a72be0c
  Add bot info for omgili, peer39, and youbot Corbin Davenport 2024-06-21 21:43:41 -04:00
• ec4610a118
  Add information for Google, Meta, and img2dataset bots Corbin Davenport 2024-06-21 20:49:25 -04:00
• 4163ca92a5
  chore: clean up unnecessary ai.txt Cory Dransfeldt 2024-06-21 08:58:44 -07:00
• 89de2d2d91
  chore: resolve conflict v1.2 Cory Dransfeldt 2024-06-20 08:12:24 -07:00
• a980ace544
  chore: update readme Cory Dransfeldt 2024-06-20 08:11:38 -07:00
• a90ee5e9f0
  chore: clean up bots and narrow scope Cory Dransfeldt 2024-06-20 08:09:21 -07:00
• 4b8499d718
  Merge pull request #16 from nisbet-hubbard/patch-1 Cory Dransfeldt 2024-06-20 07:57:16 -07:00
• 56c2285462
  Update robots.txt nisbet-hubbard 2024-06-20 11:31:08 +08:00
• 429336d725
  fix: resolve conflict v1.1 Cory Dransfeldt 2024-06-15 09:26:14 -07:00
• 3f65a93891
  chore: keeps Applebot-Extended in favor of Applebot as the latter is simply for search Cory Dransfeldt 2024-06-15 09:25:17 -07:00
• ca730a81ca
  Merge pull request #15 from christiansievers/main Cory Dransfeldt 2024-06-13 10:50:10 -07:00
• ce0b99124b
  Update table-of-bot-metrics.md Christian Sievers 2024-06-13 14:17:31 +02:00
• 1a4868f1a2
  chore: update readme Cory Dransfeldt 2024-06-12 19:05:15 -07:00
• b5d4d5bc74
  chore: update readme v1 Cory Dransfeldt 2024-06-12 19:01:23 -07:00
• 548a6686ab
  chore: add rss badge Cory Dransfeldt 2024-06-12 18:56:09 -07:00
• 1ac25b9135
  chore: propagate applebot-extended out Cory Dransfeldt 2024-06-12 18:53:26 -07:00
• c546dc048c
  Merge pull request #14 from chriskirknielsen/patch-1 Cory Dransfeldt 2024-06-12 18:51:09 -07:00
• 39363fc813
  Block Applebot-Extended Christopher Kirk-Nielsen 2024-06-12 16:43:12 -04:00
• d464f260e0
  Merge branch 'main' of github.com:ai-robots-txt/ai.robots.txt Cory Dransfeldt 2024-06-10 13:23:55 -07:00
• fd10fbda94
  chore: add cloudflare waf expression Cory Dransfeldt 2024-06-10 13:23:47 -07:00
• 7a5a21fdad
  chore: update bots table Cory Dransfeldt 2024-05-29 09:21:31 -07:00
• 2ca4154d4e
  chore: update perplexity entry Cory Dransfeldt 2024-05-29 09:14:35 -07:00
• e152c110c5
  Merge pull request #10 from kg583/main Cory Dransfeldt 2024-05-20 19:43:29 -07:00
• 04688e56fb
  Update table-of-bot-metrics.md KG 2024-05-16 18:25:46 -05:00
• 696d2acef8
  chore: add cohere-ai Cory Dransfeldt 2024-05-06 10:39:07 -07:00
• 6940db2412
  chore: add anthropic bots to table Cory Dransfeldt 2024-05-06 10:36:16 -07:00
• ec8e10a56c
  chore: add bytespider to table Cory Dransfeldt 2024-05-06 10:34:43 -07:00
• 262fe71ad9
  chore: add ccbot data to table Cory Dransfeldt 2024-05-06 10:32:05 -07:00
• dea035365f
  chore: add Diffbot and scoopit Cory Dransfeldt 2024-05-05 14:50:04 -07:00
• 29e0799332
  chore: add img2dataset to table-of-bot-metrics.md Cory Dransfeldt 2024-04-22 09:30:00 -07:00
• 5025939e5d
  chore: add img2dataset to edge func Cory Dransfeldt 2024-04-22 09:28:39 -07:00
• 8ddd2a0bb3
  chore: add img2dataset to nginx.conf Cory Dransfeldt 2024-04-22 09:28:21 -07:00
• 2b77b0b0ad
  chore: add img2dataset to apache.conf Cory Dransfeldt 2024-04-22 09:27:41 -07:00
• 118ec00126
  chore: add img2dataset to robots.txt Cory Dransfeldt 2024-04-22 09:26:59 -07:00
• 6f1a28e17a
  chore: update nginx Cory Dransfeldt 2024-04-16 08:20:32 -07:00
• 1664980617
  chore: comment Cory Dransfeldt 2024-04-14 10:04:25 -07:00
• 1e91744ed2
  feat: add edge function Cory Dransfeldt 2024-04-14 10:03:40 -07:00
• 9a92568266
  Merge branch 'main' of github.com:ai-robots-txt/ai.robots.txt Cory Dransfeldt 2024-04-14 09:56:30 -07:00
• 9d56001747
  chore: move server configs Cory Dransfeldt 2024-04-14 09:56:23 -07:00
• d05a6956b8
  Create FUNDING.yml Cory Dransfeldt 2024-04-12 11:40:44 -07:00
• 0d9b75b227
  chore: move additional resources to wiki Cory Dransfeldt 2024-04-12 09:33:19 -07:00
• 9be338094b
  chore: populate gptbot Cory Dransfeldt 2024-04-10 10:53:16 -07:00
• ef5d847e65
  chore: add gitignore; move logo file Cory Dransfeldt 2024-04-10 10:44:07 -07:00
• cc5966af5f
  chore: populate ChatGPT-User Cory Dransfeldt 2024-04-10 10:39:34 -07:00
• 0a8952b34f
  chore: add applebot Cory Dransfeldt 2024-04-10 10:18:51 -07:00
• b402ec681c
  chore: add anthropic + amazonbot Cory Dransfeldt 2024-04-10 10:15:22 -07:00
• 46b0532daf
  Merge pull request #8 from mattlinares/feature/add-table-of-crawler-detail Cory Dransfeldt 2024-04-10 09:08:22 -07:00
• db06035714
  Merge branch 'main' into feature/add-table-of-crawler-detail Cory Dransfeldt 2024-04-10 09:07:45 -07:00
• aa0edf05b3 Add Google Gemini response on AbsBot-Google for bot metrics table. Matthew Linares 2024-04-10 15:40:33 +01:00
• b6be895994
  fix: clean up syntax Cory Dransfeldt 2024-04-09 16:08:57 -07:00