377 commits

This branch

This branch

All branches

Author	SHA1	Message	Date
Corbin Davenport	fe98e41546	Update Perplexity information	2024-06-21 22:17:45 -04:00
Corbin Davenport	20dc4e16ad	Update releases feed links	2024-06-21 22:02:54 -04:00
Corbin Davenport	149a72be0c	Add bot info for omgili, peer39, and youbot	2024-06-21 21:43:41 -04:00
Corbin Davenport	ec4610a118	Add information for Google, Meta, and img2dataset bots	2024-06-21 20:49:25 -04:00
Cory Dransfeldt	4163ca92a5	chore: clean up unnecessary ai.txt	2024-06-21 08:58:44 -07:00
Cory Dransfeldt	89de2d2d91	chore: resolve conflict	2024-06-20 08:12:24 -07:00
Cory Dransfeldt	a980ace544	chore: update readme	2024-06-20 08:11:38 -07:00
Cory Dransfeldt	a90ee5e9f0	chore: clean up bots and narrow scope	2024-06-20 08:09:21 -07:00
Cory Dransfeldt	4b8499d718	Merge pull request #16 from nisbet-hubbard/patch-1 ... Remove potential false positives	2024-06-20 07:57:16 -07:00
nisbet-hubbard	56c2285462	Update robots.txt	2024-06-20 11:31:08 +08:00
Cory Dransfeldt	429336d725	fix: resolve conflict	2024-06-15 09:26:14 -07:00
Cory Dransfeldt	3f65a93891	chore: keeps Applebot-Extended in favor of Applebot as the latter is simply for search	2024-06-15 09:25:17 -07:00
Cory Dransfeldt	ca730a81ca	Merge pull request #15 from christiansievers/main ... Update table-of-bot-metrics.md	2024-06-13 10:50:10 -07:00
Christian Sievers	ce0b99124b	Update table-of-bot-metrics.md ... add link to Apple docs and description for Applebot-Extended	2024-06-13 14:17:31 +02:00
Cory Dransfeldt	1a4868f1a2	chore: update readme	2024-06-12 19:05:15 -07:00
Cory Dransfeldt	b5d4d5bc74	chore: update readme	2024-06-12 19:01:23 -07:00
Cory Dransfeldt	548a6686ab	chore: add rss badge	2024-06-12 18:56:09 -07:00
Cory Dransfeldt	1ac25b9135	chore: propagate applebot-extended out	2024-06-12 18:53:26 -07:00
Cory Dransfeldt	c546dc048c	Merge pull request #14 from chriskirknielsen/patch-1 ... Block Applebot-Extended	2024-06-12 18:51:09 -07:00
Christopher Kirk-Nielsen	39363fc813	Block Applebot-Extended ... Per [Apple's docs](https://support.apple.com/en-us/119829#datausage) ([via Matthew Bogart](https://matthewbogart.com/@matt/112605297864483766))	2024-06-12 16:43:12 -04:00
Cory Dransfeldt	d464f260e0	Merge branch 'main' of github.com:ai-robots-txt/ai.robots.txt	2024-06-10 13:23:55 -07:00
Cory Dransfeldt	fd10fbda94	chore: add cloudflare waf expression	2024-06-10 13:23:47 -07:00
Cory Dransfeldt	7a5a21fdad	chore: update bots table	2024-05-29 09:21:31 -07:00
Cory Dransfeldt	2ca4154d4e	chore: update perplexity entry	2024-05-29 09:14:35 -07:00
Cory Dransfeldt	e152c110c5	Merge pull request #10 from kg583/main ... Fix Markdown in `table-of-bot-metrics.md`	2024-05-20 19:43:29 -07:00
KG	04688e56fb	Update table-of-bot-metrics.md ... Fix Markdown	2024-05-16 18:25:46 -05:00
Cory Dransfeldt	696d2acef8	chore: add cohere-ai	2024-05-06 10:39:07 -07:00
Cory Dransfeldt	6940db2412	chore: add anthropic bots to table	2024-05-06 10:36:16 -07:00
Cory Dransfeldt	ec8e10a56c	chore: add bytespider to table	2024-05-06 10:34:43 -07:00
Cory Dransfeldt	262fe71ad9	chore: add ccbot data to table	2024-05-06 10:32:05 -07:00
Cory Dransfeldt	dea035365f	chore: add Diffbot and scoopit	2024-05-05 14:50:04 -07:00
Cory Dransfeldt	29e0799332	chore: add img2dataset to table-of-bot-metrics.md	2024-04-22 09:30:00 -07:00
Cory Dransfeldt	5025939e5d	chore: add img2dataset to edge func	2024-04-22 09:28:39 -07:00
Cory Dransfeldt	8ddd2a0bb3	chore: add img2dataset to nginx.conf	2024-04-22 09:28:21 -07:00
Cory Dransfeldt	2b77b0b0ad	chore: add img2dataset to apache.conf	2024-04-22 09:27:41 -07:00
Cory Dransfeldt	118ec00126	chore: add img2dataset to robots.txt	2024-04-22 09:26:59 -07:00
Cory Dransfeldt	6f1a28e17a	chore: update nginx	2024-04-16 08:31:36 -07:00
Cory Dransfeldt	1664980617	chore: comment	2024-04-14 10:04:25 -07:00
Cory Dransfeldt	1e91744ed2	feat: add edge function	2024-04-14 10:03:40 -07:00
Cory Dransfeldt	9a92568266	Merge branch 'main' of github.com:ai-robots-txt/ai.robots.txt	2024-04-14 09:56:30 -07:00
Cory Dransfeldt	9d56001747	chore: move server configs	2024-04-14 09:56:23 -07:00
Cory Dransfeldt	d05a6956b8	Create FUNDING.yml	2024-04-12 11:40:44 -07:00
Cory Dransfeldt	0d9b75b227	chore: move additional resources to wiki	2024-04-12 09:33:19 -07:00
Cory Dransfeldt	9be338094b	chore: populate gptbot	2024-04-10 10:53:16 -07:00
Cory Dransfeldt	ef5d847e65	chore: add gitignore; move logo file	2024-04-10 10:44:07 -07:00
Cory Dransfeldt	cc5966af5f	chore: populate ChatGPT-User	2024-04-10 10:39:34 -07:00
Cory Dransfeldt	0a8952b34f	chore: add applebot	2024-04-10 10:18:51 -07:00
Cory Dransfeldt	b402ec681c	chore: add anthropic + amazonbot	2024-04-10 10:15:22 -07:00
Cory Dransfeldt	46b0532daf	Merge pull request #8 from mattlinares/feature/add-table-of-crawler-detail ... Add Google Gemini response on AbsBot-Google for bot metrics table.	2024-04-10 09:08:22 -07:00
Cory Dransfeldt	db06035714	Merge branch 'main' into feature/add-table-of-crawler-detail	2024-04-10 09:07:45 -07:00