157 commits

This branch

This branch

All branches

Author	SHA1	Message	Date
Cory Dransfeldt	d464f260e0	Merge branch 'main' of github.com:ai-robots-txt/ai.robots.txt	2024-06-10 13:23:55 -07:00
Cory Dransfeldt	fd10fbda94	chore: add cloudflare waf expression	2024-06-10 13:23:47 -07:00
Cory Dransfeldt	7a5a21fdad	chore: update bots table	2024-05-29 09:21:31 -07:00
Cory Dransfeldt	2ca4154d4e	chore: update perplexity entry	2024-05-29 09:14:35 -07:00
Cory Dransfeldt	e152c110c5	Merge pull request #10 from kg583/main ... Fix Markdown in `table-of-bot-metrics.md`	2024-05-20 19:43:29 -07:00
KG	04688e56fb	Update table-of-bot-metrics.md ... Fix Markdown	2024-05-16 18:25:46 -05:00
Cory Dransfeldt	696d2acef8	chore: add cohere-ai	2024-05-06 10:39:07 -07:00
Cory Dransfeldt	6940db2412	chore: add anthropic bots to table	2024-05-06 10:36:16 -07:00
Cory Dransfeldt	ec8e10a56c	chore: add bytespider to table	2024-05-06 10:34:43 -07:00
Cory Dransfeldt	262fe71ad9	chore: add ccbot data to table	2024-05-06 10:32:05 -07:00
Cory Dransfeldt	dea035365f	chore: add Diffbot and scoopit	2024-05-05 14:50:04 -07:00
Cory Dransfeldt	29e0799332	chore: add img2dataset to table-of-bot-metrics.md	2024-04-22 09:30:00 -07:00
Cory Dransfeldt	5025939e5d	chore: add img2dataset to edge func	2024-04-22 09:28:39 -07:00
Cory Dransfeldt	8ddd2a0bb3	chore: add img2dataset to nginx.conf	2024-04-22 09:28:21 -07:00
Cory Dransfeldt	2b77b0b0ad	chore: add img2dataset to apache.conf	2024-04-22 09:27:41 -07:00
Cory Dransfeldt	118ec00126	chore: add img2dataset to robots.txt	2024-04-22 09:26:59 -07:00
Cory Dransfeldt	6f1a28e17a	chore: update nginx	2024-04-16 08:31:36 -07:00
Cory Dransfeldt	1664980617	chore: comment	2024-04-14 10:04:25 -07:00
Cory Dransfeldt	1e91744ed2	feat: add edge function	2024-04-14 10:03:40 -07:00
Cory Dransfeldt	9a92568266	Merge branch 'main' of github.com:ai-robots-txt/ai.robots.txt	2024-04-14 09:56:30 -07:00
Cory Dransfeldt	9d56001747	chore: move server configs	2024-04-14 09:56:23 -07:00
Cory Dransfeldt	d05a6956b8	Create FUNDING.yml	2024-04-12 11:40:44 -07:00
Cory Dransfeldt	0d9b75b227	chore: move additional resources to wiki	2024-04-12 09:33:19 -07:00
Cory Dransfeldt	9be338094b	chore: populate gptbot	2024-04-10 10:53:16 -07:00
Cory Dransfeldt	ef5d847e65	chore: add gitignore; move logo file	2024-04-10 10:44:07 -07:00
Cory Dransfeldt	cc5966af5f	chore: populate ChatGPT-User	2024-04-10 10:39:34 -07:00
Cory Dransfeldt	0a8952b34f	chore: add applebot	2024-04-10 10:18:51 -07:00
Cory Dransfeldt	b402ec681c	chore: add anthropic + amazonbot	2024-04-10 10:15:22 -07:00
Cory Dransfeldt	46b0532daf	Merge pull request #8 from mattlinares/feature/add-table-of-crawler-detail ... Add Google Gemini response on AbsBot-Google for bot metrics table.	2024-04-10 09:08:22 -07:00
Cory Dransfeldt	db06035714	Merge branch 'main' into feature/add-table-of-crawler-detail	2024-04-10 09:07:45 -07:00
Matthew Linares	aa0edf05b3	Add Google Gemini response on AbsBot-Google for bot metrics table. ... Reference: https://g.co/gemini/share/5bef4db5f87d	2024-04-10 15:40:45 +01:00
Cory Dransfeldt	b6be895994	fix: clean up syntax	2024-04-09 16:08:57 -07:00
Cory Dransfeldt	ed190d4803	fix: syntax	2024-04-09 16:08:13 -07:00
Cory Dransfeldt	0518b3e777	chore: add FriendlyCrawler to the table	2024-04-08 12:50:36 -07:00
Cory Dransfeldt	964b29d330	Merge pull request #6 from mattlinares/main ... Adds table of bot details and metrics	2024-04-08 12:48:35 -07:00
Cory Dransfeldt	e03d5f4ec2	chore: add FriendlyCrawler to nginx.conf.txt	2024-04-08 12:41:40 -07:00
Cory Dransfeldt	2d4efec184	chore: add FriendlyCrawler to apache.conf.txt	2024-04-08 12:41:19 -07:00
Cory Dransfeldt	d6d40989f4	chore: add FriendlyCrawler to robots.txt	2024-04-08 12:40:59 -07:00
Matthew Linares	048045ac6d	Adds table of bot details and metrics.	2024-04-08 20:29:45 +01:00
Cory Dransfeldt	74e4f64d89	chore: create nginx.conf.txt	2024-04-08 12:27:59 -07:00
Cory Dransfeldt	609de2af12	chore: create apache.conf.txt	2024-04-08 12:26:04 -07:00
Cory Dransfeldt	47fc45f2f9	chore: add PiplBot	2024-04-06 20:25:28 -07:00
Cory Dransfeldt	46c8c9adb3	chore: add Meltwater	2024-04-03 08:56:30 -07:00
Cory Dransfeldt	c8a6d7f02d	chore: add Seekr	2024-04-03 08:56:12 -07:00
Cory Dransfeldt	bdf10968fd	chore: add ai.txt	2024-04-01 09:56:10 -07:00
Cory Dransfeldt	d7064d23fe	chore: additional resources	2024-04-01 09:55:46 -07:00
Cory Dransfeldt	5e02ebc168	chore: update README	2024-03-29 12:14:48 -07:00
Cory Dransfeldt	2d1c4707aa	Merge pull request #2 from Explosion-Scratch/patch-1	2024-03-28 09:15:51 -07:00
--Explosion--	3e57b5ab5d	Add GoogleOther ... Used by Google to crawl for internal research and development. It’s unknown what exactly this entails, but is a generic user agent that is used when no other appropriate user agent is available. Documentation available from Google: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers	2024-03-28 10:00:58 -05:00
Cory Dransfeldt	543dfc8452	Merge pull request #1 from glyn/logo ... Add logo	2024-03-28 07:26:14 -07:00