cdransf/ai.robots.txt
1
1
Fork 0
mirror of https://github.com/ai-robots-txt/ai.robots.txt.git synced 2025-06-19 02:00:51 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 1

Update robots.json with new crawler

Browse source 
Update with Poseidon Research Crawler as found in nytimes.com/robots.txt
This commit is contained in:
Sérgio Spagnuolo 2025-06-13 17:15:13 -03:00 • committed by GitHub
parent e53d81c66d
commit b05f2fee00
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1 changed files with 7 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

7
robots.json
View file

@ -405,6 +405,13 @@
"operator": "[phind](https://www.phind.com/)",
"respect": "Unclear at this time."
},
"Poseidon Research Crawler": {
"operator": "[Poseidon Research](https://www.poseidonresearch.com)",
"description": "Lab focused on scaling the interpretability research necessary to make better AI systems possible.",
"frequency": "No explicit frequency provided.",
"function": "AI research crawler",
"respect": "Unclear at this time."
},
"QualifiedBot": {
"description": "Operated by Qualified as part of their suite of AI product offerings.",
"frequency": "No explicit frequency provided.",