Hacker News
new
|
past
|
comments
|
ask
|
show
|
jobs
|
submit
login
iririririr
26 days ago
|
parent
|
context
|
favorite
| on:
News publishers limit Internet Archive access due ...
> The AI companies won't just scrape IA once, they're keeping come back to the same pages and scraping them over and over. Even if nothing has changed.
Maybe they vibecoded the crawlers. I wish I were joking.
terminalshort
25 days ago
[–]
Isn't this just how crawlers work? How do you know if a page has changed if you don't keep visiting it?
heavyset_go
25 days ago
|
parent
[–]
HEAD requests
Findecanor
25 days ago
|
root
|
parent
[–]
Scrapers could also use the "Cache-Control", "Expires" and "If-Modified-Since" headers properly, to reduce their traffic a little, but do they?
Guidelines
|
FAQ
|
Lists
|
API
|
Security
|
Legal
|
Apply to YC
|
Contact
Search:
Maybe they vibecoded the crawlers. I wish I were joking.