What do scrapers and indexing engines have to do with it? Is it normal for them to run a full headless browser in incognito mode?
I was under the impression that they're either just doing straight HTTP requests for HTML only... or they're running a full headless browser in normal mode.
So I'm not getting what's different here?
Sites have a long history of serving up different content to different users, e.g. to paying users, or blocking certain countries based on content contracts. It's certainly not part of the "web's contract" that scrapers get paywalled content they haven't paid for.
I was under the impression that they're either just doing straight HTTP requests for HTML only... or they're running a full headless browser in normal mode.
So I'm not getting what's different here?
Sites have a long history of serving up different content to different users, e.g. to paying users, or blocking certain countries based on content contracts. It's certainly not part of the "web's contract" that scrapers get paywalled content they haven't paid for.