USE CASES
AI and LLM work
Models need public data from everywhere, and products need to work everywhere. Both require you to send an ordinary-looking request from any region.
Two jobs, two kinds of IP
Training-side work is collection: many requests, each very short, and a retry is fine. That suits a rotating residential pool billed by traffic.
Inference-side work is availability checking: you need to confirm your product opens in Tokyo, Frankfurt and São Paulo with acceptable latency. That needs a fixed exit, reused over time, so you can tell network problems apart from code problems.
Public data only, and put compliance in the pipeline
We support collection of publicly accessible data. That is not a disclaimer, it is cost advice: targets behind logins and paywalls sit in a different risk class, and forcing them with proxies usually burns both accounts and IPs.
- Read robots.txt and the terms before you start, and encode the disallowed paths in your crawler
- Send a User-Agent that identifies you, so problems reach a person instead of a block rule
- Rate-limit to what the target can absorb — a site you slow down is a site you cannot crawl next round
- Store the source URL and fetch time with every record; provenance and deletion both depend on those two fields
Choosing concurrency
Get one request working single-threaded and note the average response time, then size concurrency as target QPS multiplied by response seconds. Most public sites are safe between 5 and 20 QPS; past that, returns fall and block rates climb.