Proxy Pool Design for Scraping: How Many IPs Do You Need and How to Manage Them

What a proxy pool actually is

A proxy pool is not just “a list of IPs”. It is a control system that decides:

• which IP/session to use
• how long to keep it
• when to retire it
• how to distribute load

The 4 numbers you must define

1) Concurrency (C): how many workers run in parallel

2) Requests per minute per worker (R): your pacing

3) Session length (S): sticky minutes, if used

4) Retry budget (B): max retries per request

These drive cost and block rates.

Sizing heuristic (good starting point)

For strict targets with residential proxies:

• Start with IPs = 5–20 × C
• Use sticky sessions for multi-step flows
• Rotate on failure

Then tune by measuring 429 and CAPTCHA rate.

Health checks and eviction rules

Do not keep “bad” exits in the pool.

Recommended rules:

• Mark an exit unhealthy after N consecutive failures.
• Cooldown for 10–30 minutes.
• Permanently retire exits with repeated challenge loops.

Load distribution: avoid hot spots

Even with many IPs, you can accidentally overload a small subset.

• Use round-robin with weights.
• Cap per-IP RPS.
• Add jitter.

Observability (what to log)

At minimum log:

• domain
• geo
• session ID
• exit IP
• status code
• block type (if detected)
• latency

This lets you compute:

• success rate per domain + geo
• cost per success
• top failing exits

Questions to ask any provider before you scale

• session control limits (sticky max, rotation min)
• concurrency limits per account/zone
• geo and ASN availability
• how usage is calculated (bandwidth vs requests)
• troubleshooting features (request IDs, dashboards)

Summary

A proxy pool is an engineering system. If you size and operate it correctly, residential proxies become predictable and cost-effective instead of “expensive and flaky”.