Scrapy proxy setup

Why use proxies with Scrapy

Direct requests expose your home IP and get rate-limited fast. With Scrapy behind PinguProxy, retries and bursts come from proxy IPs instead, so targets see distributed traffic instead of one noisy client.

What you can do

• ◆Keep one static IP per job so a target can whitelist or cookie you consistently.
• ◆Rotate per request with SOCKS5h so scraping bursts look like real distributed traffic.
• ◆Pool sessions with retries for long-running jobs that survive 429s and timeouts.
• ◆Use IPv4 or IPv6 exits depending on what the target infrastructure supports.

Setup snippets

Replace username, password, and port with the credentials from your dashboard. All examples route both HTTP and HTTPS traffic through the proxy.

# middlewares.py
class PinguProxyMiddleware:
    def process_request(self, request, spider):
        request.meta["proxy"] = "http://username:password@proxy.pinguproxy.com:12933"

# settings.py
DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.PinguProxyMiddleware": 543,
}

# Rotate per request: keep the middleware but point it at the
# rotating endpoint. SOCKS5h resolves DNS on the proxy, so the exit
# IP is never leaked.
class RotatingProxyMiddleware:
    def process_request(self, request, spider):
        request.meta["proxy"] = "socks5h://username:password@proxy.pinguproxy.com:12933"

DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.RotatingProxyMiddleware": 543,
}

The port shown (12933) is the default example; your dashboard lists the exact endpoints for your proxies. SOCKS5h keeps DNS lookups on the proxy so your real DNS server never sees the target domains.

Frequently asked questions

Other setup guides