Plugging the proxy into curl, Python, Node and scrapers

Short version: every tool wants the same four things, the host, port, username and password from your dashboard. Get one curl command working first, then copy the same values into your code.

Step 1: prove it in curl

curl -sS --proxy "http://HOST:PORT" --proxy-user 'USERNAME:PASSWORD' https://api.ipify.org; echo

You should get an IP that isn't yours. If curl fails, fix that before touching your scraper: see Got a 407 or a 403? Here's what they mean.

Step 2: the same values in your tool

Python Requests

import requests

proxy = "http://USERNAME:PASSWORD@HOST:PORT"
r = requests.get("https://api.ipify.org", proxies={"http": proxy, "https": proxy}, timeout=30)
print(r.text)

Playwright (Python): keep the credentials out of server, or Chromium ignores them.

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(proxy={"server": "http://HOST:PORT", "username": "USERNAME", "password": "PASSWORD"})
    page = browser.new_page()
    page.goto("https://api.ipify.org")
    print(page.inner_text("body"))
    browser.close()

Scrapy: set it per request in meta.

import scrapy

PROXY = "http://USERNAME:PASSWORD@HOST:PORT"


class IpSpider(scrapy.Spider):
    name = "ip"

    def start_requests(self):
        yield scrapy.Request("https://api.ipify.org", meta={"proxy": PROXY})

    def parse(self, response):
        self.logger.info(response.text)

Gotchas that bite everyone once

• Use http:// in the proxy URL, even for https:// sites. The traffic to the site stays encrypted inside the tunnel. An https:// proxy URL gives you "wrong version number" errors.
• Special characters in the password (@, :, /, #) break the URL. Percent-encode them or pass the credentials separately. See Username and password: getting the proxy URL right.
• Trailing newlines. Credentials read from a .env file often end in one. Strip them.
• Connection reuse looks like a sticky IP. Sessions, clients and agents keep tunnels open, and a reused tunnel usually keeps its exit IP, even on rotating residential. Want a new IP per request? Open a new connection. Need the same IP for a whole flow? Use an ISP IP. See Rotating IPs, and keeping one IP for a login or checkout.
• Chrome ignores credentials in --proxy-server. Puppeteer needs page.authenticate(), Selenium needs an extension, Playwright takes username and password.
• Cookies and IPs travel together. A site that ties a login to an IP will notice when the cookie shows up from a new address. Keep logged-in flows on a static ISP or datacenter IP.

Full guides

Step-by-step pages with rotating and static examples, sessions and fixes for curl, Python Requests, Scrapy, Playwright, Puppeteer, Selenium, Node.js, Go, HTTPX, aiohttp, Axios, PHP, Java, C#, wget, Crawlee, scrapy-playwright, Ruby, Rust and Colly: setup guides.

The host and port shown in your dashboard always win over any example.

Setting up a browser, phone or desktop app instead? See Using the proxy in a browser, phone or app.