Puppeteer with authenticated proxies: the setup that works

Puppeteer drives Chrome, and Chrome's --proxy-server flag only accepts a host and port, not a username and password. Put credentials in the flag and they're silently ignored. The fix is a two-part setup: the proxy address goes on the command line, the credentials go through page.authenticate().

You'll need your crawlproxies username and password from the generator. Browsers work best with the HTTP gateway: geo.crawlproxies.com:8080 on Residential.

The basic setup

npm i puppeteer

import puppeteer from "puppeteer";

const browser = await puppeteer.launch({
  args: ["--proxy-server=http://geo.crawlproxies.com:8080"],
});

const page = await browser.newPage();
await page.authenticate({ username: "USERNAME-country-us", password: "PASSWORD" });

await page.goto("https://ipinfo.io/json");
console.log(await page.$eval("body", (el) => el.innerText));

await browser.close();

Call page.authenticate() before the first navigation, and on every new page you open. It answers the proxy's authentication challenge for that page only.

Choosing the location

Location and session options live in the username, so the browser flag never changes:

await page.authenticate({
  username: "USERNAME-country-de-city-berlin",
  password: "PASSWORD",
});

Countries are two-letter codes; cities and US states are lowercase without spaces. The geo-targeting guide has the full list of options.

One identity per browser context

Chrome applies --proxy-server to the whole browser, but the username can differ per browser context, and the username is where the session lives. Chrome remembers proxy credentials within a context, so two pages in the same context end up sending the same username. Give each identity its own context instead: it gets its own cookies, storage and credential cache.

async function identity(browser, sessionId) {
  const context = await browser.createBrowserContext();
  const page = await context.newPage();
  await page.authenticate({
    username: `USERNAME-country-us-session-${sessionId}-time-1800`,
    password: "PASSWORD",
  });
  return { context, page };
}

const a = await identity(browser, "acct1");
const b = await identity(browser, "acct2");
// a.page and b.page leave from different sticky IPs, with separate cookies

Close a context with await a.context.close() when you're done with it. For complete isolation (separate accounts on a strict site, for example), launch a separate browser per identity.

Rotation happens per connection. Chrome keeps connections to a proxy open and reuses them, so several pages may share an exit IP even with a rotating username. When a batch of pages must come from a new IP, open a fresh browser context for it and close the context afterwards.

Save bandwidth

Proxy plans are billed per gigabyte, and images, video and fonts are usually most of a page's weight. Turn on request interception and drop them:

await page.setRequestInterception(true);
page.on("request", (req) => {
  if (["image", "media", "font"].includes(req.resourceType())) req.abort();
  else req.continue();
});

Leave scripts and stylesheets alone; many sites need them to render, and some flag clients that never load them.

Timeouts and retries

Real residential connections vary in speed, so give navigation some headroom and retry the occasional failure:

page.setDefaultNavigationTimeout(60_000);

for (let attempt = 1; attempt <= 3; attempt++) {
  try {
    await page.goto("https://example.com/", { waitUntil: "domcontentloaded" });
    break;
  } catch (err) {
    console.log(`attempt ${attempt} failed: ${err.message}`);
  }
}

Troubleshooting

Prefer Playwright? It takes the credentials directly in the launch options: see Playwright with authenticated proxies.