Restoring an archive and scraping a live site are the same craft pointed in different directions. Both come down to one question: what is standing between you and the bytes, and what is the cheapest tool that gets past it? This is the escalation ladder, the recipes that work, and the mistakes that cost real hours.
Most scraping advice starts with a heavyweight framework. That is backwards. Every rung below is cheaper, faster and less brittle than the one under it, so start at the top and only descend when something actually blocks you.
view-source:, you are done.
cost: milliseconds · breaks on: JS rendering, bot checksIf a page loads fine when you open it yourself but 403s from a script, the problem is not the page. It is that your script does not look like a browser, does not carry your cookies, or is coming from a datacenter IP. Escalate one rung rather than adding headers one at a time.
This is the most boring item on the page and the one that wastes the most time. Links copied out of email, search results, or social platforms are frequently not the link — they are a tracking redirector wearing the link as a costume.
A URL is wrapped if the host is a search or mail provider with a /url path, or if it carries utm_*, gclid, fbclid, ved, usg or similar parameters. Unwrapping means taking the q= or url= parameter, URL-decoding it, and dropping the tracking junk.
# the wrapper, as pasted
https://www.google.com/url?q=https%3A%2F%2Fexample.com%2Fpage&sa=E&usg=AOvVaw
# what you actually want
https://example.com/page
from urllib.parse import urlparse, parse_qs, unquote
def unwrap(u):
p = urlparse(u)
if p.netloc.endswith("google.com") and p.path == "/url":
q = parse_qs(p.query)
return unquote((q.get("q") or q.get("url") or [u])[0])
return uA nightly site auditor was fed a list of URLs harvested from email. Every single check came back 403. The sites were fine. The auditor was faithfully fetching the redirector, which does not like robots, and reporting the redirector's refusal as a site outage. One unwrap call at ingest fixed a report that had been wrong for weeks.
Make it mechanical, not a judgment call: the moment a URL enters a pipeline — pasted, scraped, parsed out of a feed — run it through the unwrapper before you fetch, store, or print it. Already-clean URLs pass through untouched, so it is always safe to call.
The "Verify you are human" interstitial defeats curl and headless browsers completely. The winning combination is a real browser channel plus a persistent profile directory. You solve the checkbox once by hand; the clearance cookie lands in the profile and carries to every subsequent page in the run.
import { chromium } from 'playwright';
// persistent profile + REAL Edge — this pairing is the whole trick
const ctx = await chromium.launchPersistentContext('./profile', {
headless: false,
channel: 'msedge', // installed Edge; fall back to 'chrome'
args: ['--disable-blink-features=AutomationControlled'],
});
const page = ctx.pages()[0];
await page.goto(ROOT); // you solve the checkbox HERE, once
// poll for a known marker. do NOT reload.
while (!(await page.evaluate(m => document.body.innerText.includes(m), MARKER))) {
await page.waitForTimeout(1200);
}
// cookie now sticks in the profile — loop everything else with zero clicks
for (const url of urls) { await page.goto(url); /* … */ }Reloading re-arms the checkbox and cancels the click the human just made. The result is an infinite "still verifying" loop that looks like the challenge is unbeatable. It is not — your retry logic is fighting the user. Poll the page content instead and let it settle on its own.
| Setup | What happens |
|---|---|
| Bundled Chromium, fresh context | Re-challenged on every navigation. Fourteen pages means fourteen clicks, several of which fail. |
| Bundled Chromium, persistent profile | Better, but the browser fingerprint still trips detection on strict sites. |
| Real Edge/Chrome, fresh context | Passes the fingerprint check, then loses the cookie the moment the context closes. |
| Real Edge/Chrome, persistent profile | Solve once, then clean runs. |
Some sites block automation and sit behind a subscription. Fighting that from a script is a losing game and a bad idea. The correct move is to stop scripting and start reading — in the browser where your session already lives.
Browser-automation extensions that drive your actual Chrome profile solve this cleanly: the page opens exactly as it does for you, because it is you. Nothing to install, no credentials handled, no cookie extraction.
A regional business journal published its events calendar behind both a paywall and aggressive bot filtering. Scripted fetches returned nothing usable and the pages were written off as unreachable. Opening the same URLs in an already-logged-in browser returned the full calendar — six events with dates, times and venue addresses — on the first try, in under a minute. The block was never technical. It was a mismatch between the tool and the situation.
Two boundaries worth holding to. Read what your subscription entitles you to read, and extract facts — dates, addresses, names — rather than republishing the prose. Facts are not the publisher's property; their writing is.
Archived pages are rewritten on the way out. The archive injects its own toolbar and rewrites asset URLs to point back at itself, so naive scraping yields the archive's version of an image rather than the original bytes.
The fix is a one-character modifier in the snapshot URL. Appending if_ to the timestamp asks for the raw resource with no rewriting:
# toolbar-wrapped, rewritten
https://web.archive.org/web/20050612093000/http://old-site.com/logo.gif
# the original bytes, untouched
https://web.archive.org/web/20050612093000if_/http://old-site.com/logo.gifRelated modifiers follow the same shape: id_ for the identity payload, cs_ for stylesheets, js_ for scripts. When a restored page looks right but the images are subtly wrong or missing, this is almost always the reason. The restoration guide covers the full recovery workflow.
Run browsers on a desktop, not on your web server. This is a rule worth treating as absolute, for three reasons that each bite independently.
A headed browser will happily consume the RAM your production sites need.
Datacenter IP ranges are pre-judged. The same request that succeeds from home 403s from a cloud host.
The one-click challenge pattern needs a human at a screen. There is nobody at the server.
A sitemap fetcher failed against dozens of sites from a cloud host and worked perfectly against the same URLs from a laptop on home broadband. Nothing was wrong with the code, the user agent, or the target sites. Egress reputation alone accounted for every failure. Before debugging headers, try the request from a different network.
Once a page is rendered, resist the urge to write CSS selectors against the markup. Modern sites ship generated class names that change on the next deploy, and a selector-based parser silently returns nothing when that happens.
For text-heavy pages, document.body.innerText is far more durable. It gives you what a reader sees, in reading order, with the navigation and scripts stripped. Then anchor on content landmarks rather than structure:
// brittle — dies on the next redesign
document.querySelectorAll('.EventCard__title--x7f2k')
// durable — anchors on words a human would recognize
const text = document.body.innerText;
const blocks = text.split('VIEW THIS AWARD ON ITS OWN PAGE');
// then: CATEGORY in caps → winner → FINALISTS → address | NeighborhoodSentinel lines, all-caps headings and repeated labels survive redesigns because they are editorial decisions, not implementation details. When you do need structured data, check first whether the page already embeds JSON-LD or hydration state — parsing a JSON blob out of a script tag beats scraping rendered markup every time.
Extraction is the easy half. The half that embarrasses you is trusting what came back.
Aggregators, venue calendars and directory sites carry tentative holds, stale rows and entries that were never real. If scraped data is going onto a public page, spot-check anything uncertain against the primary source — the organization's own site, not a listing about it.
A batch of events pulled from a single venue's calendar looked plausible enough to publish. Checking that venue's page directly showed six of the seven were not listed there at all, and the one that was had a date almost a year off. Everything was formatted correctly and completely wrong. Dropping a row costs nothing; publishing a wrong date sends someone somewhere on the wrong day.
A practical habit: have your extraction carry a confidence marker and the URL it came from. Then re-fetch the source for anything below full confidence, and for anything that contradicts what you already had. When two sources disagree, the organization's own page wins.
None of the techniques above are exotic, and none of them are permission. A few lines worth holding:
Start with curl. When it fails, ask what is blocking you rather than reaching for a bigger hammer. If the answer is "a challenge," use a real browser with a persistent profile. If it is "a login you legitimately hold," use your own browser. If it is "a datacenter IP," move the job to your desk. And unwrap every URL on the way in.