restorewebpages

Wayback machine api tutorial for developers

3,076 words 14 min read Updated Sep 2026

Master the Wayback Machine CDX API, Availability API, and Save Page Now API. Includes Python + curl code examples for SEO and recovery use cases.

What Is the Wayback Machine API?

An API (Application Programming Interface) is a way for one program to ask another program a question and get a structured answer, usually in JSON, instead of scraping a web page designed for human eyes. The Wayback Machine is a digital archive of the public web operated by the Internet Archive, a non-profit founded in 1996. Instead of browsing its calendar interface, you can ask it precise questions and let your own code do the paging.

Two words you will see constantly in this field. A snapshot is one archived copy of one URL, captured at one moment. Its timestamp is a 14-digit number in the form YYYYMMDDhhmmss, always in UTC, so 20230601120000 means 12:00:00 noon on 1 June 2023. Every timestamp in every API response on this page uses that format.

Three interfaces cover almost all recovery, audit, and research work. The CDX API is the index of everything the archive holds for a given URL or pattern. The Availability API answers a single question fast: is this one URL on file, and what is the closest capture to a date I care about? The Save Page Now service captures a live page on demand so it is on file for the future. All three are free for ordinary anonymous use, and the reference documentation lives at archive.org/help/wayback_api.php.

One habit matters more than any parameter. The Internet Archive runs these services on donated infrastructure and throttles traffic to keep them available. Send a descriptive User-Agent header identifying your tool, keep concurrency low, and cache what you retrieve. A crawl that behaves politely today is still available next year; one that hammers the endpoint gets a 429 and then nothing at all.

If you need captures the Internet Archive simply does not hold, it is worth knowing that other archiving services exist. Our top 5 Wayback Machine alternatives in 2025 compares coverage and capture rules across the main options.

CDX API: Find Every Snapshot of a URL

The CDX server is the brain behind the archive. Its base endpoint is https://web.archive.org/cdx/search/cdx, and it takes a URL pattern plus options describing what you want back. Here is the smallest useful query, runnable as-is in a terminal:

curl 'https://web.archive.org/cdx/search/cdx?url=example.com/*&output=json&limit=10'

You get a JSON array. The first element is a header row of field names; every element after it is one capture, with its values in that same order. The default field set is urlkey, timestamp, original, mimetype, statuscode, digest, length. The digest is a content hash and the length is the stored record size, and both are useful for spotting duplicate captures of identical bytes.

Why each parameter exists:

  • url is the pattern you want matched. Without the trailing /* you get captures of one exact page; with it you get every page on the host. URL-encode it with urllib.parse.quote rather than pasting it raw, or query strings will break the request.
  • matchType controls how the pattern is interpreted. exact is the default and matches one URL. prefix matches everything starting with the string. host matches any path on that host. domain additionally matches subdomains, which is what you usually want for a site audit.
  • output=json returns a parseable array. Use output=txt if you only want timestamp original pairs, and output=cdx for tab-separated fields. Ask for the smallest field set you can work with; it is faster and easier to read.
  • fl chooses the fields. fl=timestamp,original,statuscode,mimetype is a good default for recovery work. Set fl=original with filter=urlkey when you want one row per unique URL rather than one row per capture.
  • filter narrows results with a field:value test, and you can repeat it to combine tests. filter=statuscode:200 keeps only successful captures. filter=mimetype:application/pdf keeps only PDFs, which is the single most useful filter for document recovery.
  • collapse deduplicates. The archive often stores the same file many times over years, so collapse=digest returns one row per unique content hash, collapse=urlkey one per URL, and collapse=timestamp:8 one per calendar day.
  • from and to bound the time range, accepting anything from 8 to 14 digits. from=2020&to=2023 gives you 2020 through 2023 inclusive, because short values are padded out to the start of that period.
  • limit caps the number of rows. A negative value such as limit=-50 asks for the final 50 rows of the result set, which is how you pull the most recent captures without reading everything first.
  • showResumeKey=true adds a resume token to the end of the response. Feed it back as resumeKey and the server continues exactly where it stopped.

That last pair deserves emphasis. Offset-style paging is fragile here: if the archive grows while you page, rows shift and you silently miss or duplicate records. The resumeKey mechanism is a server-held cursor, so use it for anything beyond a few hundred rows.

curl 'https://web.archive.org/cdx/search/cdx?url=example.com&matchType=domain&output=json&fl=timestamp,original,statuscode,mimetype&filter=statuscode:200&collapse=digest&from=2015&to=2024&limit=-25&showResumeKey=true'

Two CDX responses deserve special mention. A 403 usually means a malformed or disallowed query, and a 503 usually means the index is busy or you are asking too fast. Both are reasons to slow down rather than to retry immediately in a tight loop.

Availability API: Check If a URL Was Archived

When all you need is a yes-or-no on one URL, the CDX index is overkill. The Availability API lives on a different host, https://archive.org/wayback/available, and takes two parameters. url is required. timestamp is optional and asks for the capture closest to that moment rather than the closest to now, so you can ask what a page looked like in 2019 by passing timestamp=20190101.

curl 'https://archive.org/wayback/available?url=example.com/report.pdf&timestamp=2019'

The response is a small JSON object with a url key and an archived_snapshots object containing a closest entry. That entry carries status, available, url, and timestamp. If available is false, the archive has no capture for that URL. The status field tells you whether the archived copy was itself an error page, which matters: a 200 capture of a site’s 404 page is technically successful and practically useless.

Two limitations to plan around. It handles exactly one URL, with no wildcard or pattern matching, and it returns only the single closest capture rather than the full history. Also expect the returned snapshot url to be on the web.archive.org host and sometimes without an http:// or https:// prefix, so normalise it before you store or hand it to another tool. A sensible pattern is Availability as a fast pre-flight, CDX as the authoritative follow-up.

Save Page Now API: Capture a Page on Demand

Read, analysis, and recovery all start from the same problem: a page that is not on file. Save Page Now solves that by submitting the live URL for capture now. The service is at https://web.archive.org/save/, and you can either POST the URL to that endpoint or request https://web.archive.org/save/<URL> directly. A browser submitting a capture is redirected to the new snapshot, so the same call works from a script.

curl -X POST 'https://web.archive.org/save/' -d 'url=https://example.com/new-page'

Three behaviours will shape your expectations. First, the capture respects the site’s robots.txt, so a disallowed path returns an error rather than a snapshot. Second, an authenticated page, a paywalled article, and anything behind a login will capture as whatever an anonymous visitor sees, usually the login form. Third, the capture records the page and the assets needed to render it, but a page that builds itself with JavaScript may archive with less content than a browser shows you, because the crawler may not execute the same scripts you do.

The service is throttled, and the exact anonymous ceiling is not published in a form you should treat as a contract. In practice, an unauthenticated client that fires requests in a loop will be stopped, and the documented remedy for higher volume is an API key from the Internet Archive, passed with the request as the X-Archive-Api-Key header. Check the current Save Page Now developer documentation for the exact header name and any changed requirements before you build on it, because this part of the service has changed more than the read APIs.

Save Page Now is also the right tool for capturing a PDF you already know about. Request the PDF URL directly and the archive stores the binary alongside the surrounding page, giving you a dated, timestamped copy for your own records.

Authentication + Rate Limits (Avoiding Error 429)

HTTP 429 means Too Many Requests: the server understood the request and refused it because you are asking faster than it is willing to answer. It is not a bug in your code and not a permanent block, which is why retrying after a pause usually works. The trigger is volume per client per period, and the archive adjusts it based on load.

Be honest about the numbers here. Plenty of blog posts quote exact figures for anonymous versus keyed CDX requests, and those figures are folklore: the Internet Archive publishes guidance rather than a fixed quota, and the effective limit moves with server load. Treat any specific number, including the ones in older versions of our own 429 error guide, as a starting point to test rather than a promise to design around.

What does work reliably:

  • Start at roughly one request per second, single-threaded, and only tighten that if you genuinely need to. If you see 429s, back off rather than pushing harder.
  • Retry with exponential backoff: wait one second, then two, then four, adding a little jitter so parallel workers do not resynchronise. Honour a Retry-After header if the response includes one.
  • Request an API key for anything scheduled, repeated, or user-facing. It is the difference between a script that works in a demo and a script that works at 3am.
  • Cache aggressively on your side. A CDX query for a fixed URL and date range returns the same answer tomorrow unless someone captures a new snapshot, so there is rarely a reason to ask twice.
  • Ask for fewer rows. Narrowing with from, to, filter, or collapse reduces server work and reduces your chance of hitting a ceiling.
  • Set a real User-Agent. If something is genuinely misbehaving, an identifiable agent is far easier to sort out than an anonymous one.

Python Code Examples

The examples below use only the Python 3 standard library: urllib for HTTP, json for parsing, and time for pacing. Nothing to install with pip, and they run unchanged on any server with Python 3. The first helper queries the CDX index and follows the resume key so a long result set is not silently truncated.

import json
import time
import urllib.parse
import urllib.request

CDX = "https://web.archive.org/cdx/search/cdx"
UA = "site-audit/1.0 (contact: https://example.com/contact)"


def cdx_rows(params, max_rows=5000):
    query = dict(params)
    query.setdefault("output", "json")
    query.setdefault("fl", "timestamp,original,statuscode,mimetype")
    collected = 0
    while True:
        target = CDX + "?" + urllib.parse.urlencode(query)
        request = urllib.request.Request(target, headers={"User-Agent": UA})
        with urllib.request.urlopen(request, timeout=60) as response:
            payload = json.loads(response.read().decode("utf-8", "replace"))
        if not payload:
            return
        resume = None
        if query.get("showResumeKey") == "true" and isinstance(payload[-1], list):
            resume = payload[-1][0]
            body = payload[1:-1]
        else:
            body = payload[1:]
        header = payload[0]
        for row in body:
            if not isinstance(row, list):
                continue
            yield dict(zip(header, row))
            collected += 1
            if collected >= max_rows:
                return
        if not resume:
            return
        query["resumeKey"] = resume
        time.sleep(1.0)


params = {
    "url": "example.com",
    "matchType": "domain",
    "filter": "statuscode:200",
    "collapse": "digest",
    "from": "20150101000000",
    "to": "20241231235959",
    "showResumeKey": "true",
}

rows = list(cdx_rows(params))
print("unique captures:", len(rows))
print("newest:", rows[-1] if rows else "none")

The second helper is the Availability API as a plain yes-or-no check, useful as a pre-flight before you commit to a bigger download job.

import json
import urllib.parse
import urllib.request

UA = "site-audit/1.0 (contact: https://example.com/contact)"


def closest_snapshot(url, when=None):
    params = {"url": url}
    if when:
        params["timestamp"] = when
    target = "https://archive.org/wayback/available?" + urllib.parse.urlencode(params)
    request = urllib.request.Request(target, headers={"User-Agent": UA})
    with urllib.request.urlopen(request, timeout=30) as response:
        return json.loads(response.read().decode("utf-8", "replace"))


for candidate in ["example.com/report.pdf", "example.com/pricing/"]:
    found = closest_snapshot(candidate)
    closest = found.get("archived_snapshots", {}).get("closest", {})
    if closest.get("available"):
        print(candidate, "->", closest["timestamp"], closest["status"])
    else:
        print(candidate, "-> nothing on file")

The third downloads captures politely, with backoff, and writes each file under a name taken from its original URL so nothing is overwritten twice.

import os
import time
import urllib.parse
import urllib.request

UA = "site-audit/1.0 (contact: https://example.com/contact)"
DELAY = 1.5
OUT = "restored"


def snapshot_url(timestamp, original, mode="id_"):
    return "https://web.archive.org/web/" + timestamp + mode + "/" + original.lstrip("/")


def safe_name(original):
    tail = original.split("?")[0].rstrip("/").split("/")[-1] or "index.html"
    return urllib.parse.quote(tail, safe="._-()")


def download(url, path, attempts=4):
    for attempt in range(attempts):
        try:
            request = urllib.request.Request(url, headers={"User-Agent": UA})
            with urllib.request.urlopen(request, timeout=60) as response:
                data = response.read()
            if not data:
                raise ValueError("empty body from archive")
            with open(path, "wb") as handle:
                handle.write(data)
            return len(data)
        except Exception as exc:
            wait = DELAY * pow(2, attempt)
            print("retrying in", round(wait, 1), "s:", exc)
            time.sleep(wait)
    return 0


os.makedirs(OUT, exist_ok=True)
for row in rows[:200]:
    url = snapshot_url(row["timestamp"], row["original"])
    path = os.path.join(OUT, safe_name(row["original"]))
    size = download(url, path)
    print(path, size)
    time.sleep(DELAY)

Two details in that last script are worth copying. The id_ marker asks the archive to serve the stored bytes without rewriting links, which is what you want for anything you intend to treat as a preserved original. And the retry loop treats an empty body as a failure rather than a success, because the archive sometimes holds a record for a capture whose payload did not come back intact, and an empty file is worse than no file.

Common Use Cases: SEO Audits, Recovery, Research

SEO audits. A migration loses more than URLs. Query the old domain with matchType=domain and collapse=digest, and you get a deduplicated list of every page that ever existed, which you can compare against the new site’s live sitemap. The gap is your redirect map and your list of content that needs rewriting. Doing this by hand across a few hundred pages is exactly the kind of work the CDX API exists to remove.

Recovery. When a site goes down, find the last capture before the takedown or the domain expiry, pull the HTML and assets, and reassemble. Our guide to using the Wayback Machine to restore websites walks through the assembly side once the API has given you the list of what to fetch.

Bulk download. When the goal is a complete local copy rather than a single page, the workflow changes: you need the full inventory, a safe download cadence, and a record of which captures failed so you can retry only those. Our step-by-step bulk download guide covers that process, and the scripts in this post are the inventory and download stages of it.

Research. A domain queried without a date range gives you a timeline: what existed in which year, how long each URL survived, and which content was duplicated across versions. Because every row carries a timestamp and a digest, you can measure how often a page genuinely changed rather than being re-captured unchanged.

Monitoring. A weekly Availability check on your own key URLs is a cheap early warning. If a page you published last week is suddenly not reachable, an archived copy of the working version may already exist before anyone notices the outage.

These workflows share one practical limit: the archive only holds what its crawlers actually fetched. A page excluded by robots.txt, an internal staff area, or a section behind a login was often never captured at all, and no API can return what was never stored. If the pages you need fall outside that coverage, our general restoration walkthrough describes the wider options, and our restoration service handles sites where coverage gaps mean the API alone is not enough.

FAQ: Wayback Machine API

Is the Wayback Machine API free?

Yes, for ordinary anonymous use. The read APIs, meaning CDX and Availability, are free and unauthenticated, and Save Page Now is free as well for individual captures. What is not free of cost is the burden you place on donated infrastructure, so high-volume callers are expected to slow down, cache results, and register for an API key. The practical rule is that a handful of queries a minute is a perfectly reasonable thing to do anonymously, whereas a scheduled job that fires every five seconds forever will eventually be throttled no matter how well intentioned it is.

How many requests per minute can I make?

There is no fixed published number you can rely on, and that is the honest answer. The Internet Archive throttles by observed load rather than by a documented per-key quota, so the effective ceiling for anonymous traffic, and for keyed traffic, moves over time. Anyone quoting a precise figure is describing what they measured on one day. What you can do is start near one request per second, back off on any 429, cache aggressively, and register for a key if your work is ongoing.

Can I download every snapshot of a domain?

You can enumerate every snapshot the archive holds for a domain, yes. A CDX query with matchType=domain and no date bounds returns the full index, and adding showResumeKey=true lets you page through it without gaps. Downloading all of them is a different proposition, because a busy archive can hold hundreds of thousands of rows for one domain, and the replay host throttles downloads more tightly than it throttles index queries. In practice you almost always want collapse=digest to take one copy of each distinct file, and a date filter to cover the period you actually care about.

More guides