Export a paginated book catalog

When to use this

Use this when a catalog spans several pages and failed runs must preserve progress. Stagehand extracts typed records and resolves the Next action; a stable pagination marker proves when to stop. If your application already has reliable product locators or an API, use those for the fixed parts of the job.

Browser	Languages	Output
Local or Browserbase	TypeScript, Python, Go	`out/checkpoint.json`, `out/catalog.json`

Goal and output

Export the Books to Scrape catalog into out/catalog.json. Each entry has a nonempty title, displayed price, and availability. The file also reports pages and count. The job saves validated pages atomically to out/checkpoint.json. The final catalog appears only after pagination ends.

Agent prompt

Export a complete paginated catalog with Stagehand.

Prerequisites and inputs

Add BROWSERBASE_API_KEY and OPENAI_API_KEY to .env. The TypeScript runner can also use a local browser by setting BROWSER_ENV=LOCAL.
CATALOG_URL defaults to the two-page mystery category on Books to Scrape. MAX_PAGES defaults to 2 and accepts 1 through 100. The default completes a small category. For the full catalog, set CATALOG_URL=https://books.toscrape.com/ and raise MAX_PAGES to 50. NEXT_SELECTOR defaults to li.next a; change it when adapting the target. OUT_DIR defaults to out.
TypeScript needs Node.js 22.18+ and pnpm; Python needs Python 3.11+ and uv; Go needs Go 1.26+.

Check out and run

The source is packages/examples/cookbooks/paginated-catalog. Clone this folder with the overview command, then run one version:

TypeScript
Python
Go

cd packages/examples/cookbooks/paginated-catalog/typescript
cp .env.example .env
pnpm install --frozen-lockfile
pnpm start

cd packages/examples/cookbooks/paginated-catalog/python
cp .env.example .env
uv sync --locked
uv run --locked python main.py

cd packages/examples/cookbooks/paginated-catalog/go
cp .env.example .env
set -a; . ./.env; set +a
go run .

How the job works

Open the catalog

Launch one cloud browser and navigate to CATALOG_URL. Keep the same page for all extraction and navigation calls.

Extract typed records

Check the current URL against a visited set. Extract all visible books into a typed schema with nonempty fields.

Resolve pagination

Check NEXT_SELECTOR for a Next link. Its absence ends pagination. If it exists, ask observe() for the action, pass that action to act(), and require navigation. An empty observation while Next exists is an error.

Checkpoint progress

Save validated pages to the checkpoint before navigation. On rerun, revisit the last saved URL to find Next without extracting it again. Fail before extracting beyond the total MAX_PAGES budget.

Publish the complete catalog

Write the final file atomically after natural completion. Deduplicate identical title, price, and availability tuples. Close Stagehand and the browser.

const result = await stagehand.extract(
  "Extract each book's title, price, and availability.",
  pageSchema,
  { page },
);

if (await page.locator("li.next a").count()) {
  const next = await stagehand.observe("Find the Next pagination link", { page });
  await stagehand.act(next.data[0], { page });
}

result = await stagehand.extract(
    "Extract each book's title, price, and availability.",
    CatalogPage,
    page=page,
)

if await page.locator("li.next a").count():
    next_page = await stagehand.observe("Find the Next pagination link", page=page)
    await stagehand.act(next_page.data[0], page=page)

instruction := "Extract each book's title, price, and availability."
result, err := stagehand.Extract[CatalogPage](ctx, client, instruction,
    &stagehand.StagehandClientExtractOptions{Page: page})
if err != nil { return err }

nextCount, err := page.Locator("li.next a").Count(ctx)
if err != nil { return err }
if nextCount > 0 {
    instruction := "Find the Next pagination link"
    next, err := client.Observe(ctx, &instruction,
        &stagehand.StagehandClientObserveOptions{Page: page})
    if err != nil { return err }
    _, err = client.Act(ctx, stagehand.ObservedAction(next.Data[0]),
        &stagehand.StagehandClientActOptions{Page: page})
    if err != nil { return err }
}

These excerpts show extraction and the next-page action. The runnable project also extracts records, saves checkpoints, and checks origin, cycles, page limits, and navigation results.

Recover an interrupted export

Run again with the same CATALOG_URL and OUT_DIR. Increase MAX_PAGES when the previous run stopped at its budget. The checkpoint binds progress to the source URL and rejects corrupt records or navigation to another origin. Completed reruns rebuild the final file from the checkpoint without extracting pages again.

Use one writer per output directory. Concurrent runs can overwrite each other’s checkpoint.

Use a new OUT_DIR for a fresh snapshot or changed source. Checkpoints preserve previously extracted records; they do not refresh changing prices. The example deduplicates identical records. For a real catalog, add a product ID to the schema and use it as the deduplication key. The same checkpoint format works across TypeScript, Python, and Go.

Expected result and failure checks

A completed run prints Saved N books from P pages to out/catalog.json. The file contains pages, count, and books; count equals the array length. Set MAX_PAGES=1 against a multipage catalog to confirm the limit fails with a checkpoint and without producing a new final catalog. A one-page target stops when the configured Next marker is absent. A cycle or unchanged URL fails instead of looping.

Failure	Recovery
Page budget reached	Raise `MAX_PAGES` and rerun with the same output directory
Empty extract or failed Next action	Inspect the printed session link, fix the extraction or navigation, and rerun
Corrupt checkpoint or changed source	Use a fresh `OUT_DIR`; keep the original checkpoint for investigation
Redirect or pagination cycle	Fix the target before rerunning; do not disable the origin and cycle checks

Run credential-free recovery tests with pnpm test, uv run --locked python -m unittest, or go test ./... from the matching language folder.

Adapt to your site

Change CATALOG_URL, NEXT_SELECTOR, and the extraction schema together. Use a stable product ID as the deduplication key when different products can share displayed fields. Keep checkpoint/source validation, the page budget, and atomic final writes. Use a fresh OUT_DIR for each new snapshot. Extraction, observation, and deployment. For a recorded data-collection job, see the showcase.

First steps

The basics

Configuration

Add-ons

Integrations

Best practices

Cookbooks

Migration guide

SDK reference

Export a paginated book catalog

When to use this

Goal and output

Agent prompt

Prerequisites and inputs

Check out and run

How the job works

Recover an interrupted export

Expected result and failure checks

Adapt to your site

​When to use this

​Goal and output

​Agent prompt

​Prerequisites and inputs

​Check out and run

​How the job works

​Recover an interrupted export

​Expected result and failure checks

​Adapt to your site

​Related

When to use this

Goal and output

Agent prompt

Prerequisites and inputs

Check out and run

How the job works

Recover an interrupted export

Expected result and failure checks

Adapt to your site

Related