Scrape data from multiple pages

What I mean by scraping multiple pages

Scraping multiple pages means a bot does not stop at the first page. It walks a sequence, a list of links from a sheet, the next button on a results page, the numbered pages of a catalogue, or an endless scroll, reading the same fields off each one and writing them all into a single sheet. One page is a snippet. Many pages is a dataset.

Scraping one page is easy, the trick is the next one

Pulling data off a single page takes two steps, read it and write it. The difficulty is never that. It is everything that comes after the first page. How does the bot know there is a next page, how does it get there, and how does it stop.

There are a few shapes this takes. A list of URLs in a sheet, where the bot just visits each in turn. A next button, where it clicks through until the button is gone. Numbered pages, where it walks one to the last. And infinite scroll, where it scrolls and waits for more to load. Pick the shape your site uses, and the loop is the same idea every time, do the page, move on, stop at the end.

Who this is for

This is for anyone whose data is spread across pages, not sitting on one. Scraping a whole catalogue, a multi-page search result, a directory, a feed. If you have ever scraped one page and then wondered how to get the other forty, this is that. No-coders and coders both, since you can build it without code and drop into code when you want.

How I'd approach it

Start by spotting the shape. If you have the URLs, put them in a sheet and loop them, that is the simplest. If not, find the next-page control, a button, a page number, or the scroll, and have the bot use it until there is no more. Read the same fields on every page and append them to one sheet. Get one page right, then let it run the rest.

Find the shape, loop it, append to one sheet. I would lay out the first draft with Build with description.

Scrape multiple pages from a description

Describe the pages in plain words in the Chrome extension and it builds the steps for you. Give it a few short lines, check what it made, and run it. Explore no-code.

Scrape multiple pages in code

Build with code. You do not have to write the script by hand. Describe the scrape to our Claude skill and it generates a ready-to-run Node script you own, then debugs it with you, fixing a selector that stopped matching or a step that stalls, until the run is clean. Prefer to write it yourself? Explore the code tool. Either way, the script looks like this.

1import { AxiomApi } from "axiom-api";2 3const axiom = new AxiomApi(process.env.AXIOM_API_KEY);4 5await axiom.browserOpen();6try {7  // Walk the numbered pages one at a time8  for (let pageNum = 1; pageNum <= 5; pageNum++) {9    await axiom.goto(`https://example.com/products?page=${pageNum}`);10    const rows = await axiom.scrape(".item .name");11    console.log(rows); // append these to your sheet12    await axiom.wait(2000); // a steady pace13  }14} finally {15  await axiom.browserClose();16}17 

+

Build with a Claude skill

Build no-code or code bots with a skill.

Add the Claude skill and describe how the pages are linked. It builds the bot for you, no-code or code, looping the pages into one sheet.

Install the skillLearn more

What can you scrape?

Any set of pages, into one sheet. A couple of cases worth knowing first.

What I'd watch out for

The loop is where multi-page scrapes go wrong, so a few things matter. Here is what I would watch for.

Know how it ends

The hardest part of a loop is stopping it. Decide the end condition up front, the last link, the missing next button, a page count, so the bot does not run forever or miss the last page.

A list of URLs is easiest

If you can get the page links into a sheet, do that. Looping a known list is simpler and sturdier than chasing a next button. Build the URL list first where you can.

Infinite scroll needs patience

A feed that loads as you scroll needs the bot to scroll, wait for new items, and check it actually got more before scrolling again. Without the wait, it scrapes the same screen twice.

Pace and append

Put a pause between pages so you are not a flood, and append as you go rather than holding everything to the end, so a stop midway still leaves you the pages you got. Pick fields with the selector tool.