Scrape data behind button clicks

What I mean by scraping behind button clicks

Scraping behind button clicks means a bot reveals the data before it reads it. A lot of pages hold their content back until you act, a load-more button that adds rows, a show-more link that expands text, a tab that swaps the table, an accordion that opens a detail. The bot clicks the control, waits for what appears, and then scrapes it, so the data you only see after a click ends up in the sheet too.

Some data only shows up after a click

A plain scrape reads what is on the page. The problem is that a lot of what you want is not on the page yet. It is one click away, behind a load-more, a view-details, a tab. Read the page without clicking and you get the first slice and miss the rest.

So the move is to click first, then read. Find the control that reveals the data, have the bot use it, and wait for the new content to actually appear before scraping. If it is a load-more, click it until it is gone, then read everything. If it is a row you expand, open it, read it, close it, move on. The trick is patience, the data is there, it just arrives a moment after the click.

Who this is for

This is for anyone stuck scraping a page that hides half its data behind a button. A catalogue with load-more, a table split across tabs, a list where each row expands for the detail you actually want. If a plain scrape keeps coming back half empty, this is why, and this is the fix. No-coders and coders both, since you can build it without code and drop into code when you want.

How I'd approach it

Find the control that reveals the data, and decide what kind it is. A load-more that adds to the same page, click it until it disappears, then scrape the lot. A tab or filter that swaps the content, click each one and scrape after it settles. A row that expands, open it, read the detail, move to the next. Always wait for the new content to appear before reading.

Click to reveal, wait, then read. I would lay out the first draft with Build with description.

Scrape behind clicks from a description

Describe the click in plain words in the Chrome extension and it builds the steps for you. Give it a few short lines, check what it made, and run it. Explore no-code.

Scrape behind clicks in code

Build with code. You do not have to write the script by hand. Describe the scrape to our Claude skill and it generates a ready-to-run Node script you own, then debugs it with you, fixing a selector that stopped matching or a step that stalls, until the run is clean. Prefer to write it yourself? Explore the code tool. Either way, the script looks like this.

1import { AxiomApi } from "axiom-api";2 3const axiom = new AxiomApi(process.env.AXIOM_API_KEY);4 5await axiom.browserOpen();6try {7  await axiom.goto("https://example.com/catalogue");8  await axiom.click("button.load-more");9  await axiom.wait(1500); // let the new rows load10  const rows = await axiom.scrape(".item");11  console.log(rows); // write these to your sheet12} finally {13  await axiom.browserClose();14}15 

+

Build with a Claude skill

Build no-code or code bots with a skill.

Add the Claude skill and describe the button and what it reveals. It builds the bot for you, no-code or code, clicking to reveal the data and reading it into a sheet.

Install the skillLearn more

What can you scrape?

Whatever a click reveals, into a sheet. A couple of cases worth knowing first.

What I'd watch out for

The click and the read have to stay in step, so timing is everything. Here is what I would watch for.

Wait for what the click reveals

The data appears a moment after the click, not instantly. Wait for the new content before reading, or you will scrape the page as it was before the click landed.

Know when to stop clicking

A load-more loop needs an end. Stop when the button disappears, when no new rows appear, or at a sensible count, so the bot does not click into nothing forever.

Read after, not during

Let all the clicking finish and the page settle, then read once, rather than reading between clicks. It is simpler and you will not miss or double rows.

Pick the control reliably

Buttons move and change label. Pick the control with the selector tool so the bot still finds it after a small change.