Trigger a web scraper from Zapier and get the data back


How to trigger a web scraper from Zapier and get the data back: install the Chrome extension, click Build from description, tell the agent what you want, and follow its prompts. A working bot is built and tested in minutes.

Here is how the automation works. To trigger a web scraper from Zapier, add a Webhooks by Zapier POST action that sends the URLs to https://lar.axiom.ai/api/v3/trigger. The cloud run takes them in a Receive data from another app step, scrapes the page, and a Trigger webhook step posts the data to a second Zap with a Catch Raw Hook trigger, which sends it on. Any Zapier trigger can start it: a new row in a sheet, a form submission, a new email, a schedule.

Author
Alex Barlow
Published

Build the Zapier scraper from a description


Describe it and click the button: the agent builds and tests the scraper in your Chrome extension; the two Zaps are set up in Zapier, in the questions below. Webhooks need a plan that includes them (pricing) and your API key (Authentication).

  1. Take the URL Zapier sends when a row is added
  2. Open it and scrape the title and price
  3. Post the title, price and row number to Zapier
Click to launch the Chrome extension. Not installed? You can create an account and try it free.Opens the Chrome extension. Free account to try.

You need a Zapier account on a paid plan, where Webhooks by Zapier is, and the Google Sheet your team pastes product URLs into. Make the second Zap's Catch Raw Hook first so you can give the agent its URL, or leave the Trigger webhook step's Endpoint empty and paste it in later. Name the automation something you can type exactly, such as Scrape product pages.

Build the Zapier scraper step by step


The steps the agent builds, as the Builder shows them: each step by its name, with the fields to set under it. Add them yourself in this order in the step-by-step builder, or open the agent's result there to adjust it.

  • 1.0 Receive data from another app
    • Test data: a row with the URL and the sheet's row number, in the shape the Zap sends
  • 2.0 Loop through data
    • repeats for every row in [webhook-data]
    • 2.1 Go to page
      • URL: the first column of [webhook-data]
    • 2.2 Get data from bot's current page
      • Select: the title and price on the page, one column each
  • 3.0 Trigger webhook
    • Endpoint: the second Zap's Catch Raw Hook URL
    • Payload: [scrape-data]

Common questions about the Zapier scraper


How do I start the scraper run from a Zap with Webhooks by Zapier?

With a Webhooks by Zapier POST action after the Zap's trigger:

  1. In Zapier, click + Create and pick the trigger that suits the job, for example Google Sheets: New Spreadsheet Row.
  2. Add an action, search for Webhooks by Zapier, and set the action event to POST.
  3. Set URL to https://lar.axiom.ai/api/v3/trigger.
  4. Set Payload Type to Json.
  5. In Data, add three rows: name key with your API key, name name with the automation's exact name, and name data with the table of URLs. For one URL from the trigger, the value is [[" followed by the trigger's URL field, then "]]. To send the row number back as well, make it [[" + the URL field + ", " + the Row ID field + "]].
  6. Leave Wrap Request In Array as No and Unflatten as Yes, then click Continue.

Data takes a name in the left box and a value in the right box. The data value is a JSON table typed into the right box with the trigger's field inserted where the URL goes. Zapier passes it through as JSON because Payload Type is Json, and sends this:

{
  "key": "your-api-key-here",
  "name": "Scrape product pages",
  "data": [["https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html", "2"]]
}

Click Test step. The trigger answers straight away:

ResponseWhat it means
"OPEN LINK IN BROWSER" with a linkThe run started. Open the link to watch it live.
"status": "queued"No cloud slot is free yet. The run starts within a few minutes.
Task not foundThe name row does not match the automation's name exactly, including case and spaces.
Invalid keyThe key row is wrong. Copy the key again from the extension's settings.

How do I catch the scraped data in a second Zap?

With a Catch Raw Hook as the second Zap's trigger:

  1. In Zapier, click + Create and add a trigger. Search for Webhooks by Zapier and set the trigger event to Catch Raw Hook.
  2. Copy the URL under Your webhook URL.
  3. Paste it into the Trigger webhook step's Endpoint and save the automation.
  4. Run the automation once from the builder, then click Test trigger in the Zap. The scraped rows arrive as the raw body.
  5. Add a Formatter by Zapier step: action event Text, Transform set to Split Text, Input set to Raw Body, Separator a single speech mark, Segment Index set to All (as separate fields). Each scraped value is then a field you can send to any app.
  6. End with Google Sheets: Update Spreadsheet Row, using the row number that came back to put the title and price next to the URL.

Then turn both Zaps on and fire the first Zap's trigger, for example by adding a row to the sheet: the run starts, scrapes, and the second Zap receives the data. The automation's run reports show each run, and notifications can tell you when one fails.

Can I send several URLs to the web scraper in one Zap?

Yes. The data value is a table, so [["url-1"], ["url-2"]] sends two rows and one run scrapes both. A Looping by Zapier step that posts once per URL starts one run per URL instead, which costs more runtime.

How do I stop Zapier rows without a URL starting a run?

Add a Filter by Zapier step between the trigger and the POST so only rows with a URL start a run. Each call is one run and counts against your runtime.

Can I send different sites to different scrapers from Zapier?

Yes. A Paths by Zapier step can send Amazon URLs to one automation and eBay URLs to another, each with its own scrape and its own name.

Is my axiom.ai API key visible in Zapier?

Yes. Zapier shows the request body in task history, so use a dedicated key where you can, and rotate it from the extension's settings if it is exposed.

Can I trigger a web scraper from Zapier without code?

Yes. The scraper is built from the three lines above with no code, and Webhooks by Zapier, Formatter and the Google Sheets actions are standard Zapier steps; the only text you type is the data table in the POST action.

Can the scraper run on my own machine instead of the cloud?

Yes. For a page that needs your own browser's login, see trigger a desktop run from Zapier.

Can I trigger a web scraper from Make or n8n instead of Zapier?

Yes. See trigger a web scraper from Make and trigger a web scraper from n8n. The plain request is in trigger a web scraper from a webhook.

Where do I get help?

Get in touch with support, or ask in the axiom.ai community.

Related guides