Trigger a web scraper from a webhook


How to trigger a web scraper from a webhook: install the Chrome extension, click Build from description, tell the agent what you want, and follow its prompts. A working bot is built and tested in minutes.

Here is how the automation works. To trigger a web scraper from a webhook, send a POST to https://lar.axiom.ai/api/v3/trigger with your API key, the automation's name and a list of URLs. Its first step, Receive data from another app, holds the URLs; a loop scrapes each page in the cloud, and a Trigger webhook step posts the results to a URL you choose.

Author
Alex Barlow
Published

Build the webhook scraper from a description


Describe it and click the button: the agent builds and tests the scraper in your Chrome extension. It asks for your results URL and picks the fields to scrape. Webhooks need a plan that includes them (pricing) and your API key (Authentication).

  1. Take a list of product URLs sent to me by webhook
  2. Open each one and scrape the title and price
  3. Post everything scraped to my results webhook
Click to launch the Chrome extension. Not installed? You can create an account and try it free.Opens the Chrome extension. Free account to try.

You need a URL that can receive a POST with the results: your own endpoint, a Zapier Catch Hook, a Make custom webhook, an n8n Webhook node, or a request bin while you test. Give the automation a name you can type exactly: the trigger matches it character for character, including case and spaces.

Build the webhook scraper step by step


The steps the agent builds, as the Builder shows them: each step by its name, with the fields to set under it. Add them yourself in this order in the step-by-step builder, or open the agent's result there to adjust it.

  • 1.0 Receive data from another app
    • Test data: a table of product URLs in the shape the caller sends, one per row and no header
  • 2.0 Loop through data
    • repeats for every row in [webhook-data]
    • 2.1 Go to page
      • URL: the first column of [webhook-data]
    • 2.2 Get data from bot's current page
      • Select: the title and price on the page, one column each
  • 3.0 Trigger webhook
    • Endpoint: your results URL
    • Payload: [scrape-data]

Common questions about the webhook scraper


How do I send the webhook that starts a scraper run?

From any system that can make an HTTP request, a POST with your key, the automation's name and a data table: a list of rows, each row a list of cells, one column holding a URL and no header row.

curl -X POST https://lar.axiom.ai/api/v3/trigger \
  -H "Content-Type: application/json" \
  -d '{
    "key": "your-api-key-here",
    "name": "Scrape product pages",
    "data": [
      ["https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"],
      ["https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html"]
    ]
  }'

The trigger answers straight away with one of these responses:

ResponseWhat it means
{ "OPEN LINK IN BROWSER": "https://..." }The run started. Open the link to watch it live.
"status": "queued"No cloud slot is free yet. The run starts within a few minutes.
"status": "error" with Task not foundNo automation has that name. Check the name matches exactly, including case and spaces.
"status": "error" with Invalid keyThe API key is wrong. Copy it again from the extension's settings.

See Trigger an automation for the full response reference. Once the run finishes, your results URL receives the scraped rows. If you also want to be told when a run fails, turn on notifications for the automation.

Why does the webhook answer Task not found?

No automation has that name. Check the name in the request matches the automation exactly, including case and spaces. When a run fails, hand it to the agent and it repairs the step.

How many URLs can I send in one webhook call?

One or hundreds. The table can hold any number of rows, and each call is one run, so a caller with ten URLs sends them in one call, not ten.

Can I send more than URLs to the webhook scraper?

Yes. Add columns for a search term, an account or a reference id, and read them inside the loop from the current row. Post the id back with the results so the caller can match them up.

Can I test the webhook scraper without sending a webhook?

Yes. While Test data is on in Receive data from another app, every run from the builder uses that table, so you can run and adjust the scraper before any caller exists.

Can I file the scraped rows in a Google Sheet as well as posting them back?

Yes. Add a Write data to a Google Sheet step inside the loop with [scrape-data], and the rows are filed and posted.

Can I send the webhook from Zapier, Make or n8n?

Yes. Each tool has its own guide for the same automation: Zapier, Make and n8n.

Can the scraper run on my own machine instead of the cloud?

Yes. When the page needs your own browser's login, turn on Run via API on desktop and post to the desktop trigger instead. See trigger a desktop run from Zapier.

How do I check a run's status from code?

Poll /run-data with the automation's name to read the status of its latest run.

Can I trigger a web scraper from a webhook without code?

Yes. The scraper itself is built from the three lines above with no code; the call that starts it is one POST, which a tool such as Zapier, Make or n8n sends for you without a script.

Where do I get help?

Get in touch with support, or ask in the axiom.ai community.

Related guides