Trigger a web scraper from Make and get the data back


How to trigger a web scraper from Make and get the data back: install the Chrome extension, click Build from description, tell the agent what you want, and follow its prompts. A working bot is built and tested in minutes.

Here is how the automation works. To trigger a web scraper from Make, add an HTTP: Make a request module that posts the URLs to https://lar.axiom.ai/api/v3/trigger. The cloud run takes them in a Receive data from another app step, scrapes each page, and a Trigger webhook step posts the data to a second scenario with a Custom webhook, which sends it on. Any Make trigger will do: a schedule, a new record, a form, another app.

Author
Alex Barlow
Published

Build the Make scraper from a description


Describe it and click the button: the agent builds and tests the scraper in your Chrome extension; the two Make scenarios are set up in Make, in the questions below. Webhooks need a plan that includes them (pricing) and your API key (Authentication).

  1. Take the competitor URLs Make sends each morning
  2. Open each one and scrape the name and price
  3. Post the whole list of prices to my Make webhook
Click to launch the Chrome extension. Not installed? You can create an account and try it free.Opens the Chrome extension. Free account to try.

You need a Make account and an Airtable base listing the competitor pages to watch. Make the second scenario's custom webhook first so you can give the agent its address, or leave the Trigger webhook step's Endpoint empty and paste it in later. Name the automation something you can type exactly, such as Scrape product pages.

Build the Make scraper step by step


The steps the agent builds, as the Builder shows them: each step by its name, with the fields to set under it. Add them yourself in this order in the step-by-step builder, or open the agent's result there to adjust it.

  • 1.0 Receive data from another app
    • Test data: a table of competitor URLs in the shape the scenario sends, one per row
  • 2.0 Loop through data
    • repeats for every row in [webhook-data]
    • 2.1 Go to page
      • URL: the first column of [webhook-data]
    • 2.2 Get data from bot's current page
      • Select: the product's name and price on the page, one column each
  • 3.0 Trigger webhook
    • Endpoint: the second scenario's custom webhook address
    • Payload: [scrape-data]

Common questions about the Make scraper


How do I start the scraper run from Make with the HTTP module?

With an HTTP: Make a request module after the scenario's trigger:

  1. In Make, open Scenarios and create a new scenario, or open the one you want to add this to.
  2. Pick whichever trigger module suits the job. For the daily price check, set the scenario to run every morning in its schedule settings, start with Airtable: Search Records on the competitors table, and add an Array aggregator that collects the URL field from every record into one array.
  3. Add a module, search for HTTP, and choose Make a request.
  4. Set URL to https://lar.axiom.ai/api/v3/trigger.
  5. Set Method to POST.
  6. Set Body type to Raw and Content type to JSON (application/json).
  7. Paste the JSON below into Request content, with your own key and automation name. Where the URL goes, map the field from your trigger module. For a whole list from the aggregator, join the URLs with "],[" and wrap the result in [[" and "]], so the body reads "data": [["url-1"],["url-2"]] whatever the number of records.
  8. Save the scenario.
{
  "key": "your-api-key-here",
  "name": "Scrape product pages",
  "data": [["https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"]]
}

Run the module once. The trigger answers straight away:

ResponseWhat it means
"OPEN LINK IN BROWSER" with a linkThe run started. Open the link to watch it live.
"status": "queued"No cloud slot is free yet. The run starts within a few minutes, and the webhook still fires.
Task not foundname does not match the automation's name exactly, including case and spaces.
Invalid keykey is wrong. Copy the key again from the extension's settings.

The same call in cURL, useful for checking the key and name before you put them in Make:

curl -s -X POST https://lar.axiom.ai/api/v3/trigger -H "Content-Type: application/json" -d '{"key":"your-api-key-here","name":"Scrape product pages","data":[["https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"]]}'

How do I catch the scraped data in a second Make scenario?

With a Custom webhook as the second scenario's trigger:

  1. In Make, create a new scenario and add the Webhooks module. Choose Custom webhook.
  2. Click Add, give the webhook a name, save it, and click Copy address to clipboard. Make now waits for data so it can learn the structure.
  3. Paste the address into the Trigger webhook step's Endpoint and save the automation.
  4. Run the automation once from the builder. Make reports that the structure was determined, and the scraped rows appear as the webhook's output bundle.
  5. Add the modules that send the data on. For the daily price check, Airtable: Search Records with the scraped URL as the search formula, then Airtable: Update a Record to write the price to the record it found.

Then turn both scenarios on and fire the first scenario's trigger: the run starts, scrapes, and the second scenario receives the data. The automation's run reports show each run, and notifications can tell you when one fails.

Why send the whole list of URLs to axiom.ai in one call?

One call is one run. The aggregator turns many Airtable records into one run; without it, Make calls the trigger once per record and you pay for a run each time.

Why is a new scraped field missing in Make?

Make still has the old data structure. Open the custom webhook in Make and click Redetermine data structure before the next run, or the new column is dropped.

How do I see the trigger's reply in Make?

Open the scenario's execution log and look at the HTTP module's output. "status": "queued" there is not a failure; the run starts within a few minutes and the webhook still fires.

Can I trigger a web scraper from Make without code?

Yes. The scraper is built from the three lines above with no code, and the HTTP module and custom webhook are standard Make modules; the only text you type is the JSON body above.

Can the scraper run on my own machine instead of the cloud?

Yes. For a page that needs your own browser's login, see trigger a desktop run from Make.

Can I trigger a web scraper from Zapier or n8n instead?

Yes. See trigger a web scraper from Zapier and trigger a web scraper from n8n. The plain request is in trigger a web scraper from a webhook.

Can I send scraped data to Make without triggering the run from Make?

Yes. Run or schedule the automation in axiom.ai and post the results with one Trigger webhook step. See post scraped data to Make with a webhook.

Where do I get help?

Get in touch with support, or ask in the axiom.ai community.

Related guides