Trigger a web scraper from Make and get the data back
How to trigger a web scraper from Make and get the data back: install the Chrome extension, click Build from description, tell the agent what you want, and follow its prompts. A working bot is built and tested in minutes.
Here is how the automation works. To trigger a web scraper from Make, add an HTTP: Make a request module that posts the URLs to https://lar.axiom.ai/api/v3/trigger. The cloud run takes them in a Receive data from another app step, scrapes each page, and a Trigger webhook step posts the data to a second scenario with a Custom webhook, which sends it on. Any Make trigger will do: a schedule, a new record, a form, another app.
Build the Make scraper from a description
Describe it and click the button: the agent builds and tests the scraper in your Chrome extension; the two Make scenarios are set up in Make, in the questions below. Webhooks need a plan that includes them (pricing) and your API key (Authentication).
- Take the competitor URLs Make sends each morning
- Open each one and scrape the name and price
- Post the whole list of prices to my Make webhook
You need a Make account and an Airtable base listing the competitor pages to watch. Make the second scenario's custom webhook first so you can give the agent its address, or leave the Trigger webhook step's Endpoint empty and paste it in later. Name the automation something you can type exactly, such as Scrape product pages.
Build the Make scraper step by step
The steps the agent builds, as the Builder shows them: each step by its name, with the fields to set under it. Add them yourself in this order in the step-by-step builder, or open the agent's result there to adjust it.
- 1.0
Receive data from another appTest data: a table of competitor URLs in the shape the scenario sends, one per row
- 2.0
Loop through data- repeats for every row in
[webhook-data]
- 2.1
Go to pageURL: the first column of[webhook-data]
- 2.2
Get data from bot's current pageSelect: the product's name and price on the page, one column each
- repeats for every row in
- 3.0
Trigger webhookEndpoint: the second scenario's custom webhook addressPayload:[scrape-data]
Common questions about the Make scraper
How do I start the scraper run from Make with the HTTP module?
With an HTTP: Make a request module after the scenario's trigger:
- In Make, open Scenarios and create a new scenario, or open the one you want to add this to.
- Pick whichever trigger module suits the job. For the daily price check, set the scenario to run every morning in its schedule settings, start with Airtable: Search Records on the competitors table, and add an Array aggregator that collects the URL field from every record into one array.
- Add a module, search for HTTP, and choose Make a request.
- Set URL to
https://lar.axiom.ai/api/v3/trigger. - Set Method to POST.
- Set Body type to Raw and Content type to JSON (application/json).
- Paste the JSON below into Request content, with your own key and automation name. Where the URL goes, map the field from your trigger module. For a whole list from the aggregator, join the URLs with
"],["and wrap the result in[["and"]], so the body reads"data": [["url-1"],["url-2"]]whatever the number of records. - Save the scenario.
{
"key": "your-api-key-here",
"name": "Scrape product pages",
"data": [["https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"]]
}
Run the module once. The trigger answers straight away:
| Response | What it means |
|---|---|
"OPEN LINK IN BROWSER" with a link | The run started. Open the link to watch it live. |
"status": "queued" | No cloud slot is free yet. The run starts within a few minutes, and the webhook still fires. |
Task not found | name does not match the automation's name exactly, including case and spaces. |
Invalid key | key is wrong. Copy the key again from the extension's settings. |
The same call in cURL, useful for checking the key and name before you put them in Make:
curl -s -X POST https://lar.axiom.ai/api/v3/trigger -H "Content-Type: application/json" -d '{"key":"your-api-key-here","name":"Scrape product pages","data":[["https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"]]}'
How do I catch the scraped data in a second Make scenario?
With a Custom webhook as the second scenario's trigger:
- In Make, create a new scenario and add the Webhooks module. Choose Custom webhook.
- Click Add, give the webhook a name, save it, and click Copy address to clipboard. Make now waits for data so it can learn the structure.
- Paste the address into the Trigger webhook step's
Endpointand save the automation. - Run the automation once from the builder. Make reports that the structure was determined, and the scraped rows appear as the webhook's output bundle.
- Add the modules that send the data on. For the daily price check, Airtable: Search Records with the scraped URL as the search formula, then Airtable: Update a Record to write the price to the record it found.
Then turn both scenarios on and fire the first scenario's trigger: the run starts, scrapes, and the second scenario receives the data. The automation's run reports show each run, and notifications can tell you when one fails.
Why send the whole list of URLs to axiom.ai in one call?
One call is one run. The aggregator turns many Airtable records into one run; without it, Make calls the trigger once per record and you pay for a run each time.
Why is a new scraped field missing in Make?
Make still has the old data structure. Open the custom webhook in Make and click Redetermine data structure before the next run, or the new column is dropped.
How do I see the trigger's reply in Make?
Open the scenario's execution log and look at the HTTP module's output. "status": "queued" there is not a failure; the run starts within a few minutes and the webhook still fires.
Can I trigger a web scraper from Make without code?
Yes. The scraper is built from the three lines above with no code, and the HTTP module and custom webhook are standard Make modules; the only text you type is the JSON body above.
Can the scraper run on my own machine instead of the cloud?
Yes. For a page that needs your own browser's login, see trigger a desktop run from Make.
Can I trigger a web scraper from Zapier or n8n instead?
Yes. See trigger a web scraper from Zapier and trigger a web scraper from n8n. The plain request is in trigger a web scraper from a webhook.
Can I send scraped data to Make without triggering the run from Make?
Yes. Run or schedule the automation in axiom.ai and post the results with one Trigger webhook step. See post scraped data to Make with a webhook.
Where do I get help?
Get in touch with support, or ask in the axiom.ai community.
Related guides
Trigger a web scraper from Zapier and get the data back
- Take the URL Zapier sends when a row is added
- Open it and scrape the title and price
- Post the title, price and row number to Zapier
Trigger a web scraper from n8n and get the data back
- Take the URL someone submitted in my n8n form
- Open it and scrape the title and price
- Post what was scraped to my n8n webhook
Trigger a web scraper from a webhook
- Take a list of product URLs sent to me by webhook
- Open each one and scrape the title and price
- Post everything scraped to my results webhook