Scrape a list of URLs from Airtable back into Airtable
How to scrape a list of URLs from Airtable back into Airtable: install the Chrome extension, click Build from description, tell the agent what you want, and follow its prompts. A working bot is built and tested in minutes.
Here is how the automation works. To scrape a list of URLs from Airtable, read the table of URLs with the Read data from Airtable step, open and scrape each one in a loop, and add the results to a second table with the Write data to Airtable step. No personal access token and no script: both steps use the Airtable connection in axiom.ai.
Build the Airtable scraper from a description
Describe it and click the button: the agent builds and tests the scraper in your Chrome extension. It asks for the base and the two tables, opens the first page and asks which data you want from it, then picks the selectors.
- Read the URLs from the Pages table in my Airtable
- Open each one and ask me which data to scrape
- Add the scraped data to Airtable in Results
You need an Airtable base with a Pages table holding the URL in its first field and an empty Results table whose fields are all single line text or long text, in the order you want the scraped values written, plus the base ID (it starts with app, in the base's URL after airtable.com/) and the two table names as shown on their tabs. Airtable is connected to axiom.ai once, from Credentials and API key on the dashboard, and a step added before that shows a Grant permission button that opens the same window.
Build the Airtable scraper step by step
The steps the agent builds, as the Builder shows them: each step by its name, with the fields to set under it. Add them yourself in this order in the step-by-step builder, or open the agent's result there to adjust it.
- 1.0
Read data from AirtableBase ID: your base's IDTable ID or name: Pages
- 2.0
Loop through data- repeats for every row in
[airtable-data]
- 2.1
Go to pageURL: the first column of[airtable-data]
- 2.2
Get data from bot's current pageSelect: the data you asked for on the page, one column each
- 2.3
Write data to AirtableBase ID: your base's IDTable ID or name: ResultsDATA:[scrape-data]
- repeats for every row in
Common questions about scraping URLs from Airtable
Why does the write fail with "axiom.ai can only write to text fields"?
A field in the results table is a number, date, link or other non-text type. Change it to single line text or long text, or write to a table whose fields are all text.
Why does the Airtable step say "Axiom needs permission to interact with Airtable"?
The Airtable connection has not been made. Click Grant permission on the step and allow access to the base in Airtable's window, or connect from Credentials and API key on the dashboard (how to connect Airtable).
How do I keep the page's URL beside its results in Airtable?
Scrape it too. Add the URL column of the current row as the first value with Append or merge data, or make the page's own link the first field you scrape, and give the results table a URL field first.
Can I replace the Airtable results table each run instead of adding to it?
Yes. Move the Write data to Airtable step after the loop and switch Clear data before writing so it clears existing data, to replace the results table with the latest scrape each run. Every record in it is deleted first, so point it at a table that holds nothing else.
Can I read and write large Airtable tables?
Yes. Read data from Airtable reads every record in the table. Writes go ten records at a time with a short pause, as Airtable's rate limit requires, so a run writing hundreds of rows takes a little longer at the end.
Can I scrape URLs from Airtable without code or an access token?
Yes. Both Airtable steps use the connection you grant once in axiom.ai, so there is no personal access token to create and no script; the agent builds the automation from the three lines above.
Can I start the Airtable scraper from another app instead of a table?
Yes. To scrape URLs that arrive by webhook rather than from a table, see trigger a web scraper from a webhook. To post scraped data to Airtable from a scraper that starts elsewhere, see post data to Airtable.
Where do I get help?
Get in touch with support, or ask in the axiom.ai community.
Related guides
Post data to Airtable with the Airtable API
- Go to the page I give you
- Scrape the details I want from it
- Add each row as a new record in my Airtable table
Combine variables from different steps into one
- Go to the page I give you
- Scrape the value I point out
- Get today's date
- Join them into one row in my Google Sheet
Scrape data from a list of URLs in a Google Sheet
- Open each link in column A of my Google Sheet
- Ask me which data to scrape from each page
- Put the details in that URL's row from column B