Loop through a list of URLs
When the thing you're automating needs to happen on lots of pages, not one. You'll pass in a list of URLs, visit each one, and run the same actions on every page. Useful for product-page scraping, bulk screenshots, and any "do this 200 times" task.
- Read URLs from a Google Sheet
- Loop through urls
- Go to page
Start building opens the Chrome extension with these lines filled in. Click Try agent there and the agent opens the site, builds the steps, test-runs them and hands you the automation in the Builder. It stops and asks if it needs a login or a choice from you.
What the agent builds
Three steps, and room for yours:
- Read data from a Google Sheet: pulls in the column of URLs. Say Import a CSV or Get every link on example.com/listings instead and the agent uses that source.
- Loop through data: repeats the steps inside it once per URL.
- Inside the loop, Go to page: opens the current URL.
Add a line for whatever should happen on each page, such as Get the price or Take a screenshot, and the agent builds that step inside the loop too. Leave it out and you get the loop with an empty slot to fill in the Builder.
Build it manually
Prefer to add the steps yourself? These are, loosely, the steps the agent builds. The agent reads each page as it goes, so it adds the steps for whatever you asked it to do there, and a wait where a page is slow.
Step 1: Get the URLs into a step
Pick the source that matches where your URLs live:
- A sheet: add Read data from a Google Sheet and, in
SpreadsheetandSheet name, pick the sheet and tab with a column of URLs. - A CSV: add Import CSV File and, in
Import CSV, upload the file. - A listing page: add Get a list of links to pages from a URL, enter the listing page in
URL, then clickSelectand click one of the links you want.
Step 2: Loop once per URL
- Click Add step and search for Loop.
- Select Loop through data to add it.
- In
Loop through data, click Insert Data and choose the URL data from step 1.
Step 3: Open each URL
- Inside the loop, click Add step and search for Go to page.
- Select Go to page to add it.
- In
Enter URL, click Insert Data and choose the current loop value. This is what makes the automation visit a different page each time round.
Step 4: Do the work on each page
- Still inside the loop, below Go to page, add whatever should run on each page: a scrape, a click, a form fill, a screenshot.
Step 5: Run it
- Click Run and watch it work through the list.
- If it does, click Save.
The shape: get URLs, loop once per URL, go to that URL, do the work.
Things worth knowing
- One line per thing to do on each page. Get the price, then take a screenshot is two lines, not one.
- The whole job of the Go to page step inside a loop is to use the current URL, not the first or last one. Always use Insert Data to reference the loop row, never paste a hard-coded URL.
- Lots of sites rate-limit or block automated traffic when hit in quick succession. A Wait step inside the loop is usually a good idea.
- If any single URL fails, the whole loop stops. Wrap the page work in a Try / Catch step if you'd rather skip bad URLs and keep going.
Next steps
- The Loop through data reference covers the loop mechanics.
- The Try / Catch reference covers handling failures mid-loop.
Quick starts: build from description with one click
Automations that loop through a list of URLs and do something on each page. Each guide carries a Build from description button: one click opens the extension with the lines filled in and the agent builds it, or follow the steps yourself.
Scrape data from a list of URLs in a Google Sheet
- Open each link in column A of my Google Sheet
- Ask me which data to scrape from each page
- Put the details in that URL's row from column B
Scrape every page in a list of links into a Google Sheet
- Open each link in the Links tab of my sheet
- Scrape the title, price and description
- Add them to the Data tab
- Delete the link row once it is scraped
Screenshot a list of websites from a Google Sheet
- Open each website link in column A of my sheet
- Save a screenshot of the page to my computer
- Write today's date in column B of the same row
Export a web page as a PDF automatically
- Go to the page I give you
- Save the whole page as an A4 PDF
- Wait for the download to finish
Bulk download files from a website to a folder
- Go to the page I give you
- Download every PDF on it
- Save them in my Downloads folder
Mark Google Sheet rows as done so a run resumes where it stopped
- Read the product URLs from my Google Sheet
- Skip any rows already marked scraped
- Open each URL and scrape the fields I need
- Write what was scraped to a Results tab
- Mark each row scraped in a new column