How to loop through pages scraping content

Looping through links scraping data from web pages into Google Sheets is made simple with the template. If you don't want to use a Google Sheets, Its possible to edit this template and tailor it to your requirements.

FYI If you want to scrape links first and loop through pages, we recommend you do this in two parts:

  1. Use this Axiom to scrape the links.
  2. Then use this web scraping template.

By splitting the task up, we make it easier to manage.

Important! This scraper is best suited for scraping around 30 or 40 pages per run. If you want to scrape large volumes please use our batch scraper.

# Install the Looping page scraper bot template

To install this web scraper Axiom click 'Install template'. If you are a new user you will be required to create an Axiom.ai account before you can edit the template.

# Set up a Google Sheet

  1. Create a new Google Sheet. You can do this in your Chrome browser by entering the shortcut 'sheet.new', presuming you already have an account.
  2. Name your sheet something like 'Scraper’.
  3. Set up two tabs titled 'Links' and 'Scraped Data’.
  4. Add your links to the ‘Links’ tab on your sheet. Each link should be added to a new row.

# Set up the 'Read data from Google Sheet' step

  1. Spreadsheet - In the field called 'Spreadsheet', you can search for the Google Sheet you created. Once found, click to select.
  2. Sheet name - Choose the sheet tab called 'Links'.
  3. You should now see a preview of the data.
Axiom.ai scraping made simple

# Configure the 'Interact with a page's interface' and select the content to scrape

  1. Enter URL - Click 'Insert Data' select 'google-sheet-data' and select the column with the links in.
  2. Get data from a webpage - Select the content on the page you wish to scrape. Its as easy as point and click, this video shows you how.
  3. Find pager (optional) - Select the next button if the page has a pager. If it scrolls then do nothing.
  4. Max Results - When testing set to 2-3 results max, only toggle to all when ready to go live with your Axiom.

# Set up the 'Write Data to a Google Sheet' step

  1. Spreadsheet - In the field called 'Spreadsheet', you can search for the Google Sheet you created. Once found, click to select.
  2. Sheet name - Choose the tab you created.
  3. DATA - Select the 'Interact data'.
  4. Clear data before writing | Add to existing data - Set this option to 'Add to existing data'.

# Test run

We always recommed doing a test run - click run then check the output from the scraper. In this case in the sheet.

Try the following if your bot is not working. We would also recommend you watch the video to troubleshoot.

  • The links come out as text - Set the selector type to link not text.
  • Selectors fail and the bot stops, try a new selection.
  • The pager is not working - Re-select the next button or try custom selector.
  • No output in the sheet - In 'Write Data to a Google Sheet' check data step is connected.
  • Selectors fail you may need to use custom selectors

# More web scraping templates