Scrape links from a website into a Google Sheet


How to scrape links from a website into a Google Sheet: install the Chrome extension, click Build from description, tell the agent what you want, and follow its prompts. A working bot is built and tested in minutes.

Here is how the automation works. To scrape links from a website, point the Get a list of links to pages from a URL step at the page, select one link, and write the list to a Google Sheet with Write data to a Google Sheet. It works on listing pages, paginated results, scrolling pages and feeds.

Author
Alex Barlow
Published

Describe it and click the button: the agent builds and tests the link scraper in your Chrome extension.

  1. Go to the listing page I give you
  2. Scrape every product link on it
  3. Do every page of results
  4. Add the links to my Google Sheet
Click to launch the Chrome extension. Not installed? You can create an account and try it free.Opens the Chrome extension. Free account to try.

You need a Google Sheet to write to, with a named tab such as Links. Type sheet.new in Chrome to make one.

The video shows the agent building the link scraper from the lines above, in a few minutes.


The steps the agent builds, as the Builder shows them: each step by its name, with the fields to set under it. Add them yourself in this order in the step-by-step builder, or open the agent's result there to adjust it.

  • 1.0 Get a list of links to pages from a URL
    • URL: the listing page
    • Select: one product link
    • Find pager: the Next button
    • Max results: 10 for a test run
  • 2.0 Write data to a Google Sheet
    • Spreadsheet: your sheet
    • Sheet name: Links
    • DATA: [scraped-link-data]
    • Write options: Add to existing data

Reselect them. In Get a list of links to pages from a URL, pick a different example link. If the page's links have no stable pattern, paste a custom CSS selector instead.

Turn off retries and lower Max results. Open Configure scraper on the step and set No. of retries to 0. A Max results far above the number of links on the page also slows it down, because axiom.ai keeps looking for links that are not there.

The step is pointed at the wrong data or tab. In Write data to a Google Sheet, check that DATA is [scraped-link-data] and that the tab in Sheet name exists.

Set Write options on the sheet step to Add to existing data.

Yes. Point Find pager on Get a list of links to pages from a URL at the page's Next button and it turns the pages until it has Max results links.

Yes. The whole automation is two no-code steps in the Chrome extension: one that collects the links, one that writes them to the sheet.

Yes. Save the automation and set a schedule if the page changes often; with Add to existing data each run adds the new links below the last.

Build a second automation that reads this sheet and loops through the links: scrape data from every page in a list. Keeping the two apart makes each one easier to rerun and fix.

Yes. Swap the sheet step for Export to CSV: scrape links from a website into a CSV file walks through it.

Where do I get help?

Get in touch with support, or ask in the axiom.ai community.

Related guides