Get a list of links from a URL step
Use this step to scrape links from listing-style pages. Then pass the links into a 'Get data from a webpage' step to scrape data. Alternatively, write the links to a 'Google sheet' and create a batch scraper for large-scale scraping. We call this design pattern 'Batch scraping' Learn more about 'batching here' or try this template. You can also find a 'Link scraper template here.'
You can use this step to scrape:
- Amazon product page links, like Amazon Book pages
- Social media links
- LinkedIn profile links from Sales Navigator
- Links from any website

How to configure the Get a list of links to pages step
URL
Enter a URL or multiple URLs, each on their own line. You can also plugin data from other steps to provide URLs
Select
Click 'Select' to choose the links you wish to scrape. The display will transform into the selector tool, and it will guide you by showing you how to select links from the webpage.
The Multi-selector tool comes with several valuable features accessed by clicking custom:
- Ability to use custom CSS selectors
- 'Use element text' allows you to click buttons based on the button text i.e. 'Submit'
- Pass CSS selectors in from data sources
Watch these guides to learn more about the selector tool.
Find pager (optional)
Select the 'Next' button for the pager, if there is one. If the button features text such as 'Next,' why not try the 'Use element text' method? Click 'Custom' on the selector toolbar, then click 'Use element text'.
Max results
We set the max results to 20 to speed up the testing of your bots as you make them. It's best to do short runs while testing your bot.
Wait time between scrolls (ms)
Adjust the wait time between scrolls to increase or decrease the loading time of content. This feature is particularly useful when scrolling down listing pages with slow-loading content. However, insufficient waiting time could mean that content is not loaded. Therefore, experiment with caution.
No. of retry attempts when results not found
To speed up your runs, reduce the retry runs. But keep in mind content could be missed. Make sure to do some test runs.
Minimum wait before scraping (ms)
To speed up your runs, reduce your wait time. However, keep in mind that some content may not have finished loading yet, which is why we wait.
Page number to start scraping on
For paginated pages, you can specify a starting page. However, not all pages support this.
Specify exact number of pixels to scroll
Instead of auto-scrolling set a pixel height to scroll.
Force a re-scrape after each page change
Use if you are only getting the first page of results.
Output
A preview of the scraped data.
Guides that use this step
Scrape links from a website into a CSV file
- Go to the listing page I give you
- Scrape every product link on it
- Do every page of results
- Save the links as a CSV file
Scrape links from a website into a Google Sheet
- Go to the listing page I give you
- Scrape every product link on it
- Do every page of results
- Add the links to my Google Sheet
Automate Sales Navigator connection requests with a personal message
- Get the profile links and names from my lead list
- Open each lead's profile in Sales Navigator
- Send a connection request with my message
- Start the message with the lead's first name
Automate LinkedIn Sales Navigator messages to a saved lead list
- Read the leads from my Sales Navigator list
- Open each lead's profile
- Send them my message, using their first name
- Add a subject line when one is required
Triage Jira tickets with AI and close the spam
- Get every ticket link from my Jira support queue
- Open each ticket and read its description
- Ask AI whether it is spam or valid
- Log the spam tickets and the date to my Closed tab
Scrape Instagram users from a hashtag into a Google Sheet
- Open the Instagram hashtag page in my Google Sheet
- Collect the links to every post on it
- Open each post and get the poster's profile link
- Write the post and profile links to my sheet