Scrape

axiom.scrape(url, selector, pager, max_results, settings) navigates to a URL and extracts a list of structured records from it, paginating if you supply a pager. The same step type as the No-Code Tool's Get data from bot's current page, called from your code with selectors you choose at runtime.

Signature

const rows = await axiom.scrape(url, selector, pager, max_results, settings);
ParameterTypeRequiredDescription
urlstring | stringYesURL to scrape, or an array of URLs to scrape in sequence. scrape() navigates for you as part of the call.
selectorobject | stringYesOne column spec per field you want (see Selecting columns). A plain CSS string works for a single-column scrape.
pagerstring | nullYesCSS selector for the "next page" link. Pass null for single-page scraping. With a pager, the scraper paginates until max_results is hit or the pager stops resolving.
max_resultsnumber | nullYesCap on the number of records returned. Must be a positive whole number, or null for unlimited.
settingsobjectYesBehavioural tuning (output format, minimum wait, …). Pass {} for defaults.

Returns a 2D array — one row per matched record, one column per selector you passed.

Warning: url is required. Don't pass null to scrape the page the session is already on — the cloud driver returns a 500. Pass the URL explicitly, even when a previous step already navigated there. The browser session and its cookies persist across the navigation, so this still works after a login flow: pass the post-login page's URL.

Selecting columns

Pass an array of column specs to get more than one field per row. Each spec takes a CSS selector and an optional resultType:

resultTypeReturns
textContentThe element's text. This is the default.
innerHTMLThe element's inner HTML.
linkThe element's href.
axiom-downloadDownloads the linked file.

Scope every column to the same repeating row container so the columns line up with each other:

const rows = await axiom.scrape(
  "https://example.com/products",
  [
    { selector: "article.product h3 a",     resultType: "textContent" },
    { selector: "article.product .price",   resultType: "textContent" },
    { selector: "article.product h3 a",     resultType: "link" },
  ],
  "a.pagination-next",
  100,
  {}
);

That returns a row per product, with three columns in the order you declared them:

[
  ["Widget Pro",  "£49", "https://example.com/widget-pro"],
  ["Widget Lite", "£19", "https://example.com/widget-lite"],
  // ...
]

For a single column, a bare CSS string is enough:

const titles = await axiom.scrape("https://example.com/blog", ".post h2", null, 50, {});

Example

Scrape headlines and article links from a BBC search results page, following the pagination links until 50 results:

const rows = await axiom.scrape(
  "https://www.bbc.co.uk/search?q=Harry+Kane&d=NEWS_PS",
  [
    { selector: '[data-testid="default-promo"] p[class*="PromoHeadline"]', resultType: 'textContent' },
    { selector: '[data-testid="default-promo"] a[href]',                   resultType: 'link' },
  ],
  'nav div:nth-child(4) a[href^="/search?q=Harry+Kane&d=NEWS_PS&page="]',
  50,
  {}
);

Notes

  • Column selectors are ordinary CSS. You can write them by hand, generate them with an LLM from the page's HTML, or capture them with the No-Code Tool's selector tool and paste them in.
  • For single-value extraction (page title, OG image, one field), use axiom.scrapeMetadata() — it's faster and simpler.
  • The scraper handles lazy-loaded content by waiting briefly after each pager click. Sites with slow infinite-scroll may need a higher minWait in settings.
  • Multiple URLs in url are scraped in sequence and the results concatenated. A failure on any one URL fails the whole call.
Previous
Get clipboard contents
Next
Scrape metadata
On this page