Puppeteer (opens new window) is a Node.js library that was developed by Google that provides a high-level API to control headless (or full) Chrome, Chromium, or Firefox. It allows you to programmatically interact with a web browser as if a human user were doing so, but with greater speed and precision. It's primarily used for browser automation, but can also be used for automated testing, performance monitoring, screenshot generation and web scraping.

The good news? Puppeteer is built into Axiom.ai, making accessing the library within your automations easy. Let's dive into some of the things that you can do with the Write JavaScript step.

Why use Puppeteer

Most Puppeteer features are already built into Axiom.ai and are offered as a no-code method of interacting with web pages, however, there are instances where you may want to extend this functionality or customise it to your specific needs. For example, if you want to locate all buttons on a page but only click on buttons that have the text "My button", you could use the following code:

// https://pptr.dev/guides/page-interactions#applying-filters-on-locators

await page
    .locator('button')
    .filter(button => button.innerText === "My button")
    .click();

This allows for more flexibility than most users will need within their automations and allows you to create entirely custom interactions within your automation.

Puppeteer scripts can only be used when running your automations locally - you'll need to enable the "Run in app" option within the step.

Interaction

Axiom.ai already offers a wide range of no-code Interaction steps that can be used to interact with web pages as if a human was doing so. It's possible to extend the functionality that is already offered by using Puppeteer within the Write JavaScript step. This will allow you to create custom interaction steps within your automations - for example, if you want to just get a list of elements that are enabled on the page you can use the snippet below:

const enabled = await page
    .locator('button')
    .map(button => !button.disabled)
    .wait();

You would then be able to loop through these elements to either click on them, or scrape their content. This can make your automation creation process more robust as it will ensure that the buttons that your automation is clicking will succeed in doing so.

Another useful use case for this is when you have a list that may involve multiple click actions, for example, if you need to click on the row to open a context menu and then click within the context menu before moving onto the next row.

const rows = await page
    .locator('.context-menu-button');

for (var i = 0; i < rows.length; i++) {
    rows[i].click();
    await page.locator('.context-menu-item').click();
}

or

for (var i = 0; i < 10; i++) {
    await page.locator('.context-menu-button').click();
    await page.locator('.context-menu-item').click();
}

For the second option you may need to add in a wait to ensure that the context menu has finished loading/displaying on the page.

Scraping

As with Interaction steps, Axiom.ai offers Scraping steps that can be used to scrape data from webpages that are loaded by your automation, these are a no-code entry point. If you require something more customisable from the options that we already offer you can make use of the "Write Javascript" step to invoke Puppeteer code.

Using Puppeteer allows you to be more specific about the type of content that you wish to scrape - using the examples above in the Interaction portion of this article, it would allow you to filter out certain elements without getting too specific with your CSS selectors. This can be helpful for filtering out information that you may not wish to filter out later in the process and gives you control over the data that you scrape.

Non-CSS selectors

Utilising the Puppeteer library allows you to select elements using non-CSS selectors - something that's not currently supported by Axiom.ai. These include text, XPath and ARIA selectors, for example to click all elements with the 'Submit' ARIA label you would use the following code snippet:

await page.locator('::-p-aria(Submit)').click();

This can be helpful when building automations that use some modern frameworks that use CSS classes that may be difficult to target using traditional methods but still require some attributes to be present. Some of these frameworks will generate CSS classes at build time which often leaves them as a hash, or shortened version that may change over time.

Wrapping up

Combining Axiom.ai and Puppeteer allows you to build automations that have customisable methods of interacting with webpages and scraping data from them. This gives you the option to use custom CSS selectors beyond what is currently offered within the Axiom.ai extension to build automations that are even more powerful than the no-code method. This can allow you to interact with sites that are traditionally hard to target to power up your automations.

Using Puppeteer also allows you to make use of JavaScript functionality to filter the elements that are returned to make it easier to filter out elements that you do not want to interact with or scrape. Utilising JavaScript allows you to manipulate data during the interaction or scraping steps rather than needing to manipulate this later in your automation.