Managing Large Bot Runs

Small bots may not need much design effort - they may only execute 1 or 2 steps in a short space of time.

For large bots, processing large data sets, or scraping many pages, we recommend following these guidelines to produce robust automations.

# Use Google Sheets for large runs

We recommend using Google Sheets as your primary data source in nearly all cases.

This applies even if you have files like CSVs (they can be imported into Sheets), and even if you are sending data from Zapier or Make (see below - write data to Sheets from Zapier).

Please avoid reading and writing to a single sheet for your whole automation - this makes your automations harder to structure.

Instead, we recommend using separate 'INPUT' and 'OUTPUT' sheets, with the following (example) structure.

# Structure a bot for large runs

  1. Read Data from Google Sheet
  2. Loop Through Data
    • 2.1 Go to page
    • 2.2 Get data from a webpage (Or any other actions)
    • 2.3 Write data to a Google Sheet
    • 2.4 Delete rows from Google Sheet
  • Step 1 - Add URLs, or form input data, to your INPUT sheet
  • Step 2 - Perform any data scraping or data input actions
  • Step 2.3 - Write any scraped data to your OUTPUT sheet. If doing data entry, we recommend writing completed rows.
  • Step 2.4 - Delete completed rows from your INPUT sheet. This allows your bot to easily resume, if interrupted.

# Set a maximum runtime

Axiom contains a setting for maximum runtime. This is useful if you have a large bot and don't want it to accidentally use up all of the runtime for your plan.

You can alter this setting from the configuration page for your individual axiom.

For the cloud, the maximum runtime cannot exceed the single run limit for your plan.

# Send data from Zapier, Make & other platforms

When sending data from Zapier or Make, it's possible to either send data directly, or to write data to Google Sheets first.

Sending data directly works fine for small quantities of data.

However, if your data-set contains many columns, or more than 1 row, we recommend using Google Sheets to transfer data:

  • First, structure your bot to read from a Google Sheet
  • Secondly, Write data to Google Sheets from Zapier / Make
  • Finally, Trigger Axiom from Zapier / Make. Sending data is not required, as your bot will use the Google Sheet instead.