Batching is a great way to optimize your large bot runs that involve looping through hundreds of rows of data. For example, these could be scraping tasks or data entry.
Batching is a design pattern we use all the time at Axiom. This template will teach you how and is easily adapted to your requirements.
To fully use this template, you will need a Google Sheet with multiple rows of data as a starting point.
# Advantages to using this design pattern
- Data is retrieved continuously - on every loop
- A bot can resume if interrupted, or if an error occurs
- Batches can be scheduled at regular intervals
# Install the Batching template
Just click 'Install template'. If you are a new user you will be required to create an Axiom.ai account before you can edit the template.
# Get started open the bot for editing
On the Axiom dashboard, hover over the text 'How to configure bots for large runs' and click.
# Set up your Google Sheet
Get a Google Sheet ready with a few rows of data. You don't need a complete set of data. It's best to test with a few rows first. Also create a new tab in the sheet to use later on, why not call it 'data'.
# Set up the 'Read data from Google Sheet' step
- Spreadsheet - In the field called 'Spreadsheet', you can search for the Google Sheet you created. Once found, click to select.
- Sheet name - Choose the sheet with the data you wish to use.
- First cell - Set to 'A1'.
- Last cell - Enter the end colum of data use wish to pass for example 'AD1'.
- You should now see a single row of data in the preview.
For batching to work, you must parse a single line of data from the Google sheet.
# Interact with a page's interface
If you are scraping and want to parse links from the Google sheet in 'Go to page.' Click 'Inset data' and select the Google sheet choosing the column that contains the links in the preview which will appear.
If you are entering data, cut and paste the URL of the target page into the 'Go to Page' field.
To turn this design template into a functioning bot, start adding sub-steps For example, if you want to create a scraper, you can add a 'Get data from a Webpage' sub-step, or if you're going to enter data, you can add 'Enter text' steps and use the 'Click element' steps.
# Set up the 'Write Data to a Google Sheet' step
You can use this step to record your output, if scraping data:
- Spreadsheet - In the field called 'Spreadsheet', you can search for the Google Sheet you created. Once found, click to select.
- Sheet name - Choose the extra tab you created.
- DATA - Select the 'Interact data'.
- Clear data before writing | Add to existing data - Set this option to 'Add to existing data'.
If doing data entry, use the step to record entries sent:
- Spreadsheet - In the field called 'Spreadsheet', you can search for the Google Sheet you created. Once found, click to select.
- Sheet name - Choose the extra tab you created.
- DATA - Select the 'Google-sheet-data'.
- Clear data before writing | Add to existing data - Set this option to 'Add to existing data'.
# Set up the 'Delete rows from a Google Sheet' step
This step deletes a row from the sheet so that when the bot loops, it starts from a new row
- Spreadsheet - In the field called 'Spreadsheet', you can search for the Google Sheet you created. Once found, click to select.
- Sheet name - Choose the tab you created for links.
- First row - set to 1.
- Last row - set to 1.
# set up the 'Jump to another step' step
This steps loops the bot creating the batch you can set the amount of loops for each batch in Maxium cycles.
- Jump to step - set the step you want to jump to make the bot loop in this case step one.
- Maximum cycles - set the amount of loops the bot must perform
# Test run
We always recommed doing a test run - Set Maxium cycles to three.
# Troubleshooting
Try the following if your bot is not working. We would also recommend you watch the video to troubleshoot.
- The bot loops through the same row. Check the 'Delete rows from a Google Sheet' is set to the correct sheet.
- The bot still loops throught the same row, check in the 'Jump to another step' you are jumping back to the 'Read data from Google Sheet' step.