How to monitor a web page and trigger webhook

Monitor a website for new products then send a notification via a webhook, this bot is a simple and effective way to achieve your goal. You will need to use this bot on a listing or search page.

The bot scrapes a list of content, such as titles or prices comparing them against a previous scrape to monitor for changes.

This template is simple and adaptable to different websites, from products to stock listings. A Google Sheet is required to store data for comparison.

To use this template, you will first need to scrape a list of data into your sheet to compare against. We suggest you adpat this template.

# Advantages to using this web monitoring design pattern

  1. Adaptable to different types of website
  2. Monitoring can be scheduled at regular intervals

# Install the Monitor bot template

On this page click 'Install template'. If you are a new user you will be required to create an account before you can edit the template.

# Prepare a Google Sheet

A Google sheet is required with a list of data to compare against.

# Scrape a list of content to compare against

We suggest you adpat this template.

# Get started open the bot for editing

On the Axiom dashboard, hover over the text 'Monitor web page and send email' and click.

# Set up the 'Get data from a webpage' step

This step scrapes the data you wish to compare.

  1. URL - Insert the link of the page you wish to monitor.
  2. SELECT - Select the content you wish to monitor, Its as easy as point and click, this video shows you how.
  3. Find pager (optional) - Select the next button if the page has a pager. If it scrolls then do nothing.
  4. Max Results - When testing set to 10-20 results max.

# Set up the 'Read data from Google Sheet' step

This step stores the data for comparison on each run.

  1. Spreadsheet - In the field called 'Spreadsheet', you can search for the Google Sheet you created. Once found, click to select.
  2. Sheet name - Choose the sheet that contains the comparision data.
  3. First cell - No changed required.
  4. Last cell - No changed required.
  5. You should now see a preview of the data.

# Remove results which do not contain certain words compare data in Axiom

This steps filters out the old content leaving the new.

  1. Data - Click 'Insert data and select 'scape-data'.
  2. Words (comma separated) - lick 'Insert data and select 'google-sheet-data'.
  3. Word matching mode - leave set to 'Any'.
  4. Match on word boundary - leave set to unticked.
  5. Columns (optional) - leave on defualt.

# Continue only if a condition is met

The conditional step stops the bot from sending a message if no new content is not found.

  1. Data to check - Click insert data and then choose '[matching-word-data]'
  2. Words to check for (comma separated) - Leave set to (a,b,c,d,e,f,g,h,i,j,k,l,m,n,o,p,q,r,s,t,u,v,w,x,y,z,0,1,2,3,4,5,6,7,8,9)
  3. Word matching mode - Leave set on 'Any'
  4. Match on word boundary - Leave unticked.
  5. Fail if condition not met - Tick this box.
  6. Reverse condition - Leave unticked. condition step

# Set up the 'Write Data to a Google Sheet' step

Writes new data to the sheet.

  1. Spreadsheet - In the field called 'Spreadsheet', you can search for the Google Sheet you created. Once found, click to select.
  2. Sheet name - Choose the tab you created with the existing content to check against.
  3. DATA - Select the '[word-matching-data]'.
  4. Clear data before writing | Add to existing data - Set this option to 'Add to existing data'.

# Trigger webhook

  1. Endpoint - Insert the endpoint.
  2. Payload - Enter the payload for the request, as JSON.

See this example:

  "name": "test webhook",
  "type": 1,
  "channel_id": "199737254929760256",
  "token": "3d89bb7572e0fb30d8128367b3b1b44fecd1726de135cbe28a41f8b2f777c372ba2939e72279b94526ff5d1bd4358d65cf11",
  "avatar": null,
  "guild_id": "199737254929760256",
  "id": "223704706495545344",
  "application_id": null,
  "user": {
    "username": "test",
    "discriminator": "7479",
    "id": "190320984123768832",
    "avatar": "b004ec1740a63ca06ae2e14c5cee11f3",
    "public_flags": 131328

# Test run

We always recommend doing a test run - You should test the filter and conditional. Do the first scrape, then delete some rows to test the full bot. You can also test if the webhook is triggered.

# Troubleshooting the monitoring Axiom

As mentioned at the top of the page, you will need to scrape data to compare against before running this bot.

  • All results write to the Google Sheet, check the 'Write' Step is set to '[word-matching-data]'.
  • No message sent - Check your JSON

# More starter design templates