Bypass bot detection
A number of websites use automated bot blocking technologies to protect themselves against malicious attacks. Unfortunately, harmless bots can be caught in the crossfire. Particularly, Cloudflare is a widely-used bot blocking technology that will often automatically block automated activity of any kind.
Cloudflare and other such services detect activity that is fully automated, but will not block activity that it sees as coming from a legitimate user. By attaching your automation run to an existing Chrome window which has already validated you as a real user, you can reassure the bot blocker that the traffic is not malicious in nature.
# Download and install the Axiom.ai desktop application
To use this method of bypassing bot detection, you will need to download and install the Axiom.ai desktop application.
You can find the application here.
# Select the method to bypass bot blocking
By default, Axiom.ai does not attempt to bypass bot detection. This is because the checks slow down automations.
To turn on bot blocking, open the automation which is being blocked, from the dashboard.
Click the three dots menu in the top-right of the Axiom.ai builder, then click on the gear icon ‘Settings’ (alternatively, click the gear icon, in the vertical toolbar on the left side of the builder) and select the "Bypass bot detection" tab in the left panel.
There are three options to choose from:
How
- No bot detection bypass - (selected by default).
- Use automatic bot detection bypassing.
- Attach to existing Chrome.
# Use automatic bot detection bypassing
When this mode is switched on, Axiom will automatically detect and solve bot challenges where possible.
Although this works for many sites, it is not infallible; sometimes you may need to solve the captcha yourself. There are two ways, depending on the Captcha type.
The first is to use the captcha solving step. This is required for more complicated captchas.
The best way to do this is to record "Keystrokes":
How
- Record "Tab" the correct number of times to highlight the checkbox.
- Record "Space" to complete the challenge.
# Attach to existing Chrome
When this method is selected, you need to give Chromium permission to use your window, or Axiom.ai will not be allowed to connect.
A button to quickly open such a window can be found in the "Bypass bot detection" section of the settings page.
Alternatively, you can start your own version of Chromium manually.
To do this, an extra argument needs to be passed to Chromium when starting it. For example, you can enter this in your terminal (altering the path to your Chromium executable as required):
/path/to/Chrome/chrome --remote-debugging-port=21222
You can also create a shortcut or symbolic link with this parameter specified, for easier access.
# Solve the bot blocker's challenges
After you've started your new window, open it and navigate to the website which is deploying bot blocking, and navigate to the page or section of the website where the bot blocking is taking place.
The website will now validate whether you are a real user or not. Often this happens automatically; sometimes you will need to solve a Captcha.
Once you've been validated, we're ready to let Axiom.ai take over the browser!
# Adjust your Axiom.ai for bypassing bot detection
Depending on the automation, you may need to make some adjustments to your automation so that it starts in the correct place. The best way to maximise the chances of being allowed in is to make sure your automation is set to start on the page you have bypassed bot detection on.
For example, if this is after login (a common occurrence) then perform the login manually on your open browser window, then adjust your automation so that it performs only the steps after the login.
Once you click the "Run" button, Axiom.ai should begin to run these steps in the window which has already solved the challenges.
Note: Automations running in this mode do not close the Chrome / Chromium window when they are stopped; instead the automation will finish what it is doing and then stop running.