Build a Google search scraper

What I mean by a Google search scraper

A Google search scraper is a bot that runs a query and reads the results into a sheet. The titles, the links, the snippets, the position each result sits in. Give it a list of queries and it works down them, gathering what comes back. People use it to check where a site ranks, to track results over time, or to pull a topic's results together for research.

Google does not want to be scraped

Here is the honest truth up front. Google is the most defended search box on the internet, and it does not want to be scraped. Run too many queries too fast and it throws a CAPTCHA or an unusual-traffic block, and automated querying is against its terms. This is not a target you can hammer.

So set your expectations. For serious or large-scale search data, Google has an official Custom Search API, and there are SERP data providers built for exactly this, and those are the routes that scale. The browser is for the small and occasional, a handful of queries for your own research or rank checks, run slowly, spaced out, and stopped the moment Google pushes back. When you see a CAPTCHA, that is the signal to back off, not to push through.

Who this is for

This is for the person who needs a few search results, not a data empire. An SEO checking where a page ranks for a handful of keywords, a researcher gathering results on a topic, someone tracking how a result moves week to week. Low volume, your own research. If you need search data at scale, the API is your friend, not a scraper. No-coders and coders both, since you can build it without code and drop into code when you want.

How I'd approach it

Keep it small and slow. Put your queries in a sheet, a handful, not thousands, and have the bot run each one, read the results, and write them back. Put a real pause between queries so you look like a person, not a flood. Watch for any block, and if Google throws a CAPTCHA, stop, do not try to get around it. For anything bigger, switch to the official API.

A few queries, spaced out, stop on a block. I would lay out the first draft with Build with description.

Build a Google search scraper from a description

Describe the queries in plain words in the Chrome extension and it builds the steps for you. Give it a few short lines, check what it made, and run it. Explore no-code.

Build a Google search scraper in code

Build with code. If you would rather script it yourself, this is the path. Explore code

import { chromium } from "playwright";

const browser = await chromium.connectOverCDP(
  `wss://cdp-lb.axiom.ai/?token=${process.env.AXIOM_API_KEY}`
);

try {
  const context = browser.contexts()[0];
  const page = context.pages()[0] ?? await context.newPage();

  const queries = ["best running shoes", "no code automation"];
  const results = [];

  for (const q of queries) {
    await page.goto(`https://www.google.com/search?q=${encodeURIComponent(q)}`);
    await page.waitForSelector("#search");

    const items = await page.$$eval("#search a h3", els =>
      els.map(h => ({
        title: h.textContent.trim(),
        link:  h.closest("a")?.getAttribute("href"),
      }))
    );
    results.push({ query: q, items });

    await page.waitForTimeout(8000); // a real pause, do not flood
  }

  console.log(results); // write these to your sheet
} finally {
  await browser.close();
}

+

Build with a Claude skill

Build no-code or code bots with a skill.

Add the Claude skill and describe the queries you want. It builds the bot for you, no-code or code, running a few searches and reading the results into a sheet, slowly.

Install the skillLearn more

What can you scrape?

Small and slow, for your own research. A couple of cases worth knowing first.

What I'd watch out for

Google is the hardest scrape there is, so read this one carefully. Here is what I would watch for.

When you hit a CAPTCHA, back off

A CAPTCHA or an unusual-traffic message means you have gone too fast. The answer is to slow down or stop, not to push through it. Space your queries out, run fewer, and give it time. There is no version of beating the block that ends well.

Use the API for anything real

For volume, or anything you depend on, Google's official Custom Search API and the SERP data providers exist for a reason. They are built to be queried. A browser scraper is for the small and occasional, not a pipeline.

Results are not the same for everyone

What you scrape depends on location, language, and whether you are signed in. The ranking you see is not the ranking everyone sees, so note the conditions, and do not read too much into one run.

Go slow on purpose

Put real pauses between queries, run a few at a time, and spread them out. The slower you go, the longer it works. Pick what you read with the selector tool, since Google's result markup shifts.