Scrape data from Medium

What I mean by scraping Medium

Scraping Medium means a bot reads the public metadata of articles into a sheet. The title, the author, the clap count, the tags, the publish date, the link. Point it at a tag, a publication, or a writer's page and it gathers those fields across the list, so seeing what topics land and how engagement tracks becomes a spreadsheet instead of a scroll.

Metadata yes, the article no

Here is the line that matters on Medium. The metadata around an article, the title, the author, the claps, the tags, is fair to gather for research, and it is what tells you what is working. The article itself is someone's copyrighted writing, and scraping the full text to republish or repackage is not research, it is taking their work.

So stay on the surface. Counts, titles, tags, and links are the signal you want, and they answer the questions worth asking, what topics get read, who writes them, how engagement moves. The body of the piece is the author's. Read about the articles, do not lift the articles.

Who this is for

This is for the person studying what gets read on Medium. A writer researching topics and engagement, a marketer mapping a niche, an editor tracking a publication. Public metadata, into a sheet. If you are looking to copy articles, this is not the tool. No-coders and coders both, since you can build it without code and drop into code when you want.

How I'd approach it

Pick the tag, publication, or author page you want to study. Have the bot read the title, author, claps, and date of each article into a sheet, then scroll or page for more. Take the metadata, not the body, and keep the pace reasonable. The result is a sheet that shows what topics and writers are landing, without a word of anyone's article copied.

Metadata fields only, into a sheet. I would lay out the first draft with Build with description.

Scrape Medium from a description

Describe the page in plain words in the Chrome extension and it builds the steps for you. Give it a few short lines, check what it made, and run it. Explore no-code.

Scrape Medium in code

Build with code. If you would rather script it yourself, this is the path. Explore code

import { chromium } from "playwright";

const browser = await chromium.connectOverCDP(
  `wss://cdp-lb.axiom.ai/?token=${process.env.AXIOM_API_KEY}`
);

try {
  const context = browser.contexts()[0];
  const page = context.pages()[0] ?? await context.newPage();

  await page.goto("https://medium.com/tag/web-scraping");
  await page.waitForSelector("article");

  // Read the public metadata, not the article bodies
  const articles = await page.$$eval("article", els =>
    els.map(el => ({
      title:  el.querySelector("h2")?.textContent.trim(),
      author: el.querySelector("[data-testid='authorName']")?.textContent.trim(),
      link:   el.querySelector("a")?.getAttribute("href"),
    }))
  );

  console.log(articles); // write these to your sheet
} finally {
  await browser.close();
}

+

Build with a Claude skill

Build no-code or code bots with a skill.

Add the Claude skill and describe the Medium page you want. It builds the bot for you, no-code or code, reading article metadata into a sheet.

Install the skillLearn more

What can you scrape?

Public article metadata, into a sheet. A couple of lines to stay on the right side of.

What I'd watch out for

Medium mixes public metadata with copyrighted writing, so the line is what matters. Here is what I would watch for.

Read about the articles, not the articles

Titles, claps, and tags are the signal and fine to gather. The body is the author's copyrighted work, so do not scrape it to republish or repackage. Stay on the metadata.

Mind the paywall

Members-only posts sit behind a wall, and that content is not yours to take either. Gather the public metadata, and stop at the wall.

Wait for the counts

Claps and other numbers can load a moment after the page. Wait for them to appear before reading, or you will scrape a blank.

Pick fields reliably

Medium changes its markup, so pick what you read with the selector tool and check it holds as you scroll.