Scrape data from Medium

What I mean by scraping Medium

Scraping Medium means a bot reads the public metadata of articles into a sheet. The title, the author, the clap count, the tags, the publish date, the link. Point it at a tag, a publication, or a writer's page and it gathers those fields across the list, so seeing what topics land and how engagement tracks becomes a spreadsheet instead of a scroll.

Metadata yes, the article no

Here is the line that matters on Medium. The metadata around an article, the title, the author, the claps, the tags, is fair to gather for research, and it is what tells you what is working. The article itself is someone's copyrighted writing, and scraping the full text to republish or repackage is not research, it is taking their work.

So stay on the surface. Counts, titles, tags, and links are the signal you want, and they answer the questions worth asking, what topics get read, who writes them, how engagement moves. The body of the piece is the author's. Read about the articles, do not lift the articles.

Who this is for

This is for the person studying what gets read on Medium. A writer researching topics and engagement, a marketer mapping a niche, an editor tracking a publication. Public metadata, into a sheet. If you are looking to copy articles, this is not the tool. No-coders and coders both, since you can build it without code and drop into code when you want.

How I'd approach it

Pick the tag, publication, or author page you want to study. Have the bot read the title, author, claps, and date of each article into a sheet, then scroll or page for more. Take the metadata, not the body, and keep the pace reasonable. The result is a sheet that shows what topics and writers are landing, without a word of anyone's article copied.

Metadata fields only, into a sheet. I would lay out the first draft with Build with description.

Scrape Medium from a description

Describe the page in plain words in the Chrome extension and it builds the steps for you. Give it a few short lines, check what it made, and run it. Explore no-code.

Scrape Medium in code

Build with code. You do not have to write the script by hand. Describe the scrape to our Claude skill and it generates a ready-to-run Node script you own, then debugs it with you, fixing a selector that stopped matching or a step that stalls, until the run is clean. Prefer to write it yourself? Explore the code tool. Either way, the script looks like this.

1import { AxiomApi } from "axiom-api";2 3const axiom = new AxiomApi(process.env.AXIOM_API_KEY);4 5await axiom.browserOpen();6try {7  await axiom.goto("https://medium.com/tag/web-scraping");8  const articles = await axiom.scrape("article h2");9  console.log(articles); // write these to your sheet10} finally {11  await axiom.browserClose();12}13 

+

Build with a Claude skill

Build no-code or code bots with a skill.

Add the Claude skill and describe the Medium page you want. It builds the bot for you, no-code or code, reading article metadata into a sheet.

Install the skillLearn more

What can you scrape?

Public article metadata, into a sheet. A couple of lines to stay on the right side of.

What I'd watch out for

Medium mixes public metadata with copyrighted writing, so the line is what matters. Here is what I would watch for.

Read about the articles, not the articles

Titles, claps, and tags are the signal and fine to gather. The body is the author's copyrighted work, so do not scrape it to republish or repackage. Stay on the metadata.

Mind the paywall

Members-only posts sit behind a wall, and that content is not yours to take either. Gather the public metadata, and stop at the wall.

Wait for the counts

Claps and other numbers can load a moment after the page. Wait for them to appear before reading, or you will scrape a blank.

Pick fields reliably

Medium changes its markup, so pick what you read with the selector tool and check it holds as you scroll.