Scrape data behind a login

What I mean by scraping behind a login

Scraping behind a login means a bot reads data from a page you have to sign in to see. A dashboard's numbers, a portal's records, an account's history, the things that are not on the public site. It gets past the sign-in, lands on the page you want, and reads the same fields you would, into a sheet. The login is just the door, the data behind it is the point.

Getting in is two-thirds of the job

With a public page, you open it and scrape. Behind a login, there is a step first, the bot has to be signed in, and that is where most of the work is. Get that right and the scraping is the easy part.

There are two ways through. Carry your session, where the bot uses the cookies from a browser you are already signed into, so it never handles a password. Or sign in as steps, where the bot enters the username and password itself, with the credentials kept somewhere secure, never in plain text. Cookies are simpler and safer when they work. Steps suit a fresh cloud run with no session to borrow. Either way, once the bot is in, it reads the page like any other. Scrape only your own data, or data you are allowed to see.

Who this is for

This is for anyone whose data lives behind a sign-in. Pulling figures from a dashboard you log into, exporting records from a portal, gathering your own history from an account. The data is yours to see, you just want it in a sheet without the clicking. No-coders and coders both, since you can build it without code and drop into code when you want.

How I'd approach it

Solve the login first. Try cookies, so the bot carries your existing session and there is no password in the automation. If you need a fresh sign-in, store the credentials securely and have the bot enter them. Then point it at the page behind the wall, wait until you are actually in, and scrape the fields you want into a sheet. Build the scrape once you are reliably getting past the door.

Get in with cookies if you can, then read the page behind it. I would lay out the first draft with Build with description.

Scrape behind a login from a description

Describe the sign-in and the data in plain words in the Chrome extension and it builds the steps for you. Give it a few short lines, check what it made, and run it. Explore no-code.

Scrape behind a login in code

Build with code. You do not have to write the script by hand. Describe the login to our Claude skill and it generates a ready-to-run Node script you own, then debugs it with you, fixing a selector that stopped matching or a step that stalls, until the run is clean. Prefer to write it yourself? Explore the code tool. Either way, the script looks like this.

1import { AxiomApi } from "axiom-api";2 3const axiom = new AxiomApi(process.env.AXIOM_API_KEY);4 5await axiom.browserOpen();6try {7  await axiom.goto("https://example.com/login");8  await axiom.enterText("#email", process.env.SITE_EMAIL);9  await axiom.enterText("#password", process.env.SITE_PASSWORD);10  await axiom.click("button[type=submit]");11} finally {12  await axiom.browserClose();13}14 

+

Build with a Claude skill

Build no-code or code bots with a skill.

Add the Claude skill and describe the login and the data. It builds the bot for you, no-code or code, getting past the sign-in and reading the page into a sheet.

Install the skillLearn more

What can you scrape?

Whatever you can see once you are in. A couple of cases worth knowing first.

What I'd watch out for

The login is the part that makes or breaks this, so most of the care goes there. Here is what I would watch for.

Cookies first, passwords second

Carrying your existing session means no password in the bot at all, which is simpler and safer. Fall back to a sign-in step only when there is no session to borrow, and store the credentials securely, never in plain text.

Sessions expire

A stored session does not last forever. When the bot suddenly lands on the login page, the cookies have run out. Refresh them, or fall back to signing in, and build for the day it happens.

Wait until you are in

Scraping before the login finishes reads the wrong page. Wait for something that only appears once you are signed in before the bot starts reading.

Only your own data

Get in with your own credentials and read what you are allowed to see. This is for your own dashboards and accounts, not for reaching into someone else's. Pick fields with the selector tool.