[{"data":1,"prerenderedAt":666},["ShallowReactive",2],{"content-\u002Fguides\u002Fscrape-with-claude-code-mcp":3,"content-doc-\u002Fguides\u002Fscrape-with-claude-code-mcp-inline":417,"hero-media-\u002Fguides\u002Fscrape-with-claude-code-mcp":418},{"id":4,"title":5,"body":6,"date":398,"description":399,"extension":400,"meta":401,"navigation":402,"order":398,"path":413,"seo":414,"stem":415,"__hash__":416},"guides\u002Fguides\u002Fscrape-with-claude-code-mcp.md","How to scrape a page with the axiom.ai MCP server and Step API",{"type":7,"value":8,"toc":389},"minimark",[9,12,27,32,35,38,66,70,72,94,98,100,103,113,120,143,260,263,267,269,272,278,285,289,291,294,319,323,325,374,378,380,385],[10,11],"hero-media",{},[13,14,15,16,21,22,26],"p",{},"Scrape a page through ",[17,18,20],"a",{"href":19},"\u002Fdocs\u002Fdeveloper-hub\u002Fmcp","MCP",", whether you ask for it yourself or an agent decides it needs the data. The axiom.ai MCP server runs ",[17,23,25],{"href":24},"\u002Fdocs\u002Fdeveloper-hub\u002Fapi\u002Fstep-functions","Step API"," steps in a real browser and hands the results straight back, ready to save, check or feed into the next task. It runs on demand, with nothing to build or save first. This guide uses Claude Code, though the same tools work from any MCP client.",[28,29,31],"h2",{"id":30},"getting-started","Getting started",[33,34],"hr",{},[13,36,37],{},"You need:",[39,40,41,50,53],"ul",{},[42,43,44,45,49],"li",{},"The axiom.ai ",[17,46,48],{"href":47},"\u002Finstall-desktop-app","desktop app"," installed and logged in. The MCP server ships with it.",[42,51,52],{},"Claude Code installed.",[42,54,55,56,60,61,65],{},"The ",[17,57,59],{"href":58},"\u002Fdocs\u002Fclaude-skill","axiom.ai Claude skill",", a Claude Code plugin that turns a plain-language request into a working automation. Install it with ",[62,63,64],"code",{},"\u002Fplugin install axiom@axiom-skills"," after adding the marketplace.",[28,67,69],{"id":68},"step-1-set-up-the-mcp-server","Step 1: Set up the MCP server",[33,71],{},[13,73,74,75,79,80,84,85,88,89,93],{},"Follow ",[17,76,78],{"href":77},"\u002Fdocs\u002Fdeveloper-hub\u002Fmcp\u002Fdesktop-app-server","Desktop app MCP server"," to set it up in Claude Code. You will be asked for your API key, which you can find under ",[81,82,83],"strong",{},"Credentials and API key"," in the ",[81,86,87],{},"Dashboard",". See ",[17,90,92],{"href":91},"\u002Fdocs\u002Fdeveloper-hub\u002Fapi\u002Fauthentication#generate-an-api-key","Generate an API key"," for the full steps.",[28,95,97],{"id":96},"step-2-ask-for-the-scrape","Step 2: Ask for the scrape",[33,99],{},[13,101,102],{},"Describe what you want:",[104,105,111],"pre",{"className":106,"code":108,"language":109,"meta":110},[107],"language-text","Use the axiom MCP and the Step API scraper to get the headline\nand link from https:\u002F\u002Fwww.bbc.co.uk\u002Fsearch?q=harry+kane&d=NEWS_PS,\n5 pages, into kane.csv.\n","text","",[62,112,108],{"__ignoreMap":110},[13,114,115],{},[116,117],"img",{"alt":118,"src":119},"The scrape prompt in Claude Code, with the agent loading the axiom skill and opening a cloud browser through the MCP","\u002Fguides\u002Fclaude-code-mcp-scrape.webp",[13,121,122,123,126,127,130,131,134,135,138,139,142],{},"That's the whole prompt. The agent opens the page, works out the selectors for the headline and the link, finds the next button, and fills in the rest of the smart scrape step, which takes a ",[62,124,125],{},"url",", a ",[62,128,129],{},"selector"," list, a ",[62,132,133],{},"pager"," for the next button, a ",[62,136,137],{},"max_results"," cap and a ",[62,140,141],{},"settings"," object:",[104,144,148],{"className":145,"code":146,"language":147,"meta":110,"style":110},"language-javascript shiki shiki-themes github-light-default github-dark-default","const rows = await axiom.scrape(\n  \"https:\u002F\u002Fwww.bbc.co.uk\u002Fsearch?q=harry+kane&d=NEWS_PS\",\n  [\n    { selector: \"...\", resultType: \"textContent\" },\n    { selector: \"...\", resultType: \"link\" },\n  ],\n  pager,\n  50,\n  {}\n);\n","javascript",[62,149,150,180,190,196,214,228,234,240,248,254],{"__ignoreMap":110},[151,152,155,159,163,166,169,173,177],"span",{"class":153,"line":154},"line",1,[151,156,158],{"class":157},"sjeE4","const",[151,160,162],{"class":161},"sHrmB"," rows",[151,164,165],{"class":157}," =",[151,167,168],{"class":157}," await",[151,170,172],{"class":171},"s4rv2"," axiom.",[151,174,176],{"class":175},"sbjLL","scrape",[151,178,179],{"class":171},"(\n",[151,181,183,187],{"class":153,"line":182},2,[151,184,186],{"class":185},"sSVrQ","  \"https:\u002F\u002Fwww.bbc.co.uk\u002Fsearch?q=harry+kane&d=NEWS_PS\"",[151,188,189],{"class":171},",\n",[151,191,193],{"class":153,"line":192},3,[151,194,195],{"class":171},"  [\n",[151,197,199,202,205,208,211],{"class":153,"line":198},4,[151,200,201],{"class":171},"    { selector: ",[151,203,204],{"class":185},"\"...\"",[151,206,207],{"class":171},", resultType: ",[151,209,210],{"class":185},"\"textContent\"",[151,212,213],{"class":171}," },\n",[151,215,217,219,221,223,226],{"class":153,"line":216},5,[151,218,201],{"class":171},[151,220,204],{"class":185},[151,222,207],{"class":171},[151,224,225],{"class":185},"\"link\"",[151,227,213],{"class":171},[151,229,231],{"class":153,"line":230},6,[151,232,233],{"class":171},"  ],\n",[151,235,237],{"class":153,"line":236},7,[151,238,239],{"class":171},"  pager,\n",[151,241,243,246],{"class":153,"line":242},8,[151,244,245],{"class":161},"  50",[151,247,189],{"class":171},[151,249,251],{"class":153,"line":250},9,[151,252,253],{"class":171},"  {}\n",[151,255,257],{"class":153,"line":256},10,[151,258,259],{"class":171},");\n",[13,261,262],{},"You can write that call yourself. What the agent adds is reading the real page to fill in the values, instead of you digging through the DOM for them.",[28,264,266],{"id":265},"step-3-pin-the-selectors-once-you-know-them","Step 3: Pin the selectors once you know them",[33,268],{},[13,270,271],{},"Selector discovery costs time and tokens on every run. Once you have selectors that work, put them in the prompt and the agent skips straight to the scrape:",[104,273,276],{"className":274,"code":275,"language":109,"meta":110},[107],"Use axiom mcp and step API to scrape the headline and link from\nhttps:\u002F\u002Fwww.bbc.co.uk\u002Fsearch?q=harry+kane&d=NEWS_PS\ninto kane.csv, 5 pages (50 rows).\n\nSkip selector discovery, use these verified selectors directly:\n- Headline: div[data-testid=\"default-promo\"] p[class*=\"PromoHeadline\"] > span[aria-hidden]  (resultType: textContent)\n- Link:     div[data-testid=\"default-promo\"] a[class*=\"PromoLink\"]  (resultType: link)\n- Next page (pager): nav[aria-label=\"Page\"] div[class*=\"ArrowPageButtonContainer\"]:last-of-type a\n\nOpen ONE browser session. Make ONE scrape step call with:\n  params: [url, [headline descriptor, link descriptor], pager selector, 50, {}]\nLet the pager walk pages 1 to 5 (10 results per page). Do not open multiple\nsessions and do not use page=N URLs.\n\nTrim headlines, write a CSV with columns headline,link, close the session.\n",[62,277,275],{"__ignoreMap":110},[13,279,280,281,284],{},"Being that explicit also stops the agent taking the long way round, opening a session per page or building ",[62,282,283],{},"page=N"," URLs instead of using the pager.",[28,286,288],{"id":287},"go-further-than-scraping","Go further than scraping",[33,290],{},[13,292,293],{},"Claude picks each step from what the last one returned, and the rows land back in the conversation, so ask for more in the same prompt:",[39,295,296,299,306,313,316],{},[42,297,298],{},"Log in first. Type into the form and click submit before the scrape.",[42,300,301,302,305],{},"Drop the page count and say \"keep going until the ",[81,303,304],{},"Next"," button disappears\".",[42,307,308,309,312],{},"Pull ",[62,310,311],{},"href"," values, then visit each one in the same session.",[42,314,315],{},"Ask for a screenshot when a selector comes back empty, then fix it on the spot.",[42,317,318],{},"Send the rows somewhere other than a CSV: a table in the terminal, a JSON file you already have open, a summary of each story.",[28,320,322],{"id":321},"troubleshooting","Troubleshooting",[33,324],{},[326,327,328,341],"table",{},[329,330,331],"thead",{},[332,333,334,338],"tr",{},[335,336,337],"th",{},"Problem",[335,339,340],{},"Fix",[342,343,344,356,366],"tbody",{},[332,345,346,350],{},[347,348,349],"td",{},"Claude can't see the axiom tools",[347,351,352,353,355],{},"The desktop app isn't running, or the server was never set up. See ",[17,354,78],{"href":77},".",[332,357,358,363],{},[347,359,360],{},[62,361,362],{},"session not found",[347,364,365],{},"The session was closed, or the desktop app restarted. Open a new one.",[332,367,368,371],{},[347,369,370],{},"The scrape comes back empty",[347,372,373],{},"The page hadn't finished rendering. Ask for the text again, or take a screenshot to see what the browser sees.",[28,375,377],{"id":376},"wrapping-up","Wrapping up",[33,379],{},[13,381,382,383,355],{},"Describe the data, watch it come back, fix the selector, run it again. To drive the same steps from your own code, see the ",[17,384,25],{"href":24},[386,387,388],"style",{},"html pre.shiki code .sjeE4, html code.shiki .sjeE4{--shiki-default:#CF222E;--shiki-dark:#FF7B72}html pre.shiki code .sHrmB, html code.shiki .sHrmB{--shiki-default:#0550AE;--shiki-dark:#79C0FF}html pre.shiki code .s4rv2, html code.shiki .s4rv2{--shiki-default:#1F2328;--shiki-dark:#E6EDF3}html pre.shiki code .sbjLL, html code.shiki .sbjLL{--shiki-default:#8250DF;--shiki-dark:#D2A8FF}html pre.shiki code .sSVrQ, html code.shiki .sSVrQ{--shiki-default:#0A3069;--shiki-dark:#A5D6FF}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"title":110,"searchDepth":192,"depth":192,"links":390},[391,392,393,394,395,396,397],{"id":30,"depth":182,"text":31},{"id":68,"depth":182,"text":69},{"id":96,"depth":182,"text":97},{"id":265,"depth":182,"text":266},{"id":287,"depth":182,"text":288},{"id":321,"depth":182,"text":322},{"id":376,"depth":182,"text":377},null,"Learn how to scrape a page from Claude Code with the axiom.ai MCP server, which runs Step API steps and hands the data back for the agent to use.","md",{"pinned":402,"type":62,"format":403,"intent_type":404,"intent":405,"tags":406,"platfrom":410,"video":412},true,"guide","workflow","Extract data from websites",[407,408,409],"api","ai","data",[411],"Any","https:\u002F\u002Fwww.youtube.com\u002Fembed\u002FKfCDU9oMP-w?si=QosAUiul2RSdXSye","\u002Fguides\u002Fscrape-with-claude-code-mcp",{"title":5,"description":399},"guides\u002Fscrape-with-claude-code-mcp","kHHVjXYczBYtnuEwMFS2LVj-Oi3_5XwgeXoSoRzTQFQ",{"id":4,"title":5,"body":6,"date":398,"description":399,"extension":400,"meta":401,"navigation":402,"order":398,"path":413,"seo":414,"stem":415,"__hash__":416,"pinned":402,"type":62,"format":403,"intent_type":404,"intent":405,"tags":406,"platfrom":410,"video":412},{"id":4,"title":5,"body":419,"date":398,"description":399,"extension":400,"meta":662,"navigation":402,"order":398,"path":413,"seo":665,"stem":415,"__hash__":416},{"type":7,"value":420,"toc":653},[421,423,429,431,433,435,449,451,453,463,465,467,469,474,478,490,566,568,570,572,574,579,583,585,587,589,605,607,609,643,645,647,651],[10,422],{},[13,424,15,425,21,427,26],{},[17,426,20],{"href":19},[17,428,25],{"href":24},[28,430,31],{"id":30},[33,432],{},[13,434,37],{},[39,436,437,441,443],{},[42,438,44,439,49],{},[17,440,48],{"href":47},[42,442,52],{},[42,444,55,445,60,447,65],{},[17,446,59],{"href":58},[62,448,64],{},[28,450,69],{"id":68},[33,452],{},[13,454,74,455,79,457,84,459,88,461,93],{},[17,456,78],{"href":77},[81,458,83],{},[81,460,87],{},[17,462,92],{"href":91},[28,464,97],{"id":96},[33,466],{},[13,468,102],{},[104,470,472],{"className":471,"code":108,"language":109,"meta":110},[107],[62,473,108],{"__ignoreMap":110},[13,475,476],{},[116,477],{"alt":118,"src":119},[13,479,122,480,126,482,130,484,134,486,138,488,142],{},[62,481,125],{},[62,483,129],{},[62,485,133],{},[62,487,137],{},[62,489,141],{},[104,491,492],{"className":145,"code":146,"language":147,"meta":110,"style":110},[62,493,494,510,516,520,532,544,548,552,558,562],{"__ignoreMap":110},[151,495,496,498,500,502,504,506,508],{"class":153,"line":154},[151,497,158],{"class":157},[151,499,162],{"class":161},[151,501,165],{"class":157},[151,503,168],{"class":157},[151,505,172],{"class":171},[151,507,176],{"class":175},[151,509,179],{"class":171},[151,511,512,514],{"class":153,"line":182},[151,513,186],{"class":185},[151,515,189],{"class":171},[151,517,518],{"class":153,"line":192},[151,519,195],{"class":171},[151,521,522,524,526,528,530],{"class":153,"line":198},[151,523,201],{"class":171},[151,525,204],{"class":185},[151,527,207],{"class":171},[151,529,210],{"class":185},[151,531,213],{"class":171},[151,533,534,536,538,540,542],{"class":153,"line":216},[151,535,201],{"class":171},[151,537,204],{"class":185},[151,539,207],{"class":171},[151,541,225],{"class":185},[151,543,213],{"class":171},[151,545,546],{"class":153,"line":230},[151,547,233],{"class":171},[151,549,550],{"class":153,"line":236},[151,551,239],{"class":171},[151,553,554,556],{"class":153,"line":242},[151,555,245],{"class":161},[151,557,189],{"class":171},[151,559,560],{"class":153,"line":250},[151,561,253],{"class":171},[151,563,564],{"class":153,"line":256},[151,565,259],{"class":171},[13,567,262],{},[28,569,266],{"id":265},[33,571],{},[13,573,271],{},[104,575,577],{"className":576,"code":275,"language":109,"meta":110},[107],[62,578,275],{"__ignoreMap":110},[13,580,280,581,284],{},[62,582,283],{},[28,584,288],{"id":287},[33,586],{},[13,588,293],{},[39,590,591,593,597,601,603],{},[42,592,298],{},[42,594,301,595,305],{},[81,596,304],{},[42,598,308,599,312],{},[62,600,311],{},[42,602,315],{},[42,604,318],{},[28,606,322],{"id":321},[33,608],{},[326,610,611,619],{},[329,612,613],{},[332,614,615,617],{},[335,616,337],{},[335,618,340],{},[342,620,621,629,637],{},[332,622,623,625],{},[347,624,349],{},[347,626,352,627,355],{},[17,628,78],{"href":77},[332,630,631,635],{},[347,632,633],{},[62,634,362],{},[347,636,365],{},[332,638,639,641],{},[347,640,370],{},[347,642,373],{},[28,644,377],{"id":376},[33,646],{},[13,648,382,649,355],{},[17,650,25],{"href":24},[386,652,388],{},{"title":110,"searchDepth":192,"depth":192,"links":654},[655,656,657,658,659,660,661],{"id":30,"depth":182,"text":31},{"id":68,"depth":182,"text":69},{"id":96,"depth":182,"text":97},{"id":265,"depth":182,"text":266},{"id":287,"depth":182,"text":288},{"id":321,"depth":182,"text":322},{"id":376,"depth":182,"text":377},{"pinned":402,"type":62,"format":403,"intent_type":404,"intent":405,"tags":663,"platfrom":664,"video":412},[407,408,409],[411],{"title":5,"description":399},1790161898462]