On this article, you’ll learn to construct AI brokers that may browse and work together with actual web sites utilizing Playwright, browser-use, and LangGraph.
Matters we’ll cowl embrace:
- Why Playwright is the precise basis for browser automation in 2026, and the way it differs from Selenium.
- How you can scrape dynamic, JavaScript-rendered pages and full multi-step types reliably.
- How you can wire browser actions into LangGraph and browser-use brokers, deal with anti-bot detection, handle ready and session persistence, and deploy the end in Docker.
Constructing Browser-Utilizing AI Brokers in Python
Introduction
Most AI agent tutorials begin with an API. They present you easy methods to name OpenWeather, hit the Stripe endpoint, pull knowledge from GitHub. That may be a nice place to begin till you attempt to construct one thing actual and notice that the duty you really want performed doesn’t have an API.
Take into consideration what people do with browsers daily: submitting authorities types, studying competitor pricing, extracting analysis from websites that guard their knowledge behind JavaScript rendering, logging into portals which have by no means heard of OAuth. There are roughly 1.1 billion web sites on the web. A vanishingly small fraction of them have public APIs. The remainder solely communicate browser.
An agent that’s restricted to API calls handles perhaps 5% of the duties a human employee does day by day. Give that agent a browser, and the protection approaches every little thing. That’s the hole this text closes.
The global AI agents market stands at $10.91 billion in 2026 and is projected to succeed in $50.31 billion by 2030, with browser-capable brokers on the middle of that progress. 27.7% of enterprises are already operating agentic browsers in manufacturing, up from nearly none two years prior. The tooling has matured quick, and the patterns are settled sufficient to show correctly.
By the top of this text, you should have a working browser agent that navigates actual web sites, fills types, extracts structured knowledge, and connects to an LLM that decides what to do subsequent, all in Python.
Why Playwright, Not Selenium
When you constructed browser automation 5 years in the past, you constructed it with Selenium. Selenium continues to be broadly deployed, nonetheless works, and isn’t going anyplace. However for any new venture in 2026, Playwright is the default. The explanations are sensible, not theoretical.
Selenium communicates with the browser by sending particular person HTTP requests to a WebDriver. Each motion, click on, sort, scroll, is a separate request. Playwright makes use of a persistent WebSocket connection for the complete session. Instructions circulate via that channel with no per-action round-trip value. Unbiased benchmarks persistently present Playwright operating 30-50% sooner than Selenium on the test-suite stage and averaging ~290ms per motion versus Selenium’s ~536ms. For a browser agent that may execute a whole lot of actions, that hole compounds.
Playwright additionally bundles its personal browser binaries. While you set up it, you get pre-configured variations of Chromium, Firefox, and WebKit which can be assured to work together with your Playwright model. No driver model mismatches, no damaged CI pipelines as a result of somebody up to date Chrome. It has built-in auto-waiting earlier than it clicks a component; it verifies the aspect is seen, enabled, and never animating. You would not have to write down time.sleep(2) and hope for the most effective.
For AI brokers particularly, Playwright fires actual mouse and keyboard occasions that mirror how people work together with browsers. Websites designed to detect automation search for artificial DOM clicks. Playwright’s interplay mannequin is tougher to tell apart from real human enter.
There may be additionally the browser-use library, which sits one stage greater. Browser-use is a Python library that provides an LLM a working browser. Underneath the hood, it makes use of Playwright to drive the browser, however the LLM reads the web page state and decides what to click on, sort, and extract, no CSS selectors required. You give it a process in plain English, and it figures out the remaining. We are going to cowl each uncooked Playwright and browser-use on this article, as a result of they serve completely different wants: Playwright once you need exact, predictable management; browser-use once you need the agent to deal with navigation selections autonomously.
Setting Up the Setting
You want Python 3.10 or greater, an OpenAI API key, and about 5 minutes.
Step 1: Create a digital surroundings
|
python –m venv browser_agent_env
# macOS / Linux supply browser_agent_env/bin/activate
# Home windows browser_agent_envScriptsactivate |
Step 2: Set up dependencies
|
pip set up playwright browser–use langchain langchain–openai langgraph langchain–neighborhood python–dotenv |
Step 3: Set up the browser binaries
That is the step most individuals miss. Playwright must obtain Chromium, Firefox, and WebKit individually from the Python bundle. Run this as soon as after putting in:
|
playwright set up chromium |
If you’d like all three browser engines: playwright set up. Chromium alone is enough for many agent work and is smaller to obtain.
Step 4: Retailer your API key
Create a .env file in your venture listing:
|
OPENAI_API_KEY=your_openai_api_key_here |
Add .env to your .gitignore instantly. Don’t commit API keys.
Step 5: Confirm every little thing works
Here’s a first script that navigates to a URL, reads the heading, and saves a screenshot. Use example.com, a publicly out there check area maintained by IANA that won’t block you.
How you can run: Save as first_run.py and run python first_run.py
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 |
# first_run.py # Navigate to a URL, take a screenshot, and extract the web page title. # Stipulations: pip set up playwright && playwright set up chromium # How you can run: python first_run.py
import asyncio from playwright.async_api import async_playwright
async def most important(): async with async_playwright() as p: # Launch Chromium in headless mode (no seen browser window). # Set headless=False if you wish to watch it run throughout growth. browser = await p.chromium.launch(headless=True)
# A browser context is sort of a recent browser profile. # It isolates cookies, storage, and cache from different contexts. context = await browser.new_context( viewport={“width”: 1280, “top”: 720}, user_agent=( “Mozilla/5.0 (Home windows NT 10.0; Win64; x64) “ “AppleWebKit/537.36 (KHTML, like Gecko) “ “Chrome/120.0.0.0 Safari/537.36” ) )
web page = await context.new_page()
# Navigate to the URL and wait till the community is idle. # “networkidle” means no open community connections for 500ms. # For sooner pages, “domcontentloaded” is enough. await web page.goto(“https://instance.com”, wait_until=“networkidle”)
# Extract the web page title title = await web page.title() print(f“Web page title: {title}”)
# Extract the textual content content material of the h1 heading h1 = await web page.text_content(“h1”) print(f“H1 heading: {h1}”)
# Take a full-page screenshot and reserve it to disk await web page.screenshot(path=“screenshot.png”, full_page=True) print(“Screenshot saved to screenshot.png”)
await browser.shut()
asyncio.run(most important()) |
What this does: async_playwright() is the entry level for the complete Playwright session. The browser_context is equal to opening a recent incognito window; cookies, native storage, and cache are remoted from every little thing else. wait_until=”networkidle” tells Playwright to attend till the web page has completed all its community exercise earlier than your code continues, which is the most secure wait technique for dynamic pages.
If this runs and saves a screenshot, your surroundings is working appropriately.
Net Navigation and Scraping
The rationale you want Playwright as an alternative of requests + BeautifulSoup is JavaScript rendering. Fashionable web sites ship a skeleton of HTML after which construct the precise content material dynamically after the web page masses: React, Vue, Angular, Subsequent.js. A plain HTTP request fetches the skeleton. Playwright runs an actual browser, so it sees precisely what a human sees in any case JavaScript has executed.
The goal beneath is books.toscrape.com, a authorized scraping sandbox constructed for apply. It paginates outcomes, makes use of dynamic class names for scores, and intently mirrors the construction of actual e-commerce product pages.
How you can run: Save as scrape_books.py and run python scrape_books.py
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 |
# scrape_books.py # Scrape guide titles, costs, and scores from books.toscrape.com # It is a authorized scraping sandbox web site constructed for apply. # Stipulations: pip set up playwright && playwright set up chromium # How you can run: python scrape_books.py
import asyncio import json from playwright.async_api import async_playwright
async def scrape_books(max_pages: int = 3) -> record[dict]: “”“ Scrape guide listings from books.toscrape.com throughout a number of pages. Returns an inventory of dicts with title, worth, ranking, and web page quantity. ““” outcomes = []
async with async_playwright() as p: browser = await p.chromium.launch(headless=True) context = await browser.new_context(viewport={“width”: 1280, “top”: 720}) web page = await context.new_page()
for page_num in vary(1, max_pages + 1): url = f“https://books.toscrape.com/catalogue/page-{page_num}.html” print(f“Scraping web page {page_num}: {url}”)
await web page.goto(url, wait_until=“domcontentloaded”)
# Anticipate the product playing cards to be seen earlier than extracting. # That is crucial on JavaScript-heavy pages the place content material masses after the HTML. # timeout=10000 means wait as much as 10 seconds earlier than elevating an error. await web page.wait_for_selector(“article.product_pod”, timeout=10000)
# Get all guide playing cards on the present web page books = await web page.query_selector_all(“article.product_pod”)
for guide in books: # Extract title from the <a> tag’s title attribute title_el = await guide.query_selector(“h3 a”) title = await title_el.get_attribute(“title”) if title_el else “N/A”
# Extract worth textual content price_el = await guide.query_selector(“.price_color”) worth = await price_el.inner_text() if price_el else “N/A”
# Extract star ranking from the CSS class identify. # e.g. <p class=”star-rating Three”> → “Three” rating_el = await guide.query_selector(“p.star-rating”) rating_class = await rating_el.get_attribute(“class”) if rating_el else “” ranking = rating_class.exchange(“star-rating”, “”).strip()
outcomes.append({ “title”: title, “worth”: worth, “ranking”: ranking, “web page”: web page_num })
print(f” Extracted {len(books)} books from web page {page_num}”)
await browser.shut()
return outcomes
async def most important(): books = await scrape_books(max_pages=2) print(f“nTotal books scraped: {len(books)}”) print(json.dumps(books[:3], indent=2))
asyncio.run(most important()) |
What this does: wait_for_selector() is the important thing name right here. As an alternative of sleeping for a set time and hoping the content material has loaded, it watches the DOM and proceeds the second the goal aspect seems, or raises a TimeoutError if it doesn’t seem throughout the timeout window. That’s the proper conduct: fail quick and explicitly moderately than silently extracting from an empty web page.
The ranking extraction deserves consideration. The star ranking is encoded as a CSS class (star-rating Three), not a quantity. The code strips “star-rating” from the category string to get the textual content worth. That is the form of factor you solely know by inspecting the precise HTML. While you hand this process to a uncooked LLM with no browser, it has no approach to know what the category construction appears like. With Playwright, you may examine it instantly and extract it precisely.
Kind Completion and Multi-Step Flows
Filling types is the place browser brokers earn their hold and the place most automation scripts fail. The reason being that net types will not be simply inputs and buttons. They hearth focus, enter, change, and blur occasions in sequence. JavaScript validation listens for these occasions. When you inject a price into an enter discipline by instantly setting worth within the DOM (as older automation instruments usually do), the validation listeners by no means hearth and the shape breaks.
Playwright’s fill() and click on() strategies hearth actual browser occasions in the precise order, which is why they work on kind validation that may block lower-level approaches.
The goal beneath is the-internet.herokuapp.com/login, a public check web site maintained particularly for automation apply. It accepts tomsmith / SuperSecretPassword! as legitimate credentials and returns clear success/failure messages.
How you can run: Save as form_submit.py and run python form_submit.py
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 |
# form_submit.py # Full and submit a multi-field login kind on a public demo web site. # Goal: https://the-internet.herokuapp.com/login (public check web site) # Stipulations: pip set up playwright && playwright set up chromium # How you can run: python form_submit.py
import asyncio from playwright.async_api import async_playwright
async def login_and_verify(username: str, password: str) -> dict: “”“ Try to log in to a demo web site and return whether or not it succeeded. Handles: enter filling, button clicking, and end result verification. ““” async with async_playwright() as p: browser = await p.chromium.launch(headless=True) context = await browser.new_context() web page = await context.new_page()
await web page.goto(“https://the-internet.herokuapp.com/login”)
# Anticipate the shape to be seen earlier than interacting. # state=”seen” is the default however makes the intent express. await web page.wait_for_selector(“#username”, state=“seen”)
# fill() clears the sector first, then varieties the worth. # It fires the main focus, enter, and alter occasions so as. await web page.fill(“#username”, username) await web page.fill(“#password”, password)
# click on() fires actual mouse occasions — mousedown, mouseup, click on. # This triggers JavaScript listeners {that a} plain DOM click on misses. await web page.click on(“button[type=”submit”]”)
# Anticipate the web page to settle after kind submission await web page.wait_for_load_state(“networkidle”)
# Test which end result aspect appeared success_el = await web page.query_selector(“.flash.success”) error_el = await web page.query_selector(“.flash.error”)
if success_el: message = await success_el.inner_text() end result = {“success”: True, “message”: message.strip()} elif error_el: message = await error_el.inner_text() end result = {“success”: False, “message”: message.strip()} else: end result = {“success”: False, “message”: “Unknown end result”}
await browser.shut() return end result
async def most important(): # Legitimate credentials for the demo web site end result = await login_and_verify(“tomsmith”, “SuperSecretPassword!”) print(f“Legitimate login: {end result}”)
# Invalid credentials to confirm error dealing with result_fail = await login_and_verify(“wronguser”, “wrongpass”) print(f“Invalid login: {result_fail}”)
asyncio.run(most important()) |
What this does: The sample right here, fill() → click on() → wait_for_load_state() → verify for end result aspect, is the template for nearly any kind interplay. The wait_for_load_state(“networkidle”) after the submit is vital: with out it, you question the DOM earlier than the web page has up to date and get the pre-submission state, not the end result.
For extra advanced types with file uploads, dropdowns, and checkboxes:
|
# File add await web page.set_input_files(“#file-upload”, “/path/to/doc.pdf”)
# Choose dropdown by seen label textual content await web page.select_option(“#country-select”, label=“Nigeria”)
# Test a checkbox await web page.verify(“#agree-terms”)
# Deal with a modal dialog (affirm/alert) web page.on(“dialog”, lambda dialog: asyncio.ensure_future(dialog.settle for())) |
Device Orchestration with LangChain and LangGraph
Uncooked Playwright scripts are highly effective however mounted. They do precisely what you coded, no extra. The second a web page adjustments its construction, or the duty requires a choice the script didn’t anticipate, it breaks.
Connecting Playwright to an LLM adjustments this. Browser actions change into instruments the agent can name when it decides they’re wanted. The agent reads the duty, causes about what to do, calls a device, reads the end result, and decides what to do subsequent. That loop handles variation {that a} mounted script can not.
That is the bridge from “browser automation script” to “AI agent.”
How you can run: Save as agent_tools.py, guarantee OPENAI_API_KEY is in your .env, then run python agent_tools.py
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 |
# agent_tools.py # LangGraph agent with three browser instruments: navigate_and_extract, fill_and_submit_form, take_screenshot # Stipulations: pip set up playwright langchain langchain-openai langgraph python-dotenv # playwright set up chromium # How you can run: python agent_tools.py
import asyncio import os from dotenv import load_dotenv from langchain_openai import ChatOpenAI from langchain.instruments import device from langchain_core.messages import HumanMessage from langgraph.prebuilt import create_react_agent from playwright.async_api import async_playwright
load_dotenv()
# ── SHARED BROWSER STATE ────────────────────────────────────────────────────── # We hold a single browser occasion alive for the agent’s lifetime. # Creating and destroying a browser on each device name is gradual and wasteful. _browser = None _page = None _playwright = None
async def get_page(): “”“Return the shared web page, launching the browser if wanted.”“” international _browser, _page, _playwright if _browser is None: _playwright = await async_playwright().begin() _browser = await _playwright.chromium.launch(headless=True) context = await _browser.new_context(viewport={“width”: 1280, “top”: 720}) _page = await context.new_page() return _page
async def close_browser(): “”“Clear up browser sources when the agent session ends.”“” international _browser, _page, _playwright if _browser: await _browser.shut() await _playwright.cease() _browser = None _page = None _playwright = None
# ── BROWSER TOOLS ───────────────────────────────────────────────────────────── # Be aware: these are async instruments (async def). LangChain’s @device decorator helps # async features instantly, and the agent should be invoked with ainvoke() in order that # device calls run on the identical occasion loop as an alternative of making an attempt to start out a second one.
@device async def navigate_and_extract(url: str) -> str: “”“ Navigate to a URL and return the seen textual content content material of the web page. Use this to go to web sites and browse their content material. Enter: a full URL string together with https:// (e.g., ‘https://instance.com’). ““” web page = await get_page() await web page.goto(url, wait_until=“domcontentloaded”, timeout=15000) await web page.wait_for_load_state(“networkidle”) content material = await web page.inner_text(“physique”) # Truncate to keep away from flooding the LLM context window return content material[:3000] if len(content material) > 3000 else content material
@device async def fill_and_submit_form(selector_value_pairs: str) -> str: “”“ Fill kind fields and submit a kind on the presently loaded web page. Enter: a comma-separated string of ‘selector:worth’ pairs ending with ‘submit:button_selector’. Instance: ‘#electronic mail:person@instance.com,#password:secret,submit:button[type=submit]’ ““” web page = await get_page() attempt: pairs = selector_value_pairs.break up(“,”) submit_selector = None
for pair in pairs: key, val = pair.break up(“:”, 1) key = key.strip() val = val.strip() if key == “submit”: submit_selector = val else: await web page.fill(key, val)
if submit_selector: await web page.click on(submit_selector) await web page.wait_for_load_state(“networkidle”)
return f“Kind submitted. Present URL: {web page.url}” besides Exception as e: return f“Kind interplay failed: {str(e)}”
@device async def take_screenshot(filename: str) -> str: “”“ Take a screenshot of the present browser web page and reserve it to a file. Use this to visually confirm the present state of the web page. Enter: filename string (e.g., ‘end result.png’). ““” web page = await get_page() await web page.screenshot(path=filename, full_page=False) return f“Screenshot saved to {filename}”
# ── AGENT SETUP ───────────────────────────────────────────────────────────────
llm = ChatOpenAI( mannequin=“gpt-4o”, temperature=0, api_key=os.getenv(“OPENAI_API_KEY”) )
instruments = [navigate_and_extract, fill_and_submit_form, take_screenshot]
# create_react_agent wires collectively the LLM, the instruments, and the ReAct reasoning loop. # The agent decides which device to name, calls it, reads the end result, and continues. agent = create_react_agent(llm, instruments)
# ── DEMO ──────────────────────────────────────────────────────────────────────
async def most important(): end result = await agent.ainvoke({ “messages”: [HumanMessage( content=( “Go to https://example.com, read the page content, “ “then take a screenshot called example.png” ) )] }) print(end result[“messages”][–1].content material) await close_browser()
asyncio.run(most important()) |
What this does: The three @device-decorated features are registered with the agent. Every docstring is what the LLM reads to know what the device does and when to make use of it. Write them like job descriptions, not code feedback. The shared _browser and _page globals imply the browser stays open throughout a number of device calls, which is crucial for duties that span a number of pages in the identical session. As a result of the instruments are outlined with async def, the agent is invoked with ainvoke() moderately than invoke(), so the device calls run on the identical occasion loop that most important() is already utilizing.
A vertical circulate diagram displaying how a process request flows via the agent (click on to enlarge)
Picture by Editor
The important thing design resolution on this snippet is the shared browser occasion. If every device name launched and closed its personal browser, you’d lose all session state between calls, comparable to cookies, navigation historical past, and any kind state the agent had already constructed up. Preserving the browser alive for the complete agent session preserves that context.
Utilizing browser-use for Excessive-Degree Agent Duties
Uncooked Playwright with @device features provides you exact management. The trade-off is that you’re nonetheless writing selectors, nonetheless interested by web page construction, nonetheless dealing with each edge case manually. If the location adjustments its HTML, your selectors break.
browser-use takes a unique strategy. As an alternative of writing selectors, you give the agent a process in plain English. browser-use makes use of Playwright beneath the hood, however the LLM reads the present web page state on every step and decides what to do subsequent: which aspect to click on, what to sort, and when the duty is full. The web page construction will not be hardcoded into your code. The agent figures it out at runtime.
browser-use is a Python library that provides an LLM a working browser. The LLM reads every web page and decides what to click on, sort, and extract. This makes it resilient to web site adjustments that may break a selector-based script.
When to make use of browser-use over uncooked Playwright:
- If the duty is exploratory and the web page construction is unpredictable, use browser-use.
- In case you are operating a set, repeatable workflow the place each selector is understood and secure, uncooked Playwright is extra dependable and cheaper per run.
- A browser-use agent makes a number of LLM calls per process step; a scripted Playwright run makes none.
How you can run: Save as browser_use_agent.py, guarantee OPENAI_API_KEY is in your .env, then run python browser_use_agent.py
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 |
# browser_use_agent.py # A browser-use agent that accepts a pure language process and completes it # with none CSS selectors or hardcoded web page construction. # Stipulations: pip set up browser-use playwright python-dotenv # playwright set up chromium # How you can run: python browser_use_agent.py
import asyncio import os from dotenv import load_dotenv from langchain_openai import ChatOpenAI from browser_use import Agent
load_dotenv()
async def run_browser_task(process: str) -> str: “”“ Hand a pure language process to a browser-use agent. The agent handles navigation, clicks, and extraction with out selectors. ““” # temperature=0 retains selections deterministic and reduces hallucinated actions llm = ChatOpenAI( mannequin=“gpt-4o”, temperature=0, api_key=os.getenv(“OPENAI_API_KEY”) )
# Agent wraps the browser, the LLM, and the duty loop collectively. # max_actions_per_step limits what number of actions the agent takes earlier than # re-reading the web page — prevents runaway loops on advanced pages. agent = Agent( process=process, llm=llm, max_actions_per_step=5 )
# run() executes the complete process loop: # learn web page → resolve motion → take motion → learn up to date web page → repeat end result = await agent.run()
# final_result() returns the agent’s extracted content material or conclusion return end result.final_result() or “Activity accomplished with no extracted output.”
async def most important(): process = ( “Go to https://books.toscrape.com and discover the three costliest books “ “on the primary web page. Return their titles and costs.” ) print(f“Activity: {process}n”) output = await run_browser_task(process) print(f“End result:n{output}”)
asyncio.run(most important()) |
What this does: The whole process, navigating to the location, studying the web page, figuring out the three highest costs, and extracting them, is dealt with by the agent and not using a single CSS selector in your code. If books.toscrape.com redesigns its worth show tomorrow, the script nonetheless works. With a selector-based scraper, it could break silently.
The max_actions_per_step=5 parameter is value explaining. On every step, the agent reads the web page and may resolve to take as much as 5 actions (click on, sort, scroll, navigate) earlier than re-reading the web page. Preserving this low forces the agent to verify its work extra often, which catches errors earlier.
Dealing with the Onerous Components
Three issues break most browser brokers in manufacturing. Every has an answer, however none of them is clear till you have got already been burned.
1. Anti-Bot Detection
Web sites that don’t need to be automated detect automation in a number of methods, comparable to checking the navigator.webdriver property (which Playwright units to true by default), searching for headless browser fingerprints within the JavaScript surroundings, and analyzing interplay patterns which can be too quick or too uniform to be human.
An important mitigation is eradicating the webdriver flag. Past that, a practical person agent string, an ordinary viewport measurement, and a practical locale and timezone cowl most detection strategies wanting refined fingerprint evaluation.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 |
# hard_parts.py — Half 1: Anti-bot stealth launch # Stipulations: pip set up playwright && playwright set up chromium # How you can run: python hard_parts.py
import asyncio import json from pathlib import Path from playwright.async_api import async_playwright
async def launch_stealth_browser(playwright): “”“ Launch a browser context that appears extra like an actual human session. Covers: sensible viewport, user-agent, locale, timezone, webdriver flag. Be aware: For critical anti-bot targets, take into account a paid service like Browserbase. ““” browser = await playwright.chromium.launch( headless=True, args=[ “–disable-blink-features=AutomationControlled”, # Hides webdriver detection “–no-sandbox”, “–disable-dev-shm-usage”, ] )
context = await browser.new_context( viewport={“width”: 1366, “top”: 768}, # Widespread desktop decision user_agent=( “Mozilla/5.0 (Home windows NT 10.0; Win64; x64) “ “AppleWebKit/537.36 (KHTML, like Gecko) “ “Chrome/124.0.0.0 Safari/537.36” ), locale=“en-US”, timezone_id=“America/New_York”, java_script_enabled=True, )
# Take away the ‘webdriver’ property that Playwright injects by default. # Bot detection techniques verify for this within the browser’s JS surroundings. await context.add_init_script( “Object.defineProperty(navigator, ‘webdriver’, {get: () => undefined})” )
return browser, context |
What this does: The add_init_script() name runs earlier than any web page JavaScript executes, which implies the navigator.webdriver override is in place earlier than the location’s detection code can verify for it. The –disable-blink-features=AutomationControlled launch argument removes a separate automation flag on the browser engine stage. Collectively, these two adjustments deal with the commonest detection strategies.
For websites with aggressive fingerprinting and CAPTCHA techniques, these mitigations is not going to be sufficient. Companies like Browserbase, Spidra and Brightdata’s Scraping Browser deal with CAPTCHA fixing, residential IP rotation, and browser fingerprint administration as managed infrastructure.
2. Good Ready
The second failure mode is timing. The reflex is so as to add time.sleep() calls and enhance them when issues break. That is flawed in each instructions: too brief on gradual connections, too lengthy on quick ones, and utterly opaque when debugging.
Playwright has 4 correct wait methods. Use the one which matches what you’re truly ready for:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 |
# Half 2: Good ready methods (add to your scraper or agent instruments)
async def smart_wait_examples(web page): “”“ 4 methods to attend for the precise web page state, with out arbitrary sleeps. ““” # STRATEGY 1: Anticipate a selected aspect to seem within the DOM # Use when you understand precisely what aspect indicators content material has loaded await web page.wait_for_selector(“.product-list”, state=“seen”, timeout=10000)
# STRATEGY 2: Anticipate a selected API response # Use when the content material comes from an XHR/fetch name you may establish async with web page.expect_response( lambda r: “/api/merchandise” in r.url and r.standing == 200 ) as response_info: await web page.click on(“#load-more”) response = await response_info.worth print(f“API responded: {response.standing}”)
# STRATEGY 3: Anticipate the URL to vary after kind submission # Use when a profitable submit redirects to a brand new web page await web page.wait_for_url(“**/dashboard**”, timeout=10000)
# STRATEGY 4: Anticipate a JavaScript variable to be set # Use when no visible aspect reliably indicators the prepared state await web page.wait_for_function( “() => window.__dataLoaded === true”, timeout=10000 ) |
What this does: Every technique is tied to a selected observable occasion moderately than an arbitrary time delay. wait_for_selector watches the DOM. expect_response hooks into the community layer. wait_for_url displays navigation. wait_for_function evaluates JavaScript within the browser context. Use whichever one most instantly indicators “the factor I want is now prepared.”
3. Session and Cookie Persistence
The third failure mode is shedding session state. In case your agent logs right into a web site throughout the 1st step after which the browser context is destroyed, step two has no authentication. Recreating the login on each run is gradual and may set off charge limiting or lockout.
The answer is saving cookies to disk after login and loading them at first of each subsequent run:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
# Half 3: Session persistence throughout runs
COOKIES_FILE = Path(“session_cookies.json”)
async def save_session(context) -> None: “”“Save browser cookies to disk after a profitable login.”“” cookies = await context.cookies() COOKIES_FILE.write_text(json.dumps(cookies, indent=2)) print(f“Session saved: {len(cookies)} cookies written.”)
async def load_session(context) -> bool: “”“Load saved cookies earlier than navigating. Returns True if session was discovered.”“” if not COOKIES_FILE.exists(): print(“No saved session. Recent login required.”) return False cookies = json.masses(COOKIES_FILE.read_text()) await context.add_cookies(cookies) print(f“Session restored: {len(cookies)} cookies loaded.”) return True |
What this does: context.cookies() returns all cookies for the present browser context, together with session tokens and authentication cookies. Writing them to JSON and reloading them on the following run means the browser begins in an authenticated state. Be aware that periods expire; add a verify that falls again to a recent login if the saved session returns a redirect to the login web page.
Deploying Browser Brokers
Getting a browser agent working regionally is one factor. Operating it reliably in a cloud surroundings is one other.
The principle distinction between a Python script that works in your laptop computer and one which fails in CI is system dependencies. Playwright’s Chromium browser requires a set of shared libraries which can be current on most developer machines however absent from minimal cloud pictures. The cleanest resolution is Docker.
Dockerfile — construct a container that ships every little thing Playwright wants:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 |
# Dockerfile for headless Playwright-based browser agent # Construct: docker construct -t browser-agent . # Run: docker run –rm -e OPENAI_API_KEY=your_key browser-agent
FROM python:3.11–slim
# Set up system dependencies required by Chromium RUN apt–get replace && apt–get set up –y libnss3 libatk1.0–0 libatk–bridge2.0–0 libcups2 libdrm2 libxkbcommon0 libxcomposite1 libxdamage1 libxrandr2 libgbm1 libasound2 libpangocairo–1.0–0 libpango–1.0–0 libcairo2 libx11–6 libxext6 libxfixes3 fonts–liberation wget ca–certificates && rm –rf /var/lib/apt/lists/*
WORKDIR /app
# Set up Python dependencies first (cached layer — solely rebuilds on necessities change) COPY necessities.txt . RUN pip set up —no–cache–dir –r necessities.txt
# Set up Playwright browser binaries into the picture RUN playwright set up chromium RUN playwright set up–deps chromium
# Copy software code final (adjustments right here do not invalidate the pip/playwright layers) COPY . .
CMD [“python”, “agent_tools.py”]
necessities.txt: playwright browser–use langchain langchain–openai langgraph python–dotenv |
For concurrent workloads operating a number of browser periods in parallel, use Playwright’s async API with asyncio.collect():
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 |
# Parallel scraping with semaphore charge limiting # Runs as much as 3 browser periods concurrently
import asyncio from playwright.async_api import async_playwright
async def scrape_url(browser, url: str, semaphore: asyncio.Semaphore) -> dict: “”“Scrape a single URL, respecting the concurrency semaphore.”“” async with semaphore: context = await browser.new_context() web page = await context.new_page() await web page.goto(url, wait_until=“domcontentloaded”) title = await web page.title() await context.shut() # Shut context (not browser) to launch sources return {“url”: url, “title”: title}
async def scrape_parallel(urls: record[str], max_concurrent: int = 3) -> record[dict]: “”“Scrape an inventory of URLs in parallel, capped at max_concurrent periods.”“” semaphore = asyncio.Semaphore(max_concurrent) # Cap concurrent periods
async with async_playwright() as p: # One browser shared throughout all contexts — less expensive than one browser per URL browser = await p.chromium.launch(headless=True) duties = [scrape_url(browser, url, semaphore) for url in urls] outcomes = await asyncio.collect(*duties) await browser.shut()
return record(outcomes) |
What this does: The asyncio.Semaphore(max_concurrent) caps what number of browser contexts run on the identical time. With out it, launching 50 concurrent browser contexts will exhaust reminiscence. One browser course of is shared throughout all contexts; a context is reasonable; a full browser occasion will not be.
On the managed infrastructure facet, Amazon Nova Act launched in March 2025 as a devoted SDK for constructing browser brokers on AWS, integrating natively with Playwright for browser management. Playwright’s own MCP server provides AI assistants full browser management via the Mannequin Context Protocol, utilizing structured accessibility snapshots moderately than screenshots, which implies token prices keep low whereas the agent’s understanding of the web page stays excessive.
Placing It All Collectively
Here’s a full end-to-end agent that takes a analysis query, navigates to a public knowledge supply, extracts structured outcomes, and returns a clear abstract. It makes use of the browser instruments from Part 5 orchestrated by a LangGraph agent.
How you can run: Save as reference_agent.py, guarantee OPENAI_API_KEY is in your .env, and run python reference_agent.py
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 |
# reference_agent.py # Full browser-using AI agent: navigates, extracts, summarizes. # Goal: books.toscrape.com (public scraping sandbox) # Stipulations: pip set up playwright langchain langchain-openai langgraph python-dotenv # playwright set up chromium # How you can run: python reference_agent.py
import asyncio import os from dotenv import load_dotenv from langchain_openai import ChatOpenAI from langchain.instruments import device from langchain_core.messages import HumanMessage, SystemMessage from langgraph.prebuilt import create_react_agent from playwright.async_api import async_playwright
load_dotenv()
# ── BROWSER STATE ───────────────────────────────────────────────────────────── _browser = None _context = None _page = None _playwright = None
async def get_page(): international _browser, _context, _page, _playwright if _browser is None: _playwright = await async_playwright().begin() _browser = await _playwright.chromium.launch(headless=True) _context = await _browser.new_context( viewport={“width”: 1280, “top”: 720}, user_agent=( “Mozilla/5.0 (Home windows NT 10.0; Win64; x64) “ “AppleWebKit/537.36 (KHTML, like Gecko) “ “Chrome/120.0.0.0 Safari/537.36” ) ) # Take away webdriver fingerprint await _context.add_init_script( “Object.defineProperty(navigator, ‘webdriver’, {get: () => undefined})” ) _page = await _context.new_page() return _page
async def teardown(): international _browser, _playwright if _browser: await _browser.shut() await _playwright.cease() _browser = None _playwright = None
# ── TOOLS ─────────────────────────────────────────────────────────────────────
@device async def navigate(url: str) -> str: “”“ Navigate the browser to a URL and return the web page’s textual content content material. Use when you might want to open a web site or transfer to a brand new web page. Enter: full URL with https:// prefix. ““” web page = await get_page() await web page.goto(url, wait_until=“domcontentloaded”, timeout=20000) await web page.wait_for_load_state(“networkidle”) content material = await web page.inner_text(“physique”) return content material[:4000]
@device async def extract_structured(css_selector: str) -> str: “”“ Extract textual content from all parts matching a CSS selector on the present web page. Use when you might want to pull particular parts from the loaded web page. Enter: legitimate CSS selector string (e.g., ‘h3 a’, ‘.price_color’, ‘article.product_pod’). ““” web page = await get_page() attempt: await web page.wait_for_selector(css_selector, timeout=5000) parts = await web page.query_selector_all(css_selector) texts = [] for el in parts[:20]: # Cap at 20 parts to maintain output manageable textual content = await el.inner_text() texts.append(textual content.strip()) return “n”.be part of(texts) if texts else “No parts discovered.” besides Exception as e: return f“Extraction failed: {str(e)}”
@device async def get_current_url() -> str: “”“Return the URL the browser is presently on. No enter required.”“” web page = await get_page() return web page.url
# ── AGENT ─────────────────────────────────────────────────────────────────────
llm = ChatOpenAI( mannequin=“gpt-4o”, temperature=0, api_key=os.getenv(“OPENAI_API_KEY”) )
instruments = [navigate, extract_structured, get_current_url] agent = create_react_agent(llm, instruments)
SYSTEM = ( “You’re a browser-based analysis agent. You’ve gotten entry to an actual browser. “ “Use navigate() to open pages, extract_structured() to drag particular parts, “ “and get_current_url() to verify the place you’re. “ “All the time navigate first, then extract. Be concise in your remaining reply.” )
async def run_agent(question: str) -> str: end result = await agent.ainvoke({ “messages”: [ SystemMessage(content=SYSTEM), HumanMessage(content=query) ] }) await teardown() return end result[“messages”][–1].content material
# ── DEMO ──────────────────────────────────────────────────────────────────────
if __name__ == “__main__”: question = ( “Go to https://books.toscrape.com and extract the titles and costs “ “of the primary 5 books listed. Return them as a structured record.” ) print(f“Question: {question}n”) reply = asyncio.run(run_agent(question)) print(f“Reply:n{reply}”) |
What this does: This agent has three clear instruments: navigate, extract_structured, and get_current_url, plus a system immediate that tells it precisely when to make use of every one. The agent calls navigate to load the web page, extract_structured to drag the guide titles and costs by CSS selector, and synthesizes a structured record within the remaining reply. The teardown() name after the agent finishes closes the browser cleanly so no zombie Chromium processes are left operating.
Conclusion
The browser will not be a specialised device for automation engineers. It’s the common interface for the online, and the online is the place many of the world’s precise work will get performed. An AI agent that may use a browser doesn’t want a companion group sustaining API integrations. It will probably attain something a human can attain.
What makes this sensible now, not simply theoretically fascinating, is the maturity of the tooling. Playwright handles the arduous elements of browser interplay. browser-use removes the necessity to write selectors for exploratory duties. LangGraph provides the LLM clear device hooks and a reasoning loop that handles variable web page buildings. The patterns on this article will not be demos. They’re the identical patterns 51% of enterprises now operating AI brokers in manufacturing are constructing on.
Begin with the scraping instance. Get it operating in opposition to a web site you really want knowledge from. Add the agent layer once you want selections the script can not anticipate. Add browser-use when the web page construction is simply too dynamic for selectors. Deploy in Docker once you want it operating someplace apart from your laptop computer.
The arduous half will not be the code. It’s understanding which device to succeed in for at every layer. Hopefully this text made that clearer.
