Embed a stay AI browser agent in your React app with Amazon Bedrock AgentCore

Once you construct AI-powered purposes, your customers should perceive and belief AI brokers that navigate web sites and work together with net content material on their behalf. When an agent interacts with net content material autonomously, your customers require visibility into these actions to take care of confidence and management, which they don’t presently have.

The Amazon Bedrock AgentCore Browser BrowserLiveView element addresses this problem by offering a real-time video feed of the agent’s shopping session immediately inside your React software. This element, a part of the Bedrock AgentCore TypeScript SDK, streamlines the combination by embedding a stay browser stream with three strains of JavaScript XML (JSX).

The BrowserLiveView element makes use of the Amazon DCV protocol to render the browser session, creating transparency into agent actions. Implementation requires solely a presigned URL out of your server, with out requiring you to construct streaming infrastructure.

This submit walks you thru three steps: beginning a session and producing the Dwell View URL, rendering the stream in your React software, and wiring up an AI agent that drives the browser whereas your customers watch. On the finish, you’ll have a working pattern software you may clone and run.

Why embed Dwell View in your software

Embedding Dwell View inside your individual software unlocks further worth on your customers at scale.

With an embedded Dwell View, your customers observe each navigation, type submission, and search question because the agent performs it. They get instant visible affirmation that the agent is on the precise web page, interacting with the proper components, and progressing by means of the workflow. This real-time suggestions loop offers finish customers direct perception into agent habits with out ready for the ultimate end result.

Customers who delegate shopping duties to an AI agent are extra assured after they can observe the work. Watching the agent fill in a type subject by subject is extra reassuring than receiving a textual content affirmation. For regulated workflows, visible proof of agent actions can help audit necessities.

In workflows that require human supervision, like dealing with buyer accounts and processing delicate knowledge, a supervisor can use the embedded Dwell View to observe the agent in actual time and intervene if wanted, with out leaving your software.

Organizations additionally acquire audit path help by means of visible proof of agent actions, which proves beneficial for compliance necessities and troubleshooting eventualities. Mixed with session recordings to Amazon Easy Storage Service (Amazon S3) and console-based session replay, you get each real-time commentary and post-hoc evaluation.

The way it works

The combination has three parts.

The consumer’s net browser runs a React software containing the BrowserLiveView element, which receives a SigV4-presigned URL and establishes a persistent WebSocket connection to obtain the DCV video stream from a distant browser session. The React software handles video rendering and consumer interface presentation whereas sustaining the WebSocket connection for steady streaming.

The appliance server capabilities as an AI agent throughout the Amazon Bedrock session lifecycle, orchestrating the connection between consumer browsers and cloud-hosted browser classes. It begins classes utilizing the Amazon Bedrock AgentCore API and generates SigV4-presigned URLs that grant safe, time-limited entry to the Dwell View stream. This layer handles session administration, authentication, and stream distribution.

AWS Cloud hosts Amazon Bedrock AgentCore Browser and Amazon Bedrock providers that present the underlying browser automation and streaming capabilities. Amazon Bedrock AgentCore hosts the remoted cloud browser classes inside AWS Cloud and supplies each the automation endpoint (Playwright CDP) and the Dwell View streaming endpoint (DCV).

The important thing effectivity benefit with this structure is that the DCV Dwell View stream flows immediately from Amazon Bedrock AgentCore to the consumer’s browser. It doesn’t cross by means of your software server. Your server generates the URL and runs the agent, however the video stream is a direct WebSocket connection from AWS to the consumer. This helps decrease latency and scale back infrastructure necessities.

Determine 1: Answer structure exhibiting the info movement between three parts. The numbered arrows within the diagram symbolize the next knowledge flows:

Arrow 1 (grey, strong): The consumer sends prompts and polls standing from the Utility Server utilizing REST.

Arrow 2 (orange, strong): The Utility Server calls the Amazon Bedrock Converse API for AI mannequin reasoning.

Arrow 3 (blue, strong): The Utility Server runs browser instruments in opposition to Amazon Bedrock AgentCore Browser utilizing Playwright Chrome DevTools Protocol (CDP).

Arrow 4 (pink, dashed): The DCV Dwell View stream flows immediately from Amazon Bedrock AgentCore to the Person Browser, bypassing the Utility Server.

Stipulations

Earlier than you start, confirm that you’ve got the next:

Essential: Dwell View (Steps 1 and a pair of) requires solely Amazon Bedrock AgentCore permissions. It doesn’t rely on Amazon Bedrock or any particular AI mannequin. The AI agent in Step 3 makes use of the Amazon Bedrock Converse API, which requires further Amazon Bedrock permissions, however that is particular to our pattern. You’ll be able to substitute a mannequin supplier or agent framework of your selection. Use momentary credentials from AWS IAM Id Middle or AWS Safety Token Service (AWS STS). Don’t use long-lived entry keys. Observe the precept of least privilege when configuring AWS Id and Entry Administration (IAM) permissions.

Set up the Amazon Bedrock AgentCore TypeScript SDK:

npm set up bedrock-agentcore

For the AI agent in Step 3, you additionally want the AWS SDK for JavaScript:

npm set up @aws-sdk/client-bedrock-runtime

The code on this submit runs in two environments: the server-side code (Steps 1 and three) runs in Node.js, and the client-side code (Step 2) runs in a React software bundled with Vite. The pattern software on the finish of this submit packages the whole lot collectively.

Step-by-step implementation

1: Begin a browser session and generate the Dwell View URL

In your software server, use the Browser class to start out a session and generate the presigned URL. The API returns a session identifier and streaming URL, which the server converts right into a presigned URL with an outlined expiration time of 300 seconds by default. It comprises SigV4 credentials within the question parameters, so no secrets and techniques attain the browser. Move this URL to your frontend by means of an API endpoint.

import { Browser } from 'bedrock-agentcore/browser'

const browser = new Browser({
  area: 'us-west-2'
})
await browser.startSession({
  viewport: { width: 1920, top: 1080 }
})

const signedUrl =
  await browser.generateLiveViewUrl()
// Ship signedUrl to your frontend by way of API

2: Render the BrowserLiveView element in your React app

In your browser, import the BrowserLiveView element from the Bedrock AgentCore TypeScript SDK and render it with the presigned URL. The element handles WebSocket connection, DCV protocol negotiation, video stream decoding, and body rendering. It auto scales to suit its dad or mum container whereas preserving its facet ratio. The remoteWidth and remoteHeight should match the viewport that you simply set in Step 1. Mismatched values trigger cropping or black bars.

import { BrowserLiveView }
  from 'bedrock-agentcore/browser/live-view'

<BrowserLiveView
  signedUrl={presignedUrl}
  remoteWidth={1920}
  remoteHeight={1080}
/>

After including this element, the Dwell View begins streaming as quickly because the presigned URL is legitimate and the browser session is lively. It’s best to see the distant browser’s desktop seem throughout the element’s container. If the container stays empty, confirm that the presigned URL hasn’t expired and that the browser session remains to be working.

3: Join an AI agent to drive browser actions

With the Dwell View streaming, you want one thing attention-grabbing to observe. The next instance makes use of the Amazon Bedrock Converse API, however Dwell View is mannequin agnostic. You need to use an AI mannequin or agent framework of your option to drive the browser.

The code creates a PlaywrightBrowser consumer, which begins a brand new AgentCore Browser session and connects to it utilizing the Playwright Chrome DevTools protocol. This is similar sort of cloud browser session as Step 1 however accessed by means of the Playwright automation interface moderately than the Dwell View interface.

The mannequin decides which browser instruments to name, together with navigate, click on, sort, getText, getHtml, and pressKey. Your server runs these instruments and feeds the outcomes again to the mannequin for the following iteration.

import { BedrockRuntimeClient, ConverseCommand }
  from '@aws-sdk/client-bedrock-runtime'
import { PlaywrightBrowser }
  from 'bedrock-agentcore/browser/playwright'

const browser = new PlaywrightBrowser({
  area: 'us-west-2'
})
await browser.startSession()

// Outline browser instruments as JSON Schema
// (navigate, click on, sort, getText, and extra)

whereas (step < maxSteps) {
  const response = await bedrockClient.ship(
    new ConverseCommand({
      modelId: modelId,
      system: [{ text: systemPrompt }],
      messages,
      toolConfig: browserTools,
    })
  )

  if (response.stopReason === 'tool_use') {
    // Run browser device, add end result
    // to dialog, proceed loop
  } else {
    break // Ultimate reply from mannequin
  }
}

The mannequin is configurable. You need to use Anthropic Claude, Amazon Nova, or an Amazon Bedrock mannequin that helps device use. Each device name that the mannequin makes is seen to your consumer by means of the Dwell View. They see the browser navigate, the search field fill in, and the outcomes web page load.

Observe: The TypeScript SDK additionally features a Vercel AI SDK integration (BrowserTools) that wraps these browser operations as framework-native instruments.

Attempt it utilizing the pattern software

We constructed an entire pattern software on GitHub that places Steps 1–3 collectively. The pattern features a React dashboard with the embedded Dwell View, an exercise log exhibiting agent reasoning and actions, and a Fastify server working the AI agent. The agent navigates to Wikipedia, searches for a subject, reads the web page content material, and summarizes what it finds when you watch each step. You’ll be able to obtain it from the GitHub repository.

Determine 2: The pattern software mid-run. The left panel reveals the BrowserLiveView element streaming a Wikipedia web page that the agent has navigated to. The best panel reveals the exercise log with timestamped device calls (navigate, getText, click on). On the backside, the immediate enter subject and Launch Agent button are seen.

To clone and run the pattern software

Full the next steps to clone and run the pattern software.

1. Clone the repository and navigate to the pattern folder.

git clone https://github.com/awslabs/bedrock-agentcore-samples-typescript.git
cd bedrock-agentcore-samples-typescript
cd use-cases/browser-live-view-agent

2. Set up the dependencies.

3. Export your AWS credentials.

export AWS_ACCESS_KEY_ID=<your-access-key>
export AWS_SECRET_ACCESS_KEY=<your-secret-key>
export AWS_SESSION_TOKEN=<your-session-token>
export AWS_REGION=us-west-2

Essential: Use momentary credentials. Don’t commit credentials to supply management.

4. Begin the applying.

5. Open http://localhost:5173, enter a immediate, and select Launch Agent.

Bundler configuration

The BrowserLiveView element makes use of the Amazon DCV Internet Shopper SDK, which ships vendored recordsdata contained in the bedrock-agentcore npm package deal. You don’t have to obtain or set up DCV individually. Your Vite configuration wants three additions:

• resolve.alias factors the dcv and dcv-ui naked specifiers to the vendored SDK recordsdata.
• resolve.dedupe verifies that React and shared dependencies resolve out of your node_modules, not from the vendored path.
• viteStaticCopy copies DCV runtime recordsdata (staff, WASM decoders) to your construct output.

The pattern software’s vite.config.ts has the whole configuration prepared to make use of. For extra particulars on the BrowserLiveView element, see the live-view source directory within the TypeScript SDK.

Clear up sources

To keep away from incurring prices, cease the browser session and shut down the applying whenever you’re executed:

1. Within the software UI, select Cease Session to finish the Amazon Bedrock AgentCore Browser session.
2. In your terminal, press Ctrl+C to cease the event servers.
3. If you happen to created any IAM roles or insurance policies particularly for this demo, delete them from the IAM console.

Amazon Bedrock AgentCore Browser classes incur prices whereas lively. For pricing particulars, consult with the Amazon Bedrock AgentCore pricing web page.

Subsequent steps

Now that you’ve got a working Dwell View integration, listed here are some issues to discover.

To get began, clone the sample application, fill in your AWS credentials, and run npm run dev to see the total demo in motion. For directions, consult with the To clone and run the sample application part on this submit.

The pattern software defaults to Anthropic Claude, however you may swap to Amazon Nova or one other Amazon Bedrock mannequin that helps device use by setting the BEDROCK_MODEL_ID atmosphere variable. For an inventory of obtainable fashions and their device use capabilities, consult with the Amazon Bedrock mannequin documentation.

The React dashboard within the pattern software is a place to begin on your personal implementation. You’ll be able to adapt the structure to match your design system, combine the Dwell View into an current software, or add controls that allow customers intervene mid-workflow. For steering on constructing React purposes with the AgentCore SDK, consult with the Bedrock AgentCore TypeScript SDK documentation.

The BrowserLiveView element helps a number of situations on the identical web page, every streaming a special browser session. This functionality is helpful for monitoring dashboards. The element’s supply code, together with scaling logic and DCV authentication movement, is accessible within the live-view source directory within the TypeScript SDK.

Conclusion

On this submit, you realized how you can use the BrowserLiveView element to embed a Dwell View of an Amazon Bedrock AgentCore Browser session into your React software. The three-step implementation and structure that streams video immediately from AWS to consumer browsers makes stay agent visualization accessible with out specialised streaming experience.

For a deeper take a look at Amazon Bedrock AgentCore Browser capabilities, consult with the Amazon Bedrock AgentCore Browser documentation. When you have suggestions or questions, open a difficulty within the GitHub repository.

Essential: This pattern software is meant for native improvement and demonstration. For manufacturing use, add authentication to your API endpoints, allow HTTPS, prohibit CORS origins, implement charge limiting, and observe the AWS Properly-Architected Framework safety pillar.

In regards to the authors