Getting began with laptop use in Amazon Bedrock Brokers

Computer use is a breakthrough functionality from Anthropic that permits basis fashions (FMs) to visually understand and interpret digital interfaces. This functionality permits Anthropic’s Claude fashions to establish what’s on a display, perceive the context of UI components, and acknowledge actions that must be carried out resembling clicking buttons, typing textual content, scrolling, and navigating between purposes. Nonetheless, the mannequin itself doesn’t execute these actions—it requires an orchestration layer to soundly implement the supported actions.

At present, we’re asserting laptop use help inside Amazon Bedrock Brokers utilizing Anthropic’s Claude 3.5 Sonnet V2 and Anthropic’s Claude Sonnet 3.7 fashions on Amazon Bedrock. This integration brings Anthropic’s visible notion capabilities as a managed instrument inside Amazon Bedrock Brokers, offering you with a safe, traceable, and managed approach to implement laptop use automation in your workflows.

Organizations throughout industries wrestle with automating repetitive duties that span a number of purposes and methods of document. Whether or not processing invoices, updating buyer data, or managing human useful resource (HR) paperwork, these workflows typically require workers to manually switch info between totally different methods – a course of that’s time-consuming, error-prone, and troublesome to scale.

Conventional automation approaches require customized API integrations for every utility, creating important improvement overhead. Laptop use capabilities change this paradigm by permitting machines to understand present interfaces simply as people.

On this put up, we create a pc use agent demo that gives the important orchestration layer that transforms laptop use from a notion functionality into actionable automation. With out this orchestration layer, laptop use would solely establish potential actions with out executing them. The pc use agent demo powered by Amazon Bedrock Brokers supplies the next advantages:

• Safe execution surroundings – Execution of laptop use instruments in a sandbox surroundings with restricted entry to the AWS ecosystem and the online. It’s essential to notice that at the moment Amazon Bedrock Agent doesn’t present a sandbox surroundings
• Complete logging – Capacity to trace every motion and interplay for auditing and debugging
• Detailed tracing capabilities – Visibility into every step of the automated workflow
• Simplified testing and experimentation – Lowered threat when working with this experimental functionality by way of managed controls
• Seamless orchestration – Coordination of complicated workflows throughout a number of methods with out customized code

This integration combines Anthropic’s perceptual understanding of digital interfaces with the orchestration capabilities of Amazon Bedrock Brokers, creating a strong agent for automating complicated workflows throughout purposes. Relatively than construct customized integrations for every system, builders can now create brokers that understand and work together with present interfaces in a managed, safe means.

With laptop use, Amazon Bedrock Brokers can automate duties by way of primary GUI actions and built-in Linux instructions. For instance, your agent might take screenshots, create and edit textual content information, and run built-in Linux instructions. Utilizing Amazon Bedrock Brokers and appropriate Anthropic’s Claude fashions, you should utilize the next motion teams:

• Laptop instrument – Permits interactions with consumer interfaces (clicking, typing, scrolling)
• Textual content editor instrument – Supplies capabilities to edit and manipulate information
• Bash – Permits execution of built-in Linux instructions

Resolution overview

An instance laptop use workflow consists of the next steps:

1. Create an Amazon Bedrock agent and use pure language to explain what the agent ought to do and the way it ought to work together with customers, for instance: “You might be laptop use agent able to utilizing Firefox net browser for net search.”
2. Add the Amazon Bedrock Brokers supported laptop use motion teams to your agent utilizing CreateAgentActionGroup API.
3. Invoke the agent with a consumer question that requires laptop use instruments, for instance, “What’s Amazon Bedrock, are you able to search the online?”
4. The Amazon Bedrock agent makes use of the instrument definitions at its disposal and decides to make use of the pc motion group to click on a screenshot of the surroundings. Utilizing the return management functionality of Amazon Bedrock Brokers, the agent the responds with the instrument or instruments that it needs to execute. The return management functionality is required for utilizing laptop use with Amazon Bedrock Brokers.
5. The workflow parses the agent response and executes the instrument returned in a sandbox surroundings. The output is given again to the Amazon Bedrock agent for additional processing.
6. The Amazon Bedrock agent continues to reply with instruments at its disposal till the duty is full.

You possibly can recreate this instance within the us-west-2 AWS Area with the AWS Cloud Improvement Equipment (AWS CDK) by following the directions within the GitHub repository. This demo deploys a containerized utility utilizing AWS Fargate throughout two Availability Zones within the us-west-2 Area. The infrastructure operates inside a digital non-public cloud (VPC) containing public subnets in every Availability Zone, with an web gateway offering exterior connectivity. The structure is complemented by important supporting companies, together with AWS Key Administration Service (AWS KMS) for safety and Amazon CloudWatch for monitoring, making a resilient, serverless container surroundings that alleviates the necessity to handle underlying infrastructure whereas sustaining strong safety and excessive availability.

The next diagram illustrates the answer structure.

On the core of our answer are two Fargate containers managed by way of Amazon Elastic Container Service (Amazon ECS), every protected by its personal safety group. The primary is our orchestration container, which not solely handles the communication between Amazon Bedrock Brokers and finish customers, but in addition orchestrates the workflow that permits instrument execution. The second is the environment container, which serves as a safe sandbox the place the Amazon Bedrock agent can safely run its laptop use instruments. The surroundings container has restricted entry to the remainder of the ecosystem and the web. We make the most of service discovery to attach Amazon ECS companies with DNS names.

The orchestration container contains the next parts:

• Streamlit UI – The Streamlit UI that facilitates interplay between the top consumer and laptop use agent
• Return management loop – The workflow answerable for parsing the instruments that the agent needs to execute and returning the output of those instruments

The surroundings container contains the next parts:

• UI and pre-installed purposes – A light-weight UI and pre-installed Linux purposes like Firefox that can be utilized to finish the consumer’s duties
• Software implementation – Code that may execute laptop use instrument within the surroundings like “screenshot” or “double-click”
• Quart (RESTful) JSON API – An orchestration container that makes use of Quart to execute instruments in a sandbox surroundings

The next diagram illustrates these parts.

Conditions

1. AWS Command Line Interface (CLI), comply with directions right here. Ensure that to setup credentials, comply with directions right here.
2. Require Python 3.11 or later.
3. Require Node.js 14.15.0 or later.
4. AWS CDK CLI, comply with directions right here.
5. Allow mannequin entry for Anthropic’s Claude Sonnet 3.5 V2 and for Anthropic’s Claude Sonnet 3.7.
6. Boto3 model >= 1.37.10.

Create an Amazon Bedrock agent with laptop use

You should utilize the next code pattern to create a easy Amazon Bedrock agent with laptop, bash, and textual content editor motion teams. It’s essential to offer a appropriate motion group signature when utilizing Anthropic’s Claude 3.5 Sonnet V2 and Anthropic’s Claude 3.7 Sonnet as highlighted here.

import boto3
import time

# Step 1: Create the bedrock agent shopper

bedrock_agent = boto3.shopper("bedrock-agent", region_name="us-west-2")

# Step 2: Create an agent

create_agent_response = create_agent_response = bedrock_agent.create_agent(
        agentResourceRoleArn=agent_role_arn, # Amazon Bedrock Agent execution position
        agentName="computeruse",
        description="""Instance agent for laptop use. 
				This agent ought to solely function on 
				Sandbox environments with restricted privileges.""",
        foundationModel="us.anthropic.claude-3-7-sonnet-20250219-v1:0",      
		instruction="""You might be laptop use agent able to utilizing Firefox 
                 net browser for net search.""",
)

time.sleep(30) # await agent to be created

# Step 3.1: Create and connect laptop motion group

bedrock_agent.create_agent_action_group(
    actionGroupName="ComputerActionGroup",
    actionGroupState="ENABLED",
    agentId=create_agent_response["agent"]["agentId"],
    agentVersion="DRAFT",
    parentActionGroupSignature="ANTHROPIC.Laptop",
    parentActionGroupSignatureParams={
        "sort": "computer_20250124",
        "display_height_px": "768",
        "display_width_px": "1024",
        "display_number": "1",
    },
)

# Step 3.2: Create and connect bash motion group

bedrock_agent.create_agent_action_group(
    actionGroupName="BashActionGroup",
    actionGroupState="ENABLED",
    agentId=create_agent_response["agent"]["agentId"],
    agentVersion="DRAFT",
    parentActionGroupSignature="ANTHROPIC.Bash",
    parentActionGroupSignatureParams={
        "sort": "bash_20250124",
    },
)

# Step 3.3: Create and connect textual content editor motion group

bedrock_agent.create_agent_action_group(
    actionGroupName="TextEditorActionGroup",
    actionGroupState="ENABLED",
    agentId=create_agent_response["agent"]["agentId"],
    agentVersion="DRAFT",
    parentActionGroupSignature="ANTHROPIC.TextEditor",
    parentActionGroupSignatureParams={
        "sort": "text_editor_20250124",
    },
)

# Step 3.4 Create Climate Motion Group

bedrock_agent.create_agent_action_group(
        actionGroupName="WeatherActionGroup",
        agentId=create_agent_response["agent"]["agentId"],
        agentVersion="DRAFT",
        actionGroupExecutor = {
            'customControl': 'RETURN_CONTROL',
        },
        functionSchema = {
            'capabilities': [
                {
                    "name": "get_current_weather",
                    "description": "Get the current weather in a given location.",
                    "parameters": {
                        "location": {
                            "type": "string",
                            "description": "The city, e.g., San Francisco",
                            "required": True,
                        },
                        "unit": {
                            "type": "string",
                            "description": 'The unit to use, e.g., 
									fahrenheit or celsius. Defaults to "fahrenheit"',
                            "required": False,
                        },
                    },
                    "requireConfirmation": "DISABLED",
                }
            ]
        },
)
time.sleep(10)
# Step 4: Put together agent

bedrock_agent.prepare_agent(agentId=create_agent_response["agent"]["agentId"])

Instance use case

On this put up, we display an instance the place we use Amazon Bedrock Brokers with the pc use functionality to finish an online type. Within the instance, the pc use agent may swap Firefox tabs to work together with a customer relationship management (CRM) agent to get the required info to finish the shape. Though this instance makes use of a pattern CRM utility because the system of document, the identical method works with Salesforce, SAP, Workday, or different methods of document with the suitable authentication frameworks in place.

Within the demonstrated use case, you may observe how effectively the Amazon Bedrock agent carried out with laptop use instruments. Our implementation accomplished the client ID, buyer title, and e-mail by visually analyzing the excel information. Nonetheless, for the overview, it determined to pick the cell and duplicate the info, as a result of the data wasn’t utterly seen on the display. Lastly, the CRM agent was used to get extra info on the client.

Finest practices

The next are some methods you may enhance the efficiency to your use case:

Concerns

The pc use characteristic is made obtainable to you as a beta service as outlined within the AWS Service Phrases. It’s topic to your settlement with AWS and the AWS Service Phrases, and the relevant mannequin EULA. Laptop use poses distinctive dangers which might be distinct from normal API options or chat interfaces. These dangers are heightened when utilizing the pc use characteristic to work together with the web. To reduce dangers, think about taking precautions resembling:

• Function laptop use performance in a devoted digital machine or container with minimal privileges to attenuate direct system exploits or accidents
• To assist forestall info theft, keep away from giving the pc use API entry to delicate accounts or information
• Restrict the pc use API’s web entry to required domains to scale back publicity to malicious content material
• To implement correct oversight, preserve a human within the loop for delicate duties (resembling making selections that would have significant real-world penalties) and for something requiring affirmative consent (resembling accepting cookies, executing monetary transactions, or agreeing to phrases of service)

Any content material that you simply allow Anthropic’s Claude to see or entry can probably override directions or trigger the mannequin to make errors or carry out unintended actions. Taking correct precautions, resembling isolating Anthropic’s Claude from delicate surfaces, is important – together with to keep away from dangers associated to immediate injection. Earlier than enabling or requesting permissions essential to allow laptop use options in your personal merchandise, inform finish customers of any related dangers, and acquire their consent as acceptable.

Clear up

When you find yourself completed utilizing this answer, be sure that to wash up all of the assets. Comply with the directions within the supplied GitHub repository.

Conclusion

Organizations throughout industries face important challenges with cross-application workflows that historically require guide information entry or complicated customized integrations. The mixing of Anthropic’s laptop use functionality with Amazon Bedrock Brokers represents a transformative method to those challenges.

By utilizing Amazon Bedrock Brokers because the orchestration layer, organizations can alleviate the necessity for customized API improvement for every utility, profit from complete logging and tracing capabilities important for enterprise deployment, and implement automation options rapidly.

As you start exploring laptop use with Amazon Bedrock Brokers, think about workflows in your group that would profit from this method. From bill processing to buyer onboarding, HR documentation to compliance reporting, the potential purposes are huge and transformative.

We’re excited to see how you’ll use Amazon Bedrock Brokers with the pc use functionality to securely streamline operations and reimagine enterprise processes by way of AI-driven automation.

Sources

To be taught extra, confer with the next assets:

In regards to the Authors

Eashan Kaushik is a Specialist Options Architect AI/ML at Amazon Internet Providers. He’s pushed by creating cutting-edge generative AI options whereas prioritizing a customer-centric method to his work. Earlier than this position, he obtained an MS in Laptop Science from NYU Tandon College of Engineering. Outdoors of labor, he enjoys sports activities, lifting, and operating marathons.

Maira Ladeira Tanke is a Tech Lead for Agentic workloads in Amazon Bedrock at AWS, the place she permits prospects on their journey to develop autonomous AI methods. With over 10 years of expertise in AI/ML. At AWS, Maira companions with enterprise prospects to speed up the adoption of agentic purposes utilizing Amazon Bedrock, serving to organizations harness the ability of basis fashions to drive innovation and enterprise transformation. In her free time, Maira enjoys touring, enjoying along with her cat, and spending time along with her household someplace heat.

Raj Pathak is a Principal Options Architect and Technical advisor to Fortune 50 and Mid-Sized FSI (Banking, Insurance coverage, Capital Markets) prospects throughout Canada and the US. Raj focuses on Machine Studying with purposes in Generative AI, Pure Language Processing, Clever Doc Processing, and MLOps.

Adarsh Srikanth is a Software program Improvement Engineer at Amazon Bedrock, the place he develops AI agent companies. He holds a grasp’s diploma in laptop science from USC and brings three years of business expertise to his position. He spends his free time exploring nationwide parks, discovering new climbing trails, and enjoying numerous racquet sports activities.

Abishek Kumar is a Senior Software program Engineer at Amazon, bringing over 6 years of beneficial expertise throughout each retail and AWS organizations. He has demonstrated experience in growing generative AI and machine studying options, particularly contributing to key AWS companies together with SageMaker Autopilot, SageMaker Canvas, and AWS Bedrock Brokers. All through his profession, Abishek has proven ardour for fixing complicated issues and architecting large-scale methods that serve thousands and thousands of consumers worldwide. When not immersed in expertise, he enjoys exploring nature by way of climbing and touring adventures together with his spouse.

Krishna Gourishetti is a Senior Software program Engineer for the Bedrock Brokers crew in AWS. He’s enthusiastic about constructing scalable software program options that clear up buyer issues. In his free time, Krishna likes to go on hikes.