How TReNDS automates root-cause evaluation with Amazon Bedrock

This can be a visitor publish co-written with Vitaly Omelchenko from the TReNDS Middle at Georgia State College.

On the Center for Translational Research in Neuroimaging and Data Science (TReNDS), a joint middle of Georgia State College, Georgia Institute of Expertise, and Emory College, we develop and apply superior analytical strategies and neuroinformatics instruments for mind well being analysis. We’ve been operating our infrastructure on Amazon Net Providers (AWS) since 2019, and over time we’ve constructed a various set of purposes, together with analysis instruments and APIs, all operating on Amazon Elastic Kubernetes Service (Amazon EKS) with logs shipped to Amazon CloudWatch utilizing FluentBit.

As our software grew, so did the amount of errors we wanted to analyze. Once we began exploring Amazon Bedrock, we noticed a chance we had wished for a very long time. We might automate probably the most time-consuming a part of incident response, the root-cause investigation itself.

On this publish, we share the structure we constructed and use in manufacturing at TReNDS. It combines Amazon CloudWatch subscription filters, AWS Lambda, the Strands Agents SDK, and Amazon Bedrock to detect errors in actual time, enrich them with log context and supply code from GitHub, and ship AI-powered root-cause evaluation to our group.

The structure and suggestions on this publish mirror our group’s expertise on the TReNDS Middle and don’t signify official steering from Georgia State College, Georgia Institute of Expertise, or Emory College.

The issue we wished to resolve

Like many groups, we had alerting and monitoring in place. We knew when issues broke. Nonetheless, understanding that one thing failed and understanding why it failed are various things. Our engineers nonetheless needed to open Amazon CloudWatch Logs, learn by means of stack traces, discover the related supply recordsdata, and mentally hint the execution path. For simple errors, this took 15–half-hour. For advanced points spanning a number of providers, for much longer.

We realized that this investigation course of is precisely the type of work a basis mannequin with the precise instruments can do. The mannequin does greater than summarize the error message. It investigates the error by pulling the encompassing log context, studying the supply code, and producing a structured evaluation. That’s what we got down to construct.

Structure

Right here’s the structure we arrived at:

Determine 1 — Structure for automated root-cause evaluation

Our purposes on EKS ship logs to CloudWatch utilizing FluentBit. A CloudWatch subscription filter watches for error-level patterns (ERROR, Exception, FATAL, CRITICAL) and invokes a Lambda operate when a match happens. The Lambda runs a Strands Agent powered by Amazon Bedrock that investigates the error, then publishes the evaluation to an Amazon Easy Notification Service (Amazon SNS) subject for supply to our group.

The core of the system is Amazon Bedrock. The inspiration mannequin (FM) does the precise reasoning about errors, code, and root causes. We use the Strands Brokers SDK on high of Amazon Bedrock to deal with tool-use orchestration. We outline what instruments can be found, and the mannequin decides when and how you can name them. Given a stack hint, the agent may fetch the related supply file, understand it wants extra context, seek for associated error dealing with, and produce a structured evaluation, with out us hardcoding that investigation path.

As a result of TReNDS works with health-related analysis knowledge, knowledge residency and compliance are necessary concerns. Amazon Bedrock processes requests inside our AWS account, so log knowledge and supply code keep throughout the identical setting as the remainder of our software. The AI evaluation doesn’t require sending knowledge to exterior endpoints. This retains knowledge flows inside boundaries we already handle. That is notably necessary for our work, as a result of TReNDS handles health-related analysis knowledge which may fall beneath HIPAA necessities. For extra on Well being Insurance coverage Portability and Accountability Act (HIPAA)-eligible AWS providers, see the AWS HIPAA Eligible Providers Reference.

Whereas our setup makes use of EKS and FluentBit, this sample works with different purposes that ship logs to CloudWatch, together with ECS, Lambda, EC2, or on-premises workloads utilizing the CloudWatch Agent.

Stipulations

To implement this resolution, you want the next:

• An AWS account with entry to Amazon Bedrock (particularly Anthropic Claude Sonnet).
• An Amazon EKS cluster with purposes sending logs to CloudWatch by means of FluentBit.
• CloudWatch log teams configured with subscription filters.
• A GitHub repository containing your software supply code.
• The Strands Brokers SDK put in (accessible by means of the official Lambda layer).
• Familiarity with Python.
• An Amazon SNS subject configured for notifications.
• An AWS Lambda operate with applicable IAM permissions to entry Amazon Bedrock, CloudWatch Logs, AWS Secrets and techniques Supervisor, and SNS.

The agent’s capabilities come from the instruments we give it. Of all of the instruments we constructed, supply code retrieval is probably the most crucial. Stack traces reference file paths and line numbers, however with out entry to the precise implementation, the agent can be restricted to log sample matching. By giving the agent the flexibility to learn supply recordsdata, it could hint execution paths and establish the precise code that prompted the failure. With the Strands Brokers SDK, you outline a customized instrument by adorning a Python operate with @instrument. Right here’s the instrument we constructed to fetch supply code from our GitHub repositories:

import base64
import boto3
import requests
from strands import Agent, instrument

# Retrieve the GitHub token from AWS Secrets and techniques Supervisor
secrets_client = boto3.consumer("secretsmanager")
github_token = secrets_client.get_secret_value(
    SecretId="tendencies/github-token"
)["SecretString"]

@instrument
def fetch_source_code(file_path: str, repo: str) -> str:
    """Fetch a supply file from a GitHub repository.

    Args:
        file_path: Path to the file within the repository
        repo: Repository in 'proprietor/repo' format
    """
    response = requests.get(
        f"https://api.github.com/repos/{repo}/contents/{file_path}",
        headers={"Authorization": f"token {github_token}"}
    )
    if response.status_code != 200:
        return f"Couldn't fetch {file_path} from {repo}: HTTP {response.status_code}"
    content material = base64.b64decode(response.json()["content"])
    return content material.decode("utf-8")

The docstring and kind hints matter. Strands makes use of them to inform the mannequin what the instrument does and what parameters it expects. The mannequin then decides when to name this instrument primarily based on what it finds within the error. See the custom tools documentation for extra patterns.

For deployment, we use the Strands Brokers official Lambda layer. There’s no have to bundle the SDK manually.

How the pipeline works

When an error happens in considered one of our purposes, the pipeline strikes by means of 4 phases routinely. First, CloudWatch detects the error sample and invokes our Lambda operate with the compressed log knowledge. The Lambda decodes the occasion, and the Strands Agent takes over from there. The agent fetches further log context from the identical container, retrieves related supply code from GitHub, and causes by means of the basis trigger. Lastly, it publishes a structured evaluation to SNS for supply to our group. The next sections stroll by means of every stage intimately.

Receiving and decoding CloudWatch occasions

CloudWatch subscription filters ship base64-encoded, gzip-compressed log occasions to Lambda. Every invocation incorporates a number of log occasions that matched the filter sample inside a short while window. The Lambda handler decodes the data, extracts the log group title and matching occasions, and passes them to the agent for evaluation. See the CloudWatch Logs subscription filter documentation for the usual decoding sample.

Fetching prolonged context

The subscription filter delivers the matching log line, however a single line isn’t sufficient. The CloudWatch occasion info consists of the logStream, which identifies the precise container that produced the error. We constructed a second @instrument that fetches surrounding logs from the identical stream. This provides the agent the complete stacktrace and the request context that led to the failure, with out noise from different concurrent requests:

@instrument
def fetch_log_context(log_group: str, log_stream: str, timestamp: int,
                      window_seconds: int = 30) -> str:
    """Fetch log traces from the identical log stream surrounding an error.

    Args:
        log_group: CloudWatch Log Group title
        log_stream: Log stream that produced the error
        timestamp: Error occasion timestamp in milliseconds
        window_seconds: Time window earlier than and after the error
    """
    response = logs_client.filter_log_events(
        logGroupName=log_group,
        logStreamNames=[log_stream],
        startTime=timestamp - (window_seconds * 1000),
        endTime=timestamp + (window_seconds * 1000),
    )
    occasions = response.get("occasions", [])
    if not occasions:
        return f"No log occasions present in {log_stream} inside {window_seconds}s of the error."
    return "n".be a part of(e["message"] for e in occasions)

By scoping to the log stream, we get a clear, chronological sequence of occasions from the identical container. This consists of the request that triggered the error, previous warnings, and the complete exception hint.

Agent evaluation

The agent receives the error plus context, then autonomously decides what to analyze. Not like a rule-based system that follows predefined resolution bushes, the agent interprets the error message, identifies file paths and sophistication names within the stack hint, and determines which supply recordsdata to retrieve. If the preliminary code overview reveals that the error originates in a dependency or a shared utility, the agent follows that chain with out further prompting from us. We formed the output format by means of the system immediate:

SYSTEM_PROMPT = """You're a senior Website Reliability Engineer analyzing manufacturing errors.

Given an error and its surrounding log context:
1. Establish the basis trigger by analyzing the stacktrace
2. Use fetch_source_code to learn the related supply recordsdata
3. Present a structured evaluation with:
   - Severity (CRITICAL/HIGH/MEDIUM/LOW)
   - Root trigger rationalization
   - Related supply code context
   - Recommended repair
   - Associated areas that could be affected
"""

The system immediate defines a structured output format however leaves the investigation technique to the mannequin. The agent decides which instruments to name primarily based on what it finds within the error. A stack hint with clear file paths triggers fetch_source_code calls. An error with no stack hint may lead the agent to go looking the code base for the error message string. This flexibility is the core worth of the agentic strategy. We didn’t have to anticipate each kind of error our purposes can produce.

The Lambda handler ties every part collectively:

agent = Agent(
    mannequin=BedrockModel(model_id="us.anthropic.claude-sonnet-4-20250514"),
    system_prompt=SYSTEM_PROMPT,
    instruments=[fetch_source_code, search_github_code, fetch_log_context]
)

end result = agent(
    f"Analyze this error from {log_group}:nn{error_message}"
)

The handler creates an Agent occasion with our chosen Amazon Bedrock mannequin, the system immediate that defines the output format, and the listing of accessible instruments. It then passes the error message together with the log group title to the agent, which triggers the autonomous investigation loop.

Delivering outcomes

After the agent completes its evaluation, we publish the end result to an Amazon SNS subject and fan out to electronic mail and Slack. Right here’s what a typical notification appears like:

Error Evaluation — order-service

Severity: HIGH
Error: NullPointerException at OrderService.java:142

Root Trigger: The strategy processPayment() calls paymentGateway.cost()
which might return null on gateway timeout, however line 142 accesses
response.getTransactionId() with no null verify.

Supply Context (OrderService.java:138-148):
[relevant code shown]

Recommended Repair: Add null verify for gateway response earlier than accessing
fields. Take into account retry mechanism for gateway timeouts.

Associated: Related sample in RefundService.java:89

The agent autonomously investigated this error with out human steering. It learn the related supply code, recognized the null verify hole, and even flagged the same sample in one other file. This demonstrates the worth of the agentic strategy. Moderately than following a set guidelines, the agent adapts its investigation technique primarily based on what it discovers at every step, very similar to an skilled engineer would.

Outcomes

Since deploying this method, we have now seen a transparent influence on how our group handles manufacturing errors. Essentially the most instant change is velocity. Investigation time dropped from 15 to half-hour all the way down to beneath 60 seconds. As a result of the agent’s evaluation features a prompt repair, our engineers typically obtain a prepared resolution of their inbox. They’ll go straight to implementing the repair as an alternative of spending time on analysis.

The price of operating this method is negligible. Every evaluation incurs solely minimal Amazon Bedrock inference costs, usually involving two to a few tool-use rounds per error. For our workload, it is a fraction of what the equal engineer time would value.

Our builders obtain the agent’s evaluation by electronic mail, and the suggestions has been constantly constructive. The analyses present a transparent place to begin for decision, even for errors the engineer has not encountered earlier than. Engineers can shortly perceive what occurred and what to do about it with out further investigation.

After a launch, the identical code path can produce repeated errors. Our deduplication, which makes use of Amazon DynamoDB, makes positive that solely the primary prevalence triggers an evaluation. The remainder are silently filtered, retaining inboxes clear and Amazon Bedrock prices low.

Selecting the best Amazon Bedrock mannequin

Amazon Bedrock offers us entry to a variety of basis fashions by means of a single API. We examined a number of to seek out the most effective match for our error evaluation use case, evaluating reasoning high quality (understanding code and errors), instrument use reliability (calling our GitHub and CloudWatch instruments), latency, and price per evaluation.

We chosen Claude Sonnet as our major mannequin. In our testing, it constantly produced probably the most correct root-cause analyses. It could possibly hint by means of multi-file name chains, establish refined points like lacking null checks, and cause about concurrency issues. For groups with totally different value or latency necessities, the opposite fashions within the desk are sturdy alternate options for less complicated error patterns.

Switching fashions with Strands is a one-line change, which made our analysis simple:

# We examined a number of fashions by altering this single line.
# Test Amazon Bedrock documentation for the most recent accessible mannequin IDs.
mannequin = BedrockModel(model_id="us.anthropic.claude-sonnet-4-20250514")

Word: Mannequin IDs are up to date recurrently. See the Amazon Bedrock supported fashions documentation for present mannequin IDs.

Subsequent steps

We’re exploring a number of extensions to this method. The primary precedence is connecting Retrieval Augmented Era (RAG) with our inner runbooks.

• By integrating Amazon Bedrock Data Bases with our inner runbooks and previous incident stories, the agent will be capable of reference TReNDS-specific procedures in its evaluation.
• We additionally plan to implement a tiered mannequin technique. Easy, recognized error patterns would path to Haiku for quick, low-cost triage, whereas advanced or novel errors would escalate to Sonnet for deep evaluation. This is able to optimize each value and response time throughout our error quantity.
• Lastly, we’re working towards automated GitHub difficulty and pull request creation. When the agent identifies a possible repair, it could routinely create a GitHub difficulty with the evaluation and open a pull request with the prompt code change, decreasing the handbook steps between analysis and determination.

As we scale this additional, we’re additionally Amazon Bedrock AgentCore for managed agent runtime, observability, and identification administration. See the Strands Agents examples for multi-agent and deployment patterns.

Conclusion

We constructed this method at TReNDS as a result of we wished each error in our software to get an instantaneous, structured investigation as an alternative of solely an alert. Amazon Bedrock and the Strands Brokers SDK made it simple to implement. We outlined just a few instruments and wrote a system immediate. Now, we have now an agent that causes by means of manufacturing errors the identical method an skilled engineer would. It delivers ends in seconds.

Including a brand new functionality means writing one other @instrument operate and some traces of Python. Whether or not the necessity is a Jira integration, a GitHub PR with a prompt repair, or a instrument that connects to a operating pod for deeper investigation, the sample is identical. The inspiration mannequin handles the reasoning and orchestration, and we join it to the methods it wants.

In regards to the authors