Tracing OpenAI Agent Response Utilizing MLFLOW

MLFLOW is an open supply platform for managing and monitoring machine studying experiments. When used with the Openai Brokers SDK, MLFLOW routinely seems like this:

• File all agent interactions and API calls
• Seize device utilization, enter/output messages, and intermediate choices
• Tracks are run for debugging, efficiency evaluation, and reproducibility

That is particularly helpful when constructing multi-agent techniques the place numerous brokers dynamically invoke collaboration or performance

This tutorial offers two necessary examples. Easy handoffs between brokers and utilizing agent guardrails – whereas tracing conduct utilizing MLFLOW.

Set dependencies

Putting in the library

pip set up openai-agents mlflow pydantic pydotenv

Openai API Key

Go to to get Openai API key https://platform.openai.com/settings/organization/api-keys Generates a brand new key. For brand spanking new customers, you have to so as to add billing particulars and make a minimal of $5 funds to activate API entry.

As soon as the secret is generated, create a .ENV file and enter:

OPENAI_API_KEY = <YOUR_API_KEY>

exchange with the generated key.

Multi-agent system (multi_agent_demo.py)

This script (multi_agent_demo.py) builds a easy multi-agent assistant utilizing the Openai Brokers SDK, designed to route person queries to coding specialists or cooking specialists. Allow it mlflow.openai.autolog()Robotically observe and log all agent interactions with Openai API, together with enter, output, and agent handoffs, making system monitoring and debugging simpler. MLFLOW is configured to make use of an area file-based monitoring URI (./mlruns) and file all actions underneath the experiment identify.Agent Coding Cooking“.

import mlflow, asyncio
from brokers import Agent, Runner
import os
from dotenv import load_dotenv
load_dotenv()

mlflow.openai.autolog()                           # Auto‑hint each OpenAI name
mlflow.set_tracking_uri("./mlruns")
mlflow.set_experiment("Agent‑Coding‑Cooking")

coding_agent = Agent(identify="Coding agent",
                     directions="You solely reply coding questions.")

cooking_agent = Agent(identify="Cooking agent",
                      directions="You solely reply cooking questions.")

triage_agent = Agent(
    identify="Triage agent",
    directions="If the request is about code, handoff to coding_agent; "
                 "if about cooking, handoff to cooking_agent.",
    handoffs=[coding_agent, cooking_agent],
)

async def fundamental():
    res = await Runner.run(triage_agent,
                           enter="How do I boil pasta al dente?")
    print(res.final_output)

if __name__ == "__main__":
    asyncio.run(fundamental())

mlflow ui

To open the MLFLOW UI and consider all log agent interactions, run the next command in your new terminal:

This may begin the MLFLOW monitoring server and immediate you to see a URL and port that the UI can entry. http:// localhost:5000 By default.

You possibly can view your entire interplay movement of hint Part – From the preliminary enter of a person, to how the assistant routes the request to the suitable agent, and eventually, the response generated by that agent. This end-to-end tracing offers invaluable perception into decision-making, handoffs, and output, and helps you debug and optimize agent workflows.

Hint Guardrails (Guardrails.py)

This instance makes use of the OpenAI Agent SDK with MLFLOW traces to implement a GuardRail protected buyer assist agent. Brokers are designed to assist customers asking frequent questions, however have restricted solutions to medical-related questions. A devoted guardrail agent will verify for such enter and block requests if detected. MLFLOW captures your entire movement, together with guardrail activation, inference, and agent responses. It offers full traceability and perception into security mechanisms.

import mlflow, asyncio
from pydantic import BaseModel
from brokers import (
    Agent, Runner,
    GuardrailFunctionOutput, InputGuardrailTripwireTriggered,
    input_guardrail, RunContextWrapper)

from dotenv import load_dotenv
load_dotenv()

mlflow.openai.autolog()
mlflow.set_tracking_uri("./mlruns")
mlflow.set_experiment("Agent‑Guardrails")

class MedicalSymptons(BaseModel):
    medical_symptoms: bool
    reasoning: str


guardrail_agent = Agent(
    identify="Guardrail verify",
    directions="Test if the person is asking you for medical symptons.",
    output_type=MedicalSymptons,
)


@input_guardrail
async def medical_guardrail(
    ctx: RunContextWrapper[None], agent: Agent, enter
) -> GuardrailFunctionOutput:
    outcome = await Runner.run(guardrail_agent, enter, context=ctx.context)

    return GuardrailFunctionOutput(
        output_info=outcome.final_output,
        tripwire_triggered=outcome.final_output.medical_symptoms,
    )


agent = Agent(
    identify="Buyer assist agent",
    directions="You're a buyer assist agent. You assist clients with their questions.",
    input_guardrails=[medical_guardrail],
)


async def fundamental():
    strive:
        await Runner.run(agent, "Ought to I take aspirin if I am having a headache?")
        print("Guardrail did not journey - that is surprising")

    besides InputGuardrailTripwireTriggered:
        print("Medical guardrail tripped")


if __name__ == "__main__":
    asyncio.run(fundamental())

This script defines a buyer assist agent utilizing enter guardrails that detect medical-related questions. Use a distinct GuardRail_Agent to evaluate whether or not a person’s enter features a request for medical recommendation. If such an enter is detected, the guardrail prevents and prevents the primary agent from responding. All the course of, together with Guardrail checks and outcomes, is routinely logged and traced utilizing MLFlow.

mlflow ui

To open the MLFLOW UI and consider all log agent interactions, run the next command in your new terminal:

On this instance, the agent requested, “Do I must take aspirin if I’ve a headache?” The MLFLOW UI clearly exhibits that the enter was flagged, together with the inference offered by the GuardRail agent as to why the request was blocked.

I’m a civil engineering graduate (2022) from Jamia Milia Islamia, New Delhi, and have a powerful curiosity in knowledge science, notably neural networks and purposes in a wide range of fields.