Use {custom} metrics to guage your generative AI utility with Amazon Bedrock

With Amazon Bedrock Evaluations, you possibly can consider basis fashions (FMs) and Retrieval Augmented Era (RAG) methods, whether or not hosted on Amazon Bedrock or one other mannequin or RAG system hosted elsewhere, together with Amazon Bedrock Information Bases or multi-cloud and on-premises deployments. We lately introduced the overall availability of the massive language mannequin (LLM)-as-a-judge approach in mannequin analysis and the brand new RAG analysis device, additionally powered by an LLM-as-a-judge behind the scenes. These instruments are already empowering organizations to systematically consider FMs and RAG methods with enterprise-grade instruments. We additionally talked about that these analysis instruments don’t should be restricted to fashions or RAG methods hosted on Amazon Bedrock; with the convey your personal inference (BYOI) responses characteristic, you possibly can consider fashions or purposes if you happen to use the enter formatting necessities for both providing.

The LLM-as-a-judge approach powering these evaluations allows automated, human-like analysis high quality at scale, utilizing FMs to evaluate high quality and accountable AI dimensions with out guide intervention. With built-in metrics like correctness (factual accuracy), completeness (response thoroughness), faithfulness (hallucination detection), and accountable AI metrics corresponding to harmfulness and reply refusal, you and your staff can consider fashions hosted on Amazon Bedrock and information bases natively, or utilizing BYOI responses out of your custom-built methods.

Amazon Bedrock Evaluations affords an in depth listing of built-in metrics for each analysis instruments, however there are occasions once you would possibly need to outline these analysis metrics otherwise, or make utterly new metrics which can be related to your use case. For instance, you would possibly need to outline a metric that evaluates an utility response’s adherence to your particular model voice, or need to classify responses in line with a {custom} categorical rubric. You would possibly need to use numerical scoring or categorical scoring for numerous functions. For these causes, you want a means to make use of {custom} metrics in your evaluations.

Now with Amazon Bedrock, you possibly can develop {custom} analysis metrics for each mannequin and RAG evaluations. This functionality extends the LLM-as-a-judge framework that drives Amazon Bedrock Evaluations.

On this put up, we exhibit how you can use {custom} metrics in Amazon Bedrock Evaluations to measure and enhance the efficiency of your generative AI purposes in line with your particular enterprise necessities and analysis standards.

Overview

Customized metrics in Amazon Bedrock Evaluations provide the next options:

• Simplified getting began expertise – Pre-built starter templates can be found on the AWS Administration Console based mostly on our industry-tested built-in metrics, with choices to create from scratch for particular analysis standards.
• Versatile scoring methods – Assist is offered for each quantitative (numerical) and qualitative (categorical) scoring to create ordinal metrics, nominal metrics, and even use analysis instruments for classification duties.
• Streamlined workflow administration – It can save you {custom} metrics for reuse throughout a number of analysis jobs or import beforehand outlined metrics from JSON recordsdata.
• Dynamic content material integration – With built-in template variables (for instance, {{immediate}}, {{prediction}}, and {{context}}), you possibly can seamlessly inject dataset content material and mannequin outputs into analysis prompts.
• Customizable output management – You should utilize our advisable output schema for constant outcomes, with superior choices to outline {custom} output codecs for specialised use circumstances.

Customized metrics offer you unprecedented management over the way you measure AI system efficiency, so you possibly can align evaluations together with your particular enterprise necessities and use circumstances. Whether or not assessing factuality, coherence, helpfulness, or domain-specific standards, {custom} metrics in Amazon Bedrock allow extra significant and actionable analysis insights.

Within the following sections, we stroll by the steps to create a job with mannequin analysis and {custom} metrics utilizing each the Amazon Bedrock console and the Python SDK and APIs.

Supported knowledge codecs

On this part, we evaluation some essential knowledge codecs.

Decide immediate importing

To add your beforehand saved {custom} metrics into an analysis job, observe the JSON format within the following examples.

The next code illustrates a definition with numerical scale:

{
    "customMetricDefinition": {
        "metricName": "my_custom_metric",
        "directions": "Your full {custom} metric immediate together with no less than one {{enter variable}}",
        "ratingScale": [
            {
                "definition": "first rating definition",
                "value": {
                    "floatValue": 3
                }
            },
            {
                "definition": "second rating definition",
                "value": {
                    "floatValue": 2
                }
            },
            {
                "definition": "third rating definition",
                "value": {
                    "floatValue": 1
                }
            }
        ]
    }
}

The next code illustrates a definition with string scale:

{
    "customMetricDefinition": {
        "metricName": "my_custom_metric",
        "directions": "Your full {custom} metric immediate together with no less than one {{enter variable}}",
        "ratingScale": [
            {
                "definition": "first rating definition",
                "value": {
                    "stringValue": "first value"
                }
            },
            {
                "definition": "second rating definition",
                "value": {
                    "stringValue": "second value"
                }
            },
            {
                "definition": "third rating definition",
                "value": {
                    "stringValue": "third value"
                }
            }
        ]
    }
}

The next code illustrates a definition with no scale:

{
    "customMetricDefinition": {
        "metricName": "my_custom_metric",
        "directions": "Your full {custom} metric immediate together with no less than one {{enter variable}}"
    }
}

For extra data on defining a decide immediate with no scale, see the perfect practices part later on this put up.

Mannequin analysis dataset format

When utilizing LLM-as-a-judge, just one mannequin may be evaluated per analysis job. Consequently, you should present a single entry within the modelResponses listing for every analysis, although you possibly can run a number of analysis jobs to check totally different fashions. The modelResponses subject is required for BYOI jobs, however not wanted for non-BYOI jobs. The next is the enter JSONL format for LLM-as-a-judge in mannequin analysis. Fields marked with ? are optionally available.

{
    "immediate": string
    "referenceResponse"?: string
    "class"?: string
     "modelResponses"?: [
        {
            "response": string
            "modelIdentifier": string
        }
    ]
}

RAG analysis dataset format

We up to date the analysis job enter dataset format to be much more versatile for RAG analysis. Now, you possibly can convey referenceContexts, that are anticipated retrieved passages, so you possibly can examine your precise retrieved contexts to your anticipated retrieved contexts. You will discover the brand new referenceContexts subject within the up to date JSONL schema for RAG analysis:

{
    "conversationTurns": [{
            "prompt": {
                "content": [{
                    "text": string
                }]
            },
            "referenceResponses": [{
                "content": [{
                    "text": string
                }]
            }],
            "referenceContexts" ? : [{
                "content": [{
                    "text": string
                }]
            }],
            "output": {
                "textual content": string "modelIdentifier" ? : string "knowledgeBaseIdentifier": string "retrievedPassages": {
                    "retrievalResults": [{
                        "name" ? : string "content": {
                            "text": string
                        },
                        "metadata" ? : {
                            [key: string]: string
                        }
                    }]
                }
            }]
    }

Variables for knowledge injection into decide prompts

To make it possible for your knowledge is injected into the decide prompts in the correct place, use the variables from the next desk. We have now additionally included a information to point out you the place the analysis device will pull knowledge out of your enter file, if relevant. There are circumstances the place if you happen to convey your personal inference responses to the analysis job, we’ll use that knowledge out of your enter file; if you happen to don’t use convey your personal inference responses, then we’ll name the Amazon Bedrock mannequin or information base and put together the responses for you.

The next desk summarizes the variables for mannequin analysis.

The next desk summarizes the variables for RAG analysis (retrieve solely).

The next desk summarizes the variables for RAG analysis (retrieve and generate).

Stipulations

To make use of the LLM-as-a-judge mannequin analysis and RAG analysis options with BYOI, you should have the next stipulations:

Create a mannequin analysis job with {custom} metrics utilizing Amazon Bedrock Evaluations

Full the next steps to create a job with mannequin analysis and {custom} metrics utilizing Amazon Bedrock Evaluations:

1. On the Amazon Bedrock console, select Evaluations within the navigation pane and select the Fashions
2. Within the Mannequin analysis part, on the Create dropdown menu, select Automated: mannequin as a decide.
   
3. For the Mannequin analysis particulars, enter an analysis identify and optionally available description.
   
4. For Evaluator mannequin, select the mannequin you need to use for automated analysis.
5. For Inference supply, choose the supply and select the mannequin you need to consider.

For this instance, we selected Claude 3.5 Sonnet because the evaluator mannequin, Bedrock fashions as our inference supply, and Claude 3.5 Haiku as our mannequin to guage.

5. The console will show the default metrics for the evaluator mannequin you selected. You possibly can choose different metrics as wanted.
   
6. Within the Customized Metrics part, we create a brand new metric referred to as “Comprehensiveness.” Use the template supplied and modify based mostly in your metrics. You should utilize the next variables to outline the metric, the place solely {{prediction}} is necessary:
   1. immediate
   2. prediction
   3. ground_truth

The next is the metric we outlined in full:

Your function is to evaluate the comprehensiveness of a solution based mostly on the query and 
the prediction. Assess the standard, accuracy, and helpfulness of language mannequin response,
 and use these to evaluate how complete the response is. Award increased scores to responses
 which can be detailed and considerate.

Fastidiously consider the comprehensiveness of the LLM response for the given question (immediate)
 in opposition to all specified standards. Assign a single total rating that finest represents the 
comprehensivenss, and supply a quick clarification justifying your score, referencing 
particular strengths and weaknesses noticed.

When evaluating the response high quality, think about the next rubrics:
- Accuracy: Factual correctness of data supplied
- Completeness: Protection of essential elements of the question
- Readability: Clear group and presentation of data
- Helpfulness: Sensible utility of the response to the consumer

Consider the next:

Question:
{{immediate}}

Response to guage:
{{prediction}}

7. Create the output schema and extra metrics. Right here, we outline a scale that gives most factors (10) if the response could be very complete, and 1 if the response shouldn’t be complete in any respect.
   
8. For Datasets, enter your enter and output areas in Amazon S3.
9. For Amazon Bedrock IAM function – Permissions, choose Use an present service function and select a job.
   
10. Select Create and watch for the job to finish.
    

Concerns and finest practices

When utilizing the output schema of the {custom} metrics, notice the next:

• If you happen to use the built-in output schema (advisable), don’t add your grading scale into the principle decide immediate. The analysis service will routinely concatenate your decide immediate directions together with your outlined output schema score scale and a few structured output directions (distinctive to every decide mannequin) behind the scenes. That is so the analysis service can parse the decide mannequin’s outcomes and show them on the console in graphs and calculate common values of numerical scores.
• The totally concatenated decide prompts are seen within the Preview window in case you are utilizing the Amazon Bedrock console to assemble your {custom} metrics. As a result of decide LLMs are inherently stochastic, there may be some responses we will’t parse and show on the console and use in your common rating calculations. Nevertheless, the uncooked decide responses are at all times loaded into your S3 output file, even when the analysis service can’t parse the response rating from the decide mannequin.
• If you happen to don’t use the built-in output schema characteristic (we suggest you employ it as a substitute of ignoring it), then you’re chargeable for offering your score scale within the decide immediate directions physique. Nevertheless, the analysis service won’t add structured output directions and won’t parse the outcomes to point out graphs; you will note the total decide output plaintext outcomes on the console with out graphs and the uncooked knowledge will nonetheless be in your S3 bucket.

Create a mannequin analysis job with {custom} metrics utilizing the Python SDK and APIs

To make use of the Python SDK to create a mannequin analysis job with {custom} metrics, observe these steps (or discuss with our example notebook):

2. Arrange the required configurations, which ought to embrace your mannequin identifier for the default metrics and {custom} metrics evaluator, IAM function with applicable permissions, Amazon S3 paths for enter knowledge containing your inference responses, and output location for outcomes:

   import boto3
   import time
   from datetime import datetime
   
   # Configure information base and mannequin settings
   evaluator_model = "anthropic.claude-3-5-sonnet-20240620-v1:0"
   generator_model = "amazon.nova-lite-v1:0"
   custom_metrics_evaluator_model = "anthropic.claude-3-5-sonnet-20240620-v1:0"
   role_arn = "arn:aws:iam::<YOUR_ACCOUNT_ID>:function/<YOUR_IAM_ROLE>"
   BUCKET_NAME = "<YOUR_BUCKET_NAME>"
   
   # Specify S3 areas
   input_data = f"s3://{BUCKET_NAME}/evaluation_data/enter.jsonl"
   output_path = f"s3://{BUCKET_NAME}/evaluation_output/"
   
   # Create Bedrock shopper
   # NOTE: You possibly can change the area identify to the area of your selecting.
   bedrock_client = boto3.shopper('bedrock', region_name="us-east-1") 

3. To outline a {custom} metric for mannequin analysis, create a JSON construction with a customMetricDefinition Embody your metric’s identify, write detailed analysis directions incorporating template variables (corresponding to {{immediate}} and {{prediction}}), and outline your ratingScale array with evaluation values utilizing both numerical scores (floatValue) or categorical labels (stringValue). This correctly formatted JSON schema allows Amazon Bedrock to guage mannequin outputs persistently in line with your particular standards.

   comprehensiveness_metric ={
       "customMetricDefinition": {
           "identify": "comprehensiveness",
           "directions": """Your function is to evaluate the comprehensiveness of an 
   reply based mostly on the query and the prediction. Assess the standard, accuracy, 
   and helpfulness of language mannequin response, and use these to evaluate how complete
    the response is. Award increased scores to responses which can be detailed and considerate.
   
   Fastidiously consider the comprehensiveness of the LLM response for the given question (immediate)
    in opposition to all specified standards. Assign a single total rating that finest represents the 
   comprehensivenss, and supply a quick clarification justifying your score, referencing 
   particular strengths and weaknesses noticed.
   
   When evaluating the response high quality, think about the next rubrics:
   - Accuracy: Factual correctness of data supplied
   - Completeness: Protection of essential elements of the question
   - Readability: Clear group and presentation of data
   - Helpfulness: Sensible utility of the response to the consumer
   
   Consider the next:
   
   Question:
   {{immediate}}
   
   Response to guage:
   {{prediction}}""",
           "ratingScale": [
               {
                   "definition": "Very comprehensive",
                   "value": {
                       "floatValue": 10
                   }
               },
               {
                   "definition": "Mildly comprehensive",
                   "value": {
                       "floatValue": 3
                   }
               },
               {
                   "definition": "Not at all comprehensive",
                   "value": {
                       "floatValue": 1
                   }
               }
           ]
       }
   }

4. To create a mannequin analysis job with {custom} metrics, use the create_evaluation_job API and embrace your {custom} metric within the customMetricConfig part, specifying each built-in metrics (corresponding to Builtin.Correctness) and your {custom} metric within the metricNames array. Configure the job together with your generator mannequin, evaluator mannequin, and correct Amazon S3 paths for enter dataset and output outcomes.

   # Create the mannequin analysis job
   model_eval_job_name = f"model-evaluation-custom-metrics{datetime.now().strftime('%Y-%m-%d-%H-%M-%S')}"
   
   model_eval_job = bedrock_client.create_evaluation_job(
       jobName=model_eval_job_name,
       jobDescription="Consider mannequin efficiency with {custom} comprehensiveness metric",
       roleArn=role_arn,
       applicationType="ModelEvaluation",
       inferenceConfig={
           "fashions": [{
               "bedrockModel": {
                   "modelIdentifier": generator_model
               }
           }]
       },
       outputDataConfig={
           "s3Uri": output_path
       },
       evaluationConfig={
           "automated": {
               "datasetMetricConfigs": [{
                   "taskType": "General",
                   "dataset": {
                       "name": "ModelEvalDataset",
                       "datasetLocation": {
                           "s3Uri": input_data
                       }
                   },
                   "metricNames": [
                       "Builtin.Correctness",
                       "Builtin.Completeness",
                       "Builtin.Coherence",
                       "Builtin.Relevance",
                       "Builtin.FollowingInstructions",
                       "comprehensiveness"
                   ]
               }],
               "customMetricConfig": {
                   "customMetrics": [
                       comprehensiveness_metric
                   ],
                   "evaluatorModelConfig": {
                       "bedrockEvaluatorModels": [{
                           "modelIdentifier": custom_metrics_evaluator_model
                       }]
                   }
               },
               "evaluatorModelConfig": {
                   "bedrockEvaluatorModels": [{
                       "modelIdentifier": evaluator_model
                   }]
               }
           }
       }
   )
   
   print(f"Created mannequin analysis job: {model_eval_job_name}")
   print(f"Job ID: {model_eval_job['jobArn']}")

5. After submitting the analysis job, monitor its standing with get_evaluation_job and entry outcomes at your specified Amazon S3 location when full, together with the usual and {custom} metric efficiency knowledge.

Create a RAG system analysis with {custom} metrics utilizing Amazon Bedrock Evaluations

On this instance, we stroll by a RAG system analysis with a mixture of built-in metrics and {custom} analysis metrics on the Amazon Bedrock console. Full the next steps:

1. On the Amazon Bedrock console, select Evaluations within the navigation pane.
2. On the RAG tab, select Create.
   
3. For the RAG analysis particulars, enter an analysis identify and optionally available description.
   
4. For Evaluator mannequin, select the mannequin you need to use for automated analysis. The evaluator mannequin chosen right here can be used to calculate default metrics if chosen. For this instance, we selected Claude 3.5 Sonnet because the evaluator mannequin.
5. Embody any optionally available tags.
   
6. For Inference supply, choose the supply. Right here, you may have the choice to pick out between Bedrock Information Bases and Carry your personal inference responses. If you happen to’re utilizing Amazon Bedrock Information Bases, you will have to decide on a beforehand created information base or create a brand new one. For BYOI responses, you possibly can convey the immediate dataset, context, and output from a RAG system. For this instance, we selected Bedrock Information Base as our inference supply.
   
7. Specify the analysis kind, response generator mannequin, and built-in metrics. You possibly can select between a mixed retrieval and response analysis or a retrieval solely analysis, with choices to make use of default metrics, {custom} metrics, or each to your RAG analysis. The response generator mannequin is barely required when utilizing an Amazon Bedrock information base because the inference supply. For the BYOI configuration, you possibly can proceed and not using a response generator. For this instance, we chosen Retrieval and response technology as our analysis kind and selected Nova Lite 1.0 as our response generator mannequin.
   
8. Within the Customized Metrics part, select your evaluator mannequin. We chosen Claude 3.5 Sonnet v1 as our evaluator mannequin for {custom} metrics.
9. Select Add {custom} metrics.
   
10. Create your new metric. For this instance, we create a brand new {custom} metric for our RAG analysis referred to as information_comprehensiveness. This metric evaluates how totally and utterly the response addresses the question through the use of the retrieved data. It measures the extent to which the response extracts and incorporates related data from the retrieved passages to offer a complete reply.
11. You possibly can select between importing a JSON file, utilizing a preconfigured template, or making a {custom} metric with full configuration management. For instance, you possibly can choose the preconfigured templates for the default metrics and alter the scoring system or rubric. For our information_comprehensiveness metric, we choose the {custom} choice, which permits us to enter our evaluator immediate immediately.
    
12. For Directions, enter your immediate. For instance:

    Your function is to guage how comprehensively the response addresses the question 
    utilizing the retrieved data. Assess whether or not the response gives an intensive 
    therapy of the topic by successfully using the accessible retrieved passages.
    
    Fastidiously consider the comprehensiveness of the RAG response for the given question
     in opposition to all specified standards. Assign a single total rating that finest represents
     the comprehensiveness, and supply a quick clarification justifying your score, 
    referencing particular strengths and weaknesses noticed.
    
    When evaluating response comprehensiveness, think about the next rubrics:
    - Protection: Does the response make the most of the important thing related data from the retrieved
     passages?
    - Depth: Does the response present enough element on essential elements from the
     retrieved data?
    - Context utilization: How successfully does the response leverage the accessible
     retrieved passages?
    - Info synthesis: Does the response mix retrieved data to create
     an intensive therapy?
    
    Consider the next:
    
    Question: {{immediate}}
    
    Retrieved passages: {{context}}
    
    Response to guage: {{prediction}}

13. Enter your output schema to outline how the {custom} metric outcomes can be structured, visualized, normalized (if relevant), and defined by the mannequin.

If you happen to use the built-in output schema (advisable), don’t add your score scale into the principle decide immediate. The analysis service will routinely concatenate your decide immediate directions together with your outlined output schema score scale and a few structured output directions (distinctive to every decide mannequin) behind the scenes in order that your decide mannequin outcomes may be parsed. The totally concatenated decide prompts are seen within the Preview window in case you are utilizing the Amazon Bedrock console to assemble your {custom} metrics.

14. For Dataset and analysis outcomes S3 location, enter your enter and output areas in Amazon S3.
15. For Amazon Bedrock IAM function – Permissions, choose Use an present service function and select your function.
    
16. Select Create and watch for the job to finish.
    

Begin a RAG analysis job with {custom} metrics utilizing the Python SDK and APIs

To make use of the Python SDK for creating an RAG analysis job with {custom} metrics, observe these steps (or discuss with our example notebook):

1. Arrange the required configurations, which ought to embrace your mannequin identifier for the default metrics and {custom} metrics evaluator, IAM function with applicable permissions, information base ID, Amazon S3 paths for enter knowledge containing your inference responses, and output location for outcomes:

   import boto3
   import time
   from datetime import datetime
   
   # Configure information base and mannequin settings
   knowledge_base_id = "<YOUR_KB_ID>"
   evaluator_model = "anthropic.claude-3-5-sonnet-20240620-v1:0"
   generator_model = "amazon.nova-lite-v1:0"
   custom_metrics_evaluator_model = "anthropic.claude-3-5-sonnet-20240620-v1:0"
   role_arn = "arn:aws:iam::<YOUR_ACCOUNT_ID>:function/<YOUR_IAM_ROLE>"
   BUCKET_NAME = "<YOUR_BUCKET_NAME>"
   
   # Specify S3 areas
   input_data = f"s3://{BUCKET_NAME}/evaluation_data/enter.jsonl"
   output_path = f"s3://{BUCKET_NAME}/evaluation_output/"
   
   # Configure retrieval settings
   num_results = 10
   search_type = "HYBRID"
   
   # Create Bedrock shopper
   # NOTE: You possibly can change the area identify to the area of your selecting
   bedrock_client = boto3.shopper('bedrock', region_name="us-east-1") 

2. To outline a {custom} metric for RAG analysis, create a JSON construction with a customMetricDefinition Embody your metric’s identify, write detailed analysis directions incorporating template variables (corresponding to {{immediate}}, {{context}}, and {{prediction}}), and outline your ratingScale array with evaluation values utilizing both numerical scores (floatValue) or categorical labels (stringValue). This correctly formatted JSON schema allows Amazon Bedrock to guage responses persistently in line with your particular standards.

   # Outline our {custom} information_comprehensiveness metric
   information_comprehensiveness_metric = {
       "customMetricDefinition": {
           "identify": "information_comprehensiveness",
           "directions": """
           Your function is to guage how comprehensively the response addresses the 
   question utilizing the retrieved data. 
           Assess whether or not the response gives an intensive therapy of the topic
   by successfully using the accessible retrieved passages.
   
   Fastidiously consider the comprehensiveness of the RAG response for the given question
   in opposition to all specified standards. 
   Assign a single total rating that finest represents the comprehensiveness, and 
   present a quick clarification justifying your score, referencing particular strengths
   and weaknesses noticed.
   
   When evaluating response comprehensiveness, think about the next rubrics:
   - Protection: Does the response make the most of the important thing related data from the 
   retrieved passages?
   - Depth: Does the response present enough element on essential elements from 
   the retrieved data?
   - Context utilization: How successfully does the response leverage the accessible 
   retrieved passages?
   - Info synthesis: Does the response mix retrieved data to 
   create an intensive therapy?
   
   Consider utilizing the next:
   
   Question: {{immediate}}
   
   Retrieved passages: {{context}}
   
   Response to guage: {{prediction}}
   """,
           "ratingScale": [
               {
                   "definition": "Very comprehensive",
                   "value": {
                       "floatValue": 3
                   }
               },
               {
                   "definition": "Moderately comprehensive",
                   "value": {
                       "floatValue": 2
                   }
               },
               {
                   "definition": "Minimally comprehensive",
                   "value": {
                       "floatValue": 1
                   }
               },
               {
                   "definition": "Not at all comprehensive",
                   "value": {
                       "floatValue": 0
                   }
               }
           ]
       }
   }

3. To create a RAG analysis job with {custom} metrics, use the create_evaluation_job API and embrace your {custom} metric within the customMetricConfig part, specifying each built-in metrics (Builtin.Correctness) and your {custom} metric within the metricNames array. Configure the job together with your information base ID, generator mannequin, evaluator mannequin, and correct Amazon S3 paths for enter dataset and output outcomes.

   # Create the analysis job
   retrieve_generate_job_name = f"rag-evaluation-generate-{datetime.now().strftime('%Y-%m-%d-%H-%M-%S')}"
   
   retrieve_generate_job = bedrock_client.create_evaluation_job(
       jobName=retrieve_generate_job_name,
       jobDescription="Consider retrieval and technology with {custom} metric",
       roleArn=role_arn,
       applicationType="RagEvaluation",
       inferenceConfig={
           "ragConfigs": [{
               "knowledgeBaseConfig": {
                   "retrieveAndGenerateConfig": {
                       "type": "KNOWLEDGE_BASE",
                       "knowledgeBaseConfiguration": {
                           "knowledgeBaseId": knowledge_base_id,
                           "modelArn": generator_model,
                           "retrievalConfiguration": {
                               "vectorSearchConfiguration": {
                                   "numberOfResults": num_results
                               }
                           }
                       }
                   }
               }
           }]
       },
       outputDataConfig={
           "s3Uri": output_path
       },
       evaluationConfig={
           "automated": {
               "datasetMetricConfigs": [{
                   "taskType": "General",
                   "dataset": {
                       "name": "RagDataset",
                       "datasetLocation": {
                           "s3Uri": input_data
                       }
                   },
                   "metricNames": [
                       "Builtin.Correctness",
                       "Builtin.Completeness",
                       "Builtin.Helpfulness",
                       "information_comprehensiveness"
                   ]
               }],
               "evaluatorModelConfig": {
                   "bedrockEvaluatorModels": [{
                       "modelIdentifier": evaluator_model
                   }]
               },
               "customMetricConfig": {
                   "customMetrics": [
                       information_comprehensiveness_metric
                   ],
                   "evaluatorModelConfig": {
                       "bedrockEvaluatorModels": [{
                           "modelIdentifier": custom_metrics_evaluator_model
                       }]
                   }
               }
           }
       }
   )
   
   print(f"Created analysis job: {retrieve_generate_job_name}")
   print(f"Job ID: {retrieve_generate_job['jobArn']}")

4. After submitting the analysis job, you possibly can test its standing utilizing the get_evaluation_job methodology and retrieve the outcomes when the job is full. The output can be saved on the Amazon S3 location specified within the output_path parameter, containing detailed metrics on how your RAG system carried out throughout the analysis dimensions together with {custom} metrics.

Customized metrics are solely accessible for LLM-as-a-judge. On the time of writing, we don’t settle for {custom} AWS Lambda capabilities or endpoints for code-based {custom} metric evaluators. Human-based mannequin analysis has supported {custom} metric definition since its launch in November 2023.

Clear up

To keep away from incurring future costs, delete the S3 bucket, pocket book situations, and different sources that had been deployed as a part of the put up.

Conclusion

The addition of {custom} metrics to Amazon Bedrock Evaluations empowers organizations to outline their very own analysis standards for generative AI methods. By extending the LLM-as-a-judge framework with {custom} metrics, companies can now measure what issues for his or her particular use circumstances alongside built-in metrics. With assist for each numerical and categorical scoring methods, these {custom} metrics allow constant evaluation aligned with organizational requirements and targets.

As generative AI turns into more and more built-in into enterprise processes, the power to guage outputs in opposition to custom-defined standards is crucial for sustaining high quality and driving steady enchancment. We encourage you to discover these new capabilities by the Amazon Bedrock console and API examples supplied, and uncover how customized analysis frameworks can improve your AI methods’ efficiency and enterprise influence.

Concerning the Authors

Shreyas Subramanian is a Principal Information Scientist and helps clients through the use of generative AI and deep studying to unravel their enterprise challenges utilizing AWS companies. Shreyas has a background in large-scale optimization and ML and in using ML and reinforcement studying for accelerating optimization duties.

Adewale Akinfaderin is a Sr. Information Scientist–Generative AI, Amazon Bedrock, the place he contributes to leading edge improvements in foundational fashions and generative AI purposes at AWS. His experience is in reproducible and end-to-end AI/ML strategies, sensible implementations, and serving to international clients formulate and develop scalable options to interdisciplinary issues. He has two graduate levels in physics and a doctorate in engineering.

Jesse Manders is a Senior Product Supervisor on Amazon Bedrock, the AWS Generative AI developer service. He works on the intersection of AI and human interplay with the aim of making and bettering generative AI services and products to fulfill our wants. Beforehand, Jesse held engineering staff management roles at Apple and Lumileds, and was a senior scientist in a Silicon Valley startup. He has an M.S. and Ph.D. from the College of Florida, and an MBA from the College of California, Berkeley, Haas College of Enterprise.

Ishan Singh is a Sr. Generative AI Information Scientist at Amazon Net Companies, the place he helps clients construct modern and accountable generative AI options and merchandise. With a powerful background in AI/ML, Ishan focuses on constructing Generative AI options that drive enterprise worth. Exterior of labor, he enjoys taking part in volleyball, exploring native bike trails, and spending time along with his spouse and canine, Beau.