Open Basis Fashions (FMS) permits organizations to construct custom-made AI functions by tweaking particular domains or duties whereas managing prices and deployments. Nevertheless, as engineers must fastidiously optimize occasion sorts and configure serving parameters via cautious testing, deployment is a major a part of the hassle, and infrequently requires 30% of the challenge time. This course of may be advanced and time-consuming, requires specialised information and iterative testing, and requires attaining the specified efficiency.
Amazon Bedrock Customized Mannequin Import simplifies customized mannequin deployment by offering a easy API for mannequin deployment and invocation. You may add mannequin weights and permit AWS to deal with optimum, totally managed deployments. This ensures that the deployment is operating and is cost-effective. Amazon Bedrock Customized Mannequin imports additionally deal with autoscaling, equivalent to zero scaling. If it isn’t used and there are not any 5 minute calls, scale to zero. You’ll solely pay for something you employ in 5 minutes. It additionally handles scale-up and mechanically will increase the variety of lively mannequin copies when greater concurrency is required. With these options, Amazon Bedrock Customized Mannequin imports into a gorgeous resolution for organizations trying to make use of customized fashions with Amazon Bedrock, offering simplicity and cost-effectiveness.
You will need to use benchmark instruments to evaluate efficiency earlier than deploying these fashions into manufacturing. These instruments may help you actively detect potential manufacturing points equivalent to throttling and be certain that deployment can deal with the anticipated manufacturing load.
This publish launches a weblog collection exploring DeepSeek and Open FMS with Amazon Bedrock customized mannequin imports. We cowl the efficiency benchmarking course of for Amazon Bedrock’s customized fashions utilizing fashionable open supply instruments LLMPERF and LITELLM. Features a Notes This consists of step-by-step directions for deploying deplay deepseek-r1-distill-lama-8b The identical process applies to different fashions which are supported by importing Amazon Bedrock customized fashions.
Conditions
This publish requires an Amazon Bedrock customized mannequin. If you do not have one in your AWS account but, comply with the directions for deploying the Deepseek-R1 distilled llama mannequin with Amazon Bedrock Customized Mannequin import.
Use open supply instruments llmperf and litellm for efficiency benchmarks
Use to carry out efficiency benchmarks llmperfa well-liked open supply library for benchmark basis fashions. LLMPERF simulates load testing of the mannequin’s name API by making a concurrent ray consumer and analyzing the response. An vital benefit of LLMPERF is the broad help of the fundamental mannequin API. This consists of litellmI help it All available models On Amazon’s bedrock.
Arrange a customized mannequin name utilizing Litellm
Litellm is a flexible open supply software that can be utilized as each a Python SDK and a proxy server (AI gateway) to entry over 100 FMSs utilizing standardized codecs. Litellm standardizes inputs to go well with the precise endpoint necessities of every FM supplier. Contains Amazon Bedrock API InvokeModel FMS out there on Amazon Bedrock, together with Converse APIs and imported customized fashions.
To invoke a customized mannequin utilizing Litellm, use the mannequin parameters (see Amazon Bedrock documentation for Litellm). That is the subsequent string bedrock/provider_route/model_arn format.
provider_route Signifies the litellm implementation of the request/response specification to make use of. DeepSeek R1 fashions may be known as utilizing customized chat templates DeepSeek R1 provider routeor use llama chat templates Rama Provider Route.
model_arn The mannequin Amazon useful resource title (ARN) for the import mannequin. You may both retrieve the mannequin ARN of the imported mannequin within the console or ship a ListImportModels request.
For instance, the next script invokes a customized mannequin utilizing a DeepSeek R1 chat template:
import time
from litellm import completion
whereas True:
strive:
response = completion(
mannequin=f"bedrock/deepseek_r1/{model_id}",
messages=[{"role": "user", "content": """Given the following financial data:
- Company A's revenue grew from $10M to $15M in 2023
- Operating costs increased by 20%
- Initial operating costs were $7M
Calculate the company's operating margin for 2023. Please reason step by step."""},
{"role": "assistant", "content": "<think>"}],
max_tokens=4096,
)
print(response['choices'][0]['message']['content'])
break
besides:
time.sleep(60)After the invocation parameters for the imported mannequin have been validated, you’ll be able to configure LLMPERF for the benchmark.
Configure token benchmark exams in llmperf
Use LLMPERF to benchmark efficiency Raydistributed computing framework, simulates practical load. A number of distant shoppers are generated, every of which might ship simultaneous requests. These shoppers are carried out as actor It runs in parallel. llmperf.requests_launcher It manages request distribution throughout Ray shoppers, permitting for the simulation of varied load eventualities and concurrent request patterns. On the similar time, every consumer collects efficiency metrics in the course of the request, equivalent to latency, throughput, and error charges.
Contains two vital metrics for efficiency delay and throughput:
- Latency refers back to the period of time it takes for a single request to be processed.
- Throughput measures the variety of tokens generated per second.
Selecting the best configuration to offer an FMS usually includes carefully monitoring GPU utilization and experimenting with numerous batch sizes, inspecting components equivalent to out there reminiscence, mannequin dimension, and particular workload necessities. For extra info, see Optimizing AI Responsiveness: A Sensible Information to Amazon Bedrock Latency-Optimized Incerence. Whereas Amazon Bedrock Customized Mannequin Import simplifies this by offering a pre-optimized serving configuration, it’s nonetheless vital to test the latency and throughput of your deployment.
Begin with the settings token_benchmark.pya pattern script that makes it simpler to configure benchmark exams. The script can outline the next parameters:
- LLM API: Use Litellm to invoke the Amazon Bedrock customized import mannequin.
- Mannequin: Outline the foundation, API, and mannequin ARN and name it in the identical means as within the earlier part.
- Enter token imply/customary deviation: The parameter used within the likelihood distribution the place the variety of enter tokens is sampled.
- Output token imply/customary deviation: The parameter used within the likelihood distribution the place the variety of output tokens is sampled.
- Variety of concurrent requests: The variety of customers that your utility is more likely to help throughout use.
- Variety of accomplished requests: The overall variety of requests despatched to the LLM API within the take a look at.
The next script exhibits an instance of find out how to invoke a mannequin: look This notebook Step-by-step directions for importing customized fashions and operating benchmark exams.
python3 ${{LLM_PERF_SCRIPT_DIR}}/token_benchmark_ray.py
--model "bedrock/llama/{model_id}"
--mean-input-tokens {mean_input_tokens}
--stddev-input-tokens {stddev_input_tokens}
--mean-output-tokens {mean_output_tokens}
--stddev-output-tokens {stddev_output_tokens}
--max-num-completed-requests ${{LLM_PERF_MAX_REQUESTS}}
--timeout 1800
--num-concurrent-requests ${{LLM_PERF_CONCURRENT}}
--results-dir "${{LLM_PERF_OUTPUT}}"
--llm-api litellm
--additional-sampling-params '{{}}'On the finish of the take a look at, LLMPERF outputs two JSON recordsdata. One has an aggregation metric, and one has a separate entry for every name.
Scaling from zero to chilly begin latency
One factor to recollect is that the Amazon Bedrock customized mannequin import shrinks to zero when the mannequin just isn’t in use, so you need to first make a request to verify there may be at the very least one lively mannequin copy. If an error happens indicating that the mannequin just isn’t prepared, you have to to attend as much as 1 minute for roughly 10 seconds to arrange at the very least one lively mannequin copy. Once you’re prepared, run the take a look at name once more and proceed with the benchmark.
Deepseek-R1-Distill-Lalama-8B instance state of affairs
Contemplate A DeepSeek-R1-Distill-Llama-8B Fashions hosted with Amazon Bedrock Customized Mannequin imports help AI functions with low site visitors with fewer than two simultaneous requests. To account for variability, you’ll be able to alter the token depend parameters for prompts and completion. for instance:
- Variety of shoppers: 2
- Common enter token depend: 500
- Customary deviation enter token depend: 25
- Common output token depend: 1000
- Customary deviation output token depend: 100
- Variety of requests per consumer: 50
This instance take a look at takes about 8 minutes. On the finish of the take a look at, you get a abstract of the outcomes of the combination metric.
inter_token_latency_s
p25 = 0.010615988283217918
p50 = 0.010694698716183695
p75 = 0.010779359342088015
p90 = 0.010945443657517748
p95 = 0.01100556307365132
p99 = 0.011071086908721675
imply = 0.010710014800224604
min = 0.010364670612635254
max = 0.011485444453299149
stddev = 0.0001658793389904756
ttft_s
p25 = 0.3356793452499005
p50 = 0.3783651359990472
p75 = 0.41098671700046907
p90 = 0.46655246950049334
p95 = 0.4846706690498647
p99 = 0.6790834719300077
imply = 0.3837810468001226
min = 0.1878921090010408
max = 0.7590946710006392
stddev = 0.0828713133225014
end_to_end_latency_s
p25 = 9.885957818500174
p50 = 10.561580732000039
p75 = 11.271923759749825
p90 = 11.87688222009965
p95 = 12.139972019549713
p99 = 12.6071144856102
imply = 10.406450886010116
min = 2.6196457750011177
max = 12.626598834998731
stddev = 1.4681851822617253
request_output_throughput_token_per_s
p25 = 104.68609252502657
p50 = 107.24619111072519
p75 = 108.62997591951486
p90 = 110.90675007239598
p95 = 113.3896235445618
p99 = 116.6688412475626
imply = 107.12082450567561
min = 97.0053466021563
max = 129.40680882698936
stddev = 3.9748004356837137
number_input_tokens
p25 = 484.0
p50 = 500.0
p75 = 514.0
p90 = 531.2
p95 = 543.1
p99 = 569.1200000000001
imply = 499.06
min = 433
max = 581
stddev = 26.549294727074212
number_output_tokens
p25 = 1050.75
p50 = 1128.5
p75 = 1214.25
p90 = 1276.1000000000001
p95 = 1323.75
p99 = 1372.2
imply = 1113.51
min = 339
max = 1392
stddev = 160.9598415942952
Quantity Of Errored Requests: 0
General Output Throughput: 208.0008834264341
Quantity Of Accomplished Requests: 100
Accomplished Requests Per Minute: 11.20784995697034
Along with the overview, you’ll obtain metrics for particular person requests that can be utilized to create detailed studies equivalent to the next histogram First token time and Token Throughput.
Analyse efficiency outcomes from LLMPERF and estimate prices utilizing Amazon CloudWatch
LLMPERF supplies the power to benchmark the efficiency of customized fashions supplied by Amazon Bedrock with out inspecting the serving properties and configuration particulars of Amazon Bedrock customized fashions import deployments. This info is efficacious because it represents the anticipated end-user expertise of the appliance.
Moreover, benchmark workout routines can function a precious software for price estimating. With Amazon CloudWatch, you’ll be able to observe the variety of lively mannequin copies that Amazon Bedrock Customized Mannequin Import Scale imports scales in response to load exams. ModelCopy is printed as a CloudWatch metric within the AWS/Bedrock namespace and is reported utilizing the imported mannequin ARN as a label. Plot of ModelCopy The metrics are proven within the diagram beneath. This knowledge may help you estimate prices, as billing is predicated on the variety of lively mannequin copies at a given time.
Conclusion
Whereas importing Amazon Bedrock Customized Fashions simplifies mannequin deployment and scaling, efficiency benchmarks are important for predicting manufacturing efficiency and evaluating fashions throughout key metrics equivalent to price, latency, and throughput.
Please strive it for extra info Notebook example With a customized mannequin.
Further assets:
In regards to the Creator
Felipe Lopez I’m AWS Senior AI/ML Specialist Resolution Architect. Previous to becoming a member of AWS, Felipe labored with GE Digital and SLB to give attention to industrial utility modeling and optimization merchandise.
Rupinder Growal I’m a sophisticated AI/ML specialist resolution architect at AWS. He’s at present specializing in Amazon Sagemaker fashions and serving MLOPs. Previous to this function, he labored as a builder and internet hosting mannequin for a machine studying engineer. Outdoors of labor, he enjoys tennis and biking on the mountain trails.
Parasmera I am a senior product supervisor at AWS. He focuses on serving to Amazon construct bedrock. In her spare time, she enjoys spending time along with her household and biking across the Bay Space.
Prashant Patel I’m a senior software program growth engineer at AWS Bedrock. He’s enthusiastic about scaling large-scale language fashions for enterprise functions. Earlier than becoming a member of AWS, he labored at IBM to provide large-scale AI/ML workloads with Kubernetes. Prashant holds a Grasp’s diploma from the NYU Tandon College of Engineering. Whereas not working, he enjoys touring and enjoying together with his canine.
