Speed up customized LLM deployments: Superb-tune with Oumi and deploy to Amazon Bedrock

This publish was co-authored by Oumi’s David Stewart and Matthew individuals.

Superb-tuning open supply large-scale language fashions (LLMs) is usually caught between experimentation and manufacturing. Coaching configuration, artifact administration, and scalable deployment every require totally different instruments, creating friction when shifting from fast experimentation to a safe, enterprise-grade atmosphere.

This publish exhibits methods to fine-tune your Llama mannequin utilizing: Omi On Amazon EC2 (with the choice to create artificial information utilizing Oumi), retailer artifacts in Amazon S3 and deploy to Amazon Bedrock utilizing customized mannequin import for managed inference. This walkthrough makes use of EC2, however you may full the fine-tuning with different compute companies, reminiscent of Amazon SageMaker or Amazon Elastic Kubernetes Service, relying in your wants.

Advantages of Oumi and Amazon Bedrock

Oumi is an open supply system that streamlines the lifecycle of foundational fashions, from information preparation to coaching and analysis. As a substitute of assembling separate instruments for every stage, outline a single configuration and reuse it throughout a number of runs.

Key advantages of this workflow:

• Recipe-driven coaching: Configurations will be outlined as soon as and reused throughout experiments, lowering boilerplate and rising reproducibility.
• Versatile fine-tuning: Select full fine-tuning or parameter-efficient strategies. LoRAbased mostly on the constraints
• Built-in analysis: Rating checkpoints utilizing benchmarks or LLM-as-a-judge with out extra instruments
• Information synthesis: Generate task-specific datasets when manufacturing information is proscribed

Amazon Bedrock enhances this by providing managed serverless inference. After fine-tuning in Oumi, import the mannequin through customized mannequin import in three steps: add to S3, create an import job, and name. There isn’t a inference infrastructure to handle. The next structure diagram exhibits how these elements work collectively.

Answer overview

This workflow consists of three phases.

1. Superb adjustment Omi For EC2: Launch a GPU-optimized occasion (reminiscent of g5.12xlarge or p4d.24xlarge). Install OumiClick on to run the coaching utilizing your settings. For giant fashions, Oumi helps distributed coaching utilizing absolutely sharded information parallel (FSDP), DeepSpeed, and distributed information parallel (DDP) methods throughout multi-GPU or multi-node setups.
2. Retailer artifacts in S3. Add mannequin weights, checkpoints, and logs for sturdy storage.
3. Deploy to Amazon Bedrock. Create a customized mannequin import job that factors to your S3 artifacts. Amazon Bedrock robotically provisions your inference infrastructure. The shopper software calls the imported mannequin utilizing the Amazon Bedrock Runtime API.

This structure addresses frequent challenges when shifting fine-tuned fashions to manufacturing.

technical implementation

Let’s check out the precise workflow utilizing . metal-llama/Llama-3.2-1B-Instruct Mannequin for instance. We selected this mannequin as a result of it lends itself properly to fine-tuning on AWS. g6.12xlarge On EC2 situations, you may replicate the identical workflow throughout many different open supply fashions (notice that bigger fashions could require bigger situations or distributed coaching throughout situations). For extra data, see Oumi model fine tuning recipe Amazon Bedrock Customized Mannequin Structure.

Conditions

To finish this tutorial you have to:

Organising AWS assets

1. Clone this repository to your native machine.

git clone https://github.com/aws-samples/sample-oumi-fine-tuning-bedrock-cmi.git
cd sample-oumi-fine-tuning-bedrock-cmi

2. Run the setup script to create an IAM position, S3 bucket, and launch a GPU-optimized EC2 occasion.

./scripts/setup-aws-env.sh [--dry-run]

The script prompts on your AWS Area, S3 bucket title, EC2 key pair title, safety group ID, and creates all vital assets. Default: g6.12xlarge occasion, Deep Studying Base AMI (Amazon Linux 2023) with single CUDA, and 100 GB gp3 storage. Observe: If you do not have permissions to create an IAM position or launch an EC2 occasion, share this repository together with your IT administrator and ask them to finish this part to arrange your AWS atmosphere.

3. As soon as the occasion is operating, the script outputs the SSH command and the Amazon Bedrock import position ARN (required in step 5). SSH into your occasion and proceed to step 1 beneath.

Please check with iam/README.md Be taught extra about IAM insurance policies, scoping steerage, and validation steps.

Step 1: Arrange your EC2 atmosphere

To arrange your EC2 atmosphere, comply with these steps:

1. In your EC2 occasion (Amazon Linux 2023), replace the system and set up the bottom dependencies.

sudo yum replace -y
sudo yum set up python3 python3-pip git -y

2. Clone the companion repository.

git clone https://github.com/aws-samples/sample-oumi-fine-tuning-bedrock-cmi.git
cd sample-oumi-fine-tuning-bedrock-cmi

3. Configure the atmosphere variables (substitute the values ​​together with your precise area and bucket title within the setup script).

export AWS_REGION=us-west-2
export S3_BUCKET=your-bucket-name 
export S3_PREFIX=your-s3-prefix 
aws configure set default.area "$AWS_REGION"

4. Run the setup script to create a Python digital atmosphere, set up Oumi, confirm GPU availability, and configure Hugging Face authentication. look setup environment.sh For choices.

./scripts/setup-environment.sh
supply .venv/bin/activate

5. Authenticate with Hugging Face to entry gated mannequin weights. Generate an entry token with . hugface.co/settings/tokensthen run:

hf auth login

Step 2: Configure your coaching

The default dataset is: Tatsurabo/AlpacaIt consists of configs/oumi-config.yaml. Oumi will obtain robotically throughout coaching. No have to obtain manually. To make use of a distinct dataset, dataset_name of parameters configs/oumi-config.yaml. Please check with Oumi dataset documentation For supported codecs.

[Optional] Generate artificial coaching information utilizing Oumi.

To generate artificial information utilizing Amazon Bedrock as an inference backend, model_name placeholder for configs/synthesis-config.yaml Use an Amazon Bedrock mannequin ID that you’ve got entry to, e.g. anthropic.claude-sonnet-4-6). look Oumi data synthesis documentation For extra data. Then run:

oumi synth -c configs/synthesis-config.yaml

Step 3: Superb-tune the mannequin

Superb-tune your mannequin utilizing Oumi’s built-in coaching recipe For Rama-3.2-1B – Directions:

./scripts/fine-tune.sh --config configs/oumi-config.yaml --output-dir fashions/remaining [--dry-run]

To customise hyperparameters, edit oumi-config.yaml.

Observe: Should you generated artificial information in step 2, replace the dataset path in your configuration earlier than coaching.

Monitor GPU utilization nvidia-smi or Amazon CloudWatch agent. For long-running jobs, configure Amazon EC2 Automated Occasion Restoration to deal with occasion interruptions.

Step 4: Consider the mannequin (non-obligatory)

Commonplace benchmarks can be utilized to judge fine-tuned fashions.

oumi consider -c configs/evaluation-config.yaml

The analysis configuration specifies the mannequin path and benchmark duties (reminiscent of MMLU). Edit to customise evaluation configuration.yaml. See Oumi’s documentation for the LLM-as-a-judge strategy and extra benchmarks. Evaluation guide.

Step 5: Deploy to Amazon Bedrock

To deploy your mannequin to Amazon Bedrock, comply with these steps:

1. Add the mannequin artifacts to S3 and import the mannequin to Amazon Bedrock.

./scripts/upload-to-s3.sh --bucket $S3_BUCKET --source fashions/remaining --prefix $S3_PREFIX
./scripts/import-to-bedrock.sh --model-name my-fine-tuned-llama --s3-uri s3://$S3_BUCKET/$S3_PREFIX --role-arn $BEDROCK_ROLE_ARN --wait

2. The import script outputs the mannequin ARN upon completion. set MODEL_ARN This worth (format: arn:aws:bedrock:<REGION>:<ACCOUNT_ID>:imported-model/<MODEL_ID>).
3. Invoke a mannequin in Amazon Bedrock

./scripts/invoke-model.sh --model-id $MODEL_ARN --prompt "Translate this textual content to French: What's the capital of France?"

4. Amazon Bedrock robotically creates a managed inference atmosphere for you. For details about configuring IAM roles, see . bedrock-import-role.json.
5. To help mannequin revision rollbacks, allow S3 versioning in your bucket. For details about SSE-KMS encryption and bucket coverage hardening, see security script It is within the companion repository.

Step 6: Clear up

To keep away from ongoing prices, delete the assets created throughout this tutorial.

aws ec2 terminate-instances --instance-ids $INSTANCE_ID
aws s3 rm s3://$S3_BUCKET/$S3_PREFIX/ --recursive
aws bedrock delete-imported-model --model-identifier $MODEL_ARN

conclusion

On this publish, you realized methods to fine-tune the Llama-3.2-1B-Instruct base mannequin utilizing Oumi on EC2 and deploy it utilizing Amazon Bedrock Customized Mannequin Import. This strategy lets you use managed inference with Amazon Bedrock whereas nonetheless having full management over fine-tuning with your personal information.

companion sample-oumi-fine-tuning-bedrock-cmi The repository supplies scripts, configuration, and IAM insurance policies to get began. Deploy your customized mannequin to Amazon Bedrock by cloning and exchanging datasets.

Get began by reviewing the assets beneath to get began constructing your personal tweak-to-deployment pipeline on Oumi and AWS. Joyful constructing!

study extra

perceive

We wish to thank Pronoy Chopra and Jon Turdiev for his or her contributions.

---

In regards to the creator