Streamline machine studying workflows utilizing Skypilot on Amazon Sagemaker HyperPod

This submit was co-authored with Zhanghao Wu, co-creator of Skypilot.

Speedy advances in generator AI and primary fashions (FMS) have considerably elevated the computational useful resource necessities for machine studying (ML) workloads. Trendy ML pipelines require environment friendly techniques to distribute workloads throughout accelerated computational sources, guaranteeing builders are extremely productive. Organizations want infrastructure options that aren’t solely highly effective, but additionally versatile, resilient, and simple to handle.

Skypilot An open supply framework that simplifies the execution of ML workloads by offering an built-in layer of abstraction that helps ML engineers run their workloads on quite a lot of computing sources with out managing the complexity of the underlying infrastructure. It supplies a easy, high-level interface for useful resource provisioning, job scheduling, and managing distributed coaching throughout a number of nodes.

Amazon Sagemaker HyperPod is a devoted infrastructure for growing and deploying large-scale FMSs. Sagemaker HyperPod not solely presents the flexibleness to create and use your individual software program stack, but additionally supplies optimum efficiency via the location and built-in resilience of the identical backbone occasion. The resilience of the Sagemaker HyperPod with the effectivity of Skypilot supplies a strong framework to broaden generative AI workloads.

On this submit, we share how Sagemaker HyperPod works with Skypilot to streamline AI growth workflows. This integration makes the delicate GPU infrastructure extra accessible to ML engineers, rising productiveness and useful resource utilization.

Challenges for tuning machine studying workloads

Kubernetes Its scalability and in depth open supply instruments make it widespread in ML workloads. Organized into the Amazon Elastic Kubernetes Service (Amazon EKS), Sagemaker HyperPod combines the facility of Kubernetes with the resilient atmosphere of Sagemaker HyperPods designed for coaching large-scale fashions. Amazon EKS help for SageMaker HyperPod enhances resilience via deep well being checks, automated node restoration and automatic job show capabilities, offering uninterrupted coaching for big, long-term jobs.

ML engineers migrating from conventional VM or on-premises environments typically face a steep studying curve. The complexity of Kubernetes and cluster administration can gradual growth cycles and useful resource utilization, creating necessary challenges.

Moreover, the AI infrastructure crew confronted the problem of balancing the will to offer a user-friendly expertise for ML engineers with the necessity for classy administration instruments. They wanted an answer that might present each excessive ranges of management and ease of use for day by day operations.

Sagemaker HyperPod and Skypilot

To handle these challenges, we partnered with Skypilot to showcase options that use the strengths of each platforms. Sagemaker HyperPod excels in managing underlying computational sources and cases, offering the strong infrastructure wanted to request AI workloads. Skypilot enhances this by offering an intuitive layer for job administration, interactive growth and crew coordination.

By way of this partnership, we are able to present our prospects with the most effective world of each. It combines the highly effective and scalable infrastructure of SageMaker HyperPod with a user-friendly interface that considerably reduces the educational curve of ML engineers. For AI infrastructure groups, this integration supplies subtle administration capabilities, simplifies the expertise of ML engineers and creates a positive state of affairs for all stakeholders.

Skypilot helps AI groups run their workloads on quite a lot of infrastructures with unified high-level interfaces and highly effective administration of sources and jobs. AI engineers can herald AI frameworks and specify useful resource necessities for jobs. Skypilot intelligently schedules workloads with the most effective infrastructure. Discover accessible GPUs, provision GPUs, run jobs, and handle the lifecycle.

Resolution overview

Implementing this resolution is straightforward whether or not you might be utilizing an present Sagemaker HyperPod cluster or configuring a brand new deployment. For present clusters, you’ll be able to join utilizing AWS Command Line Interface (AWS CLI) instructions to replace Kubeconfig to confirm setup. For brand new deployments, it guides you thru the configuration of high-performance community choices equivalent to API server setup, cluster creation, and Elastic Cloth Adapter (EFA).

The next diagram illustrates the answer structure.

The following part exhibits how one can run a Skypilot job for multi-node distributed coaching on a Sagemaker HyperPod. Think about the method of making a Sagemaker HyperPod cluster, putting in Skypilot, making a Skypilot cluster, and deploying a Skypilot coaching job.

Stipulations

The next conditions are required:

• To create an present Sagemaker HyperPod cluster with Amazon eks, Deploy a HyperPod cluster). Within the subsequent part, you should provision a single ml.p5.48xlarge occasion within the code pattern.
• Accessing the AWS CLI kubectl Command line software.
• A Python atmosphere for putting in Skypilot.

Create a Sagemaker HyperPod cluster

You may create an EKS cluster utilizing a single AWS cloud formation stack following the directions in Use CloudFormationconsists of a Digital Personal Cloud (VPC) and storage sources.

You need to use the AWS Administration Console or the AWS CLI to create and handle a Sagemaker HyperPod cluster. In case you are utilizing the AWS CLI, specify the cluster configuration for the JSON file and choose the EKS cluster created from the CloudFormation stack as an orchestrator for the Sagemaker HyperPod cluster. Subsequent, create a cluster employee node NodeRecovery Set to Automated Allow automated node restoration OnStartDeepHealthChecksaddition InstanceStress and InstanceConnectivity To allow deep well being checks. See the next code:

cat > cluster-config.json << EOL
{
    "ClusterName": "hp-cluster",
    "Orchestrator": {
        "Eks": {
            "ClusterArn": "${EKS_CLUSTER_ARN}"
        }
    },
    "InstanceGroups": [
        {
            "InstanceGroupName": "worker-group-1",
            "InstanceType": "ml.p5.48xlarge",
            "InstanceCount": 2,
            "LifeCycleConfig": {
                "SourceS3Uri": "s3://${BUCKET_NAME}",
                "OnCreate": "on_create.sh"
            },
            "ExecutionRole": "${EXECUTION_ROLE}",
            "ThreadsPerCore": 1,
            "OnStartDeepHealthChecks": [
                "InstanceStress",
                "InstanceConnectivity"
            ],
        },
  ....
    ],
    "VpcConfig": {
        "SecurityGroupIds": [
            "$SECURITY_GROUP"
        ],
        "Subnets": [
            "$SUBNET_ID"
        ]
    },
    "ResilienceConfig": {
        "NodeRecovery": "Automated"
    }
}
EOL

You may add InstanceStorageConfigs to offer and mount further Amazon Elastic Block Retailer (Amazon EBS) volumes in your SageMaker HyperPod nodes.

To create a cluster utilizing the Sagemaker HyperPod API, run the next AWS CLI command:

aws sagemaker create-cluster  
--cli-input-json file://cluster-config.json

You at the moment are able to arrange Skypilot in your Sagemaker HyperPod cluster.

Connect with Sagemaker HyperPod EKS cluster

Run from the AWS CLI atmosphere aws eks update-kubeconfig It may be discovered within the command to replace the native Kube configuration file ( ~/.kube/config) Use the required {qualifications} and configuration to hook up with an eks cluster kubectl Command (supplies a particular EKS cluster identify):

aws eks update-kubeconfig --name $EKS_CLUSTER_NAME

You may confirm that you’re linked to your EKS cluster by operating the next command:

kubectl config current-context

Set up Skypilot with Kubernetes help

Use the next code to put in Skypilot with Kubernetes help utilizing PIP:

pip set up skypilot[kubernetes]

It will set up the newest construct of Skypilot, which incorporates the required Kubernetes integrations.

Verify Skypilot’s connection to EKS cluster

Verify if Skypilot can hook up with your Kubernetes cluster.

sky test k8s

The output ought to look much like the next code:

Checking credentials to allow clouds for SkyPilot.
Kubernetes: enabled [compute]

To allow a cloud, observe the hints above and rerun: sky test
If any issues stay, discuss with detailed docs at: https://docs.skypilot.co/en/newest/getting-started/set up.html

🎉 Enabled clouds 🎉
Kubernetes [compute]
Energetic context: arn:aws:eks:us-east-2:XXXXXXXXXXXXX:cluster/sagemaker-hyperpod-eks-cluster

Utilizing SkyPilot API server: http://127.0.0.1:46580

In case you are new to utilizing Skypilot on this Kubernetes cluster, chances are you’ll be prompted to create a GPU label for the node. Run the next code and observe the directions:

python -m sky.utils.kubernetes.gpu_labeler --context <your-eks-context>

This script helps Skypilot determine the accessible GPU sources for every node within the cluster. GPU Labeling Job It might take a number of minutes relying on the variety of GPU sources within the cluster.

Uncover the accessible GPUs in your cluster

To test the accessible GPU sources in your Sagemaker HyperPod cluster, use the next code:

sky show-gpus --cloud k8s

This lists the accessible GPU varieties and their counts. There are two p5.48xlarge cases, every geared up with eight nvidia H100 GPUs.

 Kubernetes GPUs
GPU REQUESTABLE_QTY_PER_NODE TOTAL_GPUS TOTAL_FREE_GPUS
H100 1, 2, 4, 8 16 16

Kubernetes per node accelerator availability
NODE_NAME GPU_NAME TOTAL_GPUS FREE_GPUS
hyperpod-i-00baa178bc31afde3 H100 8 8
hyperpod-i-038beefa954efab84 H100 8 8

Begin an interactive growth atmosphere

Skypilot lets you launch a Skypilot cluster for interactive growth.

sky launch -c dev --gpus H100

This command creates an interactive growth atmosphere (IDE) with a single H100 GPU and synchronizes the native working listing to the cluster. Skypilot handles pod creation, useful resource allocation, and setup in your IDE.

Thought-about sources (1 node):
-------------------------------------------------------------------------------------------------------------------------------------------------------------------
 CLOUD        INSTANCE            vCPUs   Mem(GB)   ACCELERATORS   REGION/ZONE                                                                 COST ($)   CHOSEN   
-------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Kubernetes   2CPU--8GB--H100:1   2       8         H100:1         arn:aws:eks:us-east-2:XXXXXXXXXX:cluster/sagemaker-hyperpod-eks-cluster   0.00          ✔     
------------------------------------------------------------------------------------------------------------------------------------------------------------------
Launching a brand new cluster 'dev'. Proceed? [Y/n]: Y
• Launching on Kubernetes.
Pod is up.
✔ Cluster launched: dev. View logs: sky api logs -1 sky-2025-05-05-15-28-47-523797/provision. log
• Syncing information.
Run instructions not specified or empty.
Helpful Instructions
Cluster identify: dey
To log into the top VM:   ssh dev
To submit a job:           sky exec dev yaml_file
To cease the cluster:       sky cease dev
To teardown the cluster:   sky down dev

As soon as booted you’ll be able to hook up with the IDE.

ssh dev

This supplies an interactive shell to your IDE, permitting you to run code, set up packages, and run ML experiments.

Run the coaching job

Skypilot lets you run distributed coaching jobs on a Sagemaker HyperPod cluster. Beneath is an instance of beginning a distributed coaching job utilizing a YAML configuration file:

First, create a file with the identify prepare.yaml Within the coaching job configuration:

sources:
    accelerators: H100

num_nodes: 1

setup: |
    git clone --depth 1 https://github.com/pytorch/examples || true
    cd examples
    git filter-branch --prune-empty --subdirectory-filter distributed/minGPT-ddp
    # SkyPilot's default picture on AWS/GCP has CUDA 11.6 (Azure 11.5).
    uv venv --python 3.10
    supply .venv/bin/activate
    uv pip set up -r necessities.txt "numpy<2" "torch"

run: |
    cd examples
    supply .venv/bin/activate
    cd mingpt
    export LOGLEVEL=INFO

    MASTER_ADDR=$(echo "$SKYPILOT_NODE_IPS" | head -n1)
    echo "Beginning distributed coaching, head node: $MASTER_ADDR"

    torchrun 
    --nnodes=$SKYPILOT_NUM_NODES 
    --nproc_per_node=$SKYPILOT_NUM_GPUS_PER_NODE 
    --master_addr=$MASTER_ADDR 
    --master_port=8008 
    --node_rank=${SKYPILOT_NODE_RANK} 
    major.py

After that, begin the coaching job.

sky launch -c prepare prepare.yaml

This creates a coaching job on a single P5.48XLARGE node geared up with an 8 H100 NVIDIA GPU. You may monitor the output with the next command:

sky logs prepare

Run a multi-node coaching job with EFA

The Elastic Cloth Adapter (EFA) is the community interface for Amazon Elastic Compute Cloud (Amazon EC2) cases that may run purposes requiring high-level internode communication on AWS by way of customized constructed working system bypass {hardware} interfaces. This permits purposes to speak instantly with community {hardware} whereas bypassing the working system kernel, considerably lowering latency and CPU overhead. This direct {hardware} entry is especially helpful for distributed ML workloads the place internode communication can typically develop into a bottleneck throughout gradient synchronization. Through the use of EFA-enabled cases equivalent to P5.48XLARGE and P6-B200.48XLARGE, knowledge scientists scale coaching jobs throughout a number of nodes, sustaining low latency, excessive bandwidth communications important for environment friendly distributed coaching, finally lowering coaching occasions and enhancing sources for big AI workloads.

The next code snippet exhibits how one can incorporate this right into a Skypilot job:

identify: nccl-test-efa

sources:
  cloud: kubernetes
  accelerators: H100:8
  image_id: docker:public.ecr.aws/hpc-cloud/nccl-tests:newest

num_nodes: 2

envs:
  USE_EFA: "true"

run: |
  if [ "${SKYPILOT_NODE_RANK}" == "0" ]; then
    echo "Head node"

    # Whole variety of processes, NP ought to be the whole variety of GPUs within the cluster
    NP=$(($SKYPILOT_NUM_GPUS_PER_NODE * $SKYPILOT_NUM_NODES))

    # Append :${SKYPILOT_NUM_GPUS_PER_NODE} to every IP as slots
    nodes=""
    for ip in $SKYPILOT_NODE_IPS; do
      nodes="${nodes}${ip}:${SKYPILOT_NUM_GPUS_PER_NODE},"
    accomplished
    nodes=${nodes::-1}
    echo "All nodes: ${nodes}"

    # Set atmosphere variables
    export PATH=$PATH:/usr/native/cuda-12.2/bin:/choose/amazon/efa/bin:/usr/bin
    export LD_LIBRARY_PATH=/usr/native/cuda-12.2/lib64:/choose/amazon/openmpi/lib:/choose/nccl/construct/lib:/choose/amazon/efa/lib:/choose/aws-ofi-nccl/set up/lib:/usr/native/nvidia/lib:$LD_LIBRARY_PATH
    export NCCL_HOME=/choose/nccl
    export CUDA_HOME=/usr/native/cuda-12.2
    export NCCL_DEBUG=INFO
    export NCCL_BUFFSIZE=8388608
    export NCCL_P2P_NET_CHUNKSIZE=524288
    export NCCL_TUNER_PLUGIN=/choose/aws-ofi-nccl/set up/lib/libnccl-ofi-tuner.so

    if [ "${USE_EFA}" == "true" ]; then
      export FI_PROVIDER="efa"
    else
      export FI_PROVIDER=""
    fi

    /choose/amazon/openmpi/bin/mpirun 
      --allow-run-as-root 
      --tag-output 
      -H $nodes 
      -np $NP 
      -N $SKYPILOT_NUM_GPUS_PER_NODE 
      --bind-to none 
      -x FI_PROVIDER 
      -x PATH 
      -x LD_LIBRARY_PATH 
      -x NCCL_DEBUG=INFO 
      -x NCCL_BUFFSIZE 
      -x NCCL_P2P_NET_CHUNKSIZE 
      -x NCCL_TUNER_PLUGIN 
      --mca pml ^cm,ucx 
      --mca btl tcp,self 
      --mca btl_tcp_if_exclude lo,docker0,veth_def_agent 
      /choose/nccl-tests/construct/all_reduce_perf 
      -b 8 
      -e 2G 
      -f 2 
      -g 1 
      -c 5 
      -w 5 
      -n 100
  else
    echo "Employee nodes"
  fi

config:
  kubernetes:
    pod_config:
      spec:
        containers:
        - sources:
            limits:
              
              vpc.amazonaws.com/efa: 32
            requests:
              
              vpc.amazonaws.com/efa: 32

cleansing

To delete a Skypilot cluster, run the next command:

sky down <cluster_name>

To delete the Sagemaker HyperPod cluster created on this submit, you should utilize the Sagemaker AI console or one of many following AWS CLI instructions for customers:

aws sagemaker delete-cluster --cluster-name <cluster_name>

It takes a couple of minutes to delete a cluster. After the cluster doesn’t seem within the Sagemaker AI console, you’ll be able to test for profitable deletions.

Should you created a useful resource utilizing the CloudFormation stack, you’ll be able to delete it utilizing the next command:

aws cloudformation delete-stack --stack-name <stack_name>

Conclusion

By combining the strong infrastructure capabilities of the Sagemaker HyperPod with Skypilot’s user-friendly interface, we have launched options that can assist groups concentrate on innovation moderately than infrastructure complexity. This strategy not solely simplifies operations, but additionally enhances productiveness and useful resource utilization throughout organizations of all sizes. To get began, see Skypilot in Amazon EKS SupportAmazon Sagemaker HyperPod Workshop.

In regards to the creator

Roy Arera I’m AWS Senior AI/ML Specialist Resolution Architect. He helps AWS prospects (from small startups to massive firms) and effectively drives and deploys primary fashions on AWS. He’s captivated with computational optimization issues and enhancing efficiency for AI workloads.

ZHANGHAO WU He’s a co-creator of the Skypilot Open Supply Challenge and holds a PhD in Laptop Science from Berkeley, California. He’s dedicated to enhancing the AI expertise for Skypilot Core, client-server structure, managed jobs, and a variety of cloud infrastructures typically.

Anniversary I’m a cell market (GTM) specialist in AWS’ senior basis mannequin. He companions with Prime Generative AI Mannequin Builder, Strategic Clients and AWS Companies crew to allow the following technology of AI/ML workloads on AWS. Ankit’s expertise contains Amazon Alexa’s product administration experience inside the monetary providers business in excessive frequency and low latency buying and selling and enterprise growth.