Introducing new Ray capabilities on SageMaker HyperPod

Distant job submission

For manufacturing workloads, you’ll be able to submit jobs remotely to Ray clusters from Studio, your laptop computer, or steady integration and steady supply (CI/CD) pipelines utilizing the toolkit-for-ray-on-sagemaker-ai Python package. The package deal handles endpoint decision and EKS API credential era by means of IAM authentication, so you utilize customary Ray job submission APIs with a SageMaker-aware deal with resolver:

$ aws eks update-kubeconfig --name <eks-cluster-name> --region <area>
$ pip set up toolkit-for-ray-on-sagemaker-ai
$ ray job submit --address sagemaker_ray://<ray-cluster-name>/<namespace> 
     --working-dir <your-code-directory> 
     --python your-code.py
# To checklist ray jobs
$ ray job checklist --address sagemaker_ray://<ray-cluster-name>/<namespace>

Interactive improvement with SageMaker Areas

Information scientists can connect a Ray cluster to a HyperPod JupyterLab or Code Editor area from Studio. The area joins the cluster as a zero-compute employee node, giving the pocket book full native Ray driver entry. You choose your cluster from a listing throughout area creation and start working instantly.

After making a HyperPod JupyterLab or Code Editor area, a Ray cluster integration possibility seems within the configuration panel.

Choose the Ray cluster you need your workspace to connect with.

Attaching a Ray cluster restarts the area. As soon as it restarts, open JupyterLab or Code Editor in your browser and name ray.init(deal with="auto"). You might be linked to your Ray cluster and might run distributed workloads as when you had been on the pinnacle node. Ray’s runtime_env parameter helps you to inject Python dependencies at runtime with out rebuilding container photos, and you’ll scale staff up or down with out recreating the cluster. For instance, a knowledge scientist coaching a mannequin can begin prototyping in a pocket book with a single employee, then scale to 4 GPU staff by altering one line in ScalingConfig. The coaching runs distributed throughout the connected Ray cluster whereas the pocket book stays interactive for monitoring progress, adjusting hyperparameters, or inspecting intermediate outcomes. Your complete workflow stays throughout the pocket book.

To study extra, see IDEs and Notebooks with Ray.

Out-of-the-box observability

Establishing Ray observability on Kubernetes with Amazon Managed Service for Prometheus and Amazon Managed Grafana is a multi-step course of that entails putting in Helm charts, creating PodMonitors and ServiceMonitors, configuring IAM roles for SigV4 signing, and manually importing dashboard JSON recordsdata.

The HyperPod Observability EKS add-on now handles all of this. It mechanically discovers Ray head and employee pods, scrapes their metrics endpoints, and provisions 4 pre-built Grafana dashboards in Amazon Managed Grafana: Ray Core, Ray Information, Ray Prepare, and Ray Serve. You don’t want to create PodMonitors, configure scrape targets, or import dashboard JSON recordsdata.

All 4 dashboards are organized below a Ray folder in Amazon Managed Grafana and assist filtering metrics by particular Ray cluster. The Open Grafana motion from the cluster checklist (proven within the first screenshot) takes you on to the metrics to your particular cluster in a single click on. The dashboards seem alongside current HyperPod infrastructure dashboards (GPU, EFA, activity governance), so operations groups see Ray workload metrics and cluster well being in a single place.

Resilient coaching

SageMaker HyperPod offers three layers of resilience for Ray coaching workloads: computerized node restoration when {hardware} fails, hung job detection when coaching stalls, and tiered checkpointing for quick restoration after both occasion.

Automated node restoration

SageMaker HyperPod repeatedly displays node well being and might mechanically change defective nodes so your Ray coaching jobs can run for so long as wanted with out guide restarts. When a node is changed, Ray reschedules employee pods onto the brand new wholesome node. In case your coaching code saves checkpoints periodically and consists of logic to renew from the newest checkpoint, the job picks up the place it left off. This requires no modifications to your current Ray coaching code. You solely must configure a enough variety of retries in your RayJob’s FailureConfig in order that jobs get better mechanically slightly than failing completely on the primary interruption. To study extra, see Automated node restoration with Ray.

Hung job detection

Distributed coaching jobs can dangle with out producing errors. A single pod fails (due to a community partition, storage mount challenge, or {hardware} fault), and each different pod blocks on the subsequent collective operation, ready indefinitely. GPUs keep allotted with reminiscence loaded however produce no helpful compute. As a result of there isn’t any error message or crash, information scientists usually uncover the issue hours later once they test job progress manually. At scale, just a few hours of undetected dangle time throughout dozens of GPUs represents vital wasted price.

SageMaker HyperPod EKS now features a per-node Job Monitoring Agent that detects these circumstances for Ray Prepare workloads mechanically, with no modifications to your code. The agent displays a number of node-level and job-level indicators to find out when a coaching job has stalled, and notifies the consumer by means of the cluster’s Amazon CloudWatch log group and the Ray Prepare Grafana dashboard provisioned by the HyperPod Observability add-on.

For customized detection guidelines, information scientists can use the toolkit-for-ray-on-sagemaker-ai library to outline log patterns and timeout thresholds. When the configured motion is cancel, HyperPod terminates the hung employee course of and Ray Prepare’s built-in FailureConfig restarts staff from the final checkpoint. To study extra, see HyperPod Hung Job Detection on Ray.

Tiered checkpointing

Quick checkpoint restoration is crucial for each node alternative and hung job restart situations. The amzn-sagemaker-checkpointing library integrates with HyperPod managed tiered checkpointing, which writes checkpoints to native disk and asynchronously uploads them to Amazon Easy Storage Service (Amazon S3). When a job restarts, the library checks HyperPod Tiered Storage first. If the checkpoint continues to be out there there, restoration is quicker than restoring from Amazon S3. For giant fashions, this will scale back restoration time in comparison with restoring straight from Amazon S3. To study extra, see HyperPod Tiered Storage on Ray.

Accelerated inference

Ray Serve is Ray’s framework for deploying ML fashions as scalable, production-ready endpoints. It helps multi-model composition, autoscaling, and works with serving engines like vLLM. SageMaker HyperPod helps Ray Serve workloads on EKS, so you’ll be able to deploy and scale inference endpoints in your HyperPod cluster.

SageMaker JumpStart offers a catalog of pre-trained fashions. With this launch, the toolkit-for-ray-on-sagemaker-ai library now features a JumpStart mannequin loader that downloads mannequin weights from the JumpStart catalog and deploys them straight into Ray Serve on HyperPod with out guide weight obtain, mannequin configuration, or container setup.

One widespread problem with giant language mannequin (LLM) serving is that inference latency grows with context size. Every new token requires recalculating consideration over all earlier tokens, making lengthy paperwork and multi-turn conversations sluggish and costly. SageMaker HyperPod addresses this with Managed Tiered KV Cache. The tiered cache shops consideration key-value vectors in CPU reminiscence on every node (L1) and on HyperPod Tiered Storage for cross-instance sharing (L2). Your Ray Serve deployments can make the most of the Tiered Storage KV caching capabilities in SageMaker HyperPod with minimal code modifications, decreasing time-to-first-token for multi-turn conversations and long-document workloads. To study extra, see Accelerated Ray Inference on SageMaker HyperPod.

Clear up

To liberate compute capability in your cluster, delete any Ray clusters you created throughout this walkthrough. From the SageMaker Studio Duties tab, choose your Ray cluster and select Delete from the Actions menu. For those who created a HyperPod cluster, you’ll be able to delete the cluster from the Amazon SageMaker AI console to cease incurring costs for the underlying compute cases. For those who put in any prerequisite add-ons for testing, uninstall them from EKS console to liberate compute capability.

Conclusion

On this publish, we walked by means of how Amazon SageMaker HyperPod now offers an entire Ray expertise on EKS, from cluster creation and interactive notebooks to resilient coaching and accelerated inference. All of this works with open-source KubeRay and customary Ray APIs, so current scripts run with out modification. We’d prefer to thank Dhawal Parkar, Pradeep Cruz, Mark Vinciguerra, and Giuseppe Angelo Porcelli for his or her contributions to this publish.

This integration is out there at this time in all AWS Areas the place SageMaker HyperPod EKS is supported. To get began, see the Amazon SageMaker HyperPod documentation and the Ray on HyperPod getting began information. You can too discover the complete workflow on this interactive demo. For those who’d like to debate how Ray on HyperPod can assist your workloads, contact an AWS consultant.

Concerning the authors