Fast ML experimentation for enterprises with Amazon SageMaker AI and Comet

This publish was written with Sarah Ostermeier from Comet.

As enterprise organizations scale their machine studying (ML) initiatives from proof of idea to manufacturing, the complexity of managing experiments, monitoring mannequin lineage, and managing reproducibility grows exponentially. That is primarily as a result of information scientists and ML engineers continuously discover totally different combos of hyperparameters, mannequin architectures, and dataset variations, producing huge quantities of metadata that have to be tracked for reproducibility and compliance. Because the ML mannequin improvement scales throughout a number of groups and regulatory necessities intensify, monitoring experiments turns into much more complicated. With growing AI laws, particularly in the EU, organizations now require detailed audit trails of mannequin coaching information, efficiency expectations, and improvement processes, making experiment monitoring a enterprise necessity and never only a finest follow.

Amazon SageMaker AI gives the managed infrastructure enterprises have to scale ML workloads, dealing with compute provisioning, distributed coaching, and deployment with out infrastructure overhead. Nonetheless, groups nonetheless want sturdy experiment monitoring, mannequin comparability, and collaboration capabilities that transcend primary logging.

Comet is a complete ML experiment administration platform that routinely tracks, compares, and optimizes ML experiments throughout all the mannequin lifecycle. It gives information scientists and ML engineers with highly effective instruments for experiment monitoring, mannequin monitoring, hyperparameter optimization, and collaborative mannequin improvement. It additionally gives Opik, Comet’s open supply platform for LLM observability and improvement.

Comet is offered in SageMaker AI as a Companion AI App, as a totally managed experiment administration functionality, with enterprise-grade safety, seamless workflow integration, and an easy procurement course of via AWS Market.

The mixture addresses the wants of an enterprise ML workflow end-to-end, the place SageMaker AI handles infrastructure and compute, and Comet gives the experiment administration, mannequin registry, and manufacturing monitoring capabilities that groups require for regulatory compliance and operational effectivity. On this publish, we display an entire fraud detection workflow utilizing SageMaker AI with Comet, showcasing reproducibility and audit-ready logging wanted by enterprises at present.

Enterprise-ready Comet on SageMaker AI

Earlier than continuing to setup directions, organizations should determine their working mannequin and based mostly on that, resolve how Comet goes to be arrange. We suggest implementing Comet utilizing a federated working mannequin. On this structure, Comet is centrally managed and hosted in a shared companies account, and every information science crew maintains absolutely autonomous environments. Every working mannequin comes with their very own units of advantages and limitations. For extra data, check with SageMaker Studio Administration Finest Practices.

Let’s dive into the setup of Comet in SageMaker AI. Massive enterprise typically have the next personas:

• Directors – Answerable for establishing the widespread infrastructure companies and surroundings to be used case groups
• Customers – ML practitioners from use case groups who use the environments arrange by platform crew to resolve their enterprise issues

Within the following sections, we undergo every persona’s journey.

Comet works properly with each SageMaker AI and Amazon SageMaker. SageMaker AI gives the Amazon SageMaker Studio built-in improvement surroundings (IDE), and SageMaker gives the Amazon SageMaker Unified Studio IDE. For this publish, we use SageMaker Studio.

Administrator journey

On this situation, the administrator receives a request from a crew engaged on a fraud detection use case to provision an ML surroundings with a totally managed coaching and experimentation setup. The administrator’s journey contains the next steps:

1. Comply with the stipulations to arrange Companion AI Apps. This units up permissions for directors, permitting Comet to imagine a SageMaker AI execution position on behalf of the customers and extra privileges for managing the Comet subscription via AWS Market.
2. On the SageMaker AI console, below Functions and IDEs within the navigation pane, select Companion AI Apps, then select View particulars for Comet.

The small print are proven, together with the contract pricing mannequin for Comet and infrastructure tier estimated prices.

Comet gives totally different subscription choices starting from a 1-month to 36-month contract. With this contract, customers can entry Comet in SageMaker. Primarily based on the variety of customers, the admin can evaluate and analyze the suitable occasion dimension for the Comet dashboard server. Comet helps 5–500 customers operating greater than 100 experiment jobs..

3. Select Go to Market to subscribe to be redirected to the Comet itemizing on AWS Market.
4. Select View buy choices.

5. Within the subscription type, present the required particulars.

When the subscription is full, the admin can begin configuring Comet.

6. Whereas deploying Comet, add the mission lead of the fraud detection use case crew as an admin to handle the admin operations for the Comet dashboard.

It takes a couple of minutes for the Comet server to be deployed. For extra particulars on this step, check with Companion AI App provisioning.

7. Arrange a SageMaker AI area following the steps in Use customized setup for Amazon SageMaker AI. As a finest follow, present a pre-signed area URL for the use case crew member to immediately entry the Comet UI with out logging in to the SageMaker console.
8. Add the crew members to this area and allow entry to Comet whereas configuring the area.

Now the SageMaker AI area is prepared for customers to log in to and begin engaged on the fraud detection use case.

Person journey

Now let’s discover the journey of an ML practitioner from the fraud detection use case. The consumer completes the next steps:

1. Log in to the SageMaker AI area via the pre-signed URL.

You can be redirected to the SageMaker Studio IDE. Your consumer title and AWS Id and Entry Administration (IAM) execution position are preconfigured by the admin.

2. Create a JupyterLab Area following the JupyterLab consumer information.
3. You can begin engaged on the fraud detection use case by spinning up a Jupyter pocket book.

The admin has additionally arrange required entry to the information via an Amazon Easy Storage Service (Amazon S3) bucket.

4. To entry Comet APIs, set up the comet_ml library and configure the required surroundings variables as described in Arrange the Amazon SageMaker Companion AI Apps SDKs.
5. To entry the Comet UI, select Companion AI Apps within the SageMaker Studio navigation pane and select Open for Comet.

Now, let’s stroll via the use case implementation.

Answer overview

This use case highlights widespread enterprise challenges: working with imbalanced datasets (on this instance, solely 0.17% of transactions are fraudulent), requiring a number of experiment iterations, and sustaining full reproducibility for regulatory compliance. To observe alongside, check with the Comet documentation and Quickstart guide for added setup and API particulars.

For this use case, we use the Credit Card Fraud Detection dataset. The dataset incorporates bank card transactions with binary labels representing fraudulent (1) or respectable (0) transactions. Within the following sections, we stroll via a few of the vital sections of the implementation. The complete code of the implementation is offered within the GitHub repository.

Stipulations

As a prerequisite, configure the mandatory imports and surroundings variables for the Comet and SageMaker integration:

# Comet ML for experiment monitoring
import comet_ml
from comet_ml import Experiment, API, Artifact
from comet_ml.integration.sagemaker import log_sagemaker_training_job_v1
AWS_PARTNER_APP_AUTH=true
AWS_PARTNER_APP_ARN=<Your_AWS_PARTNER_APP_ARN>
COMET_API_KEY=<Your_Comet_API_Key> 	
# From Particulars Web page, click on Open Comet. Within the prime #proper nook, click on on consumer -> API # Key
# Comet ML configuration
COMET_WORKSPACE = '<your-comet-workspace-name>'
COMET_PROJECT_NAME = '<your-comet-project-name>'

Put together the dataset

One among Comet’s key enterprise options is computerized dataset versioning and lineage monitoring. This functionality gives full auditability of what information was used to coach every mannequin, which is crucial for regulatory compliance and reproducibility. Begin by loading the dataset:

# Create a Comet Artifact to trace our uncooked dataset
dataset_artifact = Artifact(
    title="fraud-dataset",
    artifact_type="dataset",
    aliases=["raw"]
)
# Add the uncooked dataset file to the artifact
dataset_artifact.add_remote(s3_data_path, metadata={
    "dataset_stage": "uncooked", 
    "dataset_split": "not_split", 
    "preprocessing": "none"
})

Begin a Comet experiment

With the dataset artifact created, now you can begin monitoring the ML workflow. Creating a Comet experiment routinely begins capturing code, put in libraries, system metadata, and different contextual data within the background. You may log the dataset artifact created earlier within the experiment. See the next code:

# Create a brand new Comet experiment
experiment_1 = comet_ml.Experiment(
    project_name=COMET_PROJECT_NAME,
    workspace=COMET_WORKSPACE,
)
# Log the dataset artifact to this experiment for lineage monitoring
experiment_1.log_artifact(dataset_artifact)

Preprocess the information

The subsequent steps are customary preprocessing steps, together with eradicating duplicates, dropping unneeded columns, splitting into prepare/validation/take a look at units, and standardizing options utilizing scikit-learn’s StandardScaler. We wrap the processing code in preprocess.py and run it as a SageMaker Processing job. See the next code:

# Run SageMaker processing job
processor = SKLearnProcessor(
    framework_version='1.0-1',
    position=sagemaker.get_execution_role(),
    instance_count=1,
    instance_type="ml.t3.medium"
)
processor.run(
    code="preprocess.py",
    inputs=[ProcessingInput(source=s3_data_path, destination='/opt/ml/processing/input')],
    outputs=[ProcessingOutput(source="/opt/ml/processing/output", destination=f's3://{bucket_name}/{processed_data_prefix}')]
)

After you submit the processing job, SageMaker AI launches the compute cases, processes and analyzes the enter information, and releases the sources upon completion. The output of the processing job is saved within the S3 bucket specified.

Subsequent, create a brand new model of the dataset artifact to trace the processed information. Comet routinely variations artifacts with the identical title, sustaining full lineage from uncooked to processed information.

# Create an up to date model of the 'fraud-dataset' Artifact for the preprocessed information
preprocessed_dataset_artifact = Artifact(
    title="fraud-dataset",
    artifact_type="dataset", 
    aliases=["preprocessed"],
    metadata={
        "description": "Bank card fraud detection dataset",
        "fraud_percentage": f"{fraud_percentage:.3f}%",
        "dataset_stage": "preprocessed",
        "preprocessing": "StandardScaler + prepare/val/take a look at cut up",
    }
)
# Add our prepare, validation, and take a look at dataset recordsdata as distant property 
preprocessed_dataset_artifact.add_remote(
    uri=f's3://{bucket_name}/{processed_data_prefix}',
    logical_path="split_data"
)
# Log the up to date dataset to the experiment to trace the updates
experiment_1.log_artifact(preprocessed_dataset_artifact)

The Comet and SageMaker AI experiment workflow

Information scientists favor fast experimentation; due to this fact, we organized the workflow into reusable utility features that may be known as a number of occasions with totally different hyperparameters whereas sustaining constant logging and analysis throughout all runs. On this part, we showcase the utility features together with a quick snippet of the code contained in the operate:

    # Create SageMaker estimator
    estimator = Estimator(
        image_uri=xgboost_image,
        position=execution_role,
        instance_count=1,
        instance_type="ml.m5.giant",
        output_path=model_output_path,
        sagemaker_session=sagemaker_session_obj,
        hyperparameters=hyperparameters_dict,
        max_run=1800  # Most coaching time in seconds
    )
    # Begin coaching
    estimator.match({
        'prepare': train_channel,
        'validation': val_channel
    })

• log_training_job() – Captures the coaching metadata and metrics and hyperlinks the model asset to the experiment for full traceability:

# Log SageMaker coaching job to Comet 
    log_sagemaker_training_job_v1(
        estimator=training_estimator,
        experiment=api_experiment
    )

• log_model_to_comet() – Hyperlinks mannequin artifacts to Comet, captures the coaching metadata, and hyperlinks the model asset to the experiment for full traceability:

experiment.log_remote_model(
        model_name=model_name,
        uri=model_artifact_path,
        metadata=metadata
    )

• deploy_and_evaluate_model() – Performs mannequin deployment and analysis, and metric logging:

# Deploy to endpoint
predictor = estimator.deploy(
initial_instance_count=1,
       instance_type="ml.m5.xlarge")
# Log metrics and visualizations to Comet 
experiment.log_metrics(metrics) experiment.log_confusion_matrix(matrix=cm,labels=['Normal', 'Fraud']) 
# Log ROC curve 
fpr, tpr, _ = roc_curve(y_test, y_pred_prob_as_np_array) experiment.log_curve("roc_curve", x=fpr, y=tpr)

The whole prediction and analysis code is offered within the GitHub repository.

Run the experiments

Now you’ll be able to run a number of experiments by calling the utility features with totally different configurations and evaluate experiments to search out essentially the most optimum settings for the fraud detection use case.

For the primary experiment, we set up a baseline utilizing customary XGBoost hyperparameters:

# Outline hyperparameters for first experiment
hyperparameters_v1 = {
    'goal': 'binary:logistic',	# Binary classification
    'num_round': 100,                   # Variety of boosting rounds
    'eval_metric': 'auc',               # Analysis metric
    'learning_rate': 0.15,              # Studying price
    'booster': 'gbtree'                 # Booster algorithm
}
# Prepare the mannequin
estimator_1 = prepare(
    model_output_path=f"s3://{bucket_name}/{model_output_prefix}/1",
    execution_role=position,
    sagemaker_session_obj=sagemaker_session,
    hyperparameters_dict=hyperparameters_v1,
    train_channel_loc=train_channel_location,
    val_channel_loc=validation_channel_location
)
# log the coaching job and mannequin artifact
log_training_job(experiment_key = experiment_1.get_key(), training_estimator=estimator_1)
log_model_to_comet(experiment = experiment_1,
                   model_name="fraud-detection-xgb-v1", 
                   model_artifact_path=estimator_1.model_data, 
                   metadata=metadata)
# Deploy and consider
deploy_and_evaluate_model(experiment=experiment_1,
                          estimator=estimator_1,
                          X_test_scaled=X_test_scaled,
                          y_test=y_test
                          )

Whereas operating a Comet experiment from a Jupyter pocket book, we have to finish the experiment to ensure all the pieces is captured and continued within the Comet server. See the next code: experiment_1.finish()

When the baseline experiment is full, you’ll be able to run extra experiments with totally different hyperparameters. Take a look at the notebook to see the small print of each experiments.

When the second experiment is full, navigate to the Comet UI to match these two experiment runs.

View Comet experiments within the UI

To entry the UI, you’ll be able to find the URL within the SageMaker Studio IDE or by executing the code supplied within the pocket book: experiment_2.url

The next screenshot reveals the Comet experiments UI. The experiment particulars are for illustration functions solely and don’t signify a real-world fraud detection experiment.

This concludes the fraud detection experiment.

Clear up

For the experimentation half, SageMaker processing and coaching infrastructure is ephemeral in nature and shuts down routinely when the job is full. Nonetheless, you should nonetheless manually clear up a couple of sources to keep away from pointless prices:

1. Shut down the SageMaker JupyterLab Area after use. For directions, check with Idle shutdown.
2. The Comet subscription renews based mostly on the contract chosen. Cancel the contract when there is no such thing as a additional requirement to resume the Comet subscription.

Benefits of SageMaker and Comet integration

Having demonstrated the technical workflow, let’s study the broader benefits this integration gives.

Streamlined mannequin improvement

The Comet and SageMaker mixture reduces the handbook overhead of operating ML experiments. Whereas SageMaker handles infrastructure provisioning and scaling, Comet’s computerized logging captures hyperparameters, metrics, code, put in libraries, and system efficiency out of your coaching jobs with out extra configuration. This helps groups give attention to mannequin improvement reasonably than experiment bookkeeping.Comet’s visualization capabilities lengthen past primary metric plots. Constructed-in charts allow fast experiment comparability, and customized Python panels assist domain-specific evaluation instruments for debugging mannequin conduct, optimizing hyperparameters, or creating specialised visualizations that customary instruments can’t present.

Enterprise collaboration and governance

For enterprise groups, the mixture creates a mature platform for scaling ML tasks throughout regulated environments. SageMaker gives constant, safe ML environments, and Comet allows seamless collaboration with full artifact and mannequin lineage monitoring. This helps keep away from pricey errors that happen when groups can’t recreate earlier outcomes.

Full ML lifecycle integration

Not like level options that solely tackle coaching or monitoring, Comet paired with SageMaker helps your full ML lifecycle. Fashions could be registered in Comet’s mannequin registry with full model monitoring and governance. SageMaker handles mannequin deployment, and Comet maintains the lineage and approval workflows for mannequin promotion. Comet’s manufacturing monitoring capabilities monitor mannequin efficiency and information drift after deployment, making a closed loop the place manufacturing insights inform your subsequent spherical of SageMaker experiments.

Conclusion

On this publish, we confirmed methods to use SageMaker and Comet collectively to spin up absolutely managed ML environments with reproducibility and experiment monitoring capabilities.

To reinforce your SageMaker workflows with complete experiment administration, deploy Comet immediately in your SageMaker surroundings via the AWS Market, and share your suggestions within the feedback.

For extra details about the companies and options mentioned on this publish, check with the next sources:

In regards to the authors

Vikesh Pandey is a Principal GenAI/ML Specialist Options Architect at AWS, serving to giant monetary establishments undertake and scale generative AI and ML workloads. He’s the creator of guide “Generative AI for monetary companies.” He carries greater than 15 years of expertise constructing enterprise-grade functions on generative AI/ML and associated applied sciences. In his spare time, he performs an unnamed sport together with his son that lies someplace between soccer and rugby.

Naufal Mir is a Senior GenAI/ML Specialist Options Architect at AWS. He focuses on serving to clients construct, prepare, deploy and migrate machine studying workloads to SageMaker. He beforehand labored at monetary companies institutes creating and working methods at scale. Exterior of labor, he enjoys extremely endurance operating and biking.

Sarah Ostermeier is a Technical Product Advertising Supervisor at Comet. She focuses on bringing Comet’s GenAI and ML developer merchandise to the engineers who want them via technical content material, instructional sources, and product messaging. She has beforehand labored as an ML engineer, information scientist, and buyer success supervisor, serving to clients implement and scale AI options. Exterior of labor she enjoys touring off the overwhelmed path, writing about AI, and studying science fiction.