Inference Llama 2 fashions with real-time response streaming utilizing Amazon SageMaker

With the speedy adoption of generative AI purposes, there’s a want for these purposes to reply in time to cut back the perceived latency with increased throughput. Basis fashions (FMs) are sometimes pre-trained on huge corpora of knowledge with parameters ranging in scale of hundreds of thousands to billions and past. Giant language fashions (LLMs) are a kind of FM that generate textual content as a response of the consumer inference. Inferencing these fashions with various configurations of inference parameters could result in inconsistent latencies. The inconsistency may very well be due to the various variety of response tokens you expect from the mannequin or the kind of accelerator the mannequin is deployed on.

In both case, somewhat than ready for the complete response, you possibly can undertake the strategy of response streaming on your inferences, which sends again chunks of data as quickly as they’re generated. This creates an interactive expertise by permitting you to see partial responses streamed in actual time as a substitute of a delayed full response.

With the official announcement that Amazon SageMaker real-time inference now helps response streaming, now you can repeatedly stream inference responses again to the shopper when utilizing Amazon SageMaker real-time inference with response streaming. This resolution will provide help to construct interactive experiences for varied generative AI purposes equivalent to chatbots, digital assistants, and music turbines. This put up exhibits you tips on how to notice sooner response occasions within the type of Time to First Byte (TTFB) and scale back the general perceived latency whereas inferencing Llama 2 fashions.

To implement the answer, we use SageMaker, a totally managed service to arrange knowledge and construct, practice, and deploy machine studying (ML) fashions for any use case with totally managed infrastructure, instruments, and workflows. For extra details about the varied deployment choices SageMaker offers, discuss with Amazon SageMaker Mannequin Internet hosting FAQs. Let’s perceive how we are able to deal with the latency points utilizing real-time inference with response streaming.

Resolution overview

As a result of we need to deal with the aforementioned latencies related to real-time inference with LLMs, let’s first perceive how we are able to use the response streaming assist for real-time inferencing for Llama 2. Nevertheless, any LLM can benefit from response streaming assist with real-time inferencing.

Llama 2 is a set of pre-trained and fine-tuned generative textual content fashions ranging in scale from 7 billion to 70 billion parameters. Llama 2 fashions are autoregressive fashions with decoder solely structure. When supplied with a immediate and inference parameters, Llama 2 fashions are able to producing textual content responses. These fashions can be utilized for translation, summarization, query answering, and chat.

For this put up, we deploy the Llama 2 Chat mannequin meta-llama/Llama-2-13b-chat-hf on SageMaker for real-time inferencing with response streaming.

In relation to deploying fashions on SageMaker endpoints, you possibly can containerize the fashions utilizing specialised AWS Deep Studying Container (DLC) photographs obtainable for well-liked open supply libraries. Llama 2 fashions are textual content technology fashions; you need to use both the Hugging Face LLM inference containers on SageMaker powered by Hugging Face Text Generation Inference (TGI) or AWS DLCs for Giant Mannequin Inference (LMI).

On this put up, we deploy the Llama 2 13B Chat mannequin utilizing DLCs on SageMaker Internet hosting for real-time inference powered by G5 cases. G5 cases are a high-performance GPU-based cases for graphics-intensive purposes and ML inference. You may as well use supported occasion varieties p4d, p3, g5, and g4dn with acceptable modifications as per the occasion configuration.

Stipulations

To implement this resolution, you need to have the next:

• An AWS account with an AWS Identification and Entry Administration (IAM) function with permissions to handle sources created as a part of the answer.
• If that is your first time working with Amazon SageMaker Studio, you first must create a SageMaker area.
• A Hugging Face account. Sign up together with your e-mail in case you don’t have already got account.
  • For seamless entry of the fashions obtainable on Hugging Face, particularly gated fashions equivalent to Llama, for fine-tuning and inferencing functions, you need to have a Hugging Face account to acquire a learn entry token. After you join your Hugging Face account, log in to go to https://huggingface.co/settings/tokens to create a learn entry token.
• Entry to Llama 2, utilizing the identical e-mail ID that you simply used to join Hugging Face.
  • The Llama 2 fashions obtainable by way of Hugging Face are gated fashions. The usage of the Llama mannequin is ruled by the Meta license. To obtain the mannequin weights and tokenizer, request access to Llama and settle for their license.
  • After you’re granted entry (sometimes in a few days), you’ll obtain an e-mail affirmation. For this instance, we use the mannequin Llama-2-13b-chat-hf, however you need to have the ability to entry different variants as effectively.

Strategy 1: Hugging Face TGI

On this part, we present you tips on how to deploy the meta-llama/Llama-2-13b-chat-hf mannequin to a SageMaker real-time endpoint with response streaming utilizing Hugging Face TGI. The next desk outlines the specs for this deployment.

Deploy the mannequin

First, you retrieve the bottom picture for the LLM to be deployed. You then construct the mannequin on the bottom picture. Lastly, you deploy the mannequin to the ML occasion for SageMaker Internet hosting for real-time inference.

Let’s observe tips on how to obtain the deployment programmatically. For brevity, solely the code that helps with the deployment steps is mentioned on this part. The complete supply code for deployment is obtainable within the pocket book llama-2-hf-tgi/llama-2-13b-chat-hf/1-deploy-llama-2-13b-chat-hf-tgi-sagemaker.ipynb.

Retrieve the most recent Hugging Face LLM DLC powered by TGI by way of pre-built SageMaker DLCs. You employ this picture to deploy the meta-llama/Llama-2-13b-chat-hf mannequin on SageMaker. See the next code:

from sagemaker.huggingface import get_huggingface_llm_image_uri

# retrieve the llm picture uri
llm_image = get_huggingface_llm_image_uri(
  "huggingface",
  model="1.0.3"
)

Outline the surroundings for the mannequin with the configuration parameters outlined as follows:

instance_type = "ml.g5.12xlarge"
number_of_gpu = 4
config = {
    'HF_MODEL_ID': "meta-llama/Llama-2-13b-chat-hf", # model_id from hf.co/fashions
    'SM_NUM_GPUS': json.dumps(number_of_gpu), # Variety of GPU used per duplicate
    'MAX_INPUT_LENGTH': json.dumps(2048),  # Max size of enter textual content
    'MAX_TOTAL_TOKENS': json.dumps(4096),  # Max size of the technology (together with enter textual content)
    'MAX_BATCH_TOTAL_TOKENS': json.dumps(8192),  # Limits the variety of tokens that may be processed in parallel through the technology
    'HUGGING_FACE_HUB_TOKEN': "<YOUR_HUGGING_FACE_READ_ACCESS_TOKEN>"
}

Change <YOUR_HUGGING_FACE_READ_ACCESS_TOKEN> for the config parameter HUGGING_FACE_HUB_TOKEN with the worth of the token obtained out of your Hugging Face profile as detailed within the conditions part of this put up. Within the configuration, you outline the variety of GPUs used per duplicate of a mannequin as 4 for SM_NUM_GPUS. Then you possibly can deploy the meta-llama/Llama-2-13b-chat-hf mannequin on an ml.g5.12xlarge occasion that comes with 4 GPUs.

Now you possibly can construct the occasion of HuggingFaceModel with the aforementioned surroundings configuration:

llm_model = HuggingFaceModel(
    function=function,
    image_uri=llm_image,
    env=config
)

Lastly, deploy the mannequin by offering arguments to the deploy methodology obtainable on the mannequin with varied parameter values equivalent to endpoint_name, initial_instance_count, and instance_type:

llm = llm_model.deploy(
    endpoint_name=endpoint_name,
    initial_instance_count=1,
    instance_type=instance_type,
    container_startup_health_check_timeout=health_check_timeout,
)

Carry out inference

The Hugging Face TGI DLC comes with the power to stream responses with none customizations or code modifications to the mannequin. You should use invoke_endpoint_with_response_stream if you’re utilizing Boto3 or InvokeEndpointWithResponseStream when programming with the SageMaker Python SDK.

The InvokeEndpointWithResponseStream API of SageMaker permits builders to stream responses again from SageMaker fashions, which may help enhance buyer satisfaction by lowering the perceived latency. That is particularly essential for purposes constructed with generative AI fashions, the place quick processing is extra essential than ready for your complete response.

For this instance, we use Boto3 to deduce the mannequin and use the SageMaker API invoke_endpoint_with_response_stream as follows:

def get_realtime_response_stream(sagemaker_runtime, endpoint_name, payload):
    response_stream = sagemaker_runtime.invoke_endpoint_with_response_stream(
        EndpointName=endpoint_name,
        Physique=json.dumps(payload), 
        ContentType="software/json",
        CustomAttributes="accept_eula=false"
    )
    return response_stream

The argument CustomAttributes is ready to the worth accept_eula=false. The accept_eula parameter should be set to true to efficiently receive the response from the Llama 2 fashions. After the profitable invocation utilizing invoke_endpoint_with_response_stream, the tactic will return a response stream of bytes.

The next diagram illustrates this workflow.

You want an iterator that loops over the stream of bytes and parses them to readable textual content. The LineIterator implementation could be discovered at llama-2-hf-tgi/llama-2-13b-chat-hf/utils/LineIterator.py. Now you’re prepared to arrange the immediate and directions to make use of them as a payload whereas inferencing the mannequin.

Put together a immediate and directions

On this step, you put together the immediate and directions on your LLM. To immediate Llama 2, you need to have the next immediate template:

<s>[INST] <<SYS>>
{{ system_prompt }}
<</SYS>>

{{ user_message }} [/INST]

You construct the immediate template programmatically outlined within the methodology build_llama2_prompt, which aligns with the aforementioned immediate template. You then outline the directions as per the use case. On this case, we’re instructing the mannequin to generate an e-mail for a advertising and marketing marketing campaign as lined within the get_instructions methodology. The code for these strategies is within the llama-2-hf-tgi/llama-2-13b-chat-hf/2-sagemaker-realtime-inference-llama-2-13b-chat-hf-tgi-streaming-response.ipynb pocket book. Construct the instruction mixed with the duty to be carried out as detailed in user_ask_1 as follows:

user_ask_1 = f'''
AnyCompany just lately introduced new service launch named AnyCloud Web Service.
Write a brief e-mail concerning the product launch with Name to motion to Alice Smith, whose e-mail is alice.smith@instance.com
Point out the Coupon Code: EARLYB1RD to get 20% for 1st 3 months.
'''
directions = get_instructions(user_ask_1)
immediate = build_llama2_prompt(directions)

We move the directions to construct the immediate as per the immediate template generated by build_llama2_prompt.

inference_params = {
        "do_sample": True,
        "top_p": 0.6,
        "temperature": 0.9,
        "top_k": 50,
        "max_new_tokens": 512,
        "repetition_penalty": 1.03,
        "cease": ["</s>"],
        "return_full_text": False
    }
payload = {
    "inputs":  immediate,
    "parameters": inference_params,
    "stream": True ## <-- to have response stream.
}

We membership the inference parameters together with immediate with the important thing stream with the worth True to type a ultimate payload. Ship the payload to get_realtime_response_stream, which will likely be used to invoke an endpoint with response streaming:

resp = get_realtime_response_stream(sagemaker_runtime, endpoint_name, payload)
print_response_stream(resp)

The generated textual content from the LLM will likely be streamed to the output as proven within the following animation.

Strategy 2: LMI with DJL Serving

On this part, we display tips on how to deploy the meta-llama/Llama-2-13b-chat-hf mannequin to a SageMaker real-time endpoint with response streaming utilizing LMI with DJL Serving. The next desk outlines the specs for this deployment.

You first obtain the mannequin and retailer it in Amazon Easy Storage Service (Amazon S3). You then specify the S3 URI indicating the S3 prefix of the mannequin within the serving.properties file. Subsequent, you retrieve the bottom picture for the LLM to be deployed. You then construct the mannequin on the bottom picture. Lastly, you deploy the mannequin to the ML occasion for SageMaker Internet hosting for real-time inference.

Let’s observe tips on how to obtain the aforementioned deployment steps programmatically. For brevity, solely the code that helps with the deployment steps is detailed on this part. The complete supply code for this deployment is obtainable within the pocket book llama-2-lmi/llama-2-13b-chat/1-deploy-llama-2-13b-chat-lmi-response-streaming.ipynb.

Obtain the mannequin snapshot from Hugging Face and add the mannequin artifacts on Amazon S3

With the aforementioned conditions, obtain the mannequin on the SageMaker pocket book occasion after which add it to the S3 bucket for additional deployment:

model_name="meta-llama/Llama-2-13b-chat-hf"
# Solely obtain pytorch checkpoint recordsdata
allow_patterns = ["*.json", "*.txt", "*.model", "*.safetensors", "*.bin", "*.chk", "*.pth"]

# Obtain the mannequin snapshot
model_download_path = snapshot_download(
    repo_id=model_name, 
    cache_dir=local_model_path, 
    allow_patterns=allow_patterns, 
    token='<YOUR_HUGGING_FACE_READ_ACCESS_TOKEN>'
)

Observe that though you don’t present a legitimate entry token, the mannequin will obtain. However while you deploy such a mannequin, the mannequin serving gained’t succeed. Subsequently, it’s really useful to exchange <YOUR_HUGGING_FACE_READ_ACCESS_TOKEN> for the argument token with the worth of the token obtained out of your Hugging Face profile as detailed within the conditions. For this put up, we specify the official mannequin’s identify for Llama 2 as recognized on Hugging Face with the worth meta-llama/Llama-2-13b-chat-hf. The uncompressed mannequin will likely be downloaded to local_model_path on account of working the aforementioned code.

Add the recordsdata to Amazon S3 and acquire the URI, which will likely be later utilized in serving.properties.

You may be packaging the meta-llama/Llama-2-13b-chat-hf mannequin on the LMI container picture with DJL Serving utilizing the configuration specified by way of serving.properties. You then deploy the mannequin together with mannequin artifacts packaged on the container picture on the SageMaker ML occasion ml.g5.12xlarge. You then use this ML occasion for SageMaker Internet hosting for real-time inferencing.

Put together mannequin artifacts for DJL Serving

Put together your mannequin artifacts by making a serving.properties configuration file:

%%writefile chat_llama2_13b_hf/serving.properties
engine = MPI
possibility.entryPoint=djl_python.huggingface
possibility.tensor_parallel_degree=4
possibility.low_cpu_mem_usage=TRUE
possibility.rolling_batch=lmi-dist
possibility.max_rolling_batch_size=64
possibility.model_loading_timeout=900
possibility.model_id={{model_id}}
possibility.paged_attention=true

We use the next settings on this configuration file:

• engine – This specifies the runtime engine for DJL to make use of. The attainable values embrace Python, DeepSpeed, FasterTransformer, and MPI. On this case, we set it to MPI. Mannequin Parallelization and Inference (MPI) facilitates partitioning the mannequin throughout all of the obtainable GPUs and due to this fact accelerates inference.
• possibility.entryPoint – This feature specifies which handler supplied by DJL Serving you want to use. The attainable values are djl_python.huggingface, djl_python.deepspeed, and djl_python.stable-diffusion. We use djl_python.huggingface for Hugging Face Speed up.
• possibility.tensor_parallel_degree – This feature specifies the variety of tensor parallel partitions carried out on the mannequin. You may set to the variety of GPU gadgets over which Speed up must partition the mannequin. This parameter additionally controls the variety of employees per mannequin that will likely be began up when DJL serving runs. For instance, if we now have a 4 GPU machine and we’re creating 4 partitions, then we can have one employee per mannequin to serve the requests.
• possibility.low_cpu_mem_usage – This reduces CPU reminiscence utilization when loading fashions. We suggest that you simply set this to TRUE.
• possibility.rolling_batch – This permits iteration-level batching utilizing one of many supported methods. Values embrace auto, scheduler, and lmi-dist. We use lmi-dist for turning on steady batching for Llama 2.
• possibility.max_rolling_batch_size – This limits the variety of concurrent requests within the steady batch. The worth defaults to 32.
• possibility.model_id – You must change {{model_id}} with the mannequin ID of a pre-trained mannequin hosted inside a model repository on Hugging Face or S3 path to the mannequin artifacts.

Extra configuration choices could be present in Configurations and settings.

As a result of DJL Serving expects the mannequin artifacts to be packaged and formatted in a .tar file, run the next code snippet to compress and add the .tar file to Amazon S3:

s3_code_prefix = f"{s3_prefix}/code" # folder inside bucket the place code artifact will go
s3_code_artifact = sess.upload_data("mannequin.tar.gz", bucket, s3_code_prefix)

Retrieve the most recent LMI container picture with DJL Serving

Subsequent, you utilize the DLCs obtainable with SageMaker for LMI to deploy the mannequin. Retrieve the SageMaker picture URI for the djl-deepspeed container programmatically utilizing the next code:

from sagemaker import image_uris
inference_image_uri = image_uris.retrieve(
    framework="djl-deepspeed", area=area, model="0.25.0"
)

You should use the aforementioned picture to deploy the meta-llama/Llama-2-13b-chat-hf mannequin on SageMaker. Now you possibly can proceed to create the mannequin.

Create the mannequin

You may create the mannequin whose container is constructed utilizing the inference_image_uri and the mannequin serving code situated on the S3 URI indicated by s3_code_artifact:

from sagemaker.utils import name_from_base

model_name = name_from_base(f"Llama-2-13b-chat-lmi-streaming")

create_model_response = sm_client.create_model(
    ModelName=model_name,
    ExecutionRoleArn=function,
    PrimaryContainer={
        "Picture": inference_image_uri,
        "ModelDataUrl": s3_code_artifact,
        "Setting": {"MODEL_LOADING_TIMEOUT": "3600"},
    },
)

Now you possibly can create the mannequin config with all the main points for the endpoint configuration.

Create the mannequin config

Use the next code to create a mannequin config for the mannequin recognized by model_name:

endpoint_config_name = f"{model_name}-config"

endpoint_name = name_from_base(model_name)

endpoint_config_response = sm_client.create_endpoint_config(
    EndpointConfigName=endpoint_config_name,
    ProductionVariants=[
        {
            "VariantName": "variant1",
            "ModelName": model_name,
            "InstanceType": "ml.g5.12xlarge",
            "InitialInstanceCount": 1,
            "ModelDataDownloadTimeoutInSeconds": 3600,
            "ContainerStartupHealthCheckTimeoutInSeconds": 3600,
        },
    ],
)

The mannequin config is outlined for the ProductionVariants parameter InstanceType for the ML occasion ml.g5.12xlarge. You additionally present the ModelName utilizing the identical identify that you simply used to create the mannequin within the earlier step, thereby establishing a relation between the mannequin and endpoint configuration.

Now that you’ve outlined the mannequin and mannequin config, you possibly can create the SageMaker endpoint.

Create the SageMaker endpoint

Create the endpoint to deploy the mannequin utilizing the next code snippet:

create_endpoint_response = sm_client.create_endpoint(
    EndpointName=f"{endpoint_name}", EndpointConfigName=endpoint_config_name
)

You may view the progress of the deployment utilizing the next code snippet:

resp = sm_client.describe_endpoint(EndpointName=endpoint_name)
standing = resp["EndpointStatus"]

After the deployment is profitable, the endpoint standing will likely be InService. Now that the endpoint is prepared, let’s carry out inference with response streaming.

Actual-time inference with response streaming

As we lined within the earlier strategy for Hugging Face TGI, you need to use the identical methodology get_realtime_response_stream to invoke response streaming from the SageMaker endpoint. The code for inferencing utilizing the LMI strategy is within the llama-2-lmi/llama-2-13b-chat/2-inference-llama-2-13b-chat-lmi-response-streaming.ipynb pocket book. The LineIterator implementation is situated in llama-2-lmi/utils/LineIterator.py. Observe that the LineIterator for the Llama 2 Chat mannequin deployed on the LMI container is totally different to the LineIterator referenced in Hugging Face TGI part. The LineIterator loops over the byte stream from Llama 2 Chat fashions inferenced with the LMI container with djl-deepspeed model 0.25.0. The next helper operate will parse the response stream acquired from the inference request made by way of the invoke_endpoint_with_response_stream API:

from utils.LineIterator import LineIterator

def print_response_stream(response_stream):
    event_stream = response_stream.get('Physique')
    for line in LineIterator(event_stream):
        print(line, finish='')

The previous methodology prints the stream of knowledge learn by the LineIterator in a human-readable format.

Let’s discover tips on how to put together the immediate and directions to make use of them as a payload whereas inferencing the mannequin.

Since you’re inferencing the identical mannequin in each Hugging Face TGI and LMI, the method of making ready the immediate and directions is similar. Subsequently, you need to use the strategies get_instructions and build_llama2_prompt for inferencing.

The get_instructions methodology returns the directions. Construct the directions mixed with the duty to be carried out as detailed in user_ask_2 as follows:

user_ask_2 = f'''
AnyCompany just lately introduced new service launch named AnyCloud Streaming Service.
Write a brief e-mail concerning the product launch with Name to motion to Alice Smith, whose e-mail is alice.smith@instance.com
Point out the Coupon Code: STREAM2DREAM to get 15% for 1st 6 months.
'''

directions = get_instructions(user_ask_2)
immediate = build_llama2_prompt(directions)

Go the directions to construct the immediate as per the immediate template generated by build_llama2_prompt:

inference_params = {
        "do_sample": True,
        "top_p": 0.6,
        "temperature": 0.9,
        "top_k": 50,
        "max_new_tokens": 512,
        "return_full_text": False,
    }

payload = {
    "inputs":  immediate,
    "parameters": inference_params
}

We membership the inference parameters together with the immediate to type a ultimate payload. You then ship the payload to get_realtime_response_stream, which is used to invoke an endpoint with response streaming:

resp = get_realtime_response_stream(sagemaker_runtime, endpoint_name, payload)
print_response_stream(resp)

The generated textual content from the LLM will likely be streamed to the output as proven within the following animation.

Clear up

To keep away from incurring pointless costs, use the AWS Administration Console to delete the endpoints and its related sources that had been created whereas working the approaches talked about within the put up. For each deployment approaches, carry out the next cleanup routine:

import boto3
sm_client = boto3.shopper('sagemaker')
endpoint_name="<SageMaker_Real-time_Endpoint_Name>"
endpoint = sm_client.describe_endpoint(EndpointName=endpoint_name)
endpoint_config_name = endpoint['EndpointConfigName']
endpoint_config = sm_client.describe_endpoint_config(EndpointConfigName=endpoint_config_name)
model_name = endpoint_config['ProductionVariants'][0]['ModelName']

print(f"""
About to delete the next sagemaker sources:
Endpoint: {endpoint_name}
Endpoint Config: {endpoint_config_name}
Mannequin: {model_name}
""")

# delete endpoint
sm_client.delete_endpoint(EndpointName=endpoint_name)
# delete endpoint config
sm_client.delete_endpoint_config(EndpointConfigName=endpoint_config_name)
# delete mannequin
sm_client.delete_model(ModelName=model_name)

Change <SageMaker_Real-time_Endpoint_Name> for variable endpoint_name with the precise endpoint.

For the second strategy, we saved the mannequin and code artifacts on Amazon S3. You may clear up the S3 bucket utilizing the next code:

s3 = boto3.useful resource('s3')
s3_bucket = s3.Bucket(bucket)
s3_bucket.objects.filter(Prefix=s3_prefix).delete()

Conclusion

On this put up, we mentioned how a various variety of response tokens or a special set of inference parameters can have an effect on the latencies related to LLMs. We confirmed tips on how to deal with the issue with the assistance of response streaming. We then recognized two approaches for deploying and inferencing Llama 2 Chat fashions utilizing AWS DLCs—LMI and Hugging Face TGI.

You must now perceive the significance of streaming response and the way it can scale back perceived latency. Streaming response can enhance the consumer expertise, which in any other case would make you wait till the LLM builds the entire response. Moreover, deploying Llama 2 Chat fashions with response streaming improves the consumer expertise and makes your clients blissful.

You may discuss with the official aws-samples amazon-sagemaker-llama2-response-streaming-recipes that covers deployment for different Llama 2 mannequin variants.

References

Concerning the Authors

Pavan Kumar Rao Navule is a Options Architect at Amazon Net Providers. He works with ISVs in India to assist them innovate on AWS. He’s a broadcast creator for the guide “Getting Began with V Programming.” He pursued an Government M.Tech in Knowledge Science from the Indian Institute of Expertise (IIT), Hyderabad. He additionally pursued an Government MBA in IT specialization from the Indian Faculty of Enterprise Administration and Administration, and holds a B.Tech in Electronics and Communication Engineering from the Vaagdevi Institute of Expertise and Science. Pavan is an AWS Licensed Options Architect Skilled and holds different certifications equivalent to AWS Licensed Machine Studying Specialty, Microsoft Licensed Skilled (MCP), and Microsoft Licensed Expertise Specialist (MCTS). He’s additionally an open-source fanatic. In his free time, he likes to hearken to the good magical voices of Sia and Rihanna.

Sudhanshu Hate is principal AI/ML specialist with AWS and works with purchasers to advise them on their MLOps and generative AI journey. In his earlier function earlier than Amazon, he conceptualized, created, and led groups to construct ground-up open source-based AI and gamification platforms, and efficiently commercialized it with over 100 purchasers. Sudhanshu to his credit score a few patents, has written two books and several other papers and blogs, and has introduced his factors of view in varied technical boards. He has been a thought chief and speaker, and has been within the trade for practically 25 years. He has labored with Fortune 1000 purchasers throughout the globe and most just lately with digital native purchasers in India.