Deploy a multimodal WhatsApp ordering assistant with Amazon Bedrock AgentCore

This put up reveals the best way to deploy a multimodal WhatsApp ordering assistant constructed with Amazon Bedrock AgentCore and Amazon Nova 2. Many quick-service eating places unfold ordering throughout an app, a web site, a telephone line, and the counter. Every of these is a separate system to construct and run. Each additionally fragments the client’s historical past, making the identical individual seem like a stranger on each channel. Prospects already reside of their messaging apps. WhatsApp reaches greater than two billion individuals. A buyer who can textual content, ship a voice word, or place a name from the identical dialog doesn’t want to put in something or sign up.

A single WhatsApp Enterprise quantity hosts the assistant. A buyer can textual content the restaurant, ship a voice word, or place a voice name. An AI agent takes the order finish to finish, from greeting to affirmation. All three channels share one backend and one cross-channel reminiscence. A buyer who texts at this time and calls tomorrow is acknowledged as the identical individual.

The answer makes use of the Meta WhatsApp Business Platform because the buyer entrance door. Amazon Bedrock AgentCore hosts the brokers. Amazon Nova 2 Lite handles textual content by the Amazon Bedrock Converse API, and Amazon Nova 2 Sonic handles real-time speech on voice notes and calls. The brokers attain the restaurant backend by the Model Context Protocol (MCP). You deploy the entire system with the AWS Cloud Growth Package (AWS CDK). The channel and the ordering logic keep separate, so the backend doesn’t change if you add or take away a channel.

Resolution overview

The design retains three issues aside: (1) the WhatsApp layer handles the dialog, (2) three agent runtimes run the conversations for his or her channels, and (3) the backend holds the menu, carts, orders, and places. Inbound site visitors arrives on a single HTTPS webhook, is acknowledged with a 200 instantly, after which processed asynchronously so no request blocks the response. This separation retains every layer independently deployable and simple to cause about.

The client entrance door is the Meta WhatsApp Business Platform. It exposes the Cloud API webhook, Messages API, Media API, and Calling API. Meta manages this service. You set it up and join it as a prerequisite, not one thing this answer deploys. The AWS CDK provisions all the things on the AWS aspect. It emits the webhook URL you register with Meta.

You utilize the AWS CDK to deploy the next AWS assets as a set of stacks in dependency order, grouped right here by perform.

• Amazon API Gateway gives two REST APIs. The primary is a regional HTTPS webhook on an AWS managed certificates. It’s the one public endpoint. The second is an AWS Identification and Entry Administration (IAM)-authorized backend API in entrance of the ordering logic.
• AWS Lambda runs the webhook ingest, the webhook employee, the message sender, and the ordering enterprise logic.
• Amazon Easy Queue Service (Amazon SQS) gives the inbound queue, with a dead-letter queue, that decouples the fast acknowledgment from the remainder of the processing.
• AgentCore runtime, a functionality of Amazon Bedrock AgentCore, hosts the three brokers. Every dialog runs in its personal microVM. Classes keep remoted.
• Amazon Nova 2 Lite (textual content by the Converse API) and Amazon Nova 2 Sonic (speech-to-speech for voice) are invoked by Amazon Bedrock.
• AgentCore Gateway, a functionality of Amazon Bedrock AgentCore, is a managed MCP server that exposes the backend REST API as MCP instruments the brokers name by identify.
• AgentCore reminiscence, a functionality of Amazon Bedrock AgentCore, is one shared, cross-channel file keyed by a hashed buyer ID.
• Amazon DynamoDB shops buyer profiles, orders, menu gadgets, carts, and places. It additionally holds a final inbound window desk for the WhatsApp layer.
• Amazon Location Service handles geocoding and nearest-location lookups.
• Amazon Kinesis Video Streams (Amazon KVS) gives a signaling channel that the voice-call runtime makes use of to mint credentials for the managed Traversal Utilizing Relays round NAT (TURN) relay that carries name media.
• Amazon Digital Non-public Cloud (Amazon VPC) with a single community tackle translation (NAT) gateway is the outbound path for the voice-call runtime. That is the one runtime that requires a VPC.
• AWS Secrets and techniques Supervisor holds the Meta Entry Token, App Secret, and Confirm Token, created as empty containers you populate out-of-band. AWS Techniques Supervisor Parameter Retailer holds the customer-id pepper.
• Amazon Elastic Container Registry (Amazon ECR), AWS CodeBuild, and Amazon Easy Storage Service (Amazon S3) construct and retailer the ARM64 agent container pictures.
• Amazon CloudWatch captures logs and metrics, and AWS Key Administration Service (AWS KMS) encrypts information at relaxation.

Structure diagram

Determine 1 reveals the complete structure. The diagram organizes the answer into labeled teams A by G that carry the request path. Two supporting teams sit outdoors that path, dealing with a construct pipeline that runs as soon as at deploy time to construct the agent pictures and safety and monitoring companies that again each channel at runtime.

• A. WhatsApp ingress and supply: The webhook API Gateway, the ingest, employee, and sender Lambdas, and the Amazon SQS queue. The ingest verifies the Meta signature and enqueues the message. The employee handles the remaining processing. The Sender Lambda delivers replies.
• B. Agent runtimes: Three brokers on AgentCore runtime, every an ARM64 container, one per channel: chat (Amazon Nova 2 Lite), voice word (Amazon Nova 2 Sonic), and voice name (Amazon Nova 2 Sonic over Net Actual-Time Communication, or WebRTC). Solely the voice-call runtime runs in a VPC.
• C. Synthetic intelligence and machine studying (AI/ML): Amazon Nova 2 Lite and Amazon Nova 2 Sonic invoked by Amazon Bedrock, plus the shared AgentCore reminiscence keyed by a hashed customer_id for cross-channel continuity.
• D. MCP server (managed): AgentCore Gateway exposes the backend REST API as discoverable MCP instruments (GetMenu, AddToCart, PlaceOrder, and extra) that each runtime calls by identify.
• E. API and compute: The IAM-authorized backend API Gateway and the ordering Lambdas that maintain the enterprise logic.
• F. Knowledge storage: Amazon DynamoDB shops buyer profiles, orders, menu gadgets, carts, and places.
• G. Networking and placement: Amazon Location Service for geocoding and nearest-location lookups, the Amazon KVS managed TURN relay for voice-call media, and the VPC with a NAT gateway because the outbound path for the voice-call runtime.

Two supporting teams sit outdoors the request path. The construct pipeline (AWS CDK, AWS CodeBuild, Amazon ECR, and Amazon S3) runs as soon as at deploy time to construct and retailer the ARM64 agent pictures. It’s not within the path of any request. Safety and monitoring (AWS Secrets and techniques Supervisor, AWS Techniques Supervisor Parameter Retailer, Amazon CloudWatch, and AWS KMS) again each channel at runtime.

The next steps hint a single request finish to finish by the structure:

1. Meta delivers an inbound webhook (textual content, voice word, or name occasion) to the Webhook API Gateway and the Webhook Ingest Lambda.
2. The ingest verifies the Meta signature, enqueues to Amazon SQS, and returns 200 inside Meta’s window.
3. The Webhook Employee derives a pseudonymous customer_id utilizing the pepper in AWS Techniques Supervisor Parameter Retailer.
4. The employee fetches media from the Meta Media API and invokes the matching agent on Amazon Bedrock AgentCore runtime (chat, voice word, or voice name) with session_id = customer_id.
5. The runtime reads the client’s long-term insights from AgentCore reminiscence at session begin.
6. It runs the dialog with Amazon Nova 2 Lite (textual content) or Amazon Nova 2 Sonic (voice) by Amazon Bedrock.
7. AgentCore Gateway is the managed MCP server. It exposes the backend REST API as MCP instruments the agent calls by identify.
8. Instruments route by the backend API Gateway to AWS Lambda, Amazon DynamoDB, and Amazon Location Service. Voice-call media makes use of the Amazon KVS TURN relay (runtime in a VPC).
9. Replies exit by the Sender Lambda (textual content) or the employee (voice). Occasions are written again to reminiscence at session finish.
10. AWS CDK builds ARM64 pictures by AWS CodeBuild into Amazon ECR. Amazon CloudWatch logs elements and AWS KMS encrypts information at relaxation.

In brief, each request flows from Meta’s webhook by the ingest, queue, employee, and agent runtime to the backend instruments, then again to the client on WhatsApp.

Channel flows

All three channels share the identical entrance door, backend instruments, and reminiscence. What differs is the media on the wire and the runtime that handles it.

Textual content message: A textual content message arrives on the webhook. The employee derives customer_id and invokes the chat runtime, which reads reminiscence, streams Amazon Nova 2 Lite by the Converse API, and calls backend instruments by the MCP gateway as wanted. Replies undergo the Sender Lambda, and occasions are written to reminiscence at session finish.

Determine 2 reveals the textual content movement, from the inbound webhook by Amazon Nova 2 Lite on the Converse API to the reply delivered by the Sender Lambda.

Determine 2: Textual content movement with Amazon Nova 2 Lite

Voice word (speech-to-speech): A voice word arrives as an audio message. The employee downloads the OGG Opus bytes and invokes the voice-note runtime. After studying reminiscence, the audio is decoded to 16 kHz pulse-code modulation (PCM) and fed right into a bounded Amazon Nova 2 Sonic speech-to-speech session. Instruments can be found by the identical gateway. The spoken reply is returned as a WhatsApp voice message. There’s no transcription service within the path. It’s true voice-in, voice-out.

Determine 3 reveals the voice-note movement, a bounded Amazon Nova 2 Sonic speech-to-speech session that returns a spoken reply with no transcription within the path.

Determine 3: Voice word movement with Amazon Nova 2 Sonic

Voice name (WebRTC): The client selects Name, and Meta’s Calling API delivers a join webhook with the WebRTC Session Description Protocol (SDP) provide. The employee relays it to the voice-call runtime in turnOnly mode, as a result of it has no public IP. TURN credentials come from Amazon KVS. Meta gives no trickle Interactive Connectivity Institution (ICE) path. The aiortc answerer waits for ICE gathering and returns a single-shot SDP reply. The employee delivers that reply to Meta. Media then flows over Datagram Transport Layer Safety and Safe Actual-time Transport Protocol (DTLS/SRTP) by the KVS managed TURN relay. Amazon Nova 2 Sonic drives the dialog.

Determine 4 reveals the voice-call movement, the place WebRTC media is relayed by the Amazon KVS managed TURN relay and Amazon Nova 2 Sonic drives the dialog.

Determine 4: Voice name movement with Amazon Nova 2 Sonic

Stipulations

This answer has conditions in two areas: your AWS account and your Meta WhatsApp setup. Full each previous to working the deployment, as a result of it asks you for particular WhatsApp values and the agent can’t reply till they’re in place.

AWS conditions

• You want an lively AWS account with Amazon Bedrock mannequin entry enabled for Amazon Nova 2 Lite (amazon.nova-2-lite-v1:0) and Amazon Nova 2 Sonic (amazon.nova-2-sonic-v1:0) in your deployment Area. Your IAM person or position will need to have permissions to deploy AWS CDK stacks and create the assets this answer makes use of, together with AgentCore runtime, Gateway, and reminiscence. In your native machine, set up Node.js 24.x or later, the AWS CLI 2.x configured with credentials, and git. Lastly, bootstrap AWS CDK in your goal account and Area (npx cdk bootstrap aws://<ACCOUNT_ID>/<REGION>).

The agent containers construct inside AWS CodeBuild on ARM64, so you do not want Python, Docker, or audio toolchains domestically. Deploy in an AWS Area the place Amazon Nova 2 Lite, Amazon Nova 2 Sonic, and AgentCore runtime, Gateway, and reminiscence are all out there. The US East (N. Virginia) Area (us-east-1) is an efficient place to begin.

For mannequin availability by Area, discuss with Supported fashions by AWS Area in Amazon Bedrock.

WhatsApp Enterprise Platform (Meta) conditions

The WhatsApp aspect is about up as soon as within the Meta consoles, as a prerequisite fairly than a step on this walkthrough. No AWS API can create a Meta app for you, so full it first and have the values readily available earlier than you deploy. For a demo, the Meta sandbox take a look at quantity, which is accessible at no extra price, is sufficient. You do not want enterprise verification or a manufacturing quantity. The total process lives in Meta’s documentation, linked within the following part.

Have this stuff prepared earlier than you deploy.

• A Meta Developer App with the WhatsApp product added, linked to a Enterprise portfolio. Including the product provisions a WhatsApp Enterprise Account (WABA) and a sandbox take a look at quantity at no extra price. Seek advice from Get started with the Cloud API.
• The App ID and App Secret. The App Secret is the important thing Meta makes use of to signal every webhook, and the webhook Lambda recomputes that signature.
• An Entry Token. The momentary token works for a fast take a look at and expires in about 24 hours. For one thing longer-lived, create a System User token with the whatsapp_business_messaging and whatsapp_business_management scopes.
• The Enterprise portfolio ID (the setup CLI can auto-discover the Telephone Quantity ID and WABA ID).
• A Confirm Token that you just invent. It’s a hard-to-guess string Meta echoes again throughout the one-time webhook verification handshake.
• For voice calls, the WhatsApp Calling API enabled on the quantity.

Retailer none of those in supply management. The Entry Token, App Secret, and Confirm Token are secrets and techniques and go into AWS Secrets and techniques Supervisor throughout deployment, not into the CDK templates.

Deploy the answer with AWS CDK

The total answer is within the sample repository on GitHub. The repository incorporates three agent containers, the AWS CDK infrastructure code, and setup scripts that wire the WhatsApp webhook to your account. Clone it, run the preflight verify, then deploy with a prefix. The prefix is added to each useful resource identify, so you possibly can deploy greater than as soon as per account.

git clone https://github.com/aws-samples/sample-multimodal-whatsapp-restaurant-agent.git
cd sample-multimodal-whatsapp-restaurant-agent
./scripts/preflight-check.sh
./scripts/deploy-all.sh --deploymentPrefix qsr-wa

The script provisions each stack in dependency order. It passes every stack’s outputs to the following. First it deploys the shared VPC. Then it deploys the backend: Amazon DynamoDB, Amazon Location Service, the ordering Lambdas, and the backend REST API. Subsequent it deploys AgentCore Gateway and the shared AgentCore reminiscence. It then builds every ARM64 container with AWS CodeBuild, pushes to Amazon ECR, and deploys the three runtimes. After that it deploys the WhatsApp webhook and the order-notifier. It then seeds pattern menu and placement information. The primary construct of every container takes roughly 8–12 minutes. For a guided, browser-based expertise, use ./scripts/deploy-all.sh --interactive-web-ui.

How the WhatsApp values attain the deployment issues for safety. The CDK creates empty Secrets and techniques Supervisor containers for the three secrets and techniques. It doesn’t take a secret as a CDK parameter, which might bake it into the synthesized template. You populate the key values out-of-band, whereas the non-secret identifiers (Telephone Quantity ID, WABA ID, App ID) experience alongside as parameters. The scripts/whatsapp-setup/ CLI handles this in two flows. The pre-deploy movement validates the Entry Token and auto-discovers the WABA and Telephone Quantity ID. It generates a Confirm Token if wanted and populates the secrets and techniques. After the deploy emits the webhook URL, the post-deploy movement units the callback URL, Confirm Token, and subscribed fields in Meta. It then subscribes the WABA and completes the verification handshake.

cd scripts/whatsapp-setup
npm begin          # select "Pre-deploy", then "Submit-deploy" after deploy
node whatsapp-setup.mjs --doctor   # read-only end-to-end verify

As soon as the webhook is subscribed and the secrets and techniques are populated, the quantity is reside and the agent replies throughout all three channels.

Acknowledging quick and processing asynchronously

Meta expects a immediate HTTP response with a 200 OK standing code. Taking an order, fetching media, invoking an agent, and relaying name signaling wants somewhat greater than that temporary acknowledgment window permits. So the work splits in two. The Webhook Ingest Lambda does solely the quick, protected half, which is to confirm the signature, enqueue to Amazon SQS, and return 200. The Webhook Employee Lambda then consumes the queue and does the remainder of the processing. A burst of messages turns into a queue to work by, and the general public floor stays at one endpoint. If the employee fails on a message, that message returns to the queue and retries. After a couple of failed makes an attempt, the message lands within the dead-letter queue for later inspection.

One reminiscence throughout three channels

What makes this really feel like one assistant is shared reminiscence. A single AgentCore reminiscence useful resource is keyed by a hashed customer_id, and all three runtimes use the identical key. Every runtime reads the client’s long-term insights at session begin and writes occasions again on the finish. The insights embrace previous orders, favourite gadgets, and preferences the client talked about earlier than. As a result of each channel resolves the identical buyer to the identical reminiscence, a buyer who texts at this time and calls tomorrow is acknowledged as the identical individual over one quantity. There’s no separate cross-channel state to reconcile.

No AgentCore runtime calls the backend Lambda capabilities instantly. AgentCore Gateway is a managed MCP server. Every AgentCore runtime connects to it as an MCP consumer over HTTPS, authenticated with the runtime’s IAM position, and discovers the instruments by identify. Every runtime carries its personal position, so the gateway grants solely the entry that runtime wants. There’s no separate MCP server behind the gateway. The gateway fronts the backend’s REST API and generates one MCP software per endpoint. When an agent calls a software equivalent to PlaceOrder, the gateway interprets it right into a REST request that the backend API Gateway routes to the matching Lambda. As a result of the brokers discuss to named instruments fairly than particular capabilities, you possibly can change a handler or add a software with out altering any agent. All three channels place orders in opposition to the identical instruments and information. The cart and order instruments personal all pricing and totals. The agent doesn’t compute them, and each order is persevered in Amazon DynamoDB with channel = "whatsapp".

Storing menus, carts, and orders

Amazon DynamoDB tables cowl the workflow. They’re Prospects (profiles for recognizing returning clients), Orders (historical past with pickup location and channel), Menu (gadgets, costs, and availability), Carts (in-progress carts with a time-to-live), and Places (coordinates, hours, and tax charges for totals and suggestions). On-demand capability scales with site visitors, so there is no such thing as a throughput to handle.

Discovering a pickup location

Amazon Location Service helps a buyer discover a pickup spot with out a lot typing. The agent passes a ZIP code or tackle to be geocoded into coordinates. The backend then ranks the closest eating places and returns concrete choices the client can act on.

Recognizing a buyer with no login

A WhatsApp buyer doesn’t sign up, so the system makes use of the telephone quantity as the idea for id. It hashes the E.164 quantity with a secret pepper in AWS Techniques Supervisor Parameter Retailer, and the end result turns into the customer_id ("wa-" + sha256(E164 || Pepper)[:16]). The uncooked quantity isn’t saved in reminiscence or session state, which helps meet personally identifiable data (PII) necessities. If it matches a identified buyer, the agent greets them by identify and remembers preferences. In any other case they order as a brand new buyer. The runtime that sends replies holds no token or telephone quantity. The Sender Lambda resolves the recipient from a “final inbound” window desk at ship time. That is recognition, not id verification. A deployment that wants verification can add a step equivalent to a one-time passcode.

Ordering walkthrough

There isn’t a net UI or take a look at consumer. Message or name the WhatsApp Enterprise quantity from a telephone with WhatsApp put in. This instance reveals a typical trade (strains prefixed “Agent” are the assistant’s replies, software calls in brackets).

Buyer: What's on the menu close to 75201?

Agent:    [tools: GeocodeAddress, GetNearestLocations, GetMenu]
          Here is what's out there at Superb Burgers - Dallas:
          - Burger Combo ($8.99)
          - Rooster Tenders ($6.49)
          ...

Buyer: A burger combo and a milkshake, please.

Agent:    [tools: AddToCart, GetCart]
          Added to your cart:
          - 1x Burger Combo - $8.99
          - 1x Milkshake - $3.49
          Whole: $12.48. Need me to position the order?

Buyer: Sure.

Agent:    [tools: PlaceOrder]
          Your order is positioned and is being ready for pickup at
          Superb Burgers - Dallas. I am going to let you realize when it is prepared.

Voice notes and voice calls run the identical movement by Amazon Nova 2 Sonic. You converse your order and the agent replies with voice. Because the order advances, the order-notifier delivers proactive standing updates again to WhatsApp. You may comply with conversations in Amazon CloudWatch Logs.

Price

You pay for the AWS companies the system makes use of, and individually for Meta’s WhatsApp messaging charges. Prices scale primarily with dialog quantity and the share of voice site visitors, which consumes extra assets per session than textual content. This instance deploys all three channels (textual content, voice word, and voice name), so limiting the deployment to particular channels can cut back your baseline price. For detailed, region-specific estimates, use the AWS Pricing Calculator and discuss with Meta’s WhatsApp Business Platform pricing page. Arrange a price range in AWS Price Explorer to trace spend as site visitors grows.

Issues to think about

WhatsApp also can obtain pictures and paperwork. This answer focuses on order taking and doesn’t act on these attachments. In case your use case wants them, you possibly can configure the backend and the chat agent to deal with them, as a result of Amazon Nova 2 Lite is already multimodal. The agent may learn a photograph of a loyalty card or a PDF catering request and switch it right into a structured order.

The sample additionally reaches past eating places. The core constructing blocks are one enterprise quantity, a number of dialog channels, shared reminiscence, and MCP instruments in entrance of a backend. The identical form matches retail help, healthcare consumption, discipline service scheduling, and plenty of different domains. You adapt it by altering the backend instruments and information whereas the channel and agent layers keep the identical.

For manufacturing deployments, contemplate enabling Amazon Bedrock Guardrails so as to add content material filtering and grounding validation. This helps ensure that agent responses keep inside coverage boundaries and reduces hallucinated outputs.

Clear up assets

To keep away from ongoing expenses, take away the assets if you find yourself executed. The cleanup script destroys stacks in reverse order, every client earlier than its producers.

./scripts/cleanup-all.sh --dry-run   # preview with out eradicating something
./scripts/cleanup-all.sh             # delete each stack the deploy created

Cleanup is harmful. It deletes the order historical past in Amazon DynamoDB, the pepper in Parameter Retailer, the Secrets and techniques Supervisor secrets and techniques, and the pictures in Amazon ECR. Again up something you need to hold first. It doesn’t contact the Meta aspect, so unsubscribe the webhook and revoke tokens within the Meta console individually. When it finishes, verify within the AWS CloudFormation console that stacks are gone.

Conclusion

This put up walked by the structure and deployment of a multimodal WhatsApp ordering assistant that takes an order finish to finish over textual content, voice notes, and voice calls on a single enterprise quantity. An asynchronous webhook accepts site visitors quick and queues the remainder of the work. Three runtimes on AgentCore runtime deal with their channels. Amazon Nova 2 Lite and Amazon Nova 2 Sonic run the conversations. AgentCore Gateway connects the brokers to the backend by MCP instruments. A single AgentCore reminiscence provides every buyer one steady relationship throughout channels. To get began, clone the sample repository on GitHub and adapt it to your menu and places. Share within the feedback the way you adapt this sample in your personal channels.

In regards to the authors