Multi-tenant RAG with Amazon Bedrock Data Bases

Organizations are constantly in search of methods to make use of their proprietary information and area experience to realize a aggressive edge. With the appearance of basis fashions (FMs) and their exceptional pure language processing capabilities, a brand new alternative has emerged to unlock the worth of their knowledge property.

As organizations attempt to ship customized experiences to prospects utilizing generative AI, it turns into paramount to specialize the conduct of FMs utilizing their very own—and their prospects’—knowledge. Retrieval Augmented Technology (RAG) has emerged as a easy but efficient method to attain a desired stage of specialization.

Amazon Bedrock Data Bases is a completely managed functionality that simplifies the administration of the whole RAG workflow, empowering organizations to offer FMs and brokers contextual info from firm’s personal knowledge sources to ship extra related and correct responses tailor-made to their particular wants.

For organizations creating multi-tenant merchandise, akin to unbiased software program distributors (ISVs) creating software program as a service (SaaS) choices, the flexibility to personalize experiences for every of their prospects (tenants of their SaaS software) is especially vital. This personalization could be achieved by implementing a RAG method that selectively makes use of tenant-specific knowledge.

On this submit, we talk about and supply examples of the way to obtain personalization utilizing Amazon Bedrock Data Bases. We focus significantly on addressing the multi-tenancy challenges that ISVs face, together with knowledge isolation, safety, tenant administration, and price administration. We give attention to situations the place the RAG structure is built-in into the ISV software and never instantly uncovered to tenants. Though the particular implementations offered on this submit use Amazon OpenSearch Service as a vector database to retailer tenants’ knowledge, the challenges and structure options proposed could be prolonged and tailor-made to different vector retailer implementations.

Multi-Tenancy design issues

When architecting a multi-tenanted RAG system, organizations have to take a number of issues into consideration:

• Tenant isolation – One essential consideration in designing multi-tenanted methods is the extent of isolation between the info and sources associated to every tenant. These sources embody knowledge sources, ingestion pipelines, vector databases, and RAG shopper software. The extent of isolation is usually ruled by safety, efficiency, and the scalability necessities of your resolution, collectively along with your regulatory necessities. For instance, chances are you’ll have to encrypt the info associated to every of your tenants utilizing a special encryption key. You may additionally have to make it possible for excessive exercise generated by one of many tenants doesn’t have an effect on different tenants.
• Tenant variability – An identical but distinct consideration is the extent of variability of the options supplied to every tenant. Within the context of RAG methods, tenants may need various necessities for knowledge ingestion frequency, doc chunking technique, or vector search configuration.
• Tenant administration simplicity – Multi-tenant options want a mechanism for onboarding and offboarding tenants. This dimension determines the diploma of complexity for this course of, which could contain provisioning or tearing down tenant-specific infrastructure, akin to knowledge sources, ingestion pipelines, vector databases, and RAG shopper functions. This course of may additionally contain including or deleting tenant-specific knowledge in its knowledge sources.
• Value-efficiency – The working prices of a multi-tenant resolution depend upon the best way it offers the isolation mechanism for tenants, so designing a cost-efficient structure for the answer is essential.

These 4 issues must be fastidiously balanced and weighted to swimsuit the wants of the particular resolution. On this submit, we current a mannequin to simplify the decision-making course of. Utilizing the core isolation ideas of silo, pool, and bridge outlined within the SaaS Tenant Isolation Methods whitepaper, we suggest three patterns for implementing a multi-tenant RAG resolution utilizing Amazon Bedrock Data Bases, Amazon Easy Storage Service (Amazon S3), and OpenSearch Service.

A typical RAG resolution utilizing Amazon Bedrock Data Bases consists of a number of elements, as proven within the following determine:

The primary problem in adapting this structure for multi-tenancy is figuring out the way to present isolation between tenants for every of the elements. We suggest three prescriptive patterns that cater to totally different use circumstances and provide carrying ranges of isolation, variability, administration simplicity, and cost-efficiency. The next determine illustrates the trade-offs between these three architectural patterns when it comes to reaching tenant isolation, variability, cost-efficiency, and ease of tenant administration.

Multi-tenancy patterns

On this part, we describe the implementation of those three totally different multi-tenancy patterns in a RAG structure primarily based on Amazon Bedrock Data Bases, discussing their use circumstances in addition to their professionals and cons.

Silo

The silo sample, illustrated within the following determine, provides the very best stage of tenant isolation, as a result of the whole stack is deployed and managed independently for every single tenant.

Within the context of the RAG structure carried out by Amazon Bedrock Data Bases, this sample prescribes the next:

• A separate knowledge supply per tenant – On this submit, we contemplate the situation wherein tenant paperwork to be vectorized are saved in Amazon S3, subsequently a separate S3 bucket is provisioned per tenant. This enables for per-tenant AWS Key Administration Service (AWS KMS) encryption keys, in addition to per-tenant S3 lifecycle insurance policies to handle object expiration, and object versioning insurance policies to take care of a number of variations of objects. Having separate buckets per tenant offers isolation and permits for custom-made configurations primarily based on tenant necessities.
• A separate information base per tenant – This enables for a separate chunking technique per tenant, and it’s significantly helpful should you envision the doc foundation of your tenants to be totally different in nature. For instance, one in every of your tenants may need a doc base composed of flat textual content paperwork, which could be handled with fixed-size chunking, whereas one other tenant may need a doc base with specific sections, for which semantic chunking could be higher suited to part. Having a special information base per tenant additionally helps you to resolve on totally different embedding fashions, supplying you with the likelihood to decide on totally different vector dimensions, balancing accuracy, price, and latency. You’ll be able to select a special KMS key per tenant for the transient knowledge shops, which Amazon Bedrock makes use of for end-to-end per-tenant encryption. You may as well select per-tenant knowledge deletion insurance policies to regulate whether or not your vectors are deleted from the vector database when a information base is deleted. Separate information bases additionally imply that you could have totally different ingestion schedules per tenants, permitting you to conform to totally different knowledge freshness requirements along with your prospects.
• A separate OpenSearch Serverless assortment per tenant – Having a separate OpenSearch Serverless assortment per tenant permits you to have a separate KMS encryption key per tenant, sustaining per-tenant end-to-end encryption. For every tenant-specific assortment, you may create a separate vector index, subsequently selecting for every tenant the gap metric between Euclidean and dot product, so that you could select how a lot significance to offer to the doc size. You may as well select the particular settings for the HNSW algorithm per tenant to regulate reminiscence consumption, price, and indexing time. Every vector index, along side the setup of metadata mappings in your information base, can have a special metadata set per tenant, which can be utilized to carry out filtered searches. Metadata filtering can be utilized within the silo sample to limit the search to a subset of paperwork with a selected attribute. For instance, one in every of your tenants could be importing dated paperwork and needs to filter paperwork pertaining to a selected 12 months, whereas one other tenant could be importing paperwork coming from totally different firm divisions and needs to filter over the documentation of a selected firm division.

As a result of the silo sample provides tenant architectural independence, onboarding and offboarding a tenant means creating and destroying the RAG stack for that tenant, composed of the S3 bucket, information base, and OpenSearch Serverless assortment. You’ll sometimes do that utilizing infrastructure as code (IaC). Relying in your software structure, you might also have to replace the log sinks and monitoring methods for every tenant.

Though the silo sample provides the very best stage of tenant isolation, it is usually the most costly to implement, primarily as a result of making a separate OpenSearch Serverless assortment per tenant for the next causes:

• Minimal capability prices – Every OpenSearch Serverless assortment encrypted with a separate KMS key has a minimal of two OpenSearch Compute Models (OCUs) charged hourly. These OCUs are charged independently from utilization, which means that you’ll incur prices for dormant tenants should you select to have a separate KMS encryption key per tenant.
• Scalability overhead – Every assortment individually scales OCUs relying on utilization, in steps of 6 GB of reminiscence, and related vCPUs and quick entry storage. Because of this sources may not be totally and optimally utilized throughout tenants.

When selecting the silo sample, be aware {that a} most of 100 information bases are supported in every AWS account. This makes the silo sample favorable in your largest tenants with particular isolation necessities. Having a separate information base per tenant additionally reduces the impression of quotas on concurrent ingestion jobs (most one concurrent job per KB, 5 per account), job measurement (100 GB per job), and knowledge sources (most of 5 million paperwork per knowledge supply). It additionally improves the efficiency equity as perceived by your tenants.
Deleting a information base throughout offboarding a tenant could be time-consuming, relying on the scale of the info sources and the synchronization course of. To mitigate this, you may set the info deletion coverage in your tenants’ information bases to RETAIN. This manner, the information base deletion course of is not going to delete your tenants’ knowledge from the OpenSearch Service index. You’ll be able to delete the index by deleting the OpenSearch Serverless assortment.

Pool

In distinction with the silo sample, within the pool sample, illustrated within the following determine, the entire end-to-end RAG structure is shared by your tenants, making it significantly appropriate to accommodate many small tenants.

The pool sample prescribes the next:

• Single knowledge supply – The tenants’ knowledge is saved inside the similar S3 bucket. This suggests that the pool mannequin helps a shared KMS key for encryption at relaxation, not providing the potential for per-tenant encryption keys. To determine tenant possession downstream for every doc uploaded to Amazon S3, a corresponding JSON metadata file must be generated and uploaded. The metadata file era course of could be asynchronous, and even batched for a number of recordsdata, as a result of Amazon Bedrock Data Bases requires an specific triggering of the ingestion job. The metadata file should use the identical identify as its related supply doc file, with .metadata.json appended to the tip of the file identify, and should be saved in the identical folder or location because the supply file within the S3 bucket. The next code is an instance of the format:

{
  "metadataAttributes" : {
    "tenantId" : "tenant_1",
  ...
  }
}

Within the previous JSON construction, the important thing tenantId has been intentionally chosen, and could be modified to a key you need to use to specific tenancy. The tenancy area can be used at runtime to filter paperwork belonging to a selected tenant, subsequently the filtering key at runtime should match the metadata key within the JSON used to index the paperwork. Moreover, you may embody different metadata keys to carry out additional filtering that isn’t primarily based on tenancy. Should you don’t add the object.metadata.json file, the shopper software received’t be capable of discover the doc utilizing metadata filtering.

• Single information base – A single information base is created to deal with the info ingestion in your tenants. Because of this your tenants will share the identical chunking technique and embedding mannequin, and share the identical encryption at-rest KMS key. Furthermore, as a result of ingestion jobs are triggered per knowledge supply per KB, you’ll be restricted to supply to your tenants the identical knowledge freshness requirements.
• Single OpenSearch Serverless assortment and index – Your tenant knowledge is pooled in a single OpenSearch Service vector index, subsequently your tenants share the identical KMS encryption key for vector knowledge, and the identical HNSW parameters for indexing and question. As a result of tenant knowledge isn’t bodily segregated, it’s essential that the question shopper be capable of filter outcomes for a single tenant. This may be effectively achieved utilizing both the Amazon Bedrock Data Bases Retrieve or RetrieveAndGenerate, expressing the tenant filtering situation as a part of the retrievalConfiguration (for extra particulars, see Amazon Bedrock Data Bases now helps metadata filtering to enhance retrieval accuracy). If you wish to limit the vector search to return outcomes for tenant_1, the next is an instance shopper implementation performing RetrieveAndGenerate primarily based on the AWS SDK for Python (Boto3):

import boto3

bedrock_agent_runtime = boto3.shopper(
    service_name = "bedrock-agent-runtime"
)

tenant_filter = {
    "equals": {
        "key": "tenantId",
        "worth": "tenant_1"
    }
}

retrievalConfiguration = {
    "vectorSearchConfiguration": {
        "filter": tenant_filter
    }
}

bedrock_agent_runtime.retrieve_and_generate(
    enter = {
        'textual content': 'The unique person question'
    },
    retrieveAndGenerateConfiguration = {
        'kind': 'KNOWLEDGE_BASE',
        'knowledgeBaseConfiguration': {
            'knowledgeBaseId': <YOUR_KNOWLEDGEBASE_ID>,
            'modelArn': <FM_ARN>,
            'retrievalConfiguration': retrievalConfiguration
        }
    }
)

textual content comprises the unique person question that must be answered. Making an allowance for the doc base, <YOUR_KNOWLEDGEBASE_ID> must be substituted with the identifier of the information base used to pool your tenants, and <FM_ARN> must be substituted with the Amazon Bedrock mannequin Amazon Useful resource Title (ARN) you need to use to answer to the person question. The shopper offered within the previous code has been streamlined to current the tenant filtering performance. In a manufacturing case, we suggest implementing session and error dealing with, logging and retry logic, and separating the tenant filtering logic from the shopper invocation to make it inaccessible to builders.

As a result of the end-to-end structure is pooled on this sample, onboarding and offboarding a tenant doesn’t require you to create new bodily or logical constructs, and it’s so simple as beginning or stopping and importing particular tenant paperwork to Amazon S3. This suggests that there is no such thing as a AWS managed API that can be utilized to offboard and end-to-end overlook a selected tenant. To delete the historic paperwork belonging to a selected tenant, you may simply delete the related objects in Amazon S3. Sometimes, prospects can have an exterior software that maintains the record of obtainable tenants and their standing, facilitating the onboarding and offboarding course of.

Sharing the monitoring system and logging capabilities on this sample reduces the complexity of operations with numerous tenants. Nonetheless, it requires you to gather the tenant-specific metrics from the shopper facet to carry out particular tenant attribution.

The pool sample optimizes the end-to-end price of your RAG structure, as a result of sharing OCUs throughout tenants maximizes using every OCU and minimizes the tenants’ idle time. Sharing the identical pool of OCUs throughout tenants signifies that this sample doesn’t provide efficiency isolation on the vector retailer stage, so the most important and most lively tenants would possibly impression the expertise of different tenants.

When selecting the pool sample in your RAG structure, try to be conscious {that a} single ingestion job can ingest or delete a most of 100 GB. Moreover, the info supply can have a most of 5 million paperwork. If the answer has many tenants which might be geographically distributed, contemplate triggering the ingestion job a number of occasions a day so that you don’t hit the ingestion job measurement restrict. Additionally, relying on the quantity and measurement of your paperwork to be synchronized, the time for ingestion can be decided by the embedding mannequin invocation fee. For instance, contemplate the next situation:

• Variety of tenants to be synchronized = 10
• Common variety of paperwork per tenant = 100
• Common measurement per doc = 2 MB, containing roughly 200,000 tokens divided in 220 chunks of 1,000 tokens to permit for overlap
• Utilizing Amazon Titan Embeddings v2 on demand, permitting for two,000 RPM and 300,000 TPM

This could end result within the following:

• Complete embeddings requests = 10*100*220 = 220,000
• Complete tokens to course of = 10*100*1,000=1,000,000
• Complete time taken to embed is dominated by the RPM, subsequently 220,000/2,000 = 1 hour, 50 minutes

This implies you might set off an ingestion job 12 occasions per day to have time distribution of information to be ingested. This calculation is a best-case situation and doesn’t account for the latency launched by the FM when creating the vector from the chunk. Should you anticipate having to synchronize numerous tenants on the similar time, think about using provisioned throughput to lower the time it takes to create vector embeddings. This method may also assist distribute the load on the embedding fashions, limiting throttling of the Amazon Bedrock runtime API calls.

Bridge

The bridge sample, illustrated within the following determine, strikes a steadiness between the silo and pool patterns, providing a center floor that balances tenant knowledge isolation and safety.

The bridge sample delivers the next traits:

• Separate knowledge supply per tenant in a typical S3 bucket – Tenant knowledge is saved in the identical S3 bucket, however prefixed by a tenant identifier. Though having a special prefix per tenant doesn’t provide the potential for utilizing per-tenant encryption keys, it does create a logical separation that can be utilized to segregate knowledge downstream within the information bases.
• Separate information base per tenant – This sample prescribes making a separate information base per tenant much like the silo sample. Subsequently, the issues within the silo sample apply. Functions constructed utilizing the bridge sample often share question shoppers throughout tenants, so they should determine the particular tenant’s information base to question. They’ll determine the information base by storing the tenant-to-knowledge base mapping in an exterior database, which manages tenant-specific configurations. The next instance reveals the way to retailer this tenant-specific info in an Amazon DynamoDB desk:

  import boto3
  # Create a DynamoDB useful resource
  dynamodb = boto3.useful resource('dynamodb')
  
  table_name="tenantKbConfig"
  attribute_definitions = [
      {'AttributeName': 'tenantId', 'AttributeType': 'S'}
  ]
  
  key_schema = [
      {'AttributeName': 'tenantId', 'KeyType': 'HASH'}
  ]
  
  #Create the desk holding KB tenant configurations
  tenant_kb_config_table = dynamodb.create_table(
      TableName=table_name,
      AttributeDefinitions=attribute_definitions,
      KeySchema=key_schema,
      BillingMode="PAY_PER_REQUEST" # Use on-demand billing mode for illustration
  )
  
  #Create a tenant
      tenant_kb_config_table.put_item(
      Merchandise={
          'tenantId': 'tenant_1',
          'knowledgebaseId': <YOUR_KNOWLEDGEBASE_ID>,
          'modelArn': <FM_ARN>     }
  )

  In a manufacturing setting, your software will retailer tenant-specific parameters belonging to different performance in your knowledge shops. Relying in your software structure, you would possibly select to retailer knowledgebaseId and modelARN alongside the opposite tenant-specific parameters, or create a separate knowledge retailer (for instance, the tenantKbConfig desk) particularly in your RAG structure.

  This mapping can then be utilized by the shopper software by invoking the RetrieveAndGenerate API. The next is an instance implementation:

  import json
  import boto3
  
  # Create a DynamoDB useful resource
  dynamodb = boto3.useful resource('dynamodb')
  
  # Create a Bedrock Runtime shopper
  bedrock_runtime = boto3.shopper('bedrock-agent-runtime')
  
  # Outline the desk identify
  table_name="tenantKbConfig"
  
  # Outline perform returning tenant config
  def get_tenant_config(tenant_id):
      desk = dynamodb.Desk(table_name)
      response = desk.get_item(
          Key = {
              'tenantId': tenant_id
          }
      )
  if 'Merchandise' in response:
      return { 'knowledgebaseId':response['Item'].get('knowledgebaseId'), 'modelArn': response['Item'].get('modelArn')}
  else:
      return None
  
  # Retrieve the tenant configurations from DynamoDB
  
  tenant_config = get_tenant_config('tenant_1')
  
  #Invoke the Retrieve and Generate API
  bedrock_runtime.retrieve_and_generate(
      enter = {
          'textual content': 'What kind of information do your paperwork include?'
      },
      retrieveAndGenerateConfiguration = {
          'kind': 'KNOWLEDGE_BASE',
          'knowledgeBaseConfiguration': {
              'knowledgeBaseId': tenant_config['knowledgebaseId'],
              'modelArn': tenant_config['modelArn']
          }
      }
  )

• Separate OpenSearch Service index per tenant – You retailer knowledge inside the similar OpenSearch Serverless assortment, however you create a vector index per tenant. This suggests your tenants share the identical KMS encryption key and the identical pool of OCUs, optimizing the OpenSearch Service sources utilization for indexing and querying. The separation in vector indexes provides you the flexibleness of selecting totally different HNSM parameters per tenant, letting you tailor the efficiency of your k-NN indexing and querying in your totally different tenants.

The bridge sample helps as much as 100 tenants, and onboarding and offboarding a tenant requires the creation and deletion of a information base and OpenSearch Service vector index. To delete the info pertaining to a specific tenant, you may delete the created sources and use the tenant-specific prefix as a logical parameter in your Amazon S3 API calls. Not like the silo sample, the bridge sample doesn’t enable for per-tenant end-to-end encryption; it provides the identical stage of tenant customization supplied by the silo sample whereas optimizing prices.

Abstract of variations

The next determine and desk present a consolidated view for evaluating the traits of the totally different multi-tenant RAG structure patterns. This complete overview highlights the important thing attributes and trade-offs related to the pool, bridge, and silo patterns, enabling knowledgeable decision-making primarily based on particular necessities.

The next determine illustrates the mapping of design traits to elements of the RAG structure.

The next desk summarizes the traits of the multi-tenant RAG structure patterns.

Conclusion

This submit explored three distinct patterns for implementing a multi-tenant RAG structure utilizing Amazon Bedrock Data Bases and OpenSearch Service. The silo, pool, and bridge patterns provide various ranges of tenant isolation, variability, administration simplicity, and cost-efficiency, catering to totally different use circumstances and necessities. By understanding the trade-offs and issues related to every sample, organizations could make knowledgeable choices and select the method that finest aligns with their wants.

Get began with Amazon Bedrock Data Bases at this time.

About the Authors

Emanuele Levi is a Options Architect within the Enterprise Software program and SaaS group, primarily based in London. Emanuele helps UK prospects on their journey to refactor monolithic functions into fashionable microservices SaaS architectures. Emanuele is principally considering event-driven patterns and designs, particularly when utilized to analytics and AI, the place he has experience within the fraud-detection trade.

Mehran Nikoo is a Generative AI Go-To-Market Specialist at AWS. He leads the generative AI go-to-market technique for UK and Eire.

Dani Mitchell is a Generative AI Specialist Options Architect at AWS. He’s targeted on pc imaginative and prescient use case and helps AWS prospects in EMEA speed up their machine studying and generative AI journeys with Amazon SageMaker and Amazon Bedrock.