Introducing document-level sync stories: Enhanced information sync visibility in Amazon Kendra

Amazon Kendra is an clever search service powered by machine studying (ML). Amazon Kendra helps you combination content material from quite a lot of content material repositories right into a centralized index that allows you to rapidly search all of your enterprise information and discover probably the most correct reply.

Amazon Kendra securely connects to over 40 information sources. When utilizing your information supply, you may want higher visibility into the doc processing lifecycle throughout information supply sync jobs. They might embody understanding the standing of every doc you tried to crawl and index, in addition to with the ability to troubleshoot why sure paperwork weren’t returned with the anticipated solutions. Moreover, you may want entry to metadata, timestamps, and entry management lists (ACLs) for the listed paperwork.

We’re happy to announce a brand new characteristic now out there in Amazon Kendra that considerably improves visibility into information supply sync operations. The newest launch introduces a complete document-level report included into the sync historical past, offering directors with granular indexing standing, metadata, and ACL particulars for each doc processed throughout an information supply sync job. This enhancement to sync job observability allows directors to rapidly examine and resolve ingestion or entry points encountered whereas organising Amazon Kendra indexes. The detailed doc stories are continued within the new SYNC_RUN_HISTORY_REPORT log stream below the Amazon Kendra index log group, so important sync job particulars can be found on-demand when troubleshooting.

On this submit, we focus on the advantages of this new characteristic and the way it presents enhanced information sync visibility in Amazon Kendra.

Lifecycle of a doc in an information supply sync run job

On this part, we study the lifecycle of a doc inside an information supply sync in Amazon Kendra. This gives useful perception into the sync course of. The information supply sync includes three key levels: crawling, syncing, and indexing. Crawling includes the connector connecting to the info supply and extracting paperwork assembly the outlined sync scope in response to the info supply configuration. These paperwork are then synced to the Amazon Kendra index through the syncing part. Lastly, indexing makes the synced paperwork searchable inside the Amazon Kendra atmosphere.

The next diagram exhibits a flowchart of a sync run job.

Crawling stage

The primary stage is the crawling stage, the place the connector crawls all paperwork and their metadata from the info supply. Throughout this stage, the connector additionally compares the checksum of the doc towards the Amazon Kendra index to find out if a selected doc must be added, modified, or deleted from the index. This operation corresponds to the CrawlAction discipline within the sync run historical past report.

If the doc is unmodified, it’s marked as UNMODIFIED and skipped in the remainder of the levels. If any doc fails within the crawling stage, for instance because of throttling errors, damaged content material, or if the doc dimension is simply too large, that doc is marked within the sync run historical past report with the CrawlStatus as FAILED. If the doc was skipped because of any validation errors, its CrawlStatus is marked as SKIPPED. These paperwork aren’t despatched to the subsequent stage. All profitable paperwork are marked as SUCCESS and are despatched ahead.

We additionally seize the ACLs and metadata on every doc on this stage to have the ability to add it to the sync run historical past report.

Syncing stage

In the course of the syncing stage, the doc is shipped to Amazon Kendra ingestion service APIs like BatchPutDocument and BatchDeleteDocument. After a doc is submitted to those APIs, Amazon Kendra runs validation checks on the submitted paperwork. If any doc fails these checks, its SyncStatus is marked as FAILED. If there may be an irrecoverable error for a selected doc, it’s marked as SKIPPED and different paperwork are despatched ahead.

Indexing stage

On this step, Amazon Kendra parses the doc, processes it in response to its content material sort, and persists it within the index. If the doc fails to be continued, its IndexStatus is marked as FAILED; in any other case, it’s marked as SUCCESS.

After the statuses of all of the levels have been captured, we emit these statuses as an Amazon CloudWatch occasion to the shopper’s AWS account.

Key options and advantages of document-level stories

The next are the important thing options and advantages of the brand new document-level report in Amazon Kendra indexes:

• Enhanced sync run historical past web page – A brand new Actions column has been added to the sync run historical past web page, offering entry to the document-level report for every sync run.

• Devoted log stream – A brand new log stream named SYNC_RUN_HISTORY_REPORT has been created within the Amazon Kendra CloudWatch log group, containing the document-level report.

• Complete doc data – The document-level report contains the next data for every doc:
• Doc ID – That is the doc ID that’s inherited instantly from the info supply or mapped by the shopper within the information supply discipline mappings.
• Doc title – The title of the doc is taken from the info supply or mapped by the shopper within the information supply discipline mappings.
• Consolidated doc standing (SUCCESS, FAILED, or SKIPPED) – That is the ultimate consolidated standing of the doc. It could possibly have a price of SUCCESS, FAILED, or SKIPPED. If the doc was efficiently processed in all levels, then the worth is SUCCESS. If the doc failed or was skipped in any of the levels, then the worth of this discipline shall be FAILED or SKIPPED, respectively.
• Error message (if the doc failed) – This discipline incorporates the error message with which a doc failed. If a doc was skipped because of throttling errors, or any inside errors, this shall be proven within the error message discipline.
• Crawl standing – This discipline denotes whether or not the doc was crawled efficiently from the info supply. This standing correlates to the syncing-crawling state within the information supply sync.
• Sync standing – This discipline denotes whether or not the doc was despatched for syncing efficiently. This correlates to the syncing-indexing state within the information supply sync.
• Index standing – This discipline denotes whether or not the doc was efficiently continued within the index.
• ACLs – This discipline incorporates a listing of document-level permissions that had been crawled from the info supply. The small print of every component within the checklist are:
  • International title – That is the e-mail or consumer title of the consumer. This discipline is mapped throughout a number of information sources. For instance, if a consumer has three datasources Confluence, SharePoint, and Gmail, with the native consumer ID as confluence_user, sharepoint_user and gmail_user respectively, and their e-mail handle consumer@e-mail.com is the globalName within the ACL for all of them, then Amazon Kendra understands that every one of those native consumer IDs map to the identical international title.
  • Title – That is the native distinctive ID of the consumer, which is assigned by the info supply.
  • Sort – This discipline signifies the principal sort. This may be both USER or GROUP.
  • Is Federated – This can be a boolean flag that signifies whether or not the group is of INDEX degree (true) or DATASOURCE degree (false).
  • Entry – This discipline signifies whether or not the consumer has entry allowed or denied explicitly. Values will be both ALLOWED or DENIED.
  • Knowledge supply ID – That is the info supply ID. For federated teams (INDEX degree), this discipline shall be null.
• Metadata – This discipline incorporates the metadata fields (apart from ACL) that had been pulled from the info supply. This checklist additionally contains the metadata fields mapped by the shopper within the information supply discipline mappings in addition to further metadata fields added by the connector.
• Hashed doc ID (for troubleshooting help) – To safeguard your information privateness, we current a safe, one-way hash of the doc identifier. This encrypted worth allows the Amazon Kendra staff to effectively find and analyze the particular doc inside our logs, must you encounter any concern that requires additional investigation and backbone.
• Timestamp – The timestamp signifies when the doc standing was logged in CloudWatch.

Within the following sections, we discover completely different use instances for the logging characteristic.

Decide the optimum boosting period for current paperwork in utilizing document-level reporting

In terms of producing correct solutions, chances are you’ll wish to fine-tune the way in which Amazon Kendra prioritizes its content material. As an illustration, chances are you’ll favor to spice up current paperwork over older ones to verify probably the most up-to-date passages are used to generate a solution. To realize this, you need to use the relevance tuning characteristic in Amazon Kendra to spice up paperwork primarily based on the final replace date attribute, with a specified boosting period. Nevertheless, figuring out the optimum boosting interval will be difficult when coping with a lot of steadily altering paperwork.

Now you can use the per-document-level report back to get hold of the _last_updated_at metadata discipline data to your paperwork, which will help you establish the suitable boosting interval. For this, you utilize the next CloudWatch Logs Insights question to retrieve the _last_updated_at metadata attribute for machine studying paperwork from the SYNC_RUN_HISTORY_REPORT log stream.

filter @logStream like 'SYNC_RUN_HISTORY_REPORT/'
and Metadata like 'Machine Studying'
| parse Metadata '{"key":"_last_updated_at","worth":{"dateValue":"*"}}' as @last_updated_at
| kind @last_updated_at desc, @timestamp desc
| dedup DocumentTitle

With the previous question, you may achieve insights into the final up to date timestamps of your paperwork, enabling you to make knowledgeable selections concerning the optimum boosting interval. This method makes certain your chat responses are generated utilizing the latest and related data, enhancing the general accuracy and effectiveness of your Amazon Kendra implementation.

The next screenshot exhibits an instance end result.

Frequent doc indexing observability and troubleshooting strategies

On this part, we discover some frequent admin duties for observing and troubleshooting doc indexing utilizing the brand new document-level reporting characteristic.

Listing all efficiently listed paperwork from an information supply

To retrieve a listing of all paperwork which have been efficiently listed from a particular information supply, you need to use the next CloudWatch Logs Insights question:

fields DocumentTitle, DocumentId, @timestamp
| filter @logStream like 'SYNC_RUN_HISTORY_REPORT/your-data-source-id/'
and ConnectorDocumentStatus.Standing = "SUCCESS"
| kind @timestamp desc | dedup DocumentTitle, DocumentId

The next screenshot exhibits an instance end result.

Listing all efficiently listed paperwork from an information supply sync job

To retrieve a listing of all paperwork which have been efficiently listed throughout a particular sync job, you need to use the next CloudWatch Logs Insights question:

fields DocumentTitle, DocumentId, ConnectorDocumentStatus.Standing AS IndexStatus, @timestamp
| filter @logStream like 'SYNC_RUN_HISTORY_REPORT/your-data-source-id/run-id'
and ConnectorDocumentStatus.Standing = "SUCCESS"
| kind DocumentTitle

The next screenshot exhibits an instance end result.

Listing all failed listed paperwork from an information supply sync job

To retrieve a listing of all paperwork that did not index throughout a particular sync job, together with the error messages, you need to use the next CloudWatch Logs Insights question:

fields DocumentTitle, DocumentId, ConnectorDocumentStatus.Standing AS IndexStatus, ErrorMsg, @timestamp
| filter @logStream like 'SYNC_RUN_HISTORY_REPORT/your-data-source-id/run-id'
and ConnectorDocumentStatus.Standing = "FAILED"
| kind @timestamp desc

The next screenshot exhibits an instance end result.

Listing all paperwork that include a consumer’s ACL permission from an Amazon Kendra index

To retrieve a listing of paperwork which have a particular customers ACL permission, you need to use the next CloudWatch Logs Insights question:

filter @logStream like 'SYNC_RUN_HISTORY_REPORT/'
and Acl like 'aneesh@mydemoaws.onmicrosoft.com'
| show DocumentTitle, SourceUri

The next screenshot exhibits an instance end result.

Listing the ACL of an listed doc from an information supply sync job

To retrieve the ACL data for a particular listed doc from a sync job, you need to use the next CloudWatch Logs Insights question:

filter @logStream like 'SYNC_RUN_HISTORY_REPORT/data-source-id/run-id'
and DocumentTitle = "your-document-title"
| show DocumentTitle, Acl

The next screenshot exhibits an instance end result.

Listing metadata of an listed doc from an information supply sync job

To retrieve the metadata data for a particular listed doc from a sync job, you need to use the next CloudWatch Logs Insights question:

filter @logStream like 'SYNC_RUN_HISTORY_REPORT/data-source-id/run-id'
and DocumentTitle = "your-document-title"
| show DocumentTitle, Metadata

The next screenshot exhibits an instance end result.

Conclusion

The newly launched document-level report in Amazon Kendra gives enhanced visibility and observability into the doc processing lifecycle throughout information supply sync jobs. This characteristic addresses a important want expressed by clients for higher troubleshooting capabilities and entry to detailed details about the indexing standing, metadata, and ACLs of particular person paperwork.

The document-level report is saved in a log stream named SYNC_RUN_HISTORY_REPORT inside the Amazon Kendra index CloudWatch log group. This report incorporates complete data for every doc, together with the doc ID, title, general doc sync standing, error messages (if any), together with its ACLs and metadata data retrieved from the info sources. The information supply sync run historical past web page now contains an Actions column, offering entry to the document-level report for every sync run. This characteristic considerably improves the power to troubleshoot points associated to doc ingestion and entry management, and points associated to metadata relevance, and gives higher visibility concerning the paperwork synced with an Amazon Kendra index.

To get began with Amazon Kendra, discover the Getting began information. To be taught extra about information supply connectors and finest practices, see Creating an information supply connector.

In regards to the Authors

Aneesh Mohan is a Senior Options Architect at Amazon Net Companies (AWS), with over 20 years of expertise in architecting and delivering high-impact options for mission-critical workloads. His experience spans throughout the monetary providers trade, AI/ML, safety, and information applied sciences. Pushed by a deep ardour for expertise, Aneesh is devoted to partnering with clients to design and implement well-architected, revolutionary options that handle their distinctive enterprise wants.

Ashwin Shukla is a Software program Growth Engineer II on the Amazon Q for Enterprise and Amazon Kendra engineering staff, with 6 years of expertise in growing enterprise software program. On this function, he works on designing and growing foundational options for Amazon Q for Enterprise.