WordFinder app: Harnessing generative AI on AWS for aphasia communication

On this publish, we showcase how Dr. Kori Ramajoo, Dr. Sonia Brownsett, Prof. David Copland, from QARC, and Scott Harding, an individual residing with aphasia, used AWS providers to develop WordFinder, a cell, cloud-based answer that helps people with aphasia improve their independence via the usage of AWS generative AI expertise.

Within the spirit of giving again to the group and harnessing the artwork of the doable for optimistic change, AWS hosted the Hack For Function occasion in 2023. This hackathon introduced collectively groups from AWS clients throughout Queensland, Australia, to sort out urgent challenges confronted by social good organizations.

The College of Queensland’s Queensland Aphasia Research Centre (QARC)’s mission is to enhance entry to expertise for individuals residing with aphasia, a communication incapacity that may affect a person’s skill to specific and perceive spoken and written language.

The problem: Overcoming communication limitations

In 2023, it was estimated that greater than 140,000 individuals in Australia had been residing with aphasia. This quantity is predicted to develop to over 300,000 by 2050. Aphasia could make on a regular basis duties like on-line banking, utilizing social media, and attempting new units difficult. The purpose was to create a cell app that might help individuals with aphasia by producing a thesaurus of the objects which might be in a user-selected picture and prolong the listing with associated phrases, enabling them to discover different communication strategies.

Overview of the answer

The next screenshot exhibits an instance of navigating the WordFinder app, together with check in, picture choice, object definition, and associated phrases.

Within the previous diagram, the next situation unfolds: 

1. Sign up: The primary display screen exhibits a easy sign-in web page the place customers enter their e-mail and password. It contains choices to create an account or get better a forgotten password.
2. Picture choice: After signing in, customers are prompted to Choose a picture to look. This display screen is initially clean.
3. Photograph entry: The following display screen exhibits a popup requesting personal entry to the consumer’s photographs, with a grid of pattern pictures seen within the background.
4. Picture chosen: After a picture is chosen (on this case, an image of a koala), the app shows the picture together with some preliminary tags or classifications similar to Animal, Bear, Mammal, Wildlife, and Koala.
5. Associated phrases: The ultimate display screen exhibits an inventory of associated phrases based mostly on the collection of Associated Phrases subsequent to Koala from the earlier display screen. This step is essential for individuals with aphasia who typically have difficulties with word-finding and verbal expression. By exploring associated phrases (similar to habitat phrases like tree and eucalyptus, or descriptive phrases like fur and marsupial), customers can bridge communication gaps when the precise phrase they need isn’t instantly accessible. This semantic community method aligns with widespread aphasia remedy methods, serving to customers discover other ways to specific their ideas when particular phrases are troublesome to recall.

This move demonstrates how customers can use the app to seek for phrases and ideas by beginning with a picture, then drilling down into associated terminology—a visible method to increasing vocabulary or discovering related phrases.

The next diagram illustrates the answer structure on AWS.

Within the following sections, we talk about the move and key elements of the answer in additional element.

1. Safe entry utilizing Route 53 and Amplify 
   1. The journey begins with the consumer accessing the WordFinder app via a site managed by Amazon Route 53, a extremely out there and scalable cloud DNS net service. AWS Amplify hosts the React Native frontend, offering a seamless cross-environment expertise. 
2. Safe authentication with Amazon Cognito 
   1. Earlier than accessing the core options, the consumer should securely authenticate via Amazon Cognito. Cognito gives strong consumer id administration and entry management, ensuring that solely authenticated customers can work together with the app’s providers and sources. 
3. Picture seize and storage with Amplify and Amazon S3 
   1. After being authenticated, the consumer can seize a picture of a scene, merchandise, or situation they want to recall phrases from. AWS Amplify streamlines the method by robotically storing the captured picture in an Amazon Easy Storage Service (Amazon S3) bucket, a extremely out there, cost-effective, and scalable object storage service. 
4. Object recognition with Amazon Rekognition 
   1. As quickly because the picture is saved within the S3 bucket, Amazon Rekognition, a strong pc imaginative and prescient and machine studying service, is triggered. Amazon Rekognition analyzes the picture, figuring out objects current and returning labels with confidence scores. These labels kind the preliminary phrase immediate listing throughout the WordFinder app, kickstarting the word-finding journey. 
5. Semantic phrase associations with API Gateway and Lambda 
   1. Whereas the preliminary thesaurus generated by Amazon Rekognition gives a strong start line, the consumer may be in search of a extra particular or associated phrase. To handle this problem, the WordFinder app sends the preliminary thesaurus to an AWS Lambda perform via Amazon API Gateway, a totally managed service that securely handles API requests. 
6. Lambda with Amazon Bedrock, and generative AI and immediate engineering utilizing Amazon Bedrock
   1. The Lambda perform, appearing as an middleman, crafts a fastidiously designed immediate and submits it to Amazon Bedrock, a totally managed service that gives entry to high-performing basis fashions (FMs) from main AI corporations, together with Anthropic’s Claude mannequin.
   2. Amazon Bedrock generative AI capabilities, powered by Anthropic’s Claude mannequin, use superior language understanding and era to supply semantically associated phrases and ideas based mostly on the preliminary thesaurus. This course of is pushed by immediate engineering, the place fastidiously crafted prompts information the generative AI mannequin to supply related and contextually applicable phrase associations.

WordFinder app element particulars

On this part, we take a better take a look at the elements of the WordFinder app.

React Native and Expo

WordFinder was constructed utilizing React Native, a preferred framework for constructing cross-environment cell apps. To streamline the event course of, Expo was used, which permits for write-once, run-anywhere capabilities throughout Android and iOS working programs.

Amplify

Amplify performed an important function in accelerating the app’s growth and provisioning the required backend infrastructure. Amplify is a set of instruments and providers that allow builders to construct and deploy safe, scalable, and full stack apps. On this structure, the frontend of the phrase discovering app is hosted on Amplify. The answer makes use of a number of Amplify elements:

• Authentication and entry management: Amazon Cognito is used for consumer authentication, enabling customers to enroll and check in to the app. Amazon Cognito gives consumer id administration and entry management with entry to an Amazon S3 bucket and an API gateway requiring authenticated consumer periods.
• Storage: Amplify was used to create and deploy an S3 bucket for storage. A key element of this app is the flexibility for a consumer to take an image of a scene, merchandise, or situation that they’re in search of to recall phrases from. The answer must quickly retailer this picture for processing and evaluation. When a consumer uploads a picture, it’s saved in an S3 bucket for processing with Amazon Rekognition. Amazon S3 gives extremely out there, cost-effective, and scalable object storage.
• Picture recognition: Amazon Rekognition makes use of pc imaginative and prescient and machine studying to establish objects current within the picture and return labels with confidence scores. These labels are used because the preliminary phrase immediate listing throughout the WordFinder app.

Associated phrases

The generated preliminary thesaurus is step one towards discovering the specified phrase, however the labels returned by Amazon Rekognition won’t be the precise phrase that somebody is searching for. The mission workforce then thought of learn how to implement a thesaurus-style lookup functionality. Though the mission workforce initially explored completely different programming libraries, they discovered this method to be considerably inflexible and restricted, typically returning solely synonyms and never entities which might be associated to the supply phrase. The libraries additionally added overhead related to packaging and sustaining the library and dataset transferring ahead.

To handle these challenges and enhance responses for associated entities, the mission workforce turned to the capabilities of generative AI. By utilizing the generative AI basis fashions (FMs), the mission workforce was in a position to offload the continuing overhead of managing this answer whereas rising the flexibleness and curation of associated phrases and entities which might be returned to customers. The mission workforce built-in this functionality utilizing the next providers:

• Amazon Bedrock: Amazon Bedrock is a totally managed service that gives a alternative of high-performing FMs from main AI corporations like AI21 Labs, Anthropic, Cohere, Meta, Mistral AI, Stability AI, and Amazon via a single API, together with a broad set of capabilities to construct generative AI apps with safety, privateness, and accountable AI. The mission workforce was in a position to shortly combine with, take a look at, and consider completely different FMs, lastly settling upon Anthropic’s Claude mannequin.
• API Gateway: The mission workforce prolonged the Amplify mission and deployed API Gateway to just accept safe, encrypted, and authenticated requests from the WordFinder cell app and cross them to a Lambda perform dealing with Amazon Bedrock entry. 
• Lambda: A Lambda perform was deployed behind the API gateway to deal with incoming net requests from the cell app. This perform was accountable for taking the provided enter, constructing the immediate, and submitting it to Amazon Bedrock. This meant that integration and immediate logic may very well be encapsulated in a single Lambda perform.

Advantages of API Gateway and Lambda

The mission workforce briefly thought of utilizing the AWS SDK for JavaScript v3 and credentials sourced from Amazon Cognito to instantly interface with Amazon Bedrock. Though this might work, there have been a number of advantages related to implementing API Gateway and a Lambda perform:

• Safety: To allow the cell consumer to combine instantly with Amazon Bedrock, authenticated customers and their related AWS Identification and Entry Administration (IAM) function would should be granted permissions to invoke the FMs in Amazon Bedrock. This may very well be achieved utilizing Amazon Cognito and short-term permissions granted via roles. Consideration was given to the potential of uncontrolled entry to those fashions if the cell app was compromised. By shifting the IAM permissions and invocation dealing with to a central perform, the workforce was in a position to improve visibility and management over how and when the FMs had been invoked.
• Change administration: Over time, the underlying FM or immediate would possibly want to alter. If both was exhausting coded into the cell app, any change would require a brand new launch and each consumer must obtain the brand new app model. By finding this throughout the Lambda perform, the specifics round mannequin utilization and immediate creation are decoupled and may be tailored with out impacting customers. 
• Monitoring: By routing requests via API Gateway and Lambda, the workforce can log and monitor metrics related to utilization. This allows higher decision-making and reporting on how the app is performing. 
• Information optimization: By implementing the REST API and encapsulating the immediate and integration logic throughout the Lambda perform, the workforce to can ship the supply phrase from the cell app to the API. This implies much less knowledge is shipped over the mobile community to the backend providers. 
• Caching layer: Though a caching layer wasn’t carried out throughout the system throughout the hackathon, the workforce thought of the flexibility to implement a caching mechanism for supply and associated phrases that over time would scale back requests that should be routed to Amazon Bedrock. This may be readily queried within the Lambda perform as a preliminary step earlier than submitting a immediate to an FM.

Immediate engineering

One of many core options of WordFinder is its skill to generate associated phrases and ideas based mostly on a user-provided supply phrase. This supply phrase (obtained from the cell app via an API request) is embedded into the next immediate by the Lambda perform, changing {phrase}:

immediate = "I've Aphasia. Give me the highest 10 most typical phrases which might be associated phrases to the phrase provided within the immediate context. Your response needs to be a legitimate JSON array of simply the phrases. No surrounding context. {phrase}"

The workforce examined a number of completely different prompts and approaches throughout the hackathon, however this primary guiding immediate was discovered to provide dependable, correct, and repeatable outcomes, whatever the phrase provided by the consumer.

After the mannequin responds, the Lambda perform bundles the associated phrases and returns them to the cell app. Upon receipt of this knowledge, the WordFinder app updates and shows the brand new listing of phrases for the consumer who has aphasia. The consumer would possibly then discover their phrase, or drill deeper into different associated phrases.

To keep up environment friendly useful resource utilization and price optimization, the structure incorporates a number of useful resource cleanup mechanisms:

• Lambda automated scaling: The Lambda perform accountable for interacting with Amazon Bedrock is configured to robotically scale all the way down to zero cases when not in use, minimizing idle useful resource consumption.
• Amazon S3 lifecycle insurance policies: The S3 bucket storing the user-uploaded pictures is configured with lifecycle insurance policies to robotically expire and delete objects after a specified retention interval, releasing up cupboard space. 
• API Gateway throttling and caching: API Gateway is configured with throttling limits to assist forestall extreme requests, and caching mechanisms are carried out to cut back the load on downstream providers similar to Lambda and Amazon Bedrock.

Conclusion

The QARC workforce and Scott Harding labored carefully with AWS to develop WordFinder, a cell app that addresses communication challenges confronted by people residing with aphasia. Their profitable entry on the 2023 AWS Queensland Hackathon showcased the facility of involving these with lived experiences within the growth course of. Harding’s insights helped the tech workforce perceive the nuances and affect of aphasia, resulting in an answer that empowers customers to seek out their phrases and keep linked.

References

Concerning the Authors

Kori Ramijoo is a analysis speech pathologist at QARC. She has intensive expertise in aphasia rehabilitation, expertise, and neuroscience. Kori leads the Aphasia Tech Hub at QARC, enabling individuals with aphasia to entry expertise. She gives consultations to clinicians and gives recommendation and assist to assist individuals with aphasia acquire and preserve independence. Kori can also be researching design concerns for expertise growth and use by individuals with aphasia.

Scott Harding lives with aphasia after a stroke. He has a background in Engineering and Pc Science. Scott is likely one of the Administrators of the Australian Aphasia Affiliation and is a client consultant and advisor on numerous state authorities well being committees and nationally funded analysis initiatives. He has pursuits in the usage of AI in growing predictive fashions of aphasia restoration.

Sonia Brownsett is a speech pathologist with intensive expertise in neuroscience and expertise. She has been a postdoctoral researcher at QARC and led the aphasia tech hub in addition to a analysis program on the mind mechanisms underpinning aphasia restoration after stroke and in different populations together with adults with mind tumours and epilepsy.

David Copland is a speech pathologist and Director of QARC. He has labored for over 20 years within the discipline of aphasia rehabilitation. His work seeks to develop new methods to know, assess and deal with aphasia together with the usage of mind imaging and expertise. He has led the creation of complete aphasia remedy applications which might be being carried out into well being providers.

Mark Promnitz is a Senior Options Architect at Amazon Internet Companies, based mostly in Australia. Along with serving to his enterprise clients leverage the capabilities of AWS, he can typically be discovered speaking about Software program as a Service (SaaS), knowledge and cloud-native architectures on AWS.

Kurt Sterzl is a Senior Options Architect at Amazon Internet Companies, based mostly in Australia.  He enjoys working with public sector clients like UQ QARC to assist their analysis breakthroughs.