Designing Fully Immersive Experiences with Generative AI

The Brief

Full Immersion is the most effective way to learn languages. Despite this, most language learning apps heavily rely on translations. These designs are centred around crafting a fully immersive language learning app, along with Generative AI content, prompts, and marketing materials.

This project began with access to Emoji representations of words. However, Emojis alone lack the context needed for comprehensive language learning. Thanks to advancements in image-based Generative AI, it became possible to design a fully immersive (zero translation) experience.

UI Design

Designs were continuously updated, evolving from sketches to low-fidelity wireframes to higher-fidelity designs. These updates were driven by insights from interactive prototypes, new development considerations, usability testing, and user feedback. Updates included streamlining HomeScreen CTAs, updating interactions automatic interactions like Text-to-Speech and their options, and refining the algorithms that track a user’s progress and keeps learners engaged.

Animations

Animations can both add joy and help users formulate mental models of products. This is particularly important in a fully immersion language app, where text labels are restricted. One example of this is the use of animation to communicate to users that they have equal access to both Emoji Word puzzles, and Image-Based puzzles to progress any given section.

App Icon

The initial App Icon evolved as the priority of different concepts shifted. Concepts included playfulness, feeling of being a detective, encyclopedic index of words learnt (like a Pokédex), to an emphasis on the annotation aspect of emojis and words. In the end, the playfulness and encyclopedic aspects gained the highest priority.

A Quick Landing Page

In less than 1 hour, an old landing page was quickly redlined, improved, and implemented alongside front-end development. This was done for rapid App Store approval.

Tests & Optimizations

A/B and Multivariate tests are run on sets of App Store splash images. From this we gather insights like the fact that people are more attracted to the app when we don’t emphasize specific languages. These insights let us increase conversions which helps increase qualitative feedback.

In-Person Usability Test sessions, and Remote Beta Tests are also run on the overall app.

Human Curated Generative AI Content

Many of the design concepts explored for this app were only conceptually possible because of recent advancements in generative AI and other visual technologies. Many of the app’s human-curated images are facilitated by GenAI. OpenAI’s GPT4, GPT-4o, Dalle-3 (SD+HD) are used to holistically generate both questions and their associated images as part of a content generation pipeline. Both the questions and images are manually curated, cut, or edited. As batches of content are generated, the AI prompts to generate them are iteratively improved upon to get closer to what is humanly desired.

Overall

This app stands out with its commitment to 100% language immersion, offering over 3000+ images to facilitate intuitive learning and enabling self-paced, rapid user progression. The integration of generative AI unlocked creative possibilities that traditional methods could never achieve. The rapid design process involved continuous iteration on UI design, animations, marketing materials, and generated AI content pipelines. These dynamic adjustments were supported by UX research, interactive prototypes, and close collaborations across fields. The end result is a sophisticated, user-centered app that redefines the language learning experience.