واجهات المستخدم التوليدية: محاكاة نظام تشغيل عصبي باستخدام Gemini 2.5 Flash-Lite

The future of user interfaces may lie in generative AI, capable of creating dynamic and context-aware experiences on the fly. A new research prototype leverages Gemini 2.5 Flash-Lite to simulate an operating system where each screen is generated in real-time based on user interaction. This blog post delves into the core technical concepts behind this innovative approach to UI design, exploring how large language models can redefine the way we interact with computers.

On-the-Fly UI Generation with Contextual Awareness

The prototype uses Gemini 2.5 Flash-Lite to generate UI elements dynamically. Critical to this process is conditioning the model to understand both UI structure and user context. This is achieved using a two-part prompt:

• UI Constitution: A system prompt containing fixed rules for UI generation, defining elements such as OS-level styling, home screen format, and logic for embedding elements like maps. This establishes consistency.
• UI Interaction: A JSON object capturing the user’s most recent action (e.g., a button click). This serves as the specific query prompting the model to generate the next screen, providing contextual relevance. The object includes details like:
  • id: Unique identifier of the interacted element.
  • type: Type of interaction (e.g., ‘button_press’).
  • value: Data associated with the interaction, such as text from a text area.
  • elementType: HTML tag of the element clicked.
  • elementText: Visible text inside the clicked element.
  • appContext: The application context the user is currently in.

By combining the UI constitution and UI interaction data, the model maintains a consistent look and feel while adapting to specific, real-time user inputs.

Interaction Tracing for Enhanced Context

Beyond individual interactions, the prototype utilizes interaction tracing to consider the sequence of past actions.

• The system maintains a trace of the past N interactions.
• This allows the model to generate screens that are more contextually relevant.
• For example, the content within a calculator app can adapt based on whether the user previously interacted with a shopping cart or a travel booking app.
• The length of the interaction trace can be tuned to balance contextual accuracy with UI variability.

Streaming for a Responsive Experience

To ensure a seamless user experience, the prototype implements streaming and progressive rendering.

• Instead of waiting for the entire UI to be generated, the model streams HTML code in chunks.
• These chunks are continuously appended to the component’s state.
• React then re-renders the content, allowing the browser to display valid HTML elements as soon as they are received.
• This creates the perception of an interface materializing almost instantly, improving responsiveness.

Achieving Statefulness with a Generative UI Graph

While the model generates new screens with each input by default, this can lead to a stateless and non-deterministic experience. To address this:

• The system incorporates an in-memory cache to model a session-specific UI graph.
• When a user navigates to a screen that has already been generated, the system retrieves the stored version from the graph.
• This avoids querying Gemini again, providing a more consistent experience.
• When the user requests a new screen, the UI graph grows incrementally.
• This method provides state without sacrificing the quality of the generative output.

Potential Applications

This research prototype demonstrates a novel approach to UI design with promising applications:

• Contextual shortcuts: The system can observe user interaction patterns and generate ephemeral UI panels to accelerate tasks, such as dynamically creating buttons for comparing prices or booking flights when the user is comparing flights across multiple websites.
• “Generative mode” in existing apps: Developers can add a “generative mode” to their applications, enabling just-in-time UI generation for specific tasks. For example, in Google Calendar, a “generative mode” could present the best alternative meeting times as selectable buttons based on attendee schedules.

This exploration highlights the potential of generative interfaces and the exciting possibilities they offer for the future of human-computer interaction. As models become faster and more capable, this field promises continued innovation and development.

Source: Google