باستا: توليد صور تعاوني من خلال التعلم المعزز التكيفي للتفضيلات

In the ever-evolving landscape of text-to-image (T2I) generation, a common frustration arises: the difficulty in perfectly translating a user’s unique vision into a final image. While T2I models possess incredible power, capturing the nuances and specific desires of an individual creative intent with a single prompt remains a challenge. This blog post delves into Google Research’s innovative solution, PASTA (Preference Adaptive and Sequential Text-to-image Agent), a reinforcement learning agent designed to collaboratively refine T2I results through an interactive dialogue with the user. This approach moves away from trial-and-error prompt refinement and aims for a more intuitive and satisfying image generation experience.

Introducing PASTA: A Collaborative Image Generation Agent

PASTA utilizes reinforcement learning to learn a user’s unique preferences through multi-turn interactions, leading to more personalized and satisfying image generation. The core idea is to transform image generation from a one-shot process into a collaborative conversation between the user and the AI. This is achieved by allowing the user to iteratively guide the image generation process by selecting images that align closest with their vision.

• PASTA stands for “Preference Adaptive and Sequential Text-to-image Agent.”
• It uses reinforcement learning (RL) to refine T2I outputs based on user feedback.
• Eliminates the need for excessive prompt engineering and trial-and-error.
• Uses a novel user simulation technique to generate training data.
• A foundational dataset containing over 7,000 human rater interactions is released.

The Power of User Simulation for Training PASTA

A significant hurdle in training an AI agent to adapt to individual preferences is the need for a large and diverse dataset of user interactions. Gathering such data from real users poses challenges related to privacy and scalability. To overcome this, PASTA employs a two-stage training strategy that combines real human feedback with large-scale user simulation.

• A foundational dataset of over 7,000 real user interactions with prompt expansions generated by Gemini Flash and images from Stable Diffusion XL (SDXL) was collected.
• This dataset was used to train a sophisticated user simulator.
• The user simulator comprises a utility model (predicts user liking) and a choice model (predicts user selection).
• An expectation-maximization algorithm is used to learn user preferences and identify latent user types.
• The trained simulator generates over 30,000 simulated interaction trajectories.

How PASTA Interacts with Users

Once trained, PASTA engages users in a visually grounded dialogue to iteratively refine images based on their preferences. The process starts with an initial user prompt. PASTA then uses a candidate generator (a large multimodal model) to create a diverse set of potential prompt expansions. An RL agent then selects an optimal set of expansions, generating corresponding images that are presented to the user. The user selects the image that best aligns with their vision, providing feedback that guides PASTA’s next set of suggestions.

• User provides an initial prompt.
• PASTA’s candidate generator proposes prompt expansions.
• The RL agent selects the best slate of expansions (four in the example).
• Corresponding images are generated and presented to the user.
• User selects the preferred image, providing feedback for the next iteration.

Evaluating PASTA’s Performance

The effectiveness of PASTA was rigorously evaluated through human evaluations, comparing various training approaches to a baseline model (Gemini Flash and SDXL without further training). Three versions of PASTA were trained: on real data, on simulated data, and on a combination of both. The results revealed that training PASTA on both real and simulated data significantly outperformed the baseline and the versions trained solely on one type of data.

• PASTA was trained using implicit Q-learning (IQL).
• Three training approaches were compared: real data only, simulated data only, and a combination.
• Evaluation metrics included Pick-a-Pic accuracy, Spearman’s rank correlation, choice model accuracy, and cross-turn accuracy.
• The combined real and simulated data training yielded the best performance, surpassing the baseline in all tested metrics.
• In direct comparisons, 85% of raters preferred PASTA’s generated images over the baseline, especially with abstract prompts.

The Future of Collaborative Image Generation

PASTA demonstrates the potential of interactive, preference-adaptive, and collaborative generative AI. The user simulation techniques developed hold promise for application across various generative tasks, facilitating the creation of AI systems that better align with human needs and preferences. The researchers have open-sourced their sequential rater dataset and simulated user data to further accelerate research in this area.

In conclusion, PASTA represents a significant step towards more intuitive and personalized image generation. By leveraging reinforcement learning and innovative user simulation, PASTA empowers users to collaboratively craft images that perfectly match their unique creative vision.

Source: Google Research