Unsupervised Learning Achieves Efficient Exploration Via Self-Imposed Goals for Zero-Shot Tasks

Scientists are tackling the challenge of equipping reinforcement learning agents with the ability to learn efficiently without explicit task guidance. Octavio Pappalardo from University College London (UCL), alongside colleagues, demonstrate a novel unsupervised pre-training method that allows agents to set and pursue their own goals, accelerating learning in downstream tasks. This research is significant because it addresses the difficulty of training agents for broad distributions of tasks where solving everything beforehand is impossible , a common scenario when facing unknown or changing environments. Their approach, named ULEE, combines in-context learning with an adversarial strategy to maintain a challenging and effective training curriculum, ultimately yielding improved exploration, adaptation, and performance on XLand-MiniGrid benchmarks compared to existing methods.

This research is significant because it addresses the difficulty of training agents for broad distributions of tasks where solving everything beforehand is impossible, a common scenario when facing unknown or changing environments. Their approach, named ULEE, combines in-context learning with an adversarial strategy to maintain a challenging and effective training curriculum, ultimately yielding improved exploration, adaptation, and performance on XLand-MiniGrid benchmarks compared to existing methods.

Self-Supervised Goal Setting for Reinforcement Learning improves sample

The team achieved this breakthrough by focusing on agents that learn by autonomously setting and pursuing their own goals, addressing the core challenge of effectively generating, selecting, and learning from these self-imposed objectives. This method utilizes a post-adaptation task-difficulty metric to guide automatic goal generation and selection, contrasting with prior work that often evaluated goal desirability based solely on immediate performance. Experiments show that ULEE’s adaptive curriculum maintains goals at an intermediate difficulty, avoiding the pitfalls of overly easy or unsolvable tasks, and focuses training on more challenging scenarios to broaden the distribution of learnable tasks. Anticipating the need for test-time adaptation, the researchers leveraged meta-learning to explicitly optimize for efficient learning, bridging the gap between unsupervised settings and meta-learning approaches.
The research establishes a novel unsupervised meta-learning algorithm that meta-learns a base policy using an automatic curriculum of self-generated goals, guided by the newly developed post-adaptation difficulty metric. ULEE’s architecture comprises an in-context learner, a difficulty-prediction network estimating post-adaptation performance, an adversarial agent for proposing challenging goals, and a sampling strategy to select goals within a desired difficulty range. The resulting policy attains improved zero-shot and few-shot performance, and provides a strong initialization for longer fine-tuning processes, demonstrably outperforming learning from scratch, DIAYN pre-training, and alternative curricula. This breakthrough reveals a pathway towards foundation policies in reinforcement learning, equipping agents with transferable knowledge to address sample inefficiency and lack of generalization. The study unveils a system capable of continuously gathering useful data at scale in an unsupervised, open-ended fashion, leveraging it to acquire transferable capabilities and contend with limited information, such as partial observability or uncertain dynamics. The work opens exciting possibilities for developing agents that can quickly adapt to new tasks and environments, with implications for robotics, game playing, and other areas where autonomous learning is crucial, all while requiring minimal prior knowledge of the specific task at hand.

ULEE Adversarial Goal Generation for Reinforcement Learning enables

Scientists engineered ULEE, an unsupervised meta-learning method designed to enhance reinforcement agent pre-training and accelerate downstream task performance. To achieve this, the team developed an adversarial goal-generation network trained to propose challenging, yet achievable, goals for the pre-trained policy. This network was paired with a novel metric evaluating goal difficulty based on the agent’s performance after adaptation, contrasting with methods relying on immediate performance. Experiments employed the XLand-MiniGrid benchmark suite, utilising a curriculum guided by evolving estimates of post-adaptation performance, this allowed the system to maintain goals at intermediate difficulty, avoiding overly simple or unsolvable challenges.

The core of ULEE leverages an in-context learner, trained to solve the generated goals, and a difficulty predictor estimating performance following a fixed adaptation budget. This predictor, trained adversarially against the goal generator, provides feedback to refine goal selection and maintain a balanced curriculum. Data collection involved continuous interaction with the XLand-MiniGrid environments, with the agent autonomously generating and attempting to solve goals over multiple episodes. Performance comparisons demonstrated ULEE’s superiority, consistently outperforming learning from scratch, DIAYN pre-training, and alternative curricula, the resulting policy attained improved zero-shot and few-shot performance, and served as a strong initialisation for longer fine-tuning processes. This innovative methodology enables the development of foundation policies equipped with transferable knowledge, addressing key challenges of sample inefficiency and generalisation in reinforcement learning.

ULEE boosts reinforcement learning exploration and adaptation through

Scientists achieved substantial improvements in reinforcement learning through unsupervised pre-training, demonstrating a novel method called ULEE. The team measured exploration capabilities by assessing the percentage of goals reached from a set, μeval, under varying episode budgets of 1 to 20. Results demonstrate that ULEE’s pre-trained policy substantially outperforms random behavior and DIAYN pre-training, reaching more than twice as many goals at the 20-episode mark across all benchmarks (Fig 0.2). Ablation studies highlighted the importance of avoiding pre-training on trivial goals and showed that guiding the curriculum with post-adaptation performance, rather than immediate success, is increasingly beneficial as benchmark difficulty grows.

Variants employing an adversarial Goal-search Policy achieved the best results, while bounded goal sampling proved effective when goal search was random. Tests prove that ULEE’s pre-trained policy leverages interaction history to improve steadily during fast adaptation, reaching up to a 3× increase in mean return by the 30th episode in gradient-free, few-shot adaptation over 30-episode lifetimes (Fig 3a). Measurements confirm that ULEE outperforms all baselines and ablations in post-adaptation return, measured as the mean over the last 10 episodes by task percentile (Fig 3b). However, the most challenging out-of-distribution tasks remained difficult, with no return achieved on 60% of tasks in two of the three benchmarks.

Increasing the pre-training budget to 5 billion steps further improved ULEE’s post-adaptation performance, while DIAYN exhibited stagnation or decline. Furthermore, the study evaluated fine-tuning on fixed tasks, sampling 2048 environments from μeval. For budgets up to 1 billion steps, ULEE consistently outperformed training from scratch and DIAYN in mean, 40th percentile, and 20th percentile returns (Fig 0.4). The breakthrough delivers a strong initialization for supervised meta-reinforcement learning, with ULEE yielding higher returns on μeval tasks during meta-learning on μtrain, up to 5 billion steps (Fig 5). Finally, ULEE (fcounts) achieved a non-zero return on all classical MiniGrid environments after pre-training on 4Rooms-Small, as shown in Table 1, demonstrating generalization to new environment structures.

ULEE achieves robust transferable policy learning through diverse

Scientists have developed ULEE, an unsupervised meta-learning approach designed to create pre-trained policies with transferable capabilities. This method centres on an adversarial curriculum of self-imposed goals, training policies on tasks that are challenging yet solvable within a defined adaptation budget. Evaluations across reward-free grid worlds demonstrate ULEE’s superior performance in both zero-shot and few-shot scenarios, and it effectively initialises policies for fine-tuning on both fixed tasks and meta-learning distributions. The research establishes that ULEE generalises effectively to new goals, transition dynamics, and grid structures, scaling positively with both pre-training and adaptation resources.

The authors acknowledge a limitation in the current scope of the work, specifically regarding the handling of longer-horizon tasks. Future research will likely focus on incorporating hierarchical structures into the meta-learned policy to address these more complex scenarios. A complementary avenue for exploration involves integrating vision-language models to align pre-training with human-relevant tasks, potentially broadening the applicability of this approach to real-world problems. These developments could significantly advance the field of reinforcement learning, enabling agents to learn more efficiently and adapt more readily to novel environments and objectives.