Abstract

We create a full-featured two-dimensional side-scrolling game via context-refinement of LLM prompts. We evolve a modular architecture in order to build this agentic loop within a constrained environment. We then instruct an LLM agent to create a machine learning agent. This machine learning agent then learns to win the game via Reinforcement Learning (RL). The challenge of building the reinforcement learning engine highlights the game's playability and conceptual complexity. The resulting agent learns to beat the game's first level with consistency. With specialized training, the RL agent can complete the second level. Due to the game's challenging procedural dynamics, we further scrutinize the reward engine. The game is available to play live at: Lunar Liftoff

Introduction

Lunar Liftoff is a five-level, two-dimensional side-scrolling jump-and-shoot-style action game. To create the game, we used an iterative prompting strategy. This approach allowed us to build out the game features in a progressive manner. The game is playable and complete across all levels. The implementation includes a cinematic ending scene, rolling credits, and high score board. Each successive win of the game introduces new features. This includes new enemies, trap types, and boss skills. For example gameplay, see Appendix A (Human as player), and Appendix B (Machine Learning Agent as player).

We accomplished the entire design using vector graphics directly in the DOM. We did not use any external libraries. We prompted the LLM to create everything from scratch. This included rendering, game logic, music, sound effects, and animation sequences.

Except in a couple of isolated debugging instances, the LLM did not have access to any of the final code. This constraint restricted the model's context and enforced a modular development style. The model then struggled to remember code when files went over a certain length (~300-400 lines). Even so, this allowed us to build out the design in an iterative fashion. We extracted each successive completed feature into a module. The model was able to reason about these modules in a higher-level abstraction. This allowed us to refine the context while expanding the design.

System Geography

Here we show an LLM-generated system geography. Clusters form around key game concepts. We have granular Managers which communicate between the clusters.

Implementation Analysis & Trade-offs (Procedural Game)

Following a similar analysis to Architecting a High-Fidelity Digital Twin for Autonomous Fleet Telemetry, we analyze and categorize our agentic prompting loop. We leverage the LLM to reduce the prompts and code to six categories. We use the same nomenclature (Lines/Prompt, Prompts/File, Lines/File, Files/Prompt) as the previous article.

Agentic Reward Engineering

We took an iterative approach to designing the reward engine using Farama Foundation's Gymnasium, and DLR-RM's Stable Baselines3.

We bootstrapped the machine learning agent with a simple Python training loop. This set the expectations of the python packages to use. We then looped the LLM agent to evolve this new agent to play the game using Reinforcement Learning. The model first assessed that it could use OCR, but then realized it could access the game through the DOM.

Implementation Analysis & Trade-offs (Agentic Reward Engineering)

The first iteration (Phase 1) was a failure. The agent could not jump or collect rewards, which meant it could not advance very far. The LLM offered actionable insights. The model suggested certain improvements to the reward engine, but the memory footprint was huge, and the agent would crash when increasing the number of runs. We were also battling with an LLM with ideas of how the reward engine should evolve.

By Phase 2, the LLM offered some insights into the reward engine, but we were still learning from the screen via optical character recognition (OCR). This was inefficient and required a huge amount of memory. By Phase 3, the model learned it could exploit the DOM directly to give us information about the game. I was also impatient. The LLM recommended running for 100s of thousands to millions of iterations, which were slow due to the nature of the game levels. I preferred training in 10k steps, and by Phase 4 we had completely rewired the reward engine, including game-specific training models.

meta_agent:
  enabled: true 
  default_expert: "models/level1"
  experts:
    runner: "models/level_runner"
    boss_0: "models/level1"
    boss_1: "models/boss2_expert"

This allowed us to train the running levels (jumping with progressive velocity increase) separately from the boss fights (where jumping and firing was the ideal behavior). By fine-tuning the model on the level 2 boss, the RL agent quickly learned a "double jump" trick. This made some progress and we could now beat the level 2 boss about 2% of the time.

Conclusion

The main issue appears to be the procedural nature of the game and the limited amount of training runs. Massively increasing the amount of training runs may yield some improvement. However, the biggest improvement came from fine-tuning the training on specific game states (level running, boss fights).

Appendix A: Example Gameplay (Human as player)

Appendix B: Example Gameplay (Machine Learning Agent as player)