Lilian Weng Proposes New Path for AI Self-Improvement: Harness Engineering May Precede Model Weight Optimization

Lilian Weng, former VP of Safety Research at OpenAI and co-founder of Thinking Machines Lab, published a blog post titled "Harness Engineering for Self-Improvement," systematically outlining a practical path for recursive self-improvement (RSI) in AI. She argues that in the near term, AI self-improvement may not start with directly modifying model weights, but rather with the Harness layer—the execution system that orchestrates model reasoning, tool calls, context management, and result evaluation.

Core Thesis: Harness as a Viable Starting Point for RSI

Weng defines Harness as "the execution system built around the base model," which determines how the model observes the environment, invokes tools, manages context, stores state, and evaluates results. She points out that the success of coding agent products like Claude Code and Codex has already demonstrated the critical role of Harness in AI deployment. Compared to early agent frameworks (LLM + memory + tools + planning + action), Harness engineering adds workflow design, evaluation, permission control, and persistent state management, making it closer to runtime and software system design.

Optimization Path: From Context to Harness Code

Weng outlines a progressive chain of Harness optimization: prompt → structured context → workflow → Harness code → optimizer code. Specific approaches include:

• Context Engineering: ACE (Agentic Context Engineering) treats context as a continuously updated "operating manual," maintained by three components: generator, reflector, and curator. MCE (Meta Context Engineering) further splits "how to manage context" and "context content" into two layers of optimization.
• Workflow Design: Systems like AI Scientist, ADAS, and AFlow treat agent workflows themselves as searchable optimization objects, evolving from human-designed processes to model-involved design, and finally to workflow structures entering the search space.
• Self-Improving Harness: The Self-Harness system enables the model to analyze its own failure modes and modify Harness configurations through three steps: weakness mining, Harness proposal, and proposal validation. Tests on models like MiniMax M2.5, Qwen3.5, and GLM-5 show that this method can learn differentiated configurations tailored to each model's weaknesses.
• Evolutionary Search: DGM (Darwin Gödel Machine) allows coding agents to directly modify their own Harness code repository. Experiments show that with Claude 3.5 Sonnet as the base, starting from a simple initial configuration, the evolved DGM agent improved from 20% to 50% on SWE-bench Verified and from 14.2% to 30.7% on Polyglot, reaching or exceeding human-designed levels.

Challenges and Boundaries

Weng candidly lists current bottlenecks for RSI:

• Evaluators are too weak and vague; only tasks with clear objective feedback (e.g., code, math) can currently run self-improvement loops.
• Lifecycle management of context and memory: the more autonomous the task, the more memory needs to be managed.
• Negative results are systematically ignored; models trained on success-dominated data may not be good at judging when to abandon hypotheses.
• Diversity collapse: evolutionary and reinforcement learning loops tend to converge to local optima.
• Allowing programs to modify system-level code breaks abstraction boundaries, and reward hacking remains a problem.

Long-Term Outlook: Harness Improvements May Be Internalized

Weng believes that Harness engineering will evolve toward a "meta-methodology"—optimizing the "mechanism for obtaining better answers" itself. A mature Harness can support the research loop of model self-improvement, while smarter models can prevent over-engineering of the Harness. Eventually, many Harness-level improvements may be internalized into core model behavior, but external context and tool interfaces are likely to remain. This pattern has precedent in prompt engineering history: as instruction tuning and reasoning capabilities improve, manual prompt tricks become less central, but the need to specify goals, constraints, context, and evaluation does not disappear.