The failure may come from missing context, weak tools, incomplete retrieval, poor validation, or an environment the agent cannot inspect. Those are harness problems.

I open-sourced Harneloop to give agents a structured way to improve that system through real attempts and verifiable evidence.

How Harneloop works

The operating agent attempts the target task, captures the resulting artifacts and traces, evaluates the outcome, and follows likely mistakes back through the run.

It then creates an isolated candidate change to the harness.

The candidate can change instructions, tools, retrieval, examples, validators, environment automation, skills, or other task-specific infrastructure. It is tested separately and promoted only if the evidence supports an improvement without unacceptable regressions.

This keeps experimentation flexible while preventing the agent from silently rewriting the current working harness.

Artifact-aware evaluation

Many useful agent tasks produce outputs that cannot be judged from the transcript alone.

An interface can compile but be visually wrong. A generated document can be valid but ineffective. A scene can render while its spatial relationships are incorrect. A browser agent can execute every action but leave the application in the wrong state.

Harneloop treats those artifacts as evidence. It links them to the run that produced them so the agent can inspect the actual result and diagnose the likely system-level cause.

Portable harness units

Harneloop stores each task-specific development environment as a harness unit.

A harness unit contains:

• the task goal and current operational map;
• reusable instructions, tools, retrieval, and validators;
• candidate changes and evidence references;
• regression cases and promoted versions;
• custom infrastructure the agent needs for that task.

The framework protects lifecycle state and promotion integrity without forcing every task into a single template. Successful harness units can be restored, continued, packaged, moved, or exported into another agent environment.

Harness-first does not mean fine-tuning never matters

SWE-agent, Reflexion, and Self-Refine all provide evidence that interfaces, feedback, memory, and iterative refinement can materially change model performance without additional training.

The 2026 Self-Harness paper goes closer to Harneloop's core direction: agents mined weaknesses from execution traces, proposed harness modifications, and accepted them after regression testing.

Fine-tuning can still be useful for specialized behavior, style, latency, representations, or capabilities that cannot be supplied effectively at inference time. A fine-tuned model can also benefit from a strong harness.

The useful decision rule is:

If failures come from context, tools, retrieval, feedback, validation, memory, or environment interaction, improve and measure the harness before changing model weights.

The first measured use case

Harneloop was first used to develop ViperMesh, a Blender harness for spatial reasoning and scene generation.

The acting model was held constant at OpenAI GPT 5.5 High. Compared with the Anthropic x Blender MCP server baseline across seven comparable live tasks, the evolved ViperMesh harness:

• was faster on 6 of 7 tasks;
• achieved a 2.534x mean speedup;
• improved preliminary neutral LLM visual evaluation by 8.19 points.

Harneloop did not generate the scenes. It structured the evidence loop used to diagnose failures, develop the harness, and verify improvements.

The full case study includes the methodology and limits. This is one task family, not a universal comparison against every fine-tuning method.

Try it

Harneloop is designed for agent-led onboarding. Give the GitHub repository to a capable coding agent and describe the task you want to improve. The agent can install it, map the actual environment, and create the first harness unit.

I am looking for independent use cases beyond Blender, including negative results. Feedback about onboarding, environment mapping, evidence quality, token cost, candidate decisions, and capability limits will directly shape the framework.

Repository: https://github.com/Ker102/Harneloop
Case study: https://www.kristoferjussmann.me/case-studies/vipermesh

📉 Anti-Pattern 1: The "Emotional Blackmail" Scaffold (14%)

Caption: Anti-Pattern Prevalence in 28,000 Production System Prompts. (Full-width hero chart)

Over 14% still contain emotional appeals:

"Take a deep breath. If you miss a bug, the company will lose millions."

Why it fails: Modern RLHF has trained out the "anxiety" response. Emotional context distracts self-attention from the actual task.

🏗️ Anti-Pattern 2: The "Just in Case" Clause (62%)

62% of prompts over 300 words contained contradictory constraints. Our Study E data proved short prompts (<50 words, scoring 80.1/100) consistently outperform long ones (>300 words, 66.9/100).

🎭 Anti-Pattern 3: The "World Class Expert" Trap (80%)

Nearly 80% started with "Act as a world-class expert." Our Study C proved this provides zero lift on modern models (~78/100 with or without it).

🚀 The Fix: The 50-Word Rule

1. State the role (10 words): "Extract data from SEC filings."
2. State negatives (20 words): "Do not include pleasantries. Do not output markdown."
3. Halt.

PromptTriage compresses 500-word prompts to the optimal 50-word framework.

We spend hours tweaking the words in our prompts, but how much thought do we give to the structure? If you ask an AI to return data in JSON vs. Markdown, or if you write a concise 50-word prompt vs. a detailed 500-word prompt, does the quality of the reasoning actually change?

To find out, I ran Study E v2: The Format Wars.

I subjected 5 frontier models to 1,080 rigorous evaluations across 12 distinct task domains (coding, math, data extraction, analysis, creative writing, and more). Every single evaluation was scored blindly by a 3-judge LLM jury on a 100-point scale.

The results completely changed how I build AI applications.

🔬 The Setup: 1,080 Evaluations

We tested five heavyweight models:

• GPT-5.4 (OpenAI)
• Nemotron 3 Super 120B (Nvidia)
• Claude Sonnet 4.6 (Anthropic)
• Gemini 3.1 Pro (Google)
• Qwen 3.5 397B (Alibaba)

For each model, we ran 216 evaluations testing 18 unique prompt configurations:

• 6 Formats: Plain Text, Markdown, XML, JSON, YAML, Hybrid (Text + Code Blocks)
• 3 Lengths: Short (<50 words), Medium (~150 words), Long (>300 words)

The scoring was handled by a ruthless 3-judge panel (Llama 4 Maverick, Claude Opus 4.6, and Atla Selene Mini) grading on instruction following, reasoning quality, formatting adherence, and edge-case handling.

🏆 Finding 1: The Model Rankings

Before looking at formats, how did the models perform overall across all 18 permutations?

1. 🥇 GPT-5.4: 88.1 / 100 — Won 10 out of 12 task domains
2. 🥈 Nemotron 120B: 85.1 / 100 — Won 1 domain (Data Extraction), extremely close to GPT-5.4
3. 🥉 Claude Sonnet 4.6: 69.5 / 100
4. Gemini 3.1 Pro: 62.6 / 100 — Won 1 domain (Question Answering)
5. Qwen 397B: 61.0 / 100

Takeaway: GPT-5.4 is the undeniable reasoning king right now. But Nvidia's Nemotron 120B is a shocking powerhouse—it scored incredibly close and actually beat GPT-5.4 outright in Data Extraction tasks. If you aren't testing Nemotron in your pipelines, you are missing out.

🧱 Finding 2: The Best Format is... JSON?

If you want the highest quality reasoning and instruction following from an LLM, what format should you ask it to return?

1. YAML: 74.6 / 100
2. JSON: 74.4 / 100 (Statistical tie with YAML)
3. Hybrid: 73.5 / 100
4. XML: 73.3 / 100
5. Markdown: 72.9 / 100
6. Plain Text: 70.8 / 100

Takeaway: Asking the model to structure its output in JSON or YAML doesn't just make it easier for your code to parse—it actually improves the model's reasoning.

Why? Forcing the model into a strict structural schema (like JSON keys) acts as a cognitive scaffold. It forces the model to categorize its thoughts before generating output, leading to fewer hallucinations and better instruction adherence. Plain unstructured text performed the worst across the board.

But here's the nuance: different models prefer different formats:

Note: While JSON was the best overall, Nemotron and Qwen actually performed slightly better when outputting YAML.

📏 Finding 3: The Prompt Length Paradox

We've been trained to write massive, highly detailed "megaprompts" with endless context. But the data reveals a startling paradox:

• Short Prompts (<50 words): 80.1 / 100
• Medium Prompts (~150 words): 72.8 / 100
• Long Prompts (>300 words): 66.9 / 100

Takeaway: Across all 5 models and all 6 formats, short prompts absolutely demolished long prompts.

When you flood the context window with too many instructions, constraints, and examples, the model suffers from attention dilution. It forgets the primary objective and gets bogged down trying to satisfy secondary constraints.

The worst combination in the entire study? Qwen 397B given a Long prompt asking for Plain Text (38.8/100).

🏅 Finding 4: The Best and Worst Combinations

What are the absolute best and worst model + format + length trios?

The Golden Combo scored 92.2 / 100: GPT-5.4 + Hybrid Output + Short Prompt.

🚀 The Ultimate Prompting Formula

If you want to maximize the performance of a modern LLM, the data points to a clear formula:

1. Keep it brief: State your objective clearly in under 50 words. Drop the fluff.
2. Demand structure: Always ask the model to return its answer in JSON or YAML. Avoid asking for unstructured text.
3. Use the right model: GPT-5.4 for general reasoning/coding, Nemotron 120B for extraction.

I built PromptTriage specifically to help developers automatically refactor those bloated 500-word metaprompts down into the high-scoring "Short + Structured" format this data proves works best.

Data lovers: The full 1,080-row dataset and analysis script are open-sourced in the PromptTriage repo.