Microsoft white paper demonstrates: swapping out the Harness makes the same Claude perform noticeably worse

Author: Tencent Technology

On November 7, Microsoft released a technical white paper on Agent execution systems, including a set of somewhat unexpected test results: using the same Claude Opus 4.8, simply replacing the Harness behind Copilot Studio improved the pass rate for complex tasks by 22.1 percentage points. The new system nearly doubled the number of model calls per task, while normalized input Token consumption could be reduced by 21%.

Just ten days earlier, Microsoft had announced that it would roll out a unified Copilot Harness across multiple products and sell hosted services to developers through its cloud platform.

On November 5, Moonshot AI just released Kimi Hosted Agents, formally introducing Harness as a Service. In August, DeepSeek open-sourced its own Harness; OpenAI and Anthropic had already turned their internal runtime systems into developer-callable services even earlier.

The runtime system behind large models is starting to become the key player model companies are watching.

The Rush to Build Harnesses

Harness can be understood as an Agent runtime system built around large models. It enables models to use external tools, execute multi-step tasks continuously, and adjust subsequent actions based on execution results. For Agents that need to run for extended periods, Harness is also responsible for saving task state so that work can continue after interruptions.

It sits at a different layer from AI applications. For example, products such as Claude Code and WorkBuddy are user-facing, while the underlying Harness can be reused by different applications and offered as an independent service to developers. As the work handled by Agents becomes increasingly complex, Harness is gradually becoming an important factor affecting task success rates, execution efficiency, and operating costs.

Anthropic is one of the earlier model companies to invest in Harness development, initially to address problems encountered by Claude Code in long-running programming tasks.

At the end of 2025, Anthropic found that once Claude's context window was exhausted, a new conversation often could not continue the previous work. Engineers then improved the Harness behind Claude Code so that the Agent saved progress before the end of each stage, and the next round could read these records to continue unfinished tasks. This way, even if one model session ended, the overall programming task could keep moving forward.

In April this year, Anthropic further launched Managed Agents, offering this Agent runtime capability to developers as a cloud service. To enable enterprises to run Agents over the long term and with stability, Anthropic also adjusted the underlying architecture: Harness drives task execution, historical records can be stored separately, and the sandbox in which AI programs run can be replaced at any time. Even if the sandbox fails, the Agent can recover its previous work.

From an internal runtime system for Claude Code to a hosted service for external developers, Anthropic turned its years of accumulated Harness capabilities into a standalone product.

OpenAI has also chosen to open the Harness behind Codex to more developers. In February this year, OpenAI disclosed that Codex's web version, command-line interface, and IDE extensions actually share the same Agent runtime system. In August, it further positioned Codex as an open Harness platform, allowing third-party applications to connect directly. The Agents API launched in September is hosted and maintained by OpenAI, allowing developers to connect directly to the Agent runtime environment.

This route allows OpenAI to serve users directly through Codex while also providing underlying execution capabilities to other software companies. Developers can keep their own product interfaces and business workflows while letting Codex complete specific tasks behind the scenes.

DeepSeek open-sourced the entire Harness framework code under the MIT License on August 13. It proposed "Everything is a Plugin", nearly decomposing the entire Agent runtime system into freely combinable plugins. In addition to replacing models and external tools, developers can also modify how Agents call models and execute tasks, and even allow Agents to write and install new plugins themselves.

This Cordis-based architecture aims to let developers build different types of Agents according to their needs without having to build a runtime system from scratch.

On November 5, Kimi officially announced that Hosted Agents provide an always-on cloud environment, enabling enterprises to deploy Agents within internal systems or embed them in externally sold products, and explicitly proposing Harness as a service.

Harness Evolution

In fact, the momentum behind Harness did not start with model companies.

Earlier this year, the popularity of OpenClaw showed the industry that a company without a foundation model can still attract a large user base through an Agent runtime system. In February, OpenAI recruited OpenClaw founder Peter Steinberger to work on the development of next-generation personal Agents.

Subsequently, Nous Research launched Hermes Agent, attempting to let Agents save experience accumulated during work as reusable Skills and continuously improve in subsequent tasks.

In late March, Stanford University and other institutions proposed Meta-Harness, enabling AI to directly modify Harness code based on failure records and validate the results through testing. In a text classification experiment, accuracy increased by 7.7%, and context token consumption fell to one-quarter of the original.

Why are these explorations worth model companies' investment? In October, Nanyang Technological University in Singapore tested 66 combinations of models and Harness, finding that changing the Harness could even alter the performance rankings among models.

FrontierHarness Eval also found that for the same Kimi K3, the task pass rate was 66.7% with Codex, 63.3% with DeepSeek Harness Creator, and 56.7% with Kimi Code.

For foundation model companies, improving model performance increasingly depends not only on improving the model itself, but also on the design and optimization of the Harness.

Chart: Comparison of task pass rate and cost for the same Kimi K3 under different Harness configurations

Researchers are also trying to use experience accumulated by Harness for model training. Co-Harness, proposed in July, first improves Harness based on failure records and then uses the optimized execution data to train models. HarnessFlow, announced in November, attempts to automatically search for better Harness designs, raising average scores by 10.89 percentage points across six task categories.

Chart: Timeline of Harness technology evolution (late 2025 to November 2026)

However, automated optimization also carries risks. A study in early November found that in a specific experiment, after a Harness was optimized for new tasks, its previous SWE-bench score fell from 94% to 32%. Model upgrades may also render previously effective designs obsolete.

This means that technical accumulation in Harness is unlikely to be a one-time solution. Maintaining stability across different tasks during continuous improvement remains an unsolved challenge.

Industry sources say this is also a major pain point in deploying open-source models. As a result, model companies have begun selling Harness services. Open-source frameworks can be obtained for free, but enterprises still need to bear deployment and maintenance costs to run Agents stably over the long term. Companies such as Kimi and Anthropic have seen an opportunity to take over this work and turn Harness into a standalone business, though their approaches vary significantly.

The Hidden Value of Harness

Harness is starting to become a separately billable service, but current pricing tables show that model inference remains the primary source of charges.

Anthropic's Managed Agents charges 0.08 USD per active session hour, with tokens billed separately. According to the official one-hour programming task example, the total cost is 0.58 USD, of which the managed runtime fee accounts for only about 14%.

Kimi's fees are divided into model tokens, sandbox runtime, and paid plugins. For one published investment-banking legal analysis task, the estimated cost is about 30 CNY, with sandbox fees below 0.1 CNY. OpenAI waives additional service fees during the Agents API public beta, primarily charging based on model and tool usage. DeepSeek has open-sourced Harness under the MIT license and has no direct pricing logic.

Figure: Comparison of Harness pricing models and commercial positioning across six model companies

Therefore, the direct revenue that model companies currently earn from Harness may be negligible.

So why continue investing substantial R&D resources?

One important reason is that after enterprises actually deploy agents, demand for Harness may be more stable than demand for any specific model.

U.S. legal services company ABC Legal originally deployed agents on employees' computers and ran them via scheduled tasks. As usage expanded, CTO Brandon Fuller began looking for a platform that could centrally manage these agents. He compared multiple options and ultimately chose Anthropic's Managed Agents because the company did not want to continue maintaining fragmented runtime environments.

According to Anthropic, ABC Legal brought more than 50 agents into production within one month, and approximately 310 employees use Claude daily. Employees can create new agents through configuration without needing software development knowledge.

This shows that the appeal of managed Harness comes from real enterprise demand. Rather than developing, deploying, and maintaining runtime systems from scratch over the long term, enterprises may prefer to purchase ready-made services and direct engineering resources toward their own business.

However, enterprises may not necessarily want to be locked into a single model company as a result.

In August this year, VentureBeat published a survey of 107 enterprises. 85% of surveyed enterprises use two or more agent orchestration platforms simultaneously. When selecting platforms, 29% of enterprises value the flexibility of models and tools most, while only 10% prioritize native compatibility with a leading model. Although the sample leans toward large technology companies, it already reflects a procurement tendency: customers want to preserve the freedom to switch models.

This provides another business logic for Harness competition. If enterprises are willing to use a runtime system over the long term, providers may continue to earn revenue from hosting, maintenance, and enterprise development services even if they cannot monopolize model calls, and may build more stable relationships with customers.

In this way, Harness can access the execution process of enterprises' actual work. Compared with a simple model API, Harness can record the work trajectory of a task from start to finish, including model calls, tool executions, failure retries, and final results. These records can help enterprises troubleshoot failures and provide valuable feedback for model development.

Figure: Four categories of enterprise task execution data recorded by Harness

If model companies can continuously obtain validated real-task feedback, they have an opportunity to discover issues with models in production environments more quickly. However, this advantage is also subject to explicit restrictions on data rights.

This also raises a new question for model companies: if enterprises can freely switch underlying models and insist on controlling their own data, how much long-term advantage can Harness still establish?

For model companies, Harness may bring longer-term value than direct billing. As enterprises delegate more and more work to agents, whoever provides a reliable runtime system has the opportunity to influence model selection and how AI participates in real business.

A familiar and clichéd term: entry point.

However, this term, deeply rooted in the internet era, always touches a sensitive nerve in AI-era competition. Today, Harness in some sense carries this expectation and imagination.

Source: TechFlow