Research · Technical report · October 2026

Allocation, not automation: efficiency gains from task-specific LLM harnesses

How much of the efficiency gain from language models comes from the harness around them, and why allocating each task to the right person or machine matters more than the model.

Authors
Immersive Reality Studio Research
Published
October 2026
Sources
12 cited papers and reports

Abstract

Most of the measured gains from language models at work do not come from the model alone. They come from what surrounds it: the interface it acts through, the context it is given, the checks it passes, and the decision of which model, tool or person should take each piece of work. We call this surrounding system a harness, and its routing function the allocation layer.

We bring together published evidence from software engineering, medicine, model routing, labour economics and human–AI collaboration. The same model resolves up to 6.7 times more software issues inside a purpose-built agent interface than with retrieval alone1; a general model steered by a task-specific prompting harness exceeds a fine-tuned specialist on medical licensing questions3; learned cascades match the strongest model at 59–98% lower cost4, and learned routers cut cost by up to 3.66 times while keeping 95% of its quality5. Field experiments show large gains when work sits inside a model’s competence and losses when it does not11, and a meta-analysis of 106 experiments finds that blending human and AI judgement is, on average, worse than giving each case to whichever performs better8. We argue that the main lever for efficiency is per-task allocation between people and machines, and we describe the allocation layer we build in Mixa and Membrane.

1. The allocation layer

An organisation produces a continuous stream of work: messages, requests, documents, orders, readings from devices. Today most of it is routed by habit: whoever opens the inbox, whichever tool is to hand. A language model added to that stream as a general assistant inherits the same routing problem; it is asked everything, and it is the right executor for only part of it.

The allocation layer sits between the stream and the executors. For each unit of work it decides who or what should do it: a small model, a frontier model, a tool or API, a device, or a person. It also decides which context and instructions go with it. Formally, for a task x it chooses an executor e from people and machines to minimise expected cost subject to a quality target and to the approvals the organisation requires. This generalises two lines of work: cascades that try cheaper models before expensive ones4, and learning to defer, where a model learns when to hand a case to a human expert9.

Email and messagesRequests from staffDocumentsCalendars and ordersSensors and devicesAllocationlayerManager approvalSpecialistSmall modelFrontier modelTools and APIsDevicesPeopleMachinesIncoming work
Figure 1. The allocation layer. Work from many sources enters on the left; the layer attaches task-specific context and sends each item to a person (orange) or a machine executor (blue). Approvals return through the same layer.

2. The same model, a better harness

Benchmarks are usually read as rankings of models. Read differently, they show how much of the result belongs to the harness. On SWE-bench, which asks systems to resolve real GitHub issues2, GPT-4 Turbo resolves 2.7% of the Lite tasks when given retrieved files and asked for a patch. Given a shell, it resolves 11.0%. Given an agent-computer interface designed for the task (search, file viewing and editing commands with feedback built for a model rather than a person) it resolves 18.0%1. The interface transfers: Claude 3 Opus rises from 4.3% to 13.0% with the same harness.

Medicine shows the same pattern without any new tools. Medprompt combines nearest-neighbour example selection, model-written reasoning chains and choice-shuffled ensembling. With it, GPT-4 reaches 90.2% on MedQA, ahead of Med-PaLM 2 at 86.5%, a model fine-tuned for medicine, and nine points above GPT-4 with a standard five-shot prompt3.

SWE-bench Lite% of issues resolved

  1. SWE-agent · GPT-4 Turbo18.0
  2. SWE-agent · Claude 3 Opus13.0
  3. Shell-only agent · GPT-4 Turbo11.0
  4. Retrieval only · Claude 3 Opus4.3
  5. Retrieval only · GPT-4 Turbo2.7

Axis 0–20%. 300 tasks.

MedQA (USMLE)accuracy %

  1. GPT-4 · Medprompt harness90.2
  2. Med-PaLM 2 · fine-tuned specialist86.5
  3. GPT-4 · five-shot prompt81.4
  4. Flan-PaLM 540B67.6

Axis 50–100%. Four-option questions.

Figure 2. Task-specific harnesses versus general use of the same models. Blue bars use a harness built for the task. Left: SWE-bench Lite, data from Yang et al.1 Right: MedQA four-option, data from Nori et al.3

3. Routing across models

Not every task needs the most capable model. FrugalGPT learns a cascade that queries inexpensive models first and escalates only when an answer looks unreliable. It matches the accuracy of the best single model at a fraction of the cost: 98.3% less on a financial headlines dataset, 73.3% less on legal overruling classification and 59.2% less on conversational question answering4. RouteLLM learns a router from human preference data and reports up to 3.66 times lower cost than sending every query to GPT-4 while keeping 95% of its quality on MT Bench5.

DatasetCost to reach the best single model’s accuracySaved
HEADLINESbest single model: GPT-4GPT-4 alone · 33.1Cascade · 0.698.3%
OVERRULINGbest single model: GPT-4GPT-4 alone · 9.7Cascade · 2.673.3%
CoQAbest single model: GPT-3GPT-3 alone · 72.5Cascade · 29.659.2%

Learned router vs. GPT-4 for every querycost saving at a quality target

MT Bench3.66×at 95% of GPT-4 quality
GSM8K1.49×at 87% of GPT-4 quality
MMLU1.41×at 92% of GPT-4 quality
Figure 3. Cost of matching the strongest model. Top: cost for a learned cascade to reach the accuracy of the best single model, relative to using that model alone (FrugalGPT4). Bottom: cost saving of a learned router against GPT-4 at a fixed quality target (RouteLLM5).

4. Where the work is

The size of the opportunity depends less on the model than on what is built around it. Eloundou et al. estimate that with an LLM alone about 15% of US workers’ tasks could be completed significantly faster at the same quality; with software and tooling built on top of LLMs the share rises to between 47% and 56%6. Around 80% of the workforce is in occupations with at least a tenth of tasks exposed, and 19% in occupations with at least half6.

Observed use is already broad but uneven: roughly 36% of occupations use AI for at least a quarter of their tasks, and 57% of usage is augmentation, with a person in the loop, against 43% automation7.

~15%of tasks a worker does could be completed significantly faster at the same quality with an LLM alone

47–56%with software and tooling built on top of LLMs

Figure 4. Share of tasks, out of 100, that could be done significantly faster at the same quality. Left: a language model alone. Right: with software and tooling built on LLMs, 47% (solid) to 56% (light). Data from Eloundou et al.6

5. People and machines: allocate, don’t blend

Field experiments confirm large gains when work is inside a model’s competence. Customer support agents resolved 14% more issues per hour with an AI assistant, and novices 34% more10; professionals finished writing tasks 40% faster with 18% higher quality12. Consultants completed 12.2% more tasks, 25.1% faster and at over 40% higher quality, but on a task chosen to sit outside the frontier they were 19% less likely to be correct11.

+14%

issues resolved per hour by 5,179 support agents; +34% for novice and low-skilled agents10

−40%

time on professional writing tasks, with +18% output quality12

+12.2%

more consulting tasks completed, 25.1% faster and over 40% higher quality, inside the frontier11

−19%

less likely to reach a correct answer on a task outside the frontier when consultants used AI11

A meta-analysis of 106 experiments explains why allocation matters more than assistance8. On average, human–AI combinations performed worse than the better of the two alone. When the human was better than the AI, the combination gained; when the AI was better, the combination lost. Decision tasks lost most; content-creation tasks gained. The practical reading is to send each case to whichever executor is stronger for it, and keep people where their judgement or approval adds value, rather than putting a person and a model on every task together.

−1.0−0.50+0.5+1.0
All studiesg = -0.23
Human better than AI aloneg = +0.46
AI better than human aloneg = -0.54

Hedges’ g for human–AI synergy (combination vs. the better of human or AI alone), with 95% confidence intervals. Left of zero: the combination did worse.

Figure 5. Human–AI synergy depends on who is better at the task. Effect sizes and 95% intervals from Vaccaro, Almaatouq and Malone8.

6. What the layer has to do

Taken together, the evidence suggests five properties for an allocation layer in a real organisation:

  1. Task-specific harnesses. Each recurring kind of work gets its own interface, context and checks, as in SWE-agent and Medprompt1,3.
  2. Cost-aware routing. Cheaper models first, escalation on uncertainty4,5.
  3. Deferral to people. A learned or rule-based decision to hand a case to the person best placed to do it9.
  4. Frontier awareness. Tasks outside a model’s competence are recognised and kept with people11.
  5. Approval and record. Actions with outside effects wait for a person, and every allocation is logged with its reason.

Mixa implements these as operators with duties, hard limits and approvals, and Membrane extends them to an organisation’s systems and devices. We measure allocation by cost per completed task, quality against a reviewed reference, time to completion and the share of work that needed a person.

7. Limitations

The figures combine different benchmarks, models and years, and are not directly comparable with one another. Benchmark gains do not translate one-to-one into workplace gains, and field experiments cover specific occupations. Exposure estimates measure technical feasibility, not adoption6. Meta-analytic averages hide large heterogeneity across tasks8.

References

  1. Yang, J., Jimenez, C. E., Wettig, A., Lieret, K., Yao, S., Narasimhan, K., Press, O. (2024). SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. NeurIPS 2024. arXiv:2405.15793. arxiv.org/abs/2405.15793
  2. Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., Narasimhan, K. (2024). SWE-bench: Can Language Models Resolve Real-World GitHub Issues? ICLR 2024. arXiv:2310.06770. arxiv.org/abs/2310.06770
  3. Nori, H., Lee, Y. T., Zhang, S., Carignan, D., Edgar, R., Fusi, N., et al., Horvitz, E. (2023). Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine. arXiv:2311.16452. arxiv.org/abs/2311.16452
  4. Chen, L., Zaharia, M., Zou, J. (2023). FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance. arXiv:2305.05176. arxiv.org/abs/2305.05176
  5. Ong, I., et al. (2024). RouteLLM: Learning to Route LLMs with Preference Data. arXiv:2406.18665. arxiv.org/abs/2406.18665
  6. Eloundou, T., Manning, S., Mishkin, P., Rock, D. (2023). GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models. arXiv:2303.10130. arxiv.org/abs/2303.10130
  7. Anthropic (2025). The Anthropic Economic Index. www.anthropic.com/news/the-anthropic-economic-index
  8. Vaccaro, M., Almaatouq, A., Malone, T. (2024). When combinations of humans and AI are useful: A systematic review and meta-analysis. Nature Human Behaviour 8, 2293–2303. doi.org/10.1038/s41562-024-02024-1
  9. Mozannar, H., Sontag, D. (2020). Consistent Estimators for Learning to Defer to an Expert. ICML 2020. arXiv:2006.01862. arxiv.org/abs/2006.01862
  10. Brynjolfsson, E., Li, D., Raymond, L. R. (2023). Generative AI at Work. NBER Working Paper 31161. www.nber.org/papers/w31161
  11. Dell’Acqua, F., McFowland III, E., Mollick, E., Lifshitz-Assaf, H., Kellogg, K. C., Rajendran, S., Krayer, L., Candelon, F., Lakhani, K. R. (2023). Navigating the Jagged Technological Frontier. Harvard Business School Working Paper 24-013. www.hbs.edu/faculty/Pages/item.aspx?num=64700
  12. Noy, S., Zhang, W. (2023). Experimental evidence on the productivity effects of generative artificial intelligence. Science 381(6654), 187–192. doi.org/10.1126/science.adh2586

Work with us

We run allocation studies on real workflows with the teams that own them.

See a Demo

See a Demo

Send your enquiry to Immersive Reality Studio. All fields are required. Use a +country code for your phone. Please do not include sensitive information. Privacy notice.