Research · Technical report · October 2026
Allocation, not automation: efficiency gains from task-specific LLM harnesses
How much of the efficiency gain from language models comes from the harness around them, and why allocating each task to the right person or machine matters more than the model.
@techreport{irs2026allocation,
title={Allocation, not automation: efficiency gains from task-specific LLM harnesses},
author={{Immersive Reality Studio}},
institution={Immersive Reality Studio LLC},
year={2026},
month={oct}
}
Abstract
Most of the measured gains from language models at work do not come from the model alone. They come from what surrounds it: the interface it acts through, the context it is given, the checks it passes, and the decision of which model, tool or person should take each piece of work. We call this surrounding system a harness, and its routing function the allocation layer.
We bring together published evidence from software engineering, medicine, model routing, labour economics and human–AI collaboration. The same model resolves up to 6.7 times more software issues inside a purpose-built agent interface than with retrieval alone1; a general model steered by a task-specific prompting harness exceeds a fine-tuned specialist on medical licensing questions3; learned cascades match the strongest model at 59–98% lower cost4, and learned routers cut cost by up to 3.66 times while keeping 95% of its quality5. Field experiments show large gains when work sits inside a model’s competence and losses when it does not11, and a meta-analysis of 106 experiments finds that blending human and AI judgement is, on average, worse than giving each case to whichever performs better8. We argue that the main lever for efficiency is per-task allocation between people and machines, and we describe the allocation layer we build in Mixa and Membrane.
1. The allocation layer
An organisation produces a continuous stream of work: messages, requests, documents, orders, readings from devices. Today most of it is routed by habit: whoever opens the inbox, whichever tool is to hand. A language model added to that stream as a general assistant inherits the same routing problem; it is asked everything, and it is the right executor for only part of it.
The allocation layer sits between the stream and the executors. For each unit of work it decides who or what should do it: a small model, a frontier model, a tool or API, a device, or a person. It also decides which context and instructions go with it. Formally, for a task x it chooses an executor e from people and machines to minimise expected cost subject to a quality target and to the approvals the organisation requires. This generalises two lines of work: cascades that try cheaper models before expensive ones4, and learning to defer, where a model learns when to hand a case to a human expert9.
2. The same model, a better harness
Benchmarks are usually read as rankings of models. Read differently, they show how much of the result belongs to the harness. On SWE-bench, which asks systems to resolve real GitHub issues2, GPT-4 Turbo resolves 2.7% of the Lite tasks when given retrieved files and asked for a patch. Given a shell, it resolves 11.0%. Given an agent-computer interface designed for the task (search, file viewing and editing commands with feedback built for a model rather than a person) it resolves 18.0%1. The interface transfers: Claude 3 Opus rises from 4.3% to 13.0% with the same harness.
Medicine shows the same pattern without any new tools. Medprompt combines nearest-neighbour example selection, model-written reasoning chains and choice-shuffled ensembling. With it, GPT-4 reaches 90.2% on MedQA, ahead of Med-PaLM 2 at 86.5%, a model fine-tuned for medicine, and nine points above GPT-4 with a standard five-shot prompt3.
3. Routing across models
Not every task needs the most capable model. FrugalGPT learns a cascade that queries inexpensive models first and escalates only when an answer looks unreliable. It matches the accuracy of the best single model at a fraction of the cost: 98.3% less on a financial headlines dataset, 73.3% less on legal overruling classification and 59.2% less on conversational question answering4. RouteLLM learns a router from human preference data and reports up to 3.66 times lower cost than sending every query to GPT-4 while keeping 95% of its quality on MT Bench5.
4. Where the work is
The size of the opportunity depends less on the model than on what is built around it. Eloundou et al. estimate that with an LLM alone about 15% of US workers’ tasks could be completed significantly faster at the same quality; with software and tooling built on top of LLMs the share rises to between 47% and 56%6. Around 80% of the workforce is in occupations with at least a tenth of tasks exposed, and 19% in occupations with at least half6.
Observed use is already broad but uneven: roughly 36% of occupations use AI for at least a quarter of their tasks, and 57% of usage is augmentation, with a person in the loop, against 43% automation7.
~15%of tasks a worker does could be completed significantly faster at the same quality with an LLM alone
47–56%with software and tooling built on top of LLMs
5. People and machines: allocate, don’t blend
Field experiments confirm large gains when work is inside a model’s competence. Customer support agents resolved 14% more issues per hour with an AI assistant, and novices 34% more10; professionals finished writing tasks 40% faster with 18% higher quality12. Consultants completed 12.2% more tasks, 25.1% faster and at over 40% higher quality, but on a task chosen to sit outside the frontier they were 19% less likely to be correct11.
issues resolved per hour by 5,179 support agents; +34% for novice and low-skilled agents10
time on professional writing tasks, with +18% output quality12
more consulting tasks completed, 25.1% faster and over 40% higher quality, inside the frontier11
less likely to reach a correct answer on a task outside the frontier when consultants used AI11
A meta-analysis of 106 experiments explains why allocation matters more than assistance8. On average, human–AI combinations performed worse than the better of the two alone. When the human was better than the AI, the combination gained; when the AI was better, the combination lost. Decision tasks lost most; content-creation tasks gained. The practical reading is to send each case to whichever executor is stronger for it, and keep people where their judgement or approval adds value, rather than putting a person and a model on every task together.
Hedges’ g for human–AI synergy (combination vs. the better of human or AI alone), with 95% confidence intervals. Left of zero: the combination did worse.
6. What the layer has to do
Taken together, the evidence suggests five properties for an allocation layer in a real organisation:
- Task-specific harnesses. Each recurring kind of work gets its own interface, context and checks, as in SWE-agent and Medprompt1,3.
- Cost-aware routing. Cheaper models first, escalation on uncertainty4,5.
- Deferral to people. A learned or rule-based decision to hand a case to the person best placed to do it9.
- Frontier awareness. Tasks outside a model’s competence are recognised and kept with people11.
- Approval and record. Actions with outside effects wait for a person, and every allocation is logged with its reason.
Mixa implements these as operators with duties, hard limits and approvals, and Membrane extends them to an organisation’s systems and devices. We measure allocation by cost per completed task, quality against a reviewed reference, time to completion and the share of work that needed a person.
7. Limitations
The figures combine different benchmarks, models and years, and are not directly comparable with one another. Benchmark gains do not translate one-to-one into workplace gains, and field experiments cover specific occupations. Exposure estimates measure technical feasibility, not adoption6. Meta-analytic averages hide large heterogeneity across tasks8.
References
- Yang, J., Jimenez, C. E., Wettig, A., Lieret, K., Yao, S., Narasimhan, K., Press, O. (2024). SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. NeurIPS 2024. arXiv:2405.15793. arxiv.org/abs/2405.15793
- Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., Narasimhan, K. (2024). SWE-bench: Can Language Models Resolve Real-World GitHub Issues? ICLR 2024. arXiv:2310.06770. arxiv.org/abs/2310.06770
- Nori, H., Lee, Y. T., Zhang, S., Carignan, D., Edgar, R., Fusi, N., et al., Horvitz, E. (2023). Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine. arXiv:2311.16452. arxiv.org/abs/2311.16452
- Chen, L., Zaharia, M., Zou, J. (2023). FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance. arXiv:2305.05176. arxiv.org/abs/2305.05176
- Ong, I., et al. (2024). RouteLLM: Learning to Route LLMs with Preference Data. arXiv:2406.18665. arxiv.org/abs/2406.18665
- Eloundou, T., Manning, S., Mishkin, P., Rock, D. (2023). GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models. arXiv:2303.10130. arxiv.org/abs/2303.10130
- Anthropic (2025). The Anthropic Economic Index. www.anthropic.com/news/the-anthropic-economic-index
- Vaccaro, M., Almaatouq, A., Malone, T. (2024). When combinations of humans and AI are useful: A systematic review and meta-analysis. Nature Human Behaviour 8, 2293–2303. doi.org/10.1038/s41562-024-02024-1
- Mozannar, H., Sontag, D. (2020). Consistent Estimators for Learning to Defer to an Expert. ICML 2020. arXiv:2006.01862. arxiv.org/abs/2006.01862
- Brynjolfsson, E., Li, D., Raymond, L. R. (2023). Generative AI at Work. NBER Working Paper 31161. www.nber.org/papers/w31161
- Dell’Acqua, F., McFowland III, E., Mollick, E., Lifshitz-Assaf, H., Kellogg, K. C., Rajendran, S., Krayer, L., Candelon, F., Lakhani, K. R. (2023). Navigating the Jagged Technological Frontier. Harvard Business School Working Paper 24-013. www.hbs.edu/faculty/Pages/item.aspx?num=64700
- Noy, S., Zhang, W. (2023). Experimental evidence on the productivity effects of generative artificial intelligence. Science 381(6654), 187–192. doi.org/10.1126/science.adh2586
Work with us
We run allocation studies on real workflows with the teams that own them.
See a Demo