Research Tools & Code·arXiv cs.CL·2d ago

Open-source models lag on chained API calls, new Korean benchmark exposes gap

Open-source LLMs struggle with multi-step tool-calling workflows against live government APIs, a critical gap for data-sovereignty deployments in regulated sectors. Researchers introduce KOPA-Bench, a 145-task Korean public API benchmark, and EDGE, a synthesis method that maps tool interdependencies through actual execution and generates verified multi-step trajectories for fine-tuning. This addresses a real infrastructure bottleneck: on-premise agents must reliably chain API calls without cloud dependency, yet existing models and training data don't prepare them for this. The execution-grounded approach signals a shift toward pragmatic, deployment-aware benchmarking over lab-only metrics.

Modelwire context

Explainer

The key insight isn't just that open-source models fail at chaining API calls, but that existing training data doesn't capture the actual failure modes of real tool interdependencies. EDGE generates trajectories by executing against live APIs and recording what breaks, then uses those verified sequences for fine-tuning, which is fundamentally different from synthetic data generated offline.

This connects directly to two recent threads. BenchMIRT (Hugging Face, Sept 1) exposed that most benchmarks measure narrow task performance rather than real-world utility, and KOPA-Bench addresses that critique by grounding evaluation in actual government API workflows instead of isolated tasks. More importantly, the enterprise consolidation case from early September showed how production telemetry drives targeted post-training improvements. KOPA-Bench's execution-grounded synthesis follows the same pattern: observe where deployed systems fail, then generate training data that fixes those specific failure modes rather than hoping general fine-tuning will transfer.

If EDGE-trained models maintain their multi-step accuracy when tested against a different country's public API set (e.g., Japanese or Brazilian government APIs) within the next six months, that confirms the method generalizes beyond Korean-specific patterns. If accuracy drops significantly, it signals the approach is overfitted to Korean API design conventions and won't transfer to other regulated sectors.

Coverage we drew on

From Production Traffic to Post-Training: Building a Self-Hosted LLM That Covers the Corporate Request Mix · arXiv cs.CL

This analysis is generated by Modelwire’s editorial layer from our archive and the summary above. It is not a substitute for the original reporting. How we write it.

MentionsKOPA-Bench · EDGE · GRPO · Korean Open Public APIs

Read full story at arXiv cs.CL →(arxiv.org)

Modelwire Editorial

This synthesis and analysis was prepared by the Modelwire editorial team. We use advanced language models to read, ground, and connect the day’s most significant AI developments, providing original strategic context that helps practitioners and leaders stay ahead of the frontier.

Our mission How we write

Modelwire summarizes, we don’t republish. arXiv cs.CL originally reported this story as “Multi-Step Tool-Calling over Korean Open Public APIs: A Benchmark and a Data-Synthesis Recipe”. The full content lives on arxiv.org. If you’re a publisher and want a different summarization policy for your work, see our takedown page.