by SeraphimSerapis · Agent Tool · ★ 361
tool-eval-bench A tool-calling quality benchmark for LLMs in agentic workflows, built for self-hosted serving stacks: vLLM, SGLang, LiteLLM, llama.cpp, NInfer, and hosted Gemini. It runs 69 deterministic scenarios (plus 19 opt-in Hard Mode ones) through OpenAI-compatible endpoints, scores each as pass, partial, or fail, and writes a full conversation trace for every one. Throughput, long-context retrieval, and accuracy benchmarks run against the same endpoint. Quickstart Install Also available via Docker if you would rather not have a local Python, or as a development checkout.
| Stars | 361 |
| Forks | 47 |
| Language | Python |
| Category | Agent Tool |
| License | MIT |
| Quality Score | 36.5/100 |
| Last Updated | 2026-10-04 |
| Created | 2026-04-17 |
| Platforms | python |
| Est. Tokens | ~124k |
These tools work well together with tool-eval-bench for enhanced workflows:
Explore other popular agent tool tools:
tool-eval-bench is Tool-calling quality benchmark for LLM serving stacks. 80+ deterministic scenarios testing multi-turn orchestration, safety boundaries, and structured output. Supports vLLM, SGLang, and llama.cpp.. It is categorized as a Agent Tool with 361 GitHub stars.
tool-eval-bench is primarily written in Python.
You can find installation instructions and usage details in the tool-eval-bench GitHub repository at github.com/SeraphimSerapis/tool-eval-bench. The project has 361 stars and 47 forks, indicating an active community.
tool-eval-bench is released under the MIT license, making it free to use and modify according to the license terms.