The first reproducible benchmark for 770B open models on multi-step workflows. Get the full 10-page report with raw data.
No spam. Just the report when it's ready.
Enterprises are rapidly adopting open-weight models for complex, multi-step workflows—think automated code refactoring, deep research pipelines, and agentic process orchestration. Yet the standard leaderboards (HuggingFace, Artificial Analysis) still measure single-turn accuracy and short-context recall. That leaves a dangerous blind spot: a model that scores well on trivia can completely fail at a 20-step task with tool calls and error recovery.
Existing open models like Llama 3.1 405B and DeepSeek V3 show impressive benchmarks on paper, but practitioners report they lose coherence after 5-6 sequential steps. Tencent's Hy4 770B explicitly targets "long-horizon work," but no independent data validates this claim. Teams are left to either trust vendor marketing or burn weeks running their own evaluations—neither is acceptable when deployment decisions are on the line.
This is the perfect moment for a focused, reproducible benchmark. The signal is fresh (207 upvotes on HN, cross-posted to 3 platforms), the model is open and downloadable, and the gap in the market is clear: no one has yet published a rigorous, long-horizon comparison of these three 400B+ models. Hy4-Bench fills that void with hard numbers, not hype.
We download the open weights for Hy4 770B, Llama 3.1 405B, and DeepSeek V3, then serve each on identical vLLM instances with 8×H100 GPUs. This ensures fair inference settings and reproducible latency/throughput numbers.
Each model is evaluated on AgentBench (multi-step tool use), LongBench (extended context reasoning), and SWE-bench (real-world GitHub issue resolution). We measure task completion rate, average steps, error recovery, and token efficiency.
Results are aggregated into a 10-page PDF with full methodology, per-task breakdowns, and raw CSV data. You get a decision-ready report in 30 minutes—no need to run your own 2-week evaluation.
Three models × three test suites = nine distinct performance profiles. See exactly where Hy4 excels (AgentBench) and where it falls short (SWE-bench) with confidence intervals and step-by-step traces.
Every deployment config, prompt template, and evaluation script is open-sourced. Raw CSV data is included, so you can verify our numbers or extend the benchmark to your own test cases.
Skip the 2-week eval cycle. Our executive summary highlights the key trade-offs, so you can confidently choose a model for your long-horizon agent pipeline today.