Hacker NewsHN submitter: Edward40
Original published: Updated on DevPick:

What is the best harness for each model?

What is the best harness for each model?
AI summary
Editorial policyUpdated on DevPick:

FrontierHarness Eval v1.1 results show that 64% of inference spend across 27,280 coding agent runs went to failed executions.

Key points

  • •FrontierHarness Eval v1.1 evaluated 22 coding agent harnesses with 5 models on the same 62 tasks, 4 times each.
  • •The best-value harness differed by model, with cost ranks shifting significantly between models.
  • •With a fixed model, the most expensive harness cost 1.8 to 5.0 times as much per task as the cheapest.
#AI Agents#Generative AI Tools & Use#LLM / Model Trends#FrontierHarness Eval
See more