llm_eval 0.1.0
llm_eval: ^0.1.0 copied to clipboard
Test harness for LLM outputs: assertion checks, LLM-as-judge scoring, response caching, and CI-friendly reports.
Changelog #
0.1.0 #
Initial release.
EvalCase,EvalSuite, andEvalReportwith Markdown and JSON output.- Built-in checks:
contains,notContains,matches,isValidJson,predicate, and LLM-as-judge scoring withCheck.judge. ResponseCacheinterface in the core and a file-backedFileResponseCacheinpackage:llm_eval/io.dart(atomic writes) for deterministic reruns in CI.- Concurrent case execution with stable result order.
- Repeat runs with a flakiness rate.