A standalone, CI-first evaluation tool that runs versioned JSONL cases through declarative HTTP adapters, computes deterministic metrics, compares reviewed baselines, and writes reports.
- Python
- JSONL
- HTTP
- CI
EVIDENCE
Engineering evidence available
- 85%+
- Test coverage · 2026-07-18
- 3.11–3.14
- Python CI · 2026-07-18
SYSTEM MAP
How it works
- 01Load versioned cases
- 02Call the target system
- 03Compute deterministic metrics
- 04Compare baselines and gates
- 05Write JSON and Markdown reports
Problem
AI features are often evaluated by trying a few prompts manually. Those checks are difficult to reproduce and cannot reliably answer whether a code change degraded quality, which cases regressed, or whether a release should be blocked.
Design
Eval42 stores evaluation definitions as version-controlled configuration and JSONL datasets. A constrained HTTP Adapter or an in-process Adapter calls the target, deterministic metrics measure the result, and reviewed baselines feed stable CI exit codes. PhoneMall and GroundedSeek can consume the tool for validation without becoming runtime dependencies.
Current result
- Ships an installable Alpha CLI and a Python library interface.
- Includes offline fixtures so examples and CI do not depend on external networks or model services.
- Writes JSON and Markdown reports and distinguishes gate failures, startup errors, and unreliable partial execution.
- Verifies Python 3.11–3.14 on Linux CI, with additional Windows and macOS coverage.
- Uses the MIT License and provides an installable wheel through a GitHub prerelease.
Boundaries
Eval42 does not use an LLM judge by default, upload evaluation data, or decide how the evaluated system should change. It remains Alpha; package-index publication and broader real-system adapters will follow only after the deterministic baseline is stable.