TOGBench: A Benchmark for Automated Test Oracle Generation from 25 Java Projects

77,299 single-oracle instances derived from developer-written unit tests across 25 real-world Java projects. The TOGBench artifact, created by Tasfia Tasnim and last updated in April 2026, is a benchmark suite for automated test oracle generation built around the OE25dev dataset. It supports end-to-end executable evaluation through oracle injection, compilation, and mutation-based assessment.

Use Cases

• Benchmarking automated test oracle generation tools based on the six defined oracle categories.
• Evaluating oracle prediction accuracy and adequacy using the provided compilation and mutation scoring scripts.
• Studying patterns in developer-written test oracles across 56 modules from real-world projects.

Strengths

• Contains 77,299 single-oracle instances with preserved test context.
• Derived from 25 real-world Java projects, providing realistic evaluation scenarios.
• Supports six distinct oracle categories, including assertion and exception types.

Limitations

• Column-level documentation is absent; field semantics must be inferred after download.
• Row count for the primary decomposed dataset is unknown, which may limit suitability assessment.
• Data is specific to Java 8 and may not generalize to other languages or newer Java versions.

Provenance

Source

figshare

Collection Method

Mined from developer-written unit tests across 25 real-world Java projects.

Time Range

null

Freshness

Last updated 2026-04-30 15:55:01; freshness should be verified.

Geography

null