A preview dataset and code for the HumanPCR benchmark, which is designed for evaluating model predictions against ground truth. The benchmark includes a utility script for computing accuracy based on predictions formatted in a specific JSON structure. The dataset was authored by HumanPCR and last updated on May 23, 2025.
Use Cases
- Benchmarking model prediction accuracy based on the provided evaluation script.
- Testing model outputs against a structured ground truth for validation.
- Developing evaluation pipelines for models that generate JSON-formatted predictions.
Strengths
- Includes a dedicated evaluation script (evaluate.py) for standardized accuracy calculation.
- Provides a clear, required JSON schema for model predictions with 'id' and 'model_prediction' fields.
Limitations
- Description metadata is limited; actual data quality requires manual inspection after download.
- Column-level documentation is absent; field semantics must be inferred after download.
- Row count is unknown, which may limit suitability assessment.
Provenance
- Source
- HumanPCR
- Collection Method
- Likely created for benchmarking purposes, as indicated by the description.
- Time Range
- null
- Freshness
- Last updated 2025-05 23 09:49:35.
- Geography
- null