Replication materials for a study measuring systematic bias in large language models through repeated prompting. The dataset contains input materials, intermediate samples, and model-generated response files for experiments using benchmarks like BBQ, Discrimination Eval, and WinoBias. It was authored by Adam Wasilewski and last updated on May 26, 2026.
Use Cases
- Analyze systematic bias in LLMs based on outputs from repeated prompting across multiple models.
- Compare model behavior across families based on responses under baseline and robustness settings.
- Study the effect of prompt perturbations on model outputs based on the robustness_prompt experimental condition.
- Investigate the impact of temperature settings on response consistency based on the robustness_temp experimental condition.
- Reproduce experiments on bias measurement using the provided raw benchmark materials and processed sample sets.
Strengths
- Includes outputs from multiple prominent models, including GPT-4.1, GPT-4o, Llama-3.3-70B-Instruct, and open-weight variants.
- Organized structure enables comparison across model families and experimental variants using the same underlying sample sets.
- Contains both raw benchmark materials and processed, annotation-ready versions of data.
Limitations
- Column-level documentation is absent; field semantics must be inferred after download.
- Row count and file sizes are unknown, which may limit suitability assessment.
- Description metadata is limited; actual data quality requires manual inspection after download.
Provenance
- Source
- Harvard Dataverse
- Collection Method
- Likely contains model-generated responses and processed benchmark data from academic experiments.
- Freshness
- Last updated 2026-05-26 18:25:31; freshness should be verified.