Inference on Sample Average Treatment Effects with Large-Scale Online Experiments

Jasjeet S. Sekhon from Yale University presents new variance formulas for inference on sample average treatment effects in randomized control trials. The work includes an empirical application using hundreds of online experiments, with an average sample size of approximately 100 million observations per experiment. An R package named estCI implements the proposed estimation procedures.

Use Cases

• Estimating variance for sample average treatment effects based on the proposed theoretical framework.
• Comparing inference accuracy between SATE and SATT estimands as described in the paper.
• Applying the estCI R package to analyze data from large-scale online experiments.

Strengths

• Empirical application is based on hundreds of online experiments.
• Average sample size per experiment is approximately 100 million observations.
• Includes an R package (estCI) for implementing the proposed methods.

Limitations

• Column-level documentation is absent; field semantics must be inferred after download.
• Row count is unknown, which may limit suitability assessment.

Provenance

Source

Yale University

Collection Method

Derived from theoretical work and applied to empirical data from online experiments.