Experimental Results Of Managing Variant Calling Datasets The Big Data Way

Experimental results from a study comparing the performance of Big Data clusters for variant calling data retrieval. The data includes query execution times for different storage formats (VCF and Apache Parquet), input sizes (104 and 1144 individuals), cluster sizes (2 to 150 executor nodes), and HDFS replication factors (3, 5, 7, 9). The dataset was produced by Katerina Boufea at Wageningen University & Research and accompanies the paper 'Managing Variant Calling Datasets The Big Data Way'.

Use Cases

• Benchmarking query performance of VCF versus Apache Parquet storage for genomic data based on the described experimental setup.
• Analyzing the impact of HDFS replication factor on data retrieval times for variant calling datasets.
• Modeling the scaling behavior of cluster compute nodes on query latency for a fixed genomic region.
• Comparing data processing efficiency for small (104 individuals) versus large (1144 individuals) cohort sizes in a Big Data context.

Strengths

• Results are based on five repeated executions for each experimental condition, which likely reduces measurement noise.
• The experiment systematically varies four key performance factors: storage format, input size, cluster node count, and HDFS replication factor.
• The dataset is associated with a specific research paper ('Managing Variant Calling Datasets The Big Data Way'), providing academic context.

Limitations

• Row count and detailed column-level documentation are unknown; data structure must be inferred after download.
• Last update date is unknown; freshness unverified.
• The dataset scope is limited to performance metrics for a specific query on chromosome 6 data from a single 2014 study (Aflitos et al).

Provenance

Source

Wageningen University & Research

Collection Method

Scripted experiments querying allele frequency data, with results provided alongside a script for generating figures.