Benchmarking Notes


Benchmarking Setup

28 cores with HT (total 56 logical cores)
Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
RAM: 503GiB
Ubuntu 18.04.1 LTS

2890 WGS dataset

Results for cluster with 4 (s/w) nodes running on a single physical node

Benchmark                                 Mode  Cnt   Score   Error   Units  // Returned results/op

// Variant selection
SelectVariants.hetP53orBRCA2              avgt   25   3.228 ± 0.169   ms/op  // 200 heterozygous alleles
SelectVariants.homP53orBRCA2              avgt   25   3.497 ± 0.138   ms/op  // 200 homozygous alleles
SelectVariants.homP53orBRCA2Unlim         avgt   25   4.742 ± 0.041   ms/op  // 456 homozygous alleles

SelectVariants.snpInRegion                avgt   25   2.290 ± 0.092   ms/op  // 1 allele
SelectVariants.mitoLiverPanel             avgt   25  21.989 ± 0.388   ms/op  // 6396 alleles

// Variant selection in "virtual cohort"
SelectVariants.hetP53InSamples            avgt   25   4.252 ± 0.055   ms/op  // 61 heterozygous alleles
SelectVariants.homP53InSamples            avgt   25   3.295 ± 0.071   ms/op  // 18 homozygous alleles

// Sample selection: Het
SelectSamples.withHetP53Variants          avgt   25   2.304 ± 0.062   ms/op  // 100 heterozygous alleles
SelectSamples.withHetP53VariantsAsMulti   avgt   25   2.400 ± 0.060   ms/op  // 100 heterozygous alleles
SelectSamples.withHetP53orBRCA2Variants   avgt   25   2.546 ± 0.129   ms/op  // 200 heterozygous alleles

// Sample selection: Hom
SelectSamples.withHomP53Variants          avgt   25   2.451 ± 0.059   ms/op  // 100 homozygous alleles
SelectSamples.withHomP53VariantsAsMulti   avgt   25   2.486 ± 0.053   ms/op  // 100 homozygous alleles
SelectSamples.withHomP53orBRCA2Variants   avgt   25   2.636 ± 0.146   ms/op  // 200 homozygous alleles

// Sample selection: Hom + Het
SelectSamples.withP53Variants             avgt   25   2.377 ± 0.058   ms/op  // 100 alleles
SelectSamples.withP53VariantsAsMulti      avgt   25   2.417 ± 0.049   ms/op  // 100 alleles
SelectSamples.withP53orBRCA2Variants      avgt   25   2.592 ± 0.142   ms/op  // 200 alleles

// Variant selection
SelectVariants.hetP53orBRCA2             thrpt   25   0.311 ± 0.017  ops/ms  // 200 heterozygous alleles
SelectVariants.homP53orBRCA2             thrpt   25   0.292 ± 0.017  ops/ms  // 200 homozygous alleles
SelectVariants.homP53orBRCA2Unlim        thrpt   25   0.206 ± 0.006  ops/ms  // 456 homozygous alleles

SelectVariants.snpInRegion               thrpt   25   0.441 ± 0.019  ops/ms  // 1 allele
SelectVariants.mitoLiverPanel            thrpt   25   0.046 ± 0.001  ops/ms  // 6396 alleles

// Variant selection in "virtual cohort"
SelectVariants.hetP53InSamples           thrpt   25   0.234 ± 0.002  ops/ms  // 61 heterozygous alleles
SelectVariants.homP53InSamples           thrpt   25   0.304 ± 0.007  ops/ms  // 18 homozygous alleles

// Sample selection: Het
SelectSamples.withHetP53Variants         thrpt   25   0.433 ± 0.036  ops/ms  // 100 heterozygous alleles
SelectSamples.withHetP53VariantsAsMulti  thrpt   25   0.423 ± 0.031  ops/ms  // 100 heterozygous alleles
SelectSamples.withHetP53orBRCA2Variants  thrpt   25   0.393 ± 0.030  ops/ms  // 200 heterozygous alleles

// Sample selection: Hom
SelectSamples.withHomP53Variants         thrpt   25   0.416 ± 0.036  ops/ms  // 100 homozygous alleles
SelectSamples.withHomP53VariantsAsMulti  thrpt   25   0.409 ± 0.031  ops/ms  // 100 homozygous alleles
SelectSamples.withHomP53orBRCA2Variants  thrpt   25   0.379 ± 0.028  ops/ms  // 200 homozygous alleles

// Sample selection: Hom + Het
SelectSamples.withP53Variants            thrpt   25   0.431 ± 0.030  ops/ms  // 100 alleles
SelectSamples.withP53VariantsAsMulti     thrpt   25   0.410 ± 0.021  ops/ms  // 100 alleles
SelectSamples.withP53orBRCA2Variants     thrpt   25   0.384 ± 0.024  ops/ms  // 200 alleles

Results for cluster with 16 (s/w) nodes running on a single physical node

Benchmark                                 Mode  Cnt   Score   Error   Units  // Returned results/op

// Variant selection
SelectVariants.hetP53orBRCA2              avgt   25   3.286 ± 0.179   ms/op  // 200 heterozygous alleles
SelectVariants.homP53orBRCA2              avgt   25   3.360 ± 0.130   ms/op  // 200 homozygous alleles
SelectVariants.homP53orBRCA2Unlim         avgt   25   4.592 ± 0.041   ms/op  // 456 homozygous alleles

SelectVariants.snpInRegion                avgt   25   2.329 ± 0.082   ms/op  // 1 allele
SelectVariants.mitoLiverPanel             avgt   25  18.671 ± 0.083   ms/op  // 6396 alleles

// Variant selection in "virtual cohort"
SelectVariants.hetP53InSamples            avgt   25   4.418 ± 0.038   ms/op  // 61 heterozygous alleles
SelectVariants.homP53InSamples            avgt   25   3.204 ± 0.059   ms/op  // 18 homozygous alleles

// Sample selection: Het
SelectSamples.withHetP53Variants          avgt   25   2.350 ± 0.060   ms/op  // 100 heterozygous alleles
SelectSamples.withHetP53VariantsAsMulti   avgt   25   2.398 ± 0.072   ms/op  // 100 heterozygous alleles
SelectSamples.withHetP53orBRCA2Variants   avgt   25   2.563 ± 0.133   ms/op  // 200 heterozygous alleles

// Sample selection: Hom
SelectSamples.withHomP53Variants          avgt   25   2.448 ± 0.060   ms/op  // 100 homozygous alleles
SelectSamples.withHomP53VariantsAsMulti   avgt   25   2.504 ± 0.061   ms/op  // 100 homozygous alleles
SelectSamples.withHomP53orBRCA2Variants   avgt   25   2.579 ± 0.132   ms/op  // 200 homozygous alleles

// Sample selection: Hom + Het
SelectSamples.withP53Variants             avgt   25   2.388 ± 0.050   ms/op  // 100 alleles
SelectSamples.withP53VariantsAsMulti      avgt   25   2.411 ± 0.074   ms/op  // 100 alleles
SelectSamples.withP53orBRCA2Variants      avgt   25   2.499 ± 0.183   ms/op  // 200 alleles

// Variant selection
SelectVariants.hetP53orBRCA2             thrpt   25   0.304 ± 0.021  ops/ms  // 200 heterozygous alleles
SelectVariants.homP53orBRCA2             thrpt   25   0.296 ± 0.021  ops/ms  // 200 homozygous alleles
SelectVariants.homP53orBRCA2Unlim        thrpt   25   0.216 ± 0.003  ops/ms  // 456 homozygous alleles

SelectVariants.snpInRegion               thrpt   25   0.430 ± 0.017  ops/ms  // 1 allele
SelectVariants.mitoLiverPanel            thrpt   25   0.054 ± 0.001  ops/ms  // 6396 alleles

// Variant selection in "virtual cohort"
SelectVariants.hetP53InSamples           thrpt   25   0.225 ± 0.003  ops/ms  // 61 heterozygous alleles
SelectVariants.homP53InSamples           thrpt   25   0.312 ± 0.006  ops/ms  // 18 homozygous alleles

// Sample selection: Het
SelectSamples.withHetP53Variants         thrpt   25   0.423 ± 0.037  ops/ms  // 100 heterozygous alleles
SelectSamples.withHetP53VariantsAsMulti  thrpt   25   0.420 ± 0.029  ops/ms  // 100 heterozygous alleles
SelectSamples.withHetP53orBRCA2Variants  thrpt   25   0.390 ± 0.030  ops/ms  // 200 heterozygous alleles

// Sample selection: Hom
SelectSamples.withHomP53Variants         thrpt   25   0.418 ± 0.039  ops/ms  // 100 homozygous alleles
SelectSamples.withHomP53VariantsAsMulti  thrpt   25   0.414 ± 0.033  ops/ms  // 100 homozygous alleles
SelectSamples.withHomP53orBRCA2Variants  thrpt   25   0.390 ± 0.036  ops/ms  // 200 homozygous alleles

// Sample selection: Hom + Het
SelectSamples.withP53Variants            thrpt   25   0.433 ± 0.032  ops/ms  // 100 alleles
SelectSamples.withP53VariantsAsMulti     thrpt   25   0.420 ± 0.030  ops/ms  // 100 alleles
SelectSamples.withP53orBRCA2Variants     thrpt   25   0.404 ± 0.028  ops/ms  // 200 alleles

Traffic Notes


gnomAD v3.1 dataset

Results for cluster with 4 (s/w) nodes running on a single physical node

Benchmark                                 Mode  Cnt    Score   Error   Units  // Returned results/op

// Variant selection
SelectVariants.hetP53orBRCA2              avgt   25    3.143 ± 0.498   ms/op  // 200 heterozygous alleles
SelectVariants.homP53orBRCA2              avgt   25    3.497 ± 0.238   ms/op  // 200 homozygous alleles
SelectVariants.homP53orBRCA2Unlim         avgt   25   13.378 ± 0.237   ms/op  // 1215 homozygous alleles

SelectVariants.snpInRegion                avgt   25    2.278 ± 0.275   ms/op  // 1 allele

// Variant selection in "virtual cohort"
SelectVariants.hetP53InSamples            avgt   25   11.511 ± 0.163   ms/op  // 51 heterozygous alleles
SelectVariants.homP53InSamples            avgt   25    5.330 ± 0.082   ms/op  // 14 homozygous alleles

// Sample selection: Het
SelectSamples.withHetP53Variants          avgt   25    2.416 ± 0.132   ms/op  // 100 heterozygous alleles
SelectSamples.withHetP53VariantsAsMulti   avgt   25    2.463 ± 0.154   ms/op  // 100 heterozygous alleles
SelectSamples.withHetP53orBRCA2Variants   avgt   25    2.528 ± 0.170   ms/op  // 200 heterozygous alleles

// Sample selection: Hom
SelectSamples.withHomP53Variants          avgt   25    2.383 ± 0.075   ms/op  // 100 homozygous alleles
SelectSamples.withHomP53VariantsAsMulti   avgt   25    2.430 ± 0.054   ms/op  // 100 homozygous alleles
SelectSamples.withHomP53orBRCA2Variants   avgt   25    2.657 ± 0.151   ms/op  // 200 homozygous alleles

// Sample selection: Hom + Het
SelectSamples.withP53Variants             avgt   25    2.442 ± 0.078   ms/op  // 100 alleles
SelectSamples.withP53VariantsAsMulti      avgt   25    2.537 ± 0.085   ms/op  // 100 alleles
SelectSamples.withP53orBRCA2Variants      avgt   25    2.465 ± 0.095   ms/op  // 200 alleles

// Variant selection
SelectVariants.hetP53orBRCA2             thrpt   25    0.329 ± 0.045  ops/ms  // 200 heterozygous alleles
SelectVariants.homP53orBRCA2             thrpt   25    0.277 ± 0.017  ops/ms  // 200 homozygous alleles
SelectVariants.homP53orBRCA2Unlim        thrpt   25    0.074 ± 0.002  ops/ms  // 1215 homozygous alleles

SelectVariants.snpInRegion               thrpt   25    0.449 ± 0.054  ops/ms  // 1 allele

// Variant selection in "virtual cohort"
SelectVariants.hetP53InSamples           thrpt   25    0.087 ± 0.001  ops/ms  // 51 heterozygous alleles
SelectVariants.homP53InSamples           thrpt   25    0.188 ± 0.002  ops/ms  // 14 homozygous alleles

// Sample selection: Het
SelectSamples.withHetP53Variants         thrpt   25    0.412 ± 0.040  ops/ms  // 100 heterozygous alleles
SelectSamples.withHetP53VariantsAsMulti  thrpt   25    0.411 ± 0.033  ops/ms  // 100 heterozygous alleles
SelectSamples.withHetP53orBRCA2Variants  thrpt   25    0.389 ± 0.031  ops/ms  // 200 heterozygous alleles

// Sample selection: Hom
SelectSamples.withHomP53Variants         thrpt   25    0.419 ± 0.035  ops/ms  // 100 homozygous alleles
SelectSamples.withHomP53VariantsAsMulti  thrpt   25    0.416 ± 0.030  ops/ms  // 100 homozygous alleles
SelectSamples.withHomP53orBRCA2Variants  thrpt   25    0.374 ± 0.025  ops/ms  // 200 homozygous alleles

// Sample selection: Hom + Het
SelectSamples.withP53Variants            thrpt   25    0.411 ± 0.028  ops/ms  // 100 alleles
SelectSamples.withP53VariantsAsMulti     thrpt   25    0.397 ± 0.021  ops/ms  // 100 alleles
SelectSamples.withP53orBRCA2Variants     thrpt   25    0.402 ± 0.023  ops/ms  // 200 alleles

Traffic Notes


51 WGS 1KG dataset

Dataset with 51 WGS samples from 1000 Genomes Project. It gives a glimpse of what to expect on a commodity laptop.

Results for cluster with 4 (s/w) nodes running on a single physical node (laptop)

Benchmark                                 Mode  Cnt  Score   Error   Units  // Returned results/op

// Variant selection
SelectVariants.hetP53orBRCA2Unlim         avgt   25  3.446 ± 0.111   ms/op  // 613 heterozygous alleles
SelectVariants.homP53orBRCA2Unlim         avgt   25  3.017 ± 0.096   ms/op  // 263 homozygous alleles
SelectVariants.snpInRegion                avgt   25  2.212 ± 0.067   ms/op  // 1 allele

// Variant selection in "virtual cohort"
SelectVariants.hetP53InSamples            avgt   25  2.663 ± 0.082   ms/op  // 54 heterozygous alleles
SelectVariants.homP53InSamples            avgt   25  2.632 ± 0.077   ms/op  // 38 homozygous alleles

// Sample selection
SelectSamples.withHetP53orBRCA2Variants   avgt   25  2.872 ± 0.086   ms/op  // 51 samples
SelectSamples.withHomP53orBRCA2Variants   avgt   25  2.813 ± 0.078   ms/op  // 51 samples

// Variant selection
SelectVariants.hetP53orBRCA2Unlim        thrpt   25  0.287 ± 0.010  ops/ms  // 613 heterozygous alleles
SelectVariants.homP53orBRCA2Unlim        thrpt   25  0.334 ± 0.012  ops/ms  // 263 homozygous alleles
SelectVariants.snpInRegion               thrpt   25  0.453 ± 0.016  ops/ms  // 1 allele

// Variant selection in "virtual cohort"
SelectVariants.hetP53InSamples           thrpt   25  0.376 ± 0.016  ops/ms  // 54 heterozygous alleles
SelectVariants.homP53InSamples           thrpt   25  0.380 ± 0.015  ops/ms  // 38 homozygous alleles

// Sample selection
SelectSamples.withHetP53orBRCA2Variants  thrpt   25  0.349 ± 0.014  ops/ms  // 51 samples
SelectSamples.withHomP53orBRCA2Variants  thrpt   25  0.351 ± 0.013  ops/ms  // 51 samples

Traffic Notes


Load scalability

Same tests, dataset and environment as for 51 WGS dataset, with more than a double load [*]. Two JMH benchmarks querying the same cluster node, at the same phase and time, one running outside cluster node container and another inside the container.

On benchmark outside the container, throughput decreased 10% to ~26% throughout the tests (which reflects roughly half of the whole throughput of the system under load, another half served benchmark inside container). Latency increased 15% to 29%. The smaller numbers are taken as a base for % evaluation.

Benchmark results outside container, cluster with 4 (s/w) nodes

Benchmark                                 Mode  Cnt  Score   Error   Units

// Variant selection
SelectVariants.hetP53orBRCA2Unlim         avgt   25  4.446 ± 0.090   ms/op
SelectVariants.homP53orBRCA2Unlim         avgt   25  3.787 ± 0.146   ms/op
SelectVariants.snpInRegion                avgt   25  2.563 ± 0.133   ms/op

// Variant selection in "virtual cohort"
SelectVariants.hetP53InSamples            avgt   25  3.247 ± 0.156   ms/op
SelectVariants.homP53InSamples            avgt   25  3.146 ± 0.197   ms/op

// Sample selection
SelectSamples.withHetP53orBRCA2Variants   avgt   25  3.690 ± 0.163   ms/op
SelectSamples.withHomP53orBRCA2Variants   avgt   25  3.626 ± 0.170   ms/op

// Variant selection
SelectVariants.hetP53orBRCA2Unlim        thrpt   25  0.228 ± 0.003  ops/ms
SelectVariants.homP53orBRCA2Unlim        thrpt   25  0.278 ± 0.013  ops/ms
SelectVariants.snpInRegion               thrpt   25  0.413 ± 0.029  ops/ms

// Variant selection in "virtual cohort"
SelectVariants.hetP53InSamples           thrpt   25  0.332 ± 0.018  ops/ms
SelectVariants.homP53InSamples           thrpt   25  0.339 ± 0.020  ops/ms

// Sample selection
SelectSamples.withHetP53orBRCA2Variants  thrpt   25  0.273 ± 0.010  ops/ms
SelectSamples.withHomP53orBRCA2Variants  thrpt   25  0.285 ± 0.017  ops/ms

[*] Workload on the cluster more than doubles as the benchmark from inside the container generates more load than the one from outside.


Alternative solutions

Apache Spark

We tested Spark queries on datasets fully allocated in memory, mainly for illustrative purposes. Using Spark as an in-memory OLAP database is not a proper use case for Spark. Also, placing the whole genomic dataset in data frame format in memory is not feasible in real world applications.

We ran simple benchmark tests counting the number of variants in gene panels on 51 WGS samples dataset on test bed. We used StorageLevel.MEMORY_ONLY and made sure the whole dataset fits in memory.

Across 100s tests for each panel,

Unrolling queries

To find out the limits of what the Spark query optimisers can do, we unrolled Mito11 query. It's not a feasible approach for the real world applications, but it provided us with the results which must be close to the best possible performance. Across 100s of tests, the majority of best results were around 250 ms.

RAM disk

The above Spark tests were run with the whole dataset allocated in RAM in deserialized data frame format, which is an unrealistic scenario for real world applications. To test in more realistic settings we put the dataset in parquet format (430Mb) on RAM disk (type tmpfs) and ran tests without persisting in memory on the same single node. Across 32 tests:

The same tests for 2890 samples dataset - 22Gb in parquet format on RAM disk:


Dnaerys

The same query for mitochondrial liver disease genes (Mito11 panel with 11 nuclear genes) in Dnaerys:


Google Genomics

A similar to gene panels query seems to take from tens of seconds to minutes in Google Cloud Life Sciences.