Performance - Dnaerys

Benchmarking Notes

Take the notes with a grain of salt. Run your own tests. Make your own conclusions.
Every response carries information about execution time in elapsed_ms and elapsed_db_ms.
- elapsed_ms comes in the last message in a response stream and reflects all time from receiving gRPC request to sending the last message in response. This includes all internal communication overheads, but does not include Protocol Buffers serialization/deserialization and communication overheads from/to client, which are not exposed to the database. Those overheads can be derived from end-to-end time known on the client side.
- elapsed_db_ms reflects processing time in the database engine without internal communication overheads. Difference between elapsed_ms and elapsed_db_ms gives the cost of cluster's internal communications, including transferring results to response node. Each message in a response stream from replying nodes carries elapsed_db_ms.
End-to-end latency and throughput on a client side for a given query/cluster/dataset/testbed can be evaluated with JMH tests.
- 100+ integration tests, which cover entire gRPC API, provide more examples for client code that can be used for additional tests.
For throughput evaluation of massive amount of parallel requests consider to distribute requests evenly across nodes in the cluster. Load balancing is handled automatically in Kubernetes deployments.
- Any node can respond to any query, regardless of data distribution in the cluster, but every response is routed back to the client through the node of contact, creating additional load on this node.

Benchmarking Setup

Dnaerys 1.8.0 (JRE16), broadcast mode = false (default)
JMH version: 1.32
- all JMH settings = defaults
- running: java -jar benchmarks.jar
- java version "1.8.0_131"
- benchmarks sources
Hardware & OS
- 2890 WGS & gnomAD dataset tests performed on a single physical node with resources:

28 cores with HT (total 56 logical cores)
Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
RAM: 503GiB
Ubuntu 18.04.1 LTS

2890 WGS dataset

2 890 WGS samples
~86×10⁶ unique variants
~2890×86×10⁶ = 2.48×10¹¹ total number of all genotypes

Results for cluster with 4 (s/w) nodes running on a single physical node

Benchmark                                 Mode  Cnt   Score   Error   Units  // Returned results/op

// Variant selection
SelectVariants.hetP53orBRCA2              avgt   25   3.228 ± 0.169   ms/op  // 200 heterozygous alleles
SelectVariants.homP53orBRCA2              avgt   25   3.497 ± 0.138   ms/op  // 200 homozygous alleles
SelectVariants.homP53orBRCA2Unlim         avgt   25   4.742 ± 0.041   ms/op  // 456 homozygous alleles

SelectVariants.snpInRegion                avgt   25   2.290 ± 0.092   ms/op  // 1 allele
SelectVariants.mitoLiverPanel             avgt   25  21.989 ± 0.388   ms/op  // 6396 alleles

// Variant selection in "virtual cohort"
SelectVariants.hetP53InSamples            avgt   25   4.252 ± 0.055   ms/op  // 61 heterozygous alleles
SelectVariants.homP53InSamples            avgt   25   3.295 ± 0.071   ms/op  // 18 homozygous alleles

// Sample selection: Het
SelectSamples.withHetP53Variants          avgt   25   2.304 ± 0.062   ms/op  // 100 heterozygous alleles
SelectSamples.withHetP53VariantsAsMulti   avgt   25   2.400 ± 0.060   ms/op  // 100 heterozygous alleles
SelectSamples.withHetP53orBRCA2Variants   avgt   25   2.546 ± 0.129   ms/op  // 200 heterozygous alleles

// Sample selection: Hom
SelectSamples.withHomP53Variants          avgt   25   2.451 ± 0.059   ms/op  // 100 homozygous alleles
SelectSamples.withHomP53VariantsAsMulti   avgt   25   2.486 ± 0.053   ms/op  // 100 homozygous alleles
SelectSamples.withHomP53orBRCA2Variants   avgt   25   2.636 ± 0.146   ms/op  // 200 homozygous alleles

// Sample selection: Hom + Het
SelectSamples.withP53Variants             avgt   25   2.377 ± 0.058   ms/op  // 100 alleles
SelectSamples.withP53VariantsAsMulti      avgt   25   2.417 ± 0.049   ms/op  // 100 alleles
SelectSamples.withP53orBRCA2Variants      avgt   25   2.592 ± 0.142   ms/op  // 200 alleles

// Variant selection
SelectVariants.hetP53orBRCA2             thrpt   25   0.311 ± 0.017  ops/ms  // 200 heterozygous alleles
SelectVariants.homP53orBRCA2             thrpt   25   0.292 ± 0.017  ops/ms  // 200 homozygous alleles
SelectVariants.homP53orBRCA2Unlim        thrpt   25   0.206 ± 0.006  ops/ms  // 456 homozygous alleles

SelectVariants.snpInRegion               thrpt   25   0.441 ± 0.019  ops/ms  // 1 allele
SelectVariants.mitoLiverPanel            thrpt   25   0.046 ± 0.001  ops/ms  // 6396 alleles

// Variant selection in "virtual cohort"
SelectVariants.hetP53InSamples           thrpt   25   0.234 ± 0.002  ops/ms  // 61 heterozygous alleles
SelectVariants.homP53InSamples           thrpt   25   0.304 ± 0.007  ops/ms  // 18 homozygous alleles

// Sample selection: Het
SelectSamples.withHetP53Variants         thrpt   25   0.433 ± 0.036  ops/ms  // 100 heterozygous alleles
SelectSamples.withHetP53VariantsAsMulti  thrpt   25   0.423 ± 0.031  ops/ms  // 100 heterozygous alleles
SelectSamples.withHetP53orBRCA2Variants  thrpt   25   0.393 ± 0.030  ops/ms  // 200 heterozygous alleles

// Sample selection: Hom
SelectSamples.withHomP53Variants         thrpt   25   0.416 ± 0.036  ops/ms  // 100 homozygous alleles
SelectSamples.withHomP53VariantsAsMulti  thrpt   25   0.409 ± 0.031  ops/ms  // 100 homozygous alleles
SelectSamples.withHomP53orBRCA2Variants  thrpt   25   0.379 ± 0.028  ops/ms  // 200 homozygous alleles

// Sample selection: Hom + Het
SelectSamples.withP53Variants            thrpt   25   0.431 ± 0.030  ops/ms  // 100 alleles
SelectSamples.withP53VariantsAsMulti     thrpt   25   0.410 ± 0.021  ops/ms  // 100 alleles
SelectSamples.withP53orBRCA2Variants     thrpt   25   0.384 ± 0.024  ops/ms  // 200 alleles

Results for cluster with 16 (s/w) nodes running on a single physical node

Benchmark                                 Mode  Cnt   Score   Error   Units  // Returned results/op

// Variant selection
SelectVariants.hetP53orBRCA2              avgt   25   3.286 ± 0.179   ms/op  // 200 heterozygous alleles
SelectVariants.homP53orBRCA2              avgt   25   3.360 ± 0.130   ms/op  // 200 homozygous alleles
SelectVariants.homP53orBRCA2Unlim         avgt   25   4.592 ± 0.041   ms/op  // 456 homozygous alleles

SelectVariants.snpInRegion                avgt   25   2.329 ± 0.082   ms/op  // 1 allele
SelectVariants.mitoLiverPanel             avgt   25  18.671 ± 0.083   ms/op  // 6396 alleles

// Variant selection in "virtual cohort"
SelectVariants.hetP53InSamples            avgt   25   4.418 ± 0.038   ms/op  // 61 heterozygous alleles
SelectVariants.homP53InSamples            avgt   25   3.204 ± 0.059   ms/op  // 18 homozygous alleles

// Sample selection: Het
SelectSamples.withHetP53Variants          avgt   25   2.350 ± 0.060   ms/op  // 100 heterozygous alleles
SelectSamples.withHetP53VariantsAsMulti   avgt   25   2.398 ± 0.072   ms/op  // 100 heterozygous alleles
SelectSamples.withHetP53orBRCA2Variants   avgt   25   2.563 ± 0.133   ms/op  // 200 heterozygous alleles

// Sample selection: Hom
SelectSamples.withHomP53Variants          avgt   25   2.448 ± 0.060   ms/op  // 100 homozygous alleles
SelectSamples.withHomP53VariantsAsMulti   avgt   25   2.504 ± 0.061   ms/op  // 100 homozygous alleles
SelectSamples.withHomP53orBRCA2Variants   avgt   25   2.579 ± 0.132   ms/op  // 200 homozygous alleles

// Sample selection: Hom + Het
SelectSamples.withP53Variants             avgt   25   2.388 ± 0.050   ms/op  // 100 alleles
SelectSamples.withP53VariantsAsMulti      avgt   25   2.411 ± 0.074   ms/op  // 100 alleles
SelectSamples.withP53orBRCA2Variants      avgt   25   2.499 ± 0.183   ms/op  // 200 alleles

// Variant selection
SelectVariants.hetP53orBRCA2             thrpt   25   0.304 ± 0.021  ops/ms  // 200 heterozygous alleles
SelectVariants.homP53orBRCA2             thrpt   25   0.296 ± 0.021  ops/ms  // 200 homozygous alleles
SelectVariants.homP53orBRCA2Unlim        thrpt   25   0.216 ± 0.003  ops/ms  // 456 homozygous alleles

SelectVariants.snpInRegion               thrpt   25   0.430 ± 0.017  ops/ms  // 1 allele
SelectVariants.mitoLiverPanel            thrpt   25   0.054 ± 0.001  ops/ms  // 6396 alleles

// Variant selection in "virtual cohort"
SelectVariants.hetP53InSamples           thrpt   25   0.225 ± 0.003  ops/ms  // 61 heterozygous alleles
SelectVariants.homP53InSamples           thrpt   25   0.312 ± 0.006  ops/ms  // 18 homozygous alleles

// Sample selection: Het
SelectSamples.withHetP53Variants         thrpt   25   0.423 ± 0.037  ops/ms  // 100 heterozygous alleles
SelectSamples.withHetP53VariantsAsMulti  thrpt   25   0.420 ± 0.029  ops/ms  // 100 heterozygous alleles
SelectSamples.withHetP53orBRCA2Variants  thrpt   25   0.390 ± 0.030  ops/ms  // 200 heterozygous alleles

// Sample selection: Hom
SelectSamples.withHomP53Variants         thrpt   25   0.418 ± 0.039  ops/ms  // 100 homozygous alleles
SelectSamples.withHomP53VariantsAsMulti  thrpt   25   0.414 ± 0.033  ops/ms  // 100 homozygous alleles
SelectSamples.withHomP53orBRCA2Variants  thrpt   25   0.390 ± 0.036  ops/ms  // 200 homozygous alleles

// Sample selection: Hom + Het
SelectSamples.withP53Variants            thrpt   25   0.433 ± 0.032  ops/ms  // 100 alleles
SelectSamples.withP53VariantsAsMulti     thrpt   25   0.420 ± 0.030  ops/ms  // 100 alleles
SelectSamples.withP53orBRCA2Variants     thrpt   25   0.404 ± 0.028  ops/ms  // 200 alleles

Traffic Notes

limit is a free parameter in queries
Variant selection
- hetP53orBRCA2 results are limited to 100/node (= 200 total, p53 & BRCA2 are on different nodes)
- homP53orBRCA2 results are limited to 100/node (= 200 total, p53 & BRCA2 are on different nodes)
- homP53orBRCA2Unlim results unlimited = returns all 456 homozygous alleles in p53 or BRCA2
- mitoLiverPanel results unlimited = returns 6 396 alleles in 11 genes
Variant selection in "virtual cohort"
- hetP53InSamples returns 61 heterozygous alleles which exist in 2 samples in p53
- homP53InSamples returns 18 homozygous alleles which exist in 2 samples in p53
Sample selection: Het
- withHetP53Variants results are limited to 100/node (= 100 total, p53 is on a single node)
- withHetP53VariantsAsMulti same as above, but via SelectSamplesInMultiRegions call, which returns results as a stream
- withHetP53orBRCA2Variants results are limited to 100/node (= 200 total, p53 & BRCA2 are on different nodes)
Sample selection: Hom
- withHomP53Variants results are limited to 100/node (= 100 total, p53 is on a single node)
- withHomP53VariantsAsMulti same as above, but via SelectSamplesInMultiRegions call, which returns results as a stream
- withHomP53orBRCA2Variants results are limited to 100/node (= 200 total, p53 & BRCA2 are on different nodes)
Sample selection: Hom + Het
- withP53Variants results are limited to 100/node (= 100 total, p53 is on a single node)
- withP53VariantsAsMulti same as above, but via SelectSamplesInMultiRegions call, which returns results as a stream
- withP53orBRCA2Variants results are limited to 100/node (= 200 total, p53 & BRCA2 are on different nodes)

gnomAD v3.1 dataset

gnomAD v3.1
76 156 WGS samples
~759×10⁶ unique variants
415,071,703,278 homozygous alt + heterozygous genotypes, modelled by HWE with 0.005 no-call rate
76156*759×10⁶ = ~57.802404×10¹² total number of genotypes (including hom ref + missing)

Results for cluster with 4 (s/w) nodes running on a single physical node

Benchmark                                 Mode  Cnt    Score   Error   Units  // Returned results/op

// Variant selection
SelectVariants.hetP53orBRCA2              avgt   25    3.143 ± 0.498   ms/op  // 200 heterozygous alleles
SelectVariants.homP53orBRCA2              avgt   25    3.497 ± 0.238   ms/op  // 200 homozygous alleles
SelectVariants.homP53orBRCA2Unlim         avgt   25   13.378 ± 0.237   ms/op  // 1215 homozygous alleles

SelectVariants.snpInRegion                avgt   25    2.278 ± 0.275   ms/op  // 1 allele

// Variant selection in "virtual cohort"
SelectVariants.hetP53InSamples            avgt   25   11.511 ± 0.163   ms/op  // 51 heterozygous alleles
SelectVariants.homP53InSamples            avgt   25    5.330 ± 0.082   ms/op  // 14 homozygous alleles

// Sample selection: Het
SelectSamples.withHetP53Variants          avgt   25    2.416 ± 0.132   ms/op  // 100 heterozygous alleles
SelectSamples.withHetP53VariantsAsMulti   avgt   25    2.463 ± 0.154   ms/op  // 100 heterozygous alleles
SelectSamples.withHetP53orBRCA2Variants   avgt   25    2.528 ± 0.170   ms/op  // 200 heterozygous alleles

// Sample selection: Hom
SelectSamples.withHomP53Variants          avgt   25    2.383 ± 0.075   ms/op  // 100 homozygous alleles
SelectSamples.withHomP53VariantsAsMulti   avgt   25    2.430 ± 0.054   ms/op  // 100 homozygous alleles
SelectSamples.withHomP53orBRCA2Variants   avgt   25    2.657 ± 0.151   ms/op  // 200 homozygous alleles

// Sample selection: Hom + Het
SelectSamples.withP53Variants             avgt   25    2.442 ± 0.078   ms/op  // 100 alleles
SelectSamples.withP53VariantsAsMulti      avgt   25    2.537 ± 0.085   ms/op  // 100 alleles
SelectSamples.withP53orBRCA2Variants      avgt   25    2.465 ± 0.095   ms/op  // 200 alleles

// Variant selection
SelectVariants.hetP53orBRCA2             thrpt   25    0.329 ± 0.045  ops/ms  // 200 heterozygous alleles
SelectVariants.homP53orBRCA2             thrpt   25    0.277 ± 0.017  ops/ms  // 200 homozygous alleles
SelectVariants.homP53orBRCA2Unlim        thrpt   25    0.074 ± 0.002  ops/ms  // 1215 homozygous alleles

SelectVariants.snpInRegion               thrpt   25    0.449 ± 0.054  ops/ms  // 1 allele

// Variant selection in "virtual cohort"
SelectVariants.hetP53InSamples           thrpt   25    0.087 ± 0.001  ops/ms  // 51 heterozygous alleles
SelectVariants.homP53InSamples           thrpt   25    0.188 ± 0.002  ops/ms  // 14 homozygous alleles

// Sample selection: Het
SelectSamples.withHetP53Variants         thrpt   25    0.412 ± 0.040  ops/ms  // 100 heterozygous alleles
SelectSamples.withHetP53VariantsAsMulti  thrpt   25    0.411 ± 0.033  ops/ms  // 100 heterozygous alleles
SelectSamples.withHetP53orBRCA2Variants  thrpt   25    0.389 ± 0.031  ops/ms  // 200 heterozygous alleles

// Sample selection: Hom
SelectSamples.withHomP53Variants         thrpt   25    0.419 ± 0.035  ops/ms  // 100 homozygous alleles
SelectSamples.withHomP53VariantsAsMulti  thrpt   25    0.416 ± 0.030  ops/ms  // 100 homozygous alleles
SelectSamples.withHomP53orBRCA2Variants  thrpt   25    0.374 ± 0.025  ops/ms  // 200 homozygous alleles

// Sample selection: Hom + Het
SelectSamples.withP53Variants            thrpt   25    0.411 ± 0.028  ops/ms  // 100 alleles
SelectSamples.withP53VariantsAsMulti     thrpt   25    0.397 ± 0.021  ops/ms  // 100 alleles
SelectSamples.withP53orBRCA2Variants     thrpt   25    0.402 ± 0.023  ops/ms  // 200 alleles

Traffic Notes

limit is a free parameter in queries
Variant selection
- hetP53orBRCA2 results are limited to 100/node (= 200 total, p53 & BRCA2 are on different nodes)
- homP53orBRCA2 results are limited to 100/node (= 200 total, p53 & BRCA2 are on different nodes)
- homP53orBRCA2Unlim results unlimited = returns all 1 215 homozygous alleles in p53 or BRCA2
Variant selection in "virtual cohort"
- hetP53InSamples returns 51 heterozygous alleles which exist in 2 samples in p53
- homP53InSamples returns 14 homozygous alleles which exist in 2 samples in p53
Sample selection: Het
- withHetP53Variants results are limited to 100/node (= 100 total, p53 is on a single node)
- withHetP53VariantsAsMulti same as above, but via SelectSamplesInMultiRegions call, which returns results as a stream
- withHetP53orBRCA2Variants results are limited to 100/node (= 200 total, p53 & BRCA2 are on different nodes)
Sample selection: Hom
- withHomP53Variants results are limited to 100/node (= 100 total, p53 is on a single node)
- withHomP53VariantsAsMulti same as above, but via SelectSamplesInMultiRegions call, which returns results as a stream
- withHomP53orBRCA2Variants results are limited to 100/node (= 200 total, p53 & BRCA2 are on different nodes)
Sample selection: Hom + Het
- withP53Variants results are limited to 100/node (= 100 total, p53 is on a single node)
- withP53VariantsAsMulti same as above, but via SelectSamplesInMultiRegions call, which returns results as a stream
- withP53orBRCA2Variants results are limited to 100/node (= 200 total, p53 & BRCA2 are on different nodes)

51 WGS 1KG dataset

Dnaerys 1.11.1 (JRE17), broadcast mode = false (default)
JMH version: 1.32, default settings
Hardware: Lenovo ThinkPad T14s, i7-10510U CPU @ 1.80GHz, 4 cores/8HT
OS: Ubuntu 21.10
Dataset
- 51 WGS samples
- 19 725 105 unique variants
- ~51×19×10⁶ = ~1×10⁹ genotypes

Results for cluster with 4 (s/w) nodes running on a single physical node (laptop)

Benchmark                                 Mode  Cnt  Score   Error   Units  // Returned results/op

// Variant selection
SelectVariants.hetP53orBRCA2Unlim         avgt   25  3.446 ± 0.111   ms/op  // 613 heterozygous alleles
SelectVariants.homP53orBRCA2Unlim         avgt   25  3.017 ± 0.096   ms/op  // 263 homozygous alleles
SelectVariants.snpInRegion                avgt   25  2.212 ± 0.067   ms/op  // 1 allele

// Variant selection in "virtual cohort"
SelectVariants.hetP53InSamples            avgt   25  2.663 ± 0.082   ms/op  // 54 heterozygous alleles
SelectVariants.homP53InSamples            avgt   25  2.632 ± 0.077   ms/op  // 38 homozygous alleles

// Sample selection
SelectSamples.withHetP53orBRCA2Variants   avgt   25  2.872 ± 0.086   ms/op  // 51 samples
SelectSamples.withHomP53orBRCA2Variants   avgt   25  2.813 ± 0.078   ms/op  // 51 samples

// Variant selection
SelectVariants.hetP53orBRCA2Unlim        thrpt   25  0.287 ± 0.010  ops/ms  // 613 heterozygous alleles
SelectVariants.homP53orBRCA2Unlim        thrpt   25  0.334 ± 0.012  ops/ms  // 263 homozygous alleles
SelectVariants.snpInRegion               thrpt   25  0.453 ± 0.016  ops/ms  // 1 allele

// Variant selection in "virtual cohort"
SelectVariants.hetP53InSamples           thrpt   25  0.376 ± 0.016  ops/ms  // 54 heterozygous alleles
SelectVariants.homP53InSamples           thrpt   25  0.380 ± 0.015  ops/ms  // 38 homozygous alleles

// Sample selection
SelectSamples.withHetP53orBRCA2Variants  thrpt   25  0.349 ± 0.014  ops/ms  // 51 samples
SelectSamples.withHomP53orBRCA2Variants  thrpt   25  0.351 ± 0.013  ops/ms  // 51 samples

Traffic Notes

limit is not set - all results returned
Variant selection
- hetP53orBRCA2Unlim unlimited = returns all 613 heterozygous alleles in p53 or BRCA2
- homP53orBRCA2Unlim unlimited = returns all 263 homozygous alleles in p53 or BRCA2
Variant selection in "virtual cohort"
- hetP53InSamples returns 54 heterozygous alleles which exist in 2 samples in p53
- homP53InSamples returns 38 homozygous alleles which exist in 2 samples in p53
Sample selection: Het
- withHetP53orBRCA2Variants unlimited = returns all 51 samples
Sample selection: Hom
- withHomP53orBRCA2Variants unlimited = returns all 51 samples

Load scalability

Same tests, dataset and environment as for 51 WGS dataset, with more than a double load [*]. Two JMH benchmarks querying the same cluster node, at the same phase and time, one running outside cluster node container and another inside the container.

On benchmark outside the container, throughput decreased 10% to ~26% throughout the tests (which reflects roughly half of the whole throughput of the system under load, another half served benchmark inside container). Latency increased 15% to 29%. The smaller numbers are taken as a base for % evaluation.

Benchmark results outside container, cluster with 4 (s/w) nodes

Benchmark                                 Mode  Cnt  Score   Error   Units

// Variant selection
SelectVariants.hetP53orBRCA2Unlim         avgt   25  4.446 ± 0.090   ms/op
SelectVariants.homP53orBRCA2Unlim         avgt   25  3.787 ± 0.146   ms/op
SelectVariants.snpInRegion                avgt   25  2.563 ± 0.133   ms/op

// Variant selection in "virtual cohort"
SelectVariants.hetP53InSamples            avgt   25  3.247 ± 0.156   ms/op
SelectVariants.homP53InSamples            avgt   25  3.146 ± 0.197   ms/op

// Sample selection
SelectSamples.withHetP53orBRCA2Variants   avgt   25  3.690 ± 0.163   ms/op
SelectSamples.withHomP53orBRCA2Variants   avgt   25  3.626 ± 0.170   ms/op

// Variant selection
SelectVariants.hetP53orBRCA2Unlim        thrpt   25  0.228 ± 0.003  ops/ms
SelectVariants.homP53orBRCA2Unlim        thrpt   25  0.278 ± 0.013  ops/ms
SelectVariants.snpInRegion               thrpt   25  0.413 ± 0.029  ops/ms

// Variant selection in "virtual cohort"
SelectVariants.hetP53InSamples           thrpt   25  0.332 ± 0.018  ops/ms
SelectVariants.homP53InSamples           thrpt   25  0.339 ± 0.020  ops/ms

// Sample selection
SelectSamples.withHetP53orBRCA2Variants  thrpt   25  0.273 ± 0.010  ops/ms
SelectSamples.withHomP53orBRCA2Variants  thrpt   25  0.285 ± 0.017  ops/ms

[*] Workload on the cluster more than doubles as the benchmark from inside the container generates more load than the one from outside.