Skip to content

About

What is Dnaerys Variant Store ?

Dnaerys is a state-of-the-art, distributed, horizontally scalable, in-memory low latency genome variant store — a specialised analytical DBMS designed to store genetic variations and execute genomic algorithms at scale.

It provides an environment for low-latency genomics:

  • Real-time Genome Exploration at Scale: millisecond-range queries response time
  • Horizontal Scalability: scales to hundreds of thousands of WGS samples while maintaining strict latency constraints
  • In-Memory Analytics: executes compute-heavy genetic algorithms inside database engine
  • LLM-native: seamless access to WGS datasets for LLMs via MCP servers

Why a Specialised Solution ?

Historically, horizontally scalable variant stores relied on general-purpose, disk-based architectures (Data Warehouses and Data Lakes). Disk-oriented design, often coupled with remote storage, puts them at orders-of-magnitude disadvantage compared to domain-optimised, specialised in-memory systems.

Thesis

General-purpose DBMS architectures are sub-optimal for low-latency, scalable variant stores.

A few main reasons:

I. Query Plan Optimisation and Code Generation
Specialised systems avoid the execution stages mandatory in general-purpose engines:

  • The Optimiser Burden: Generating an optimal plan for arbitrary queries is an NP-Complete problem. General-purpose optimisers rely on heuristics that often result in sub-optimal paths for genomic data.
  • Direct Execution: Since the genomic data model and queries are known a priori, the need for runtime query optimisers, code generation, or plan interpretation is fully eliminated.

II. Storage Model Efficiency
Traditional Relational DBMS models face significant scaling hurdles in genomics:

  • Row-oriented (NSM): These are unfit for in-memory genomic data; the structural overhead makes RAM requirements prohibitively expensive.
  • Column-oriented (DSM/PAX): While better at compression, the main memory required for large genomic datasets remains beyond the limit of practical feasibility in most general-purpose implementations.

III. Primary Data Allocation and Access Latency

  • Architectural Overheads: managing internal buffer pools and other subsystems in disk-based systems incur significant overheads. It is simple to see by placing all database files on RAM disks, thus eliminating disk I/O, and comparing with main-memory DBMS.
  • Physical Access Times: persistent storage is slower than RAM by approximately three orders of magnitude
  • The Cloud Penalty: solutions based on object stores are affected by:
    • Network Overheads: latency from serialisation and TLS
    • Data Locality: using object stores implies pulling data to the queries, instead of pushing queries to the data