Design Principles

Dnaerys is engineered for predictable latency and high availability in distributed environments.

I. Distributed Architecture

  • Homogeneous Distributed System: Dnaerys runs identical software on all nodes. Every node is a peer with equal roles; any node can accept and coordinate queries.
  • Shared-Nothing & In-Memory: To eliminate I/O bottlenecks, each node stores its entire partition of data in RAM, utilizing a shared-nothing architecture to maximize independent scaling.
  • Horizontal Scalability: Designed so that query latency remains within the same constraints even as the data volume and the number of nodes grow.
  • Column-Oriented OLAP: Optimized for analytical workloads, using columnar storage to minimize memory footprint and accelerate scan-heavy genomic queries.

II. High-Performance Execution & Specialization

  • Code Specialization:
    • No Runtime Overhead: Since the data model is known a priori, query logic is pre-written. This avoids the latency penalties of runtime code generation or compilation (JIT).
    • Hardware-Aware: Leverages the JVM for low-level specialization, including SIMD and NUMA.
  • Modern Processing Model:
    • Uses a materialization query execution model with a bottom-to-top approach.
    • Utilizes push-based task assignment to minimize scheduling overhead.
  • Parallelization: Employs a hybrid Actor-based and Multi-Threaded (MT) models to maximize CPU utilization across distributed nodes.
  • Low-latency network stack: Utilises Aeron UDP for inter-node communications.

III. Resilience & The PACELC Model
Dnaerys operates primarily as a PA/EC system (Availability over Consistency during partitions; Consistency over Latency during normal operation).

  • Network Partition Handling (PA): In the event of a network split, sub-clusters continue to serve clients. Responses are transparently marked as "potentially incomplete" if the query targets data on unreachable nodes.
  • Configurable Consistency: Can be tuned to a PC/EC system via an optional split-brain resolver for environments requiring strict consistency.
  • Advanced Fault Detection:
    • φ Accrual Failure Detector: A flexible, adaptive detector that adjusts to varying network latencies and deviations.
    • Push-Pull Gossip Protocol: Ensures rapid cluster-state awareness across all nodes without a central coordinator.
  • Data Integrity: Multi-layer validation and integrity checks, including non-cryptographic hashes, ensure detection of data corruption.

IV. Cloud-Scale Operations

  • Deployment Flexibility: Runs seamlessly on Kubernetes clusters and is equally at home on infrastructure ranging from laptops to enterprise clouds.
  • Observability-First: Designed with deep focus on Observability and Ops, ensuring that performance bottlenecks are visible in real-time.
  • Graceful Degradation: If a node fails, the cluster remains operational. The system prioritizes uptime, serving all available data while flagging missing segments to the requester.