mökv

mökv is a distributed, in-memory key-value store. It utilizes Raft for consensus, serf for discvoery, gRPC for communication, and TLS for security.

Note

This is a project to learn more about distributed systems and Go.

Features

• Distributed Architecture: Data is replicated across multiple nodes for fault tolerance.
• In-Memory Storage: Provides fast read and write operations.
• Raft Consensus: Ensures data consistency across the cluster.
• gRPC Interface: Offers a well-defined API for interacting with the store.
• TLS Encryption: Secures communication between nodes and clients.
• Access Control: Uses Casbin for authorization, enabling fine-grained control over data access.
• Metrics: Exposes Prometheus metrics for monitoring cluster health and performance.
• Service Discovery: Uses serf for automatic node discovery and membership management.
• Load Balancing: Implements gRPC client-side load balancing, directing write operations to the leader and read operations to followers.

Getting Started

To run mökv:

Prerequisites

• Go
• cfssl (for generating TLS certificates)
• ghz (for performance testing. Optional)

Installation

1. Clone the repository:

   git clone git@github.com:dynamic-calm/mokv.git
   cd mokv

2. Generate TLS Certificates:

   make gencert

   This command uses cfssl to generate the necessary TLS certificates in the $HOME/.mokv directory.

3. Compile the code:

   make build

   This will create an executable binary mokv in the bin/ directory.

Configuration

Configuration is done through command-line flags or a configuration file. A sample configuration file (example/config.yaml) is provided. Copy certs/model.conf and certs/policy.csv to $HOME/.mokv.

Here's an example config.yaml:

data-dir: /tmp/mokv-data
node-name: node1
bind-addr: 127.0.0.1:8401
rpc-port: 8400
start-join-addrs: []
bootstrap: true
acl-model-file: $HOME/.mokv/model.conf
acl-policy-file: $HOME/.mokv/policy.csv
server-tls-cert-file: $HOME/.mokv/server.pem
server-tls-key-file: $HOME/.mokv/server-key.pem
server-tls-ca-file: $HOME/.mokv/ca.pem
peer-tls-cert-file: $HOME/.mokv/root-client.pem
peer-tls-key-file: $HOME/.mokv/root-client-key.pem
peer-tls-ca-file: $HOME/.mokv/ca.pem
metrics-port: 4000

Running mökv

1. Start the first node:

   bin/mokv --config-file example/config.yaml

2. Start additional nodes:

   Modify the example/config.yaml file with the appropriate node-name, bind-addr, and rpc-port. Crucially, set start-join-addrs to the address of the first node (e.g., 127.0.0.1:8401). Also set bootstrap: false for the additional nodes. Then, run the command again:

   bin/mokv --config-file example/config2.yaml  # Example config for the second node

   Refer to example/start_nodes.sh for a convenient script to start a cluster.

Usage

mökv exposes a gRPC API defined in internal/api/kv.proto. You can use a gRPC client to interact with the store.

service KV {
    rpc Get(GetRequest) returns (GetResponse) {}
    rpc Set(SetRequest) returns (SetResponse) {}
    rpc Delete(DeleteRequest) returns (DeleteResponse) {}
    rpc List(google.protobuf.Empty) returns (stream GetResponse) {}
    rpc GetServers(google.protobuf.Empty) returns (GetServersResponse){}
}

How it Works: Core Components and Data Flow

mökv combines Serf for node discovery and Raft for consistent data replication. Here's how the key components interact:

• Serf: Dynamic Membership: Serf uses UDP to monitor cluster membership. When a node joins, the serf.EventMemberJoin event triggers the Join function (internal/kv/kv.go), adding the node as a Raft voter. This ensures the Raft cluster reflects the current active nodes.

• Raft: Consensus and the FSM: Raft guarantees data consistency. One node is Leader, handling all write operations. Write operations become Raft log entries, replicated to Followers. The Finite State Machine (FSM) is the core of Raft's operation:

  • Applying Log Entries: When a log entry is committed (acknowledged by a quorum), the Apply method of the FSM (in internal/kv/kv.go) is invoked. The Apply method handles different request types:

    • Set Request: Updates the in-memory key-value store (kv.store) with the new key-value pair.
    • Delete Request: Removes the specified key from the in-memory store.

  • Data Flow for Writes: gRPC -> Raft Leader -> Log Entry -> Replication to Followers -> FSM Apply -> kv.store.

• Persistence (raft-boltdb): mökv uses raft-boltdb to persist Raft's log, stable state, and periodic snapshots to disk. This enables recovery after node failures.

  • Snapshotting: The FSM's Snapshot method creates a snapshot of the current in-memory state.
  • Restoring State: After a crash, the FSM's Restore method loads the latest snapshot and replays any subsequent log entries, reconstructing the in-memory kv.store to a consistent state. This entire process happens automatically when setupRaft is called during startup.

gRPC

mökv uses gRPC for efficient communication between clients and the cluster, secured with TLS client certificates for authentication and Casbin for authorization.

• API Definition: The core gRPC service, KV, is defined in internal/api/kv.proto, exposing methods like Get, Set, Delete, List, and GetServers.

• gRPC Server: The server implementation resides in internal/server/server.go, handling gRPC requests.

• Interceptors: gRPC Interceptors are used to handle:

  • Logging: Each incoming request is logged for monitoring.
  • Authentication: Authenticates clients using TLS client certificates. The certificate's Common Name (CN) is extracted and used as the subject for authorization (see below). This is handled in the authenticate function in internal/server/server.go.

• Authorization (Casbin): The internal/auth/auth.go enforces access control. Casbin determines if the authenticated user (identified by the CN) has permission to perform an action ("produce" for writes, "consume" for reads).

• Client-Side Load Balancing (Name Resolution and Picker): mökv uses client-side load balancing.

  • Name Resolver (internal/discovery/resolver.go): The name resolver periodically calls GetServers to discover available mökv nodes and their roles (Leader/Follower). It updates the list of available servers with the is_leader attribute.

  • Picker (internal/discovery/picker.go): The Picker directs requests based on the operation type and the leader status of available connections:

    • Writes (Set, Delete): These are routed to the Leader node to ensure consistency.
    • Reads (Get, List): These are balanced among available Follower nodes for improved read performance.