⚡ Synap

High-Performance In-Memory Key-Value Store & Message Broker

Synap is a modern, high-performance data infrastructure system built in Rust, combining the best features of Redis, RabbitMQ, and Kafka into a unified platform for real-time applications.

🎯 Overview

Synap provides multiple core capabilities in a single, cohesive system:

💾 Memory Key-Value Store - Radix-tree based in-memory storage with O(k) lookup
#️⃣ Hash Data Structure - Field-value maps within keys (Redis-compatible HSET, HGET, etc.) ✅ v0.6.0
📋 List Data Structure - Ordered sequences with LPUSH, RPOP, LRANGE (Redis-compatible) ✅ v0.6.0
🔷 Set Data Structure - Unordered unique collections with SADD, SREM, SINTER, SUNION (Redis-compatible) ✅ v0.6.0
📊 Sorted Set Data Structure - Scored members with ranking (ZADD, ZRANGE, ZRANK, ZINTER/ZUNION) ✅ v0.7.0
🔢 HyperLogLog - Probabilistic cardinality estimation with ~0.81% error (~12KB memory) ✅ v0.7.0-rc1
🗺️ Geospatial Indexes - Redis-compatible GEO commands (GEOADD, GEORADIUS, GEOSEARCH) ✅ v0.7.0-rc2
🔢 Bitmap Operations - Redis-compatible bit manipulation (SETBIT, GETBIT, BITCOUNT, BITOP) ✅ v0.7.0-rc2
📝 String Extensions - APPEND, GETRANGE, SETRANGE, GETSET, MSETNX ✅ v0.7.0-rc1
📜 Lua Scripting - Server-side scripting with EVAL/EVALSHA and redis.call() bridge ✅ v0.7.0
🔄 Transactions - MULTI/EXEC/WATCH/DISCARD with optimistic locking ✅ v0.7.0
📨 Acknowledgment Queues - RabbitMQ-style message queues with delivery guarantees
📡 Event Streams - Kafka-style partitioned topics with consumer groups and retention
🔔 Pub/Sub Messaging - Topic-based publish/subscribe with wildcard support

✨ Key Features

⚡ Performance

🚀 Sub-microsecond Operations: 87ns for GET operations (20,000x better than target)
📈 High Throughput: 10M+ ops/sec sequential writes (200x better than baseline)
💾 Efficient Memory: 92MB for 1M keys (54% reduction vs baseline)
🔄 64-Way Sharding: Linear scalability with CPU core count
⚙️ Async I/O: Built on Tokio for non-blocking operations
🗜️ Smart Compression: LZ4/Zstd compression with minimal CPU overhead
🔥 Hot Data Cache: Decompressed cache for frequently accessed data

🔐 Security & Authentication (✅ PRODUCTION READY - Jan 2025)

🔒 Authentication System - Root user, user management, API keys
🛡️ Fine-grained Permissions - Resource-based permissions (RabbitMQ-style)
📝 Audit Logging - Track all authentication events (login, API key usage, permission denials)
🔑 Password Validation - Configurable password requirements (SHA512 hashing)
🌐 IP Filtering - Restrict API keys to specific IP addresses
✅ Basic Auth & Bearer Token - Support for both authentication methods
🔐 MCP Authentication - Full authentication support for MCP protocol

💪 Durability (✅ COMPLETE - Oct 2025)

💾 Full Persistence: WAL + Snapshots for KV, Queue, and Stream
🔄 OptimizedWAL: Redis-style batching (10K ops/batch, 100µs window)
📨 Queue Persistence: RabbitMQ-style durable messaging with ACK tracking
📡 Stream Persistence: Kafka-style append-only logs per room
⚖️ PACELC Model: PC/EL (Consistency during partition, Latency in normal operation)
⏱️ Recovery Time: 1-10 seconds from snapshots + WAL replay

🛡️ Reliability & High Availability

🔄 Master-Slave Replication: 1 write master + N read replicas (✅ PRODUCTION READY)
- TCP binary protocol with length-prefixed framing
- Full sync via snapshot transfer (CRC32 verified)
- Partial sync via replication log (incremental updates)
- Auto-reconnect with intelligent resync
- 51 comprehensive tests (98% passing)
- Stress tested: 5000 operations
✅ Message Acknowledgment: Guaranteed message delivery with ACK/NACK
🔁 Event Replay: Stream history and replay capabilities
🔀 Manual Failover: Promote replica to master capability

📊 Monitoring & Observability

📈 INFO Command - Redis-style server introspection (server, memory, stats, replication, keyspace)
🐌 SLOWLOG - Slow query logging with configurable threshold (default 10ms)
💾 MEMORY USAGE - Per-key memory tracking across all data types
👥 CLIENT LIST - Active connection tracking and management
📊 Prometheus Metrics - 17 metric types for comprehensive monitoring

👨‍💻 Developer Experience

🤖 AI Integration: MCP support for Cursor, Claude Desktop, and AI assistants
🌊 StreamableHTTP Protocol: Simple HTTP-based streaming protocol
🔌 WebSocket Support: Persistent connections for real-time updates
📚 Multi-language SDKs: TypeScript, Python, Rust (with reactive PubSub), PHP, and C# clients with full authentication support
📖 Rich Examples: Chat, event broadcasting, task queues, authentication examples, and more

🔗 Protocol Support

🤖 MCP (Model Context Protocol): ✅ PRODUCTION READY - Configurable tools (KV, Hash, List, Set, Queue, Sorted Set) at /mcp endpoint with authentication support
🌐 UMICP (Universal Matrix Inter-Communication Protocol): ✅ PRODUCTION READY - 13 operations via MCP bridge with TLS support
📡 REST API: Standard HTTP endpoints for all operations
🔌 WebSocket API: Real-time bidirectional communication

📊 Scalability

📖 Read Scaling: Multiple replica nodes for distributed reads
🏠 Event Rooms: Isolated event streams per room/channel
🎯 Partitioned Topics: Kafka-style horizontal scaling with multiple partitions
👥 Consumer Groups: Coordinated consumption with automatic rebalancing
🔀 Topic Routing: Efficient pub/sub with wildcard matching
🔗 Connection Pooling: Client-side connection management

🏗️ Architecture

┌─────────────────────────────────────────────────────────┐
│                     Synap Server                        │
├─────────────────────────────────────────────────────────┤
│  StreamableHTTP/WebSocket Protocol Layer               │
├─────────────────────────────────────────────────────────┤
│  ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐  │
│  │ Key-Value│ │  Queue   │ │  Event   │ │  Pub/Sub │  │
│  │  Store   │ │  System  │ │  Stream  │ │          │  │
│  └──────────┘ └──────────┘ └──────────┘ └──────────┘  │
├─────────────────────────────────────────────────────────┤
│            Replication Log (Append-Only)                │
├─────────────────────────────────────────────────────────┤
│  Master Node              Replica Nodes (Read-Only)     │
└─────────────────────────────────────────────────────────┘

🚀 Quick Start

📦 Installation

From GitHub Releases (Recommended):

# Download pre-built binaries from GitHub Releases
# https://github.com/hivellm/synap/releases

# Linux (x86_64)
wget https://github.com/hivellm/synap/releases/download/v0.3.0/synap-linux-x86_64.tar.gz
tar -xzf synap-linux-x86_64.tar.gz
cd synap-linux-x86_64
./synap-server --config config.yml

# macOS (Intel)
wget https://github.com/hivellm/synap/releases/download/v0.3.0/synap-macos-x86_64.tar.gz
tar -xzf synap-macos-x86_64.tar.gz
cd synap-macos-x86_64
./synap-server --config config.yml

# macOS (Apple Silicon)
wget https://github.com/hivellm/synap/releases/download/v0.3.0/synap-macos-aarch64.tar.gz
tar -xzf synap-macos-aarch64.tar.gz
cd synap-macos-aarch64
./synap-server --config config.yml

# Windows (x86_64)
# Download synap-windows-x86_64.zip from releases page
# Extract and run synap-server.exe

🐳 Docker:

Quick Start (Docker Hub):

# Pull and run latest image
docker pull hivehub/synap:latest
docker run -d \
  --name synap \
  -p 15500:15500 \
  -p 15501:15501 \
  -v synap-data:/data \
  hivehub/synap:latest

# Check health
curl http://localhost:15500/health

Build Locally:

# Clone and build
git clone https://github.com/hivellm/synap.git
cd synap
docker build -t hivehub/synap:latest .

# Run container
docker run -d \
  --name synap-server \
  -p 15500:15500 \
  -p 15501:15501 \
  -v synap-data:/data \
  hivehub/synap:latest

With Authentication:

docker run -d \
  --name synap-server \
  -p 15500:15500 \
  -p 15501:15501 \
  -v synap-data:/data \
  -e SYNAP_AUTH_ENABLED=true \
  -e SYNAP_AUTH_REQUIRE_AUTH=true \
  -e SYNAP_AUTH_ROOT_USERNAME=admin \
  -e SYNAP_AUTH_ROOT_PASSWORD=SecurePassword123! \
  -e SYNAP_AUTH_ROOT_ENABLED=true \
  hivehub/synap:latest

Multi-Architecture Build:

# Build and push multi-arch images (AMD64 + ARM64)
./scripts/docker-publish.sh 0.8.1

# Or using PowerShell
.\scripts\docker-publish.ps1 0.8.1

Docker Compose:

# Use docker-compose for replication setup
docker-compose up -d

# With authentication (set environment variables)
export SYNAP_AUTH_ENABLED=true
export SYNAP_AUTH_REQUIRE_AUTH=true
export SYNAP_AUTH_ROOT_USERNAME=admin
export SYNAP_AUTH_ROOT_PASSWORD=SecurePassword123!
docker-compose up -d

Available Images:

hivehub/synap:latest - Latest stable release
hivehub/synap:0.8.1 - Specific version
Supports linux/amd64 and linux/arm64 architectures

📖 For detailed Docker documentation, see DOCKER_README.md

🛠️ From Source:

# Clone repository
git clone https://github.com/hivellm/synap.git
cd synap

# Build from source (requires Rust nightly 1.85+)
cargo build --release

# Run server
./target/release/synap-server --config config.yml

See Development Guide for detailed build instructions.

💻 Basic Usage

# Start server (default port 15500)
synap-server

# Key-Value Operations
curl -X POST http://localhost:15500/kv/set \
  -H "Content-Type: application/json" \
  -d '{"key": "user:1", "value": "John Doe", "ttl": 3600}'

curl http://localhost:15500/kv/get/user:1

# Queue Operations
curl -X POST http://localhost:15500/queue/publish \
  -d '{"queue": "tasks", "message": "process-video", "priority": 1}'

curl http://localhost:15500/queue/consume/tasks

# Event Stream
curl -X POST http://localhost:15500/stream/publish \
  -d '{"room": "chat-room-1", "event": "message", "data": "Hello!"}'

# Pub/Sub
curl -X POST http://localhost:15500/pubsub/publish \
  -d '{"topic": "notifications.email", "message": "New order"}'

# With Authentication (if enabled)
curl -X POST http://localhost:15500/kv/set \
  -H "Authorization: Basic $(echo -n 'admin:password' | base64)" \
  -H "Content-Type: application/json" \
  -d '{"key": "user:1", "value": "John Doe"}'

# Or with API Key
curl -X POST http://localhost:15500/kv/set \
  -H "Authorization: Bearer sk_your_api_key_here" \
  -H "Content-Type: application/json" \
  -d '{"key": "user:1", "value": "John Doe"}'

🔒 Authentication & Security (✅ Production Ready)

Authentication is disabled by default for development. Enable it for production:

Features:

✅ Root User Management - Configurable root user with full permissions
✅ User Management - Create, delete, enable/disable users
✅ API Key Management - Generate, revoke, and manage API keys with expiration
✅ Fine-grained Permissions - Resource-based permissions (RabbitMQ-style)
✅ Basic Auth & Bearer Token - Support for both authentication methods
✅ Audit Logging - Track all authentication events (login, API key usage, permission denials)
✅ Password Validation - Configurable password requirements (length, complexity)
✅ IP Filtering - Restrict API keys to specific IP addresses

Via Config File (config.yml):

auth:
  enabled: true
  require_auth: true
  root:
    username: "root"
    password: "your_secure_password"
    enabled: true

Via Docker Environment Variables:

docker run -d -p 15500:15500 \
  -e SYNAP_AUTH_ENABLED=true \
  -e SYNAP_AUTH_REQUIRE_AUTH=true \
  -e SYNAP_AUTH_ROOT_USERNAME=root \
  -e SYNAP_AUTH_ROOT_PASSWORD=your_secure_password \
  synap:latest

Using Authentication:

# Basic Auth
curl -u root:password http://localhost:15500/kv/get/user:1

# Bearer Token (API Key)
curl -H "Authorization: Bearer sk_XXXXX..." http://localhost:15500/kv/get/user:1

# Query Parameter
curl "http://localhost:15500/kv/get/user:1?api_key=sk_XXXXX..."

See Authentication Guide for complete details.

🔄 Replication Setup

Synap supports master-slave replication for high availability and read scaling.

Quick Start with Docker

# Start 1 master + 3 replicas
docker-compose up -d

# Master available at: localhost:15500
# Replica 1 at: localhost:15501
# Replica 2 at: localhost:15502
# Replica 3 at: localhost:15503

Manual Setup

Master Node Configuration (config-master.yml):

server:
  host: "0.0.0.0"
  port: 15500

replication:
  enabled: true
  role: "master"
  replica_listen_address: "0.0.0.0:15501"
  heartbeat_interval_ms: 1000
  max_lag_ms: 10000
  buffer_size_kb: 256
  replica_timeout_secs: 30

persistence:
  enabled: true
  wal:
    enabled: true
    path: "./data/wal/synap.wal"
  snapshot:
    enabled: true
    directory: "./data/snapshots"

Replica Node Configuration (config-replica.yml):

server:
  host: "0.0.0.0"
  port: 15500

replication:
  enabled: true
  role: "replica"
  master_address: "master:15501"  # Master's replication port
  heartbeat_interval_ms: 1000
  max_lag_ms: 10000
  buffer_size_kb: 256
  auto_reconnect: true
  reconnect_delay_ms: 5000

persistence:
  enabled: true
  wal:
    enabled: true
    path: "./data/wal/synap.wal"
  snapshot:
    enabled: true
    directory: "./data/snapshots"

Start Nodes:

# Terminal 1: Start master
synap-server --config config-master.yml

# Terminal 2: Start replica 1
synap-server --config config-replica-1.yml

# Terminal 3: Start replica 2
synap-server --config config-replica-2.yml

Usage Patterns

Write to Master:

# All writes go to master
curl -X POST http://localhost:15500/kv/set \
  -H "Content-Type: application/json" \
  -d '{"key": "user:100", "value": "Alice", "ttl": 3600}'

Read from Replicas (Load Balancing):

# Read from replica 1 (eventually consistent, ~5ms lag)
curl http://localhost:15501/kv/get/user:100

# Read from replica 2
curl http://localhost:15502/kv/get/user:100

# Read from replica 3
curl http://localhost:15503/kv/get/user:100

Monitor Replication Status:

# Check replication health on master
curl http://localhost:15500/health/replication

# Check replication status on replica
curl http://localhost:15501/health/replication

Consistency Guarantees

Master Reads: Strongly consistent (immediate)
Replica Reads: Eventually consistent (~5-10ms lag typical)
Write Durability: Writes confirmed after master commit
Replication: Asynchronous to replicas
Lag Monitoring: Real-time offset tracking

See docs/specs/REPLICATION.md for complete replication documentation.

For detailed Docker deployment guide, see:

DOCKER_README.md - Complete Docker Hub documentation with examples
docs/DOCKER_DEPLOYMENT.md - Advanced deployment guide

🎯 Use Cases

💬 Real-Time Chat

Use event streams for room-based messaging with message history and guaranteed delivery.

📋 Task Distribution

Leverage acknowledgment queues for distributed task processing with retry logic.

⚡ Cache Layer

Utilize key-value store as a high-speed cache with TTL support.

📡 Event Broadcasting

Implement pub/sub for system-wide notifications and event distribution.

🔄 Microservices Communication

Use queues for reliable inter-service messaging with delivery guarantees.

🛠️ Technology Stack

Language: Rust (Edition 2024)
Runtime: Tokio (async/await)
Web Framework: Axum
Data Structure: radix_trie (memory-efficient key-value)
Serialization: serde (JSON, MessagePack)
Protocols: StreamableHTTP + WebSocket + MCP + UMICP

📚 Documentation

📖 Getting Started

User Guide - Complete getting started guide (Installation, Quick Start, Operations)
Admin Guide - Operations handbook (Deployment, Monitoring, HA, Security)
Tutorials - 8 practical tutorials (Chat, Queues, Caching, Pub/Sub)

🔧 Core Documentation

Architecture - System architecture and components
Roadmap - Development roadmap and timeline
Configuration - Complete configuration reference
CLI Guide - Synap CLI usage and commands

🔒 Security & Authentication

Authentication - Complete auth guide (users, roles, API keys, ACL)
Queue Concurrency - Zero-duplicate guarantees

🌐 API & Protocols

REST API - Complete REST API documentation
OpenAPI Spec - OpenAPI 3.0 specification (YAML/JSON)
StreamableHTTP - StreamableHTTP protocol
MCP Integration - Model Context Protocol ✅ PRODUCTION READY
MCP Test Results - Live testing via Cursor AI
UMICP Integration - UMICP protocol ✅ PRODUCTION READY

📊 Performance & Testing

Benchmark Results - KV performance metrics
Testing Strategy - Test coverage and approach
Phase 1 Summary - Phase 1 implementation details

🔧 Development

Development Guide - Setup and contribution guide
Design Decisions - Technical choices
Project DAG - Component dependencies
Deployment - Production deployment (planned)
Packaging - Distribution packages (planned)

📋 Project Planning

Roadmap - Development roadmap and timeline
Project DAG - Component dependencies and implementation order

🧩 Component Specifications

Key-Value Store - Radix-tree storage system
Queue System - Message queues with ACK
Event Stream - Room-based broadcasting
Pub/Sub - Topic-based messaging
Replication - Master-slave architecture

📦 SDKs

TypeScript SDK - Node.js and browser support
Python SDK - Async/sync Python client
Rust SDK - Native Rust client library

💡 Examples

Real-Time Chat - Multi-room chat application
Event Broadcasting - System-wide events
Task Queue - Distributed task processing
Pub/Sub Pattern - Notification system

📊 Performance

✅ Performance with Persistence Enabled (October 2025) ⚡

🚀 Realistic Benchmarks - Fair Comparison ✅

Operation	Synap (Periodic fsync)	Redis/RabbitMQ	vs Competitor
KV Write	44K ops/s (22.5µs)	50-100K ops/s	🟰 Competitive (2x slower)
KV Read	12M ops/s (83ns)	80-100K ops/s	✅ 120x faster
Queue Publish	19.2K msgs/s (52µs)	0.1-0.2K msgs/s	✅ 100x faster
Queue Consume+ACK	607µs	5-10ms	✅ 8-16x faster
Memory (1M keys)	92MB	~200MB	✅ 54% reduction
Recovery (1K ops)	120ms	50-200ms	🟰 Similar
Concurrent Ops	64x parallel	Single-thread	✅ Linear scaling

Note: Persistence enabled with fsync_mode: periodic (10ms batching) - Production config

📈 Optimization Results

Metric	Before	After	Improvement
Memory (1M keys)	200MB	92MB	54% reduction
Write throughput	50K ops/s	10M+ ops/s	200x faster
Read latency P99	2-5ms	87ns	20,000x faster
Concurrent ops	Limited	64x parallel	Linear scaling
TTL cleanup CPU	100%	1-10%	10-100x reduction

🔜 Planned

Operation	Target	Status
Event Publish	< 1ms	🔄 In Progress
Pub/Sub Publish	< 0.5ms	🔵 Planned
Replication Lag	< 10ms	🔵 Planned

Test Coverage: 456+ tests passing (99.30% coverage)

Scripts: ./scripts/test-performance.ps1 (full suite), ./scripts/quick-test.ps1 (fast validation)

⚖️ Comparison

Feature	Synap	Redis	RabbitMQ	Kafka
Key-Value	✅	✅	❌	❌
Hashes	✅ (v0.6.0)	✅	❌	❌
Lists	✅ (v0.6.0)	✅	❌	❌
Sets	✅ (v0.6.0)	✅	❌	❌
Sorted Sets	✅ (v0.7.0)	✅	❌	❌
Geospatial	✅ (v0.7.0-rc2)	✅	❌	❌
Bitmaps	✅ (v0.7.0-rc2)	✅	❌	❌
HyperLogLog	✅ (v0.7.0-rc1)	✅	❌	❌
Lua Scripting	✅ (v0.7.0)	✅	❌	❌
Transactions	✅ (v0.7.0)	✅	❌	❌
String Extensions	✅ (v0.7.0-rc1)	✅	❌	❌
Queues (ACK)	✅	❌	✅	❌
Priority Queues	✅ (0-9)	❌	✅	❌
Dead Letter Queue	✅	❌	✅	❌
Event Streams	✅	✅ (Limited)	❌	✅
Partitioned Topics	✅	❌	❌	✅
Consumer Groups	✅	❌	❌	✅
Retention Policies	✅ (5 types)	✅ (2 types)	✅ (1 type)	✅ (2 types)
Pub/Sub	✅	✅	✅	✅
Authentication	✅ (Users+API Keys)	✅ (ACL)	✅ (Users)	✅ (SASL)
RBAC	✅	✅ (Limited)	✅	✅
API Key Expiration	✅	❌	❌	❌
IP Filtering	✅	✅	❌	❌
Replication	✅ (Master-Slave)	✅	✅	✅
Persistence	✅ (WAL+Snapshot)	✅ (AOF/RDB)	✅ (Disk)	✅ (Log)
PACELC Model	PC/EL	PC/EL	PC/EC	PA/EL
Native Compression	✅ (LZ4/Zstd)	❌	❌	✅ (Snappy)
Hot Data Cache	🔄 (L1/L2)	✅ (Single)	❌	❌
StreamableHTTP	✅	❌	❌	❌
MCP Support	✅ (Configurable, Auth)	❌	❌	❌
UMICP Support	✅ (13 ops, TLS)	❌	❌	❌
Enhanced Monitoring	✅ (INFO, SLOWLOG, MEMORY)	✅ (INFO)	❌	❌
Password Hashing	✅ (SHA512)	✅ (SHA256)	✅ (bcrypt)	✅ (SASL)
AI Integration	✅ (MCP+UMICP)	❌	❌	❌
Matrix Operations	✅ (via UMICP)	❌	❌	❌
Single Binary	✅	✅	❌	❌
Zero-Duplicate Guarantee	✅ (Tested)	N/A	✅	✅

Legend: ✅ Implemented | 🔄 In Progress | ❌ Not Available

📄 License

Apache License 2.0 - See LICENSE for details.

🤝 Contributing

See DEVELOPMENT.md for development setup and contribution guidelines.

📊 Project Status

Status: ✅ Phase 1-3 Complete | ✅ Redis Phase 1-2 Complete (Hash, List, Set, Sorted Set, Geospatial, Bitmap, String Extensions)
Version: 0.8.1 (Dependency updates + SDK fixes)
Edition: Rust 2024 (nightly 1.85+)
Last Updated: November 12, 2025

✅ Implementation Complete

🎯 Phase 1: Foundation (v0.1.0-alpha)

✅ Radix tree-based key-value store
✅ GET/SET/DELETE + Atomic (INCR/DECR)
✅ Batch operations (MSET/MGET/MDEL)
✅ TTL support with background cleanup
✅ Extended commands (KEYS, SCAN, FLUSH, EXPIRE, PERSIST)
✅ HTTP REST API (4 KV endpoints)
✅ StreamableHTTP Protocol (15+ commands)
✅ Comprehensive error handling
✅ Advanced logging (JSON + Pretty formats)

✅ Phase 2: Core Features (v0.2.0-beta) - COMPLETE (Oct 2025)

📨 Queue System ✅ COMPLETE

✅ FIFO with priority support (0-9)
✅ ACK/NACK mechanism + retry logic
✅ Dead Letter Queue (DLQ)
✅ Background deadline checker
✅ 9 REST API endpoints (create, publish, consume, ack, nack, etc.)
✅ Zero-duplicate guarantee (5 concurrency tests)
✅ Performance: 7,500+ msg/s with 50 concurrent consumers

🔒 Authentication & Authorization ✅ COMPLETE

✅ User management (SHA512 password hashing)
✅ Role-Based Access Control (admin, readonly, custom roles)
✅ API Keys (expiration, IP filtering, usage tracking)
✅ ACL system (resource-based permissions)
✅ Basic Auth + Bearer Token authentication
✅ Multi-tenant support via permission patterns
✅ 23 security tests (100% auth module coverage)

🗜️ Compression ✅ COMPLETE

✅ LZ4 (fast compression)
✅ Zstandard (better ratio)
✅ Configurable thresholds
✅ 6 comprehensive tests

📡 Event Streams ✅ COMPLETE + KAFKA-STYLE PARTITIONING ✅ NEW

✅ Ring buffer implementation (VecDeque, 10K msg/room)
✅ Room-based isolation (multi-tenant)
✅ Message history (offset-based replay)
✅ Offset-based consumption (Kafka-style)
✅ Automatic compaction (retention policy)
✅ Kafka-style persistence (append-only logs per room)
✅ Stream recovery from disk logs
✅ Master-Slave replication (full + partial sync)
✅ Snapshot integration (stream data in full sync)
✅ Partitioned Topics (multiple partitions per topic) ✅ NEW
✅ Consumer Groups (coordinated consumption with rebalancing) ✅ NEW
✅ Key-Based Routing (hash-based partition assignment) ✅ NEW
✅ Advanced Retention (time, size, count, combined, infinite) ✅ NEW
✅ Assignment Strategies (round-robin, range, sticky) ✅ NEW
✅ Offset Management (commit/checkpoint positions) ✅ NEW
✅ 6 simple stream endpoints + 17 Kafka-style endpoints
✅ Performance: 12.5M msgs/s consume, 2.3 GiB/s publish, 10K+ events/sec per partition

🔔 Pub/Sub System ✅ COMPLETE

✅ Topic routing (Radix Trie)
✅ Wildcard subscriptions (* and #)
✅ Fan-out messaging (concurrent delivery)
✅ Hierarchical topics
✅ Performance: 850K msgs/s, 1.2µs latency

💾 Persistence ✅ COMPLETE - All Subsystems

✅ OptimizedWAL (Redis-style batching, 10K ops/batch) ✅ NEW
✅ Queue Persistence (RabbitMQ-style ACK tracking) ✅ NEW
✅ Stream Persistence (Kafka-style append-only logs) ✅ NEW
✅ AsyncWAL with group commit (3-5x throughput)
✅ Streaming snapshot v2 (O(1) memory)
✅ Automatic recovery on startup (KV + Queue + Stream)
✅ 3 fsync modes: Always, Periodic, Never
✅ Manual snapshot endpoint (POST /snapshot)

🔄 Phase 3: Replication (v0.3.0-rc) - COMPLETE (Oct 2025)

Master-Slave Replication ✅ COMPLETE

✅ TCP communication layer (length-prefixed binary protocol)
✅ Full sync (snapshot transfer with CRC32 verification)
✅ Partial sync (incremental replication log updates)
✅ Circular replication log (1M operations buffer, like Redis)
✅ Lag monitoring (real-time offset tracking)
✅ Auto-reconnect (intelligent full/partial resync)
✅ Manual failover (promote replica to master)
✅ Stream replication (Event Streams included in sync) ✅ NEW
✅ Multi-subsystem sync (KV + Queue + Streams) ✅ NEW
✅ 67 comprehensive tests (25 unit + 16 extended + 10 integration + 16 KV operations)
✅ Stress tested (5000 operations validated)
✅ Multiple replicas (3+ replicas tested simultaneously)

Performance:

Snapshot creation: 1000 keys < 50ms
Large values: 100KB transfers validated
Multiple replicas: 3 replicas sync concurrently
Stress test: 5000 ops in ~4-5 seconds

🧪 Testing & Quality

✅ 456+ tests passing (99.30% test coverage)
- 128 library tests (KV, Queue, Streams, Partitioning, Consumer Groups, Persistence, Auth, Compression)
- 67 replication tests (25 unit + 16 extended + 10 integration TCP + 16 KV ops)
- 21 integration tests (performance, hybrid storage, persistence e2e)
- 7 Kafka-style integration tests (partition, consumer groups, retention)
- 40+ geospatial tests (23 unit + 17 integration)
- 12 bitmap integration tests
- 30 Lua scripting tests
- 11 transaction tests
- 58 authentication tests
- Protocol tests across REST, StreamableHTTP, WebSocket
✅ 11 comprehensive benchmark suites
- kv_bench: Memory, sharding, TTL, concurrency
- queue_bench: Arc sharing, priority, pending messages
- persistence_bench: AsyncWAL, snapshots, recovery
- hybrid_bench: Adaptive storage (HashMap/RadixTrie)
- stream_bench: Publish, consume, overflow, multi-subscriber
- pubsub_bench: Wildcards, fan-out, hierarchy
- compression_bench: LZ4/Zstd performance
- kv_persistence_bench: With disk I/O (3 fsync modes)
- queue_persistence_bench: RabbitMQ-style durability
- geospatial_bench: GEO operations performance ✅ NEW
- bitmap_bench: Bit manipulation performance ✅ NEW
✅ 99.30% test coverage
✅ Clean cargo fmt and cargo clippy
✅ WebSocket tests with graceful shutdown (s2s-tests feature)

🚀 Quick Start

# Clone and build
git clone https://github.com/hivellm/synap.git
cd synap
cargo build --release

# Run tests (456+ passing)
cargo test

# Run server
./target/release/synap-server --config config.yml
# Server starts at http://0.0.0.0:15500

# Use CLI client
./target/release/synap-cli
synap> SET mykey "Hello World"
synap> GET mykey

# Or via HTTP API
curl -X POST http://localhost:15500/kv/set \
  -H "Content-Type: application/json" \
  -d '{"key": "test", "value": "hello", "ttl": 3600}'

curl http://localhost:15500/kv/get/test
# Returns: "hello" (plain value, not wrapped JSON)

# Or via MCP (Cursor/Claude Desktop)
# Just ask: "Get the value of key 'test' from Synap"
# MCP tool synap_kv_get will be called automatically

📨 Queue System Examples

# Create queue
curl -X POST http://localhost:15500/queue/jobs \
  -H "Content-Type: application/json" \
  -d '{"max_depth": 10000, "ack_deadline_secs": 30}'

# Publish message
curl -X POST http://localhost:15500/queue/jobs/publish \
  -H "Content-Type: application/json" \
  -d '{"payload": [72,101,108,108,111], "priority": 9, "max_retries": 3}'

# Consume message
curl http://localhost:15500/queue/jobs/consume/worker-1

# Acknowledge (ACK)
curl -X POST http://localhost:15500/queue/jobs/ack \
  -H "Content-Type: application/json" \
  -d '{"message_id": "xxx-xxx-xxx"}'

🔒 Authentication Examples

# Basic Auth (Redis-style)
curl -u admin:password http://localhost:15500/queue/private/stats

# API Key (Bearer Token)
curl -H "Authorization: Bearer sk_XXXXX..." http://localhost:15500/queue/list

# API Key (Query Parameter)
curl http://localhost:15500/queue/list?api_key=sk_XXXXX...

See docs/AUTHENTICATION.md for complete authentication guide.

🔜 Next Phases

✅ Phase 2 (Q4 2025): Event Streams, Pub/Sub, Persistence - COMPLETE
✅ Phase 3 (Q1 2026): Master-Slave Replication with TCP - COMPLETE
✅ Redis Phase 1 (Oct 2025): Hash, List, Set Data Structures + 5 SDKs - COMPLETE 🎉
✅ Redis Phase 2 (Jan 2025): Sorted Sets, String Extensions, Geospatial, Bitmap - COMPLETE 🎉
✅ MCP Integration: Model Context Protocol with Authentication & Configurable Tools - COMPLETE
✅ UMICP Integration: Universal Matrix Inter-Communication Protocol with TLS - COMPLETE
✅ Enhanced Monitoring: INFO, SLOWLOG, MEMORY USAGE, CLIENT LIST - COMPLETE
⏳ Phase 4 (Q2 2026): Clustering, Sharding, GUI Dashboard, Distribution packages

See docs/ROADMAP.md for details.

Name		Name	Last commit message	Last commit date
Latest commit History 324 Commits
.cargo		.cargo
.cursor		.cursor
.github		.github
docs		docs
gui		gui
helm/synap		helm/synap
rulebook		rulebook
scripts		scripts
sdks		sdks
synap-cli		synap-cli
synap-migrate		synap-migrate
synap-server		synap-server
tests/load		tests/load
.codespellignore		.codespellignore
.cursorrules		.cursorrules
.dockerignore		.dockerignore
.gitignore		.gitignore
.markdownlint.json		.markdownlint.json
.rulebook		.rulebook
AGENTS.md		AGENTS.md
CHANGELOG.md		CHANGELOG.md
CLAUDE.md		CLAUDE.md
Cargo.toml		Cargo.toml
DOCKER_README.md		DOCKER_README.md
Dockerfile		Dockerfile
LICENSE		LICENSE
README.md		README.md
_typos.toml		_typos.toml
config-cluster.yml		config-cluster.yml
config-master.yml		config-master.yml
config-replica.yml		config-replica.yml
config.development.yml		config.development.yml
config.example.yml		config.example.yml
config.production.yml		config.production.yml
config.yml		config.yml
deny.toml		deny.toml
docker-compose.yml		docker-compose.yml
rust-toolchain.toml		rust-toolchain.toml
tatus --short		tatus --short

License

hivellm/synap

Folders and files

Latest commit

History

Repository files navigation

⚡ Synap

🎯 Overview

✨ Key Features

⚡ Performance

🔐 Security & Authentication (✅ PRODUCTION READY - Jan 2025)

💪 Durability (✅ COMPLETE - Oct 2025)

🛡️ Reliability & High Availability

📊 Monitoring & Observability

👨‍💻 Developer Experience

🔗 Protocol Support

📊 Scalability

🏗️ Architecture

🚀 Quick Start

📦 Installation

💻 Basic Usage

🔒 Authentication & Security (✅ Production Ready)

🔄 Replication Setup

Quick Start with Docker

Manual Setup

Usage Patterns

Consistency Guarantees

🎯 Use Cases

💬 Real-Time Chat

📋 Task Distribution

⚡ Cache Layer

📡 Event Broadcasting

🔄 Microservices Communication

🛠️ Technology Stack

📚 Documentation

📖 Getting Started

🔧 Core Documentation

🔒 Security & Authentication

🌐 API & Protocols

📊 Performance & Testing

🔧 Development

📋 Project Planning

🧩 Component Specifications

📦 SDKs

💡 Examples

📊 Performance

✅ Performance with Persistence Enabled (October 2025) ⚡

📈 Optimization Results

🔜 Planned

⚖️ Comparison

📄 License

🤝 Contributing

📊 Project Status

✅ Implementation Complete

🎯 Phase 1: Foundation (v0.1.0-alpha)

✅ Phase 2: Core Features (v0.2.0-beta) - COMPLETE (Oct 2025)

🔄 Phase 3: Replication (v0.3.0-rc) - COMPLETE (Oct 2025)

🧪 Testing & Quality

🚀 Quick Start

📨 Queue System Examples

🔒 Authentication Examples

🔜 Next Phases

About

Topics

Resources

License

Code of conduct

Contributing

Security policy

Uh oh!

Stars

Watchers

Forks

Releases 4

Packages 0

Contributors 2

Uh oh!

Languages

Packages