- Add access control system for replica authorization
- Implement persistence of replica information
- Add stale replica detection
- Create comprehensive tests for replica registration
- Update ReplicationServiceServer to use new components
- Add checksums for WAL entries and WAL entry batches
- Implement robust retry and circuit breaker patterns for reliability
- Add comprehensive tests for message processing and reliability features
- Enhance error handling and timeout management
This commit adds comprehensive reliability features to the replication transport layer:
- Add retry logic with exponential backoff for all network operations
- Implement circuit breaker pattern to prevent cascading failures
- Add reconnection handling with automatic recovery
- Implement proper timeout handling for all network operations
- Add comprehensive logging for connection issues
- Improve error handling with temporary error classification
- Enhance stream processing with automatic recovery
This commit implements the replication transport layer as part of Phase 2 of the replication plan.
Key components include:
- Add protocol buffer definitions for replication services
- Implement WALReplicator extension for processor management
- Create replication service server implementation
- Add replication client and server transport implementations
- Implement storage snapshot interface for bootstrap operations
- Standardize package naming across replication components
- Add replication-specific interfaces to pkg/transport
- Create ReplicationClient and ReplicationServer interfaces
- Add replication message types for WAL entries and bootstrap
- Create Protobuf schema for replication in proto/kevo/replication.proto
- Update transport registry to support replication components
- Add Lamport clock implementation for logical timestamps
- Define ReplicationHook interface in WAL package
- Extend WAL to use Lamport clock for timestamps
- Add notification hooks for WAL entries and batches
- Update WAL initialization to support replication
- Add tests for replication hooks and clocks
Adds a complete LSM-based storage engine with these features:
- Single-writer based architecture for the storage engine
- WAL for durability, and hey it's configurable
- MemTable with skip list implementation for fast read/writes
- SSTable with block-based structure for on-disk level-based storage
- Background compaction with tiered strategy
- ACID transactions
- Good documentation (I hope)
