docs: added idea, plan, and todo docs
This commit is contained in:
commit
ee23a47a74
GPG Key ID:
1278B36BA6F5D5E4
52
IDEA.md
Normal file
52
IDEA.md
Normal file
@ -0,0 +1,52 @@
|
|||||||
|
# Go Storage: A Minimalist LSM Storage Engine
|
||||||
|
|
||||||
|
## Vision
|
||||||
|
|
||||||
|
Build a clean, composable, and educational storage engine in Go that follows Log-Structured Merge Tree (LSM) principles, focusing on simplicity while providing the building blocks needed for higher-level database implementations.
|
||||||
|
|
||||||
|
## Goals
|
||||||
|
|
||||||
|
### 1. Extreme Simplicity
|
||||||
|
- Create minimal but complete primitives that can support various database paradigms (KV, relational, graph)
|
||||||
|
- Prioritize readability and educational value over hyper-optimization
|
||||||
|
- Use idiomatic Go with clear interfaces and documentation
|
||||||
|
- Implement a single-writer architecture for simplicity and reduced concurrency complexity
|
||||||
|
|
||||||
|
### 2. Durability + Performance
|
||||||
|
- Implement the LSM architecture pattern: Write-Ahead Log → MemTable → SSTables
|
||||||
|
- Provide configurable durability guarantees (sync vs. batched fsync)
|
||||||
|
- Optimize for both point lookups and range scans
|
||||||
|
|
||||||
|
### 3. Configurability
|
||||||
|
- Store all configuration parameters in a versioned, persistent manifest
|
||||||
|
- Allow tuning of memory usage, compaction behavior, and durability settings
|
||||||
|
- Support reproducible startup states across restarts
|
||||||
|
|
||||||
|
### 4. Composable Primitives
|
||||||
|
- Design clean interfaces for fundamental operations (reads, writes, snapshots, iteration)
|
||||||
|
- Enable building of higher-level abstractions (SQL, Gremlin, custom query languages)
|
||||||
|
- Support both transactional and analytical workloads
|
||||||
|
- Provide simple atomic write primitives that can be built upon:
|
||||||
|
- Leverage read snapshots from immutable LSM structure
|
||||||
|
- Support basic atomic batch operations
|
||||||
|
- Ensure crash recovery through proper WAL handling
|
||||||
|
|
||||||
|
## Target Use Cases
|
||||||
|
|
||||||
|
1. **Educational Tool**: Learn and teach storage engine internals
|
||||||
|
2. **Embedded Storage**: Applications needing local, durable storage with predictable performance
|
||||||
|
3. **Prototype Foundation**: Base layer for experimenting with novel database designs
|
||||||
|
4. **Go Ecosystem Component**: Reusable storage layer for Go applications and services
|
||||||
|
|
||||||
|
## Non-Goals
|
||||||
|
|
||||||
|
1. **Feature Parity with Production Engines**: Not trying to compete with RocksDB, LevelDB, etc.
|
||||||
|
2. **Multi-Node Distribution**: Focusing on single-node operation
|
||||||
|
3. **Complex Query Planning**: Leaving higher-level query features to layers built on top
|
||||||
|
|
||||||
|
## Success Criteria
|
||||||
|
|
||||||
|
1. **Correctness**: Data is never lost or corrupted, even during crashes
|
||||||
|
2. **Understandability**: Code is clear enough to serve as an educational reference
|
||||||
|
3. **Performance**: Reasonable throughput and latency for common operations
|
||||||
|
4. **Extensibility**: Can be built upon to create specialized database engines
|
||||||
154
PLAN.md
Normal file
154
PLAN.md
Normal file
@ -0,0 +1,154 @@
|
|||||||
|
# Implementation Plan for Go Storage Engine
|
||||||
|
|
||||||
|
## Architecture Overview
|
||||||
|
|
||||||
|
```
|
||||||
|
┌─────────────┐ ┌─────────────┐ ┌─────────────────────────┐
|
||||||
|
│ Client API │────▶│ MemTable │────▶│ Immutable SSTable Files │
|
||||||
|
└─────────────┘ └─────────────┘ └─────────────────────────┘
|
||||||
|
│ ▲ ▲
|
||||||
|
│ │ │
|
||||||
|
▼ │ │
|
||||||
|
┌─────────────┐ │ ┌─────────────────────────┐
|
||||||
|
│ Write- │────────────┘ │ Background Compaction │
|
||||||
|
│ Ahead Log │ │ Process │
|
||||||
|
└─────────────┘ └─────────────────────────┘
|
||||||
|
│ │
|
||||||
|
│ │
|
||||||
|
▼ ▼
|
||||||
|
┌─────────────────────────────────────────────────────────────────┐
|
||||||
|
│ Persistent Storage │
|
||||||
|
└─────────────────────────────────────────────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
## Package Structure
|
||||||
|
|
||||||
|
```
|
||||||
|
go-storage/
|
||||||
|
├── cmd/
|
||||||
|
│ └── storage-bench/ # Benchmarking tool
|
||||||
|
│
|
||||||
|
├── pkg/
|
||||||
|
│ ├── config/ # Configuration and manifest
|
||||||
|
│ ├── wal/ # Write-ahead logging with transaction markers
|
||||||
|
│ ├── memtable/ # In-memory table implementation
|
||||||
|
│ ├── sstable/ # SSTable read/write
|
||||||
|
│ │ ├── block/ # Block format implementation
|
||||||
|
│ │ └── footer/ # File footer and metadata
|
||||||
|
│ ├── compaction/ # Compaction strategies
|
||||||
|
│ ├── iterator/ # Merged iterator implementation
|
||||||
|
│ ├── transaction/ # Transaction management with Snapshot + WAL
|
||||||
|
│ │ ├── snapshot/ # Read snapshot implementation
|
||||||
|
│ │ └── txbuffer/ # Transaction write buffer
|
||||||
|
│ └── engine/ # Main engine implementation with single-writer architecture
|
||||||
|
│
|
||||||
|
└── internal/
|
||||||
|
├── checksum/ # Checksum utilities (xxHash64)
|
||||||
|
└── utils/ # Shared internal utilities
|
||||||
|
```
|
||||||
|
|
||||||
|
## Development Phases
|
||||||
|
|
||||||
|
### Phase A: Foundation (1-2 weeks)
|
||||||
|
1. Set up project structure and Go module
|
||||||
|
2. Implement config package with serialization/deserialization
|
||||||
|
3. Build basic WAL with:
|
||||||
|
- Append operations (Put/Delete)
|
||||||
|
- Replay functionality
|
||||||
|
- Configurable fsync modes
|
||||||
|
4. Write comprehensive tests for WAL durability
|
||||||
|
|
||||||
|
### Phase B: In-Memory Layer (1 week)
|
||||||
|
1. Implement MemTable with:
|
||||||
|
- Skip list data structure
|
||||||
|
- Sorted key iteration
|
||||||
|
- Size tracking for flush threshold
|
||||||
|
2. Connect WAL replay to MemTable restore
|
||||||
|
3. Test concurrent read/write scenarios
|
||||||
|
|
||||||
|
### Phase C: Persistent Storage (2 weeks)
|
||||||
|
1. Design and implement SSTable format:
|
||||||
|
- Block-based layout with restart points
|
||||||
|
- Checksummed blocks
|
||||||
|
- Index and metadata in footer
|
||||||
|
2. Build SSTable writer:
|
||||||
|
- Convert MemTable to blocks
|
||||||
|
- Generate sparse index
|
||||||
|
- Write footer with checksums
|
||||||
|
3. Implement SSTable reader:
|
||||||
|
- Block loading and validation
|
||||||
|
- Binary search through index
|
||||||
|
- Iterator interface
|
||||||
|
|
||||||
|
### Phase D: Basic Engine Integration (1 week)
|
||||||
|
1. Implement Level 0 flush mechanism:
|
||||||
|
- MemTable to SSTable conversion
|
||||||
|
- File management and naming
|
||||||
|
2. Create read path that merges:
|
||||||
|
- Current MemTable
|
||||||
|
- Immutable MemTables awaiting flush
|
||||||
|
- Level 0 SSTable files
|
||||||
|
|
||||||
|
### Phase E: Compaction (2 weeks)
|
||||||
|
1. Implement a single, efficient compaction strategy:
|
||||||
|
- Simple tiered compaction approach
|
||||||
|
2. Handle tombstones and key deletion
|
||||||
|
3. Manage file obsolescence and cleanup
|
||||||
|
4. Build background compaction scheduling
|
||||||
|
|
||||||
|
### Phase F: Basic Atomicity and Advanced Features (2-3 weeks)
|
||||||
|
1. Implement merged iterator across all levels
|
||||||
|
2. Add snapshot capability for reads:
|
||||||
|
- Point-in-time view of the database
|
||||||
|
- Consistent reads across MemTable and SSTables
|
||||||
|
3. Implement simple atomic batch operations:
|
||||||
|
- Support atomic multi-key writes
|
||||||
|
- Ensure proper crash recovery for batch operations
|
||||||
|
- Design interfaces that can be extended for full transactions
|
||||||
|
4. Add basic statistics and metrics
|
||||||
|
|
||||||
|
### Phase G: Optimization and Benchmarking (1 week)
|
||||||
|
1. Develop benchmark suite for:
|
||||||
|
- Random vs sequential writes
|
||||||
|
- Point reads vs range scans
|
||||||
|
- Compaction overhead and pauses
|
||||||
|
2. Optimize critical paths based on profiling
|
||||||
|
3. Tune default configuration parameters
|
||||||
|
|
||||||
|
### Phase H: Optional Enhancements (as needed)
|
||||||
|
1. Add Bloom filters to reduce disk reads
|
||||||
|
2. Create monitoring hooks and detailed metrics
|
||||||
|
3. Add crash recovery testing
|
||||||
|
|
||||||
|
## Testing Strategy
|
||||||
|
|
||||||
|
1. **Unit Tests**: Each component thoroughly tested in isolation
|
||||||
|
2. **Integration Tests**: End-to-end tests for complete workflows
|
||||||
|
3. **Property Tests**: Generate randomized operations and verify correctness
|
||||||
|
4. **Crash Tests**: Simulate crashes and verify recovery
|
||||||
|
5. **Benchmarks**: Measure performance across different workloads
|
||||||
|
|
||||||
|
## Implementation Notes
|
||||||
|
|
||||||
|
### Error Handling
|
||||||
|
- Use descriptive error types and wrap errors with context
|
||||||
|
- Implement recovery mechanisms for all critical operations
|
||||||
|
- Validate checksums at every read opportunity
|
||||||
|
|
||||||
|
### Concurrency
|
||||||
|
- Implement single-writer architecture for the main write path
|
||||||
|
- Allow concurrent readers (snapshots) to proceed without blocking
|
||||||
|
- Use appropriate synchronization for reader-writer coordination
|
||||||
|
- Ensure proper isolation between transactions
|
||||||
|
|
||||||
|
### Batch Operation Management
|
||||||
|
- Use WAL for atomic batch operation durability
|
||||||
|
- Leverage LSM's natural versioning for snapshots
|
||||||
|
- Provide simple interfaces that can be built upon for transactions
|
||||||
|
- Ensure proper crash recovery for batch operations
|
||||||
|
|
||||||
|
### Go Idioms
|
||||||
|
- Follow standard Go project layout
|
||||||
|
- Use interfaces for component boundaries
|
||||||
|
- Rely on Go's GC but manage large memory allocations carefully
|
||||||
|
- Use context for cancellation where appropriate
|
||||||
198
TODO.md
Normal file
198
TODO.md
Normal file
@ -0,0 +1,198 @@
|
|||||||
|
# Go Storage Engine Todo List
|
||||||
|
|
||||||
|
This document outlines the implementation tasks for the Go Storage Engine, organized by development phases. Follow these guidelines:
|
||||||
|
|
||||||
|
- Work on tasks in the order they appear
|
||||||
|
- Check off exactly one item (✓) before moving to the next unchecked item
|
||||||
|
- Each phase must be completed before starting the next phase
|
||||||
|
- Test thoroughly before marking an item complete
|
||||||
|
|
||||||
|
## Phase A: Foundation
|
||||||
|
|
||||||
|
- [ ] Setup project structure and Go module
|
||||||
|
- [ ] Create directory structure following the package layout in PLAN.md
|
||||||
|
- [ ] Initialize Go module and dependencies
|
||||||
|
- [ ] Set up testing framework
|
||||||
|
|
||||||
|
- [ ] Implement config package
|
||||||
|
- [ ] Define configuration struct with serialization/deserialization
|
||||||
|
- [ ] Include configurable parameters for durability, compaction, memory usage
|
||||||
|
- [ ] Create manifest loading/saving functionality
|
||||||
|
- [ ] Add versioning support for config changes
|
||||||
|
|
||||||
|
- [ ] Build Write-Ahead Log (WAL)
|
||||||
|
- [ ] Implement append-only file with atomic operations
|
||||||
|
- [ ] Add Put/Delete operation encoding
|
||||||
|
- [ ] Create replay functionality with error recovery
|
||||||
|
- [ ] Implement both synchronous (default) and batched fsync modes
|
||||||
|
- [ ] Add checksumming for entries
|
||||||
|
|
||||||
|
- [ ] Write WAL tests
|
||||||
|
- [ ] Test durability with simulated crashes
|
||||||
|
- [ ] Verify replay correctness
|
||||||
|
- [ ] Benchmark write performance with different sync options
|
||||||
|
- [ ] Test error handling and recovery
|
||||||
|
|
||||||
|
## Phase B: In-Memory Layer
|
||||||
|
|
||||||
|
- [ ] Implement MemTable
|
||||||
|
- [ ] Create skip list data structure aligned to 64-byte cache lines
|
||||||
|
- [ ] Add key/value insertion and lookup operations
|
||||||
|
- [ ] Implement sorted key iteration
|
||||||
|
- [ ] Add size tracking for flush threshold detection
|
||||||
|
|
||||||
|
- [ ] Connect WAL replay to MemTable
|
||||||
|
- [ ] Create recovery logic to rebuild MemTable from WAL
|
||||||
|
- [ ] Implement consistent snapshot reads during recovery
|
||||||
|
- [ ] Handle errors during replay with appropriate fallbacks
|
||||||
|
|
||||||
|
- [ ] Test concurrent read/write scenarios
|
||||||
|
- [ ] Verify reader isolation during writes
|
||||||
|
- [ ] Test snapshot consistency guarantees
|
||||||
|
- [ ] Benchmark read/write performance under load
|
||||||
|
|
||||||
|
## Phase C: Persistent Storage
|
||||||
|
|
||||||
|
- [ ] Design SSTable format
|
||||||
|
- [ ] Define 16KB block structure with restart points
|
||||||
|
- [ ] Create checksumming for blocks (xxHash64)
|
||||||
|
- [ ] Define index structure with entries every ~64KB
|
||||||
|
- [ ] Design file footer with metadata (version, timestamp, key count, etc.)
|
||||||
|
|
||||||
|
- [ ] Implement SSTable writer
|
||||||
|
- [ ] Add functionality to convert MemTable to blocks
|
||||||
|
- [ ] Create sparse index generator
|
||||||
|
- [ ] Implement footer writing with checksums
|
||||||
|
- [ ] Add atomic file creation for crash safety
|
||||||
|
|
||||||
|
- [ ] Build SSTable reader
|
||||||
|
- [ ] Implement block loading with validation
|
||||||
|
- [ ] Create binary search through index
|
||||||
|
- [ ] Develop iterator interface for scanning
|
||||||
|
- [ ] Add error handling for corrupted files
|
||||||
|
|
||||||
|
## Phase D: Basic Engine Integration
|
||||||
|
|
||||||
|
- [ ] Implement Level 0 flush mechanism
|
||||||
|
- [ ] Create MemTable to SSTable conversion process
|
||||||
|
- [ ] Implement file management and naming scheme
|
||||||
|
- [ ] Add background flush triggering based on size
|
||||||
|
|
||||||
|
- [ ] Create read path that merges data sources
|
||||||
|
- [ ] Implement read from current MemTable
|
||||||
|
- [ ] Add reads from immutable MemTables awaiting flush
|
||||||
|
- [ ] Create mechanism to read from Level 0 SSTable files
|
||||||
|
- [ ] Build priority-based lookup across all sources
|
||||||
|
|
||||||
|
## Phase E: Compaction
|
||||||
|
|
||||||
|
- [ ] Implement tiered compaction strategy
|
||||||
|
- [ ] Create file selection algorithm based on overlap/size
|
||||||
|
- [ ] Implement merge-sorted reading from input files
|
||||||
|
- [ ] Add atomic output file generation
|
||||||
|
- [ ] Create size ratio and file count based triggering
|
||||||
|
|
||||||
|
- [ ] Handle tombstones and key deletion
|
||||||
|
- [ ] Implement tombstone markers
|
||||||
|
- [ ] Create logic for tombstone garbage collection
|
||||||
|
- [ ] Test deletion correctness across compactions
|
||||||
|
|
||||||
|
- [ ] Manage file obsolescence and cleanup
|
||||||
|
- [ ] Implement safe file deletion after compaction
|
||||||
|
- [ ] Create consistent file tracking
|
||||||
|
- [ ] Add error handling for cleanup failures
|
||||||
|
|
||||||
|
- [ ] Build background compaction
|
||||||
|
- [ ] Implement worker pool for compaction tasks
|
||||||
|
- [ ] Add rate limiting to prevent I/O saturation
|
||||||
|
- [ ] Create metrics for monitoring compaction progress
|
||||||
|
- [ ] Implement priority scheduling for urgent compactions
|
||||||
|
|
||||||
|
## Phase F: Basic Atomicity and Features
|
||||||
|
|
||||||
|
- [ ] Implement merged iterator across all levels
|
||||||
|
- [ ] Create priority merging iterator
|
||||||
|
- [ ] Add efficient seeking capabilities
|
||||||
|
- [ ] Implement proper cleanup for resources
|
||||||
|
|
||||||
|
- [ ] Add snapshot capability
|
||||||
|
- [ ] Create point-in-time view mechanism
|
||||||
|
- [ ] Implement consistent reads across all data sources
|
||||||
|
- [ ] Add resource tracking and cleanup
|
||||||
|
- [ ] Test isolation guarantees
|
||||||
|
|
||||||
|
- [ ] Implement atomic batch operations
|
||||||
|
- [ ] Create batch data structure for multiple operations
|
||||||
|
- [ ] Implement atomic batch commit to WAL
|
||||||
|
- [ ] Add crash recovery for batches
|
||||||
|
- [ ] Design extensible interfaces for future transaction support
|
||||||
|
|
||||||
|
- [ ] Add basic statistics and metrics
|
||||||
|
- [ ] Implement counters for operations
|
||||||
|
- [ ] Add timing measurements for critical paths
|
||||||
|
- [ ] Create exportable metrics interface
|
||||||
|
- [ ] Test accuracy of metrics
|
||||||
|
|
||||||
|
## Phase G: Optimization and Benchmarking
|
||||||
|
|
||||||
|
- [ ] Develop benchmark suite
|
||||||
|
- [ ] Create random/sequential write benchmarks
|
||||||
|
- [ ] Implement point read and range scan benchmarks
|
||||||
|
- [ ] Add compaction overhead measurements
|
||||||
|
- [ ] Build reproducible benchmark harness
|
||||||
|
|
||||||
|
- [ ] Optimize critical paths
|
||||||
|
- [ ] Profile and identify bottlenecks
|
||||||
|
- [ ] Optimize memory usage patterns
|
||||||
|
- [ ] Improve cache efficiency in hot paths
|
||||||
|
- [ ] Reduce GC pressure for large operations
|
||||||
|
|
||||||
|
- [ ] Tune default configuration
|
||||||
|
- [ ] Benchmark with different parameters
|
||||||
|
- [ ] Determine optimal defaults for general use cases
|
||||||
|
- [ ] Document configuration recommendations
|
||||||
|
|
||||||
|
## Phase H: Optional Enhancements
|
||||||
|
|
||||||
|
- [ ] Add Bloom filters
|
||||||
|
- [ ] Implement configurable Bloom filter
|
||||||
|
- [ ] Add to SSTable format
|
||||||
|
- [ ] Create adaptive sizing based on false positive rates
|
||||||
|
- [ ] Benchmark improvement in read performance
|
||||||
|
|
||||||
|
- [ ] Create monitoring hooks
|
||||||
|
- [ ] Add detailed internal event tracking
|
||||||
|
- [ ] Implement exportable metrics
|
||||||
|
- [ ] Create health check mechanisms
|
||||||
|
- [ ] Add performance alerts
|
||||||
|
|
||||||
|
- [ ] Add crash recovery testing
|
||||||
|
- [ ] Build fault injection framework
|
||||||
|
- [ ] Create randomized crash scenarios
|
||||||
|
- [ ] Implement validation for post-recovery state
|
||||||
|
- [ ] Test edge cases in recovery
|
||||||
|
|
||||||
|
## API Implementation
|
||||||
|
|
||||||
|
- [ ] Implement Engine interface
|
||||||
|
- [ ] `Put(ctx context.Context, key, value []byte, opts ...WriteOption) error`
|
||||||
|
- [ ] `Get(ctx context.Context, key []byte, opts ...ReadOption) ([]byte, error)`
|
||||||
|
- [ ] `Delete(ctx context.Context, key []byte, opts ...WriteOption) error`
|
||||||
|
- [ ] `Batch(ctx context.Context, ops []Operation, opts ...WriteOption) error`
|
||||||
|
- [ ] `NewIterator(opts IteratorOptions) Iterator`
|
||||||
|
- [ ] `Snapshot() Snapshot`
|
||||||
|
- [ ] `Close() error`
|
||||||
|
|
||||||
|
- [ ] Implement error types
|
||||||
|
- [ ] `ErrIO` - I/O errors with recovery procedures
|
||||||
|
- [ ] `ErrCorruption` - Data integrity issues
|
||||||
|
- [ ] `ErrConfig` - Configuration errors
|
||||||
|
- [ ] `ErrResource` - Resource exhaustion
|
||||||
|
- [ ] `ErrConcurrency` - Race conditions
|
||||||
|
- [ ] `ErrNotFound` - Key not found
|
||||||
|
|
||||||
|
- [ ] Create comprehensive documentation
|
||||||
|
- [ ] API usage examples
|
||||||
|
- [ ] Configuration guidelines
|
||||||
|
- [ ] Performance characteristics
|
||||||
|
- [ ] Error handling recommendations
|
||||||
Loading…
Reference in New Issue
Block a user