kevo/IDEA.md

Go Storage: A Minimalist LSM Storage Engine

Vision

Build a clean, composable, and educational storage engine in Go that follows Log-Structured Merge Tree (LSM) principles, focusing on simplicity while providing the building blocks needed for higher-level database implementations.

Goals

1. Extreme Simplicity

• Create minimal but complete primitives that can support various database paradigms (KV, relational, graph)
• Prioritize readability and educational value over hyper-optimization
• Use idiomatic Go with clear interfaces and documentation
• Implement a single-writer architecture for simplicity and reduced concurrency complexity

2. Durability + Performance

• Implement the LSM architecture pattern: Write-Ahead Log → MemTable → SSTables
• Provide configurable durability guarantees (sync vs. batched fsync)
• Optimize for both point lookups and range scans

3. Configurability

• Store all configuration parameters in a versioned, persistent manifest
• Allow tuning of memory usage, compaction behavior, and durability settings
• Support reproducible startup states across restarts

4. Composable Primitives

• Design clean interfaces for fundamental operations (reads, writes, snapshots, iteration)
• Enable building of higher-level abstractions (SQL, Gremlin, custom query languages)
• Support both transactional and analytical workloads
• Provide simple atomic write primitives that can be built upon:
  • Leverage read snapshots from immutable LSM structure
  • Support basic atomic batch operations
  • Ensure crash recovery through proper WAL handling

Target Use Cases

1. Educational Tool: Learn and teach storage engine internals
2. Embedded Storage: Applications needing local, durable storage with predictable performance
3. Prototype Foundation: Base layer for experimenting with novel database designs
4. Go Ecosystem Component: Reusable storage layer for Go applications and services

Non-Goals

1. Feature Parity with Production Engines: Not trying to compete with RocksDB, LevelDB, etc.
2. Multi-Node Distribution: Focusing on single-node operation
3. Complex Query Planning: Leaving higher-level query features to layers built on top

Success Criteria

1. Correctness: Data is never lost or corrupted, even during crashes
2. Understandability: Code is clear enough to serve as an educational reference
3. Performance: Reasonable throughput and latency for common operations
4. Extensibility: Can be built upon to create specialized database engines