diff --git a/IDEA.md b/IDEA.md index 2c69200..83888c1 100644 --- a/IDEA.md +++ b/IDEA.md @@ -23,13 +23,14 @@ Build a clean, composable, and educational storage engine in Go that follows Log - Support reproducible startup states across restarts ### 4. Composable Primitives -- Design clean interfaces for fundamental operations (reads, writes, snapshots, iteration) +- Design clean interfaces for fundamental operations (reads, writes, iteration, transactions) - Enable building of higher-level abstractions (SQL, Gremlin, custom query languages) - Support both transactional and analytical workloads - Provide simple atomic write primitives that can be built upon: - - Leverage read snapshots from immutable LSM structure + - Leverage reader-writer lock for concurrency control (SQLite-inspired approach) - Support basic atomic batch operations - Ensure crash recovery through proper WAL handling + - Enable read operations during writes via WAL (following SQLite's concurrency model) ## Target Use Cases diff --git a/PLAN.md b/PLAN.md index 7b34e37..8212e6c 100644 --- a/PLAN.md +++ b/PLAN.md @@ -37,8 +37,7 @@ go-storage/ │ │ └── footer/ # File footer and metadata │ ├── compaction/ # Compaction strategies │ ├── iterator/ # Merged iterator implementation -│ ├── transaction/ # Transaction management with Snapshot + WAL -│ │ ├── snapshot/ # Read snapshot implementation +│ ├── transaction/ # Transaction management with WAL and locks │ │ └── txbuffer/ # Transaction write buffer │ └── engine/ # Main engine implementation with single-writer architecture │ @@ -98,9 +97,10 @@ go-storage/ ### Phase F: Basic Atomicity and Advanced Features (2-3 weeks) 1. Implement merged iterator across all levels -2. Add snapshot capability for reads: - - Point-in-time view of the database - - Consistent reads across MemTable and SSTables +2. Implement SQLite-inspired reader-writer concurrency: + - Reader-writer lock for basic isolation + - WAL-based reads for active transactions + - Support for read operations during writes 3. Implement simple atomic batch operations: - Support atomic multi-key writes - Ensure proper crash recovery for batch operations @@ -137,15 +137,17 @@ go-storage/ ### Concurrency - Implement single-writer architecture for the main write path -- Allow concurrent readers (snapshots) to proceed without blocking -- Use appropriate synchronization for reader-writer coordination -- Ensure proper isolation between transactions +- Allow concurrent readers through reader-writer lock mechanism +- Use SQLite-inspired WAL approach for reader-writer coordination: + - Writers write to WAL, readers can read from either main database or WAL + - Use appropriate synchronization with reader-writer lock +- Ensure proper isolation between transactions through lock-based exclusion ### Batch Operation Management - Use WAL for atomic batch operation durability -- Leverage LSM's natural versioning for snapshots +- Leverage reader-writer lock for transactional isolation - Provide simple interfaces that can be built upon for transactions -- Ensure proper crash recovery for batch operations +- Ensure proper crash recovery for batch operations through WAL replay ### Go Idioms - Follow standard Go project layout diff --git a/TODO.md b/TODO.md index 5d13551..9815add 100644 --- a/TODO.md +++ b/TODO.md @@ -43,12 +43,12 @@ This document outlines the implementation tasks for the Go Storage Engine, organ - [x] Connect WAL replay to MemTable - [x] Create recovery logic to rebuild MemTable from WAL - - [x] Implement consistent snapshot reads during recovery + - [x] Implement consistent state reconstruction during recovery - [x] Handle errors during replay with appropriate fallbacks - [x] Test concurrent read/write scenarios - [x] Verify reader isolation during writes - - [x] Test snapshot consistency guarantees + - [x] Test consistency guarantees with concurrent operations - [x] Benchmark read/write performance under load ## Phase C: Persistent Storage @@ -121,11 +121,11 @@ This document outlines the implementation tasks for the Go Storage Engine, organ - [x] Add efficient seeking capabilities - [x] Implement proper cleanup for resources -- [ ] Add snapshot capability - - [ ] Create point-in-time view mechanism - - [ ] Implement consistent reads across all data sources - - [ ] Add resource tracking and cleanup - - [ ] Test isolation guarantees +- [ ] Implement SQLite-inspired reader-writer concurrency + - [ ] Add reader-writer lock for basic isolation + - [ ] Implement WAL-based reads during active write transactions + - [ ] Design clean API for transaction handling + - [ ] Test concurrent read/write operations - [ ] Implement atomic batch operations - [ ] Create batch data structure for multiple operations @@ -186,7 +186,7 @@ This document outlines the implementation tasks for the Go Storage Engine, organ - [ ] `Delete(ctx context.Context, key []byte, opts ...WriteOption) error` - [ ] `Batch(ctx context.Context, ops []Operation, opts ...WriteOption) error` - [ ] `NewIterator(opts IteratorOptions) Iterator` - - [ ] `Snapshot() Snapshot` + - [ ] `BeginTransaction(readOnly bool) (Transaction, error)` - [ ] `Close() error` - [ ] Implement error types