Jeremy Tregunna
6fc3be617d
Some checks failed
Go Tests / Run Tests (1.24.2) (push) Has been cancelled
Adds a complete LSM-based storage engine with these features: - Single-writer based architecture for the storage engine - WAL for durability, and hey it's configurable - MemTable with skip list implementation for fast read/writes - SSTable with block-based structure for on-disk level-based storage - Background compaction with tiered strategy - ACID transactions - Good documentation (I hope)
3.0 KiB
3.0 KiB
Storage Benchmark Utility
This utility benchmarks the performance of the Kevo storage engine under various workloads.
Usage
go run ./cmd/storage-bench/... [flags]
Available Flags
-type: Type of benchmark to run (write, read, scan, mixed, tune, or all) [default: all]-duration: Duration to run each benchmark [default: 10s]-keys: Number of keys to use [default: 100000]-value-size: Size of values in bytes [default: 100]-data-dir: Directory to store benchmark data [default: ./benchmark-data]-sequential: Use sequential keys instead of random [default: false]-cpu-profile: Write CPU profile to file [optional]-mem-profile: Write memory profile to file [optional]-results: File to write results to (in addition to stdout) [optional]-tune: Run configuration tuning benchmarks [default: false]
Example Commands
Run all benchmarks with default settings:
go run ./cmd/storage-bench/...
Run only write benchmark with 1 million keys and 1KB values for 30 seconds:
go run ./cmd/storage-bench/... -type=write -keys=1000000 -value-size=1024 -duration=30s
Run read and scan benchmarks with sequential keys:
go run ./cmd/storage-bench/... -type=read,scan -sequential
Run with profiling enabled:
go run ./cmd/storage-bench/... -cpu-profile=cpu.prof -mem-profile=mem.prof
Run configuration tuning benchmarks:
go run ./cmd/storage-bench/... -tune
Benchmark Types
- Write Benchmark: Measures throughput and latency of key-value writes
- Read Benchmark: Measures throughput and latency of key lookups
- Scan Benchmark: Measures performance of range scans
- Mixed Benchmark: Simulates real-world workload with 75% reads, 25% writes
- Compaction Benchmark: Tests compaction throughput and overhead (available through code API)
- Tuning Benchmark: Tests different configuration parameters to find optimal settings
Result Interpretation
Benchmark results include:
- Operations per second (throughput)
- Average latency per operation
- Hit rate for read operations
- Throughput in MB/s for compaction
- Memory usage statistics
Configuration Tuning
The tuning benchmark tests various configuration parameters including:
MemTableSize: Sizes tested: 16MB, 32MBSSTableBlockSize: Sizes tested: 8KB, 16KBWALSyncMode: Modes tested: None, BatchCompactionRatio: Ratios tested: 10.0, 20.0
Tuning results are saved to:
tuning_results.json: Detailed benchmark metrics for each configurationrecommendations.md: Markdown file with performance analysis and optimal configuration recommendations
The recommendations include:
- Optimal settings for write-heavy workloads
- Optimal settings for read-heavy workloads
- Balanced settings for mixed workloads
- Additional configuration advice
Profiling
Use the -cpu-profile and -mem-profile flags to generate profiling data that can be analyzed with:
go tool pprof cpu.prof
go tool pprof mem.prof