2d1e42b4d6
feat: implement integrity validation with checksums for replication transport
...
- Add checksums for WAL entries and WAL entry batches
- Implement robust retry and circuit breaker patterns for reliability
- Add comprehensive tests for message processing and reliability features
- Enhance error handling and timeout management
2025-04-26 14:07:31 -06:00
61858f595e
feat: implement reliability features for replication transport
...
This commit adds comprehensive reliability features to the replication transport layer:
- Add retry logic with exponential backoff for all network operations
- Implement circuit breaker pattern to prevent cascading failures
- Add reconnection handling with automatic recovery
- Implement proper timeout handling for all network operations
- Add comprehensive logging for connection issues
- Improve error handling with temporary error classification
- Enhance stream processing with automatic recovery
2025-04-26 13:32:23 -06:00
5963538bc5
feat: implement replication transport layer
...
Go Tests / Run Tests (1.24.2) (pull_request) Successful in 9m49s
This commit implements the replication transport layer as part of Phase 2 of the replication plan.
Key components include:
- Add protocol buffer definitions for replication services
- Implement WALReplicator extension for processor management
- Create replication service server implementation
- Add replication client and server transport implementations
- Implement storage snapshot interface for bootstrap operations
- Standardize package naming across replication components
2025-04-26 13:05:54 -06:00
ed991ae00d
feat: add replication transport interfaces and protocol schema
...
- Add replication-specific interfaces to pkg/transport
- Create ReplicationClient and ReplicationServer interfaces
- Add replication message types for WAL entries and bootstrap
- Create Protobuf schema for replication in proto/kevo/replication.proto
- Update transport registry to support replication components
2025-04-26 12:33:38 -06:00
33ddfeeb64
fix: use Lamport clocks consistently across WAL operations
...
Ensure proper Lamport clock integration across batch operations and sequence number handling:
- Add Lamport clock support to Batch.Write for consistent replication timestamps
- Fix potential sequence number inconsistencies in WAL operations
- Update WAL tests to properly verify sync mode behaviors
- Fix Sequence numbering to appropriately handle Lamport clock timestamps
2025-04-26 12:24:50 -06:00
5cd1f5c5f8
feat: implement WAL applier for replication
...
- Add WALApplier component for applying entries on replica nodes
- Implement logical timestamp ordering with Lamport clocks
- Add support for handling out-of-order entry delivery
- Add error handling and recovery mechanisms
- Implement comprehensive testing for all applier functions
2025-04-26 12:02:53 -06:00
02febadf5d
feat: implement WAL replicator and entry serialization
...
- Add WAL replicator component with entry capture, buffering, and subscriptions
- Implement WAL entry serialization with checksumming
- Add batch serialization for network-efficient transfers
- Implement proper concurrency control with mutex protection
- Add utility functions for entry size estimation
- Create comprehensive test suite
2025-04-26 11:54:19 -06:00
5b2ecdd08c
fix: add the retry logic
2025-04-26 11:48:51 -06:00
c0bfd835f7
feat: implement replication hook point in WAL and Lamport clocks…
...
Go Tests / Run Tests (1.24.2) (pull_request) Failing after 5m1s
- Add Lamport clock implementation for logical timestamps
- Define ReplicationHook interface in WAL package
- Extend WAL to use Lamport clock for timestamps
- Add notification hooks for WAL entries and batches
- Update WAL initialization to support replication
- Add tests for replication hooks and clocks
2025-04-26 11:42:49 -06:00
0e5d60cb83
feat: add lamport timestamp
2025-04-26 11:28:44 -06:00
9fb40779c7
docs: update readme, remove multi-node distribution as a non-goal
Go Tests / Run Tests (1.24.2) (pull_request) Failing after 5m2s
2025-04-26 10:03:19 -06:00
8e04c2cea3
fix: fix the build error from before
Go Tests / Run Tests (1.24.2) (push) Failing after 5m2s
2025-04-26 04:11:32 -06:00
ae75f2935f
fix: fix the WAL is closed errors on rotation in tight loops, increase durability
Go Tests / Run Tests (1.24.2) (push) Failing after 5m5s
2025-04-26 00:15:50 -06:00
7e226825df
fix: engine refactor bugfix fest, go fmt
Go Tests / Run Tests (1.24.2) (push) Successful in 9m48s
2025-04-25 23:36:08 -06:00
169c1f78a4
chore: update the layout of the interface design in readme
Go Tests / Run Tests (1.24.2) (push) Failing after 5m2s
2025-04-25 13:04:08 -06:00
0637c40a40
feat: big refactor cleaning up the engine code
Go Tests / Run Tests (1.24.2) (push) Failing after 5m4s
2025-04-23 22:45:16 -06:00
7dd816bdf5
feat: add a standard stats collector, needs integration
2025-04-23 20:01:18 -06:00
d5a90cf2e4
perf: added bloom filters for sstables to improve performance
Go Tests / Run Tests (1.24.2) (push) Failing after 5m2s
2025-04-23 19:04:10 -06:00
2e3c70e913
test: add a concurrent read/write test on memtables
Go Tests / Run Tests (1.24.2) (push) Successful in 9m48s
2025-04-23 18:20:21 -06:00
00b2566464
perf: updated skiplist find algorithm to be more efficient and cache aware
Go Tests / Run Tests (1.24.2) (push) Has been cancelled
2025-04-23 18:16:44 -06:00
2b90635021
test: add tests for edge case putting a key after delete
2025-04-23 16:49:38 -06:00
6f83fa1ade
feat: add suffix scanning
Go Tests / Run Tests (1.24.2) (push) Successful in 9m49s
2025-04-23 09:00:26 -06:00
3219e7527e
docs: remove extra character in contribution guidelines
2025-04-23 03:14:19 -06:00
f06357a2b0
docs: fix some grammar in contribution guidelines
2025-04-23 03:13:54 -06:00
a8c72fccfa
feat: add logger finally...
Go Tests / Run Tests (1.24.2) (push) Successful in 9m48s
2025-04-23 02:28:45 -06:00
139c33533e
docs: forgot that emdash's trigger some people
...
Don't want folks to think of this as AI slop, so remove the emdash, use a poor man's emdash (--) instead.
2025-04-23 02:23:32 -06:00
2d07166c4f
docs: create contributing guidelines
...
A frank, and good enough set of guidelines for how this project will be run.
2025-04-23 02:21:01 -06:00
a0a1c0512f
chore: formatting
Go Tests / Run Tests (1.24.2) (push) Successful in 9m50s
2025-04-22 14:09:54 -06:00
e7974e008d
feat: enhance wal recover statistics
2025-04-22 14:09:45 -06:00
dependabot[bot]
3b3d1c27a4
chore(deps): bump golang.org/x/net from 0.35.0 to 0.38.0
...
Bumps [golang.org/x/net](https://github.com/golang/net ) from 0.35.0 to 0.38.0.
- [Commits](https://github.com/golang/net/compare/v0.35.0...v0.38.0 )
---
updated-dependencies:
- dependency-name: golang.org/x/net
dependency-version: 0.38.0
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com>
2025-04-22 11:10:02 -06:00
91448203e6
chore: add sdk links to readme
Go Tests / Run Tests (1.24.2) (push) Successful in 9m49s
2025-04-22 11:08:50 -06:00
f77889873e
chore: add license file
2025-04-22 11:05:08 -06:00
fbea5da627
chore: renaming packages after move to github org
2025-04-22 11:01:33 -06:00
6138f4e1d5
fix: scan family of rpc calls were not returning any results
2025-04-22 11:01:33 -06:00
c46da3dfce
feat: add client sdk and fix tests
2025-04-22 11:01:33 -06:00
cb8776dcc9
feat: add gRPC service
2025-04-22 11:01:33 -06:00
e60bf8ad56
docs: add usage information to cli
2025-04-22 11:01:33 -06:00
0ce19934ba
feat: add common transport interface and server mode to kevo
2025-04-22 11:01:33 -06:00
001934e7b5
refactor: optimize two pass deduping into a single pass across the codebase
Go Tests / Run Tests (1.24.2) (push) Successful in 9m38s
2025-04-21 16:27:54 -06:00
1a0de922af
refactor: use memtable sorted order iterator to write and dedup in a single pass
2025-04-21 14:53:03 -06:00
4e813aa3fd
fix: flushing a memtable dedups keys as its processing them before creating the SSTable
2025-04-21 14:53:03 -06:00
87285d931e
docs: fix link in readme on how to go get
Go Tests / Run Tests (1.24.2) (push) Successful in 9m37s
2025-04-20 18:22:31 -06:00
dependabot[bot]
90ebdd4de9
chore(deps): bump golang.org/x/sys
...
Bumps [golang.org/x/sys](https://github.com/golang/sys ) from 0.0.0-20220310020820-b874c991c1a5 to 0.1.0.
- [Commits](https://github.com/golang/sys/commits/v0.1.0 )
---
updated-dependencies:
- dependency-name: golang.org/x/sys
dependency-version: 0.1.0
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com>
2025-04-20 18:19:48 -06:00
477cfa1817
chore: renamed cli from gs to kevo
Go Tests / Run Tests (1.24.2) (push) Has been cancelled
2025-04-20 18:16:26 -06:00
3450873ef0
chore: forgot one artifact when renaming
Go Tests / Run Tests (1.24.2) (push) Successful in 9m36s
2025-04-20 14:59:43 -06:00
25fe3b1771
chore: i am an idiot, forgot to change the full path parts
Go Tests / Run Tests (1.24.2) (push) Has been cancelled
2025-04-20 14:53:39 -06:00
d6d164752c
chore: rename from private repo to public GH repo in readme
Go Tests / Run Tests (1.24.2) (push) Failing after 4m50s
2025-04-20 14:29:54 -06:00
9752914c7e
docs: fix license link so it renders as markdown
Go Tests / Run Tests (1.24.2) (push) Failing after 4m48s
2025-04-20 14:10:14 -06:00
6fc3be617d
feat: Initial release of kevo storage engine.
...
Go Tests / Run Tests (1.24.2) (push) Has been cancelled
Adds a complete LSM-based storage engine with these features:
- Single-writer based architecture for the storage engine
- WAL for durability, and hey it's configurable
- MemTable with skip list implementation for fast read/writes
- SSTable with block-based structure for on-disk level-based storage
- Background compaction with tiered strategy
- ACID transactions
- Good documentation (I hope)
2025-04-20 14:06:50 -06:00
