thomasjungblut/go-sstables

go-sstables

go-sstables is a Go library that contains NoSQL database building blocks like a sequential record format (recordio), a sorted string table (sstable), a write-ahead-log (WAL), and a memory store (memstore) that stores key/value pairs in memory using a skip list.

You can frequently find those in embedded key-value stores or databases as well, notable examples are RocksDB or LevelDB.

There is an example embedded key-value database in this library, you can find it in the simpledb folder - please don't use it for any production workload.

While plain []byte are at the core of this library, there are wrappers and bindings for protobuf to enable more convenient serialization.

Installation

This is a library as it does not contain any installable binary, which means you can just directly add it to your dependency via go get:

go get -d github.com/thomasjungblut/go-sstables

Documentation

RocksDB has a great overview of how the components usually play together to get an idea:

You will find basically all of those mentioned pieces above and all of them stitched together as SimpleDB. The documentation is now separated by package for easier browsing, each of those READMEs contain examples - there is also /examples when you prefer browsing the code directly.

• RecordIO
  • Benchmark
• sstables
  • Benchmark
• Memstore
• SkipList
• WriteAheadLog
• SimpleDB
  • Benchmark

You can also find all interface and method documentation on pkg.go.dev .

Kaitai Support

As you might want to read the data and files in other languages, I've added support for Kaitai.
Kaitai is a declarative schema file to define a binary format. From that ksy file you can generate code for a lot of other languages and read the data.

Currently, there is support for:

• RecordIO (v2)
• RecordIO (v3)
• RecordIO (v4)

You can find more information on how to generate Kaitai readers in kaitai/README.md.

Development on go-sstables

Thank you

I want to deeply thank Sébastien Heitzmann (@sebheitzmann) for his continued support, reviews and usage of this library.

Generating protobufs

This needs some pre-requisites installed, namely the protobuf compiler and the go generator plugin. The latter can be installed as a go package:

go install google.golang.org/protobuf/cmd/protoc-gen-go

Full installation details can be found in the protobuf dev documentation .

Once installed, one can generate the protobuf structs using:

make compile-proto

Releasing the Go Module

General Guidance

In short, run these commands:

make unit-test
make release
git push --tags