ankur-anand/isledb
ankur-anand/isledb has 60 GitHub stars and 11 forks. GitHot measured 0 in 7 days and 0 in 15 days. Last checked 2026-11-15 15:58 UTC. Last push 2026-10-31.
Measured star growth
- in 24h
- Update delayed
- in 7 days
- 0
- in 15 days
- 0
- in 1 month
- Update delayed
Net star changes over overlapping periods, not daily totals. Missing measurements are left blank. Stars measure attention, not software quality.
About
A key-value database for object storage (S3, GCS, Azure Blob), in Go.
DataREADME.md
OriginalIsleDB
A key-value database for object storage, written in Go.
Store durable key-value data on Amazon S3, Google Cloud Storage, Azure Blob Storage, or any S3-compatible store such as MinIO; local files and memory work for development. Add reader services independently and run compaction outside your application processes.
IsleDB provides:
- point reads, range scans, iterators, snapshots, TTLs, and deletes;
- one fenced writer with any number of independent reader processes, usable
from any language through an optional HTTP write service (
isledb writer); - configurable local caching for readers, and an optional shared read cache
per region (
isledb cache); - optional keys-only or full-value change feeds;
- compaction and storage cleanup that can run in a separate process or scheduled job.
Use IsleDB as a Go library, or run it as services that any language reaches over HTTP:
Use it as a Go library · Run it as services
When IsleDB fits
IsleDB is a good fit when:
- object storage is the durability and capacity layer;
- writes can be buffered and published in batches;
- readers should scale without maintaining database replicas;
- second-scale freshness is acceptable;
- compaction should run away from application servers.
Choose a different database when you need:
- consistent sub-10-millisecond read-after-write latency;
- multi-key transactions or compare-and-swap updates;
- secondary indexes, joins, or a query language;
- high-frequency updates to a small, hot dataset.
Use it as a Go library
Install
go get github.com/ankur-anand/isledb
IsleDB requires Go 1.25 or newer.
Quick start
This example uses an Amazon S3 bucket; credentials come from the standard AWS
environment (AWS_PROFILE, AWS_ACCESS_KEY_ID, an instance role, and so on).
Put buffers the mutation and Flush makes it durable and visible to readers
that refresh.
package main
import (
"context"
"fmt"
"log"
"path/filepath"
"github.com/ankur-anand/isledb"
)
func main() {
if err := run(context.Background()); err != nil {
log.Fatal(err)
}
}
func run(ctx context.Context) error {
db, err := isledb.Open(ctx, "s3://my-bucket?region=us-east-1", isledb.DBOptions{
Prefix: "example",
})
if err != nil {
return err
}
defer func() { _ = db.Close() }()
writerOptions := isledb.DefaultWriterOptions()
// Commit only on Flush, for this example; by default the writer also
// commits every second.
writerOptions.Flush.Interval = 0
writer, err := db.OpenWriter(ctx, writerOptions)
if err != nil {
return err
}
if _, err := writer.Put(ctx, []byte("hello"), []byte("world")); err != nil {
return err
}
if err := writer.Flush(ctx); err != nil {
return err
}
if err := writer.Close(ctx); err != nil {
return err
}
cacheDir, err := filepath.Abs("./isledb-cache")
if err != nil {
return err
}
reader, err := db.OpenReader(
ctx,
isledb.DefaultReaderOpenOptions(cacheDir),
)
if err != nil {
return err
}
defer func() { _ = reader.Close() }()
value, found, err := reader.Get(ctx, []byte("hello"))
if err != nil {
return err
}
if found {
fmt.Printf("hello = %s\n", value)
}
return nil
}
The same API works with cloud bucket URLs:
| Provider | Example bucket URL |
|---|---|
| Amazon S3 | s3://my-bucket?region=us-east-1 |
| Google Cloud Storage | gs://my-bucket |
| Azure Blob Storage | azblob://my-container |
Credentials and provider-specific options come from the corresponding Go Cloud bucket driver. Use a unique prefix for each database.
Local stores are for development only.
file://andmem://buckets behave correctly within one process, which is enough for local experiments and tests. Across processes, their conditional writes are not atomic, so never point two processes at the samefile://directory, and do not use them in production. Use S3, Google Cloud Storage, Azure Blob Storage, or an S3-compatible store.
Behavior to know
Put,PutWithTTL, andDeletebuffer mutations in memory and return each mutation's sequence.WaitCommitted(ctx, seq)returns once that write is in object storage; use it before acting on a write you cannot lose.- If storage is down, writes wait in memory and are retried; once
MaxPendingMemtablesmemtables are waiting,PutreturnsErrBackpressure. - A successful
Flush, background flush, orWriter.Closeis the durability and visibility boundary.Flushcan be retried;Closemakes one attempt and finishes the writer either way, reporting any writes not known to be committed. To stop gracefully, callDrain, thenClosewith a deadline, thenDB.Close; see Shutting down. - One writer owns a database prefix at a time. Writer ownership is fenced across processes.
- A reader uses a consistent loaded view. It refreshes according to its view
policy; call
Refreshwhen a newly committed write must be discovered immediately. Writer.Statereports whether the writer is open, stopped, closed or fenced, and how much is not yet committed, for readiness probes and dashboards.- Reader services scale independently by opening the same bucket and prefix with their own local cache directories.
- Run maintenance in production so compaction, checkpoints, configured retention, and physical cleanup continue to make progress.
Shutting down
Drain stops intake and commits everything accepted, retrying until its
context ends; Close is the final step:
if err := writer.Drain(ctx); err != nil { // Put and Delete now return ErrWritesStopped
log.Printf("drain: %v", err)
}
closeCtx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
defer cancel()
if err := writer.Close(closeCtx); err != nil {
log.Printf("writer closed with unconfirmed writes: %v", err) // the error names them
}
_ = db.Close()
DB.Close alone closes an open writer with a fixed 30-second deadline and no
retry; it is cleanup, not the durability step. Set Flush.Interval so a
crash loses at most one interval of writes.
See the Go API guide for scans, iterators, snapshots, prefetching, configuration, metrics, and error handling.
Change feed
Enable the change feed when another service needs committed mutations in writer order:
db, err := isledb.Open(ctx, bucketURL, isledb.DBOptions{
Prefix: "prod/accounts",
ChangeFeed: &isledb.ChangeFeedOptions{
Payload: isledb.ChangeFeedFullValues,
},
})
Use ChangeFeedKeysOnly for invalidation or downstream re-fetching. Use
ChangeFeedFullValues when consumers need the committed PUT values. The mode
is persisted when the feed is enabled and cannot later be changed for that
database prefix.
Consumers use OpenChangeReader with an opaque, persistent cursor. See
Change feed
for the complete example and retention behavior.
Maintenance
Maintenance can run continuously in its own service or periodically as a job. It opens the same database prefix as the writer and readers.
func runMaintenance(ctx context.Context, bucketURL string) error {
db, err := isledb.Open(ctx, bucketURL, isledb.DBOptions{
Prefix: "prod/accounts",
})
if err != nil {
return err
}
defer func() { _ = db.Close() }()
maintenance, err := db.OpenMaintenance(ctx, isledb.DefaultMaintenanceOptions())
if err != nil {
return err
}
defer func() { _ = maintenance.Close(context.Background()) }()
return maintenance.Run(ctx) // until ctx ends or another process takes over maintenance
}
The active writer publishes maintenance results, while cleanup proceeds at its
own bounded pace. Use RunOnce instead of Run for a scheduled job.
To run it without writing code, see the maintenance service.
Reading through the shared cache
With an isledb cache in the region, point readers and
change-feed consumers at it:
opts := isledb.DefaultReaderOpenOptions(cacheDir)
opts.ReadCache = isledb.ReadCacheOptions{
URL: "http://isledb-cache:8090",
Token: token,
}
reader, err := db.OpenReader(ctx, opts)
feed, err := db.OpenChangeReader(ctx, isledb.ChangeReaderOptions{
ReadCache: isledb.ReadCacheOptions{URL: "http://isledb-cache:8090", Token: token},
})
Run it as services
One binary, isledb, runs each service. Get it from the
latest release,
for Linux (amd64, arm64) and macOS (arm64), or run the container image:
docker run ghcr.io/ankur-anand/isledb:0.20.0 help
The image is tagged with each release's exact version, such as 0.20.0, and
with its minor version, such as 0.20; there is no latest tag. With Go
installed, you can also build it:
go install github.com/ankur-anand/isledb/cmd/isledb@latest
| Service | Does | Listens on | Run |
|---|---|---|---|
isledb writer |
Takes writes over HTTP | :8070 |
One per database |
isledb reader |
Serves reads and scans over HTTP | :8080 |
As many as you need |
isledb maintenance |
Compacts and cleans up; no API | admin only, 127.0.0.1:8061 |
One per database |
isledb cache |
Optional regional read cache | :8090 |
One or more per region |
Each also serves health checks and metrics on its admin address. Run
isledb help for the list, and isledb <service> -h for a service's flags.
Writer service
isledb writer holds the writer for one database and serves writes over
HTTP, so services in any language can write:
isledb writer \
--bucket-url 's3://my-bucket?region=us-east-1' \
--prefix prod/accounts \
--auth-token-file /etc/isledb/tokens
curl -X PUT -H "Authorization: Bearer $TOKEN" \
--data '{"name":"a"}' 'http://isledb-writer:8070/v1/kv/user/42?ttl=24h'
# {"write_id":"1.1","revision":"2"}
A write is answered once it is durable in object storage; batches of puts and
deletes go to POST /v1/kv and apply atomically. Run one writer per
database. See Running isledb writer and the
writer HTTP API.
Reader service
isledb reader serves reads over HTTP. Replicas share nothing, so you scale
reads by adding them behind a load balancer:
isledb reader \
--bucket-url 's3://my-bucket?region=us-east-1' \
--prefix prod/accounts \
--cache-dir /var/cache/isledb \
--auth-token-file /etc/isledb/tokens
curl -H "Authorization: Bearer $TOKEN" 'http://isledb-reader:8080/v1/kv/user/42'
# {"name":"a"}
curl -H "Authorization: Bearer $TOKEN" 'http://isledb-reader:8080/v1/kv?prefix=user/'
# {"items":[{"key":"user/42","value":"{\"name\":\"a\"}"}]}
If the database has a change feed, readers also serve it at
/v1/changes?from=…, so consumers in any language can follow it.
The paths match the writer's, so one gateway can route GET to readers and
writes to the writer. A read is usually within a second of the writer (about
two on Google Cloud Storage), and its Isledb-Revision header says which
writes it includes. See
Running isledb reader and the
reader HTTP API.
Maintenance service
isledb maintenance compacts the database, keeps its metadata short and
deletes what is no longer needed. It has no API. Run exactly one per
database, alongside the writer:
isledb maintenance \
--bucket-url 's3://my-bucket?region=us-east-1' \
--prefix prod/accounts \
--scratch-dir /var/lib/isledb/scratch
A second one for the same database takes over, and the first stops working
and reports not ready. See
Running isledb maintenance.
Shared read cache
isledb cache is an optional service that sits between readers and object
storage, typically one per region. It serves immutable objects (SSTs,
manifest pages, change batches) from chunks on its local disk, and relays
CURRENT so readers learn of new commits through a long poll instead of
polling storage.
Run it next to the readers:
isledb cache \
--bucket-url 's3://my-bucket?region=us-east-1' \
--db prod/accounts \
--disk-dir /var/cache/isledb --disk-bytes 200GiB \
--auth-token-file /etc/isledb/tokens
The cache only makes reads faster and cheaper; it never decides what readers
see. Readers check what it returns against checksums, reject a copy of
CURRENT that is too old, and read storage directly whenever the cache is
slow, down, or wrong.
See Running isledb cache and the
read cache HTTP API.
An isledb reader uses it through read_cache.url; a Go program, through
ReadCacheOptions.
Versions and support
IsleDB is before 1.0. A minor release, such as 0.19 to 0.20, may change the Go API, the services' flags and configuration, the HTTP API, or metrics; its release notes say what changed and how to move to it. A patch release, such as 0.19.0 to 0.19.1, only fixes problems.
Fixes go into the next release; earlier releases are not patched. Run a recent release, and name its exact version wherever you deploy it.
Documentation
Library:
- Go API guide
- Go package reference
- Local quickstart
- Separate writer, reader, and maintenance services with MinIO
- Restartable change-feed consumer with Azurite
Services:
- Running the writer service and its HTTP API
- Running the reader service and its HTTP API
- Running the maintenance service
- Running the shared read cache and its HTTP API
Storage:
Acknowledgments
IsleDB uses the SSTable implementation from Pebble. Thanks to the CockroachDB team for building and open-sourcing it.
License
IsleDB is licensed under the Apache License 2.0.