Skip to content
Projects

ankur-anand/isledb has 60 GitHub stars and 11 forks. GitHot measured 0 in 7 days and 0 in 15 days. Last checked 2026-11-15 15:58 UTC. Last push 2026-10-31.

How these counts are measured

Measured star growth

in 24h
Update delayed
in 7 days
0
in 15 days
0
in 1 month
Update delayed

Net star changes over overlapping periods, not daily totals. Missing measurements are left blank. Stars measure attention, not software quality.

About

A key-value database for object storage (S3, GCS, Azure Blob), in Go.

Data

README.md

Original

IsleDB

IsleDB

CI Tests Coverage Status Go Reference License: Apache 2.0

A key-value database for object storage, written in Go.

Store durable key-value data on Amazon S3, Google Cloud Storage, Azure Blob Storage, or any S3-compatible store such as MinIO; local files and memory work for development. Add reader services independently and run compaction outside your application processes.

IsleDB provides:

  • point reads, range scans, iterators, snapshots, TTLs, and deletes;
  • one fenced writer with any number of independent reader processes, usable from any language through an optional HTTP write service (isledb writer);
  • configurable local caching for readers, and an optional shared read cache per region (isledb cache);
  • optional keys-only or full-value change feeds;
  • compaction and storage cleanup that can run in a separate process or scheduled job.
IsleDB architecture: one writer commits to object storage; maintenance compacts it; reader processes read through their local cache, then an optional shared read cache, then object storage; the shared cache tells readers about new commits.

Use IsleDB as a Go library, or run it as services that any language reaches over HTTP:

Use it as a Go library · Run it as services

When IsleDB fits

IsleDB is a good fit when:

  • object storage is the durability and capacity layer;
  • writes can be buffered and published in batches;
  • readers should scale without maintaining database replicas;
  • second-scale freshness is acceptable;
  • compaction should run away from application servers.

Choose a different database when you need:

  • consistent sub-10-millisecond read-after-write latency;
  • multi-key transactions or compare-and-swap updates;
  • secondary indexes, joins, or a query language;
  • high-frequency updates to a small, hot dataset.

Use it as a Go library

Install

go get github.com/ankur-anand/isledb

IsleDB requires Go 1.25 or newer.

Quick start

This example uses an Amazon S3 bucket; credentials come from the standard AWS environment (AWS_PROFILE, AWS_ACCESS_KEY_ID, an instance role, and so on). Put buffers the mutation and Flush makes it durable and visible to readers that refresh.

package main

import (
	"context"
	"fmt"
	"log"
	"path/filepath"

	"github.com/ankur-anand/isledb"
)

func main() {
	if err := run(context.Background()); err != nil {
		log.Fatal(err)
	}
}

func run(ctx context.Context) error {
	db, err := isledb.Open(ctx, "s3://my-bucket?region=us-east-1", isledb.DBOptions{
		Prefix: "example",
	})
	if err != nil {
		return err
	}
	defer func() { _ = db.Close() }()

	writerOptions := isledb.DefaultWriterOptions()
	// Commit only on Flush, for this example; by default the writer also
	// commits every second.
	writerOptions.Flush.Interval = 0

	writer, err := db.OpenWriter(ctx, writerOptions)
	if err != nil {
		return err
	}
	if _, err := writer.Put(ctx, []byte("hello"), []byte("world")); err != nil {
		return err
	}
	if err := writer.Flush(ctx); err != nil {
		return err
	}
	if err := writer.Close(ctx); err != nil {
		return err
	}

	cacheDir, err := filepath.Abs("./isledb-cache")
	if err != nil {
		return err
	}
	reader, err := db.OpenReader(
		ctx,
		isledb.DefaultReaderOpenOptions(cacheDir),
	)
	if err != nil {
		return err
	}
	defer func() { _ = reader.Close() }()

	value, found, err := reader.Get(ctx, []byte("hello"))
	if err != nil {
		return err
	}
	if found {
		fmt.Printf("hello = %s\n", value)
	}
	return nil
}

The same API works with cloud bucket URLs:

Provider Example bucket URL
Amazon S3 s3://my-bucket?region=us-east-1
Google Cloud Storage gs://my-bucket
Azure Blob Storage azblob://my-container

Credentials and provider-specific options come from the corresponding Go Cloud bucket driver. Use a unique prefix for each database.

Local stores are for development only. file:// and mem:// buckets behave correctly within one process, which is enough for local experiments and tests. Across processes, their conditional writes are not atomic, so never point two processes at the same file:// directory, and do not use them in production. Use S3, Google Cloud Storage, Azure Blob Storage, or an S3-compatible store.

Behavior to know

  • Put, PutWithTTL, and Delete buffer mutations in memory and return each mutation's sequence. WaitCommitted(ctx, seq) returns once that write is in object storage; use it before acting on a write you cannot lose.
  • If storage is down, writes wait in memory and are retried; once MaxPendingMemtables memtables are waiting, Put returns ErrBackpressure.
  • A successful Flush, background flush, or Writer.Close is the durability and visibility boundary. Flush can be retried; Close makes one attempt and finishes the writer either way, reporting any writes not known to be committed. To stop gracefully, call Drain, then Close with a deadline, then DB.Close; see Shutting down.
  • One writer owns a database prefix at a time. Writer ownership is fenced across processes.
  • A reader uses a consistent loaded view. It refreshes according to its view policy; call Refresh when a newly committed write must be discovered immediately.
  • Writer.State reports whether the writer is open, stopped, closed or fenced, and how much is not yet committed, for readiness probes and dashboards.
  • Reader services scale independently by opening the same bucket and prefix with their own local cache directories.
  • Run maintenance in production so compaction, checkpoints, configured retention, and physical cleanup continue to make progress.

Shutting down

Drain stops intake and commits everything accepted, retrying until its context ends; Close is the final step:

if err := writer.Drain(ctx); err != nil { // Put and Delete now return ErrWritesStopped
    log.Printf("drain: %v", err)
}
closeCtx, cancel := context.WithTimeout(context.Background(), 2*time.Minute)
defer cancel()
if err := writer.Close(closeCtx); err != nil {
    log.Printf("writer closed with unconfirmed writes: %v", err) // the error names them
}
_ = db.Close()

DB.Close alone closes an open writer with a fixed 30-second deadline and no retry; it is cleanup, not the durability step. Set Flush.Interval so a crash loses at most one interval of writes.

See the Go API guide for scans, iterators, snapshots, prefetching, configuration, metrics, and error handling.

Change feed

Enable the change feed when another service needs committed mutations in writer order:

db, err := isledb.Open(ctx, bucketURL, isledb.DBOptions{
	Prefix: "prod/accounts",
	ChangeFeed: &isledb.ChangeFeedOptions{
		Payload: isledb.ChangeFeedFullValues,
	},
})

Use ChangeFeedKeysOnly for invalidation or downstream re-fetching. Use ChangeFeedFullValues when consumers need the committed PUT values. The mode is persisted when the feed is enabled and cannot later be changed for that database prefix.

Consumers use OpenChangeReader with an opaque, persistent cursor. See Change feed for the complete example and retention behavior.

Maintenance

Maintenance can run continuously in its own service or periodically as a job. It opens the same database prefix as the writer and readers.

func runMaintenance(ctx context.Context, bucketURL string) error {
	db, err := isledb.Open(ctx, bucketURL, isledb.DBOptions{
		Prefix: "prod/accounts",
	})
	if err != nil {
		return err
	}
	defer func() { _ = db.Close() }()

	maintenance, err := db.OpenMaintenance(ctx, isledb.DefaultMaintenanceOptions())
	if err != nil {
		return err
	}
	defer func() { _ = maintenance.Close(context.Background()) }()

	return maintenance.Run(ctx) // until ctx ends or another process takes over maintenance
}

The active writer publishes maintenance results, while cleanup proceeds at its own bounded pace. Use RunOnce instead of Run for a scheduled job.

To run it without writing code, see the maintenance service.

Reading through the shared cache

With an isledb cache in the region, point readers and change-feed consumers at it:

opts := isledb.DefaultReaderOpenOptions(cacheDir)
opts.ReadCache = isledb.ReadCacheOptions{
	URL:   "http://isledb-cache:8090",
	Token: token,
}
reader, err := db.OpenReader(ctx, opts)

feed, err := db.OpenChangeReader(ctx, isledb.ChangeReaderOptions{
	ReadCache: isledb.ReadCacheOptions{URL: "http://isledb-cache:8090", Token: token},
})

Run it as services

One binary, isledb, runs each service. Get it from the latest release, for Linux (amd64, arm64) and macOS (arm64), or run the container image:

docker run ghcr.io/ankur-anand/isledb:0.20.0 help

The image is tagged with each release's exact version, such as 0.20.0, and with its minor version, such as 0.20; there is no latest tag. With Go installed, you can also build it:

go install github.com/ankur-anand/isledb/cmd/isledb@latest
Service Does Listens on Run
isledb writer Takes writes over HTTP :8070 One per database
isledb reader Serves reads and scans over HTTP :8080 As many as you need
isledb maintenance Compacts and cleans up; no API admin only, 127.0.0.1:8061 One per database
isledb cache Optional regional read cache :8090 One or more per region

Each also serves health checks and metrics on its admin address. Run isledb help for the list, and isledb <service> -h for a service's flags.

Writer service

isledb writer holds the writer for one database and serves writes over HTTP, so services in any language can write:

isledb writer \
  --bucket-url 's3://my-bucket?region=us-east-1' \
  --prefix prod/accounts \
  --auth-token-file /etc/isledb/tokens
curl -X PUT -H "Authorization: Bearer $TOKEN" \
  --data '{"name":"a"}' 'http://isledb-writer:8070/v1/kv/user/42?ttl=24h'
# {"write_id":"1.1","revision":"2"}

A write is answered once it is durable in object storage; batches of puts and deletes go to POST /v1/kv and apply atomically. Run one writer per database. See Running isledb writer and the writer HTTP API.

Reader service

isledb reader serves reads over HTTP. Replicas share nothing, so you scale reads by adding them behind a load balancer:

isledb reader \
  --bucket-url 's3://my-bucket?region=us-east-1' \
  --prefix prod/accounts \
  --cache-dir /var/cache/isledb \
  --auth-token-file /etc/isledb/tokens
curl -H "Authorization: Bearer $TOKEN" 'http://isledb-reader:8080/v1/kv/user/42'
# {"name":"a"}
curl -H "Authorization: Bearer $TOKEN" 'http://isledb-reader:8080/v1/kv?prefix=user/'
# {"items":[{"key":"user/42","value":"{\"name\":\"a\"}"}]}

If the database has a change feed, readers also serve it at /v1/changes?from=…, so consumers in any language can follow it.

The paths match the writer's, so one gateway can route GET to readers and writes to the writer. A read is usually within a second of the writer (about two on Google Cloud Storage), and its Isledb-Revision header says which writes it includes. See Running isledb reader and the reader HTTP API.

Maintenance service

isledb maintenance compacts the database, keeps its metadata short and deletes what is no longer needed. It has no API. Run exactly one per database, alongside the writer:

isledb maintenance \
  --bucket-url 's3://my-bucket?region=us-east-1' \
  --prefix prod/accounts \
  --scratch-dir /var/lib/isledb/scratch

A second one for the same database takes over, and the first stops working and reports not ready. See Running isledb maintenance.

Shared read cache

isledb cache is an optional service that sits between readers and object storage, typically one per region. It serves immutable objects (SSTs, manifest pages, change batches) from chunks on its local disk, and relays CURRENT so readers learn of new commits through a long poll instead of polling storage.

Run it next to the readers:

isledb cache \
  --bucket-url 's3://my-bucket?region=us-east-1' \
  --db prod/accounts \
  --disk-dir /var/cache/isledb --disk-bytes 200GiB \
  --auth-token-file /etc/isledb/tokens

The cache only makes reads faster and cheaper; it never decides what readers see. Readers check what it returns against checksums, reject a copy of CURRENT that is too old, and read storage directly whenever the cache is slow, down, or wrong. See Running isledb cache and the read cache HTTP API.

An isledb reader uses it through read_cache.url; a Go program, through ReadCacheOptions.

Versions and support

IsleDB is before 1.0. A minor release, such as 0.19 to 0.20, may change the Go API, the services' flags and configuration, the HTTP API, or metrics; its release notes say what changed and how to move to it. A patch release, such as 0.19.0 to 0.19.1, only fixes problems.

Fixes go into the next release; earlier releases are not patched. Run a recent release, and name its exact version wherever you deploy it.

Documentation

Library:

Services:

Storage:

Acknowledgments

IsleDB uses the SSTable implementation from Pebble. Thanks to the CockroachDB team for building and open-sourcing it.

License

IsleDB is licensed under the Apache License 2.0.