google/sedpack
google/sedpack has 39 GitHub stars and 14 forks. GitHot measured 0 stars in 24 hours. Last checked 2026-11-15 21:46 UTC. Last push 2026-11-09.
Measured star growth
- in 24h
- 0
- in 7 days
- Collecting history
- in 15 days
- Collecting history
- in 1 month
- Collecting history
Net star changes over overlapping periods, not daily totals. Missing measurements are left blank. Stars measure attention, not software quality.
About
Sedpack - Scalable and efficient data packing
Machine LearningREADME.md
OriginalSedpack - Scalable and efficient data packing
Mainly refactored from the SCAAML project.
Available components
See the documentation website: https://google.github.io/sedpack/.
Install
Package install
pip install sedpack
TensorFlow 2.x is an optional dependency
required only when using the "tfrec" shard format or Dataset.as_tfdataset:
pip install "sedpack[tf]"
Development install
Development prerequisites:
python3-devandgccfor xxhash- Install Rust (required by
maturinto buildsedpack._sedpack_rs)
Steps:
Clone the repository and create a virtual environment:
git clone https://github.com/google/sedpack cd sedpack python3 -m venv .venv source .venv/bin/activateInstall pinned dependencies (optional):
python3 -m pip install --require-hashes -r requirements.txtInstall the package in editable mode:
python3 -m pip install --editable ".[dev,tf]"Or for a minimal installation without TensorFlow and development tools:
python3 -m pip install --editable .Rebuild the Rust extension after changes in
rust/and run unit tests:maturin develop --release python -m pytestNo tests should be skipped when TensorFlow is installed via
pip install -e ".[dev,tf]".
Update dependencies
With an active virtual environment, regenerate requirements.txt and commit the
updated lock file:
uv pip compile pyproject.toml --generate-hashes -o requirements.txt
Tutorial
Tutorials and documentation are available at https://google.github.io/sedpack/.
Code for the tutorials is available in the docs/tutorials directory. For a
"hello world" example, see the
MNIST tutorial.
AI Agents and LLM Skills
Sedpack includes built-in documentation for AI coding assistants and LLM agent frameworks:
Using
sedpackin your project (SKILL.md): The high-levelsedpack.ioAPI skill is bundled inside the package atsrc/sedpack/.agents/skills/sedpack/SKILL.md(and declared inskills.sh.jsonat the repository root).Option 1 — Link from your Python environment via Library Skills (recommended for
.venvprojects): Scans your project's installed Python packages and creates a relative directory symlink in.agents/skills/sedpackso the skill always matches your installedsedpackversion and updates automatically onpip install -U sedpack:# Interactive discovery and installation: uvx library-skills # Non-interactive install into .agents/skills/sedpack: uvx library-skills install --skill sedpack --yes # Also link into .claude/skills/sedpack, or copy files instead of symlinking: uvx library-skills install --skill sedpack --yes --claude uvx library-skills install --skill sedpack --yes --copy # Inspect status, check symlinks in CI, or repair after recreating .venv: uvx library-skills list uvx library-skills --check uvx library-skills --yeslibrary-skillsautomatically detects.venv(venv,virtualenv,uv,pdm, and in-projectpoetry/pipenv) and Conda (CONDA_PREFIX). For external virtual environments (such as Poetry's default cache, Pipenv, or Hatch), passUV_PROJECT_ENVIRONMENT:UV_PROJECT_ENVIRONMENT="$(poetry env info --path)" uvx library-skills install --skill sedpack --yesOption 2 — Install from GitHub via Agent Skills CLI (
npx skills): Installs the skill directly fromgoogle/sedpack(or a local checkout) into project-local or user-global (-g) directories across 40+ coding agents:# Interactive project-local install from GitHub: npx skills add google/sedpack # Non-interactive install (optionally targeting a specific agent): npx skills add google/sedpack --skill sedpack -y npx skills add google/sedpack --skill sedpack --agent gemini-cli -y npx skills add google/sedpack --skill sedpack --agent claude-code -y # Global install into ~/.agents/skills/sedpack: npx skills add google/sedpack --skill sedpack -g -y # List available skills in the repo, or check/update installed skills: npx skills add google/sedpack --list npx skills check npx skills updateOption 3 — Link with Gemini CLI: Link the bundled skill directory directly from your active Python environment (into
~/.gemini/skillsor project-local.gemini/skillswith--scope workspace):gemini skills link "$(python -c 'import importlib.resources as r; print(r.files("sedpack") / ".agents/skills/sedpack")')"Updating skills when
sedpackupdates (automatic vs. manual, local.venvvs. global~/):Project-local
.venvsymlink (uvx library-skills) — Automatic: Because.agents/skills/sedpackis a directory symlink intosite-packages/sedpack/.agents/skills/sedpack, upgradingsedpackin-place (pip install -U sedpack,uv sync --upgrade-package sedpack, orpoetry update sedpack) updatesSKILL.mdimmediately with no extra command, and keeps each project isolated to its own.venvversion.Recreating
.venvor upgrading Python minor versions (3.12->3.13): If.venvis recreated with the same Python minor version, the relative symlink resolves again as soon assedpackis reinstalled. If you upgrade Python minor versions (which changeslib/python3.X/site-packages), repair the symlink with:uvx library-skills --yesProject-local file copy (
uvx library-skills --copy) — Manual: Copied files are static snapshots and do not update onpip install -U sedpack. Re-sync after upgradingsedpack:rm -rf .agents/skills/sedpack uvx library-skills install --skill sedpack --yes --copyGitHub-installed skills (
npx skills, local or global-g) — Manual:npx skillstracks thegoogle/sedpackGitHub repository rather than your.venv(and-gshares a single global version across all projects). Check for and pull updates from GitHub with:npx skills check npx skills updateGemini CLI link (
gemini skills link): Updates automatically on in-placepip install -U sedpackin the linked environment (run/skills reloadin an active Gemini CLI session), or ongit pullfor an editable install (pip install -e .). Re-rungemini skills linkif the linked.venvis moved or upgraded to a new Python minor version.
Contributing to
sedpack(AGENTS.md): Repository architecture, build commands, test suites, and linter rules for coding agents working onsedpackitself are documented inAGENTS.mdat the repository root.
Disclaimer
This is not an official Google product.