nycmeshnet/meshdb
nycmeshnet/meshdb has 38 GitHub stars and 32 forks. GitHot measured 0 stars in 24 hours and 0 in 7 days. Last checked 2026-11-22 06:36 UTC. Last push 2026-11-20.
Progression mesurée des étoiles
- in 24h
- 0
- in 7 days
- 0
- in 15 days
- Historique en cours
- in 1 month
- Historique en cours
Variations nettes sur des périodes qui se chevauchent, pas des totaux quotidiens. Les mesures absentes restent vides. Les étoiles indiquent l’intérêt, pas la qualité.
À propos
A convenient, stable, and sane database for tracking Members and Nodes for use with robots and humans
DonnéesREADME.md
OriginalMeshDB
A Database for Tracking NYCMesh Member Installs
We use MeshDB to track information about Buildings, Members, Installs, Nodes,
Devices, and Links; Any info we need in order to get hardware on a rooftop near you lives in here.
This project aims to provide a convenient, stable, and sane interface for use with robots and humans. For more information, check the wiki
To start contributing, find a good first issue.
Setting up a Development Environment
Requirements
Build tooling
uv - installation instructions
External services
We have a handy docker-compose file that allows you to set all the external services up with a single command. See Setting up the required external services
[!NOTE] In order to use docker compose you must have docker or podman installed
Fork Repo
Fork this repo.
[!NOTE] If you cloned nycmeshnet/meshdb, you can change your origin by doing the following:
git remote remove origin git remote add origin https://github.com/<your_username>/meshdb git remote add upstream https://github.com/nycmeshnet/meshdb
Development environment options
Dev Container
If you would like to develop in a Dev Container
- Make sure you have VS Code installed.
- Install the Dev Containers extension:
ms-vscode-remote.remote-containers - Open the repo folder in the container.
- Continue on the VS Code terminal (where your project is opened) follow normal developer setup.
Locally on your machine
[!NOTE] If you are using NixOS, activate the dev environment with
nix-shell, and then proceed with the below instructions.
Set up the project using uv. The --all extras option
ensures that all development packages are also installed
uv sync --all extras
Set Environment Variables
Next, copy the out the .env.sample file to .env and load it into
your environment.
cp .env.sample .env
nano .env # Or your favorite text editor, fill in the blank variables
You're gonna need a DJANGO_SECRET_KEY, one is already provided in the
sample file, but if you want to generate your follow the directions
below.
Generating DJANGO_SECRET_KEY (optional)
There's already a secret key for you in the .env.sample, but here's how to generate a new one
uv run python -c 'from django.core.management.utils import get_random_secret_key; print(get_random_secret_key())'
[!IMPORTANT] Make sure you're running in Debug mode if you want to see detailed traces. Set DEBUG=True in your
.envfile.
Setting up the required external services
You will need the services identified in External Services which have been pre-defined in docker compose
docker compose up -d postgres pelias redis
You might have to run the migrations. This will set up the DB for you.
uv run python src/manage.py makemigrations
uv run python src/manage.py migrate
You'll probably want an admin account
uv run python src/manage.py createsuperuser
Loading a Database Export
Use these commands.
docker compose down -v #tear down the containers and delete the volumes
docker compose up -d postgres pelias redis #boot up
docker cp data.psql.bin meshdb-postgres-1:/ #copy
docker exec -it meshdb-postgres-1 pg_restore -d meshdb -U meshdb -j 6 --verbose scrambled2.psql.bin #restore
python src/manage.py runserver #run meshdb
[!WARNING] This is real member data. DO NOT share this database with anyone under any circumstances.
mkdir data/
cp <path_to_data_dump>/full_dump.sql data/
./scripts/import_datadump.sh
Advanced - MinIO for Local Dev
If you are going to use minio for local S3 bucket emulation (not required for most tasks), also
start the minio related containers with docker compose up -d minio createbuckets. To have your local DB instance
use Minio, you will also need to set S3_ENDPOINT="http://127.0.0.1:9000" in your .env file.
[!NOTE] You only need
createbucketsonce. It will initialize the bucket that MinIO talks to
Background Workers
If you want to do work with celery, you'll need to run a worker as well as a beat.
You can do this in two other terminals with these commands. DEBUG level is recommended
for the beat to see what beats are going to run
uv run celery -A meshdb worker -l INFO
uv run celery -A meshdb beat -s /tmp/celerybeat-schedule -l DEBUG
You can also run the celery containers in docker containers with the below command.
First copy your .env to .env.container and update the Redis and
Postgres URLs according to the comments in the file newly created
.env.container file.
docker compose up -d celery-worker celery-beat
[!NOTE] On Apple silicon macs, you may need to run the worker with the
--pool=soloto avoid segfaults. i.e.uv run celery -A meshdb worker -l INFO --pool=solo
Accessing the server
Now you can get crackin'. Start up the development server
uv run python src/manage.py runserver
Once the server is running should now be able to access the API:
curl http://127.0.0.1:8000/api/v1/ # Should echo "We're Meshin" to indicate 200 status
Running the Unit Tests
We use pytest with the pytest-django plugin
uv run pytest
Code Coverage
To view coverage add the --cov flag which will print per-file coverage
the the console.
uv run pytest --cov
Adding Tests
Unit Tests
Tests live in src/meshapi/tests/. It might make sense to add your test to an existing file within that directory, depending on what it's doing, or you can add a whole new file. See the django documentation for details on how to write a test, or check the directory for examples.
Integration Tests
[!WARNING] Don't run the integration tests against the production copy of MeshDB. These tests make fake objects and could pollute the real database.
Integration tests are defined in subdirectories within the integ-tests/ directory.
They are automatically run by the deployment pipeline to prevent issues
from reaching our production stages. To run them locally, first edit
the .env file to set the SITE_BASE_URL environment variable to the
Integration tests are defined in subdirectories within the
integ-tests/ directory.
To run them locally, first edit
the .env file to set the SITE_BASE_URL environment variable to the
http://localhost:8000to run them against a local copy of MeshDB (manage.py runserver)https://devdb.nycmesh.netto run them against the dev copy of MeshDB
You will also need a MeshDB API token which is appropriate for the stage you are running against,
set this via the MESHDB_API_TOKEN variable
Finally, run the integration tests with
uv run pytest integ-tests
Database Access
If you ever need to get into the database directly, it's easy to do so.
Get a shell on the postgres container:
docker exec -it meshdb-postgres-1 bash
Switch to postgres user.
su postgres
Run psql
psql -U meshdb
Cleaning up
When you're done, you can stop the server with Ctrl+C, and run
docker compose down to take down the containers.
[!NOTE] To spin things back up again later, just run:
docker-compose up -d postgres pelias redis uv run python src/manage.py runserver
Emulating the Production Environment
The production environment relies on Nginx and Gunicorn.
Clone the package with git and create the expected .env file (or
otherwise configure the environment variables specified in
.env.sample as appropriate to your environment).
git clone https://github.com/andybaumgar/nycmesh-database
cp .env.sample .env
nano .env # Fill in any missing values
[!NOTE] Check the above instructions if you need a
DJANGO_SECRET_KEY
Finally, start the application and database servers using docker compose
docker compose up
After a few minutes for image download & database setup, the development server
should be available at 127.0.0.1:8080:
# Should return "We're meshin'."
curl http://127.0.0.1:8080/api/v1
Authentication (Permissions, Groups, and Tokens)
We have very simple permission levels.
- Unauthenticated: A user using a route without authenticating
- Installer: Can view all fields, provision NNs, and edit installs
- Admin: Full access
We use Django Rest Framework's basic Auth Token implementation. To add a token,
you need a user, which can be created at /admin/auth/user/.
To determine what permissions the user has, add them to one of the pre-existing groups.
(Superuser and Staff are DRF-specific and should be restricted to people maintaining the instance)
For software apps, create a new users for each application with the format
PersonName-ApplicationName. Grant the minimum necessary permissions directly on the user object using the admin UI.
Auth tokens can be created at /admin/authtoken/tokenproxy/.
To use them, you can include them as an HTTP header like so:
curl -X GET http://127.0.0.1:8000/api/v1/members/ \
-H 'Authorization:Token <auth_token>'
Admin Map
In .env.sample, the admin map frontend assets are configured to pull
from the production endpoint (map data will still be pulled from your
local database). If you wish to pull the admin map assets from a local
endpoint, host the map locally with:
# In the map repo on the meshdb-admin branch
docker build -t meshdb-admin-map .
docker run -p "3000:3000" meshdb-admin-map
then edit the relevant .env variable to reflect the URL of the desired endpoint:
ADMIN_MAP_BASE_URL=http://localhost:3000
How to add environment variables
Follow this PR
Making Data Exports for New Devs
To make importable data exports for new devs, first obtain a local copy of the data you want to share (see Backups below). Then:
Run the scramble script to obfuscate PII:
uv run python src/manage.py scramble_members
Clear the data from the historical tables (so that we don't leak the data we just scrambled via a diff)
scripts/clear_history_tables.sh
[!WARNING] Be sure that you spot check the data to make sure the scramble process worked as expected.
Finally, create an importable datadump with:
scripts/create_importable_datadump.sh
The file will be written to data/full_dump.sql, share this with the new devs
Invoke Tasks
For convenience, this we use Invoke to wrap common tasks into one-line commands. For example:
uv run invoke format
Will automatically format the code, including sorting imports
You can also quickly peform all the relevant lint checks locally using
uv run invoke lint
See tasks.py for a complete list of the tasks available
Backups
[!NOTE] For local backups set environment variable
LOCALBACKUP_FILEto the path of the backup directory
The Proper Way
We have a Celery job that runs hourly in production to back up to an S3 bucket.
To restore from a backup in production:
Get a shell in the meshdb container
docker exec -it meshdb-meshdb-1 bashFind the backup you want to restore
uv run python manage.py listbackups Name Datetime default-09855fadfa7e-2024-03-29-015116.psql.bin 03/29/24 01:51:16 default-0c9b0a412baf-2024-03-31-170000.psql.bin 03/31/24 17:00:00 default-12db99e5ec1d-2024-03-31-142422.psql.bin 03/31/24 14:24:22 ... default-bd0acc253775-2024-03-31-163520.psql.bin 03/31/24 16:35:20In a separate terminal, drop the old database
echo 'drop database meshdb; create database meshdb;' | docker exec -i meshdb-postgres-1 psql -U meshdb -d postgresRestore the backup
uv run python manage.py dbrestore -i default-bd0acc253775-2024-03-31-163520.psql.bin --database default
The Quick 'n Dirty Way
Export:
docker exec -it meshdb-postgres-1 pg_dump -d meshdb -U meshdb >> Downloads/meshdb_dev.sql
Import:
cat ~/Downloads/meshdb_dev.sql | docker exec -i meshdb-postgres-1 psql -U meshdb
Explorer des projets similaires
Chargement…