Skip to content
Projects

gettyimages/docker-spark has 667 GitHub stars and 358 forks. GitHot measured 0 stars in 24 hours. Last checked 2026-11-15 22:35 UTC. Last push 2021-12-30.

How these counts are measured

Measured star growth

in 24h
0
in 7 days
Update delayed
in 15 days
Update delayed
in 1 month
Update delayed

Net star changes over overlapping periods, not daily totals. Missing measurements are left blank. Stars measure attention, not software quality.

About

Docker build for Apache Spark

Developer Tools

README.md

Original

spark

A debian:stretch based Spark container. Use it in a standalone cluster with the accompanying docker-compose.yml, or as a base for more complex recipes.

docker example

To run SparkPi, run the image with Docker:

docker run --rm -it -p 4040:4040 gettyimages/spark bin/run-example SparkPi 10

To start spark-shell with your AWS credentials:

docker run --rm -it -e "AWS_ACCESS_KEY_ID=YOURKEY" -e "AWS_SECRET_ACCESS_KEY=YOURSECRET" -p 4040:4040 gettyimages/spark bin/spark-shell

To do a thing with Pyspark

echo -e "import pyspark\n\nprint(pyspark.SparkContext().parallelize(range(0, 10)).count())" > count.py
docker run --rm -it -p 4040:4040 -v $(pwd)/count.py:/count.py gettyimages/spark bin/spark-submit /count.py

docker-compose example

To create a simplistic standalone cluster with docker-compose:

docker-compose up

The SparkUI will be running at http://${YOUR_DOCKER_HOST}:8080 with one worker listed. To run pyspark, exec into a container:

docker exec -it docker-spark_master_1 /bin/bash
bin/pyspark

To run SparkPi, exec into a container:

docker exec -it docker-spark_master_1 /bin/bash
bin/run-example SparkPi 10

license

MIT