Awesome Parquet

Open-source resources for using the Parquet format. This list only includes resources for the generic Parquet format. If you are looking for GeoParquet resources (tools, libraries, data providers), please check out geoparquet.org.

Contents

• Libraries
  • C GLib
  • C++
  • Dart
  • Go
  • Java
  • JavaScript
  • Julia
  • .NET
  • PHP
  • Python
  • R
  • Ruby
  • Rust
  • Swift
  • VBA
• Tools
  • Command-line
  • Desktop applications
  • Plugins and extensions
  • Terminal UI
  • Web
• Resources
  • Blogs
  • Documentation
  • Educative resources
  • Parquet engineering
  • Tests
• Related formats

Libraries

C GLib

• Arrow GLib - A wrapper library for Arrow C++.
• DuckDB - An in-process database library that supports reading and writing Parquet files.

C++

• Apache Arrow C++ - A library with support for reading and writing Parquet files.
• DuckDB C++ API - Internal DuckDB C++ API.
• libcudf - A GPU-accelerated DataFrame library for tabular data processing.

Dart

• DuckDB.Dart - DuckDB Dart bindings.

Go

• duckdb-go - DuckDB Go client.
• parquet - Official Go implementation of Apache Arrow.
• parsyl/parquet - A Go library for reading and writing Parquet files.

Java

• cudf - Java bindings for cudf, to be able to process large amounts of data on a GPU.
• duckdb-java - DuckDB Java/JDBC API.
• hardwood - A minimal dependency implementation of Apache Parquet.
• parquet-carpet - A Java library for serializing and deserializing Parquet files efficiently using Java records.
• parquet-java - A Java implementation of the Parquet format, owned by the Apache Software Foundation.

JavaScript

• duckdb-node-neo - DuckDB Node.js client.
• duckdb-wasm - WebAssembly version of DuckDB.
• hyparquet - A lightweight, dependency-free, pure JavaScript library for parsing Apache Parquet files.
• lakeql - Pure JavaScript duck-compatible SQL query engine for Parquet and Iceberg data in object storage.
• parquet-wasm - WebAssembly bindings to read and write the Apache Parquet format to and from Apache Arrow using the Rust parquet and arrow crates.

Julia

• DuckDB - Official DuckDB Julia package.
• Parquet.jl - Julia implementation of Parquet columnar file format reader.

.NET

• Parquet.Net - A fully managed Parquet library for .NET.
• ParquetSharp - A .NET wrapper over the C++ Parquet library that integrates with .NET Arrow.

PHP

• duckdb-php - DuckDB API for PHP.

Python

• duckdb-python - DuckDB Python client.
• fastparquet - A Python implementation of the Parquet columnar file format.
• pyarrow - A Python API for functionality provided by the Arrow C++ libraries, along with tools for Arrow integration and interoperability with Pandas, NumPy, and other software in the Python ecosystem.
• pylibcudf - A lightweight Cython interface to libcudf that provides near-zero overhead for GPU-accelerated data processing in Python.
• rugo - A lightweight, dependency-free Python library for Apache Parquet files.

R

• arrow - The arrow package provides an Arrow C++ backend to dplyr, and access to the Arrow C++ library through familiar base R and tidyverse functions, or R6 classes.
• duckdb-r - DuckDB R package.
• nanoparquet - A reader and writer for a common subset of Parquet files.

Ruby

• Red Parquet - The Ruby bindings of Apache Parquet, based on GObject Introspection.

Rust

• datafusion - An extensible query engine written in Rust that can read/write Parquet files using SQL or a DataFrame API.
• duckdb-rs - DuckDB Rust client.
• parquet - The official Native Rust implementation of Apache Parquet, part of the Apache Arrow project.
• Polars - A DataFrame interface on top of an OLAP Query Engine that supports reading and writing Parquet files, with bindings for Python.

Swift

• duckdb-swift - DuckDB Swift client.

VBA

• duckdb-vba - Excel/VBA bridge for DuckDB, enabling users to read, query, transform, and export Parquet files directly from Excel through a native DLL bridge.

Tools

Command-line

• Agentsor File Contracts - Python CLI for checking a Parquet file against an explicit TOML contract.
• DataFusion CLI - A single, dependency-free executable that can read and write Parquet files, with a SQL interface.
• DuckDB CLI - A single, dependency-free executable that can read and write Parquet files, with a SQL interface.
• nail - Command-line tool for analyzing, transforming, and exploring data files.
• ODBC to Parquet - A command-line tool to query an ODBC data source and write the result into a parquet file.
• OLake - Open-source replicator that writes Postgres, MySQL, MongoDB, Oracle, SQL Server and Kafka data as Parquet files or Apache Iceberg tables. Written in Go.
• parquet-cli - Java-based CLI tool for exploring parquet files.
• parquet-cli-standalone - A JAR file for the parquet-cli tool which can be run without any dependencies.
• parquet-grep - A CLI tool to search for strings in Parquet files.
• parquet-tools - Python-based CLI tool for exploring parquet files (part of Apache Arrow).
• Spark - A multi-language engine for executing data engineering, data science, and machine learning on single-node machines or clusters.

Desktop applications

• DBConvert Streams - A desktop SQL tool for querying Parquet, CSV, JSON, databases, and S3-compatible storage in one workspace.
• Munquet - A desktop tool to convert CSV files to Parquet.
• Pink Parquet - A free and open-source, user-friendly viewer for Parquet files for Windows.
• Tad - An application for viewing and analyzing tabular data sets.

Plugins and extensions

• KoldStore - PostgreSQL tiered storage that moves historical rows to Parquet while keeping the original table fully queryable and supporting updates and deletes.
• nf-parquet - A Nextflow plugin able to read and write parquet files.

Terminal UI

• Datanomy - A terminal-based tool for visualizing a Parquet file's metadata and structure.
• DataTUI - A keyboard-first terminal UI for exploring Parquet with tabs, sorting, filtering, SQL (Polars), and more.
• parqeye - Peek inside Parquet files right from your terminal.
• parquetlens - Parquet previewer with a csvlens-style TUI.
• Tabiew - A lightweight TUI application to view and query tabular data files, such as CSV, TSV, and parquet.

Web

• Datasette - A tool to explore datasets, with support for reading Parquet files.
• DataStudio - Explore and visualize data, entirely in your browser.
• DBConvert Streams Parquet Viewer - A browser-local viewer for inspecting Parquet metadata and rows, running read-only SQL, and exporting results.
• GeoParquet Viewer - A table and map viewer for Parquet files in the browser.
• Hyparquet demo - A drag-and-drop interface for loading parquet files into the browser, including data table, metadata and file layout views.
• Onyxia Data Explorer - A web-based tool to explore Parquet files in the browser.
• Parquet File Visualizer - Claude-code generated parquet metadata visualizer that runs in your browser.
• Parquet Viewer - View parquet files online.
• Parquet X-ray - See how a Parquet file is laid out, byte by byte. Only the footer and indexes are downloaded.
• parquet.to - Convert Parquet to and from CSV, JSON and Excel, with a viewer, SQL editor, chart builder and row editor.
• ParquetKit - View, query with SQL, and convert Parquet files entirely in the browser, powered by DuckDB-Wasm and hyparquet.
• Parquetly - View Parquet files in the browser, with schema, row group and column chunk statistics, SQL via DuckDB-Wasm, and export to CSV or JSON, all without uploading the file.
• ParquetView - Lightweight, mobile-friendly Parquet viewer and CSV/JSON converter; the file is parsed in the browser and never uploaded.
• Quak - A scalable data profiler for quickly scanning large tables.

Resources

Blogs

• icem7 - Un blog sur les outils de data science, avec des articles de fond sur Parquet.
• Hyparquet: The Quest for Instant Data - 6 optimization tricks to read Parquet files faster in the browser.
• Querying Parquet with Precision Using DuckDB - Describes how DuckDB optimizes queries to a Parquet file using projection & filter pushdown.
• Why Parquet Is the Go-To Format for Data Engineers - A graphical description of the Parquet format with optimization and best practices.
• Column Storage for the AI Era - A proposal by the creator of Parquet to better support AI workloads by adding encodings and metadata.
• I spent 8 hours learning Parquet. Here’s what I discovered - A graphical description of the Parquet format.

Documentation

• Parquet - The specification for Apache Parquet and Apache Thrift definitions to read and write Parquet metadata.
• Apache Parquet Documentation - The official documentation for Apache Parquet.

Educative resources

• ssphub - Un atelier de l'Insee illustrant l'utilisation des données du recensement 🇫🇷 diffusées au format Parquet.

Parquet engineering

• Best Practices for Distributing GeoParquet - Recommandations about compression, row group size, ordering, and partitioning, not limited to GeoParquet.
• Embedding User-Defined Indexes in Apache Parquet Files - Explains how user-defined (custom) indexes in Parquet help speeding up query processing.
• Handling Parquet Files - Recommendations about the row group size and the Parquet file sizes.
• Les filtres de Bloom dans Parquet - Un article de fond sur les filtres de Bloom dans Parquet, utiles pour indexer des colonnes non triées, à forte cardinalité.
• Paging Through a Parquet File in DuckDB: file_row_number or OFFSET? - In-depth investigation about paging through a Parquet file, with recommendations about row group size.
• Tips for Writing Parquet Files - Tips for choosing the right parameters when writing Parquet files, such as the row group size and the number of row groups per file.

Tests

• parquet-testing - Testing Data and Utilities for Apache Parquet.

Related formats

• F3 - A data file format that is designed with efficiency, interoperability, and extensibility in mind.
• GeoParquet - Specification for storing geospatial vector data (point, line, polygon) in Parquet.
• Iceberg - A high-performance format for huge analytic tables that supports Parquet as one of its storage formats.
• Lance - Modern columnar data format for ML and LLMs.
• Nimble - File format for storage of large columnar datasets.
• ORC - Self-describing type-aware columnar file format designed for Hadoop workloads.
• Vortex - A columnar file format designed for high-performance data processing.

Contributing

Contributions welcome! Read the contribution guidelines first.