Skip to content
Catégories de projets

Données · GitHub

Comparez bases de données, analyses, SQL et projets d’ingénierie des données.

Projets GitHub utiles

6,818 projets

data-engineering

sophie-nguyenthuthuy

56étoiles

100+ data engineering projects from scratch — streaming, CDC, table formats, query engines, consensus, governance. 2,500+ tests, mypy strict.

125étoiles
mcp-reticle preview

Reticle intercepts, visualizes, and profiles JSON-RPC traffic between your LLM and MCP servers in real-time, with zero latency overhead. Stop debugging blind. Start seeing everything.

30étoiles
real-time_crypto_data_pipeline_using_kafka preview

I am using confluent Kafka cluster to produce and consume scraped data. In this project, I've created a real-time data pipeline that utilizes Kafka to scrape, process, and load data onto S3 in JSON format. With a producer-consumer architecture, I ensure that the data is in the right format for loading onto S3 by performing minor transformations

DonnéesPythonCC0-1.0GitHub

defradb

sourcenetwork

898étoiles
defradb preview

DefraDB is a Peer-to-Peer Edge-First Database. It's the core data storage system for the Source Ecosystem.

mysql

sous-chefs

344étoiles

Development repository for the mysql cookbook

DonnéesRubyApache-2.0GitHub
223étoiles

C#: EPSON Robot Remote TCP/IP Control,Motion Control(GoogolTech/LeadShine),Advantech(PCI1752/PCI1754),Keyence Laser,RS232C,RS485,TCP/IP Server/Client(Sync/Async),OPC Client/Server,CAN, EtherCAT,XML,Excel,MYSQL/ACCESS Database,Common functions,Delay(time), etc.C#开发的工业自动化控制类,大部分已经写完且验证过: 一、EPSON Robot Remote TCP/IP Control: 用于对EPSON机械手进行远程以太网控制,建立以太网通讯后发送命令给爱普生机械手控制器,然后依照返回的命令处理相应结果; 二、Motion Control(GoogolTech/LeadShine):【二次封装】固高GTS系列运动控制卡运动卡控制类:初始化运动卡和更新IO,控制固高运动控制器轴运动;这是对固高和雷赛运动控制卡的DLL文件中的函数进行第二次封装,使在实际编程的时候更加方便、快捷; 三、Advantech: PCI1752:研华数据采集卡PCI1752更新输出及输出状态回读; PCI1754:研华数据采集卡PCI1754更新输入信号; 四、Keyence Laser:基恩士激光读取数据; 五、RS232C:RS232C串口通讯,包括界面类; 六、RS485:RS485串口通讯,还没有完成; 七、TCP/IP Server/Client(Sync/Async):以太网通讯客户端和服务器端,支持同步和异步通讯; 八、OPC Client/Server:还没有完成; 九、CAN, EtherCAT:还没有完成; 十、XML:XML文件操作,包括写和读取的相关实用函数; 十一、Excel:Excel文件的相关操作,包括快速写大量数据到EXCEL文件;其中处理条件格式的代码还在调试中; 十二、MYSQL/ACCESS Database:数据库操作; 十三、Common functions:操作系统和文件相关操作、获取TcpClient连接的本地/对方IP地址和端口、通用TCP/IP函数、跨线程安全的委托、进制转换、通用RS232C函数; 十四、Delay(time):延时计时器,等待指定时间,如果计算机不支持高性能计数器,则只支持整数秒的时间 十五、ListView:针对ListView控件的各项增加、删除、查找操作;

264étoiles

MSE5540/6640 Materials Informatics course at the University of Utah. Learn how data science tools are revolutionizing materials science!

DonnéesJupyter NotebookMITGitHub

minarrow

SpaceCell

103étoiles

Fast, minimal, and ergonomic zero-copy Apache Arrow tabular data implementation in Rust, with PyO3, Python bindings, NdArray, XArray, shared-memory and low-latency SIMD for data-engineering and columnar workloads. Bridge to Polars and Arrow-rs over FFI effortlessly.

DonnéesRustApache-2.0GitHub

Wired

SpaceCell

44étoiles
Wired preview

Fast data transport : Apache Arrow, Protobuf and Message Pack data on the one stream. Measured faster than Arrow Flight on all open benchmarks. HTTP, TCP, QUIC, Websocket, Webtransport, Stdio, UDS, all interchangeable, with 64-byte SIMD compatibilty maintained over the network and to/from disk via custom Arrow Mmap and Parquet Readers and Writers

WorkloadTools

spaghettidba

265étoiles
WorkloadTools preview

A collection of tools to collect, analyze and replay SQL Server workloads, on premises and in the cloud

jackcess

spannm

30étoiles

A pure Java library for reading / writing Microsoft Access databases

  • Pure Java: 100% Java implementation. No native dependencies, no DLLs, no MS Access installation required.
  • Wide Version Support: Supports Access versions 2000, 2003, 2007, 2010, 2013, 2016, and 2019.
DonnéesJavaApache-2.0GitHub
107étoiles

Open-source pure Java JDBC driver for Microsoft Access databases (*.accdb, *.mdb). Enables client programs (e.g. OpenOffice, LibreOffice, Matlab, SQuirreL) to read and write Access databases.

  • Pure Java Power: Zero native dependencies. Runs anywhere Java 11+ is supported.
  • Drop-in Replacement: Fully compatible with previous UCanAccess versions.
DonnéesJavaApache-2.0GitHub

sparkdq

sparkdq-community

77étoiles

A lightweight, declarative PySpark framework for data quality validation — check columns, rows, and entire datasets directly in your Spark pipelines

DonnéesPythonApache-2.0GitHub

sparklabx

sparklabx

32étoiles
sparklabx preview

Self-hosted multi-user Apache Spark notebooks — PySpark + Scala, per-user kernel pods, MinIO IAM storage, OAuth. One Helm install. Apache-2.0.

  • Spark notebooks in the browser — PySpark, Scala (Almond), with Monaco editor, Markdown cells, kernel restart, and a Libraries dialog that wraps spark.jars.packages / import $ivy so users add Maven coordinates without editing code.
  • Embedded Spark UI + DataFlint — open the kernel's Spark Web UI (DAGs, stages, SQL plans, metrics) in-app, no port-forward. DataFlint (a Spark performance plugin) is on by default for PySpark and Scala, adding a performance tab that flags slow stages, skew, and oversized scans.
DonnéesTypeScriptApache-2.0GitHub

gsocanalyzer

Sparsh1212

76étoiles
gsocanalyzer preview

A blazingly fast tool to analyze all the selected organizations in Google Summer of Code in the form of graphical analytics.

DonnéesJavaScriptMITGitHub

gorm-zero

SpectatorNan

291étoiles

go zero gorm extension / Integrated processing of db model and Redis cache

myzql

speed2exe

83étoiles

MySQL and MariaDB driver in native Zig

  • Native Zig code, no external dependencies
  • TCP and Unix socket connections