项目
在 GitHub 查看
tradewelltech/beavers
tradewelltech/beavers has 41 GitHub stars and 2 forks. GitHot measured 0 stars in 24 hours and 0 in 7 days. Last checked 2026-11-22 00:43 UTC. Last push 2026-07-28.
可以做什么
- Works in real time (eg: reading from Kafka) and replay mode (eg: reading from Parquet files).
- Optimized for analytics, using micro-batches (instead of processing records one by one).
- Similar to incremental, it updates nodes in a dag incrementally.
- Stream: ephemeral micro-batches of events (cleared after every cycle).
实测星标增长
- 24小时
- 0
- 7天
- 0
- 15天
- 正在收集历史
- 1个月
- 正在收集历史
各周期内的星标净变化,周期相互重叠,并非每日新增。缺失的测量保持空白。星标代表关注度,不代表软件质量。
关于
Python stream processing for analytics
数据README.md
原文Beavers
Documentation / Installation / Repository / PyPI
Beavers is a python library for stream processing, optimized for analytics.
It is used at Tradewell Technologies, to calculate analytics and serve model predictions, for both realtime and batch jobs.
Key Features
- Works in real time (eg: reading from Kafka) and replay mode (eg: reading from Parquet files).
- Optimized for analytics, using micro-batches (instead of processing records one by one).
- Similar to incremental, it updates nodes in a dag incrementally.
- Taking inspiration from kafka streams, there are two types of nodes in the dag:
- Stream: ephemeral micro-batches of events (cleared after every cycle).
- State: durable state derived from streams.
- Clear separation between the business logic and the IO. So the same dag can be used in real time mode, replay mode or can be easily tested.
- Functional interface: no inheritance or decorator required.
- Support for complicated joins, not just "linear" data flow.
Limitations
- No concurrency support. To speed up calculation use libraries like pandas, pyarrow or polars.
- No async code. To speed up IO use kafka driver native thread or parquet IO thread pool.
- No support for persistent state. Instead of saving state, replay historic data from kafka to prime stateful nodes.
Talks
探索相关项目
加载中…