statmike/gcp-mlops
statmike/gcp-mlops has 719 GitHub stars and 315 forks. GitHot measured 0 stars in 24 hours, 0 in 7 days, and -1 in 15 days. Last checked 2026-11-22 06:32 UTC. Last push 2026-10-21.
Crecimiento de estrellas medido
- in 24h
- 0
- in 7 days
- 0
- in 15 days
- -1
- in 1 month
- Recopilando historial
Cambios netos en períodos que se solapan, no totales diarios. Los datos no disponibles se dejan en blanco. Las estrellas indican interés, no calidad.
Acerca de
Google Cloud Platform Vertex AI end-to-end workflows for machine learning operations
Aprendizaje automáticoREADME.md
Original|
View on GitHub |
|
Share On:
|
|
Connect With Author On:
|
|
|
ML, MLOps, Data + AI, and Agents on Google Cloud
Data Science Workflows for GCP
Google Cloud's AI platform has gone through several names over the years — Cloud ML Engine, then AI Platform, then Vertex AI, then Gemini Enterprise Agent Platform, and unified under the Gemini Enterprise umbrella (which brings together the developer Agent Platform, the employee-facing Gemini Enterprise app formerly known as Agentspace, and Gemini Enterprise for Customer Experience). Each name change tracked a real expansion in scope: from custom model training, to a unified ML platform with AutoML and managed notebooks, to the current platform where model building, agent development, evaluation, and enterprise governance live side by side.
This repository (statmike/gcp-mlops, formerly vertex-ai-mlops) has grown along that same path. It started as a collection of end-to-end training and serving workflows on Vertex AI and has expanded across the Google Cloud data and AI stack:
- In-database ML, GenAI, and causal inference in BigQuery (
CREATE MODEL,ML.*, andAI.*functions) — now also packaged as the standalone repositorystatmike/bigquery-data-scienceand on PyPI (bigquery-data-science) - AI agents and multi-agent architectures built with ADK, MCP, MCP Toolbox for Databases, and A2A, deployed on Agent Engine
- Cross-service MLOps infrastructure — online and batch serving, five feature store architectures, Vertex AI Pipelines (KFP), Managed Airflow (formerly Cloud Composer), Dataflow (
RunInference), and Dataproc Serverless Spark - Database-backed AI across Spanner, AlloyDB, Cloud SQL, Bigtable, Memorystore/Valkey, and Firestore for feature serving, vector search, and SQL-based inference
- Packaged Agent Skills, CLI & Stdio MCP Server (
SKILL.mdwith 4-Lens progressive disclosure) distilled from the notebooks and reference guides so AI coding agents can use the verified patterns and gotchas directly (uvx bigquery-data-science installornpx skills add statmike/bigquery-data-science)
Across this repository and its companion statmike/bigquery-data-science repository you will find 580+ interactive notebooks, 85+ standalone SQL scripts, and deep reference guides. Every workflow is written to be a clear, runnable starting point — heavy compute happens on Google Cloud services rather than your local machine, and related notebooks share datasets so you can compare approaches head to head.
How the Repository Is Organized
%%{init: {'flowchart': {'wrappingWidth': 650, 'nodeSpacing': 25, 'rankSpacing': 30}}}%%
flowchart TD
Start["statmike/gcp-mlops<br/>580+ Notebooks · 85+ SQL Scripts · Packaged Agent Skills"]
P1["1. Data + AI & In-Database ML (data+ai/ + statmike/bigquery-data-science)<br/>BigQuery ML (CREATE MODEL · ML.* Functions · Causal & MLOps Pipelines)<br/>BigQuery AI (AI.* & Remote ML.* Functions · RAG & Hybrid Search) · Dataflow · Dataproc · Iceberg"]
P2["2. Agent Skills, CLI & MCP (agent-skills/ → PyPI: bigquery-data-science)<br/>3 4-Lens Skills: bigquery-data-science · bigquery-ml · bigquery-ai"]
P3["3. Applied ML, AI Agents & Solutions (Applied ML/ — 29 notebooks · 60+ guides)<br/>9 AI Agent Projects (Gemini Enterprise Tour · 1,800-Cell NL2SQL MCP Benchmark)<br/>3 Solution Prototypes (Doc Fraud · 9-Way Taxonomy · Forecasting) · Vizier Optimization"]
P4["4. MLOps Infrastructure (MLOps/ — 100 notebooks)<br/>5 Feature Stores (BigQuery · Spanner · Bigtable · Valkey · Vertex AI)<br/>6 Model Serving Patterns (Online · Batch · SQL · Containers · Triton · vLLM) · KFP"]
P5["5. Applied GenAI, Frameworks & Forecasting (117 notebooks)<br/>11-Database Vector Retrieval Comparison (Applied GenAI/ — 58 notebooks)<br/>Framework Workflows (PyTorch · R · AutoML · Keras · CatBoost · Flax) · Forecasting (20)"]
Start --> P1 --> P2 --> P3 --> P4 --> P5
classDef root fill:#1a73e8,stroke:#0b57d0,stroke-width:2px,color:#ffffff
classDef primary fill:#e8f0fe,stroke:#1a73e8,stroke-width:1.5px,color:#174ea6
classDef secondary fill:#f1f5f9,stroke:#475569,stroke-width:1.5px,color:#0f172a
class Start root
class P1,P3,P4 primary
class P2,P5 secondary
Quick Navigation by Question
| If you want to... | Go here |
|---|---|
| Train, evaluate, explain, or schedule models directly in BigQuery SQL | data+ai/bq-ml/ · statmike/bigquery-data-science/bigquery-ml |
| Call Gemini, generate embeddings, run hybrid search, or zero-shot forecast in SQL | data+ai/bq-ai-functions/ · statmike/bigquery-data-science/bigquery-ai |
| Give your AI coding agent (Claude Code, Antigravity, Gemini CLI, Codex) verified BigQuery skills, CLI & MCP | agent-skills/ · statmike/bigquery-data-science · PyPI |
| Build, govern, benchmark, and deploy agents with ADK, MCP, A2A, and Agent Engine | Applied ML/AI Agents/ |
| Compare 5 feature store architectures (BigQuery, Spanner, Bigtable, Valkey, Vertex AI) | MLOps/Feature Store/ |
| Compare serving patterns (Online, Batch, SQL, Cloud Run, GKE, Triton, vLLM) | MLOps/Serving/ |
| Compare 11 Google Cloud databases for vector search and RAG retrieval | Applied GenAI/Retrieval/ |
| Compare ML training & batch inference across 7 GCP compute services on the same dataset | data+ai/overview/ml-training/ |
| See end-to-end multi-service applications (Document Fraud, 9-Way Taxonomy, Forecasting) | Applied ML/Solution Prototypes/ |
| Train and serve with PyTorch, Keras, CatBoost, Flax, AutoML, or R across GCP runtimes | Framework Workflows/ |
Repository Pillars
Each section below has its own detailed readme.md with vertical architecture diagrams and full notebook/SQL inventories:
1. Data + AI (data+ai/) & statmike/bigquery-data-science
Machine learning, generative AI, causal inference, and data engineering across Google Cloud's data services — bringing model training, foundation-model inference, and orchestration directly to where the data lives. The BigQuery ML and BigQuery AI curricula live in their dedicated repository (statmike/bigquery-data-science) and Python package (bigquery-data-science on PyPI) with local signpost hubs here in data+ai/:
- BigQuery ML (
bq-ml/) (GitHub Hub · Overview · Reference) — Per-model lifecycle deep dives (CREATE MODEL), model-freeML.*function suites, end-to-end SQL workflows (including Causal Inference & Econometrics: Propensity Score Matching/IPTW, Double ML Price Elasticity, Uplift/CATE, Difference-in-Differences with Callaway–Sant'Anna, Synthetic Control,AI.CAUSAL_EFFECT, and Survival Analysis, plus Point-in-Time Feature Store leakage quantification and Hierarchical Forecasting), and production orchestration pipelines (SQL Scripting, Scheduled Queries, Dataform, Cloud Workflows, dbt, Vertex KFP, and Managed Airflow). - BigQuery AI (
bq-ai-functions/) (GitHub Hub · Overview · Reference) — Function-by-function guides (AI.GENERATE*,AI.CLASSIFY,AI.IF,AI.SCORE,AI.AGG,AI.EMBED,VECTOR_SEARCH,AI.SEARCH,AI.FORECAST,AI.DETECT_ANOMALIES,AI.PREDICTTabFM,AI.CAUSAL_EFFECT,AI.KEY_DRIVERS, and pre-trained Cloud AI services) plus composed multi-function workflows (RAG, Hybrid Catalog Search with Rank Fusion, Web-Grounded Enrichment, Document & Time Series Intelligence, and Image Deduplication). - Cross-Service Data + AI — 7-Way ML Training & Batch Inference Comparison, Dataflow
RunInference& GPU Benchmark, Dataproc Serverless Spark, Managed Airflow, BigQuery Managed Iceberg, BigQuery Continuous Queries, Tabular Data Read Benchmarks, and SQL inference in AlloyDB & Spanner. - $\rightarrow$ Full catalog & Data + AI decision flow:
data+ai/readme.md
2. Agent Skills, CLI & Stdio MCP (agent-skills/) — bigquery-data-science on GitHub & PyPI
Packaged, cross-tool Agent Skills distilled from live-tested BigQuery ML and BigQuery AI notebooks into 4 progressive lenses (SKILL.md routers $\to$ reference/*.md decision guides $\to$ <25 KB syntax/*.md cards $\to$ narrative/*.md textbook walkthroughs), shipped as a zero-dependency Python package (bigquery-data-science on PyPI) and standalone repository (statmike/bigquery-data-science):
bigquery-data-science(umbrella triage router +ML.*vsAI.*namespace matrix) ·bigquery-ml·bigquery-ai.- Install into any repository or global agent config with
uvx bigquery-data-science installornpx skills add statmike/bigquery-data-science, query from the terminal (uvx bigquery-data-science triage | lookup | search), or attach as a stdio MCP server (uvx bigquery-data-science mcp). - $\rightarrow$ Full guide & installation options:
agent-skills/README.md·statmike/bigquery-data-science· PyPI
3. Applied ML, AI Agents & Solution Prototypes (Applied ML/) — 29 notebooks & 60+ guides
Where MLOps/ focuses on serving and infrastructure, Applied ML/ focuses on what you build with models — autonomous multi-agent systems, multi-service business applications, and black-box optimization:
- AI Agents (
Applied ML/AI Agents/) (9 projects built with ADK, MCP, MCP Toolbox for Databases, and A2A) — Led by Agent Building: A Tour of the Gemini Enterprise Agent Platform (exercising Agent Runtime, Memory Bank, Identity, Registry, Model Armor, and a mixed Gemini 3 + Claude fleet) and the Data MCP Sandbox (a controlled 1,800-cell / 12-arm empirical NL2SQL benchmark measuring the exact accuracy and token-cost impact of Dataplex/Looker governance and MCP tool schemas), plus BigQuery Context Discovery, Data Onboarding, RFI Response Agent, Image-to-Graph Agent, Concept-BQ, Conversational Analytics API, and Travel Planner (A2A). - Solution Prototypes (
Applied ML/Solution Prototypes/) (13 notebooks) — Document Processing & Fraud Detection (8 notebooks), Product Hierarchy Classification (9 approaches across flat, top-down cascade, and Keras Multi-Task Learning, both Python-first and SQL-first), and Conversational Time Series Forecasting (3 notebooks). - Optimization with Vertex AI Vizier, Hierarchical Forecasting, and Anomaly Detection.
- $\rightarrow$ Full catalog & Applied ML architecture map:
Applied ML/readme.md
4. MLOps Infrastructure (MLOps/) — 100 notebooks
Everything between training a model and operating a fleet of models in production:
- Feature Stores (
MLOps/Feature Store/) (45 notebooks + 5 Architecture Slide Decks) — Five architectures built on the same BigQuery offline data across the latency/cost spectrum: BigQuery (3), Spanner (12), Bigtable (15), Valkey / Memorystore (9), and Vertex AI Feature Store (6). - Model Serving (
MLOps/Serving/) (32 notebooks) — Online Endpoints (7), Batch Inference (4), SQL Inference (5), Cloud Run / GKE / Cloud Functions (4), NVIDIA Triton (5), and vLLM (6). - Vertex AI Pipelines / KFP (13 notebooks), Model Evaluation (3), Model Monitoring (2), Experiment Tracking (1), and Development Environments (2).
- $\rightarrow$ Full catalog & MLOps stack diagram:
MLOps/readme.md
5. Applied GenAI (Applied GenAI/) — 58 active notebooks
Grounding generative models with your own data across the full RAG and multimodal pipeline:
- Vector Search & Retrieval Across 11 Databases (
Retrieval/) (14 notebooks comparing BigQuery, Vertex Vector Search, Feature Store, Spanner, AlloyDB, Cloud SQL, Memorystore/Valkey, Firestore, Bigtable, Local, and ChromaDB), Generate (13), Embeddings (9), Solutions & Hybrid Search (Dense + BM25 + RRF) (7), RAG Engine (1), Structure-Aware Chunking (3), Evaluation (2), Ranking (1), and Validate (1). - $\rightarrow$ Full catalog & GenAI pipeline diagram:
Applied GenAI/readme.md
6. Framework Workflows, Applied Forecasting & Foundational Series
- Framework Workflows (
Framework Workflows/) (39 notebooks) — Shared BigQuery dataset, Feature Store, and Model Registry exercised across PyTorch (20 notebooks across 19 serving targets), R on Google Cloud (6 notebooks across 4 runtimes + Iceberg/arrow), Vertex AI AutoML (6), Keras 3 (3), CatBoost (3), and Flax / JAX (1). - Applied Forecasting (
Applied Forecasting/) (20 notebooks) — Progressive time-series curriculum on NYC Citi Bike data:ARIMA_PLUS,ARIMA_PLUS_XREG, regression lags, AutoML Forecasting, Seq2Seq+, TFT, TiDE, Prophet (training, containers, online CPR), TimesFM & TimesFM 2.5, and KFP forecasting tournaments. - Foundational Numbered Series & Utilities:
00 - Setup·01 - Data Sources·02 - Vertex AI AutoML(6) ·03 - BigQuery ML (BQML)(26) ·04 - scikit-learn(12) ·05 - TensorFlow(28) ·Working With/Document AI(8) ·Tips(13) ·architectures/(28) ·readme-legacy.md(complete historical index).
How to Use These Notebooks
Every notebook in this repository is designed around three practical rules:
- Self-contained and portable — Narrative explanations, architecture diagrams, setup cells, and cleanup cells live in a single
.ipynbfile (with one-click links in the header to open in Colab, Colab Enterprise, BigQuery Studio, or Vertex AI Workbench). - Cloud-orchestrated compute — Notebooks orchestrate BigQuery, Vertex AI, Dataflow, Dataproc, and Cloud Run rather than crunching large datasets locally, so almost every notebook runs comfortably on the smallest standard runtime.
- Application Default Credentials (ADC) only — No service account JSON keys are ever created or stored; local workflows use
gcloud auth application-default loginand cloud runtimes use attached service accounts.
Contributing & License
- Contributing: Issues, discussions, and pull requests are welcome — see CONTRIBUTING.md, CODE_OF_CONDUCT.md, and SECURITY.md.
- License: Apache 2.0 — see LICENSE.
This repository is maintained by Mike Henderson (@statmike). It is built on Google Cloud products for learning and reference and is not an officially supported Google product.
Explorar proyectos relacionados
Cargando…