diff --git a/blog/2026-07-02-ivorysql-htap/index.md b/blog/2026-07-02-ivorysql-htap/index.md new file mode 100644 index 0000000..1c979ff --- /dev/null +++ b/blog/2026-07-02-ivorysql-htap/index.md @@ -0,0 +1,116 @@ +--- +slug: ivorysql-htap +title: "IvorySQL HTAP Real-Time Lakehouse Access Engine" +authors: [陶郑] +category: IvorySQL +image: img/blog/covers/ivorysql-htap.png +tags: [IvorySQL, HTAP, DuckDB, PostgreSQL, Lakehouse, Mooncake] +--- + +> This article is based on Tao Zheng's (IvorySQL Core Contributor) presentation at HOW 2026. +> Video replay: https://www.youtube.com/watch?v=n2GZRLCiabg + +## 1. The Evolution of Data Architecture + +### 1.1 Limitations of Traditional Architecture + +Business data must flow through multiple systems before analysis. Two main architectures dominate: + +- **Linear Architecture**: data enters transactional databases → ETL → data lake → data warehouse → AI query. Long pipeline, many processing steps. +- **Variant Architecture**: data is written simultaneously to transactional and analytical databases, but AI queries still cross multiple systems. + +Both face common challenges: complex processing pipelines creating minutes-to-hours latency; data fragmentation between queries; high multi-system operational costs. + +### 1.2 Three Generations of Evolution + +- **Gen 1 (2000s — Data Warehouse)**: Designed for specific business needs, but costly to restructure when new analysis requirements emerge. +- **Gen 2 (2010s — Data Lake)**: Reduced storage costs by storing raw data in cheap lakes, but increased pipeline complexity. +- **Gen 3 (circa 2020 — Lakehouse)**: Merged warehouse and lake benefits, but fundamentally still a two-system combination. + +### 1.3 Native HTAP: The Next-Generation Architecture + +The latest trend is moving from "lakehouse" to "native HTAP"—storing transactional and analytical data in a single database, eliminating ETL entirely. + +| Dimension | Lakehouse | Native HTAP | +|-----------|----------|-------------| +| Storage base | Object storage | Unified high-performance storage engine | +| Data freshness | Minutes/hours | Seconds/milliseconds | +| Core capability | Lake + warehouse features | Database with warehouse capacity | + +A single database, single API for AI, no cross-database queries, lower operational costs, real-time data. + +## 2. Three Pain Points of Traditional Architecture in the AI Era + +- **Pain Point 1: Unpredictable query patterns** — Agent-generated queries are random; traditional architectures can't efficiently handle both point queries and aggregations simultaneously. +- **Pain Point 2: Data freshness sensitivity** — ETL latency means data may change between queries, unacceptable in risk control, recommendation, and finance. +- **Pain Point 3: Cross-system latency stacking** — When a single task requires both point queries and aggregations, cross-system latency compounds. + +## 3. Native HTAP: Industry Trends & IvorySQL's Positioning + +### 3.1 Industry Signals: Databricks' Two Key Acquisitions + +- **May 2025 — Acquired Neon**: Serverless PostgreSQL provider, signaling OLAP giants entering OLTP. +- **October 2025 — Acquired Mooncake Labs**: Core project pg_mooncake embeds DuckDB's columnar engine into PostgreSQL, enabling HTAP without ETL. IvorySQL's solution is built on pg_mooncake. + +### 3.2 Two Gaps Left by pg_mooncake + +- **Gap 1: Open-source maintenance void** — Updates slowed significantly after acquisition. The community needs a vendor-independent implementation. +- **Gap 2: Oracle migration capability gap** — pg_mooncake never considered Oracle compatibility, yet many Chinese enterprises still run core data on Oracle. + +### 3.3 IvorySQL's Position + +IvorySQL is an open-source PostgreSQL fork, community-maintained, fully PostgreSQL-compatible, with Oracle syntax, data type, and function support. Its HTAP engine: +- Fills Gap 1: Independently maintained pg_mooncake evolution, community-driven roadmap +- Fills Gap 2: Full HTAP capabilities in Oracle migration scenarios +- Real-time lakehouse access engine: one dataset, two query paths, real-time availability + +## 4. Technical Architecture + +### 4.1 Row vs. Column Storage + +PostgreSQL uses row storage — ideal for transactions. Column storage organizes data by column — ideal for large-scale aggregation. HTAP combines both, automatically selecting the optimal execution path based on query characteristics. + +### 4.2 Core Architecture + +**Smart routing**: Embedded via PostgreSQL hooks (non-invasive), the Planner identifies query patterns and routes to row store (OLTP) or column store (OLAP). + +**Dual storage engines**: Row store uses PG Heap files (OLTP); column store uses DuckDB + Parquet format (OLAP). + +**In-memory sharing & auto-sync**: The same data is accessible through both row and column paths. Columnar data persists in Parquet format; synchronization between stores is automatic and real-time, completely transparent to the application layer. + +### 4.3 Why ETL is Eliminated + +Traditional: Database → ETL → Warehouse (minutes-to-hours delay). Native HTAP: one copy of data, two access paths — no ETL, no delay. + +### 4.4 Write Path & Query Routing + +1. Application SQL: standard SQL, no modifications needed +2. PostgreSQL query layer: Parser → Planner → Executor, pg_mooncake embedded via hooks +3. Auto routing: Planner identifies query type, auto-selects optimal path +4. Dual engine execution: row → PG Heap; column → DuckDB + Parquet +5. Auto sync: data remains unified across both stores + +## 5. Application Scenarios + +### Scenario 1: Oracle Data Warehouse Migration + +Oracle types, syntax, and function behaviors are fully compatible. HTAP capabilities arrive with migration — no need to rebuild the analytics pipeline. + +### Scenario 2: AI Agent Mixed Queries + +Point queries go through row store, aggregations through column store — one connection for everything. Real-time consistent data, improved Agent response and decision quality. + +### Scenario 3: Real-Time Risk Control + +Transactions written while column store queries simultaneously. Risk rules trigger in real time, dropping from minutes to sub-second latency. + +## 6. Open Source Release Plan + +- **Release**: Planned alongside IvorySQL 6.0 +- **Components**: Three core modules — ivy_mooncake, ivy_duckdb, ivy_moonlink + +> Note: A preview version [1.0 beta1](https://github.com/IvorySQL/ivy_mooncake/releases/tag/IvorySQL_1.0_beta1) is already available. + +## 7. Summary + +The IvorySQL HTAP Real-Time Lakehouse Access Engine is a strategic move by the IvorySQL community in database architecture evolution. Built on pg_mooncake and validated by industry leaders like Databricks, it fills the twin gaps of independent open-source maintenance and Oracle migration capabilities. Through row-column store fusion, smart routing, and automatic synchronization, it achieves real-time unification of transactions and analytics in a single database — delivering practical solutions for AI Agent queries, real-time risk control, and Oracle migration. With the open-source release of version 6.0, the IvorySQL community will provide enterprises with a truly open, independently evolvable, Oracle-compatible HTAP infrastructure. diff --git a/blog/authors.yml b/blog/authors.yml index 5a8ad76..9c20256 100644 --- a/blog/authors.yml +++ b/blog/authors.yml @@ -21,6 +21,4 @@ ShunWah: ZhangChen: name: ZhangChen 陶郑: - name: 陶郑 -Oreo Yang: - name: Oreo Yang \ No newline at end of file + name: 陶郑 \ No newline at end of file diff --git a/i18n/zh-CN/docusaurus-plugin-content-blog/2026-07-02-ivorysql-htap/index.md b/i18n/zh-CN/docusaurus-plugin-content-blog/2026-07-02-ivorysql-htap/index.md new file mode 100644 index 0000000..3d9d563 --- /dev/null +++ b/i18n/zh-CN/docusaurus-plugin-content-blog/2026-07-02-ivorysql-htap/index.md @@ -0,0 +1,120 @@ +--- +slug: ivorysql-htap +title: "IvorySQL HTAP 实时湖仓接入引擎" +authors: [陶郑] +category: IvorySQL +image: img/blog/covers/ivorysql-htap.png +tags: [IvorySQL, HTAP, DuckDB, PostgreSQL, Lakehouse, Mooncake] +--- + +> 本文章整理于 HOW 2026 中陶郑(IvorySQL 核心贡献者)演讲内容。 +> 视频回放:https://www.bilibili.com/video/BV1KdLq6sEV3/ + +## 一、数据架构的演进历程 + +### 1.1 传统数据架构的局限 + +业务数据从产生到被分析使用,中间需要经过多个系统的流转。当前行业主要存在两种架构模式: + +- **线性架构**:业务数据产生后进入事务数据库,经过 ETL 数据迁徙进入数据湖,再经处理进入数据仓库,最终由 AI 调用数据仓库接口进行分析。这一链路冗长,数据处理环节众多。 +- **变形架构**:数据同时写入事务数据库和分析型数据库(即数据库与数据仓库并存),但本质上仍是多个数据库系统,AI 查询仍需跨系统访问。 + +这两类架构共同面临的困境是:数据处理流程复杂导致分钟级甚至小时级的延迟,且多次查询间原始数据可能已经发生变化,造成数据割裂。与此同时,多套系统的运维成本也居高不下。 + +### 1.2 架构演进的三代路径 + +- **第一代(2000年代——数据仓库)**:随着数据量增长,企业开始按业务需求设计数据结构并存入数据仓库。但后期业务扩展时,新增数据分析需求往往面临数据仓库改造的高昂成本。 +- **第二代(2010年代——数据湖)**:为降低成本,企业开始将全量增量数据保存至廉价的数据湖中,仅在需要时提取至数据仓库。这虽降低了存储成本,却使分析链路更加复杂,维护难度加大。 +- **第三代(2020年前后——湖仓一体)**:融合数据仓库与数据湖的优势,但本质上仍是两个系统的组合。 + +### 1.3 HTAP 原生化:下一代架构 + +当前最新趋势是从"湖仓一体"向"HTAP 原生化"演进——让事务数据与分析数据存储在同一数据库中,彻底消除数据迁徙与 ETL 环节。 + +两代架构的核心差异: + +| 维度 | 湖仓一体 | HTAP 原生化 | +|------|----------|-------------| +| 存储底座 | 对象存储 | 统一高性能存储引擎 | +| 数据新鲜度 | 分钟级/小时级 | 秒级/毫秒级 | +| 核心能力 | 数据湖增加数仓功能 | 数据库具备数仓体量 | + +这意味着多个数据库整合为一个,AI 可通过单一 API 调用,无需跨库查询,运维成本显著降低,数据实时性大幅提升。 + +## 二、AI 时代传统架构的三大痛点 + +- **痛点一:查询模式不可预测**——Agent 生成的查询是随机的,传统架构无法同时高效应对点查和聚合分析。 +- **痛点二:数据新鲜度敏感**——ETL 延迟意味着 Agent 查询时数据可能已变化,在风控、推荐、金融等场景中不可接受。 +- **痛点三:跨系统延迟叠加**——当单次任务同时涉及点查和聚合分析时,跨系统的延迟相互叠加。 + +## 三、HTAP 原生架构:行业动向与 IvorySQL 的定位 + +### 3.1 行业风向标:Databricks 的两笔关键并购 + +- **2025年5月——收购 Neon**:无服务器 PostgreSQL 服务提供商,意味着 OLAP 巨头主动进入 OLTP 领域。 +- **2025年10月——收购 Mooncake Labs**:核心项目 pg_mooncake——在 PostgreSQL 内嵌入 DuckDB 列存引擎,使同一份数据同时支持事务查询与分析查询,彻底无需 ETL。IvorySQL 的方案正是基于 pg_mooncake 构建。 + +### 3.2 pg_mooncake 留下的两个空白 + +- **空白一:开源生态的维护缺口**——被收购后更新频率明显放缓,开源社区需要一个独立维护的实现。 +- **空白二:Oracle 迁移场景的能力缺口**——pg_mooncake 从未考虑过 Oracle 兼容场景,而中国大量企业的核心数据仍运行在 Oracle 上。 + +### 3.3 IvorySQL 的定位与填补 + +IvorySQL 是一个开源的 PostgreSQL 分支,由社区独立维护,完整兼容 PostgreSQL 生态,同时支持 Oracle 语法、数据类型和函数行为。基于这一基础,IvorySQL 的 HTAP 引擎做到: +- 填补空白一:社区独立维护的 pg_mooncake 持续演进版本,迭代方向不受商业公司绑定 +- 填补空白二:全面支持 Oracle 迁移场景的 HTAP 能力 +- 实时湖仓接入引擎:同一份数据同时支持事务与分析两条路径,实时可查 + +## 四、技术路径与架构原理 + +### 4.1 行存与列存:理解 HTAP 的技术基础 + +PostgreSQL 是行存数据库,数据按整行连续存储,适合事务型操作。列存则按列独立排列,查询时可直接提取目标列,无需整行扫描,在大数据量聚合分析场景下性能优势显著。HTAP 的核心思路是:**在同一数据库中同时拥有行存与列存两种存储形态,根据查询特征自动选择最优执行路径。** + +### 4.2 整体架构 + +IvorySQL HTAP 引擎基于 pg_mooncake 构建,核心机制如下: + +**智能路由**:通过 Hooks 方式嵌入 PostgreSQL 内核(不侵入内核代码),Planner 自动识别查询特征,判断走行存(点查/事务)还是列存(聚合/分析)效率更高,自动路由至最优执行路径。 + +**双存储引擎**:行存采用 PG Heap 文件(OLTP),列存采用 DuckDB 引擎 + Parquet 列存格式(OLAP)。 + +**内存共享与自动同步**:在内存中,同一份数据既可通过 PG 行存访问,也可转化为列存供 DuckDB 查询。持久化层面,数据通过 Parquet 等列存格式保存,行存与列存之间自动实时同步,应用层完全无感知。 + +### 4.3 为什么不需要 ETL + +传统方案中,数据库 → ETL → 数仓,延迟达分钟级甚至小时级。HTAP 原生方案中,数据始终只有一份,行存和列存是同一份数据的两种访问路径——ETL 消失了,延迟也消失了。 + +### 4.4 写入路径与查询路由 + +1. 应用层 SQL:标准 SQL 语句,无需任何改造 +2. PostgreSQL 查询层:Parser → Planner → Executor,pg_mooncake 以 Hook 方式嵌入 +3. 自动路由:Planner 识别查询特征,自动决定行存或列存 +4. 双存储引擎执行:行存走 PG Heap,列存走 DuckDB + Parquet +5. 自动同步:两种存储之间自动同步,数据始终只有一份 + +## 五、应用场景 + +### 场景一:Oracle 数仓迁移 + +迁移后,Oracle 类型、语法、函数行为全面兼容,HTAP 能力随迁移一并到位,分析链路不需要重建。 + +### 场景二:AI Agent 混合查询 + +点查走行存,聚合走列存,一个连接搞定所有查询。数据实时一致,Agent 响应速度与决策质量同步提升。 + +### 场景三:实时风控 + +交易写入的同时,列存即可查询。风控规则实时触发,从分钟级降到秒级以内。 + +## 六、开源发布计划 + +- **发布时间**:计划伴随 IvorySQL 6.0 版本正式发布 +- **开源内容**:代码完全开源,包含三个核心模块——ivy_mooncake、ivy_duckdb、ivy_moonlink + +> 注:现在已经发布了预览版 [1.0 beta1](https://github.com/IvorySQL/ivy_mooncake/releases/tag/IvorySQL_1.0_beta1) + +## 七、总结 + +IvorySQL HTAP 实时湖仓接入引擎,是 IvorySQL 社区在数据库架构演进方向上的重要布局。项目紧跟 Databricks 等行业巨头验证的技术方向,基于 pg_mooncake 构建,同时填补了开源独立维护与 Oracle 迁移场景两大空白。通过行存与列存的融合、智能路由与自动同步机制,实现了事务与分析在同一数据库中的实时统一,为 AI Agent 时代的数据查询、实时风控、Oracle 迁移等场景提供了切实可行的解决方案。随着 6.0 版本的开源发布,IvorySQL 社区将为企业用户提供一套真正开放、可独立演进、兼容 Oracle 生态的 HTAP 基础设施。 diff --git a/i18n/zh-CN/docusaurus-plugin-content-blog/authors.yml b/i18n/zh-CN/docusaurus-plugin-content-blog/authors.yml index 5a8ad76..9c20256 100644 --- a/i18n/zh-CN/docusaurus-plugin-content-blog/authors.yml +++ b/i18n/zh-CN/docusaurus-plugin-content-blog/authors.yml @@ -21,6 +21,4 @@ ShunWah: ZhangChen: name: ZhangChen 陶郑: - name: 陶郑 -Oreo Yang: - name: Oreo Yang \ No newline at end of file + name: 陶郑 \ No newline at end of file diff --git a/static/img/blog/covers/ivorysql-htap.png b/static/img/blog/covers/ivorysql-htap.png new file mode 100644 index 0000000..3e363e3 Binary files /dev/null and b/static/img/blog/covers/ivorysql-htap.png differ