1. Introduction
Schema Evolution (table structure evolution) is an unavoidable core capability in the data lake scenario. Hudi has provided Schema Evolution support since version 0.x. However, in actual use, there are many compatibility constraints, some operations are not supported, and the coupling with the engine is high.
Hudi 1.x version systematically reconstructs Schema Evolution, introducing mature applications of internal Schema representation (Internal Schema), more complete type promotion (Type Promotion) rules, native support for column renaming and reordering, and Schema compatibility processing that is decoupled from storage formats.
2. Hudi Schema Evolution core mechanism
Hudi 1.x introduced an independent internal Schema representation system (InternalSchema), does not directly rely on Avro Schema or Parquet Schema, but establishes a layer of abstraction:
InternalSchema
├── fields: List<InternalField>
│ ├── id (unique field ID, Does not change with renaming)
│ ├── name
│ ├── type (InternalType)
│ └── isOptional
├── schemaId (version number, and instant time association)
└── record structure (Support nesting)
Core design idea: use field ID (rather than field name) as the unique identifier of the column. This makes column renaming possible - as long as the ID remains the same, historical data will still be mapped correctly even if the name changes.
Hudi supports two strategies at the same time:
- Schema-on-Write (Schema check when writing): When writing data, verify the compatibility of the incoming Schema and the table Schema. If it is not compatible, the write will be rejected.
- Schema-on-Read (Schema alignment when reading): When reading, align the historical file Schema with the current Table Schema, fill in NULL for missing columns, and trim redundant columns.
Schema compatibility check process:
Type-safe promotion paths supported by Hudi:
INT ──────────▶ LONG
LONG ─────────▶ FLOAT ──────────▶ DOUBLE
FLOAT ────────▶ DOUBLE
DECIMAL ──────▶ DECIMAL (Improved accuracy, such as Decimal(10,2) → Decimal(20,2))
STRING ◀────── DATE/TIMESTAMP (In some scenarios, caution is required)
3. Hudi 1.x vs 0.x: Key differences in Schema Evolution
| Dimensions | Hudi 0.x | Hudi 1.x |
|---|---|---|
| Schema identification method | Name-based | ID-based |
| Column rename | Not supported / limited support | Native support (tracking via ID) |
| Column reordering | Not supported | support |
| Nested structure evolution | Partially supported (only add columns at the top level) | Support nested Struct/Map/Array internal changes |
| Internal Schema | 0.11+ experimental introduction | Mature and enabled by default |
| Schema storage location | Avro Schema is stored in .commit file | Internal Schema + Avro dual storage |
| Engine compatibility | Strong dependence on Spark | Improved multi-engine support (Spark/Flink/Presto) |
| Configuration complexity | Multiple configurations need to be enabled manually | Default behavior is more reasonable and configuration is simplified |
| DDL support | LIMITED ALTER TABLE | More complete DDL support |
4. Detailed explanation of the execution path of Schema Evolution
1.Write Path
┌─────────────────────────────────────────────────────────────┐
│ write path Schema deal with │
└─────────────────────────────────────────────────────────────┘
DataFrame / Record
(with incoming schema)
│
▼
┌─────────────────┐
│ HoodieWriteClient│
│ .startCommit() │
└─────────────────┘
│
▼
┌─────────────────────────────┐
│ TableSchemaResolver │
│ 1. read Timeline latest in │
│ committed schema │
│ 2. build InternalSchema │
└─────────────────────────────┘
│
▼
┌─────────────────────────────┐
│ SchemaCompatibilityCheck │
│ - canEvolve(old, new)? │
│ - identify: New/delete/Type change │
│ - application Type Promotion rule │
└─────────────────────────────┘
│
┌────┴────┐
▼ ▼
[compatible] [Not compatible]
│ │
▼ ▼
Merge build abnormal termination
evolvedSchema
│
▼
┌─────────────────────────────┐
│ Write data file │
│ (Parquet/ORC with new schema)│
└─────────────────────────────┘
│
▼
┌─────────────────────────────┐
│ Commit Metadata writing Timeline │
│ (Include evolvedSchema) │
└─────────────────────────────┘
2. Read Path
┌─────────────────────────────────────────────────────────────┐
│ read path Schema Alignment │
└─────────────────────────────────────────────────────────────┘
Query Request (e.g., SELECT * FROM hudi_table)
│
▼
┌─────────────────────────────┐
│ get Latest Table Schema │ ◀── Query Schema (reader expected)
└─────────────────────────────┘
│
▼
┌─────────────────────────────┐
│ File-level Schema get │
│ (each Parquet The file has its own │
│ file schema, OK on writing) │
└─────────────────────────────┘
│
▼
┌─────────────────────────────────────────┐
│ Schema Alignment (Reconciliation) │
│ │
│ File Schema Query Schema │
│ ┌──────────┐ ┌──────────────┐ │
│ │ id:1 name│ │ id:1 username│ │
│ │ id:2 age │ │ id:2 age │ │
│ │ │ │ id:3 email │ │
│ └──────────┘ └──────────────┘ │
│ │
│ Alignment rules: │
│ - id:1 exist at both ends → Read and map to │
│ "username" (according to ID match) │
│ - id:2 normal reading │
│ - id:3 does not exist in file → filling NULL │
│ │
└─────────────────────────────────────────┘
│
▼
Return the aligned data to the query engine
5. Best Practices
1. Choose the appropriate table type
- COW (Copy-on-Write) table: Schema Evolution is relatively intuitive. Every time compaction/write generates a new file, a new Schema is adopted.
- MOR (Merge-on-Read) table: It should be noted that the Schema of the base file and the log file may be inconsistent, and the merging overhead during reading is slightly higher. For scenarios with frequent Schema changes, increase the compaction frequency appropriately.
2. Reserve reasonable field types
-- Recommendation: Reserve enough accuracy to reduce subsequent type improvements
CREATE TABLE orders (
order_id BIGINT, -- rather than INT, Avoid subsequent promotions
amount DECIMAL(20,4), -- Reserve precision space
created_at TIMESTAMP
);
3. Add new column (safety operation)
-- Spark SQL
ALTER TABLE hudi_table ADD COLUMNS (email STRING);
-- Nested structure new
ALTER TABLE hudi_table ADD COLUMNS (address.zipcode STRING);
4. Type promotion (pay attention to compatibility)
-- safe type promotion
ALTER TABLE hudi_table ALTER COLUMN age TYPE BIGINT; -- INT → BIGINT
-- not safe / Unsupported changes (will be rejected)
ALTER TABLE hudi_table ALTER COLUMN name TYPE INT; -- STRING → INT ❌
5. Column renaming (1.x feature)
-- 1.x support
ALTER TABLE hudi_table RENAME COLUMN amt TO amount;
6. Precautions for production environment
┌───────────────────────────────────────────────────────────┐
│ Schema Evolution Production Practice Checklist │
├───────────────────────────────────────────────────────────┤
│ │
│ ✅ before change │
│ ├── Confirm whether the change type is within the supported scope │
│ ├── Assess scope of impact (downstream operations, BI Report, API) │
│ ├── Verify in test environment Schema compatibility │
│ └── backup Timeline metadata │
│ │
│ ✅ When changing │
│ ├── use DDL(ALTER TABLE)rather than modifying it directly Avro Schema document │
│ ├── Make a single change as atomic as possible (avoid renaming at the same time) + type promotion) │
│ └── Monitor write jobs Schema Compatibility check log │
│ │
│ ✅ After change │
│ ├── Verify that historical data can be read normally │
│ ├── Confirm incremental query(Incremental Query)working normally │
│ ├── trigger Compaction(MOR table) to ensure data consistency │
│ └── Update the data directory (e.g. Hive Metastore / AWS Glue) │
│ │
└───────────────────────────────────────────────────────────┘
7. Frequently asked questions and solutions
| question | reason | solution |
|---|---|---|
| Schema incompatibility error reported when writing | Unsupported type changes (e.g. STRING→INT) | Check type promotion rule table, use safe path |
| Historical data reads too many NULLs | After adding the column, the old file has no corresponding data. | Expected behavior, can be set to default value or ETL backfill |
| Compaction failed | MOR table log conflicts with the Schema of base | Check Schema version alignment and upgrade Hudi version |
| Downstream error after column renaming | Downstream consumption by listing | Coordinate downstream synchronization modifications or use the View compatibility layer |
| Hive Metastore Schema is out of sync | DDL operations not synchronized to HMS | Run run_sync_tool or turn on automatic synchronization |