Iceberg Metadata Tables

This topic describes how to inspect the metadata information of Iceberg tables in StarRocks.

Overview

From V3.4.1 onwards, StarRocks supports Iceberg metadata tables. These metadata tables contain a variety of information about Iceberg tables, such as table change history, snapshots, and manifests. You can query each metadata table by appending the metadata table name to the original table name.

Currently, StarRocks supports the following Iceberg metadata tables:

Metadata table	Description
history	Shows a log of metadata changes made to the table.
metadata_log_entries	Shows the metadata log entries for the table.
snapshots	Shows details about the table snapshots.
manifests	Shows an overview of the manifests associated with the snapshots in the table’s log.
partitions	Shows details about the partitions in the table.
files	Shows details about the data files and delete files in the current snapshot of the table.
entries	Shows the manifest entries of the current snapshot, including the entries of removed files.
all_data_files	Shows the data files of all snapshots of the table.
all_delete_files	Shows the delete files of all snapshots of the table.
all_manifests	Shows the manifests of all snapshots of the table.
all_entries	Shows the manifest entries of all snapshots of the table.
refs	Shows details about the Iceberg references, including branches and tags.
properties	Shows the table properties as key-value pairs.

Iceberg v3 Row Lineage Metadata Columns

From v4.1 onwards, for Iceberg v3 tables (format-version = 3), StarRocks supports querying the following Row Lineage metadata columns:

Metadata Column	Description
_row_id	Unique row identifier within the table (BIGINT). Format: firstRowId + row_position.
_last_updated_sequence_number	The commit sequence number when the row was last updated (BIGINT).

Usage:

SELECT _row_id, _last_updated_sequence_number, * FROM [<catalog>.][<database>.]table;

note

• The _row_id column requires data files to have firstRowId metadata. If a data file is missing firstRowId, the query will fail with an error.
• For newly inserted data, _row_id is computed as firstRowId + row_position and _last_updated_sequence_number is the file-level dataSequenceNumber.
• After compaction (e.g., Iceberg OPTIMIZE / rewrite-data-files), if the compactor writes _row_id and _last_updated_sequence_number as physical columns in the data files (as required by the Iceberg v3 spec), StarRocks reads the per-row values from the physical columns, preserving row lineage across compaction.
• These metadata columns are only available for Iceberg v3 tables (format-version = 3).

history table

Usage:

SELECT * FROM [<catalog>.][<database>.]table$history;

Output:

Field	Description
made_current_at	The time when the snapshot became the current snapshot.
snapshot_id	The ID of the snapshot.
parent_id	The ID of the parent snapshot.
is_current_ancestor	Whether this snapshot is an ancestor of the current snapshot.

metadata_log_entries table

Usage:

SELECT * FROM [<catalog>.][<database>.]table$metadata_log_entries;

Output:

Field	Description
timestamp	The time when the metadata was recorded.
file	The location of the metadata file.
latest_snapshot_id	The ID of the latest snapshot when the metadata was updated.
latest_schema_id	The ID of the latest schema when the metadata was updated.
latest_sequence_number	The data sequence number of the metadata file.

snapshots table

Usage:

SELECT * FROM [<catalog>.][<database>.]table$snapshots;

Output:

Field	Description
committed_at	The time when the snapshot was committed.
snapshot_id	The ID for the snapshot.
parent_id	The ID for the parent snapshot.
operation	The type of operation performed on the Iceberg table. Valid values: append: New data is appended. replace: Files are removed and replaced without changing the data in the table. overwrite: Old data is overwritten by new data. delete: Data is deleted from the table.
manifest_list	The list of Avro manifest files that contain detailed information about snapshot changes.
summary	A summary of the changes made from the previous snapshot to the current snapshot.

manifests table

Usage:

SELECT * FROM [<catalog>.][<database>.]table$manifests;

Output:

Field	Description
path	The location of the manifest file.
length	The length of the manifest file.
partition_spec_id	The ID for the partition specification that is used to write the manifest file.
added_snapshot_id	The ID of the snapshot during which this manifest entry has been added.
added_data_files_count	The number of data files with status ADDED in the manifest file.
added_rows_count	The total number of rows in all data files with status ADDED in the manifest file.
existing_data_files_count	The number of data files with status EXISTING in the manifest file.
existing_rows_count	The total number of rows in all data files with status EXISTING in the manifest file.
deleted_data_files_count	The number of data files with status DELETED in the manifest file.
deleted_rows_count	The total number of rows in all data files with status DELETED in the manifest file.
partitions	Partition range metadata.
content	The content type of the manifest file. Valid values: DATA(0), DELETES(1). For a delete manifest, the *_data_files_count and *_rows_count fields count its delete files and deleted rows.

partitions table

Usage:

SELECT * FROM [<catalog>.][<database>.]table$partitions;

Output:

Field	Description
partition_value	The mapping of the partition column names to the partition column values.
spec_id	The partition Spec ID of files.
record_count	The number of records in the partition.
file_count	The number of files mapped in the partition.
total_data_file_size_in_bytes	The size of all the data files in the partition.
position_delete_record_count	The total row count of Position Delete files in the partition.
position_delete_file_count	The number of Position Delete files in the partition.
equality_delete_record_count	The total row count of Equality Delete files in the partition.
equality_delete_file_count	The number of Position Equality files in the partition.
last_updated_at	The time when the partition was updated most recently.

files table

Usage:

SELECT * FROM [<catalog>.][<database>.]table$files;

Output:

Field	Description
content	The type of content stored in the file. Valid values: DATA(0), POSITION_DELETES(1), and EQUALITY_DELETES(2).
file_path	The location of the data file.
file_format	The format of the data file.
spec_id	The Spec ID that is used to track the file containing a row.
partition	The partition values of the file, as a struct over the partition fields of all partition specs of the table. A field that the file's partition spec does not have is NULL. This field is absent if the table has never been partitioned. partition is a reserved word, so enclose it in backticks in queries.
record_count	The number of entries contained in the data file.
file_size_in_bytes	The size of the data file.
column_sizes	The mapping between the Iceberg column ID and its corresponding size in the file.
value_counts	The mapping between the Iceberg column ID and its corresponding count of entries in the file.
null_value_counts	The mapping between the Iceberg column ID and its corresponding count of NULL values in the file.
nan_value_counts	The mapping between the Iceberg column ID and its corresponding count of non- numerical values in the file.
lower_bounds	The mapping between the Iceberg column ID and its corresponding lower bound in the file.
upper_bounds	The mapping between the Iceberg column ID and its corresponding upper bound in the file.
split_offsets	The list of recommended split locations.
sort_id	The ID representing sort order for this file.
equality_ids	The set of field IDs used for equality comparison in equality delete files.
key_metadata	The metadata about the encryption key that is used to encrypt this file, if applicable.
first_row_id	The ID of the first row in a data file of a format version 3 table. NULL for delete files.
referenced_data_file	The location of the data file that a deletion vector or a file-scoped position delete file applies to. NULL for data files.
content_offset	The offset of a deletion vector in its Puffin file. NULL for other files.
content_size_in_bytes	The size of a deletion vector in its Puffin file. NULL for other files.

entries table

Usage:

SELECT * FROM [<catalog>.][<database>.]table$entries;

Output:

Field	Description
status	The status of the entry. Valid values: EXISTING(0), ADDED(1), and DELETED(2).
snapshot_id	The ID of the snapshot in which the file was added or removed.
sequence_number	The data sequence number of the file.
file_sequence_number	The sequence number of the snapshot in which the file was added.
data_file	The file, as a struct with the fields of the files table. Its sort order ID field is named sort_order_id.

all_data_files and all_delete_files tables

Usage:

SELECT * FROM [<catalog>.][<database>.]table$all_data_files;
SELECT * FROM [<catalog>.][<database>.]table$all_delete_files;

These tables have the same fields as the files table. They list the data files or the delete files of the manifests of all snapshots of the table, so a file can appear more than once.

all_manifests table

Usage:

SELECT * FROM [<catalog>.][<database>.]table$all_manifests;

This table has the fields of the manifests table and the following field. A manifest appears once for each snapshot that contains it.

Field	Description
reference_snapshot_id	The ID of the snapshot that contains the manifest.

all_entries table

Usage:

SELECT * FROM [<catalog>.][<database>.]table$all_entries;

This table has the same fields as the entries table. It lists the entries of the manifests of all snapshots of the table.

refs table

Usage:

SELECT * FROM [<catalog>.][<database>.]table$refs;

Output:

Field	Description
name	The name of the reference.
type	The type of the reference. Valid values: BRANCH or TAG.
snapshot_id	The snapshot ID of the reference.
max_reference_age_in_ms	The maximum age of the reference before it could be expired.
min_snapshots_to_keep	For branch only, the minimum number of snapshots to keep in a branch.
max_snapshot_age_in_ms	For branch only, the max snapshot age allowed in a branch. Older snapshots in the branch will be expired.

properties table

Usage:

SELECT * FROM [<catalog>.][<database>.]table$properties;

Output:

Field	Description
key	The name of the table property.
value	The value of the table property.

Example output:

key	value
location	s3://bucket/warehouse/db/my_table
write.format.default	parquet
write.parquet.compression-codec	zstd