メインコンテンツまでスキップ
バージョン: 4.1 Preview

PhoenixAI version 4.1

v4.1 was first released on April 9, 2026, and has been maintained since; the most recent patch, 4.1.6, was released on September 30, 2026.

This page describes the 4.1 line as a whole rather than patch by patch. The Upgrade and compatibility section separates what changes when you move to 4.1 from what changed later within the line, because a cluster that has been on 4.1 since launch and a cluster arriving on 4.1.6 today face different lists. Everything else is organized by capability: a feature that arrived in 4.1.0 and was extended in a later patch appears once, in its finished form.

Upgrade and compatibility​

Before you upgrade to 4.1​

Upgrade to the newest 4.1 patch, not to 4.1.0. 4.1.0 was released before 4.0.9, and several later 4.0 patches added things that 4.1.0 does not recognize. If you created SQL user-defined functions on 4.0.11 or later, a 4.1.0 FE cannot load the metadata that holds them; support returns in 4.1.1. Settings added in 4.0 patches are silently ignored by a 4.1 patch that predates them: 4.1.0 does not recognize scan_lake_partition_num_limit or binary_encoding_format (both back in 4.1.1), and LDAP case-insensitive matching (authentication_ldap_case_insensitive) and authentication-failure caching, added in 4.0.16, arrive in the 4.1 line only with 4.1.6. Choose a 4.1 patch released after the 4.0 patch you run now, and after upgrading confirm that every setting you rely on is still recognized, in fe.conf, in be.conf, and in global variables set with SET GLOBAL. Container deployments in particular should skip 4.1.0: its BE and CN processes can crash at startup, which 4.1.1 fixes.

Take a fresh metadata image before you upgrade. 4.1 no longer recognizes the journal record that 4.0 writes when a table's in_memory property is altered. If such a record has not yet been folded into an image, the 4.1 FE stops on it at startup. Run ALTER SYSTEM CREATE IMAGE on the leader FE and wait until the image reaches every follower before upgrading the first FE.

Downgrade is restricted. After the upgrade, 4.1 writes metadata records that no 4.0 patch recognizes, including records it writes on its own when the recycle bin purges dropped partitions and databases. Roll back only to 4.0.6 or later. To roll back, set metadata_ignore_unknown_operation_type and metadata_ignore_unknown_subtype to true in fe.conf. Objects that only 4.1 understands do not survive a rollback: range-distributed tables, tablet split and merge jobs, SQL digest blacklist entries and external cluster snapshots. In shared-data clusters, Primary Key tables created on 4.1 use a new primary key encoding that 4.0 does not have; contact support before rolling back a cluster that contains such tables.

Settings removed in 4.1 must be renamed or dropped. The session variable enable_pre_agg_top_n_push_down is gone, and materialized_view_subuqery_text_match_max_count is now spelled materialized_view_subquery_text_match_max_count. SET on a removed variable, or naming one in a SET_VAR hint, fails with Unknown system variable, and values set with SET GLOBAL before the upgrade are dropped, recorded only as an error in fe.log, so re-apply them under the new names. The FE configuration item tablet_reshard_split_size is ignored if it remains in fe.conf, and ADMIN SET FRONTEND CONFIG on it returns an error. If you are upgrading from 4.0.8 or earlier, note also two changes that reached the 4.0 line in 4.0.9: dump_log_dir is gone, with fe.dump.log written to sys_log_dir, and log_plan_cancelled_by_crash_be is deprecated in favor of log_plan_on_query_failure.

Defaults that change on upgrade to 4.1​

These take effect without any action on your part. None of them is in any 4.0 patch.

  • Materialized views decide how many partitions to refresh per run. default_mv_partition_refresh_strategy moves from strict to adaptive. This applies to every partitioned materialized view without its own partition_refresh_strategy property, existing views included. Each refresh task now takes as many partitions as fit within about 100 million rows, 20 GB and 64 partitions of base-table data, so refreshes finish in fewer, heavier task runs. To restore the 4.0 behavior cluster-wide, set default_mv_partition_refresh_strategy = strict; for one view, run ALTER MATERIALIZED VIEW <mv> SET ("partition_refresh_strategy" = "strict").

  • A refresh task can take up to 64 partitions, up from 10. mv_max_partitions_num_per_refresh moves from 10 to 64. Set it back to 10 if refreshes compete with queries for resources.

  • When a refresh is split across task runs, the newest partitions go first. materialized_view_refresh_ascending moves from true to false. Recent data appears in the view sooner after a backfill; a monitoring job that judges progress by the oldest partition needs changing. Set it back to true to refresh oldest-first. Separately, max_mv_task_run_meta_message_values_length moves from 16 to 8, so partition lists in task-run messages are truncated sooner.

  • Shared-nothing clusters use the slot-based Query Queue v2. enable_query_queue_v2 moves from false to true. Nothing is queued unless query queues are enabled (enable_query_queue_select, enable_query_queue_load or enable_query_queue_statistic); if they are, admission is computed from slots, and the global variables query_queue_mem_used_pct_limit and query_queue_cpu_used_permille_limit no longer take effect. To keep the v1 queue, set enable_query_queue_v2 = false in fe.conf and restart the FEs; the item cannot be changed at runtime. Shared-data clusters configure queueing per warehouse and are unaffected.

  • ALTER TABLE ... DISTRIBUTED BY runs online in shared-nothing clusters (4.1.0 to 4.1.4). enable_online_optimize_table moves from false to true, so changing a table's hash distribution or bucket count no longer blocks loads. 4.1.5 sets it back to false; see Other operational changes within the line.

  • Tablet splitting targets 10 GB tablets. tablet_reshard_split_size (4 GB) is replaced by tablet_reshard_target_size (10 GB), and one tablet can split into up to 1024 tablets rather than 8 (tablet_reshard_max_split_count). A tablet is split once it reaches twice the target size. Scripts that pass tablet_reshard_split_size to ALTER TABLE ... SPLIT TABLET now fail with Unknown properties; use tablet_reshard_target_size.

  • New shared-data tables use Fast Schema Evolution v2. Tables created on 4.1 have cloud_native_fast_schema_evolution_v2 = true, so adding, dropping or widening a column updates only FE metadata. Tables created before the upgrade keep the previous behavior until you set the property to true. See New features.

  • VARBINARY without a length is stored as VARBINARY(1048576). The documented default is now written into the metadata of new tables and added columns, so SHOW CREATE TABLE and DESC show the length. CTAS and materialized views keep their previous types.

  • KILL QUERY surfaces as error 1317 (ER_QUERY_INTERRUPTED, SQLSTATE 70100) instead of a generic error, so client retry logic can tell a deliberate cancel from a failure.

  • <string> IN (<date>, ...) compares as dates. When every item in an IN list is a DATE or DATETIME of the same type and the left side is a string, the string is now cast to that type rather than the dates being cast to strings. '2024-12-09 00:00:00' IN (DATE '2024-12-09') now matches. Strings that do not parse as dates compare as NULL. Review filters that relied on exact string matching.

  • Spark Load uses the 4.1 DPP library. spark_dpp_version moves from 1.0.0 to the 4.1 version, and the first Spark Load after the upgrade uploads the new library to the Spark repository. Remove any pin of spark_dpp_version in fe.conf.

Changes within the 4.1 line that can alter query results​

Each of these corrects behavior that was wrong. The previous results were incorrect, so a query whose output changes was returning the wrong answer before — but the change is still visible, and a report or downstream job built on the old output needs checking. Where the same fix shipped in a 4.0 patch, a cluster upgrading from that patch or later already has it.

  • Routine Load no longer skips messages in newly added Kafka partitions (4.1.3). When property.kafka_default_offsets is not set and a partition appears after a job has started, the job now reads that partition from the beginning instead of from the end, which silently skipped whatever was produced before the partition was discovered. Expect a burst of older messages the first time a topic gains partitions after the upgrade. A job's first discovery still starts at the end. Set kafka_default_offsets explicitly to choose the behavior.

  • A shared-data Primary Key load that mixes UPSERT and DELETE for the same key keeps their order (4.1.5). A delete inside a transaction was previously applied after all of that transaction's upserts, wherever it appeared, so a load that upserted a key after deleting it lost the row. lake_enable_pk_preserve_txn_delete_order now defaults to true. Before rolling back to 4.1.3 or earlier, or running a mixed-version cluster, set it to false on every BE and CN.

  • Incremental materialized views that use COUNT(<nullable column>) no longer count NULLs (4.1.2). Views created before the fix hold inflated counts and must be dropped and re-created.

  • String-to-number schema changes follow CAST semantics (4.1.4). ALTER TABLE ... MODIFY COLUMN from a string to a numeric type now stores NULL for values that overflow or do not parse. Previously it stored a wrapped value — '99999999999' to INT gave 1215752191 — or 0 for 'abc'.

  • TABLESAMPLE and ANALYZE SAMPLE TABLE take effect on shared-data tables (4.1.4). The sample options never reached the scan, so TABLESAMPLE returned every row and sampled statistics were computed from a full scan. (Also in 4.0.17.)

  • rewrite_data_files with a WHERE clause no longer loses data (4.1.6). Upgrade before compacting Iceberg tables selectively. (Also in 4.0.15.)

  • Iceberg and Delta Lake partitions stored as date strings are pruned and deleted correctly (4.1.4). When a partition column is a string holding a date and is compared with a DATE or DATETIME value, partition pruning, metadata DELETE and equality-delete application now behave correctly; equality deletes were under-applied, so deleted rows came back.

  • An Iceberg time-travel read uses the schema and partition spec of the snapshot it targets (4.1.4), not the table's current ones, and a materialized view is no longer used to answer a time-travel query (4.1.5). Materialized view rewrite also stops serving stale results after an Iceberg rollback_to_snapshot (4.1.4; also in 4.0.14).

  • Iceberg v3 tables with deletion vectors, and encrypted Iceberg tables, are rejected (4.1.1). On 4.1.0 a query could ignore deletion vectors and return deleted rows. The query now fails with an error that names the unsupported feature.

  • Oracle columns in JDBC catalogs change type (4.1.0). An Oracle NUMBER without declared precision, previously a string, maps to DECIMAL(38, 6); the scale is set by the catalog property oracle.number.default-scale. Oracle TIMESTAMP types map to VARCHAR(64) rather than VARCHAR(1048576), or to DATETIME with oracle.temporal.to-datetime (or oracle.timestamptz.to-datetime) set to true. Downstream schemas that expected strings need checking.

  • Unsigned Parquet INT32 values are zero-extended (4.1.3); values with the high bit set used to load as negative numbers. Pre-1970 Parquet and ORC timestamps keep their sub-second part on load.

  • Parquet INT64 timestamps written as timezone-naive are no longer shifted (4.1.3). When isAdjustedToUTC=false, FILES() and broker or stream loads shifted values by the session timezone offset; they are now read as wall-clock values. INT96 timestamps nested in STRUCT, ARRAY or MAP now receive the same session-timezone shift as top-level INT96 columns. (Also in 4.0.12.)

  • LIKE escaping on constant operands follows MySQL 8 (4.1.3), and LIKE with the _ wildcard returns correct rows on columns with a GIN inverted index. (Also in 4.0.13.)

  • VARCHAR(N) length is preserved through CTAS (4.1.3) when the source carries an explicit length, instead of being widened to VARCHAR(1048576). (Also in 4.0.12.)

  • VARBINARY values nested inside ARRAY, MAP or STRUCT are hex-encoded in MySQL result sets (4.1.1). Raw bytes broke text-protocol parsing. binary_encoding_format (hex, base64 or raw) and binary_encoding_level (nested or all) control the encoding; top-level VARBINARY columns are unchanged by default. (Also in 4.0.10.)

  • Ranger row-filter and column-masking policies apply to Hive views in external catalogs (4.1.2). Queries through such a view previously bypassed them. (Also in 4.0.12.)

  • A division with a non-constant divisor is no longer treated as monotonic (4.1.4), so Zone Map pruning no longer drops rows that satisfy a predicate such as 10 DIV c, and NOT MATCH on a GIN inverted index no longer returns NULL rows (4.1.4). (Both also in 4.0.14.)

  • List partition pruning no longer discards a partition because its value list contains NULL (4.1.6). !=, NOT IN and IS NOT NULL predicates, including the IS NOT NULL a planner derives on a join key, previously dropped a whole multi-value partition. Range predicates also map onto expression-partition columns only when the expression preserves order. (List partition fix also in 4.0.16.)

  • Query cache results are keyed correctly (4.1.5). The cache digest now includes the scan's schema and join-predicate sub-expressions, and incomplete per-tablet results are no longer stored, so a cached result can no longer be served for a different query. (Partly also in 4.0.14 and 4.0.15.)

  • Date and string functions (4.1.6). months_diff and years_diff return 0 within the same month or year rather than -1 in some cases, and substr on invalid UTF-8 no longer returns bytes from neighboring rows. (Also in 4.0.16.)

  • Searched CASE WHEN returning a collection type evaluates each branch only on the rows that select it (4.1.6), so a branch that would raise an error no longer does when no row selects it. Set the BE configuration case_when_selective_eval_ratio to 0 to restore the previous evaluation.

  • Data Cache keys for external-table files no longer depend on the CPU (4.1.6). 4.1.0–4.1.5 derived these keys from a CRC hash on x86 CPUs with SSE4.2; they are now always MurmurHash3, as in 4.0. On x86 nodes, entries cached by 4.1.0–4.1.5 are not reused, so the external-table Data Cache re-warms once after the upgrade.

Changes within the 4.1 line that can cause a statement to fail​

  • Partition guards cover Iceberg and Delta Lake (4.1.3). allow_lake_without_partition_filter (formerly allow_hive_without_partition_filter) and scan_lake_partition_num_limit previously applied to Hive, Hudi and Paimon only. Both defaults are permissive, but a cluster that set allow_hive_without_partition_filter = false globally now also rejects Iceberg and Delta Lake queries without a partition predicate, with Partition pruning is invalid. On 4.1.0 only, scan_lake_partition_num_limit is not recognized; 4.1.1 restores it.

  • Incremental materialized views reject what they cannot maintain correctly (4.1.1). "refresh_mode" = "AUTO" is no longer accepted; only INCREMENTAL and PCT are. Paimon tables can no longer be base tables. REFRESH ... FORCE, partition-scoped REFRESH and switching refresh mode with ALTER are rejected; drop and re-create the view instead. CREATE rejects aggregates outside a supported list, DISTINCT aggregates and, from 4.1.2, HAVING. Incremental views no longer serve automatic query rewrite. Check any incremental view created on 4.1.0.

  • CREATE STORAGE VOLUME and ALTER STORAGE VOLUME validate the location first (4.1.2). In shared-data clusters, invalid credentials, endpoints or network access fail the statement immediately instead of being persisted and failing later. Set enable_storage_volume_access_check = false to skip the check.

  • Iceberg NOT NULL columns are enforced on write (4.1.5). A NULL now fails with NULL value in non-nullable column '<name>', where it was previously written.

  • remove_orphan_files rejects an older_than less than 24 hours old (4.1.5). A cutoff that recent can delete files a concurrent write has not yet committed. If no concurrent writer can be affected, lower iceberg_remove_orphan_files_min_retention_seconds (default 86400, dynamically modifiable).

  • A materialized view whose Iceberg or external base table drifts becomes inactive at its next refresh (4.1.2) — a changed column type, a dropped column or a dropped table — instead of silently producing NULL rows.

  • Sort key columns of a type the BE cannot encode are rejected (4.1.5), including JSON, complex, floating-point, VARIANT and TIME types, where the BE previously crashed on rewrite. On range-distributed tables, ALTER TABLE ... ORDER BY that reorders key columns of a key-derived sort key is rejected (4.1.4).

  • ARRAY and MAP constructors fail rather than corrupt (4.1.4). A constructor whose result exceeds the addressable size of one chunk raises CapacityLimitExceed instead of returning corrupted values. pipeline_enable_large_column_checker is also enabled by default (4.1.2), so a query producing an oversized column fails with Large column detected rather than proceeding. Reduce chunk_size or split the expression. (The checker default is also in 4.0.12, the constructor check in 4.0.14.)

  • A static partition clause in an INSERT into an Iceberg or Hive table is validated even when a column list is given (4.1.5). A clause naming a non-partition column is rejected instead of ignored. (Also in 4.0.14.)

  • TRUNCATE TABLE <catalog>.<db>.<table> checks privileges against the named catalog (4.1.6) rather than the session's current catalog.

  • Analysis rejects statements that previously reached the BE (4.1.6): SELECT DISTINCT ... ORDER BY a column not in the select list, a DISTINCT aggregation over PERCENTILE, and an ASOF JOIN temporal condition that does not compare one column from each side.

Other operational changes within the line​

  • Online optimize is off by default again (4.1.5). enable_online_optimize_table returns to false, so on a cluster upgrading from 4.1.0–4.1.4, ALTER TABLE ... DISTRIBUTED BY blocks writes to the table while it runs, as in 4.0. Set it to true to keep the online behavior.
  • Query Queue v2 estimates slots from parallelism by default (4.1.4). query_queue_slots_estimator_strategy defaults to PBE instead of MAX, and a CPU-cost-based estimator, CBE, is added. MAX and MIN no longer exist; a cluster or warehouse configured with either is treated as PBE without an error. Set MBE or CBE to weigh memory or CPU cost.
  • Shared-data tablet metadata and transaction logs carry a checksum by default (4.1.5). lake_enable_protobuf_file_checksum moves to true, so corruption is detected when the file is read. Files written this way cannot be read by 4.1.0–4.1.2 or by 4.0 patches before 4.0.13. Before rolling back to one of those versions, set it to false on every BE and CN; files already written in the new format remain unreadable to the older reader.
  • Copy-on-write column optimization is off (4.1.2). enable_cow_optimization moves from true to false because a design flaw could crash the BE. Expect a small CPU cost in exchange; turning it back on is not recommended.
  • Loads into automatic-partition tables open more partitions up front (4.1.2). max_load_initial_open_partition_number moves from 32 to 4096 and becomes a ceiling rather than a fixed count: LIST-partitioned tables, and RANGE-partitioned tables loaded by INSERT, Broker Load or Spark Load, open all partitions up to it, while Stream Load and Routine Load into RANGE tables keep 32. The table property load_initial_open_partition_number overrides it per table.
  • Shared-data vacuum batches more deletions per request (4.1.3). lake_vacuum_min_batch_delete_size moves from 100 to 200.
  • Data Cache disk quota grows back after 3 minutes of low usage instead of 2 hours (4.1.6). datacache_disk_idle_seconds_for_expansion moves from 7200 to 180, and a quota that auto-adjust shrank to zero now recovers.
  • The AWS SDK's own HTTP client is used again (4.1.6). enable_poco_client_for_aws_sdk defaults to false; set it to true to restore the previous client.
  • jemalloc's dedicated oversize arena is enabled by default (4.1.6), so allocations of 128 MB or more no longer land in the calling thread's per-CPU arena. This improves reuse of large buffers and reduces fragmentation. Set oversize_threshold:0 inside jemalloc_conf to restore the previous layout.
  • Shared-data Primary Key tablets no longer start a base compaction on their own when deletes accumulate (4.1.6). In 4.1.4 and 4.1.5 the delete thresholds triggered it automatically, which rewrote large update-heavy tablets repeatedly. Set lake_pk_compaction_base_enable_delete_trigger = true to restore the trigger; ALTER TABLE ... COMPACT runs a base compaction on demand either way.
  • An exception that escapes a BE thread-pool task stops the process (4.1.5) instead of being logged and ignored, which could report a task as successful when it had produced nothing. As a temporary mitigation for a crash loop, set enable_threadpool_catch_task_exception = true.
  • Result delivery to a slow client has a timeout (4.1.2). mysql_send_packet_timeout_ms (default 60 seconds) bounds how long the FE waits for a client to read results; set it to 0 for the previous unbounded wait.
  • Several settings are renamed; the old names keep working. scan_hive_partition_num_limit becomes scan_lake_partition_num_limit (4.1.1) and allow_hive_without_partition_filter becomes allow_lake_without_partition_filter (4.1.3). On the FE, files_enable_insert_push_down_schema becomes files_enable_insert_push_down_column_type (4.1.1), azure_adls2_oauth2_oauth2_client_endpoint becomes azure_adls2_oauth2_client_endpoint (4.1.2), and slow_lock_log_every_ms becomes slow_lock_log_l2_info_interval_ms (4.1.2). On the BE, txn_commit_rpc_timeout_ms becomes stream_load_thrift_rpc_timeout_ms (4.1.2).
  • Some configuration items are removed. lock_manager_enabled (4.1.2) is gone, so FE metadata locks always use the lock manager. skip_pk_preload (4.1.1), enable_pk_index_eager_build (4.1.4) and segment_sort_key_sample_row_interval (4.1.6, replaced by sort_key_max_samples_per_tablet) are also gone. A leftover entry in fe.conf or be.conf is ignored at startup, but a runtime update that names one fails, so remove them from automation scripts.
  • The FE starts listening only after startup materialized view activation completes (4.1.5), controlled by enable_mv_startup_activation_gate (default true). An FE with many materialized views takes longer to start serving, but is usable once it does.
  • A statistics health figure pinned near 1.0 for tables with many stale partitions is corrected (4.1.5), so auto-ANALYZE may re-analyze such tables after the upgrade.
  • paimon.option.cache-enabled = false disables the Paimon metadata cache again (4.1.6), and HDFS files are no longer hsynced before close.

New features​

Enterprise-only capabilities​

These are available in PhoenixAI and not in open-source StarRocks.

  • Cross-region disaster recovery for shared-data clusters (Beta)

    Cluster snapshots have protected against losing a cluster, but not against losing a region: the snapshot sat in the same storage as the data it protected. From 4.1, an automated cluster snapshot can be external — ADMIN SET AUTOMATED CLUSTER SNAPSHOT ON ... PROPERTIES ("snapshot_scope" = "external"). Each run then writes a full FE metadata image and copies the data files that changed since the previous run to a storage volume in a second region, and a cluster in that region can restore from the latest snapshot without the primary cluster or its storage. The amount of data at risk is bounded by the snapshot interval. A warehouse property chooses which warehouse does the copying.

    Later patches hardened the feature: the interval survives FE restarts and snapshots stay restorable across tablet split and merge (4.1.3–4.1.4), each snapshot records its scope and source storage volumes and restore is replay-safe (4.1.5), and cleanup is retryable (4.1.6). Run 4.1.6 or later if you rely on it.

  • Resource groups scoped to warehouses

    A resource group previously applied to every warehouse in a cluster. From 4.1.0 the warehouses attribute restricts a group to the named warehouses: the FE considers it only for queries running there, and nodes in other warehouses hold it inactive. A group scoped this way sets its CPU share with cpu_weight_percent or exclusive_cpu_percent, which express the soft weight and the hard reservation as a percentage of each node's cores, so one definition behaves the same in warehouses of different sizes.

  • Query queue policy per warehouse

    From 4.1.4, Query Queue v2 can be tuned for each warehouse with ALTER WAREHOUSE: query_queue_slots_estimator_strategy, query_queue_v2_concurrency_level, query_queue_v2_mem_bytes_per_slot, query_queue_v2_cpu_costs_per_slot and query_queue_v2_schedule_strategy. Properties left unset fall back to the FE configuration, so an interactive warehouse and a batch warehouse no longer have to share one admission policy.

  • The real client address survives a load balancer

    Behind a TCP load balancer, every connection appeared to come from the balancer, so host-restricted accounts, SHOW PROCESSLIST and the audit log all saw the wrong address. From 4.1.2 the FE accepts a PROXY protocol v1 header on the MySQL port from the networks listed in mysql_proxy_protocol_networks, and uses the client address it carries for authentication, the process list and auditing. It is off by default. List only your load balancers' ranges, because any peer allowed to send the header can claim any client address. Only v1 is supported, so a load balancer that emits v2 is not compatible.

Shared-data architecture and storage​

  • Tablets split and merge as the data changes (Beta)

    A hash-bucketed table fixes its bucket count when it is created. When data grows unevenly — tenants of different sizes, or a hot key range — some tablets become too large while others stay nearly empty, and the only remedy was to rebuild the table. 4.1 adds range-based distribution for shared-data clusters: rows are ordered by the sort key, each tablet covers a contiguous key range, and a tablet that grows past tablet_reshard_target_size (10 GB by default) is split automatically. Adjacent tablets can be merged when they shrink, and the new ALTER TABLE ... MERGE TABLET merges on demand alongside the existing ALTER TABLE ... SPLIT TABLET.

    The line extended the feature in every patch. From 4.1.1, range-distributed tables can form colocate groups and take part in colocate joins, extended to outer, semi and anti joins in 4.1.4. From 4.1.2, the first bulk load into a range-distributed table samples the incoming sort keys and pre-splits its tablets so the load runs in parallel from the start; this covers INSERT ... SELECT FROM FILES() and Broker Load in 4.1.2, INSERT ... SELECT from another table in 4.1.3, and Iceberg sources and overwrite loads in 4.1.5. Pre-splitting is on by default for eligible tables and can be turned off per session with enable_tablet_pre_split = false.

    Range distribution is opt-in in 4.1: set the FE configuration enable_range_distribution to true, and new tables created without a DISTRIBUTED BY clause use it. Merging is also off by default; set tablet_reshard_enable_tablet_merge to true to allow it.

  • Large tablets no longer bottleneck on a single thread (Beta)

    In a shared-data cluster each tablet was compacted, published and indexed by one thread, so a large tablet became a serial bottleneck however many cores the Compute Node had. 4.1 parallelizes work inside a tablet across the write path. Compaction splits a large tablet's input into parallel subtasks, by default up to three per tablet, set cluster-wide by lake_compaction_max_parallel_default or per table by the lake_compaction_max_parallel property. MemTable finalize, flush and spill merge run in parallel during loads; the cloud-native Primary Key index is range-split and compacted in parallel; and Primary Key publish, partial updates and conditional updates run in parallel within the tablet (4.1.1). The practical effect is that fewer, larger tablets — with less FE metadata and fewer small files — no longer cost load latency or compaction backlog. These are on by default.

    From 4.1.4, Primary Key tablets also support base compaction, which rewrites delete-heavy data to reclaim deleted rows. Run it with ALTER TABLE ... COMPACT. In 4.1.4 and 4.1.5 it is also triggered automatically by delete volume; from 4.1.6 that trigger is opt-in through lake_pk_compaction_base_enable_delete_trigger.

  • Column changes on shared-data tables complete when the statement returns

    With Fast Schema Evolution v2, ALTER TABLE adding, dropping or widening a column updates only FE metadata rather than rewriting tablet metadata in object storage, and finishes in seconds however many partitions and tablets the table has. In 4.1 this is controlled per table by the cloud_native_fast_schema_evolution_v2 property, which is on for tables created on 4.1; set it to true on older tables with ALTER TABLE ... SET. ALTER MATERIALIZED VIEW ... ADD COLUMN and DROP COLUMN let a single-table aggregate materialized view gain or lose a column without being rebuilt.

  • Full-text inverted indexes on shared-data tables (Beta)

    Inverted indexes used CLucene, which works only in shared-nothing clusters. 4.1 adds a built-in implementation that works in both architectures and is chosen automatically in shared-data clusters, so INDEX ... USING GIN accelerates MATCH and text-filter predicates on cloud-native Duplicate Key and Primary Key tables. The dict_gram_num property adds an n-gram index over the term dictionary, which speeds up LIKE and wildcard searches, and MATCH_ALL joins MATCH_ANY. As with other inverted indexes, set the FE configuration enable_experimental_gin to true before creating one.

  • Restore a single table from a cluster snapshot

    Recovering one table that had been dropped or damaged meant restoring the whole cluster to a snapshot and losing everything written since. From 4.1.0, RESTORE TABLE <source> FROM SNAPSHOT <snapshot> TO TABLE <new_table> rebuilds one table, as it was in a finished cluster snapshot, under a new name beside the live data, so you can compare it with the current table or swap it in. The target table must not already exist.

  • Migrate between shared-data clusters

    The cross-cluster migration tool previously moved data from shared-nothing to shared-data clusters only. From 4.1 it also replicates tables, including range-distributed tables, from one shared-data cluster to another, copying files in parallel, verifying file integrity and skipping files already present on the target. The target must run 4.1 or later.

  • A publish queue stuck on one transaction can be cleared

    When a committed transaction on a shared-data table can never be published — because its transaction log or a segment file was lost — every later load to the table queues behind it. From 4.1.2, ADMIN SKIP COMMITTED TRANSACTION <txn_id> REASON '...' finalizes that one transaction as a no-op so the queue drains without dropping the partition. It applies to load and compaction transactions on tables with file_bundling enabled. The transaction's data is discarded irreversibly, so the statement requires the SYSTEM OPERATE privilege and is rejected unless enable_admin_skip_committed_txn is set to true.

  • Cache state is visible per tablet

    The [_CACHE_STATS_] table hint, as in SELECT * FROM <table> PARTITION(<p>) [_CACHE_STATS_] (the square brackets are literal), returns, for each tablet of a cloud-native table, the bytes cached locally and the tablet's total size, which shows whether a slow query reads from object storage and how much of a table a CACHE SELECT warmed. CACHE SELECT on a Primary Key table now also warms the Primary Key index, and dropping a table or partition evicts its cached data immediately (lake_enable_drop_tablet_cache, on by default).

  • Stream Load and Merge Commit honor the user's default warehouse. A request without a warehouse header runs in the warehouse set as the user's session.warehouse property, rather than always in default_warehouse.

  • Routine Load can load message metadata (4.1.4). INCLUDE METADATA (...) in CREATE ROUTINE LOAD exposes Kafka or Pulsar metadata such as partition, offset and timestamp as loadable columns.

  • Shared-data scans skip segments by sort-key range. A scan skips segments whose sort-key range cannot satisfy the query's predicates without opening them (enable_lake_segment_metadata_filter, on by default).

  • Multi-table transactions can be batch-published (4.1.5) with lake_enable_batch_publish_multi_table = true, which removes the serialization CDC micro-batches fanning out to many tables hit. Enable it only after every FE runs 4.1.5 or later.

Lakehouse​

  • Rows can be deleted from Iceberg tables

    A row-level change to an Iceberg table had to go through another engine. From 4.1.0, DELETE FROM <iceberg_table> WHERE ... runs in PhoenixAI: it writes Iceberg position delete files and commits them as a new snapshot, without rewriting data files. The WHERE clause is required and can include subqueries; partition clauses, USING and WITH are not supported, and the table's data files must be Parquet. Each commit records the engine, its version and the PhoenixAI user in the snapshot summary.

  • TRUNCATE TABLE empties Iceberg tables and Hive managed tables

    From 4.1.0, TRUNCATE TABLE applies to Iceberg tables and to Hive managed tables in an external catalog. On Iceberg it commits a snapshot that removes every data file, so history stays available for time travel until the snapshots expire; Iceberg truncates whole tables only. On Hive, TRUNCATE TABLE <t> PARTITION (col = value, ...) truncates the named partitions; external Hive tables are rejected. Delta Lake and Paimon tables do not support TRUNCATE.

  • Iceberg maintenance gains procedures and named arguments

    Iceberg maintenance in 4.0 was a fixed set of ALTER TABLE ... EXECUTE operations taking positional arguments. From 4.1.0 each operation is a procedure that also accepts named arguments, as in EXECUTE expire_snapshots(older_than = '...', retain_last = 2). New in 4.1 are retain_last for expire_snapshots; location for remove_orphan_files, which limits the scan to a subdirectory; rewrite_manifests, which merges small manifests by partition; add_files, which adds existing Parquet or ORC files to an Iceberg table from a Hive table or a storage location without copying them; and CALL <catalog>.system.register_table(...), which registers an existing table from its metadata file. Together these let a table be migrated from Hive or adopted from another catalog and then kept healthy from SQL.

  • Iceberg v3 row lineage and default values

    On Iceberg v3 tables, every row has a stable identity: _row_id and _last_updated_sequence_number can be selected like ordinary columns, which is what change-tracking and deduplication jobs need. From 4.1.1, rewrite_data_files writes these fields into rewritten files, so row identity survives compaction. Columns on v3 tables can carry a default value, set in CREATE TABLE or with ALTER TABLE ... ADD COLUMN ... DEFAULT; an INSERT that omits the column writes the default, and files that predate the column read it back as the default. New tables are still created as v2 unless you set "format-version" = "3".

  • VARIANT for semi-structured Iceberg data (Beta)

    Iceberg tables whose Parquet files use the variant encoding can be queried directly. VARIANT columns are recognized automatically, values are extracted with variant_query and the typed get_variant_* functions, and they can be cast to scalar, ARRAY, MAP or STRUCT types. Only unshredded variant values are supported, and VARIANT is not available for native tables.

  • Incremental materialized views over Iceberg (Beta)

    A partition-based refresh recomputes every partition that changed, so a few thousand appended rows can cost a full partition's worth of work. With "refresh_mode" = "INCREMENTAL", a materialized view over Iceberg tables consumes only the snapshots appended since its last refresh and merges the result into the view, which is stored as a Primary Key table. The first refresh builds a full baseline. Supported queries are filters, projections, UNION ALL, inner and cross joins, and aggregations from a supported list that includes COUNT, SUM, AVG, MIN, MAX, ARRAY_AGG, approximate and bitmap distinct counts, and HLL, bitmap and percentile unions (extended in 4.1.1 through 4.1.3). An unsupported query is rejected at CREATE time with the reason. Incremental views do not take part in automatic query rewrite, so query them directly; see Upgrade and compatibility for what changed after 4.1.0.

  • Concurrent writes to one Iceberg table no longer collide at commit

    Iceberg commits are optimistic, so two loads committing to one table at the same moment could fail with a conflict. From 4.1.0, commits to the same table are queued and applied one at a time while different tables still commit in parallel. This is on by default and controlled by enable_iceberg_commit_queue, iceberg_commit_queue_timeout_seconds (default 300) and iceberg_commit_queue_max_size (default 1000). Global shuffle on write now routes rows by the transformed partition value for bucket, truncate and time transforms, so each partition is written by fewer writers and produces fewer, larger files; from 4.1.1, tables partitioned by a transform of a TIMESTAMPTZ column can be written as well.

  • Paimon time travel and views

    A query can pin a Paimon table to a snapshot ID, to the latest snapshot at or before a timestamp, or to a branch or tag ('branch:<name>', 'tag:<name>'), which gives reproducible reads and lets you compare branches. From 4.1.2, Paimon views can be created, listed, queried and dropped through a Paimon catalog, with table references inside them resolved against the Paimon catalog.

  • AWS Glue tables with partition projection are readable. Athena partition projection computes partitions from table properties rather than storing them in the catalog, so such tables appeared unpartitioned or failed in 4.0. Enum, integer, date and injected projections are now evaluated during partition listing and pruning.

  • JDBC catalogs map more source types and hold connections better. PostgreSQL TIME, timetz, timestamptz, json and jsonb columns map to TIME, DATETIME and JSON, PostgreSQL column comments are shown, and MySQL BINARY and VARBINARY map to VARBINARY. Pooled connections are recycled before the remote side times them out (jdbc_connection_max_lifetime_ms, jdbc_connection_keepalive_time_ms), database lists are cached, and from 4.1.3 row counts for MySQL, PostgreSQL and ClickHouse tables are fetched asynchronously for planning. See Upgrade and compatibility for the Oracle type changes.

  • FILES() takes an explicit schema (4.1.2). "schema" = "col TYPE, ..." declares the columns to read and their types instead of inferring them, which keeps a pipeline over drifting or deeply nested files stable. From 4.1.1, the INSERT property "enable_push_down_schema" = "true" reshapes the inferred schema to the target table's columns.

  • More Hive layouts are readable. Hive tables stored as OpenX JSON, Hive tables partitioned on FLOAT, DOUBLE or DECIMAL columns, and, from 4.1.1, Avro tables through a native reader that supports avro.schema.literal and avro.schema.url. The native Avro reader is opt-in: set avro_use_jni_reader = false. The ORC writer supports MAP.

  • One cluster can reach several HDFS HA clusters (4.1.1). HA settings such as dfs.nameservices can be declared in a catalog's PROPERTIES, and both FE and BE apply them, so each catalog can point at a different HDFS HA cluster.

  • Iceberg REST catalogs without an STS work with vended credentials. When a catalog returns no per-table credentials, PhoenixAI uses the storage credentials the catalog publishes in its configuration endpoint before falling back to the credentials set on the catalog.

  • Elasticsearch 9 is supported by Elasticsearch catalogs and external tables (4.1.6).

Query execution​

  • Recursive common table expressions

    WITH RECURSIVE lets a query refer to itself, so an organizational chart, a bill of materials or a graph of linked records can be walked in SQL rather than in application code or with a fixed number of self-joins. The anchor member produces the first level, and the recursive member runs again on each level's output until it returns no rows or reaches recursive_cte_max_depth (default 5). Reaching the depth limit raises an error rather than silently truncating the result. The feature is off by default; set enable_recursive_cte = true to use it.

  • ORDER BY <aggregate> LIMIT n gets cheaper

    For a query that ranks groups by an aggregate and keeps the top few, the aggregation now maintains a heap of the current top values and publishes it as a runtime filter, so groups that cannot reach the top N are discarded before they are fully aggregated and shuffled. TopN operators can also spill to disk, so a large ORDER BY ... LIMIT no longer fails for lack of memory when spilling is enabled.

  • Shared expressions in join conditions are computed once

    When a join's ON clause repeats an expensive expression — the same CAST, regexp_extract or JSON path in several conjuncts — it is evaluated once per row and reused. Reuse previously applied to scan and outer-join WHERE predicates; it now also covers the ON predicates of inner hash joins and nested-loop joins.

  • Query Queue v2 shows its work. With enable_extended_explain = true, EXPLAIN reports how many slots a query would request and the state of the queue, so you can see why a query would wait before running it.

  • ETL execution mode does more. SET exec_mode = 'ETL' now enables dependent-event waiting and Query Queue v2 admission alongside spilling and the phased scheduler, and switching back to the default mode restores your global settings rather than forcing spilling and phased scheduling off. The mode is not on by default.

  • The window-function skew hint can name the skewed column and value (4.1.1), so a hot PARTITION BY value is split into its own branch rather than processed by one driver.

  • Shared-data scans spread work evenly across pipeline drivers, so a single large tablet no longer leaves most drivers idle.

SQL and functions​

  • New functions: array_top_n, arrays_zip, json_pretty, json_set, map_entries, sum_map, initcap, current_timezone, sec_to_time, weekday, raise_error and ai_query; uuid_v7 and uuid_v7_numeric (4.1.1) for time-ordered UUIDs; query_id() (4.1.2); and get_query_dump_from_query_id (4.1.1).
  • SQL UDFs (4.1.1): CREATE FUNCTION f(a INT, b INT) RETURNS a + b defines a function as a SQL expression, with no jar to build. Java UDF files can be loaded from S3, Java UDFs accept DECIMAL and variable-length argument lists, and Python UDFs accept nested ARRAY and MAP.
  • Syntax: WHERE, ORDER BY and LIMIT on most SHOW statements; ALTER TASK; ORDER BY ALL; DATE_PART as an alias for EXTRACT; expression column defaults, including ARRAY, MAP and STRUCT literals and defaults for JSON, HLL, BITMAP and VARBINARY columns; REFRESH CONNECTIONS, which pushes changed global variables to open sessions; and CREATE TABLE ... AS WITH ... SELECT in the Trino dialect (4.1.1).
  • The STRUCT_CAST_BY_NAME SQL mode matches STRUCT fields by name when casting, so UNION ALL branches that declare the same fields in different orders can be combined.
  • The default argument of lag and lead can be a column, percentile_approx accepts an array of percentiles, min_by and max_by accept STRUCT inputs, and array_generate works over DATE ranges.

Security and authentication​

  • A query pattern can be blocked by its digest

    The SQL blacklist matches regular expressions against the statement text, which makes it awkward to block one runaway report without catching statements that resemble it, or to keep blocking it when its literals change. From 4.1.0, ADD SQL DIGEST BLACKLIST <digest> (the digest unquoted) blocks every statement with the same SQL digest whatever its constants; SHOW SQL DIGEST BLACKLIST lists the entries and DELETE SQL DIGEST BLACKLIST removes them. Entries are persisted and apply on every FE. Enforcement requires enable_sql_blacklist = true and SQL digest computation (enable_sql_digest = true), and covers queries, INSERT and CTAS.

  • The HTTP APIs can be closed to anonymous callers

    From 4.1.3, enable_http_auth = true on FEs and BEs requires Basic Auth on every external HTTP endpoint and restricts the license and warehouse endpoints to privileged users. Internal traffic and health and metrics probes stay exempt, so monitoring keeps working. It is off by default and requires a restart to change; before enabling it, check scripts that call /api/... anonymously.

  • LDAP and Active Directory logins behave the way the directory does (4.1.6). Groups can be resolved from a user's memberOf attribute, group names are matched without regard to case, and authentication_ldap_case_insensitive = true matches user names case-insensitively as well. ALTER GROUP PROVIDER changes a provider's properties in place, users from an LDAP security integration can authenticate over HTTP, Stream Load and Arrow Flight, and LDAP and group-provider calls have timeouts. (All but ALTER GROUP PROVIDER is also in 4.0.16.)

  • Hardening switches, off by default (4.1.6): authorization_enable_query_profile_access_check = true limits a user to their own query profiles unless they hold SYSTEM OPERATE, and files_require_explicit_credentials = true stops FILES() from falling back to the node's own credentials. (Both also in 4.0.16.)

  • Literal values can be kept out of logs. enable_sql_desensitize_in_log = true replaces literal values in SQL before it reaches the audit log, query details and profiles; query dumps (enable_desensitize_query_dump) and EXPLAIN output (enable_desensitize_explain) have their own switches. Credentials in SUBMIT TASK statements are masked in the task views, and UDF file paths no longer appear in SHOW FUNCTIONS.

  • TLS cipher suites are configurable. ssl_cipher_whitelist and ssl_cipher_blacklist take comma-separated cipher names or regular expressions, so weak suites can be removed from the FE's TLS endpoints.

  • BEs and CNs can authenticate to S3 with an AWS WebIdentity token file (4.1.2), matching the FE.

Observability​

  • FE threads can be inspected with SQL. information_schema.fe_threads lists every thread on every FE with its state and CPU time, so a stuck or CPU-bound FE can be diagnosed with a query before reaching for jstack.
  • Materialized view refresh is observable (4.1.3). information_schema.materialized_view_refresh_jobs shows each refresh as a job, and information_schema.materialized_views reports each view's refresh mode, trigger, warehouse, resource group, last refresh time and why it is not used for rewrite.
  • The audit log says why a statement failed (4.1.6). It records the error message the client received, truncated at audit_log_error_message_max_length (default 1024), and DML into Iceberg, Hive or FILES() and asynchronous materialized view refreshes report real scan and CPU figures. custom_session_name (4.1.1) tags a session in the audit log, and audit_stmt_before_execute (4.1.2, also in 4.0.12) can record a statement when it starts.
  • Internal queries — statistics collection, task runs and materialized view refreshes — appear in SHOW PROC '/current_queries' and can be stopped with KILL QUERY (4.1.2), and load profiles can be written to fe.profile.log (enable_print_load_profile_to_log).
  • The FE web UI can analyze a query profile and show its formatted SQL, and process profiles cover BE nodes as well as FEs. GET /api/v2/cluster_summary returns database and table counts, disk capacity and node counts as JSON.
  • New metrics cover Data Cache usage on /metrics, FE and lake compaction, ALTER TABLE column operations and Thrift server queue timeouts; EXPLAIN COSTS can be included in the query profile (enable_explain_in_profile, 4.1.5).

Improvements​

Six patch releases carry several hundred further improvements. The themes most likely to be visible in your workload:

  • Optimizer and execution

    Scans read predicate columns first, ordered by selectivity, and materialize the remaining columns only for surviving rows. Full sorts compare strings through a prefix-inlined layout, shuffle exchange uses a faster hash function, and join reordering covers conditions built from expressions over both inputs. Skew is detected more precisely and can be forced for GROUP BY with enable_force_group_by_skew_eliminate_when_skewed (4.1.1). UNNEST avoids copying nullable arrays and expands multi-array input incrementally, which removes a common out-of-memory pattern for wide arrays (4.1.6), and Parquet, ORC and CSV reads are bounded by the query memory limit.

  • External-table statistics

    Statistics on external tables are cheaper to collect: PhoenixAI tracks which columns of an Iceberg, Hive or other external table appear in predicates, ANALYZE SAMPLE TABLE ... PREDICATE COLUMNS collects only those (4.1.5), all requested columns are gathered in one scan, and Iceberg collection stops at a configurable byte and file budget. Hive and Hudi row counts account for file compression, and Elasticsearch, Kudu and FILES() sources are estimated from metadata or file sizes. Better estimates mostly show up as better join orders over external data.

  • Iceberg reads

    TopN queries with aggregation push a runtime filter into Iceberg scans (4.1.1), a compound AND predicate is pushed down to file pruning even when only some of its terms translate (4.1.2), and catalog-cache lookups lock per table rather than per catalog, which removes a bottleneck under concurrent queries.

  • Memory

    jemalloc decay can tighten as resident memory nears the memory limit (enable_jemalloc_decay_under_rss_pressure, off by default, 4.1.6), and on arm64 a 4K-page jemalloc variant is chosen at startup with a jemalloc_page_size override (4.1.5). A guard stops collecting global dictionaries that keep being invalidated (enable_dict_thrash_guard, on by default, 4.1.6).

  • Security dependency upgrades

    Across the line: Netty to 4.1.137.Final, Thrift to 0.24.0, transitive Jetty libraries pinned to patched versions, gRPC aligned for CVE-2025-55163, Apache HttpCore 5.4.3 and HttpClient 5.6.4, Apache Ranger 2.9.0, Bouncy Castle 1.85, ZooKeeper 3.8.7 and 3.9.6, the PostgreSQL JDBC driver to 42.7.12, and the Broker's Tomcat to 9.0.121 and commons-lang3 to 3.20.0 for CVE-2025-48924; and removal of vulnerable transitive dependencies, including bcprov-jdk15on, the OkHttp 2.x line and avro-ipc.