BE Configuration - Shared-data, Data Lake, and Others
Some BE configuration items are dynamic parameters which you can set interactively when BE nodes are online. The rest of them are static parameters. You can only set the static parameters of a BE node by changing them in the PhoenixAI Cloud console.
- Sign in to the PhoenixAI Cloud console.
- On the Clusters page, click the classic cluster that you want to configure.
- On the cluster details page, click the Cluster parameters tab.
- In the BE Node static configuration section, click View all parameter list.
- In the dialog box that appears, click New parameter.
- Enter the name of the static parameter you want to configure in the Parameter key field, and the parameter value you want to set in the Value field. For the details of the parameters you can configure, see Usage notes.
- Click the save button next to the Value field to save the record of the parameter change.
- After configuring all the parameters you want to change, click Save changes to save the changes.
- To allow the changes to take effect, you can either manually suspend and resume the cluster whenever you find suitable, or click Apply to all nodes in the BE Node static configuration section to restart the cluster instantly.
Follow these steps to configure Compute Node parameters of a warehouse in the PhoenixAI Cloud console:
- Sign in to the PhoenixAI Cloud console.
- On the Clusters page, click the elastic cluster that you want to configure.
- On the cluster details page, click the Warehouses tab.
- On the Warehouses tab, click the warehouse that you want to configure.
- On the warehouse details page, click the Warehouse parameters tab.
- In the Compute Node static configuration section, click View all parameter list.
- In the dialog box that appears, click New parameter.
- Enter the name of the parameter you want to configure in the Parameter key field, and the parameter value you want to set in the Value field.
note
PhoenixAI provides validation checks on parameter keys and value types. Only valid parameter keys and values can be applied. For the details of the parameters you can configure, see Usage notes.
- Click the save button next to the Value field to save the record of the parameter change.
- After configuring all the parameters you want to change, click Save changes to save the changes.
- To allow the changes to take effect, you can either manually suspend and resume the warehouse whenever you find it suitable, or click Apply to all nodes in the Compute Node static configuration section to restart the warehouse instantly. For
default_warehouse, you can only restart the cluster.
View BE configuration items
You can view the BE configuration items using the following command:
SELECT * FROM information_schema.be_configs [WHERE NAME LIKE "%<name_pattern>%"]
Configure BE parameters
Configure BE dynamic parameters
You can configure a dynamic parameter of a BE node by using the curl command.
curl -XPOST http://be_host:http_port/api/update_config?<configuration_item>=<value>
Configure BE static parameters
You can only set the static parameters of a BE by changing them in the PhoenixAI Cloud console, and restarting the BE to allow the changes to take effect.
This topic introduces the following types of BE configurations:
Shared-data
cloud_native_pk_index_rebuild_files_threshold
- Default: 50
- Type: Int
- Unit: -
- Is mutable: Yes
- Description: The maximum number of segment files that need to be rebuilt in cloud-native Primary Key index. If the number of files that need to be rebuilt during index recovery exceeds this threshold, StarRocks will flush the in-memory MemTable immediately to reduce the number of segments that must be replayed. Set to
0to disable this early-flush strategy. - Introduced in: -
cloud_native_pk_index_rebuild_rows_threshold
- Default: 10000000
- Type: Long
- Unit: Rows
- Is mutable: Yes
- Description: The maximum number of rows that need to be rebuilt in cloud-native Primary Key index. If the number of rows that need to be rebuilt during index recovery exceeds this threshold, StarRocks will flush the in-memory MemTable immediately to reduce the rebuild overhead. Set to
0to disable this early-flush strategy. Works in conjunction withcloud_native_pk_index_rebuild_files_threshold; a flush is triggered if either threshold is exceeded. The row count includes segment rows plus the tombstone (delete) rows recorded in del files, so a delete-heavy workload that produces a few large del files also counts toward this threshold; del files written by older versions that did not record a row count contribute 0. - Introduced in: -
cloud_native_tablet_metadata_ancestors_recorded
- Default: 5
- Type: Int
- Unit: -
- Is mutable: Yes
- Description: Number of historical tablet-metadata ancestor versions retained per tablet; sets the trackable horizon for CHANGES queries. A larger value lets CHANGES reach further-back base versions at the cost of metadata storage.
- Introduced in: -
download_buffer_size
- Default: 4194304
- Type: Int
- Unit: Bytes
- Is mutable: Yes
- Description: Size (in bytes) of the in-memory copy buffer used when downloading snapshot files. SnapshotLoader::download passes this value to fs::copy as the per-transfer chunk size when reading from the remote sequential file into the local writable file. Larger values can improve throughput on high-bandwidth links by reducing syscall/IO overhead; smaller values reduce peak memory use per active transfer. Note: this parameter controls buffer size per stream, not the number of download threads—total memory consumption = download_buffer_size * number_of_concurrent_downloads.
- Introduced in: v3.2.13
graceful_exit_wait_for_frontend_heartbeat
- Default: false
- Type: Boolean
- Unit: -
- Is mutable: Yes
- Description: Determines whether to await at least one frontend heartbeat response indicating SHUTDOWN status before completing graceful exit. When enabled, the graceful shutdown process remains active until a SHUTDOWN confirmation is responded via heartbeat RPC, ensuring the frontend has sufficient time to detect the termination state between two regular heartbeat intervals.
- Introduced in: v3.4.5
lake_compaction_stream_buffer_size_bytes
- Default: 1048576
- Type: Int
- Unit: Bytes
- Is mutable: Yes
- Description: The reader's remote I/O buffer size for cloud-native table compaction in a shared-data cluster. The default value is 1MB. You can increase this value to accelerate compaction process.
- Introduced in: v3.2.3
lake_dump_tablet_metadata_per_request_memory_limit_bytes
- Default: 268435456
- Type: Long
- Unit: Bytes
- Is mutable: Yes
- Description: The tracked memory budget for synchronous allocations made while processing one
/api/cloudnative/dump_tablet_metadatarequest on a CN. It uses the standard MemTracker accounting granularity rather than enforcing a byte-exact ceiling. It covers metadata inspection, sensitive-field redaction, and JSON serialization, but not the metadata already stored in the CN metadata cache or the asynchronous HTTP output buffer. Each request uses the value captured when it is admitted. If this value is less than or equal to0, new requests fail closed. - Introduced in: v26.2
lake_dump_tablet_metadata_per_request_json_size_limit_bytes
- Default: 33554432
- Type: Long
- Unit: Bytes
- Is mutable: Yes
- Description: The maximum size of the complete JSON response for one
/api/cloudnative/dump_tablet_metadatarequest. The size is checked before the response is handed to the HTTP output buffer. Each request uses the value captured when it is admitted. If this value is less than or equal to0, new requests fail closed. - Introduced in: v26.2
lake_dump_tablet_metadata_max_concurrency
- Default: 1
- Type: Int
- Unit: Requests
- Is mutable: Yes
- Description: The maximum number of
/api/cloudnative/dump_tablet_metadatarequests admitted concurrently on one CN. A request continues to occupy one slot until its HTTP request is released. Increasing the value applies immediately to new requests. Decreasing it does not cancel admitted requests, and new requests are rejected until the active count is below the new limit. If this value is less than or equal to0, new requests fail closed. - Introduced in: v26.2
lake_enable_del_file_crc_check
- Default: true
- Type: Boolean
- Unit: -
- Is mutable: Yes
- Description: Whether to verify a Primary Key delete file (
.del) of a shared-data cluster against the CRC32C recorded in its metadata when the file is read back during transaction publish or Primary Key index rebuild. When a mismatch is detected, the operation fails with a corruption error instead of erasing the wrong primary keys. Delete files written before the checksum existed carry none and are always accepted, so this item has no effect on data written by an earlier version. Writing the checksum is not controlled by this item; this item only controls verification. - Introduced in: v4.2
lake_enable_pk_preserve_txn_delete_order
- Default: true
- Type: Boolean
- Unit: -
- Is mutable: Yes
- Description: Whether to preserve the in-transaction upsert/delete order for Primary Key tables in a shared-data cluster. When a single load transaction contains both a
DELETEand a later re-UPSERTof the same key, enabling this makes the re-upsert win (consistent with shared-nothing clusters). It is enabled by default. For downgrade safety, set it tofalsebefore rolling back to (or running a mixed cluster with) a BE version without this fix: when enabled, a load can persist on-disk metadata that a pre-fix BE would misinterpret, potentially producing duplicate primary keys. When disabled, deletes fall back to the legacy behavior (applied after all upserts in the transaction): if a load deletes a key and then upserts it again, and the delete and the upsert are flushed separately (for example, in a large load), the key ends up deleted and the upsert is lost. A Primary Key table whose sort key (ORDER BY) differs from its primary key always preserves the order, whatever this item is set to. - Introduced in: v4.1.4
lake_enable_protobuf_file_checksum
- Default: true
- Type: Boolean
- Unit: -
- Is mutable: Yes
- Description: Whether to write tablet metadata and transaction log files of a shared-data cluster with an Adler-32 checksum, so that corruption of these files can be detected when they are read. Regardless of this item, readers always detect and verify the checksum automatically when it is present; this item only controls the write format. Set it to
falseonly while the cluster may still be downgraded to a version that predates the checksummed format. During a rolling upgrade or a downgrade, an earlier BE or CN uses the legacy reader and cannot parse files written in the new format. - Introduced in: v4.2
lake_enable_publish_partial_meta
- Default: true
- Type: Boolean
- Unit: -
- Is mutable: Yes
- Description: Whether a publish of a shared-data table with file bundling keeps the result of each tablet that the aggregator may not have received. A tablet whose publish finishes more than the publish RPC timeout plus 3 seconds after the request arrived, or whose publish is a retry, keeps its result in the local Data Cache of the node, never in object storage. A later retry on that node reads it back instead of applying the transactions again, so tablets that already finished are not redone on every retry of a slow partition. The entries are evicted like any other Data Cache entry. Takes effect only when Data Cache is enabled. Set the same value on all BE and CN nodes.
- Introduced in: v4.2
lake_enable_txn_vlog_group
-
Default: true
-
Type: Boolean
-
Unit: -
-
Is mutable: Yes
-
Description: Whether schema-change publish in a shared-data cluster writes an ordered statement group into a separate
{tablet}_{version}.group.vlogfile for a multi-statement transaction. Legacy single logs use{tablet}_{version}.vlog. Readers in this version always support both legacy single-log files and statement groups, regardless of this setting. With the defaulttrue, each statement keeps its own schema, segments, rewrite metadata, and delete positions, so a DELETE followed by an INSERT in one transaction no longer causes mismatched partial-update metadata during shadow publish. Setting it tofalserestores the legacy merged-log writer, including that known failure. Changing the setting does not convert versioned transaction logs that have already been written.CNs of versions without group readers cannot read statement groups. When you upgrade from an earlier version in which this setting does not exist or defaults to
false, setlake_enable_txn_vlog_group = falseincn.confbefore you start each upgraded CN, and keep itfalseuntil every CN that may perform publish, replay, or cleanup has been upgraded, including background workers and images used for replacement nodes. Then enable it as described below. If the earlier version cannot read statement groups, this step is required: enabling group writes while older CNs remain can silently lose committed statements. After a group writer deletes some source logs, an older CN can retry using only the remaining sources and write an incomplete legacy log, which replay prefers over the group file.Before changing this setting in either direction, pause the relevant schema-change and multi-statement publish work, drain in-flight requests, update all affected CNs consistently, and then resume the work. The setting does not negotiate reader capabilities or provide an atomic cluster-wide switch.
Setting it to
falseonly stops subsequent group writes; it does not convert, drain, or remove existing grouped logs. Before downgrading to a version without group readers, list every affected storage root'stxnlogdirectory and verify that no*.group.vlogfiles remain, including unused group files next to valid legacy logs: older tablet cleanup also scans these files and cannot decode them. ALTER reachingFINISHED, transactions becomingVISIBLE, or routine vacuum completing does not establish this. With file bundling, successful replay leaves these logs in place, and routine vacuum does not reclaim version logs. If any remain, keep group-capable readers until safe cleanup has been completed. Disabling this setting alone does not make a downgrade safe. -
Introduced in: -
lake_pk_compaction_max_input_rowsets
- Default: 500
- Type: Int
- Unit: -
- Is mutable: Yes
- Description: The maximum number of input rowsets allowed in a Primary Key table compaction task in a shared-data cluster. The default value of this parameter is changed from
5to1000since v3.2.4 and v3.1.10, and to500since v3.3.1 and v3.2.9. After the Sized-tiered Compaction policy is enabled for Primary Key tables (by settingenable_pk_size_tiered_compaction_strategytotrue), StarRocks does not need to limit the number of rowsets for each compaction to reduce write amplification. Therefore, the default value of this parameter is increased. - Introduced in: v3.1.8, v3.2.3
lake_pk_compaction_max_rowsets_per_level
- Default: 20
- Type: Int
- Unit: -
- Is mutable: Yes
- Description: The number of rowsets that still need compaction at which a size level of a Primary Key table in a shared-data cluster is merged even if it does not have the highest score. The Size-tiered Compaction policy merges only the level with the highest score in each round. While loads keep arriving, the level of the newest and smallest rowsets has the highest score in every round, so a level of larger rowsets (outputs of earlier compactions and medium-sized loads) keeps growing. This increases the number of segments that queries read, and shows up as a spike in the compaction score right after a compaction finishes. When a level that is not picked holds this many rowsets that still need compaction, and these rowsets fit in the same compaction task as the picked ones, the level is merged in that compaction task. Set it to
0to disable this behavior. - Introduced in: -
lake_pk_compaction_base_enable_delete_trigger
- Default: false
- Type: Boolean
- Unit: -
- Is mutable: Yes
- Description: Whether automatic compaction of a shared-data Primary Key tablet can select Base Compaction when its delete ratio or delete-row count reaches a threshold. This switch only controls automatic Base Compaction triggered by deletes and does not affect ordinary Cumulative Compaction. It is disabled by default because large tablets under continuous updates or deletes can frequently reach the absolute delete-row threshold and repeatedly rewrite data. Manual
ALTER TABLE ... COMPACTrequests bypass this switch and both delete thresholds; tablets without deleted rows fall back to ordinary compaction selection. Neither automatic nor manual requests guarantee a full-tablet merge in one execution. - Introduced in: v4.1.7, v26.2.1
lake_pk_compaction_base_delete_ratio_threshold
- Default: 0.5
- Type: Double
- Unit: -
- Is mutable: Yes
- Description: One of the two thresholds for automatically selecting Base Compaction by deletes for a shared-data Primary Key tablet. Base Compaction can be selected when the aggregate delete ratio across all current Rowsets,
sum(num_dels) / sum(num_rows), reaches this value, or when the total delete-row count reacheslake_pk_compaction_base_delete_rows_threshold. In v4.1.4–v4.1.6, both thresholds are checked directly. From v4.1.7,lake_pk_compaction_base_enable_delete_trigger = trueis also required. ManualALTER TABLE ... COMPACTrequests bypass both thresholds. Reaching a threshold does not guarantee immediate FE scheduling. - Introduced in: v4.1.4
lake_pk_compaction_base_delete_rows_threshold
- Default: 10000000
- Type: Int
- Unit: -
- Is mutable: Yes
- Description: The absolute delete-row threshold for automatically selecting Base Compaction by deletes for a shared-data Primary Key tablet. It is combined with
lake_pk_compaction_base_delete_ratio_thresholdusing OR. The delete-row count issum(num_dels)across all current Rowsets in the tablet. In v4.1.4–v4.1.6, this threshold is checked directly. From v4.1.7,lake_pk_compaction_base_enable_delete_trigger = trueis also required. Manual requests bypass this threshold. This condition covers tablets with a low aggregate delete ratio but many delete marks in delete vectors. Raise it to make automatic Base Compaction less frequent, or lower it to select delete reclamation sooner. - Introduced in: v4.1.4
enable_lake_pk_compaction_score_gate
- Default: true
- Type: Boolean
- Unit: -
- Is mutable: Yes
- Description: Master switch for the Primary Key size-tiered compaction score gate in a shared-data cluster. When enabled (the default), low-value sparse mid-tier compaction picks are skipped: a picked level whose compaction score is below
lake_pk_compaction_min_level_score, and which trips none of the override conditions, is left uncompacted to avoid rewriting large amounts of base data for a negligible reduction in file count. Set tofalseto turn off the whole gate in one step; every picked level then compacts unconditionally (the behavior before the gate existed), and the threshold parameters below have no effect. - Introduced in: v4.2
lake_pk_compaction_min_level_score
- Default: 2.0
- Type: Double
- Unit: -
- Is mutable: Yes
- Description: Score threshold of the Primary Key compaction score gate (see
enable_lake_pk_compaction_score_gate). When the size-tiered selector picks a level whose total compaction score is below this value and no override condition fires, the compaction is skipped. The per-rowset score isio_count * 1MB / read_bytessummed across the level: levels with many small or overlapped rowsets score high (useful compaction), while a few large non-overlapped rowsets score low (a near-pure base rewrite). Set to0to make the gate a no-op, because a level's score is always greater than or equal to0. - Introduced in: v4.2
lake_pk_compaction_min_benefit_cost_ratio
- Default: 0.005
- Type: Double
- Unit: -
- Is mutable: Yes
- Description: An override of the Primary Key compaction score gate. The minimum benefit/cost ratio (segments saved per MB rewritten) at which a below-threshold level is still compacted. The benefit folds in delete-vector cleanup pressure (see
lake_pk_compaction_delvec_benefit_weight). Set to0to disable this override so that onlylake_pk_compaction_min_level_scoreapplies. - Introduced in: v4.2
lake_pk_compaction_emergency_score
- Default: 50.0
- Type: Double
- Unit: -
- Is mutable: Yes
- Description: An override of the Primary Key compaction score gate. When the tablet-wide read-pressure score (the sum of all rowset scores) reaches this value, compaction proceeds even if the picked level is below
lake_pk_compaction_min_level_score, so that hot tablets with heavy read amplification are not starved. Set to0to disable this override. - Introduced in: v4.2
lake_pk_compaction_delvec_benefit_weight
- Default: 12.0
- Type: Double
- Unit: -
- Is mutable: Yes
- Description: The weight that converts a level's delete pressure into segment-equivalent benefit units in the benefit/cost ratio used by
lake_pk_compaction_min_benefit_cost_ratio, computed asbenefit = real_benefit_segments + delete_ratio * input_segments * weight. A higher value makes compaction more aggressive at cleaning up delete vectors; a lower value favors reducing write amplification. Set to0to drop the delete contribution so that the ratio reflects segment-count benefit only. - Introduced in: v4.2
lake_pk_compaction_size_overflow_ratio
- Default: 2.0
- Type: Double
- Unit: -
- Is mutable: Yes
- Description: An override of the Primary Key compaction score gate. When a below-threshold level's total bytes exceed
ratio * largest_rowset_bytes * size_tiered_level_multiple(that is,ratiotimes the natural next-tier promotion target), compaction is forced to bound long-tail mid-tier accumulation. The default2.0tolerates twice the natural promotion threshold before forcing a merge. Set to0to disable this override, so that there is no size cap. - Introduced in: v4.2
enable_lake_prepared_split_pre_refinement
- Default: true
- Type: Boolean
- Unit: -
- Is mutable: Yes
- Description: Whether a Prepared Physical Split Lake scan (see the session variable
enable_lake_prepared_physical_split_scan) issues an extra coarse-range morsel over an un-pruned segment range while the seed page-pruning is still running, so otherwise-idle drivers stay busy until the refined ranges land. Disabling it never drops data (the coarse range is always a superset that the refined ranges subtract from); it only trades early parallelism for less redundant coarse scanning. - Introduced in: v4.2
lake_prepared_split_max_splitted_scan_rows
- Default: 262144
- Type: Int
- Unit: Rows
- Is mutable: Yes
- Description: The upper bound on
splitted_scan_rows(the number of rows scanned per split morsel) applied only when the Prepared Physical Split Lake Scan is enabled (see the session variableenable_lake_prepared_physical_split_scan). The effective bound ismin(tablet_internal_parallel_max_splitted_scan_rows, this), so it can only make split morsels finer -- cutting a large tablet into more sub-range morsels that fill otherwise-idle drivers -- never coarser. Takes effect only in a shared-data cluster. - Introduced in: v4.2
lake_put_txn_log_timeout_guard_ms
- Default: -1
- Type: Int64
- Unit: Milliseconds
- Is mutable: Yes
- Description: Timeout guard for writing a transaction log to object storage in a shared-data cluster (the
put_txn_logandput_combined_txn_logpaths). If writing a transaction log takes longer than this value, StarRocks dumps the stack trace of the slow thread to the BE log to help diagnose slow object-storage writes. Disabled by default (a value less than or equal to0disables the guard); set it to a positive value such as4000(4 seconds) to enable. - Introduced in: -
lake_rows_mapper_read_parallelism
- Default: 32
- Type: Int
- Unit: sub-chunks
- Is mutable: Yes
- Description: Maximum number of in-flight
.lcrm(lake compaction rows-mapper) sub-chunk reads kept byRowsMapperIteratorduring light Primary Key compaction publish in a shared-data cluster. Each sub-chunk islake_rows_mapper_sub_chunk_bytesin size and never crosses a segment boundary; the iterator submits up to this many reads to the PK index execution thread pool and pipelines them against the caller's per-segment processing. Memory bound islake_rows_mapper_read_parallelism * lake_rows_mapper_sub_chunk_bytes. Set to1to disable pipelining and fall back to sequential reads.
lake_rows_mapper_sub_chunk_bytes
- Default: 4194304
- Type: Int
- Unit: Bytes
- Is mutable: Yes
- Description: Sub-chunk granularity for
RowsMapperIteratorpipelined reads of.lcrmfiles during light Primary Key compaction publish in a shared-data cluster. Each output segment is split intoceil(segment_bytes / lake_rows_mapper_sub_chunk_bytes)sub-chunks pipelined independently. Smaller values raise the achievable parallelism for few-but-large output segments at the cost of more range reads and an extra memcpy on consume. Defaults to 4 MiB to align with the starcache disk-tier block size.
lake_segment_metadata_cache_limit_percent
- Default: 10
- Type: Int
- Unit: -
- Is mutable: Yes
- Description: The capacity of the segment metadata cache in a shared-data cluster, as a percentage of the BE process memory limit. This cache holds opened segments (their footers, indexes, and column readers), delete vectors, and tablet schemas. It is separate from the tablet metadata cache (see
lake_tablet_metadata_cache_limit_percent), so loading many segments does not evict tablet metadata or transaction logs. Values outside the range 0–100 are treated as 0 or 100. Set this item to0to disable the cache. Together withlake_tablet_metadata_cache_limit_percent, this item replaceslake_metadata_cache_limit. This value is the largest capacity of the cache: the cache shrinks below it when the BE process runs short of memory (seelake_tablet_metadata_cache_min_limit). - Introduced in: -
lake_segment_metadata_cache_min_limit
- Default: 1073741824
- Type: Long
- Unit: Bytes
- Is mutable: Yes
- Description: The capacity below which the segment metadata cache in a shared-data cluster does not shrink when it gives memory back under memory pressure (see
lake_tablet_metadata_cache_min_limit). If this value is larger than the capacity set bylake_segment_metadata_cache_limit_percent, the cache stays at that capacity. - Introduced in: -
lake_shard_info_cache_ttl_sec
- Default: 60
- Type: Int
- Unit: Seconds
- Is mutable: Yes
- Description: How long a BE or CN in a shared-data cluster keeps the storage location of a tablet it does not own, so that touching the tablet again within this time does not request the location from StarMgr. A node keeps two kinds of locations. The first kind is what it requested from StarMgr, counted in
starrocks_be_staros_shard_info_fallback_total, when it read or wrote the files of a tablet assigned to another node, for example when several nodes write the same tablet in one load, or when a vacuum reads the metadata of every tablet in a partition; lookups answered by them are counted instaros_shard_info_fallback_cache_hits_total. The second kind is what the nodes that own the tablets hand over to the node that coordinates a load into a table withfile_bundlingenabled, together with their transaction logs, so that the coordinating node writes the combined transaction log of each partition without requesting the location; lookups answered by them are counted inlake_tablet_location_handoff_hits_total. The node that owns a tablet receives storage volume updates pushed by StarMgr, while a kept location picks them up only after it expires, so this item also bounds how long such a node can keep using outdated storage volume settings, such as rotated credentials. The kept locations include the credentials of the storage volume.0disables both kinds: nothing is handed over or kept, and every such lookup requests the location from StarMgr. - Introduced in: -
lake_tablet_metadata_cache_limit_percent
- Default: 5
- Type: Int
- Unit: -
- Is mutable: Yes
- Description: The capacity of the tablet metadata cache in a shared-data cluster, as a percentage of the BE process memory limit. This cache holds tablet metadata and transaction logs. It is separate from the segment metadata cache (see
lake_segment_metadata_cache_limit_percent), so loading many segments does not evict them. Values outside the range 0–100 are treated as 0 or 100. Set this item to0to disable the cache. Together withlake_segment_metadata_cache_limit_percent, this item replaceslake_metadata_cache_limit. This value is the largest capacity of the cache: the cache shrinks below it when the BE process runs short of memory (seelake_tablet_metadata_cache_min_limit). - Introduced in: -
lake_tablet_metadata_cache_min_limit
- Default: 1073741824
- Type: Long
- Unit: Bytes
- Is mutable: Yes
- Description: The capacity below which the tablet metadata cache in a shared-data cluster does not shrink. When the BE process memory usage exceeds
memory_high_level, the tablet metadata cache and the segment metadata cache lower their capacities below their current usage and evict their least recently used entries. The memory to give back is shared between the two caches in proportion to how far each cache's usage is above its floor (this item andlake_segment_metadata_cache_min_limit). The usage abovememory_high_levelis released gradually over about 50 seconds. If the usage also exceedsmemory_urgent_level, the part abovememory_urgent_levelis released at once. After the usage drops belowmemory_high_level, the capacities grow back over about 200 seconds to the values set bylake_tablet_metadata_cache_limit_percentandlake_segment_metadata_cache_limit_percent. The check runs every 10 seconds. If this value is larger than the capacity set bylake_tablet_metadata_cache_limit_percent, the cache stays at that capacity. - Introduced in: -
lake_vacuum_enable_task_timeout
- Default: true
- Type: Boolean
- Unit: -
- Is mutable: Yes
- Description: Whether Vacuum tasks honor the timeout carried in the request (
VacuumRequest.timeout_ms) and abort themselves once it elapses. Set this item tofalseto let Vacuum tasks always run to completion no matter how long the FE caller waits.
lake_vacuum_min_batch_delete_size
- Default: 200
- Type: Int64
- Unit: Number of files
- Is mutable: Yes
- Description: The number of stale files Vacuum batches into a single
DeleteObjectsrequest on a shared-data cluster. A larger batch amortizes per-call HTTP / auth / signing overhead and reduces the prefix-level request rate against the object store, at the cost of higher single-call latency and a larger replay cost when a transient error retries. Users running on AWS S3 are encouraged to raise this further (up to the protocol cap of 1000) where per-request server time is nearly batch-size insensitive.
loop_count_wait_fragments_finish
- Default: 2
- Type: Int
- Unit: -
- Is mutable: Yes
- Description: The number of loops to be waited when the BE/CN process exits. Each loop is a fixed interval of 10 seconds. You can set it to
0to disable the loop wait. From v3.4 onwards, this item is changed to mutable and its default value is changed from0to2. - Introduced in: v2.5
max_client_cache_size_per_host
- Default: 10
- Type: Int
- Unit: entries (cached client instances) per host
- Is mutable: No
- Description: The maximum number of cached client instances retained for each remote host by BE-wide client caches. This single setting is used when creating BackendServiceClientCache, FrontendServiceClientCache, and BrokerServiceClientCache during ExecEnv initialization, so it limits the number of client stubs/connections kept per host across those caches. Raising this value reduces reconnects and stub creation overhead at the cost of increased memory and file-descriptor usage; lowering it saves resources but may increase connection churn. The value is read at startup and cannot be changed at runtime. Currently one shared setting controls all client cache types; separate per-cache configuration may be introduced later.
- Introduced in: v3.2.0
primary_key_index_memory_high_percent
- Default: 75
- Type: Int
- Unit: -
- Is mutable: Yes
- Description: The level to which a background thread periodically shrinks the primary key index cache in a shared-data cluster, expressed as a percentage of the cache capacity. The cache capacity equals the memory limit for updates (see
update_memory_limit_percent). When the cache holds more than this percentage, the least recently used primary key indexes that are not in use are evicted until the cache is within this percentage. Indexes in use are never evicted. Set it to a value lower thanprimary_key_index_memory_urgent_percent. - Introduced in: -
primary_key_index_memory_middle_percent
- Default: 50
- Type: Int
- Unit: -
- Is mutable: Yes
- Description: The level, expressed as a percentage of the cache capacity, to which the primary key index cache is shrunk when a publish of a Primary Key table in a shared-data cluster fails because it exceeds the memory limit. The primary key indexes that are not in use are evicted until the cache is within this percentage, which frees memory for the retried publish. Set it to a value lower than
primary_key_index_memory_high_percent. - Introduced in: -
primary_key_index_memory_urgent_percent
- Default: 85
- Type: Int
- Unit: -
- Is mutable: Yes
- Description: The upper limit of the primary key index cache in a shared-data cluster, expressed as a percentage of the cache capacity. It is checked after every publish that does not fail because it exceeds the memory limit: when the cache holds more than this percentage, the least recently used primary key indexes that are not in use are evicted until the cache is within this percentage. It is a backstop for when the cache grows faster than the background thread brings it down to
primary_key_index_memory_high_percent, so set it to a value higher than that item. - Introduced in: -
starlet_filesystem_instance_cache_capacity
- Default: 10000
- Type: Int
- Unit: -
- Is mutable: Yes
- Description: The cache capacity of starlet filesystem instances.
- Introduced in: v3.2.16, v3.3.11, v3.4.1
starlet_filesystem_instance_cache_ttl_sec
- Default: 86400
- Type: Int
- Unit: Seconds
- Is mutable: Yes
- Description: The cache expiration time of starlet filesystem instances.
- Introduced in: v3.3.15, 3.4.5
starlet_fslib_azure_storage_max_single_part_size
- Default: 104857600
- Type: Int64
- Unit: Bytes
- Is mutable: Yes
- Description: In a shared-data cluster, the object size above which files written to Azure Blob Storage or ADLS Gen2 use a multipart upload instead of a single upload request. Both Azure filesystems share this item. BE buffers up to this many bytes in memory per output stream before switching to multipart, so raising it raises memory usage, multiplied by the number of concurrent output streams. Raising it high enough to avoid multipart entirely can also fail uploads outright if it exceeds the object store's single-request upload limit; the upload then fails with the backend's own error. Changing it takes effect on uploads already in progress. The value must be greater than 0; a non-positive value is rejected. A valid value here overrides an equivalent
--fslib_*gflag passed on the BE command line. A value rejected at BE startup is not applied at all: BE keeps the previously effective value and logs a warning, so this item can report a value that is not the one in effect. - Introduced in: v4.1.5, v4.2
starlet_fslib_azure_storage_min_upload_part_size
- Default: 5242880
- Type: Int64
- Unit: Bytes
- Is mutable: Yes
- Description: In a shared-data cluster, the block size used for multipart uploads to Azure Blob Storage or ADLS Gen2. Both Azure filesystems share this item. BE buffers up to this many bytes in memory per output stream between block uploads, so raising it raises memory usage, multiplied by the number of concurrent output streams. Setting it very low increases request count and can exceed the block limits of the storage service. Changing it takes effect on uploads already in progress. The value must be greater than 0; a non-positive value is rejected. A valid value here overrides an equivalent
--fslib_*gflag passed on the BE command line. A value rejected at BE startup is not applied at all: BE keeps the previously effective value and logs a warning, so this item can report a value that is not the one in effect. - Introduced in: v4.1.5, v4.2
starlet_fslib_gcs_max_single_part_size
- Default: 104857600
- Type: Int64
- Unit: Bytes
- Is mutable: Yes
- Description: In a shared-data cluster, the object size above which files written to Google Cloud Storage use a streaming upload instead of a single insert request. BE buffers up to this many bytes in memory per output stream before switching to streaming, so raising it raises memory usage, multiplied by the number of concurrent output streams. Raising it high enough to avoid streaming entirely can also fail uploads outright if it exceeds the object store's single-request upload limit; the upload then fails with the backend's own error. Changing it takes effect on uploads already in progress. The value must be greater than 0; a non-positive value is rejected. A valid value here overrides an equivalent
--fslib_*gflag passed on the BE command line. A value rejected at BE startup is not applied at all: BE keeps the previously effective value and logs a warning, so this item can report a value that is not the one in effect. - Introduced in: v4.1.5, v4.2
starlet_fslib_s3_max_single_part_size
- Default: 104857600
- Type: Int64
- Unit: Bytes
- Is mutable: Yes
- Description: In a shared-data cluster, the object size above which files written to S3 or an S3-compatible object store use a multipart upload instead of a single
PutObjectrequest. BE buffers up to this many bytes in memory per output stream before switching to multipart, so raising it raises memory usage, multiplied by the number of concurrent output streams. Raising it too high to avoid multipart entirely can also fail uploads outright: AWS S3 rejects a singlePutObjectlarger than 5 GiB, and the limits of S3-compatible object stores differ. Changing it takes effect on uploads already in progress. The value must be greater than 0; a non-positive value is rejected. A valid value here overrides an equivalent--fslib_*gflag passed on the BE command line. A value rejected at BE startup is not applied at all: BE keeps the previously effective value and logs a warning, so this item can report a value that is not the one in effect. This item controls starlet uploads in a shared-data cluster and is unrelated to theexperimental_s3_*items. - Introduced in: v4.1.5, v4.2
starlet_fslib_s3_min_upload_part_size
- Default: 5242880
- Type: Int64
- Unit: Bytes
- Is mutable: Yes
- Description: In a shared-data cluster, the part size used for multipart uploads to S3 or an S3-compatible object store. BE buffers up to this many bytes in memory per output stream between part uploads, so raising it raises memory usage, multiplied by the number of concurrent output streams. Setting it very low increases request count and can exceed the 10,000-part limit or fall below the 5 MB minimum part size that AWS S3 enforces; the limits of other S3-compatible stores differ. Changing it takes effect on uploads already in progress. The value must be greater than 0; a non-positive value is rejected. A valid value here overrides an equivalent
--fslib_*gflag passed on the BE command line. A value rejected at BE startup is not applied at all: BE keeps the previously effective value and logs a warning, so this item can report a value that is not the one in effect. This item controls starlet uploads in a shared-data cluster and is unrelated to theexperimental_s3_*items. - Introduced in: v4.1.5, v4.2
starlet_port
- Default: 9070
- Type: Int
- Unit: -
- Is mutable: No
- Description: An extra agent service port for BE and CN.
- Introduced in: -
starlet_star_cache_disk_size_percent
- Default: 80
- Type: Int
- Unit: -
- Is mutable: No
- Description: The percentage of disk capacity that Data Cache can use at most in a shared-data cluster. Only takes effect when
datacache_unified_instance_enableisfalse. - Introduced in: v3.1
starlet_star_cache_skip_fresh_block_checksum_verification
- Default: true
- Type: Boolean
- Unit: -
- Is mutable: No
- Description: In a shared-data cluster, whether a Data Cache disk block that this CN process wrote and has not evicted since skips the first full-block checksum read when a later request touches it. That verification re-reads the whole block from disk to check data that the process has just written and still tracks in memory, which costs one extra disk read on the first access to every freshly cached block. Set this item to
falseto verify every block on first access, which is only worth its cost where the cache disk is suspected of corrupting data at rest. Blocks inherited from a previous run of the process are always verified regardless of this item, because nothing in the current process witnessed their write. This item is read once when Data Cache initializes. - Introduced in: v4.2.0
starlet_star_cache_write_overwrite
- Default: false
- Type: Boolean
- Unit: -
- Is mutable: Yes
- Description: In a shared-data cluster, whether a Data Cache write replaces an object that is already cached under the same key. When this item is
false, the existing object is kept and the write is reported as successful, which suits the read-through fill path, where the bytes being written are the ones already cached. Set this item totrueto let a re-read refresh a block that is suspected to be stale or damaged. Data Cache reads this item on every write, so a change takes effect on the next write without re-initializing the cache. - Introduced in: -
starlet_starmgr_client_compression_type
- Default: none
- Type: String
- Unit: -
- Is mutable: Yes
- Description: In a shared-data cluster, the compression applied to the worker heartbeat that this CN sends to StarMgr. The heartbeat carries one entry per tablet on the CN, which makes it the largest recurring request in the cluster. Valid values:
noneandzstd.zstdcompresses the payload on the heartbeat thread, within the same RPC timeout budget, and the FE decompresses it.nonesends the heartbeat uncompressed. The FE must be able to decompress the payload before any CN is switched, because an FE that predates this mechanism discards the compressed heartbeat, so upgrade the FEs first and only then set this item on the CNs. An unrecognized value is reported and falls back tononeinstead of preventing the CN from starting. - Introduced in: v4.2.0
starlet_starmgr_client_rpc_timeout_ms
- Default: 5000
- Type: Int
- Unit: Milliseconds
- Is mutable: Yes
- Description: In a shared-data cluster, the timeout of each RPC that this CN sends to StarMgr, including GetShard, the worker heartbeat, and the tablet replica information update. The CN sends GetShard when it accesses a tablet that it does not hold locally, for example when a single CN vacuums all tablets of a partition in a table with file bundling enabled. The value must be greater than
0. A dynamic update to a non-positive value is rejected. At startup, a non-positive value is not applied and a warning is logged, and the default value of 5000 remains in effect. - Introduced in: v4.2.0
starlet_use_star_cache
- Default: true
- Type: Boolean
- Unit: -
- Is mutable: Yes
- Description: Whether to enable Data Cache in a shared-data cluster.
trueindicates enabling this feature andfalseindicates disabling it. The default value is set fromfalsetotruefrom v3.2.3 onwards. - Introduced in: v3.1
starlet_write_file_with_tag
- Default: false
- Type: Boolean
- Unit: -
- Is mutable: Yes
- Description: In a shared-data cluster, whether to tag files written to object storage with object storage tags for convenient custom file management.
- Introduced in: v3.5.3
table_schema_service_max_retries
- Default: 3
- Type: Int
- Unit: -
- Is mutable: Yes
- Description: The maximum number of retries for Table Schema Service requests.
- Introduced in: v4.1
Data Lake
datacache_block_buffer_enable
- Default: true
- Type: Boolean
- Unit: -
- Is mutable: No
- Description: Whether to enable Block Buffer to optimize Data Cache efficiency. When Block Buffer is enabled, the system reads the Block data from the Data Cache and caches it in a temporary buffer, thus reducing the extra overhead caused by frequent cache reads.
- Introduced in: v3.2.0
datacache_disk_adjust_interval_seconds
- Default: 10
- Type: Int
- Unit: Seconds
- Is mutable: Yes
- Description: The interval of Data Cache automatic capacity scaling. At regular intervals, the system checks the cache disk usage, and triggers Automatic Scaling when necessary.
- Introduced in: v3.3.0
datacache_disk_idle_seconds_for_expansion
- Default: 180
- Type: Int
- Unit: Seconds
- Is mutable: Yes
- Description: The minimum wait time for Data Cache automatic expansion. Automatic scaling up is triggered only if the disk usage remains below
datacache_disk_low_levelfor longer than this duration. - Introduced in: v3.3.0
datacache_disk_size
- Default: 100%
- Type: String
- Unit: -
- Is mutable: Yes
- Description: The maximum amount of data that can be cached on a single disk. You can set it as a percentage (for example,
80%) or a physical limit (for example,2T,500G). For example, if you use two disks and set the value of thedatacache_disk_sizeparameter as21474836480(20 GB), a maximum of 40 GB data can be cached on these two disks. A percentage is relative to the capacity of the disk that holds eachstorage_root_path. The default value is100%. Whenenable_datacache_disk_auto_adjustistrue(default), the system adjusts the actual disk cache quota based on disk usage, and this item only sets the upper limit of that quota.0disables the disk cache. - Introduced in: -
datacache_enable
- Default: true
- Type: Boolean
- Unit: -
- Is mutable: No
- Description: Whether to enable Data Cache.
trueindicates Data Cache is enabled, andfalseindicates Data Cache is disabled. The default value is changed totruefrom v3.3. - Introduced in: -
datacache_eviction_policy
- Default: slru
- Type: String
- Unit: -
- Is mutable: No
- Description: The eviction policy of Data Cache. Valid values:
lru(least recently used) andslru(Segmented LRU). - Introduced in: v3.4.0
datacache_inline_item_count_limit
- Default: 130172
- Type: Int
- Unit: -
- Is mutable: Yes
- Description: The maximum number of inline cache items in Data Cache. For some particularly small cache blocks, Data Cache stores them in
inlinemode, which caches the block data and metadata together in memory. - Introduced in: v3.4.0
datacache_lru_shard_evict_fallback_enable
- Default: true
- Type: Boolean
- Unit: -
- Is mutable: Yes
- Description: Whether an LRU shard of Data Cache can evict entries from the other shards when it is being refilled or cannot supply the whole charge by itself.
falserestores the behavior that a shard only evicts its own entries, anddatacache_lru_shard_refill_low_watermark_percentanddatacache_lru_shard_refill_high_watermark_percentare ignored. - Introduced in: -
datacache_lru_shard_refill_low_watermark_percent
- Default: 50
- Type: Int
- Unit: -
- Is mutable: Yes
- Description: An LRU shard of Data Cache is refilled from the largest shards instead of evicting its own entries once its usage drops below this percentage of its capacity, until it reaches
datacache_lru_shard_refill_high_watermark_percent. The value must not exceeddatacache_lru_shard_refill_high_watermark_percent. Setting both parameters to0disables the refill. - Introduced in: -
datacache_lru_shard_refill_high_watermark_percent
- Default: 70
- Type: Int
- Unit: -
- Is mutable: Yes
- Description: The percentage of the shard capacity at which the refill of an LRU shard of Data Cache stops. Valid values:
datacache_lru_shard_refill_low_watermark_percentto100. Setting both parameters to0disables the refill. - Introduced in: -
datacache_mem_size
- Default: 20%
- Type: String
- Unit: -
- Is mutable: Yes
- Description: The maximum amount of data that can be cached in memory. You can set it as a percentage (for example,
10%) or a physical limit (for example,10G,21474836480). - Introduced in: -
datacache_min_disk_quota_for_adjustment
- Default: 10737418240
- Type: Int
- Unit: Bytes
- Is mutable: Yes
- Description: The minimum effective capacity for Data Cache Automatic Scaling. If the system tries to adjust the cache capacity to less than this value, the cache capacity will be directly set to
0to prevent suboptimal performance caused by frequent cache fills and evictions due to insufficient cache capacity. - Introduced in: v3.3.0
disk_high_level
- Default: 90
- Type: Int
- Unit: -
- Is mutable: Yes
- Description: The upper limit of disk usage (in percentage) that triggers the automatic scaling up of the cache capacity. When the disk usage exceeds this value, the system automatically evicts cache data from the Data Cache. From v3.4.0 onwards, the default value is changed from
80to90. This item is renamed fromdatacache_disk_high_leveltodisk_high_levelfrom v4.0 onwards. - Introduced in: v3.3.0
disk_low_level
- Default: 60
- Type: Int
- Unit: -
- Is mutable: Yes
- Description: The lower limit of disk usage (in percentage) that triggers the automatic scaling down of the cache capacity. When the disk usage remains below this value for the period specified in
datacache_disk_idle_seconds_for_expansion, and the space allocated for Data Cache is fully utilized, the system will automatically expand the cache capacity by increasing the upper limit. This item is renamed fromdatacache_disk_low_leveltodisk_low_levelfrom v4.0 onwards. - Introduced in: v3.3.0
disk_safe_level
- Default: 80
- Type: Int
- Unit: -
- Is mutable: Yes
- Description: The safe level of disk usage (in percentage) for Data Cache. When Data Cache performs automatic scaling, the system adjusts the cache capacity with the goal of maintaining disk usage as close to this value as possible. From v3.4.0 onwards, the default value is changed from
70to80. This item is renamed fromdatacache_disk_safe_leveltodisk_safe_levelfrom v4.0 onwards. - Introduced in: v3.3.0
enable_connector_sink_spill
- Default: true
- Type: Boolean
- Unit: -
- Is mutable: Yes
- Description: Whether to enable Spilling for writes to external tables. Enabling this feature prevents the generation of a large number of small files as a result of writing to an external table when memory is insufficient. Currently, this feature only supports writing to Iceberg tables.
- Introduced in: v4.0.0
enable_datacache_disk_auto_adjust
- Default: true
- Type: Boolean
- Unit: -
- Is mutable: Yes
- Description: Whether to enable Automatic Scaling for Data Cache disk capacity. When it is enabled, the system dynamically adjusts the cache capacity based on the current disk usage rate. This item is renamed from
datacache_auto_adjust_enabletoenable_datacache_disk_auto_adjustfrom v4.0 onwards. - Introduced in: v3.3.0
datacache_unified_instance_enable
- Default: true
- Type: Boolean
- Unit: -
- Is mutable: No
- Description: Whether to use a unified Data Cache instance to manage data caching for both internal catalog and external catalog in a shared-data cluster.
- Introduced in: v3.4.0
external_table_clear_corrupted_cache_data
- Default: true
- Type: Boolean
- Unit: -
- Is mutable: Yes
- Description: Whether to allow the system to clear the corrupted data cache of external table files (for example, Hive and Iceberg tables). When a scan fails to decode a file whose content was read from the data cache, the system drops the cached blocks it read, so that the next read of the file recovers from the remote storage. For Parquet files, if no row has been returned yet, the system first reads the file again bypassing the data cache, and drops the cached blocks only if this read succeeds. This item does not affect the internal tables of shared-data clusters, which are controlled by
lake_clear_corrupted_cache_data. - Introduced in: -
jdbc_connection_idle_timeout_ms
- Default: 600000
- Type: Int
- Unit: Milliseconds
- Is mutable: No
- Description: The length of time after which an idle connection in the JDBC connection pool expires. If the connection idle time in the JDBC connection pool exceeds this value, the connection pool closes idle connections beyond the number specified in the configuration item
jdbc_minimum_idle_connections. If no query uses a connection pool for this length of time, the BE closes the whole connection pool, including the connections kept byjdbc_minimum_idle_connections, and creates a new one on the next query. This releases the connection pools of dropped or altered JDBC catalogs and resources. - Introduced in: -
jdbc_connection_pool_size
- Default: 8
- Type: Int
- Unit: -
- Is mutable: No
- Description: The JDBC connection pool size. On each BE node, queries that access the external table with the same
jdbc_urlshare the same connection pool. - Introduced in: -
jdbc_minimum_idle_connections
- Default: 1
- Type: Int
- Unit: -
- Is mutable: No
- Description: The minimum number of idle connections in the JDBC connection pool.
- Introduced in: -
jdbc_connection_max_lifetime_ms
- Default: 300000
- Type: Long
- Unit: Milliseconds
- Is mutable: No
- Description: Maximum lifetime of a connection in the JDBC connection pool. Connections are recycled before this timeout to prevent stale connections. Minimum allowed value is 30000 (30 seconds).
- Introduced in: -
jdbc_connection_keepalive_time_ms
- Default: 30000
- Type: Long
- Unit: Milliseconds
- Is mutable: No
- Description: Keepalive interval for idle JDBC connections. Idle connections are tested at this interval to detect stale connections proactively. Set to 0 to disable keepalive probing. When enabled, must be >= 30000 and less than
jdbc_connection_max_lifetime_ms. Invalid enabled values are silently disabled (reset to 0). - Introduced in: -
lake_clear_corrupted_cache_data
- Default: true
- Type: Boolean
- Unit: -
- Is mutable: Yes
- Description: Whether to allow the system to clear the corrupted data cache in a shared-data cluster.
- Introduced in: v3.4
lake_dump_corrupted_cache
- Default: true
- Type: Boolean
- Unit: -
- Is mutable: Yes
- Description: Whether to dump the locally cached blocks of a file before an automatic corruption-recovery path drops them in a shared-data cluster. The dump preserves the exact bytes the local cache was serving, so the corruption can be analyzed offline while the drop still lets the read recover from remote storage. Blocks are read from the local cache only, so a dump never contains bytes fetched from remote storage and never changes the cache. A dump that cannot be taken is logged and never blocks the drop. Only meaningful together with
lake_clear_corrupted_cache_metaandlake_clear_corrupted_cache_data: nothing is preserved when nothing is dropped. Dumps are written tolake_cache_dump_dirand bounded bylake_cache_dump_interval_seconds,lake_cache_dump_max_bytes_per_event, andlake_cache_dump_max_dumps. - Introduced in: v4.2.0
lake_cache_dump_dir
- Default: none
- Type: String
- Unit: -
- Is mutable: Yes
- Description: The directory that holds local cache dumps. An empty value means
<the first path in storage_root_path>/cache_dump, because a dump is data and belongs on a storage disk rather than on the log disk; one dump root is used even when several storage paths are configured. Each dump creates one<file name>.<unix milliseconds>subdirectory holdingmanifest.json, one<block id>.blkper cached block, and the object's copy of the same byte range under the file's own name. Only the directory this item resolves to is garbage collected: after you change it, the dumps under the previous root stay on disk, and adump_dirpassed to the/api/cloudnative/cacheAPI is never swept. - Introduced in: v4.2.0
lake_cache_dump_interval_seconds
- Default: 600
- Type: Int
- Unit: Seconds
- Is mutable: Yes
- Description: The minimum interval between two automatic cache dumps. The limit is process-wide and independent of which file failed, because a failing disk makes every file a first offender. Dumps requested through the
/api/cloudnative/cacheAPI are not subject to it. - Introduced in: v4.2.0
lake_cache_dump_max_bytes_per_event
- Default: 134217728
- Type: Long
- Unit: Bytes
- Is mutable: Yes
- Description: The upper bound on everything a single automatic cache dump writes. A detection site that knows which page failed dumps only the cache blocks covering that page, usually one block, so this bound only clips whole-file dumps, which are then marked
"truncated": truein the manifest. Dumps requested through the/api/cloudnative/cacheAPI are not clipped. A dump saves the range twice -- once as the cache served it, once as the object holds it -- so the range it covers is half this value. - Introduced in: v4.2.0
lake_cache_dump_max_dumps
- Default: 50
- Type: Int
- Unit: -
- Is mutable: Yes
- Description: How many cache dumps are kept under
lake_cache_dump_dir. Before a dump is taken, the oldest dumps are removed until there is room for it, so the limit holds the most recent dumps rather than the first ones to arrive: one file that keeps failing must not make the next corruption invisible. A count rather than a byte budget: what an operator needs is the last N corruption events, and the disk footprint of the automatic dumps is already bounded bylake_cache_dump_max_bytes_per_event. Dumps written to an operator-supplieddump_dirthrough the API are never swept, so a dump worth keeping can be put out of reach of this limit. Worst case on disk is this count timeslake_cache_dump_max_bytes_per_event. - Introduced in: v4.2.0
lake_cache_dump_retention_seconds
- Default: 172800
- Type: Int
- Unit: Seconds
- Is mutable: Yes
- Description: How long a cache dump is kept before it is removed. Expired dumps are swept when a new dump is taken.
- Introduced in: v4.2.0
lake_pk_index_sst_verify_checksum
- Default: true
- Type: Boolean
- Unit: -
- Is mutable: Yes
- Description: Whether to verify sstable block checksums when reading the cloud-native Primary Key index (point lookups and compaction merges) in a shared-data cluster. When enabled, corrupted bytes (usually from a corrupted local cache copy) deterministically fail as a Corruption error and trigger the corrupted-cache cleanup, instead of being misparsed or silently returning wrong index values. Note that the automatic cleanup of the corrupted cache additionally requires
lake_clear_corrupted_cache_datato be enabled. - Introduced in: v4.1
lake_clear_corrupted_cache_meta
- Default: true
- Type: Boolean
- Unit: -
- Is mutable: Yes
- Description: Whether to allow the system to clear the corrupted metadata cache in a shared-data cluster. This covers the local cache of tablet metadata files and transaction log files. When a read of such a file fails with a corruption error, the system drops the cached copy and reads the file again from remote storage once.
- Introduced in: v3.3
lake_enable_horizontal_compaction_fill_data_cache
- Default: false
- Type: Boolean
- Unit: -
- Is mutable: Yes
- Description: Whether to allow horizontal compaction tasks to cache the input data they read on local disks in a shared-data cluster. Horizontal compaction reads each input segment exactly once and its input rowsets are replaced right after the task finishes, so caching that data mainly evicts query-hot data from the local cache. Set this item to
trueto restore the previous behavior of always caching. - Introduced in: v4.2
lake_enable_vertical_compaction_fill_data_cache
- Default: true
- Type: Boolean
- Unit: -
- Is mutable: Yes
- Description: Whether to allow vertical compaction tasks to cache data on local disks in a shared-data cluster.
- Introduced in: v3.1.7, v3.2.3
lake_replication_read_buffer_size
- Default: 16777216
- Type: Long
- Unit: Bytes
- Is mutable: Yes
- Description: The read buffer size used when downloading lake segment files during lake replication. This value determines the per-read allocation for reading remote files; the implementation uses the larger of this setting and a 1 MB minimum. A larger value reduces the number of read calls and can improve throughput but increases memory used per concurrent download; a smaller value lowers memory usage at the cost of more I/O calls. Tune according to network bandwidth, storage I/O characteristics, and the number of parallel replication threads.
- Introduced in: v4.1.2
lake_replication_max_file_copy_retry
- Default: 3
- Type: Int
- Unit: -
- Is mutable: Yes
- Description: Maximum number of retry attempts for non-segment file copy (
.sst,.delvec,.del,.cols) during lake-to-lake (shared-data) cross-cluster replication. Each attempt verifies the copied file size matches the source to detect truncated copies caused by transient object storage issues. Increase this value if experiencing intermittent file corruption during replication over unreliable storage. - Introduced in: v4.1.2
lake_replication_file_copy_threads
- Default: 16
- Type: Int
- Unit: -
- Is mutable: No
- Description: Number of threads in the CN-wide dedicated thread pool used by lake-to-lake (shared-data) cross-cluster replication for per-file copy. The default caps concurrent copy tasks and their read buffers independently of CPU count.
0meanscpu_cores * 4; negative values mean-value * cpu_cores. The pool is built once at startup, so CN restart is required to change its size. - Introduced in: v4.1.2
lake_replication_max_parallel_files_per_tablet
- Default: 4
- Type: Int
- Unit: -
- Is mutable: Yes
- Description: Maximum number of files copied concurrently for one tablet during lake-to-lake (shared-data) cross-cluster replication. All tablets share the CN-wide
lake_replication_file_copy_threadspool, so this setting prevents one tablet from monopolizing the pool. Values less than or equal to1serialize file copy within each tablet. - Introduced in: v4.1.4
lake_replication_parallel_copy_min_file_count
- Default: 2
- Type: Int
- Unit: -
- Is mutable: Yes
- Description: Minimum number of files in a tablet required to copy that tablet's files concurrently during lake-to-lake (shared-data) cross-cluster replication. Set this parameter to
0to disable the dedicated file copy pool and execute file copy in the calling replication task. - Introduced in: v4.1.2
lake_service_max_concurrency
- Default: 0
- Type: Int
- Unit: -
- Is mutable: No
- Description: The maximum concurrency of RPC requests in a shared-data cluster. Incoming requests will be rejected when this threshold is reached. When this item is set to
0, no limit is imposed on the concurrency. - Introduced in: -
max_hdfs_scanner_num
- Default: 50
- Type: Int
- Unit: -
- Is mutable: No
- Description: Limits the maximum number of concurrently running connector (HDFS/remote) scanners that a ConnectorScanNode can have. During scan startup the node computes an estimated concurrency (based on memory, chunk size and scanner_row_num) and then caps it with this value to determine how many scanners and chunks to reserve and how many scanner threads to start. It is also consulted when scheduling pending scanners at runtime (to avoid oversubscription) and when deciding how many pending scanners can be re-submitted considering file-handle limits. Lowering this reduces threads, memory and open-file pressure at the cost of potential throughput; increasing it raises concurrency and resource usage.
- Introduced in: v3.2.0
query_max_memory_limit_percent
- Default: 90
- Type: Int
- Unit: -
- Is mutable: No
- Description: The maximum memory that the Query Pool can use. It is expressed as a percentage of the Process memory limit.
- Introduced in: v3.1.0
rocksdb_max_write_buffer_memory_bytes
- Default: 1073741824
- Type: Int64
- Unit: -
- Is mutable: No
- Description: It is the max size of the write buffer for meta in rocksdb. Default is 1GB.
- Introduced in: v3.5.0
rocksdb_write_buffer_memory_percent
- Default: 5
- Type: Int64
- Unit: -
- Is mutable: No
- Description: It is the memory percent of write buffer for meta in rocksdb. default is 5% of system memory. However, aside from this, the final calculated size of the write buffer memory will not be less than 64MB nor exceed 1G (rocksdb_max_write_buffer_memory_bytes)
- Introduced in: v3.5.0
Other
default_mv_resource_group_concurrency_limit
- Default: 0
- Type: Int
- Unit: -
- Is mutable: No
- Description: The maximum concurrency (per BE node) of the materialized view refresh tasks in the resource group
default_mv_wg. The default value0indicates no limits. Deprecated: from v3.4 onwards, this item no longer takes effect because the FE storesdefault_mv_wgand sends it to the BEs. UseALTER RESOURCE GROUP default_mv_wg WITH (...)instead. - Introduced in: v3.1
default_mv_resource_group_cpu_limit
- Default: 1
- Type: Int
- Unit: -
- Is mutable: No
- Description: The maximum number of CPU cores (per BE node) that can be used by the materialized view refresh tasks in the resource group
default_mv_wg. Deprecated: from v3.4 onwards, this item no longer takes effect because the FE storesdefault_mv_wgand sends it to the BEs. UseALTER RESOURCE GROUP default_mv_wg WITH (...)instead. - Introduced in: v3.1
default_mv_resource_group_memory_limit
- Default: 0.8
- Type: Double
- Unit:
- Is mutable: No
- Description: The maximum memory proportion (per BE node) that can be used by the materialized view refresh tasks in the resource group
default_mv_wg. The default value indicates 80% of the memory. Deprecated: from v3.4 onwards, this item no longer takes effect because the FE storesdefault_mv_wgand sends it to the BEs. UseALTER RESOURCE GROUP default_mv_wg WITH (...)instead. - Introduced in: v3.1
default_mv_resource_group_spill_mem_limit_threshold
- Default: 0.8
- Type: Double
- Unit: -
- Is mutable: No
- Description: The memory usage threshold before a materialized view refresh task in the resource group
default_mv_wgtriggers intermediate result spilling. The default value indicates 80% of the memory. Deprecated: from v3.4 onwards, this item no longer takes effect because the FE storesdefault_mv_wgand sends it to the BEs. UseALTER RESOURCE GROUP default_mv_wg WITH (...)instead. - Introduced in: v3.1
enable_resolve_hostname_to_ip_in_load_error_url
- Default: false
- Type: Boolean
- Unit: -
- Is mutable: Yes
- Description: For
error_urlsdebugging, whether to allow operators to choose between using original hostnames from FE heartbeat or forcing resolution to IP addresses based on their environment needs.true: Resolve hostnames to IPs.false(Default): Keeps the original hostname in the error URL.
- Introduced in: v4.0.1
enable_retry_apply
- Default: true
- Type: Boolean
- Unit: -
- Is mutable: Yes
- Description: When enabled, Tablet apply failures that are classified as retryable (for example transient memory-limit errors) are rescheduled for retry instead of immediately marking the tablet in error. The retry path in TabletUpdates schedules the next attempt using
retry_apply_interval_secondmultiplied by the current failure count and clamped to a 600s maximum, so backoff grows with successive failures. Explicitly non-retryable errors (for example corruption) bypass retries and cause the apply process to enter the error state immediately. Retries continue until an overall timeout/terminal condition is reached, after which the apply will enter the error state. Turning this off disables automatic rescheduling of failed apply tasks and causes failed applies to transition to error state without retries. - Introduced in: v3.2.9
enable_token_check
- Default: true
- Type: Boolean
- Unit: -
- Is mutable: Yes
- Description: A boolean value to control whether to enable the token check.
trueindicates enabling the token check, andfalseindicates disabling it. - Introduced in: -
es_scroll_keepalive
- Default: 5m
- Type: String
- Unit: Minutes (string with suffix, e.g. "5m")
- Is mutable: No
- Description: The keep-alive duration sent to Elasticsearch for scroll search contexts. The value is used verbatim (for example "5m") when building the initial scroll URL (
?scroll=<value>) and when sending subsequent scroll requests (via ESScrollQueryBuilder). This controls how long the ES search context is retained before garbage collection on the ES side; setting it longer keeps scroll contexts alive for more time but prolongs resource usage on the ES cluster. The value is read at startup by the ES scan reader and is not changeable at runtime. - Introduced in: v3.2.0
jemalloc_page_size
- Default: auto
- Type: String
- Unit: -
- Is mutable: No
- Description: Forces
bin/start_backend.shto load a specific jemalloc page-size variant on arm64 (aarch64) hosts instead of auto-detecting the host's page size viagetconf PAGESIZE. Accepted values areauto(default, auto-detect),4k/4096(forces the 4K-page build), and64k/65536(forces the 64K-page build); any other value falls back to auto-detection with a warning logged. A forced4kis still rejected in favor of the 64K build (with a warning) if agetconf PAGESIZEcheck succeeds and reports a size other than 4096, since jemalloc aborts at startup when the runtime page size is larger than its build-time one; a failed page-size check does not block the forced value. If4kis requested but the 4K-page jemalloc build was not packaged, BE falls back to the 64K build with a warning instead of failing to start. This value is read directly from be.conf/cn.conf bybin/start_backend.shbefore the BE process starts (to select which jemalloc.soto load viaLD_LIBRARY_PATH), so it does not reconfigure jemalloc at runtime and changing it requires a BE restart. Ignored on non-arm64 hosts, where only one jemalloc build is produced. - Introduced in: -
load_replica_status_check_interval_ms_on_failure
- Default: 2000
- Type: Int
- Unit: Milliseconds
- Is mutable: Yes
- Description: The interval that the secondary replica checks it's status on the primary replica if the last check rpc fails.
- Introduced in: v3.5.1
load_replica_status_check_interval_ms_on_success
- Default: 15000
- Type: Int
- Unit: Milliseconds
- Is mutable: Yes
- Description: The interval that the secondary replica checks it's status on the primary replica if the last check rpc successes.
- Introduced in: v3.5.1
max_length_for_bitmap_function
- Default: 1000000
- Type: Int
- Unit: Bytes
- Is mutable: Yes
- Description: The maximum length of input values for bitmap functions.
- Introduced in: -
max_length_for_to_base64
- Default: 200000
- Type: Int
- Unit: Bytes
- Is mutable: Yes
- Description: The maximum length of input values for the to_base64() function.
- Introduced in: -
memory_high_level
- Default: 75
- Type: Long
- Unit: Percent
- Is mutable: Yes
- Description: High water memory threshold expressed as a percentage of the process memory limit. When total memory consumption rises above this percentage, BE begins to free memory gradually (currently by evicting data cache and update cache, and by shrinking the shared-data metadata caches) to relieve pressure. The monitor uses this value to compute memory_high = mem_limit * memory_high_level / 100 and, if consumption
>memory_high, performs controlled eviction guided by the GC advisor; if consumption exceeds memory_urgent_level (a separate config), more aggressive immediate reductions occur. This value is also consulted to disable certain memory‑intensive operations (for example, primary-key preload) when the threshold is exceeded. Must satisfy validation with memory_urgent_level (memory_urgent_level>memory_high_level, memory_high_level>=1, memory_urgent_level<=100). - Introduced in: v3.2.0
report_exec_rpc_request_retry_num
- Default: 10
- Type: Int
- Unit: -
- Is mutable: Yes
- Description: The retry times of rpc request to report exec rpc request to FE. The default value is 10, which means that the rpc request will be retried 10 times if it fails only if it's fragment instatnce finish rpc. Report exec rpc request is important for load job, if one fragment instance finish report failed, the load job will be hang until timeout.
- Introduced in: -
sleep_one_second
- Default: 1
- Type: Int
- Unit: Seconds
- Is mutable: No
- Description: A small, global sleep interval (in seconds) used by BE agent worker threads as a one-second pause when the master address/heartbeat is not yet available or when a short retry/backoff is needed. In the codebase it is referenced by several report worker pools (e.g., ReportDiskStateTaskWorkerPool, ReportOlapTableTaskWorkerPool, ReportWorkgroupTaskWorkerPool) to avoid busy-waiting and reduce CPU consumption while retrying. Increasing this value slows the retry frequency and responsiveness to master availability; reducing it increases polling rate and CPU usage. Adjust only with awareness of the trade-off between responsiveness and resource use.
- Introduced in: v3.2.0
small_file_dir
- Default:
${STARROCKS_HOME}/lib/small_file/ - Type: String
- Unit: -
- Is mutable: No
- Description: The directory used to store the files downloaded by the file manager.
- Introduced in: -
upload_buffer_size
- Default: 4194304
- Type: Int
- Unit: Bytes
- Is mutable: Yes
- Description: Buffer size (in bytes) used by file copy operations when uploading snapshot files to remote storage (broker or direct FileSystem). In the upload path (snapshot_loader.cpp) this value is passed to fs::copy as the read/write chunk size for each upload stream. The default is 4 MiB. Increasing this value can improve throughput on high-latency or high-bandwidth links but increases memory usage per concurrent upload; decreasing it reduces per-stream memory but may reduce transfer efficiency. Tune together with upload_worker_count and overall available memory.
- Introduced in: v3.2.13
user_function_dir
- Default:
${STARROCKS_HOME}/lib/udf - Type: String
- Unit: -
- Is mutable: No
- Description: The directory used to store User-defined Functions (UDFs).
- Introduced in: -
web_log_bytes
- Default: 1048576 (1 MB)
- Type: long
- Unit: Bytes
- Is mutable: No
- Description: Maximum number of bytes to read from the INFO logfile and show on the BE debug webserver's log page. The handler uses this value to compute a seek offset (showing the last N bytes) to avoid reading or serving very large log files. If the logfile is smaller than this value the whole file is shown. Note: in the current implementation the code that reads and serves the INFO log is commented out and the handler reports that the INFO log file couldn't be opened, so this parameter may have no effect unless the log-serving code is enabled.
- Introduced in: v3.2.0