Skip to main content
Version: 26.2 Preview

BE Configuration - Query and Loading

Some BE configuration items are dynamic parameters which you can set interactively when BE nodes are online. The rest of them are static parameters. You can only set the static parameters of a BE node by changing them in the PhoenixAI Cloud console.

  1. Sign in to the PhoenixAI Cloud console.
  2. On the Clusters page, click the classic cluster that you want to configure.
  3. On the cluster details page, click the Cluster parameters tab.
  4. In the BE Node static configuration section, click View all parameter list.
  5. In the dialog box that appears, click New parameter.
  6. Enter the name of the static parameter you want to configure in the Parameter key field, and the parameter value you want to set in the Value field. For the details of the parameters you can configure, see Usage notes.
  7. Click the save button next to the Value field to save the record of the parameter change.
  8. After configuring all the parameters you want to change, click Save changes to save the changes.
  9. To allow the changes to take effect, you can either manually suspend and resume the cluster whenever you find suitable, or click Apply to all nodes in the BE Node static configuration section to restart the cluster instantly.

Follow these steps to configure Compute Node parameters of a warehouse in the PhoenixAI Cloud console:

  1. Sign in to the PhoenixAI Cloud console.
  2. On the Clusters page, click the elastic cluster that you want to configure.
  3. On the cluster details page, click the Warehouses tab.
  4. On the Warehouses tab, click the warehouse that you want to configure.
  5. On the warehouse details page, click the Warehouse parameters tab.
  6. In the Compute Node static configuration section, click View all parameter list.
  7. In the dialog box that appears, click New parameter.
  8. Enter the name of the parameter you want to configure in the Parameter key field, and the parameter value you want to set in the Value field.
    note

    PhoenixAI provides validation checks on parameter keys and value types. Only valid parameter keys and values can be applied. For the details of the parameters you can configure, see Usage notes.

  9. Click the save button next to the Value field to save the record of the parameter change.
  10. After configuring all the parameters you want to change, click Save changes to save the changes.
  11. To allow the changes to take effect, you can either manually suspend and resume the warehouse whenever you find it suitable, or click Apply to all nodes in the Compute Node static configuration section to restart the warehouse instantly. For default_warehouse, you can only restart the cluster.

View BE configuration items​

You can view the BE configuration items using the following command:

SELECT * FROM information_schema.be_configs [WHERE NAME LIKE "%<name_pattern>%"]

Configure BE parameters​

Configure BE dynamic parameters​

You can configure a dynamic parameter of a BE node by using the curl command.

curl -XPOST http://be_host:http_port/api/update_config?<configuration_item>=<value>

Configure BE static parameters​

You can only set the static parameters of a BE by changing them in the PhoenixAI Cloud console, and restarting the BE to allow the changes to take effect.

This topic introduces the following types of BE configurations:

Query​

agg_hash_map_key_prefetch_dist​

  • Default: 8
  • Type: Int
  • Unit: Rows
  • Is mutable: Yes
  • Description: Distance (in rows) of a second software prefetch stage for aggregations grouped by multiple columns or by a non-fixed-size key (serialized keys). The hash table slot of such a key only points at the key bytes, so the key comparison on a hit waits on a cache miss that agg_hash_map_prefetch_dist cannot hide. At this many rows ahead, the loop reads the slot the first stage has already prefetched, finds the entry whose cached hash matches without comparing keys, and prefetches its key bytes. It takes effect only when it is greater than 0 and smaller than agg_hash_map_prefetch_dist; any other value disables it. It also runs only once the estimated key bytes in the hash table (groups times the average key length) reach L3_size / 2 * agg_prefetch_l2_ratio, because smaller keys are already cache-resident and the extra stage is pure overhead. The value is read once per chunk.
  • Introduced in: -

agg_hash_map_prefetch_dist​

  • Default: 16
  • Type: Int
  • Unit: Rows
  • Is mutable: Yes
  • Description: Software prefetch distance (in rows) for the aggregation hash-map / hash-set probe loop. While building the aggregation hash table, the loop prefetches the bucket for the row this many positions ahead of the one it is currently processing, hiding memory latency on large tables. Setting it to 0 disables software prefetch. The value is read once per chunk, so changes take effect on the next chunk. The default 16 is empirical for L3-resident tables; raise it for DRAM-resident workloads and lower it for cache-resident ones. Prefetch is additionally gated by agg_prefetch_l2_ratio: regardless of this distance, no prefetch is issued while the hash table still fits in L2.
  • Introduced in: -

agg_prefetch_l2_ratio​

  • Default: 1.0
  • Type: Double
  • Unit: -
  • Is mutable: Yes
  • Description: Gates aggregation hash-table software prefetch on L2 residency. Prefetch is enabled only once the bucket array spills L2, that is, when bucket_count * slot_bytes >= L2_size * agg_prefetch_l2_ratio, where the L2 size is detected at runtime (falling back to 1 MiB if detection fails). Below this point the table is L2-resident and prefetching is a net loss. Lower the ratio on contended deployments that run many drivers per core, where the effective per-table share of L2 is smaller than the nominal per-core size; raising it above 1.0 delays prefetch until the table is well past L2. The same ratio also scales the key-bytes threshold of agg_hash_map_key_prefetch_dist. See also agg_hash_map_prefetch_dist.
  • Introduced in: -

clear_udf_cache_when_start​

  • Default: false
  • Type: Boolean
  • Unit: -
  • Is mutable: No
  • Description: When enabled, the BE's UserFunctionCache will clear all locally cached user function libraries on startup. During UserFunctionCache::init, the code calls _reset_cache_dir(), which removes UDF files from the configured UDF library directory (organized into kLibShardNum subdirectories) and deletes files with Java/Python UDF suffixes (.jar/.py). When disabled (default), the BE loads existing cached UDF files instead of deleting them. Enabling this forces UDF binaries to be re-downloaded on first use after restart (increasing network traffic and first-use latency).
  • Introduced in: v4.0.0

dictionary_speculate_min_chunk_size​

  • Default: 10000
  • Type: Int
  • Unit: Rows
  • Is mutable: No
  • Description: Minimum number of rows (chunk size) used by StringColumnWriter and DictColumnWriter to trigger dictionary-encoding speculation. If an incoming column (or the accumulated buffer plus incoming rows) has size larger than or equal dictionary_speculate_min_chunk_size the writer will run speculation immediately and set an encoding (DICT, PLAIN or BIT_SHUFFLE) rather than buffering more rows. Speculation uses dictionary_encoding_ratio for string columns and dictionary_encoding_ratio_for_non_string_column for numeric/non-string columns to decide whether dictionary encoding is beneficial. A string value larger than 1 MiB also triggers immediate speculation and forces plain encoding. In addition, a column byte size larger than or equal to UINT32_MAX forces immediate speculation to avoid BinaryColumn<uint32_t> overflow.
  • Introduced in: v3.2.0

disable_storage_page_cache​

  • Default: false
  • Type: Boolean
  • Unit: -
  • Is mutable: Yes
  • Description: A boolean value to control whether to disable PageCache.
    • When PageCache is enabled, StarRocks caches the recently scanned data.
    • PageCache can significantly improve the query performance when similar queries are repeated frequently.
    • true indicates disabling PageCache.
    • The default value of this item has been changed from true to false since StarRocks v2.4.
  • Introduced in: -

enable_glm_lookup_http_fallback​

  • Default: true
  • Type: Boolean
  • Unit: -
  • Is mutable: Yes
  • Description: Whether to enable the HTTP fallback RPC for GLM lookup responses. Keep this disabled during rolling upgrades or downgrades until all BE and CN nodes support lookup_via_http. When disabled, GLM lookup fetches use the regular lookup RPC.
  • Introduced in: -

enable_segment_replication_http_fallback​

  • Default: true
  • Type: Boolean
  • Unit: -
  • Is mutable: Yes
  • Description: Whether to enable the HTTP fallback RPC for segment replication. Keep this disabled during rolling upgrades or downgrades until all participating BE nodes support tablet_writer_add_segment_via_http. When disabled, segment replication uses the regular tablet_writer_add_segment RPC.
  • Introduced in: -

enable_bitmap_index_memory_page_cache​

  • Default: true
  • Type: Boolean
  • Unit: -
  • Is mutable: Yes
  • Description: Whether to enable memory cache for Bitmap index. Memory cache is recommended if you want to use Bitmap indexes to accelerate point queries.
  • Introduced in: v3.1

enable_compaction_flat_json​

  • Default: True
  • Type: Boolean
  • Unit:
  • Is mutable: Yes
  • Description: Whether to enable compaction for Flat JSON data.
  • Introduced in: v3.3.3

enable_json_flat​

  • Default: true
  • Type: Boolean
  • Unit:
  • Is mutable: Yes
  • Description: Whether to enable the Flat JSON feature. After this feature is enabled, newly loaded JSON data will be automatically flattened, improving JSON query performance.
  • Introduced in: v3.3.0

enable_lazy_dynamic_flat_json​

  • Default: True
  • Type: Boolean
  • Unit:
  • Is mutable: Yes
  • Description: Whether to enable Lazy Dyamic Flat JSON when a query misses Flat JSON schema in read process. When this item is set to true, StarRocks will postpone the Flat JSON operation to calculation process instead of read process.
  • Introduced in: v3.3.3

enable_ordinal_index_memory_page_cache​

  • Default: true
  • Type: Boolean
  • Unit: -
  • Is mutable: Yes
  • Description: Whether to enable memory cache for ordinal index. Ordinal index is a mapping from row IDs to data page positions, and it can be used to accelerate scans.
  • Introduced in: -

enable_spill_sort_events​

  • Default: false
  • Type: Boolean
  • Unit: -
  • Is mutable: Yes
  • Description: Enables the pipeline event scheduler for the spilling sort operators instead of poll-spinning.
  • Introduced in: -

enable_string_prefix_zonemap​

  • Default: true
  • Type: Boolean
  • Unit: -
  • Is mutable: Yes
  • Description: Whether to enable ZoneMap for string (CHAR/VARCHAR) columns using prefix-based min/max. For non-key string columns, the min/max values are truncated to a fixed prefix length configured by string_prefix_zonemap_prefix_len. If any value in a CHAR/VARCHAR column exceeds 1 MiB, StarRocks does not write a ZoneMap for that column in the entire segment, regardless of this setting.
  • Introduced in: -

enable_zonemap_index_memory_page_cache​

  • Default: true
  • Type: Boolean
  • Unit: -
  • Is mutable: Yes
  • Description: Whether to enable memory cache for zonemap index. Memory cache is recommended if you want to use zonemap indexes to accelerate scan.
  • Introduced in: -

exchg_node_buffer_size_bytes​

  • Default: 10485760
  • Type: Int
  • Unit: Bytes
  • Is mutable: Yes
  • Description: The maximum buffer size on the receiver end of an exchange node for each query. This configuration item is a soft limit. A backpressure is triggered when data is sent to the receiver end with an excessive speed.
  • Introduced in: -

exec_state_report_max_threads​

  • Default: 2
  • Type: Int
  • Unit: Threads
  • Is mutable: Yes
  • Description: Maximum number of threads for the exec-state-report thread pool. This pool is used by ExecStateReporter to asynchronously send non-priority execution status reports (such as fragment completion and error status) from BE to FE via RPC. The actual pool size at startup is max(1, exec_state_report_max_threads). Changing this config at runtime triggers update_max_threads on the pool in every executor set (shared and exclusive). The pool has a fixed task queue size of 1000; report submissions are silently dropped when all threads are busy and the queue is full. Paired with priority_exec_state_report_max_threads for the high-priority pool. Increase this value when delayed or dropped exec-state reports are observed under high query concurrency.
  • Introduced in: v4.1.0, v4.0.8, v3.5.15

file_descriptor_cache_capacity​

  • Default: 16384
  • Type: Int
  • Unit: -
  • Is mutable: No
  • Description: The number of file descriptors that can be cached.
  • Introduced in: -

flamegraph_tool_dir​

  • Default: ${STARROCKS_HOME}/bin/flamegraph
  • Type: String
  • Unit: -
  • Is mutable: No
  • Description: Directory of the flamegraph tool, which should contain pprof, stackcollapse-go.pl, and flamegraph.pl scripts for generating flame graphs from profile data.
  • Introduced in: -

fragment_pool_queue_size​

  • Default: 2048
  • Type: Int
  • Unit: -
  • Is mutable: No
  • Description: The upper limit of the query number that can be processed on each BE node.
  • Introduced in: -

fragment_pool_thread_num_max​

  • Default: 4096
  • Type: Int
  • Unit: -
  • Is mutable: No
  • Description: The maximum number of threads used for query.
  • Introduced in: -

fragment_pool_thread_num_min​

  • Default: 64
  • Type: Int
  • Unit: Minutes -
  • Is mutable: No
  • Description: The minimum number of threads used for query.
  • Introduced in: -

hdfs_client_enable_hedged_read​

  • Default: false
  • Type: Boolean
  • Unit: -
  • Is mutable: No
  • Description: Specifies whether to enable the hedged read feature.
  • Introduced in: v3.0

hdfs_client_hedged_read_threadpool_size​

  • Default: 128
  • Type: Int
  • Unit: -
  • Is mutable: No
  • Description: Specifies the size of the Hedged Read thread pool on your HDFS client. The thread pool size limits the number of threads to dedicate to the running of hedged reads in your HDFS client. It is equivalent to the dfs.client.hedged.read.threadpool.size parameter in the hdfs-site.xml file of your HDFS cluster.
  • Introduced in: v3.0

hdfs_client_hedged_read_threshold_millis​

  • Default: 2500
  • Type: Int
  • Unit: Milliseconds
  • Is mutable: No
  • Description: Specifies the number of milliseconds to wait before starting up a hedged read. For example, you have set this parameter to 30. In this situation, if a read from a block has not returned within 30 milliseconds, your HDFS client immediately starts up a new read against a different block replica. It is equivalent to the dfs.client.hedged.read.threshold.millis parameter in the hdfs-site.xml file of your HDFS cluster.
  • Introduced in: v3.0

io_coalesce_adaptive_lazy_active​

  • Default: true
  • Type: Boolean
  • Unit: -
  • Is mutable: Yes
  • Description: Based on the selectivity of predicates, adaptively determines whether to combine the I/O of predicate columns and non-predicate columns.
  • Introduced in: v3.2

jit_lru_cache_size​

  • Default: 0
  • Type: Int
  • Unit: Bytes
  • Is mutable: Yes
  • Description: The LRU cache size for JIT compilation. It represents the actual size of the cache if it is set to greater than 0. If it is set to less than or equal to 0, the system will adaptively set the cache using the formula jit_lru_cache_size = min(mem_limit*0.01, 1GB) (while mem_limit of the node must be greater or equal to 16 GB).
  • Introduced in: -

json_flat_column_max​

  • Default: 100
  • Type: Int
  • Unit:
  • Is mutable: Yes
  • Description: The maximum number of sub-fields that can be extracted by Flat JSON. This parameter takes effect only when enable_json_flat is set to true.
  • Introduced in: v3.3.0

json_flat_create_zonemap​

  • Default: true
  • Type: Boolean
  • Unit:
  • Is mutable: Yes
  • Description: Whether to create ZoneMaps for flattened JSON sub-columns during write. This parameter takes effect only when enable_json_flat is set to true.
  • Introduced in: -

json_flat_null_factor​

  • Default: 0.3
  • Type: Double
  • Unit:
  • Is mutable: Yes
  • Description: The proportion of NULL values in the column to extract for Flat JSON. A column will not be extracted if its proportion of NULL value is higher than this threshold. This parameter takes effect only when enable_json_flat is set to true.
  • Introduced in: v3.3.0

json_flat_sparsity_factor​

  • Default: 0.3
  • Type: Double
  • Unit:
  • Is mutable: Yes
  • Description: The proportion of columns with the same name for Flat JSON. Extraction is not performed if the proportion of columns with the same name is lower than this value. This parameter takes effect only when enable_json_flat is set to true.
  • Introduced in: v3.3.0

lake_tablet_ignore_invalid_delete_predicate​

  • Default: false
  • Type: Boolean
  • Unit: -
  • Is mutable: Yes
  • Description: A boolean value to control whether ignore invalid delete predicates in tablet rowset metadata which may be introduced by logic deletion to a duplicate key table after the column name renamed.
  • Introduced in: v4.0

late_materialization_ratio​

  • Default: 10
  • Type: Int
  • Unit: -
  • Is mutable: Yes
  • Description: Integer ratio in range [0-1000] that controls the use of late materialization in the SegmentIterator (vector query engine). A value of 0 (or ≤ 0) disables late materialization; 1000 (or ≥ 1000) forces late materialization for all reads. Values > 0 and < 1000 enable a conditional strategy where both late and early materialization contexts are prepared and the iterator selects behavior based on predicate filter ratios (higher values favor late materialization). When a segment contains complex metric types, StarRocks uses metric_late_materialization_ratio instead. If lake_io_opts.cache_file_only is set, late materialization is disabled.
  • Introduced in: v3.2.0

max_hdfs_file_handle​

  • Default: 1000
  • Type: Int
  • Unit: -
  • Is mutable: Yes
  • Description: The maximum number of HDFS file descriptors that can be opened.
  • Introduced in: -

max_hdfs_scanner_num​

  • Default: 50
  • Type: Int
  • Unit: -
  • Is mutable: No
  • Description: Maximum number of concurrent remote scanners (HDFS, object storage, etc.) that ConnectorScanNode can run simultaneously. This value caps estimated concurrency at startup and also limits pending-scanner scheduling at runtime, controlling thread, memory, and file-handle pressure.
  • Introduced in: v3.2.0

max_memory_sink_batch_count​

  • Default: 20
  • Type: Int
  • Unit: -
  • Is mutable: Yes
  • Description: The maximum number of Scan Cache batches.
  • Introduced in: -

max_pushdown_conditions_per_column​

  • Default: 1024
  • Type: Int
  • Unit: -
  • Is mutable: Yes
  • Description: The maximum number of conditions that allow pushdown in each column. If the number of conditions exceeds this limit, the predicates are not pushed down to the storage layer.
  • Introduced in: -

max_scan_key_num​

  • Default: 1024
  • Type: Int
  • Unit: -
  • Is mutable: Yes
  • Description: The maximum number of scan keys segmented by each query.
  • Introduced in: -

metric_late_materialization_ratio​

  • Default: 1000
  • Type: Int
  • Unit: -
  • Is mutable: No
  • Description: Controls when the late-materialization row access strategy is used for reads that include complex metric columns. Valid range: [0-1000]. 0 disables late materialization; 1000 forces late materialization for all applicable reads. Values 1–999 enable a conditional strategy where both late and early materialization contexts are prepared and chosen at runtime based on predicate/selectivity. When complex metric types exist, metric_late_materialization_ratio overrides the general late_materialization_ratio. Note: cache_file_only I/O mode will cause late materialization to be disabled regardless of this setting.
  • Introduced in: v3.2.0

min_file_descriptor_number​

  • Default: 60000
  • Type: Int
  • Unit: -
  • Is mutable: No
  • Description: The minimum number of file descriptors in the BE process.
  • Introduced in: -

object_storage_client_cache_size​

  • Default: 8
  • Type: Int
  • Unit: -
  • Is mutable: Yes
  • Description: The maximum number of object storage clients (S3-compatible and Azure Blob) cached per client factory. The value is read on each client creation, so lowering it takes effect gradually as cached clients are evicted during subsequent creations. Values below 1 are treated as 1.
  • Introduced in: v4.1.4, v4.0.14

object_storage_connect_timeout_ms​

  • Default: -1
  • Type: Int
  • Unit: Milliseconds
  • Is mutable: No
  • Description: Timeout duration to establish socket connections with object storage. -1 indicates to use the default timeout duration of the SDK configurations.
  • Introduced in: v3.0.9

enable_poco_client_for_aws_sdk​

  • Default: false
  • Type: Boolean
  • Unit: -
  • Is mutable: No
  • Description: Whether to use the Poco HTTP client for the AWS SDK. true replaces the AWS SDK's default curl HTTP client with Poco. false uses the default curl client.
  • Introduced in: -

object_storage_request_timeout_ms​

  • Default: 10000
  • Type: Int
  • Unit: Milliseconds
  • Is mutable: Yes
  • Description: How long a request to object storage may stay stalled before the client aborts and retries. This is not a deadline on the request: for the curl client it is the low speed time, the time a transfer may stay below 1 byte/s, and for the Poco client it is the socket send/receive timeout. A transfer that keeps making progress is never cut off, however long it runs. 0 disables the check; a negative value leaves the client on its own default, which on the Poco path is 60 seconds.
  • Introduced in: v3.0.9

parquet_late_materialization_enable​

  • Default: true
  • Type: Boolean
  • Unit: -
  • Is mutable: Yes
  • Description: A boolean value to control whether to enable the late materialization of Parquet reader to improve performance. true indicates enabling late materialization, and false indicates disabling it.
  • Introduced in: -

parquet_page_index_enable​

  • Default: true
  • Type: Boolean
  • Unit: -
  • Is mutable: No
  • Description: A boolean value to control whether to enable the pageindex of Parquet file to improve performance. true indicates enabling pageindex, and false indicates disabling it.
  • Introduced in: v3.3

parquet_reader_bloom_filter_enable​

  • Default: true
  • Type: Boolean
  • Unit: -
  • Is mutable: Yes
  • Description: A boolean value to control whether to enable the bloom filter of Parquet file to improve performance. true indicates enabling the bloom filter, and false indicates disabling it. You can also control this behavior on session level using the system variable enable_parquet_reader_bloom_filter. Bloom filters in Parquet are maintained at the column level within each row group. If a Parquet file contains bloom filters for certain columns, queries can use predicates on those columns to efficiently skip row groups.
  • Introduced in: v3.5

parquet_runtime_filter_push_down_enable​

  • Default: true
  • Type: Boolean
  • Unit: -
  • Is mutable: Yes
  • Description: A boolean value to control whether to evaluate join runtime filters against decoded rows inside the Parquet reader, so that non-matching rows are dropped before lazy columns are materialized. true indicates enabling this row-level pushdown, and false indicates disabling it, in which case runtime filters are only used for row group and page statistics pruning, and the row-level probe happens in the downstream scan operator instead.
  • Introduced in: -

path_gc_check_step​

  • Default: 1000
  • Type: Int
  • Unit: -
  • Is mutable: Yes
  • Description: The maximum number of files that can be scanned continuously each time.
  • Introduced in: -

path_gc_check_step_interval_ms​

  • Default: 10
  • Type: Int
  • Unit: Milliseconds
  • Is mutable: Yes
  • Description: The time interval between file scans.
  • Introduced in: -

path_scan_interval_second​

  • Default: 86400
  • Type: Int
  • Unit: Seconds
  • Is mutable: Yes
  • Description: The time interval at which GC cleans expired data.
  • Introduced in: -

pipeline_connector_scan_thread_num_per_cpu​

  • Default: 8
  • Type: Double
  • Unit: -
  • Is mutable: Yes
  • Description: The number of scan threads assigned to Pipeline Connector per CPU core in the BE node. This configuration is changed to dynamic from v3.1.7 onwards.
  • Introduced in: -

pipeline_enable_large_column_checker​

  • Default: true
  • Type: Boolean
  • Unit: -
  • Is mutable: Yes
  • Description: Whether to enable large column detection in the pipeline execution framework. When enabled, queries fail with a capacity limit error if an intermediate column reaches the chunk capacity limit in pipeline execution or spill serialization.
  • Introduced in: v4.0.0

pipeline_poller_timeout_guard_ms​

  • Default: -1
  • Type: Int
  • Unit: Milliseconds
  • Is mutable: Yes
  • Description: When this item is set to greater than 0, if a driver takes longer than pipeline_poller_timeout_guard_ms for a single dispatch in the poller, then the information of the driver and operator is printed.
  • Introduced in: -

pipeline_prepare_thread_pool_queue_size​

  • Default: 102400
  • Type: Int
  • Unit: -
  • Is mutable: No
  • Description: The maximum queue lenggth of PREPARE fragment thread pool for Pipeline execution engine.
  • Introduced in: -

pipeline_prepare_thread_pool_thread_num​

  • Default: 0
  • Type: Int
  • Unit: -
  • Is mutable: No
  • Description: Number of threads in the pipeline execution engine PREPARE fragment thread pool. 0 indicates the value is equal to the number of system VCPU core number.
  • Introduced in: -

pipeline_prepare_timeout_guard_ms​

  • Default: -1
  • Type: Int
  • Unit: Milliseconds
  • Is mutable: Yes
  • Description: When this item is set to greater than 0, if a plan fragment exceeds pipeline_prepare_timeout_guard_ms during the PREPARE process, a stack trace of the plan fragment is printed.
  • Introduced in: -

pipeline_scan_thread_pool_queue_size​

  • Default: 102400
  • Type: Int
  • Unit: -
  • Is mutable: No
  • Description: The maximum task queue length of SCAN thread pool for Pipeline execution engine.
  • Introduced in: -

case_when_selective_eval_ratio​

  • Default: 2
  • Type: Int
  • Unit: -
  • Is mutable: Yes
  • Description: Controls whether each THEN branch of a searched CASE WHEN is evaluated only on the rows that branch owns, instead of being evaluated over the whole chunk and having its rows picked afterwards. It only applies when the CASE returns a collection (ARRAY/MAP/STRUCT) or VARIANT type, where materializing a row is expensive enough to pay for compacting the branch's input rows into a sub-chunk. A branch is compacted only when owned_rows * case_when_selective_eval_ratio < chunk_rows, that is, when the branch owns less than 1 / case_when_selective_eval_ratio of the chunk; a branch above the threshold is still evaluated over the whole chunk, because compacting its input would copy more than the skipped evaluation saves. Setting this item to 1 compacts every branch that does not own the whole chunk. Setting it to 0 or a negative value turns the optimization off entirely and restores the previous behavior, including the behavior change it carries: a THEN or ELSE that would raise an error is no longer evaluated when no row selects it.
  • Introduced in: -

enable_lock_free_scan_task_queue​

  • Default: true
  • Type: Boolean
  • Unit: -
  • Is mutable: Yes
  • Description: Controls whether ScanExecutor uses LockFreeWorkGroupScanTaskQueue for OLAP scan and connector scan task scheduling. When enabled, scan tasks are scheduled through a lock-free per-workgroup queue plus a workgroup-level coordinator, which reduces contention compared with the legacy mutex-based WorkGroupScanTaskQueue. Set this item to false to fall back to the legacy scan task queue implementation during troubleshooting or rollback.
  • Introduced in: v4.1.0

pk_index_parallel_get_threadpool_size​

  • Default: 1048576
  • Type: Int
  • Unit: -
  • Is mutable: Yes
  • Description: Sets the maximum queue size (number of pending tasks) for the "cloud_native_pk_index_get" thread pool used by PK index parallel get operations in shared-data (cloud-native/lake) mode. The actual thread count for that pool is controlled by pk_index_parallel_get_threadpool_max_threads; this setting only limits how many tasks may be queued awaiting execution. The very large default (2^20) effectively makes the queue unbounded; lowering it prevents excessive memory growth from queued tasks but may cause task submissions to block or fail when the queue is full. Tune together with pk_index_parallel_get_threadpool_max_threads based on workload concurrency and memory constraints.
  • Introduced in: -

priority_exec_state_report_max_threads​

  • Default: 2
  • Type: Int
  • Unit: Threads
  • Is mutable: Yes
  • Description: Maximum number of threads for the high-priority exec-state-report thread pool. This pool is used by ExecStateReporter to asynchronously send high-priority execution status reports (such as urgent fragment failures) from BE to FE via RPC. Unlike the normal exec-state-report pool, this pool has an unbounded task queue. The actual pool size at startup is max(1, priority_exec_state_report_max_threads). Changing this config at runtime triggers update_max_threads on the priority pool in every executor set (shared and exclusive). Paired with exec_state_report_max_threads for the normal pool. Increase this value when high-priority reports are delayed under heavy concurrent query loads.
  • Introduced in: v4.1.0, v4.0.8, v3.5.15

priority_queue_remaining_tasks_increased_frequency​

  • Default: 512
  • Type: Int
  • Unit: -
  • Is mutable: Yes
  • Description: Controls how often the BlockingPriorityQueue increases ("ages") the priority of all remaining tasks to avoid starvation. Each successful get/pop increments an internal _upgrade_counter; when _upgrade_counter exceeds priority_queue_remaining_tasks_increased_frequency, the queue increments every element's priority, rebuilds the heap, and resets the counter. Lower values cause more frequent priority aging (reducing starvation but increasing CPU cost due to iterating and re-heapifying); higher values reduce that overhead but delay priority adjustments. The value is a simple operation count threshold, not a time duration.
  • Introduced in: v3.2.0

query_cache_capacity​

  • Default: 536870912
  • Type: Int
  • Unit: Bytes
  • Is mutable: No
  • Description: The size of the query cache in the BE. The default size is 512 MB. The size cannot be less than 4 MB. If the memory capacity of the BE is insufficient to provision your expected query cache size, you can increase the memory capacity of the BE.
  • Introduced in: -

query_pool_spill_mem_limit_threshold​

  • Default: 1.0
  • Type: Double
  • Unit: -
  • Is mutable: No
  • Description: If automatic spilling is enabled, when the memory usage of all queries exceeds query_pool memory limit * query_pool_spill_mem_limit_threshold, intermediate result spilling will be triggered.
  • Introduced in: v3.2.7

query_scratch_dirs​

  • Default: ${STARROCKS_HOME}
  • Type: string
  • Unit: -
  • Is mutable: No
  • Description: Comma-separated list of writable scratch directories used by query execution to spill intermediate data (for example, external sorts, hash joins, and other operators). Specify one or more paths separated by ; (e.g. /mnt/ssd1/tmp;/mnt/ssd2/tmp). Directories should be accessible and writable by the BE process and have sufficient free space; StarRocks will pick among them to distribute spill I/O. Changes require a restart to take effect. If a directory is missing, not writable, or full, spilling may fail or degrade query performance.
  • Introduced in: v3.2.0

result_buffer_cancelled_interval_time​

  • Default: 300
  • Type: Int
  • Unit: Seconds
  • Is mutable: Yes
  • Description: The wait time before BufferControlBlock releases data.
  • Introduced in: -

scan_context_gc_interval_min​

  • Default: 5
  • Type: Int
  • Unit: Minutes
  • Is mutable: Yes
  • Description: The time interval at which to clean the Scan Context.
  • Introduced in: -

scanner_row_num​

  • Default: 16384
  • Type: Int
  • Unit: -
  • Is mutable: Yes
  • Description: The maximum row count returned by each scan thread in a scan.
  • Introduced in: -

scanner_thread_pool_queue_size​

  • Default: 102400
  • Type: Int
  • Unit: -
  • Is mutable: No
  • Description: The number of scan tasks supported by the storage engine.
  • Introduced in: -

scanner_thread_pool_thread_num​

  • Default: 48
  • Type: Int
  • Unit: -
  • Is mutable: Yes
  • Description: The number of threads which the storage engine used for concurrent storage volume scanning. All threads are managed in the thread pool.
  • Introduced in: -

string_prefix_zonemap_prefix_len​

  • Default: 16
  • Type: Int
  • Unit: -
  • Is mutable: Yes
  • Description: Prefix length used for string ZoneMap min/max when enable_string_prefix_zonemap is enabled.
  • Introduced in: -

jvm_call_thread_pool_size​

  • Default: 4
  • Type: Int
  • Unit: Threads
  • Is mutable: No
  • Description: Sets the size of the JVM call PriorityThreadPool used for internal JNI work that must run on pthreads, such as HDFS/libhdfs close and stat operations and JNI global reference cleanup. This pool is separate from udf_thread_pool_size so generic JVM work does not compete with Java UDF execution.
  • Introduced in: -

udf_thread_pool_size​

  • Default: 1
  • Type: Int
  • Unit: Threads
  • Is mutable: No
  • Description: Sets the size of the Java UDF call PriorityThreadPool owned by JavaEnv (used for executing Java UDF-related tasks). The value is used as the pool thread count and also as the pool queue capacity when constructing the thread pool (PriorityThreadPool("udf", thread_num, queue_size)). Increase to allow more concurrent Java UDF executions; keep small to avoid excessive CPU and memory contention.
  • Introduced in: v3.2.0

update_memory_limit_percent​

  • Default: 60
  • Type: Int
  • Unit: Percent
  • Is mutable: No
  • Description: Fraction of the BE process memory reserved for update-related memory and caches. During startup RuntimeEnv computes the MemTracker for updates as process_mem_limit * clamp(update_memory_limit_percent, 0, 100) / 100. UpdateManager also uses this percentage to size its primary-index/index-cache capacity (index cache capacity = RuntimeEnv::process_mem_limit * update_memory_limit_percent / 100). The HTTP config update logic registers a callback that calls update_primary_index_memory_limit on the update managers, so changes would be applied to the update subsystem if the config were changed. Increasing this value gives more memory to update/primary-index paths (reducing memory available for other pools); decreasing it reduces update memory and cache capacity. Values are clamped to the range 0–100.
  • Introduced in: v3.2.0

enable_vector_index_block_cache​

  • Default: true
  • Type: Boolean
  • Unit: -
  • Is mutable: Yes
  • Description: Whether to cache IVFPQ indexes by inverted-list block. When enabled, StarRocks loads and caches only the IVFPQ blocks needed by a query. When disabled, it caches the whole IVFPQ index file. This configuration does not change HNSW caching.
  • Introduced in: -

config_vector_index_build_concurrency​

  • Default: 8
  • Type: Int
  • Unit: Threads
  • Is mutable: Yes
  • Description: Number of OpenMP threads used by each vector index build. StarRocks clamps the effective value to at least 1. Together with vector_index_build_max_cpu_ratio, this item also determines the size of the asynchronous vector index build thread pool.
  • Introduced in: -

config_vector_index_default_build_threshold​

  • Default: 10000
  • Type: Int
  • Unit: Rows
  • Is mutable: Yes
  • Description: Default minimum number of rows in a Segment required to build a vector index. The index property index_build_threshold overrides this value. For IVFPQ, the effective threshold is at least nlist so that the index has enough rows for training.
  • Introduced in: -

vector_index_build_max_cpu_ratio​

  • Default: 0.5
  • Type: Double
  • Unit: Ratio
  • Is mutable: Yes
  • Description: Target fraction of BE/CN CPU cores available to asynchronous vector index builds. The build CPU budget is max(2, floor(cpu_cores * value)), and the build pool size is derived from this budget and config_vector_index_build_concurrency. Because the minimum budget is 2, small machines or small ratios can exceed the configured fraction.
  • Introduced in: -
  • Default: true
  • Type: Boolean
  • Unit: -
  • Is mutable: Yes
  • Description: Master switch for adaptive ef_search scaling on HNSW vector indexes. When enabled, BE scales the effective ef_search per segment based on its row count, so recall is preserved when compaction enlarges segments without forcing manual ef_search retuning. Set to false to disable scaling and use the user-supplied ef_search literally.
  • Introduced in: -

vector_query_cache_capacity​

  • Default: 20%
  • Type: String
  • Unit: Bytes, with unit suffix (K/M/G/T) or percentage of BE mem_limit (%)
  • Is mutable: Yes
  • Description: Total capacity of the SR-owned vector index cache, which holds HNSW whole-index entries and IVF-PQ per-list block entries (when enable_vector_index_block_cache=true) in a single LRU. Applied at BE startup and on every HTTP /api/update_config call. Accepts absolute bytes (e.g. 4294967296), units (4G, 512M), or a percentage of the BE process memory limit (20%). Behavior change in v4.2.0: prior versions accepted only an absolute byte count (default 512MB); upgrading without changing this config will resize the cache to 20% of BE memory.
  • Introduced in: v3.4.0

enable_vector_index_cache_on_build​

  • Default: false
  • Type: Boolean
  • Unit: -
  • Is mutable: Yes
  • Description: Whether building a vector index also inserts the index it just built into the vector index cache sized by vector_query_cache_capacity. Disabled by default, because the cache is sized for the query working set: letting loads and compactions push freshly built indexes into it evicts entries that running queries depend on, in exchange for warming indexes that may never be queried. The query path still populates the cache on demand the first time an index is read. Enable this only when index build and queries run on the same node and newly built data is queried immediately, to save that first read-back of the index file. A change takes effect for index builds started after it.
  • Introduced in: v4.2.0

vector_index_cache_expire_sec​

  • Default: 900
  • Type: Int
  • Unit: Seconds
  • Is mutable: Yes
  • Description: Idle time before an unused vector index cache entry expires. The timer starts when the last cache handle is released, and a cache hit refreshes it when that handle is later released. Entries pinned by running queries are not removed. IVF-PQ list blocks are released together with their owning index entry and do not have independent TTLs. Values less than or equal to 0 disable expiration. A runtime change applies to subsequent handle releases and does not rewrite existing entry deadlines.
  • Introduced in: v4.2.0

enable_vector_index_cache_async_load_on_miss​

  • Default: false
  • Type: Boolean
  • Unit: -
  • Is mutable: Yes
  • Description: Whether a top-level vector index cache miss makes the current query fall back to brute-force search while the index is loaded into the cache in the background. Concurrent misses for the same index share one background load and continue to use brute-force search until the index is ready. When this item is false, a cache miss loads the index synchronously. Enable it only when the expected cold-load latency is higher than the brute-force search cost and the cache has enough capacity for the query working set. A runtime change affects vector index readers initialized after the change.
  • Introduced in: v4.2.0

vector_index_cache_async_load_threads​

  • Default: 8
  • Type: Int
  • Unit: Threads
  • Is mutable: No
  • Description: Maximum number of workers in the vector index cache background-load thread pool. Workers are created on demand and retire after being idle for 60 seconds. Values less than or equal to 0 are adjusted to 1. This item is read when StorageEnv initializes and requires a BE/CN restart to change.
  • Introduced in: v4.2.0

vector_index_cache_loading_wait_timeout_ms​

  • Default: 5000
  • Type: Int
  • Unit: Milliseconds
  • Is mutable: Yes
  • Description: Maximum time that each synchronous cache caller waits for an in-progress vector index load. When the wait times out, the caller receives a cache miss so the query path can fall back to brute-force search. The existing loader is not canceled and continues loading the index in the background. Each later caller starts its own wait with the configured timeout. Values less than or equal to 0 disable waiting. A runtime change applies to subsequent waits.
  • Introduced in: v4.2.0

vector_adaptive_ef_alpha​

  • Default: 1.0
  • Type: Double
  • Unit: -
  • Is mutable: Yes
  • Description: Growth slope for adaptive ef_search. The scaling factor is 1 + alpha * log2(segment_rows / vector_adaptive_ef_baseline_rows). Higher values raise recall more aggressively at the cost of higher CPU per query; lower values trim CPU at the cost of recall on large segments. Effective only when enable_vector_adaptive_search is true and segment_rows > vector_adaptive_ef_baseline_rows.
  • Introduced in: -

vector_adaptive_ef_baseline_rows​

  • Default: 300000
  • Type: Int
  • Unit: Rows
  • Is mutable: Yes
  • Description: Segment row count below which adaptive ef_search does not scale. Segments with fewer rows than this threshold use the user-supplied ef_search directly; segments above this threshold receive a higher effective ef_search to compensate for the larger HNSW graph. Effective only when enable_vector_adaptive_search is true.
  • Introduced in: -

vector_adaptive_ef_cap​

  • Default: 8.0
  • Type: Double
  • Unit: -
  • Is mutable: Yes
  • Description: Upper bound on the adaptive ef_search multiplier. Caps the worst-case CPU and latency cost of the scaling formula even on extremely large segments. Effective only when enable_vector_adaptive_search is true.
  • Introduced in: -

vector_index_brute_selectivity_threshold​

  • Default: 0.01
  • Type: Double
  • Unit: Ratio
  • Is mutable: Yes
  • Description: Selectivity threshold for routing a vector query with residual scalar filters to exact scoring instead of a filtered HNSW traversal. If the matched rows are no more than this fraction of the Segment, StarRocks scores the filtered candidates directly. Set to 0 to disable this ratio check. The separate short circuit for a candidate count no greater than the query k remains enabled.
  • Introduced in: -

enable_vector_index_topk_underfill_fallback​

  • Default: false
  • Type: Boolean
  • Unit: -
  • Is mutable: Yes
  • Description: Whether to fall back to exact brute-force scoring when a filtered top-k vector index search returns fewer rows than min(k, matched candidate count). When this parameter is true, StarRocks rescans the matched candidates exactly to fill the result up to k. When it is false, StarRocks returns the vector index result without this count-based fallback. This parameter applies only to pre-filtered top-k searches; it does not apply to range searches, where fewer results can legitimately mean that no more candidates satisfy the requested radius.
  • Introduced in: -

vector_index_build_flush_threshold_rows​

  • Default: 262144
  • Type: Int
  • Unit: Rows
  • Is mutable: Yes
  • Description: Maximum rows held in each vector index builder's staging buffer before the rows are added to the in-memory index. This bounds the staging-buffer memory used when building HNSW Flat indexes, but does not limit the trained index itself. Set to 0 to disable intermediate flushing and buffer the whole Segment.
  • Introduced in: -

vector_chunk_size​

  • Default: 4096
  • Type: Int
  • Unit: Rows
  • Is mutable: No
  • Description: The number of rows per vectorized chunk (batch) used throughout the execution and storage code paths. This value controls Chunk and RuntimeState batch_size creation, affects operator throughput, memory footprint per operator, spill and sort buffer sizing, and I/O heuristics (for example, ORC writer natural write size). Increasing it can improve CPU and I/O efficiency for wide/CPU-bound workloads but raises peak memory usage and can increase latency for small-result queries. Tune only when profiling shows batch-size is a bottleneck; otherwise keep the default for balanced memory and performance.
  • Introduced in: v3.2.0

Loading and unloading​

clear_transaction_task_worker_count​

  • Default: 1
  • Type: Int
  • Unit: -
  • Is mutable: No
  • Description: The number of threads used for clearing transaction.
  • Introduced in: -

column_mode_partial_update_insert_batch_size​

  • Default: 4096
  • Type: Int
  • Unit: -
  • Is mutable: Yes
  • Description: Batch size for column mode partial update when processing inserted rows. If this item is set to 0 or negative, it will be clamped to 1 to avoid infinite loop. This item controls the number of newly inserted rows processed in each batch. Larger values can improve write performance but will consume more memory.
  • Introduced in: v3.5.10, v4.0.2

partial_update_memory_limit_per_worker​

  • Default: 2147483648
  • Type: Int
  • Unit: Bytes
  • Is mutable: Yes
  • Description: Maximum memory per worker thread for partial update operations. Controls the memory footprint of individual worker threads when processing partial updates.
  • Introduced in: -

enable_load_spill_parallel_merge​

  • Default: true
  • Type: Boolean
  • Unit: -
  • Is mutable: Yes
  • Description: Specifies whether to enable parallel spill merge within a single tablet. Enabling this can improve the performance of spill merge during data loading.
  • Introduced in: -

enable_parallel_memtable_finalize​

  • Default: true
  • Type: Boolean
  • Unit: -
  • Is mutable: Yes
  • Description: Specifies whether to enable parallel memtable finalize when loading data to lake tables (shared-data mode). When enabled, the memtable finalize operation (sort/aggregate) is moved from the write thread to the flush thread, allowing the write thread to continue inserting data into a new memtable while the previous one is being finalized and flushed in parallel. This can significantly improve load throughput by overlapping CPU-intensive finalize operations with I/O-bound flush operations. Note that this optimization is automatically disabled when auto-increment columns need to be filled, as auto-increment ID assignment must happen before the memtable is submitted for flush.
  • Introduced in: -

allow_list_object_for_random_bucketing_on_cache_miss​

  • Default: true
  • Type: Boolean
  • Unit: -
  • Is mutable: Yes
  • Description: Controls whether to allow object-storage LIST fallback when lake metadata cache misses during random bucketing size checks. true means fallback to LIST metadata files to compute base size (historical behavior, more accurate size estimation). false means skip LIST and use base_size = 0, which reduces LIST object requests but may delay immutable marking due to less accurate size estimation.
  • Introduced in: 4.1.0, 4.0.7, 3.5.15

enable_stream_load_verbose_log​

  • Default: false
  • Type: Boolean
  • Unit: -
  • Is mutable: Yes
  • Description: Specifies whether to log the HTTP requests and responses for Stream Load jobs.
  • Introduced in: v2.5.17, v3.0.9, v3.1.6, v3.2.1

flush_thread_num_per_store​

  • Default: 2
  • Type: Int
  • Unit: -
  • Is mutable: Yes
  • Description: Number of threads that are used for flushing MemTable in each store.
  • Introduced in: -

lake_flush_thread_num_per_store​

  • Default: 0
  • Type: Int
  • Unit: -
  • Is mutable: Yes
  • Description: Number of threads that are used for flushing MemTable in each store in a shared-data cluster. When this value is set to 0, the system uses twice of the CPU core count as the value. When this value is set to less than 0, the system uses the product of its absolute value and the CPU core count as the value.
  • Introduced in: v3.1.12, 3.2.7

load_data_reserve_hours​

  • Default: 4
  • Type: Int
  • Unit: Hours
  • Is mutable: No
  • Description: The reservation time for the files produced by small-scale loadings.
  • Introduced in: -

load_error_log_reserve_hours​

  • Default: 48
  • Type: Int
  • Unit: Hours
  • Is mutable: Yes
  • Description: The time for which data loading logs are reserved.
  • Introduced in: -

enable_rejected_record_sync​

  • Default: false
  • Type: Boolean
  • Unit: -
  • Is mutable: Yes
  • Description: Feature flag for the rejected records sync daemon. When true, the BE-resident RejectedRecordSyncDaemon scans per-fragment JSON Lines files produced by RejectedRecordWriter and batch-ships them to the _statistics_.rejected_records system table via merge-commit Stream Load. Defaults to false during the phased rollout so a cluster upgrading to a binary that contains the daemon does not start writing to the new system table until the operator explicitly opts in.
  • Introduced in: -

rejected_record_sync_interval_sec​

  • Default: 30
  • Type: Int
  • Unit: Seconds
  • Is mutable: Yes
  • Description: How often the RejectedRecordSyncDaemon wakes to scan local JSON Lines files. A tick with no new files is a no-op. Smaller values reduce the latency with which rejected rows become queryable in _statistics_.rejected_records; larger values reduce filesystem pressure.
  • Introduced in: -

rejected_record_sync_max_batch_rows​

  • Default: 10000
  • Type: Int
  • Unit: Rows
  • Is mutable: Yes
  • Description: Row cap on a single merge-commit Stream Load batch. Enforced per-line while the daemon concatenates files, so a single giant file still obeys the cap; larger backlogs are split across consecutive ticks. The cap bounds Stream Load transaction size so one oversized flush cannot back-pressure unrelated loads.
  • Introduced in: -

rejected_record_sync_max_batch_bytes​

  • Default: 33554432 (32 MiB)
  • Type: Int64
  • Unit: Bytes
  • Is mutable: Yes
  • Description: Byte cap on the Stream Load payload the daemon assembles per post. Enforced together with rejected_record_sync_max_batch_rows so loads with very wide rows or very long error messages cannot build a multi-gigabyte payload in memory. When the cap is hit mid-file the daemon commits the current batch and starts a fresh payload; the file is retained until all of its rows have shipped. Keep at or below the FE's streaming_load_max_mb to avoid FE-side rejection.
  • Introduced in: -

rejected_record_sync_max_backoff_sec​

  • Default: 600
  • Type: Int
  • Unit: Seconds
  • Is mutable: Yes
  • Description: Upper bound on the tick interval when post_to_stream_load has been failing persistently. The daemon doubles its sleep after every failed tick (rejected_record_sync_interval_sec << consecutive_failures) until this cap is reached, then holds there until a tick succeeds. Prevents a multi-hour FE outage from producing one log line + one _sync_failures increment every rejected_record_sync_interval_sec seconds. A Uninitialized status ("master FE not yet known") never advances the backoff counter.
  • Introduced in: -

rejected_record_local_retention_hours​

  • Default: 24
  • Type: Int
  • Unit: Hours
  • Is mutable: Yes
  • Description: Maximum age for per-fragment JSON Lines files before the daemon garbage-collects them. Intended as a last-resort cap: sync failures normally retry on every tick, but a misconfigured cluster (FE unreachable, system table dropped, auth broken) cannot slowly fill the store path when this setting is in effect.
  • Introduced in: -

rejected_record_sync_post_timeout_sec​

  • Default: 60
  • Type: Int
  • Unit: Seconds
  • Is mutable: Yes
  • Description: Per-request timeout for the daemon's Stream Load PUT to the FE. Merge-commit batches can briefly sit in the FE-side commit queue while other BEs' concurrent posts coalesce into the same transaction, so this is intentionally set looser than the default query timeout.
  • Introduced in: -

load_process_max_memory_limit_bytes​

  • Default: 107374182400
  • Type: Int
  • Unit: Bytes
  • Is mutable: No
  • Description: The maximum size limit of memory resources that can be taken up by all load processes on a BE node.
  • Introduced in: -

load_spill_memory_usage_per_merge​

  • Default: 1073741824
  • Type: Int
  • Unit: Bytes
  • Is mutable: Yes
  • Description: The maximum memory usage per merge operation during spill merge. Default is 1 GB (1073741824 bytes). This parameter controls the memory consumption of individual merge tasks during data loading spill merge to prevent excessive memory usage.
  • Introduced in: -

max_consumer_num_per_group​

  • Default: 3
  • Type: Int
  • Unit: -
  • Is mutable: Yes
  • Description: The maximum number of consumers in a consumer group of Routine Load.
  • Introduced in: -

max_runnings_transactions_per_txn_map​

  • Default: 100
  • Type: Int
  • Unit: -
  • Is mutable: Yes
  • Description: The maximum number of transactions that can run concurrently in each partition.
  • Introduced in: -

number_tablet_writer_threads​

  • Default: 0
  • Type: Int
  • Unit: -
  • Is mutable: Yes
  • Description: The number of tablet writer threads used in ingestion, such as Stream Load, Broker Load and Insert. When the parameter is set to less than or equal to 0, the system uses half of the number of CPU cores, with a minimum of 16. When the parameter is set to greater than 0, the system uses that value. This configuration is changed to dynamic from v3.1.7 onwards.
  • Introduced in: -

push_worker_count_high_priority​

  • Default: 3
  • Type: Int
  • Unit: -
  • Is mutable: No
  • Description: The number of threads used to handle a load task with HIGH priority.
  • Introduced in: -

push_worker_count_normal_priority​

  • Default: 3
  • Type: Int
  • Unit: -
  • Is mutable: No
  • Description: The number of threads used to handle a load task with NORMAL priority.
  • Introduced in: -

streaming_load_max_batch_size_mb​

  • Default: 100
  • Type: Int
  • Unit: MB
  • Is mutable: Yes
  • Description: The maximum size of a JSON file that can be streamed into StarRocks.
  • Introduced in: -

streaming_load_max_mb​

  • Default: 102400
  • Type: Int
  • Unit: MB
  • Is mutable: Yes
  • Description: The maximum size of a file that can be streamed into StarRocks. From v3.0, the default value has been changed from 10240 to 102400.
  • Introduced in: -

streaming_load_rpc_max_alive_time_sec​

  • Default: 1200
  • Type: Int
  • Unit: Seconds
  • Is mutable: No
  • Description: The RPC timeout for Stream Load.
  • Introduced in: -

transaction_publish_version_thread_pool_num_min​

  • Default: 0
  • Type: Int
  • Unit: Threads
  • Is mutable: Yes
  • Description: Minimum number of threads in the Publish Version thread pool. The pool can shrink to this value when idle. 0 means no fixed lower bound.
  • Introduced in: -

transaction_publish_version_thread_pool_idle_time_ms​

  • Default: 60000
  • Type: Int
  • Unit: Milliseconds
  • Is mutable: No
  • Description: The idle time before a thread is reclaimed by the Publish Version thread pool.
  • Introduced in: -

transaction_publish_version_worker_count​

  • Default: 0
  • Type: Int
  • Unit: -
  • Is mutable: Yes
  • Description: The maximum number of threads used to publish a version. When this value is set to less than or equal to 0, the system uses the CPU core count as the value, so as to avoid insufficient thread resources when import concurrency is high but only a fixed number of threads are used. From v2.5, the default value has been changed from 8 to 0.
  • Introduced in: -

use_mmap_allocate_chunk​

  • Default: false
  • Type: Boolean
  • Unit: -
  • Is mutable: No
  • Description: Whether to use anonymous mmap (MAP_ANONYMOUS | MAP_PRIVATE) for chunk allocation. When enabled, many VM mappings are created; you must raise vm.max_map_count (e.g., echo 262144 > /proc/sys/vm/max_map_count) and set a large chunk_reserved_bytes_limit, otherwise frequent map/unmap operations will cause severe performance degradation.
  • Introduced in: v3.2.0

write_buffer_size​

  • Default: 104857600
  • Type: Int
  • Unit: Bytes
  • Is mutable: Yes
  • Description: The buffer size of MemTable in the memory. This configuration item is the threshold to trigger a flush.
  • Introduced in: -

broker_write_timeout_seconds​

  • Default: 30
  • Type: int
  • Unit: Seconds
  • Is mutable: No
  • Description: Timeout (in seconds) used by backend broker operations for write/IO RPCs. The value is multiplied by 1000 to produce millisecond timeouts and is passed as the default timeout_ms to BrokerFileSystem and BrokerServiceConnection instances (e.g., file export and snapshot upload/download). Increase this when brokers or network are slow or when transferring large files to avoid premature timeouts; decreasing it may cause broker RPCs to fail earlier. This value is defined in common/config and is applied at process start (not dynamically reloadable).
  • Introduced in: v3.2.0

enable_load_channel_rpc_async​

  • Default: true
  • Type: Boolean
  • Unit: -
  • Is mutable: Yes
  • Description: When enabled, handling of load-channel open RPCs (for example, PTabletWriterOpen) is offloaded from the BRPC worker to a dedicated thread pool: the request handler creates a ChannelOpenTask and submits it to the internal _async_rpc_pool instead of running LoadChannelMgr::_open inline. This reduces work and blocking inside BRPC threads and allows tuning concurrency via load_channel_rpc_thread_pool_num and load_channel_rpc_thread_pool_queue_size. If the thread pool submission fails (when pool is full or shut down), the request is canceled and an error status is returned. The pool is shut down on LoadChannelMgr::close(), so consider capacity and lifecycle when you want to enable this feature so as to avoid request rejections or delayed processing.
  • Introduced in: v3.5.0

enable_load_chunk_all_null_encoding​

  • Default: true
  • Type: Boolean
  • Unit: -
  • Is mutable: No
  • Description: Whether this BE takes part in the compact encoding of all-NULL columns on the tablet sink RPC, the hop on which a load routes rows to the BE that holds the target tablet. When enabled, a receiving BE advertises support for the encoding in its tablet writer open response, and a sending BE that sees the advertisement sends a row count in place of the payload of any column whose rows are all NULL, rather than a null flag and an offset for every row. This mainly helps wide tables in which most columns hold no data, and reduces both the bytes sent on this hop and the serialization work on either end. The encoding is used only when both ends of the RPC agree on it, so a mixed-version cluster or a downgrade falls back to the original layout automatically. Set this to false on either the sending or the receiving BE to turn the encoding off for loads that pass through that node. This item is not dynamically configurable: a receiving BE uses it both to advertise support and to decide whether to apply the encoding a sender declares, and changing it at runtime would leave those two decisions inconsistent.
  • Introduced in: v4.2.0

enable_load_diagnose​

  • Default: true
  • Type: Boolean
  • Unit: -
  • Is mutable: Yes
  • Description: When enabled, StarRocks will attempt an automated load diagnosis from BE OlapTableSink/NodeChannel after a brpc timeout matching "[E1008]Reached timeout". The code creates a PLoadDiagnoseRequest and sends an RPC to the remote LoadChannel to collect a profile and/or stack trace (controlled by load_diagnose_rpc_timeout_profile_threshold_ms and load_diagnose_rpc_timeout_stack_trace_threshold_ms). The diagnose RPC uses load_diagnose_send_rpc_timeout_ms as its timeout. Diagnosis is skipped if a diagnose request is already in progress. Enabling this produces additional RPCs and profiling work on target nodes; disable on sensitive production workloads to avoid extra overhead.
  • Introduced in: v3.5.0

enable_load_segment_parallel​

  • Default: false
  • Type: Boolean
  • Unit: -
  • Is mutable: No
  • Description: When enabled, rowset segment loading and rowset-level reads are performed concurrently using StarRocks background thread pools (ExecEnv::load_segment_thread_pool and ExecEnv::load_rowset_thread_pool). Rowset::load_segments and TabletReader::get_segment_iterators submit per-segment or per-rowset tasks to these pools, falling back to serial loading and logging a warning if submission fails. Enable this to reduce read/load latency for large rowsets at the cost of increased CPU/IO concurrency and memory pressure. Note: parallel loading can change the load completion order of segments and therefore prevents partial compaction (code checks _parallel_load and disables partial compaction when enabled); consider implications for operations that rely on segment order.
  • Introduced in: v3.3.0, v3.4.0, v3.5.0

enable_streaming_load_thread_pool​

  • Default: true
  • Type: Boolean
  • Unit: -
  • Is mutable: Yes
  • Description: Controls whether streaming load scanners are submitted to the dedicated streaming load thread pool. When enabled and a query is a LOAD with TLoadJobType::STREAM_LOAD, ConnectorScanNode submits scanner tasks to the streaming_load_thread_pool (which is configured with INT32_MAX threads and queue sizes, i.e. effectively unbounded). When disabled, scanners use the general thread_pool and its PriorityThreadPool submission logic (priority computation, try_offer/offer behavior). Enabling isolates streaming-load work from regular query execution to reduce interference; however, because the dedicated pool is effectively unbounded, enabling may increase concurrent threads and resource usage under heavy streaming-load traffic. This option is on by default and typically does not require modification.
  • Introduced in: v3.2.0

es_http_timeout_ms​

  • Default: 5000
  • Type: Int
  • Unit: Milliseconds
  • Is mutable: No
  • Description: HTTP connection timeout (in milliseconds) used by the ES network client in ESScanReader for Elasticsearch scroll requests. This value is applied via network_client.set_timeout_ms() before sending subsequent scroll POSTs and controls how long the client waits for an ES response during scrolling. Increase this value for slow networks or large queries to avoid premature timeouts; decrease to fail faster on unresponsive ES nodes. This setting complements es_scroll_keepalive, which controls the scroll context keep-alive duration.
  • Introduced in: v3.2.0

es_index_max_result_window​

  • Default: 10000
  • Type: Int
  • Unit: -
  • Is mutable: No
  • Description: Limits the maximum number of documents StarRocks will request from Elasticsearch in a single batch. StarRocks sets the ES request batch size to min(es_index_max_result_window, chunk_size) when building KEY_BATCH_SIZE for the ES reader. If an ES request exceeds the Elasticsearch index setting index.max_result_window, Elasticsearch returns HTTP 400 (Bad Request). Adjust this value when scanning large indexes or increase the ES index.max_result_window on the Elasticsearch side to permit larger single requests.
  • Introduced in: v3.2.0

ignore_load_tablet_failure​

  • Default: false
  • Type: Boolean
  • Unit: -
  • Is mutable: No
  • Description: When this item is set to false, the system will treat any tablet header load failures (non-NotFound and non-AlreadyExist errors) as fatal: the code logs the error and calls LOG(FATAL) to stop the BE process. When it is set to true, the BE continues startup despite such per-tablet load errors — failed tablet IDs are recorded and skipped while successful tablets are still loaded. Note that this parameter does NOT suppress fatal errors from the RocksDB meta scan itself, which always cause the process to quit.
  • Introduced in: v3.2.0

load_channel_abort_clean_up_delay_seconds​

  • Default: 600
  • Type: Int
  • Unit: Seconds
  • Is mutable: Yes
  • Description: Controls how long (in seconds) the system keeps the load IDs of aborted load channels before removing them from _aborted_load_channels. When a load job is cancelled or fails, the load ID stays recorded so any late-arriving load RPCs can be rejected immediately; once the delay expires, the entry is cleaned during the periodic background sweep (minimum sweep interval is 60 seconds). Setting the delay too low risks accepting stray RPCs after an abort, while setting it too high may retain state and consume resources longer than necessary. Tune this to balance correctness of late-request rejection and resource retention for aborted loads.
  • Introduced in: v3.5.11, v4.0.4

load_channel_rpc_thread_pool_num​

  • Default: -1
  • Type: Int
  • Unit: Threads
  • Is mutable: Yes
  • Description: Maximum number of threads for the load-channel async RPC thread pool. When set to less than or equal to 0 (default -1) the pool size is auto-set to the number of CPU cores (CpuInfo::num_cores()). The configured value is used as ThreadPoolBuilder's max threads and the pool's min threads is set to min(5, max_threads). The pool queue size is controlled separately by load_channel_rpc_thread_pool_queue_size. This setting was introduced to align the async RPC pool size with brpc workers' default (brpc_num_threads) so behavior remains compatible after switching load RPC handling from synchronous to asynchronous. Changing this config at runtime triggers ExecEnv::GetInstance()->load_channel_mgr()->async_rpc_pool()->update_max_threads(...).
  • Introduced in: v3.5.0

load_channel_rpc_thread_pool_queue_size​

  • Default: 1024000
  • Type: int
  • Unit: Count
  • Is mutable: No
  • Description: Sets the maximum pending-task queue size for the Load channel RPC thread pool created by LoadChannelMgr. This thread pool executes asynchronous open requests when enable_load_channel_rpc_async is enabled; the pool size is paired with load_channel_rpc_thread_pool_num. The large default (1024000) aligns with brpc workers' defaults to preserve behavior after switching from synchronous to asynchronous handling. If the queue is full, ThreadPool::submit() will fail and the incoming open RPC is cancelled with an error, causing the caller to receive a rejection. Increase this value to buffer larger bursts of concurrent open requests; reducing it tightens backpressure but may cause more rejections under load.
  • Introduced in: v3.5.0

load_diagnose_rpc_timeout_profile_threshold_ms​

  • Default: 60000
  • Type: Int
  • Unit: Milliseconds
  • Is mutable: Yes
  • Description: When a load RPC times out (error contains "[E1008]Reached timeout") and enable_load_diagnose is true, this threshold controls whether a full profiling diagnose is requested. If the request-level RPC timeout _rpc_timeout_ms is greater than load_diagnose_rpc_timeout_profile_threshold_ms, profiling is enabled for that diagnose. For smaller _rpc_timeout_ms values, profiling is sampled once every 20 timeouts to avoid frequent heavy diagnostics for real-time/short-timeout loads. This value affects the profile flag in the PLoadDiagnoseRequest sent; stack-trace behavior is controlled separately by load_diagnose_rpc_timeout_stack_trace_threshold_ms and send timeout by load_diagnose_send_rpc_timeout_ms.
  • Introduced in: v3.5.0

load_diagnose_rpc_timeout_stack_trace_threshold_ms​

  • Default: 600000
  • Type: Int
  • Unit: Milliseconds
  • Is mutable: Yes
  • Description: Threshold (in ms) used to decide when to request remote stack traces for long-running load RPCs. When a load RPC times out with a timeout error and the effective RPC timeout (_rpc_timeout_ms) exceeds this value, OlapTableSink/NodeChannel will include stack_trace=true in a load_diagnose RPC to the target BE so the BE can return stack traces for debugging. LocalTabletsChannel::SecondaryReplicasWaiter also triggers a best-effort stack-trace diagnose from the primary if waiting for secondary replicas exceeds this interval. This behavior requires enable_load_diagnose and uses load_diagnose_send_rpc_timeout_ms for the diagnose RPC timeout; profiling is gated separately by load_diagnose_rpc_timeout_profile_threshold_ms. Lowering this value increases how aggressively stack traces are requested.
  • Introduced in: v3.5.0

load_diagnose_send_rpc_timeout_ms​

  • Default: 2000
  • Type: Int
  • Unit: Milliseconds
  • Is mutable: Yes
  • Description: Timeout (in milliseconds) applied to diagnosis-related brpc calls initiated by BE load paths. It is used to set the controller timeout for load_diagnose RPCs (sent by NodeChannel/OlapTableSink when a LoadChannel brpc call times out) and for replica-status queries (used by SecondaryReplicasWaiter / LocalTabletsChannel when checking primary replica state). Choose a value high enough to allow the remote side to respond with profile or stack-trace data, but not so high that failure handling is delayed. This parameter works together with enable_load_diagnose, load_diagnose_rpc_timeout_profile_threshold_ms, and load_diagnose_rpc_timeout_stack_trace_threshold_ms which control when and what diagnostic information is requested.
  • Introduced in: v3.5.0

load_fp_brpc_timeout_ms​

  • Default: -1
  • Type: Int
  • Unit: Milliseconds
  • Is mutable: Yes
  • Description: Overrides the per-channel brpc RPC timeout used by OlapTableSink when the node_channel_set_brpc_timeout fail point is triggered. If set to a positive value, NodeChannel will set its internal _rpc_timeout_ms to this value (in milliseconds) causing open/add-chunk/cancel RPCs to use the shorter timeout and enabling simulation of brpc timeouts that produce the "[E1008]Reached timeout" error. Default (-1) disables the override. Changing this value is intended for testing and fault injection; small values may produce false timeouts and trigger load diagnostics (see enable_load_diagnose, load_diagnose_rpc_timeout_profile_threshold_ms, load_diagnose_rpc_timeout_stack_trace_threshold_ms, and load_diagnose_send_rpc_timeout_ms).
  • Introduced in: v3.5.0

load_fp_tablets_channel_add_chunk_block_ms​

  • Default: -1
  • Type: Int
  • Unit: Milliseconds
  • Is mutable: Yes
  • Description: When enabled (set to a positive milliseconds value) this fail-point configuration makes TabletsChannel::add_chunk sleep for the specified time during load processing. It is used to simulate BRPC timeout errors (e.g., "[E1008]Reached timeout") and to emulate an expensive add_chunk operation that increases load latency. A value less than or equal to 0 (default -1) disables the injection. Intended for testing fault handling, timeouts, and replica synchronization behavior — do not enable in normal production workloads as it delays write completion and can trigger upstream timeouts or replica aborts.
  • Introduced in: v3.5.0

load_segment_thread_pool_num_max​

  • Default: 128
  • Type: Int
  • Unit: -
  • Is mutable: No
  • Description: Sets the maximum number of worker threads for BE load-related thread pools. This value is used by ThreadPoolBuilder to limit threads for both load_rowset_pool and load_segment_pool in exec_env.cpp, controlling concurrency for processing loaded rowsets and segments (e.g., decoding, indexing, writing) during streaming and batch loads. Increasing this value raises parallelism and can improve load throughput but also increases CPU, memory usage, and potential contention; decreasing it limits concurrent load processing and may reduce throughput. Tune together with load_segment_thread_pool_queue_size and streaming_load_thread_pool_idle_time_ms. Change requires BE restart.
  • Introduced in: v3.3.0, v3.4.0, v3.5.0

load_segment_thread_pool_queue_size​

  • Default: 10240
  • Type: Int
  • Unit: Tasks
  • Is mutable: No
  • Description: Sets the maximum queue length (number of pending tasks) for the load-related thread pools created as "load_rowset_pool" and "load_segment_pool". These pools use load_segment_thread_pool_num_max for their max thread count and this configuration controls how many load segment/rowset tasks can be buffered before the ThreadPool's overflow policy takes effect (further submissions may be rejected or blocked depending on the ThreadPool implementation). Increase to allow more pending load work (uses more memory and can raise latency); decrease to limit buffered load concurrency and reduce memory usage.
  • Introduced in: v3.3.0, v3.4.0, v3.5.0

max_pulsar_consumer_num_per_group​

  • Default: 10
  • Type: Int
  • Unit: -
  • Is mutable: Yes
  • Description: Controls the maximum number of Pulsar consumers that may be created in a single data consumer group for routine load on a BE. Because cumulative acknowledge is not supported for multi-topic subscriptions, each consumer subscribes exactly one topic/partition; if the number of partitions in pulsar_info->partitions exceeds this value, group creation fails with an error advising to increase max_pulsar_consumer_num_per_group on the BE or add more BEs. This limit is enforced when constructing a PulsarDataConsumerGroup and prevents a BE from hosting more than this many consumers for one routine load group. For Kafka routine load, max_consumer_num_per_group is used instead.
  • Introduced in: v3.2.0

pull_load_task_dir​

  • Default: ${STARROCKS_HOME}/var/pull_load
  • Type: string
  • Unit: -
  • Is mutable: No
  • Description: Filesystem path where the BE stores data and working files for "pull load" tasks (downloaded source files, task state, temporary output, etc.). The directory must be writable by the BE process and have sufficient disk space for incoming loads. The default is relative to STARROCKS_HOME; tests create and expect this directory to exist (see test configuration).
  • Introduced in: v3.2.0

routine_load_kafka_timeout_second​

  • Default: 10
  • Type: Int
  • Unit: Seconds
  • Is mutable: No
  • Description: Timeout in seconds used for Kafka-related routine load operations. When a client request does not specify a timeout, routine_load_kafka_timeout_second is used as the default RPC timeout (converted to milliseconds) for get_info. It is also used as the per-call consume poll timeout for the librdkafka consumer (converted to milliseconds and capped by remaining runtime). Note: the internal get_info path reduces this value to 80% before passing it to librdkafka to avoid FE-side timeout races. Set this to a value that balances timely failure reporting and sufficient time for network/broker responses; changes require a restart because the setting is not mutable.
  • Introduced in: v3.2.0

routine_load_pulsar_timeout_second​

  • Default: 10
  • Type: Int
  • Unit: Seconds
  • Is mutable: No
  • Description: Default timeout (in seconds) that the BE uses for Pulsar-related routine load operations when the request does not supply an explicit timeout. Specifically, PInternalServiceImplBase::get_pulsar_info multiplies this value by 1000 to form the millisecond timeout passed to the routine load task executor methods that fetch Pulsar partition metadata and backlog. Increase to allow slower Pulsar responses at the cost of longer failure detection; decrease to fail faster on slow brokers. Analogous to routine_load_kafka_timeout_second used for Kafka.
  • Introduced in: v3.2.0

streaming_load_thread_pool_idle_time_ms​

  • Default: 2000
  • Type: Int
  • Unit: Milliseconds
  • Is mutable: No
  • Description: Sets the thread idle timeout (in milliseconds) for streaming-load related thread pools. The value is used as the idle timeout passed to ThreadPoolBuilder for the stream_load_io pool and also for load_rowset_pool and load_segment_pool. Threads in these pools are reclaimed when idle for this duration; lower values reduce idle resource usage but increase thread creation overhead, while higher values keep threads alive longer. The stream_load_io pool is used when enable_streaming_load_thread_pool is enabled.
  • Introduced in: v3.2.0

streaming_load_thread_pool_num_min​

  • Default: 0
  • Type: Int
  • Unit: -
  • Is mutable: No
  • Description: Minimum number of threads for the streaming load IO thread pool ("stream_load_io") created during ExecEnv initialization. The pool is built with set_max_threads(INT32_MAX) and set_max_queue_size(INT32_MAX) so it is effectively unbounded to avoid deadlocks for concurrent streaming loads. A value of 0 lets the pool start with no threads and grow on demand; setting a positive value reserves that many threads at startup. This pool is used when enable_streaming_load_thread_pool is true and its idle timeout is controlled by streaming_load_thread_pool_idle_time_ms. Overall concurrency is still constrained by fragment_pool_thread_num_max and webserver_num_workers; changing this value is rarely necessary and may increase resource usage if set too high.
  • Introduced in: v3.2.0