Data loading

📄️Loading Overview

Data loading is the process of cleansing and transforming raw data from various data sources based on your business requirements and loading the resulting data into StarRocks to facilitate analysis.

📄️Batch load data from Amazon S3

PhoenixAI provides the following options for loading data from AWS S3:

📄️Batch load data from Azure cloud storage

PhoenixAI provides two options for loading data from Microsoft Azure Storage:

📄️Load data from a local file system

PhoenixAI allows you to load data from a local file system by using Stream Load.

🗃Load from Confluent Cloud

2 items

📄️Load data from Amazon MSK

This topic introduces how to create a Routine Load job to stream messages (events) from Amazon Managed Streaming for Apache Kafka (Amazon MSK) into PhoenixAI.

📄️Load data using INSERT

This topic describes how to load data into StarRocks by using a SQL statement - INSERT.

📄️Load data from Amazon Kinesis

This topic describes how to load data from Amazon Kinesis into PhoenixAI.

📄️Load data using Stream Load transaction interface

From v2.4 onwards, StarRocks provides a Stream Load transaction interface to implement two-phase commit (2PC) for transactions that are run to load data from external systems such as Apache Flink® and Apache Kafka®. The Stream Load transaction interface helps improve the performance of highly concurrent stream loads.

📄️Realtime synchronization from MySQL

StarRocks supports multiple methods to synchronize data from MySQL to StarRocks in real time, delivering low latency real-time analytics of massive data.

📄️Realtime synchronization from PostgreSQL

How to use a Flink CDC pipeline to capture PostgreSQL changes and stream them into StarRocks Primary Key tables in real time.

📄️Continuously load data from Apache Flink®

StarRocks provides a self-developed connector named StarRocks Connector for Apache Flink® (Flink connector for short) to help you load data into a StarRocks table by using Flink. The basic principle is to accumulate the data and then load it all at a time into StarRocks through STREAM LOAD.

📄️Change data through loading

Primary Key tables provided by StarRocks allow you to make data changes to StarRocks tables by running Stream Load, Broker Load, or Routine Load jobs. These data changes include inserts, updates, and deletions. However, Primary Key tables do not support changing data by using Spark Load or INSERT.

📄️Strict mode

Strict mode is an optional property that you can configure for data loads. It affects the loading behavior and the final loaded data.