Skip to main content

Deploy Warehouse

From PhoenixAI Operator v1.9.0, PhoenixAIWarehouse CRD is introduced to manage the warehouse. This document describes how to deploy a warehouse.

A warehouse can be split further into CN Groups, each an independent copy of the same compute pool, typically one per availability zone. See Deploy CN Groups.

1. Prerequisites​

  1. PhoenixAI Operator >= v1.9.0. The latest version of the operator is recommended.

  2. An installed PhoenixAI Cluster. See Install with kubectl or Install with Helm for more details.

  3. PhoenixAI enterprise version >= v3.2.0.

  4. The PhoenixAIWarehouse CRD, present before the operator started. A fresh installation of the current charts satisfies this on its own: the operator chart ships the CRD next to the cluster one, and Install with kubectl applies both in step 1.1. No operator restart is needed there.

    An upgraded or older operator will not have it

    Operator charts only began shipping the warehouse CRD in v2.0.0, and helm upgrade never installs a chart's crds/ directory — so an operator upgraded from an earlier version does not gain the CRD from the chart, whatever version it now runs. The same holds for an operator installed with --skip-crds, or from a manifest that applied only the cluster CRD.

    Check rather than assume:

    kubectl get crd phoenixaiwarehouses.phoenixdata.ai
    kubectl -n phoenixai logs deployment/kube-anywhere-operator | grep PhoenixAIWarehouse

    The operator decides whether to run the warehouse controller once, at startup. If it started without the CRD, warehouses are never reconciled and nothing reports it: kubectl get paw shows the object with an empty STATUS, there are no pods and no events, and the grep above returns nothing. Install the CRD, then restart the operator:

    kubectl apply -f https://raw.githubusercontent.com/celerdata/phoenixai-kubernetes-operator/main/deploy/phoenixdata.ai_phoenixaiwarehouses.yaml
    kubectl -n phoenixai rollout restart deployment kube-anywhere-operator

    The grep then reports Starting Controller.

    The warehouse chart does not carry the CRD — only the operator chart does, because that is the one installed early enough for the operator to see it. So a warehouse chart installed against an operator that lacks the CRD fails outright with no matches for kind "PhoenixAIWarehouse", rather than appearing to succeed and then never producing pods.

2. Deploy Warehouse​

You can choose one of the following methods to deploy a warehouse:

  1. Deploy Warehouse by YAML Manifest
  2. Deploy Warehouse by Helm Chart

The CN nodes you deploy with a PhoenixAI cluster are added to the default_warehouse by default. To add CN nodes to that warehouse, scale the cluster's CNs or deploy a CN group without phoenixAIWarehouse. A Warehouse CR whose name resolves to default_warehouse, such as default-warehouse, is rejected: that warehouse is reserved for the cluster's own CNs.

2.1 Deploy Warehouse by YAML Manifest​

Deploy a warehouse by the following YAML manifest.

# wh1.yaml
apiVersion: phoenixdata.ai/v1
kind: PhoenixAIWarehouse
metadata:
# A warehouse will be created with this name in PhoenixAI Cluster. If you are using dash(-) in the name, the warehouse
# name created by PhoenixAI will be replaced with underscore(_).
name: wh1

spec:
# Make sure the PhoenixAI cluster exists in the same namespace.
# You can check it by running `kubectl -n phoenixai get phoenixaiclusters.phoenixdata.ai`.
phoenixAICluster: kube-anywhere
template:
envVars:
- name: TZ
value: UTC
image: us-west1-docker.pkg.dev/phoenix-ai-images/enterprise/cn-ubuntu:<database-image-tag>
replicas: 1
limits:
cpu: 8
memory: 8Gi
requests:
cpu: 8
memory: 8Gi

You can see api.md for more details about the PhoenixAIWarehouse CRD fields. The spec part is very similar to the PhoenixAICnSpec of PhoenixAICluster, so see deploy_a_phoenixai_cluster_with_cn.yaml for more fields.

Apply the YAML manifest:

kubectl -n phoenixai apply -f wh1.yaml

2.2. Deploy Warehouse by Helm Chart​

We also support deploying a warehouse by Helm chart. You can also see Warehouse Chart for how to deploy it.

First, prepare a values.yaml file for Warehouse chart.

# wh1-values.yaml
spec:
# Make sure the PhoenixAI cluster exists in the same namespace.
# You can check it by running `kubectl -n phoenixai get phoenixaiclusters.phoenixdata.ai`.
phoenixAIClusterName: kube-anywhere
replicas: 1
image:
repository: us-west1-docker.pkg.dev/phoenix-ai-images/enterprise/cn-ubuntu
tag: "<database-image-tag>"
resources:
limits:
cpu: 8
memory: 8Gi
requests:
cpu: 8
memory: 8Gi

Then deploy a warehouse by the following command:

# Use the above values.yaml to deploy a warehouse in namespace phoenixai
helm -n phoenixai install wh1 phoenixai/warehouse -f wh1-values.yaml

2.3 Give the warehouse the cluster's root password​

A warehouse's compute nodes register themselves in the cluster's coordinator over SQL as root. If the cluster was installed with a root password, they need it, and the warehouse chart does not inherit the cluster chart's initPassword setting — there is no warehouse value for it. Without the password the compute node loops on:

[...] Add myself (wh1-warehouse-cn-0...:9050) into FE ...
ERROR 1045 (28000): Access denied for user 'root' (using password: NO)

and the pod never becomes ready. Pass the password as an environment variable instead. Point it at the same Secret the cluster used, so there is one copy of the password:

# wh1-values.yaml
spec:
# ... the rest of your warehouse values ...
envVars:
- name: MYSQL_PWD
valueFrom:
secretKeyRef:
# The Secret named by phoenixai.initPassword.passwordSecret when the cluster
# was installed.
name: phoenixai-root-password
key: password

Clusters whose root has no password need none of this — leave envVars unset.

If a warehouse is already stuck in this state, adding the block above and re-running helm upgrade is enough; the compute node registers within seconds of restarting.

3. Manage Warehouse​

3.1. Show the deployed warehouse​

If you have deployed the above warehouse, you can see it by using the following SQL command:

# A warehouse has been created with the name `wh1`.
mysql> show warehouses;
+-------+-------------------+-----------+-----------+---------------------+-----------------+-----------------+------------+-----------+-----------+---------------------+---------------------+----------------------------------------------+
| Id | Name | State | NodeCount | CurrentClusterCount | MaxClusterCount | StartedClusters | RunningSql | QueuedSql | CreatedOn | ResumedOn | UpdatedOn | Comment |
+-------+-------------------+-----------+-----------+---------------------+-----------------+-----------------+------------+-----------+-----------+---------------------+---------------------+----------------------------------------------+
| 0 | default_warehouse | AVAILABLE | 0 | 1 | 1 | 1 | 0 | 0 | NULL | 2024-05-11 16:49:37 | 2024-05-11 17:53:30 | An internal warehouse init after FE is ready |
| 35030 | wh1 | AVAILABLE | 1 | 1 | 1 | 1 | 0 | 0 | NULL | NULL | NULL | NULL |
+-------+-------------------+-----------+-----------+---------------------+-----------------+-----------------+------------+-----------+-----------+---------------------+---------------------+----------------------------------------------+
2 rows in set (0.00 sec)

3.2 Upgrade Deployment​

We strongly recommend you to upgrade deployment by modifying the YAML Manifest file or values.yaml file. For example, you can update any fields in the file, e.g. the image version, replicas, and resources.

We don't suggest you to modify the deployment of warehouse by kubectl edit.

When you change the image version, the warehouse and its cluster have to move in order: the cluster's FE may never run a newer version than any compute node, the warehouse's included. Upgrade the warehouse before the cluster, and downgrade it after. In the other order the operator holds the change with InvalidImageOrder until the second one is applied. CN groups in cnGroups do not follow the warehouse's image: change each entry's image.tag in the same step. See Warehouses in their own Helm releases.

3.2.1 Update the YAML manifest​

For example, upgrade the image version:

apiVersion: phoenixdata.ai/v1
kind: PhoenixAIWarehouse
metadata:
# A warehouse will be created with this name in PhoenixAI Cluster. If you are using dash(-) in the name, the warehouse
# name created by PhoenixAI will be replaced with underscore(_).
name: wh1

spec:
# Make sure the PhoenixAI cluster exists in the same namespace.
# You can check it by running `kubectl -n phoenixai get phoenixaiclusters.phoenixdata.ai`.
phoenixAICluster: kube-anywhere
template:
envVars:
- name: TZ
value: UTC
image: us-west1-docker.pkg.dev/phoenix-ai-images/enterprise/cn-ubuntu:<new-database-image-tag> # this line is updated
replicas: 1
limits:
cpu: 8
memory: 8Gi
requests:
cpu: 8
memory: 8Gi

Apply the updated YAML manifest:

kubectl -n phoenixai apply -f wh1.yaml

3.2.2 Update values.yaml for Helm chart​

For example, upgrade the image version:

# wh1-values.yaml
spec:
# Make sure the PhoenixAI cluster exists in the same namespace.
# You can check it by running `kubectl -n phoenixai get phoenixaiclusters.phoenixdata.ai`.
phoenixAIClusterName: kube-anywhere
replicas: 1
image:
repository: us-west1-docker.pkg.dev/phoenix-ai-images/enterprise/cn-ubuntu
tag: "<new-database-image-tag>" # this line is updated
resources:
limits:
cpu: 8
memory: 8Gi
requests:
cpu: 8
memory: 8Gi

Then upgrade the warehouse by the following command:

helm -n phoenixai upgrade wh1 phoenixai/warehouse -f wh1-values.yaml

4. Delete the Warehouse​

If you deployed the warehouse by YAML manifest, you can delete it by running the following command:

kubectl delete -f wh1.yaml

If you deployed the warehouse by Helm chart, you can delete it by running the following command:

helm -n phoenixai uninstall wh1

Deleting a warehouse CR normally also removes the warehouse from the cluster: before it lets the warehouse's CN StatefulSet go, the operator runs DROP WAREHOUSE in the FE. Everything set on the warehouse inside the database (its properties and the privileges granted on it) is dropped with it.

If the FE is unreachable or rejects the statement, the warehouse stays in the cluster. From v2.1, the operator keeps retrying for up to 10 minutes, then lets the StatefulSet go anyway and records a Warning event on it. v2.0.x tries once and lets the StatefulSet go right away, reporting the failure only in the operator log. Either way, after deleting a warehouse CR, run SHOW WAREHOUSES to confirm it is gone, and drop it with DROP WAREHOUSE if it is still listed.

The one exception is a warehouse that still has CN groups. The operator keeps such a warehouse in the FE on purpose, because dropping it would take the groups' FE state with it, and drops it once the groups are deleted (see Deploy CN Groups). Do not drop it by hand while any PhoenixAICnGroup still points at it.

4.1 Delete the CR but keep the warehouse in the cluster​

Sometimes the CR is deleted only to be applied again under the same name, for example when moving the warehouse between operators or rolling such a move back. To keep the warehouse, its properties and its grants in the cluster across that, annotate the CR first.

This needs PhoenixAI operator v2.0.2 or later. Older operators ignore the annotation without any error and still drop the warehouse when the CR is deleted, which is why you check below that it has taken effect.

kubectl -n phoenixai annotate phoenixaiwarehouse wh1 starrocks.com/skip-fe-cleanup=true

Or put the annotation in the manifest:

apiVersion: phoenixdata.ai/v1
kind: PhoenixAIWarehouse
metadata:
name: wh1
namespace: phoenixai
annotations:
starrocks.com/skip-fe-cleanup: "true"

For a warehouse deployed by Helm chart, use the kubectl annotate command above; helm upgrade leaves the annotation in place.

The annotation takes effect once the operator has copied it onto the warehouse's CN StatefulSet and removed its protection finalizer there. Check it before you delete the CR:

kubectl -n phoenixai get sts wh1-warehouse-cn \
-o jsonpath='{.metadata.finalizers}{"\n"}{.metadata.annotations.starrocks\.com/skip-fe-cleanup}{"\n"}'

The first line must not list phoenixdata.ai.phoenixaiwarehouse/protection, and the second line must print true. Adding the annotation changes only the StatefulSet's metadata, so no CN pod restarts.

From then on, deleting the CR is an ordinary Kubernetes delete. The operator sends nothing to the FE, and the CN StatefulSet is removed even if the operator is not running at that moment. In the cluster, the warehouse stays as it was and its compute nodes show Alive = false in SHOW COMPUTE NODES until a same-named warehouse brings them back. If the new CR has fewer replicas than the old one, the extra compute nodes stay offline; drop them with ALTER SYSTEM DROP COMPUTE NODE.

If the warehouse has CN groups, the operator also records the opt-out on each group's StatefulSet the next time it reconciles the group, which takes up to about two minutes. Check every group before you delete the warehouse CR; for a group wh1-az-a:

kubectl -n phoenixai get sts wh1-az-a-cngroup-cn \
-o jsonpath='{.metadata.annotations.phoenixdata\.ai/warehouse-skip-fe-cleanup}{"\n"}'

It must print true. The warehouse then stays in the cluster even after its last group is deleted. The opt-out covers the warehouse, not its groups: deleting a group CR still drops that group in the cluster, and a same-named group CR applied later creates it again.

To delete the warehouse for good later, remove the annotation first:

kubectl -n phoenixai annotate phoenixaiwarehouse wh1 starrocks.com/skip-fe-cleanup-

Then wait until the operator has put the protection finalizer back, and only then delete the CR. Run the same check as above: the first line must list phoenixdata.ai.phoenixaiwarehouse/protection, and the second line must be empty. If you delete the CR before that, the StatefulSet still carries the opt-out, so the operator skips DROP WAREHOUSE and the warehouse stays in the cluster. The same holds for CN groups: the check above must print nothing for each of them before the last one is deleted.

If you delete an annotated CR and no longer need the warehouse, drop it yourself with DROP WAREHOUSE.