Deploy Warehouse
From PhoenixAI Operator v1.9.0, PhoenixAIWarehouse CRD is introduced to manage the warehouse. This document describes how to deploy a warehouse.
A warehouse can be split further into CN Groups, each an independent copy of the same compute pool, typically one per availability zone. See Deploy CN Groups.
1. Prerequisites
-
PhoenixAI Operator >= v1.9.0. The latest version of the operator is recommended.
-
An installed PhoenixAI Cluster. See Install with kubectl or Install with Helm for more details.
-
PhoenixAI enterprise version >= v3.2.0.
-
The
PhoenixAIWarehouseCRD, present before the operator started. A fresh installation of the current charts satisfies this on its own: the operator chart ships the CRD next to the cluster one, and Install with kubectl applies both in step 1.1. No operator restart is needed there.An upgraded or older operator will not have itOperator charts only began shipping the warehouse CRD in v2.0.0, and
helm upgradenever installs a chart'scrds/directory — so an operator upgraded from an earlier version does not gain the CRD from the chart, whatever version it now runs. The same holds for an operator installed with--skip-crds, or from a manifest that applied only the cluster CRD.Check rather than assume:
kubectl get crd phoenixaiwarehouses.phoenixdata.aikubectl -n phoenixai logs deployment/kube-anywhere-operator | grep PhoenixAIWarehouseThe operator decides whether to run the warehouse controller once, at startup. If it started without the CRD, warehouses are never reconciled and nothing reports it:
kubectl get pawshows the object with an emptySTATUS, there are no pods and no events, and thegrepabove returns nothing. Install the CRD, then restart the operator:kubectl apply -f https://raw.githubusercontent.com/celerdata/phoenixai-kubernetes-operator/main/deploy/phoenixdata.ai_phoenixaiwarehouses.yamlkubectl -n phoenixai rollout restart deployment kube-anywhere-operatorThe
grepthen reportsStarting Controller.The warehouse chart does not carry the CRD — only the operator chart does, because that is the one installed early enough for the operator to see it. So a warehouse chart installed against an operator that lacks the CRD fails outright with
no matches for kind "PhoenixAIWarehouse", rather than appearing to succeed and then never producing pods.
2. Deploy Warehouse
You can choose one of the following methods to deploy a warehouse:
- Deploy Warehouse by YAML Manifest
- Deploy Warehouse by Helm Chart
The CN nodes you deploy with a PhoenixAI cluster are added to the
default_warehouseby default. To add CN nodes to that warehouse, scale the cluster's CNs or deploy a CN group withoutphoenixAIWarehouse. A Warehouse CR whose name resolves todefault_warehouse, such asdefault-warehouse, is rejected: that warehouse is reserved for the cluster's own CNs.
2.1 Deploy Warehouse by YAML Manifest
Deploy a warehouse by the following YAML manifest.
# wh1.yaml
apiVersion: phoenixdata.ai/v1
kind: PhoenixAIWarehouse
metadata:
# A warehouse will be created with this name in PhoenixAI Cluster. If you are using dash(-) in the name, the warehouse
# name created by PhoenixAI will be replaced with underscore(_).
name: wh1
spec:
# Make sure the PhoenixAI cluster exists in the same namespace.
# You can check it by running `kubectl -n phoenixai get phoenixaiclusters.phoenixdata.ai`.
phoenixAICluster: kube-anywhere
template:
envVars:
- name: TZ
value: UTC
image: us-west1-docker.pkg.dev/phoenix-ai-images/enterprise/cn-ubuntu:<database-image-tag>
replicas: 1
limits:
cpu: 8
memory: 8Gi
requests:
cpu: 8
memory: 8Gi
You can see api.md for more details about the PhoenixAIWarehouse CRD fields. The spec part is very similar to the PhoenixAICnSpec of PhoenixAICluster, so see deploy_a_phoenixai_cluster_with_cn.yaml for more fields.
Apply the YAML manifest:
kubectl -n phoenixai apply -f wh1.yaml
2.2. Deploy Warehouse by Helm Chart
We also support deploying a warehouse by Helm chart. You can also see Warehouse Chart for how to deploy it.
First, prepare a values.yaml file for Warehouse chart.
# wh1-values.yaml
spec:
# Make sure the PhoenixAI cluster exists in the same namespace.
# You can check it by running `kubectl -n phoenixai get phoenixaiclusters.phoenixdata.ai`.
phoenixAIClusterName: kube-anywhere
replicas: 1
image:
repository: us-west1-docker.pkg.dev/phoenix-ai-images/enterprise/cn-ubuntu
tag: "<database-image-tag>"
resources:
limits:
cpu: 8
memory: 8Gi
requests:
cpu: 8
memory: 8Gi
Then deploy a warehouse by the following command:
# Use the above values.yaml to deploy a warehouse in namespace phoenixai
helm -n phoenixai install wh1 phoenixai/warehouse -f wh1-values.yaml
2.3 Give the warehouse the cluster's root password
A warehouse's compute nodes register themselves in the cluster's coordinator over SQL as root. If
the cluster was installed with a root password, they need it, and the warehouse chart does not
inherit the cluster chart's initPassword setting — there is no warehouse value for it. Without
the password the compute node loops on:
[...] Add myself (wh1-warehouse-cn-0...:9050) into FE ...
ERROR 1045 (28000): Access denied for user 'root' (using password: NO)
and the pod never becomes ready. Pass the password as an environment variable instead. Point it at the same Secret the cluster used, so there is one copy of the password:
# wh1-values.yaml
spec:
# ... the rest of your warehouse values ...
envVars:
- name: MYSQL_PWD
valueFrom:
secretKeyRef:
# The Secret named by phoenixai.initPassword.passwordSecret when the cluster
# was installed.
name: phoenixai-root-password
key: password
Clusters whose root has no password need none of this — leave envVars unset.
If a warehouse is already stuck in this state, adding the block above and re-running
helm upgrade is enough; the compute node registers within seconds of restarting.
3. Manage Warehouse
3.1. Show the deployed warehouse
If you have deployed the above warehouse, you can see it by using the following SQL command:
# A warehouse has been created with the name `wh1`.
mysql> show warehouses;
+-------+-------------------+-----------+-----------+---------------------+-----------------+-----------------+------------+-----------+-----------+---------------------+---------------------+----------------------------------------------+
| Id | Name | State | NodeCount | CurrentClusterCount | MaxClusterCount | StartedClusters | RunningSql | QueuedSql | CreatedOn | ResumedOn | UpdatedOn | Comment |
+-------+-------------------+-----------+-----------+---------------------+-----------------+-----------------+------------+-----------+-----------+---------------------+---------------------+----------------------------------------------+
| 0 | default_warehouse | AVAILABLE | 0 | 1 | 1 | 1 | 0 | 0 | NULL | 2024-05-11 16:49:37 | 2024-05-11 17:53:30 | An internal warehouse init after FE is ready |
| 35030 | wh1 | AVAILABLE | 1 | 1 | 1 | 1 | 0 | 0 | NULL | NULL | NULL | NULL |
+-------+-------------------+-----------+-----------+---------------------+-----------------+-----------------+------------+-----------+-----------+---------------------+---------------------+----------------------------------------------+
2 rows in set (0.00 sec)
3.2 Upgrade Deployment
We strongly recommend you to upgrade deployment by modifying the YAML Manifest file or values.yaml file. For example, you can update any fields in the file, e.g. the image version, replicas, and resources.
We don't suggest you to modify the deployment of warehouse by
kubectl edit.
When you change the image version, the warehouse and its cluster have to move in order: the
cluster's FE may never run a newer version than any compute node, the warehouse's included. Upgrade
the warehouse before the cluster, and downgrade it after. In the other order the operator holds the
change with InvalidImageOrder until the second one is applied. CN groups in cnGroups do not follow
the warehouse's image: change each entry's image.tag in the same step. See
Warehouses in their own Helm releases.
3.2.1 Update the YAML manifest
For example, upgrade the image version:
apiVersion: phoenixdata.ai/v1
kind: PhoenixAIWarehouse
metadata:
# A warehouse will be created with this name in PhoenixAI Cluster. If you are using dash(-) in the name, the warehouse
# name created by PhoenixAI will be replaced with underscore(_).
name: wh1
spec:
# Make sure the PhoenixAI cluster exists in the same namespace.
# You can check it by running `kubectl -n phoenixai get phoenixaiclusters.phoenixdata.ai`.
phoenixAICluster: kube-anywhere
template:
envVars:
- name: TZ
value: UTC
image: us-west1-docker.pkg.dev/phoenix-ai-images/enterprise/cn-ubuntu:<new-database-image-tag> # this line is updated
replicas: 1
limits:
cpu: 8
memory: 8Gi
requests:
cpu: 8
memory: 8Gi
Apply the updated YAML manifest:
kubectl -n phoenixai apply -f wh1.yaml
3.2.2 Update values.yaml for Helm chart
For example, upgrade the image version:
# wh1-values.yaml
spec:
# Make sure the PhoenixAI cluster exists in the same namespace.
# You can check it by running `kubectl -n phoenixai get phoenixaiclusters.phoenixdata.ai`.
phoenixAIClusterName: kube-anywhere
replicas: 1
image:
repository: us-west1-docker.pkg.dev/phoenix-ai-images/enterprise/cn-ubuntu
tag: "<new-database-image-tag>" # this line is updated
resources:
limits:
cpu: 8
memory: 8Gi
requests:
cpu: 8
memory: 8Gi
Then upgrade the warehouse by the following command:
helm -n phoenixai upgrade wh1 phoenixai/warehouse -f wh1-values.yaml
4. Delete the Warehouse
If you deployed the warehouse by YAML manifest, you can delete it by running the following command:
kubectl delete -f wh1.yaml
If you deployed the warehouse by Helm chart, you can delete it by running the following command:
helm -n phoenixai uninstall wh1
Deleting a warehouse CR normally also removes the warehouse from the cluster: before it lets the
warehouse's CN StatefulSet go, the operator runs DROP WAREHOUSE in the FE. Everything set on the
warehouse inside the database (its properties and the privileges granted on it) is dropped with it.
If the FE is unreachable or rejects the statement, the warehouse stays in the cluster. From v2.1, the
operator keeps retrying for up to 10 minutes, then lets the StatefulSet go anyway and records a
Warning event on it. v2.0.x tries once and lets the StatefulSet go right away, reporting the failure
only in the operator log. Either way, after deleting a warehouse CR, run SHOW WAREHOUSES to confirm
it is gone, and drop it with DROP WAREHOUSE if it is still listed.
The one exception is a warehouse that still has CN groups. The operator keeps such a warehouse in the
FE on purpose, because dropping it would take the groups' FE state with it, and drops it once the
groups are deleted (see Deploy CN Groups). Do not drop it by hand while
any PhoenixAICnGroup still points at it.
4.1 Delete the CR but keep the warehouse in the cluster
Sometimes the CR is deleted only to be applied again under the same name, for example when moving the warehouse between operators or rolling such a move back. To keep the warehouse, its properties and its grants in the cluster across that, annotate the CR first.
This needs PhoenixAI operator v2.0.2 or later. Older operators ignore the annotation without any error and still drop the warehouse when the CR is deleted, which is why you check below that it has taken effect.
kubectl -n phoenixai annotate phoenixaiwarehouse wh1 starrocks.com/skip-fe-cleanup=true
Or put the annotation in the manifest:
apiVersion: phoenixdata.ai/v1
kind: PhoenixAIWarehouse
metadata:
name: wh1
namespace: phoenixai
annotations:
starrocks.com/skip-fe-cleanup: "true"
For a warehouse deployed by Helm chart, use the kubectl annotate command above; helm upgrade leaves
the annotation in place.
The annotation takes effect once the operator has copied it onto the warehouse's CN StatefulSet and removed its protection finalizer there. Check it before you delete the CR:
kubectl -n phoenixai get sts wh1-warehouse-cn \
-o jsonpath='{.metadata.finalizers}{"\n"}{.metadata.annotations.starrocks\.com/skip-fe-cleanup}{"\n"}'
The first line must not list phoenixdata.ai.phoenixaiwarehouse/protection, and the second line must
print true. Adding the annotation changes only the StatefulSet's metadata, so no CN pod restarts.
From then on, deleting the CR is an ordinary Kubernetes delete. The operator sends nothing to the FE,
and the CN StatefulSet is removed even if the operator is not running at that moment. In the cluster,
the warehouse stays as it was and its compute nodes show Alive = false in SHOW COMPUTE NODES until a
same-named warehouse brings them back. If the new CR has fewer replicas than the old one, the extra
compute nodes stay offline; drop them with ALTER SYSTEM DROP COMPUTE NODE.
If the warehouse has CN groups, the operator also records the opt-out on each group's StatefulSet the
next time it reconciles the group, which takes up to about two minutes. Check every group before you
delete the warehouse CR; for a group wh1-az-a:
kubectl -n phoenixai get sts wh1-az-a-cngroup-cn \
-o jsonpath='{.metadata.annotations.phoenixdata\.ai/warehouse-skip-fe-cleanup}{"\n"}'
It must print true. The warehouse then stays in the cluster even after its last group is deleted.
The opt-out covers the warehouse, not its groups: deleting a group CR still drops that group in the
cluster, and a same-named group CR applied later creates it again.
To delete the warehouse for good later, remove the annotation first:
kubectl -n phoenixai annotate phoenixaiwarehouse wh1 starrocks.com/skip-fe-cleanup-
Then wait until the operator has put the protection finalizer back, and only then delete the CR. Run the
same check as above: the first line must list phoenixdata.ai.phoenixaiwarehouse/protection, and the
second line must be empty. If you delete the CR before that, the StatefulSet still carries the opt-out,
so the operator skips DROP WAREHOUSE and the warehouse stays in the cluster. The same holds for CN
groups: the check above must print nothing for each of them before the last one is deleted.
If you delete an annotated CR and no longer need the warehouse, drop it yourself with DROP WAREHOUSE.