Change root user password

The password is empty for the root user when deploying a PhoenixAI cluster from fresh installation. This can be a security concern. This document describes steps to change root password and still the operator can manage the cluster correctly.

In the following examples, mysql_password is taken as the password for the root user, it can be replaced with any password chosen for the root.

Prerequisites

A PhoenixAI cluster is deployed and up with empty root password by the operator.

1. Change the password for root user

Connect to PhoenixAI FE with a MySQL client and change the root user password.

mysql
-h <FE_IP/FE_SERVICE> -P 9030 -u root

# change root password to `mysql_password`
MySQL [(none)]> SET PASSWORD = PASSWORD('mysql_password');

2. Inject MYSQL_PWD environment variable to PhoenixAI components

There are two ways to deploy PhoenixAI cluster:

1. Deploy PhoenixAI cluster with PhoenixAICluster CR yaml.
2. Deploy PhoenixAI cluster with Helm chart.

Therefore, there are two ways to inject the MYSQL_PWD environment variable into PhoenixAI components.

2.1 inject MYSQL_PWD environment variable with PhoenixAICluster CRD yaml

1. Create a secret rootcredential with the key password to store the root password

   kubectl create secret generic rootcredential --from-literal=password=mysql_password

2. Add the following snippets to phoenixAIFeSpec/phoenixAICnSpec respectively if the corresponding components are deployed.

   # for phoenixAIFeSpec
   feEnvVars:
   - name: "MYSQL_PWD"
     valueFrom:
       secretKeyRef:
         name: rootcredential
         key: password
   
   # for phoenixAICnSpec
   cnEnvVars:
   - name: "MYSQL_PWD"
     valueFrom:
       secretKeyRef:
         name: rootcredential
         key: password

3. Apply the crd yaml

   kubectl apply -f <crd_yaml>

It will trigger a rolling restart of the cluster, wait until the cluster restart completed.

2.2 Inject MYSQL_PWD environment variable with helm chart

If you are using the kube-anywhere Helm chart, add the following snippets to values.yaml.

phoenixai:
  # create secrets if necessary.
  secrets:
    - name: rootcredential
      data:
        password: mysql_password

  phoenixAIFeSpec:
    feEnvVars:
      - name: "MYSQL_PWD"
        valueFrom:
          secretKeyRef:
            name: rootcredential
            key: password

  phoenixAICnSpec:
    cnEnvVars:
      - name: "MYSQL_PWD"
        valueFrom:
          secretKeyRef:
            name: rootcredential
            key: password

If you are using the phoenixai Helm chart, add the following snippets to values.yaml.

# create secrets if necessary.
secrets:
- name: rootcredential
  data:
    password: mysql_password

phoenixAIFeSpec:
  feEnvVars:
  - name: "MYSQL_PWD"
    valueFrom:
      secretKeyRef:
        name: rootcredential
        key: password

phoenixAICnSpec:
  cnEnvVars:
  - name: "MYSQL_PWD"
    valueFrom:
      secretKeyRef:
        name: rootcredential
        key: password

Run the following command to upgrade the cluster.

helm upgrade <release_name> <chart_path> -f values.yaml

It will trigger a rolling restart of the cluster, wait until the cluster restart completed.

3. Verify the password is all set

After the pods are restarted, run the following command to check the correctness of the password.

kubectl exec <podName> -- sh -c 'echo $MYSQL_PWD'

4. What happens when the password in the CR does not match FE

Once FE is ready and before it creates or updates a CN workload, the operator logs in to the FE query port with the root password that CN spec declares in MYSQL_PWD (spec.phoenixAICnSpec.cnEnvVars on a cluster, spec.template.envVars on a PhoenixAIWarehouse, spec.phoenixAICnSpec.cnEnvVars on a PhoenixAICnGroup) — the same password those CN pods will register with. If the login fails, the CN workload is not touched and the object reports why:

• On a cluster, status.phase becomes failed and status.reason starts with error from CN controller: operator cannot log in to fe:. On a warehouse or CN group, status.reason starts with operator cannot log in to fe:.
• The reason names the cause: the FE rejected the password (and that it came from the MYSQL_PWD env var of the CN spec, or that none is set), the FE service name did not resolve, the query port refused the connection or timed out, the --fe-ssl-mode of the operator does not match the FE SSL configuration, or the referenced Secret could not be read.

The check is retried like any other CN reconcile error, and every object is reconciled again by the operator's periodic resync (every 2 minutes), so fixing the Secret value, the MYSQL_PWD entry in the CR, or the password in FE is enough for the operator to resume within a couple of minutes; editing the CR triggers the retry at once. A successful login is remembered for 10 minutes, so the frequent reconciles of a healthy cluster do not each log in to FE (and do not each leave a root login in FE's audit log). The memory is keyed by the FE address, the SSL mode and the password: changing MYSQL_PWD in the CR or in its Secret is checked at once, while a password changed only inside FE with SET PASSWORD is noticed on the first check after those 10 minutes.

A typical mismatch after step 1 but before step 2 looks like this:

kubectl get phoenixaicluster <name> -o jsonpath='{.status.phase}{"\n"}{.status.reason}{"\n"}'
failed
error from CN controller: operator cannot log in to fe: FE at <name>-fe-service.<namespace>:9030 rejected the root password (Error 1045 (28000): Access denied for user 'root' (using password: NO)); the CN spec has no MYSQL_PWD env var, so an empty password was used; if the root password has been changed, declare it there via MYSQL_PWD