Change root user password

The password is empty for the root user when deploying a PhoenixAI cluster from fresh installation. This can be a security concern. This document describes steps to change root password and still the operator can manage the cluster correctly.

In the following examples, mysql_password is taken as the password for the root user, it can be replaced with any password chosen for the root.

Prerequisites

A PhoenixAI cluster is deployed and up with empty root password by the operator.

1. Change the password for root user

Connect to PhoenixAI FE with a MySQL client and change the root user password.

mysql
-h <FE_IP/FE_SERVICE> -P 9030 -u root

# change root password to `mysql_password`
MySQL [(none)]> SET PASSWORD = PASSWORD('mysql_password');

2. Inject MYSQL_PWD environment variable to PhoenixAI components

There are two ways to deploy PhoenixAI cluster:

1. Deploy PhoenixAI cluster with PhoenixAICluster CR yaml.
2. Deploy PhoenixAI cluster with Helm chart.

Therefore, there are two ways to inject the MYSQL_PWD environment variable into PhoenixAI components.

2.1 inject MYSQL_PWD environment variable with PhoenixAICluster CRD yaml

1. Create a secret rootcredential with the key password to store the root password

   kubectl create secret generic rootcredential --from-literal=password=mysql_password

2. Add the following snippets to phoenixAIFeSpec/phoenixAICnSpec respectively if the corresponding components are deployed.

   # for phoenixAIFeSpec
   feEnvVars:
   - name: "MYSQL_PWD"
     valueFrom:
       secretKeyRef:
         name: rootcredential
         key: password
   
   # for phoenixAICnSpec
   cnEnvVars:
   - name: "MYSQL_PWD"
     valueFrom:
       secretKeyRef:
         name: rootcredential
         key: password

3. Apply the crd yaml

   kubectl apply -f <crd_yaml>

It will trigger a rolling restart of the cluster, wait until the cluster restart completed.

2.2 Inject MYSQL_PWD environment variable with helm chart

If you are using the kube-anywhere Helm chart, add the following snippets to values.yaml.

phoenixai:
  # create secrets if necessary.
  secrets:
    - name: rootcredential
      data:
        password: mysql_password

  phoenixAIFeSpec:
    feEnvVars:
      - name: "MYSQL_PWD"
        valueFrom:
          secretKeyRef:
            name: rootcredential
            key: password

  phoenixAICnSpec:
    cnEnvVars:
      - name: "MYSQL_PWD"
        valueFrom:
          secretKeyRef:
            name: rootcredential
            key: password

If you are using the phoenixai Helm chart, add the following snippets to values.yaml.

# create secrets if necessary.
secrets:
- name: rootcredential
  data:
    password: mysql_password

phoenixAIFeSpec:
  feEnvVars:
  - name: "MYSQL_PWD"
    valueFrom:
      secretKeyRef:
        name: rootcredential
        key: password

phoenixAICnSpec:
  cnEnvVars:
  - name: "MYSQL_PWD"
    valueFrom:
      secretKeyRef:
        name: rootcredential
        key: password

Run the following command to upgrade the cluster.

helm upgrade <release_name> <chart_path> -f values.yaml

It will trigger a rolling restart of the cluster, wait until the cluster restart completed.

3. Verify the password is all set

After the pods are restarted, run the following command to check the correctness of the password.

kubectl exec <podName> -- sh -c 'echo $MYSQL_PWD'

4. What happens when the password in the CR does not match FE

Every time the operator itself logs in to FE, it uses the MYSQL_PWD of the FE spec (spec.phoenixAIFeSpec.feEnvVars of the PhoenixAICluster). That covers the login check below, the FE feature probe, the version check that orders FE and CN image rollouts, and the SQL it sends when it scales in CN or deletes a warehouse or CN group. The MYSQL_PWD of a CN spec, a warehouse or a CN group is what those CN pods register with; the operator does not log in with it. This is why step 2 puts the password in the FE spec as well as in the CN spec.

Once FE is ready and before it creates or updates a CN workload of a cluster, warehouse or CN group, the operator logs in to the FE query port with that password. If the login fails, the CN workload is not touched and the object reports why:

• On a cluster, status.phase becomes failed and status.reason starts with error from CN controller: operator cannot log in to fe:. On a warehouse or CN group, status.reason starts with operator cannot log in to fe:.
• The reason names the cause: the FE rejected the password (and that it came from the MYSQL_PWD env var of the FE StatefulSet, or that none is set), the FE service name did not resolve, the query port refused the connection or timed out, the --fe-ssl-mode of the operator does not match the FE SSL configuration, or the referenced Secret could not be read.

The check is retried like any other CN reconcile error, and every object is reconciled again by the operator's periodic resync (every 2 minutes), so fixing the Secret value, the MYSQL_PWD entry in the FE spec, or the password in FE is enough for the operator to resume within a couple of minutes. A successful login is remembered for 10 minutes, so the frequent reconciles of a healthy cluster do not each log in to FE (and do not each leave a root login in FE's audit log). The memory is keyed by the FE address, the SSL mode and the MYSQL_PWD entry of the FE spec as written (its value, or the Secret it references): changing that entry is checked at once, while a new value inside the same Secret, or a password changed only inside FE with SET PASSWORD, is noticed on the first check after those 10 minutes.

A typical mismatch after step 1 but before step 2 looks like this:

kubectl get phoenixaicluster <name> -o jsonpath='{.status.phase}{"\n"}{.status.reason}{"\n"}'
failed
error from CN controller: operator cannot log in to fe: FE at <name>-fe-service.<namespace>:9030 rejected the root password (Error 1045 (28000): Access denied for user 'root' (using password: NO)); FE StatefulSet <name>-fe has no MYSQL_PWD env var, so an empty password was used; if the root password has been changed, declare it in spec.phoenixAIFeSpec.feEnvVars of the PhoenixAICluster