Playing around with starting a MySQL pod with an environment variable populated from a secret on Kubernetes, I experienced a gotcha with an error message that I couldn’t easily find googling around:

The Error Message

mysqladmin: [ERROR] unknown option '--"'.

The Investigation

Since I had previously started a MySQL pod with a non-secret env variable without any problems, I suspected an issue with my configuration:

apiVersion: v1
kind: Secret
metadata:
  name: mysql-root-password
type: Opaque
data:
  MYSQL_ROOT_PASSWORD: cGFzc3dvcmQK
---
apiVersion: v1
kind: Pod
metadata:
  name: db
spec:
  containers:
    - name: mysql
      image: mysql
      envFrom:
        - secretRef:
            name: mysql-root-password

The value “cGFzc3dvcmQK” comes from base64-encoding the password, in this case the word “password”:

$ echo "password" | base64
cGFzc3dvcmQK

But this is actually incorrect, since echo will implicitly add a newline character, which gets base64-encoded into the string! When this string later gets base64-decoded inside Kubernetes, the environment variables in the MySQL container look like this:

$ kubectl exec -it db printenv
HOSTNAME=db
MYSQL_ROOT_PASSWORD=password

MYSQL_MAJOR=8.0
MYSQL_VERSION=8.0.20-1debian10
...

That newline character is included, and MySQL fails to start attempting to apply an option for an empty environment variable (‘–“‘), causing that somewhat-confusing error message to appear in the container logs.

The Takeaway

Be sure to base64-encode secrets without the newline character. When generating the secret with echo, you should use the “-n” flag to strip the newline character:

$ echo -n "password" | base64
cGFzc3dvcmQ=

Using this encoded string will prevent empty environment variables being injected into the MySQL container and allow MySQL to start:

$ kubectl exec -it db printenv
HOSTNAME=db
MYSQL_ROOT_PASSWORD=password
MYSQL_MAJOR=8.0
MYSQL_VERSION=8.0.20-1debian10
...

I did come across this issue which describes the gotcha affecting other applications as well, even 4 years after it was originally filed. But since I couldn’t find any posts with this exact MySQL error log, I thought I’d post my experience.