Playing around with starting a MySQL pod with an environment variable populated from a secret on Kubernetes, I experienced a gotcha with an error message that I couldn’t easily find googling around:
The Error Message
mysqladmin: [ERROR] unknown option '--"'.
The Investigation
Since I had previously started a MySQL pod with a non-secret env variable without any problems, I suspected an issue with my configuration:
apiVersion: v1
kind: Secret
metadata:
name: mysql-root-password
type: Opaque
data:
MYSQL_ROOT_PASSWORD: cGFzc3dvcmQK
---
apiVersion: v1
kind: Pod
metadata:
name: db
spec:
containers:
- name: mysql
image: mysql
envFrom:
- secretRef:
name: mysql-root-password
The value “cGFzc3dvcmQK” comes from base64-encoding the password, in this case the word “password”:
$ echo "password" | base64
cGFzc3dvcmQK
But this is actually incorrect, since echo will implicitly add a newline character, which gets base64-encoded into the string! When this string later gets base64-decoded inside Kubernetes, the environment variables in the MySQL container look like this:
$ kubectl exec -it db printenv
HOSTNAME=db
MYSQL_ROOT_PASSWORD=password
MYSQL_MAJOR=8.0
MYSQL_VERSION=8.0.20-1debian10
...
That newline character is included, and MySQL fails to start attempting to apply an option for an empty environment variable (‘–“‘), causing that somewhat-confusing error message to appear in the container logs.
The Takeaway
Be sure to base64-encode secrets without the newline character. When generating the secret with echo, you should use the “-n” flag to strip the newline character:
$ echo -n "password" | base64
cGFzc3dvcmQ=
Using this encoded string will prevent empty environment variables being injected into the MySQL container and allow MySQL to start:
$ kubectl exec -it db printenv
HOSTNAME=db
MYSQL_ROOT_PASSWORD=password
MYSQL_MAJOR=8.0
MYSQL_VERSION=8.0.20-1debian10
...
I did come across this issue which describes the gotcha affecting other applications as well, even 4 years after it was originally filed. But since I couldn’t find any posts with this exact MySQL error log, I thought I’d post my experience.