Intermittent login failure using Kubernetes operator

Help
1

Hi,

I’ve got the operator deploying a cluster ok, and it’s sort of working except that every so often connections are failing. sufficient retries will eventually get through, which makes the behaviour more confusing. It’s running a single node and aggregator with height 0.5, all operations are working and the db is functioning except for this issue.

I would have said it was something competing on the port, except that in the logs for the “node-memsql-cluster-leaf-ag1-0” node I can see: “ERROR: ProcessHandshakeResponsePacket() failed. Sending back 1045: Access denied for user ‘admin’@‘172...**’ (using password: YES)”. So it’s as though the leaf node is intermittently rejecting the password.

Is there anything that might explain this behaviour? If I create a different user and log in I get the same behaviour, and the application pods are logging out the same issue. How can I investigate this further?

2

I’ve since noticed that the logs for the failed login always come from the leaf node. The master node doesn’t have any failures unless I enter an incorrect password, in which case the login failures are distributed between the two. So it seems that the user is not replicating to the leaf node. Does anyone know why this may be?

3

I’ve found that this was being caused because I was connecting to the non ddl endpoint. Connecting to “svc-memsql-cluster” causes this problem, connecting to “svc-memsql-cluster-ddl” does not.