Leaf node partition went offline due to ssh failure

divyank1 · January 12, 2023, 5:55am

Hi,

I am having 1 SS Cluser installed in linux with version:7.5. containing 4 leaf nodes and 1 master node.
Yesterday suddenly one of leaf node ssh passwordless authentication failed and leaf partion went offline for particular leaf node.

To restore passwordless ssh access for that particular VM (Leaf node), I need to restart that VM.

So if I restart that VM, Will the leaf node will start running normally without any data loss??
I don’t want to lose data in that leaf node,also as ssh is not working I could not even stop or perform any activity in that leaf node.

Snap:

Let me know if I can do restart of VM of affected nodes without any data loss or any workaround.

Logs:

 $ ./sdb-admin list-nodes
✘ Failed to list nodes on all hosts: failed to list nodes on host < >: SSH Master process failed to start:
Error: error running command: `/usr/bin/ssh -oControlMaster=yes -oControlPath=/tmp/singlestoredb-toolbox194403858/1.socket -N -oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oServerAliveInterval=60 -oServerAliveCountMax=5 -oBatchMode=no < >`: exit status 255
Stdout:
Stderr: Warning: Permanently added ' <...> ' (ECDSA) to the list of known hosts.
Permission denied, please try again.
Received disconnect from <...>  port <>:2: Too many authentication failures
Disconnected from <...> port <>

divyank1 · January 12, 2023, 11:07am

Update:

Able to restore ssh passwordless access to leaf node without restarting VM.
But when I checked the list nodes, afftected leaf node showing “RecoverFailed” State.
Pls let me know how to bring the leaf node online again. I did not change any SS config change.

SS GUI showing DB inside the affected node is corrupted, Pls let me know how to make that afected node online again and restore data Recovery state to “online”.

Snap: