Installed a new cluster, 6 nodes with 4 leafs. Install went fine. Rebooted the servers and 2 leaf nodes do not start. Message is
✘ Failed to start node on 192.168.0.215 (1/1)
error running memsqlctl: error running command: `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/run/user/1000/singlestoredb-toolbox582155734/4.socket 192.168.0.215 sudo -S -k -- /bin/sh -c $1`
where $1 = `printf ae798195-0281-43bf-832a-c5e5ef777ef4 && exec memsqlctl --json --yes --verbose-json start-node --memsql-id 1B2658DFA6CF9254D98126C9FDE9CBC993E965FC`: exit status 1:
stdout: Latest errors from MemSQL tracelog:
03502345 2022-01-01 22:01:05.931 ERROR: open('/opt/singlestoredb-server-7.6.6-63e226ba05/lib/license', O_RDONLY, S_IRUSR | S_IWUSR) failed. Error No such file or directory (2)
03502386 2022-01-01 22:01:05.931 ERROR: Cannot open license key file at '/opt/singlestoredb-server-7.6.6-63e226ba05/lib/license'
03511710 2022-01-01 22:01:05.941 ERROR: ProcessHandshakeResponsePacket() failed. Sending back 1045: Access denied for user 'distributed'@'192.168.0.211' (using password: YES)
03511797 2022-01-01 22:01:05.941 ERROR: ProcessHandshakeResponsePacket() failed. Sending back 1045: Access denied for user 'distributed'@'192.168.0.211' (using password: YES)
03512025 2022-01-01 22:01:05.941 ERROR: ProcessHandshakeResponsePacket() failed. Sending back 1045: Access denied for user 'distributed'@'192.168.0.211' (using password: YES)
03512186 2022-01-01 22:01:05.941 ERROR: ProcessHandshakeResponsePacket() failed. Sending back 1045: Access denied for user 'root'@'localhost' (using password: YES)
: Failed to connect to MemSQL: Could not establish connection: Error 1045: Access denied for user 'root'@'localhost' (using password: YES)
stderr: (empty)
When I look at the available nodes, I only see 5
$ sdb-admin list-nodes
+------------+------------+---------------+------+---------------+--------------+---------+----------------+--------------------+--------------+
| MemSQL ID | Role | Host | Port | Process State | Connectable? | Version | Recovery State | Availability Group | Bind Address |
+------------+------------+---------------+------+---------------+--------------+---------+----------------+--------------------+--------------+
| 3A322F3CA0 | Master | 192.168.0.211 | 3306 | Running | True | 7.6.6 | Online | | 0.0.0.0 |
| AED62FE775 | Aggregator | 192.168.0.212 | 3306 | Running | True | 7.6.6 | Online | | 0.0.0.0 |
| C46109686E | Leaf | 192.168.0.213 | 3306 | Running | True | 7.6.6 | Online | 1 | 0.0.0.0 |
| A36863C9F3 | Leaf | 192.168.0.216 | 3306 | Running | True | 7.6.6 | Online | 1 | 0.0.0.0 |
| 1B2658DFA6 | Unknown | 192.168.0.215 | 3306 | Running | False | | Unknown | | 0.0.0.0 |
+------------+------------+---------------+------+---------------+--------------+---------+----------------+--------------------+--------------+
That license file does not exist on any nodes so I am not sure why it is needed for these 2 when the other 4 start without it.