Add Leaf to Existing SingleStore Cluster

Hi @rizvi and @hanson.

I was able to get past the “no host found” by using a variation of the solution found in this post. Here is what I did

$ memsql-toolbox-config register-host --host 192.168.0.215  --memsqlctl-config-path /var/lib/memsql/nodes.hcl
✓ Successfully performed host check. No duplicate hosts found.
Toolbox is about to register the following host:
  · Host: 192.168.0.215
  · Localhost: false
  · Memsqlctl Config Path: /var/lib/memsql/nodes.hcl

Would you like to continue? [y/N]: y
Toolbox is about to perform the following actions:
  · Update specific system files on you host for the initial cluster configuration.

Files to be added:

 - /etc/sysctl.d/90-singlestore.conf
Files to be updated:

 - /sys/kernel/mm/transparent_hugepage/defrag
 - /sys/kernel/mm/transparent_hugepage/enabled
 - /etc/security/limits.conf
Add a new systemd service, if available:

 - /etc/systemd/system/singlestoreConfigOS.service - .service file
 - /etc/singlestoreConfigOS - service logic file

To skip this step, re-run this command with the --skip-auto-config flag.

The host may require a reboot for these changes to take effect. To restart your cluster after a reboot, run 'sdb-admin restart-node --all'.


Would you like to continue? [y/N]: y
✓ Successfully registered host 192.168.0.215
+---------------+------------+---------------+---------------+
|     Host      | Local Host |  SSH address  | Identity File |
+---------------+------------+---------------+---------------+
| 192.168.0.215 | No         | 192.168.0.215 |               |
+---------------+------------+---------------+---------------+

Now I get different issues when trying to run sdb-admin create-node --availability-group 1 --host 192.168.0.215 --role leaf. I created the new VM for the node by cloning an existing leaf and clearing out the /var/lib/memsql directory. With no /var/lib/memsql/nodes.hcl on the new host. I had errors

Enter the password for the database user: 
could not determine memsqlctl user: error running memsqlctl: error running command: `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/run/user/1000/singlestoredb-toolbox745496104/1.socket 192.168.0.215 memsqlctl --json --yes --config /var/lib/memsql/nodes.hcl env`: exit status 1: 
stdout: invalid argument "/var/lib/memsql/nodes.hcl" for "-c, --config" flag: stat /var/lib/memsql/nodes.hcl: no such file or directory

So I copied that file to the new host from an exsiting leaf node. New issue

sdb-admin create-node --availability-group 1 --host 192.168.0.215 --role leaf
Enter the password for the database user: 
Toolbox is about to perform the following actions on host 192.168.0.215:
  · Run 'memsqlctl create-node --password ●●●●●●'

Would you like to continue? [y/N]: y
failed to encrypt password: error running memsqlctl: error running command: `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/run/user/1000/singlestoredb-toolbox241824068/1.socket 192.168.0.215 memsqlctl --json --yes --config /var/lib/memsql/nodes.hcl upsert-secure-key`: exit status 1: 
stdout: open /var/lib/memsql/nodes.hcl.lock: permission denied
stderr: (empty)

So I copied the nodes.hcl.lock file from the existing leaf to the new node. Working through that now.

Given that I created the new node VM via clone, your issue may be different but thought I would share this post in case it helps.