We already have existing cluster, with two leaf node. We want to add another three leaf node. Already setup the machine for that, can ssh without password and all connection is good.
So what is the next step to add this three clean node to existing running cluster ? What is the step?
Need to sdb-admin add-leaf or sdb-admin register node ?
and sometimes we got this error when try to add leaf or register node no host found with hostname xxxx
We try to find at documentation but still struggle to arrange the right step to do this.
I was able to get past the “no host found” by using a variation of the solution found in this post. Here is what I did
$ memsql-toolbox-config register-host --host 192.168.0.215 --memsqlctl-config-path /var/lib/memsql/nodes.hcl
✓ Successfully performed host check. No duplicate hosts found.
Toolbox is about to register the following host:
· Host: 192.168.0.215
· Localhost: false
· Memsqlctl Config Path: /var/lib/memsql/nodes.hcl
Would you like to continue? [y/N]: y
Toolbox is about to perform the following actions:
· Update specific system files on you host for the initial cluster configuration.
Files to be added:
- /etc/sysctl.d/90-singlestore.conf
Files to be updated:
- /sys/kernel/mm/transparent_hugepage/defrag
- /sys/kernel/mm/transparent_hugepage/enabled
- /etc/security/limits.conf
Add a new systemd service, if available:
- /etc/systemd/system/singlestoreConfigOS.service - .service file
- /etc/singlestoreConfigOS - service logic file
To skip this step, re-run this command with the --skip-auto-config flag.
The host may require a reboot for these changes to take effect. To restart your cluster after a reboot, run 'sdb-admin restart-node --all'.
Would you like to continue? [y/N]: y
✓ Successfully registered host 192.168.0.215
+---------------+------------+---------------+---------------+
| Host | Local Host | SSH address | Identity File |
+---------------+------------+---------------+---------------+
| 192.168.0.215 | No | 192.168.0.215 | |
+---------------+------------+---------------+---------------+
Now I get different issues when trying to run sdb-admin create-node --availability-group 1 --host 192.168.0.215 --role leaf. I created the new VM for the node by cloning an existing leaf and clearing out the /var/lib/memsql directory. With no /var/lib/memsql/nodes.hcl on the new host. I had errors
Enter the password for the database user:
could not determine memsqlctl user: error running memsqlctl: error running command: `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/run/user/1000/singlestoredb-toolbox745496104/1.socket 192.168.0.215 memsqlctl --json --yes --config /var/lib/memsql/nodes.hcl env`: exit status 1:
stdout: invalid argument "/var/lib/memsql/nodes.hcl" for "-c, --config" flag: stat /var/lib/memsql/nodes.hcl: no such file or directory
So I copied that file to the new host from an exsiting leaf node. New issue
sdb-admin create-node --availability-group 1 --host 192.168.0.215 --role leaf
Enter the password for the database user:
Toolbox is about to perform the following actions on host 192.168.0.215:
· Run 'memsqlctl create-node --password ●●●●●●'
Would you like to continue? [y/N]: y
failed to encrypt password: error running memsqlctl: error running command: `/usr/bin/ssh -oBatchMode=yes -q -tt -oControlPath=/run/user/1000/singlestoredb-toolbox241824068/1.socket 192.168.0.215 memsqlctl --json --yes --config /var/lib/memsql/nodes.hcl upsert-secure-key`: exit status 1:
stdout: open /var/lib/memsql/nodes.hcl.lock: permission denied
stderr: (empty)
So I copied the nodes.hcl.lock file from the existing leaf to the new node. Working through that now.
Given that I created the new node VM via clone, your issue may be different but thought I would share this post in case it helps.
Thinking it might be an issue with using a clone, I created a new VM from scratch and then went through the Cluster Expansion steps: sdb-toolbox-config register-host, sdb-deploy install, and sdb-admin create-node. But I ended back up at the stdout: open /var/lib/memsql/nodes.hcl.lock: permission denied problem.