Hi, I’m trying to enable autostart in singlestore leaf nodes, because many of our leaftnodes are repeatedly being rebooted for maintenance purposes. But the leaf nodes fail to restart. They work fine when started with sdb-admin, but not with the autostart. How to resolve this?
Logs:
[memsql@host-2 ~]$ sdb-admin enable-nodes-autostart --all -y
✓ Successfully connected to host xx.xx.xx.xx
✓ Successfully connected to host xx.xx.xx.xx
sudo privileges are required to run this command. Do you want to continue?
If this user does not have sudo privileges, select "N" and the command will continue with the '--create-service-only' flag. [Y/n]:
Automatically selected yes, non-interactive mode enabled
Toolbox will perform the following actions
· Write the service file and enable the service on the following hosts
- xx.xx.xx.xx
- xx.xx.xx.xx
Would you like to continue? [Y/n]:
Automatically selected yes, non-interactive mode enabled
✓ Successfully wrote service file to "xx.xx.xx.xx"
✓ Successfully wrote service file to "xx.xx.xx.xx"
✓ Successfully enabled nodes to start on reboot on host xx.xx.xx.xx
✓ Successfully enabled nodes to start on reboot on host xx.xx.xx.xx
[memsql@host-1 ~]$ sudo systemctl reboot -i
Connection to host-1 closed by remote host.
Connection to host-1 closed.
[memsql@host-2 ~]$ ssh memsql@host-1
[memsql@host-1 ~]$ sudo systemctl status memsql.service
● memsql.service - SingleStore
Loaded: loaded (/home/memsql/memsql/memsql.service; enabled; vendor preset: d> Active: failed (Result: exit-code) since Wed 2023-02-08 06:16:48 UTC; 2min 14> Process: 865 ExecStart=/home/memsql/memsql/singlestoredb-server-8.0.5-249cb543> Main PID: 865 (code=exited, status=203/EXEC)
Feb 08 06:16:48 host-1 systemd[1]: Starting SingleStore...
Feb 08 06:16:48 host-1 systemd[1]: memsql.service: Main process exited, code=exit>Feb 08 06:16:48 host-1 systemd[1]: memsql.service: Failed with result 'exit-code'.Feb 08 06:16:48 host-1 systemd[1]: Failed to start SingleStore.
[memsql@host-1 ~]$ sudo systemctl enable memsql.service
[memsql@host-1 ~]$ sudo systemctl start memsql.service
Job for memsql.service failed because the control process exited with error code.
[memsql@host-1 ~]$ sudo systemctl status memsql.service
● memsql.service - SingleStore
Loaded: loaded (/home/memsql/memsql/memsql.service; enabled; vendor preset: d> Active: failed (Result: exit-code) since Wed 2023-02-08 06:19:35 UTC; 1min 57> Main PID: 4984 (code=exited, status=203/EXEC)
Feb 08 06:19:35 host-1 systemd[1]: Starting SingleStore...
Feb 08 06:19:35 host-1 systemd[1]: memsql.service: Main process exited, code=exit>Feb 08 06:19:35 host-1 systemd[1]: memsql.service: Failed with result 'exit-code'.Feb 08 06:19:35 host-1 systemd[1]: Failed to start SingleStore.
[memsql@host-1 ~]$ sudo systemctl start memsql.service
Job for memsql.service failed because the control process exited with error code.
[memsql@host-1 ~]$ sudo journalctl -xe
Feb 08 06:21:32 host-1 systemd[1]: Configuration file /home/memsql/memsql/memsql.service is marked executable. Please remove ex>Feb 08 06:21:41 host-1 sudo[5155]: pam_unix(sudo:session): session closed for user root
Feb 08 06:21:57 host-1 sudo[5166]: memsql : TTY=pts/0 ; PWD=/home/memsql ; USER=root ; COMMAND=/bin/systemctl start memsql.se>Feb 08 06:21:57 host-1 sudo[5166]: pam_systemd(sudo:session): Cannot create session: Already running in a session or user slice
Feb 08 06:21:57 host-1 sudo[5166]: pam_unix(sudo:session): session opened for user root by memsql(uid=0)
Feb 08 06:21:57 host-1 systemd[1]: Starting SingleStore...
-- Subject: Unit memsql.service has begun start-up
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit memsql.service has begun starting up.
Feb 08 06:21:57 host-1 systemd[5170]: memsql.service: Failed to execute command: Permission denied
Feb 08 06:21:57 host-1 systemd[5170]: memsql.service: Failed at step EXEC spawning /home/memsql/memsql/singlestoredb-server-8.0>-- Subject: Process /home/memsql/memsql/singlestoredb-server-8.0.5-249cb54305/memsqlctl could not be executed
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- The process /home/memsql/memsql/singlestoredb-server-8.0.5-249cb54305/memsqlctl could not be executed and failed.
--
-- The error number returned by this process is 13.
Feb 08 06:21:57 host-1 systemd[1]: memsql.service: Main process exited, code=exited, status=203/EXEC
Feb 08 06:21:57 host-1 systemd[1]: memsql.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- The unit memsql.service has entered the 'failed' state with result 'exit-code'.
Feb 08 06:21:57 host-1 systemd[1]: Failed to start SingleStore.
-- Subject: Unit memsql.service has failed
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit memsql.service has failed.
--
-- The result is failed.
Feb 08 06:21:57 host-1 sudo[5166]: pam_unix(sudo:session): session closed for user root
Feb 08 06:22:05 host-1 dockerd[1151]: time="2023-02-08T06:22:05.089461788Z" level=info msg="NetworkDB stats host-1(e1557a24aec6)>Feb 08 06:22:05 host-1 dockerd[1151]: time="2023-02-08T06:22:05.090047558Z" level=info msg="NetworkDB stats host-1(e1557a24aec6)>Feb 08 06:22:20 host-1 sudo[5188]: memsql : TTY=pts/0 ; PWD=/home/memsql ; USER=root ; COMMAND=/bin/journalctl -xe
Feb 08 06:22:20 host-1 sudo[5188]: pam_systemd(sudo:session): Cannot create session: Already running in a session or user slice
Feb 08 06:22:20 host-1 sudo[5188]: pam_unix(sudo:session): session opened for user