Unable to Take backup using NFS in singlestore


SS Version: 8.0.5 with 1 Leaf and 1 Master Aggregator
I am using NFS File system to take backups of DB and store it in NFS.
While taking backup of test DB , getting below error:


[memsql@rnd-2 singlestoredb-toolbox-1.14.4-e432b273e6]$ sudo exportfs -v
/nfs/general    <>(sync,wdelay,hide,no_subtree_check,sec=sys,rw,secure,no_root_squash,no_all_squash)
/home           <>(sync,wdelay,hide,no_subtree_check,sec=sys,rw,secure,no_root_squash,no_all_squash)
[memsql@rnd-2 singlestoredb-toolbox-1.14.4-e432b273e6]$ vim /etc/exports
[memsql@rnd-2 singlestoredb-toolbox-1.14.4-e432b273e6]$ ll /nfs
total 0
drwxr-xr-x. 3 memsql memsql 138 Jan 23 10:20 general
[memsql@rnd-2 singlestoredb-toolbox-1.14.4-e432b273e6]$ sudo chmod 777 -R /nfs
[memsql@rnd-2 singlestoredb-toolbox-1.14.4-e432b273e6]$ ./sdb-admin create-backup -r file:///nfs/general test
Toolbox will perform the following actions:
  · backup "test" to file:///nfs/general/2023-01-23T103008-04688-12324301643728956639-test

Would you like to continue? [y/N]: y
✘ failed to backup "test" to file:///nfs/general/2023-01-23T103008-04688-12324301643728956639-test:
error running memsqlctl: error running command: `/home/memsql/memsql/singlestoredb-server-8.0.5-249cb54305/memsqlctl --json --yes --config /home/memsql/memsql/memsqlctl.hcl query --memsql-id 284B9D52BBECE935EF0CAA3109684E3017D70904 --sql BACKUP DATABASE `test` TO "/nfs/general/2023-01-23T103008-04688-12324301643728956639-test" --row-timeout 24h0m0s --strict-mode`: exit status 1:
stdout: (empty)
stderr: could not process query results: Error 1789: Leaf Error (<....:3311>): Backup operation could not be completed because path '/nfs/general/2023-01-23T103008-04688-12324301643728956639-test/test_0.backup' on node '<..>:3311' is inaccessible or points to a file that already exists.  Error returned from target:Failed to open file: /nfs/general/2023-01-23T103008-04688-12324301643728956639-test/test_0.backup (13:Permission denied)
failed to create 1 backup(s) out of 1

In SS GUI Events sections, it’s saying backup completed.


NFS back point:

df -h
<master_aggregator_ip>:/nfs/general   80G   48G   33G  59% /nfs/general

NFS is installed in Master Aggregator VM.

Backup Logs Stored in NFS::

[memsql@rnd-1 nfs]$ cd general/
[memsql@rnd-1 general]$ ls
2023-01-23T110240-463f9-12324301643728956639-test  2023-01-23T110507-4e7fb-12324301643728956639-test  file
[memsql@rnd-1 general]$ cd 2023-01-23T110240-463f9-12324301643728956639-test/
[memsql@rnd-1 2023-01-23T110240-463f9-12324301643728956639-test]$ ls
BACKUP_INCOMPLETE  test.backup  test.backup_columns0.tar  test.backup.manifest
[memsql@rnd-1 2023-01-23T110240-463f9-12324301643728956639-test]$ pwd
[memsql@rnd-1 2023-01-23T110240-463f9-12324301643728956639-test]$ sudo cat BACKUP_INCOMPLETE
{"Backup_Id":"1002","Cluster_Name":"memsql_cluster","Cluster_Id":"12324301643728956639","Database_Name":"test","Start_Timestamp":"2023-01-23 11:02:44","End_Timestamp":"2023-01-23 11:02:45","Num_Partitions":"-1","Backup_Path":"/nfs/general/2023-01-23T110240-463f9-12324301643728956639-test/test.backup","Checksum":"","Status":"Failure","Size":"0","Incr_Backup_Id":"-1","Type":"FULL","Server_Version":"8.0.5","Memory_Usage":"0","Implementation":"Lockfree","Lockfree_Timestamp":"0"}[memsql@rnd-1 2023-01-23T110240-463f9-12324301643728956639-test]$

I have mounted /nfs in leaf1 node:

When I tried to take local back as well as backup in nfs mounted drive, the backup is running.I run SS GUI commands as root user. While running commands inside vm, we are getting error.


When I tried to restore DB xyz from NFS Mounted drive, I am getting below errorr:

SS Commands:

RESTORE DATABASE xyz FROM "/nfs/general";


RESTORE DATABASE xyz FROM "/nfs/general"
ERROR 1792 ER_BACKUP_IO_ERROR: Failed to stat file: /nfs/general/Arcion_test_0.backup (13:Permission denied)

When I tried to restore DB from Local backup, Restoration of DB is successful.

RESTORE DATABASE xyz FROM "/home/memsql/backup"


Hey divyank1,

I hear ya, thanks for reporting/sharing the detail, and I’ll get someone to look into that ASAP.
Stand by …