"Failed to stat" error while creating a backup

Off-Topic
1

I am trying to create a backup of my database using the command:

BACKUP dev TO "/media/bigdrive/backups/dev-backup"

However, I am encountering the following error:

ERROR 1899 (HY000) at line 1: Leaf Error (0.0.0.0:3307): Failed to stat 'blobs/2138/8326/48/2815_891': 2 (No such file or directory).

I checked the tracelog, and saw the following:

28663415109 2021-01-17 18:52:17.929   INFO: BACKUP DATABASE dev
28663415169 2021-01-17 18:52:17.929   INFO: Kicking off a distributed backup for database `dev` to directory '/media/bigdrive/backups/dev-backup/dev.backup'
28663493884 2021-01-17 18:52:18.008   INFO: Thread 99923: TakeDatabaseSnapshotHelper: Starting snapshot for db `dev`.
28663493949 2021-01-17 18:52:18.008   INFO: Thread 99923: BeginSnapshot: `dev` log: Setting snapshot term to 3987.
28663493976 2021-01-17 18:52:18.008   INFO: Thread 99923: BeginSnapshot: `dev` log: Taking snapshot `/media/bigdrive/backups/dev-backup/dev.backup`, LSN 0x2000000e2, term 0xf93, version 0x7f, prevEpoch 0.
28663528909 2021-01-17 18:52:18.043   INFO: Thread 99923: SerializeUserTable: BEGIN SERIALIZE TABLE dev._$!$_md_columnstore_sample
28663528963 2021-01-17 18:52:18.043   INFO: Thread 99923: operator(): END SERIALIZE TABLE dev._$!$_md_columnstore_sample
28663540983 2021-01-17 18:52:18.055   INFO: Thread 99923: TakeDatabaseSnapshotHelper: Snapshot for db `dev` succeeded.
28663541052 2021-01-17 18:52:18.055  ERROR: Thread 99923: memsql_rmdir: Failed to remove directory [/media/bigdrive/backups/dev-backup/dev.backup_columns] error: No such file or directory (2)
28919247738 2021-01-17 18:56:33.762  ERROR: Failed taking a distributed backup for database `dev` to directory '/media/bigdrive/backups/dev-backup/dev.backup' failed with (1899:Leaf Error (0.0.0.0:3307): Failed to stat 'blobs/2138/8326/48/2815_891': 2 (No such file or directory).)

Any idea what might be causing this error?

2

Hi nahmed,

Which version of Singlestore are you seeing this error on (select @@memsql_version).

-Adam

3

This is version 7.0.9. I was worried about doing any sort of upgrade before ensuring the backup works.

4

Hi nahmed,

What does this return:

select * from information_schema.columnar_segments where file like '%/2815_891';

-Adam

5

It just returns 'Empty set (14.29 sec)".

By the way, I was able to work around this by running a mysqldump then re-importing the data from the dumped sql file onto another server. Backups work properly now.

6

Yeah, I suspected the issue was with the data in your tables and not with the backup itself.

Newer versions of memsql have commands to check the data + its checksums (and repair them if possible): SingleStoreDB Cloud ยท SingleStore Documentation

-Adam