Restore DB in cluster-in-a-box fails

Hi SingleStore,

I’m trying to restore a 2GB backup from GCS on a fresh cluster-in-a-box (7.5) locally (docker with 6 CPU + 10GB RAM).

After few minutes of processing the query crashes with:

ERROR 1970 (HY000): Leaf Error (127.0.0.1:3307): Remote connection timed out waiting for a response from the external endpoint.  No stderr returned.

It’s impossible for me to dev locally anymore.

How to fix this please?

Thanks

Hi Pierre,

That error is a result of very slow download from GCS (I believe the default timeout is subprocess_io_idle_timeout_ms before giving that error). Is the failure consistent ?

We’re likely going to need a cluster report to check on things in a bit more detail.

After inscreasing the subprocess_io_idle_timeout_ms to 10 minutes, I still have the same error, and the first error in the logs is:
ERROR: Thread 99995: createDatabaseHelper: my_db_name: db creation fails because 'HandleBackupTargetForCreateDb' failed with error 5

Also, I tried to download manually the GCS backup, and restore it with RESTORE DATABASE xxx FROM ‘/path’ but it fails to construct to proper path to locate the backup file.

How to RESTORE an S3/GCS backup from a local path please?

Hi Pierre,

Local backups unfortunately have a different format than blob store backups, so directly restoring a S3/GCS backup from a local path will not work. However, it is possible to hack a S3/GCS backup to be able to be restored locally. If you append “.backup” to all the files except for the manifest files and BACKUP_COMPLETE, and restore from /path/[dbname].backup, the restore should work. For example, change db1_0 to db1_0.backup …etc.

As for GCS failing to restore, it is not super clear at the moment. Could you try setting subprocess_io_idle_timeout to 30 minutes as a sanity test and if that fails generate a cluster report?

Thanks @jhuang for the trick, with subprocess_io_idle_timeout to 30 mins, the DB RESTORE worked, tried twice and it completed.
The RESTORE query took a bit more than 10 mins… as long as it works, for a dev env it’s fine!