Hello, I’m trying to use the BACKUP command with the INIT option (for later incremental) to an S3 storage, but the query runs extremely slow.
My database has a size of 50 GB compressed. The query uploads about to ~4 GB per DAY to the storage. It would take about two weeks to complete the INIT one.
The database server is not overloaded. It keeps ~10% of CPU and ~20% of RAM. Trying to do a local backup takes only ~3 minutes to complete.
My SingleStore version: 7.8.13.
Incremental backup performance was improved in 7.8.18.
This also applies to the INIT backup part. But 4GB per day seems incredibly slow so we’re not sure that is the only issue. But give it a try and comment back whether it worked, if you don’t mind.
Also, if it does not fix things, please give more details about your HW/network, and what kind of S3-compatible storage you are using. Thanks.
I just upgraded to SingleStore 8.0.15 and it seems it’s now fixed! I just made the INIT backup in 14 minutes.
Thanks @hanson !
Fantastic! Thank you for writing back to let us know that it worked.
@hanson It seems the RESTORE DATABASE, even using the latest version, is extremely slow (It will take 2 weeks to complete). I tried using the AWS CLI, and the backup was downloaded in less than 1 hour.
I’m currently using Vultr Object Storage (Amsterdan location). My server is AX101 from Hetzner (Finland location).
Current SingleStore version: 8.0.16
So, you are backing up to to Vultr from your Hetzner (cloud service provider) over the internet? Is that how you ran your INIT backup in 14 minutes? And now you are trying to restore from that backup?
Yes. I’m trying to restore the backup in another server on hetzner (same specs & same location).
Got it. I passed this on to our devs. We’re going to investigate this but don’t have a timetable for for speeding up your RESTORE scenario. Their suggestion was to do a full backup and restore, not incremental (or INIT). That should work faster (fewer, larger files would be transferred).
Thanks @hanson. It doesn’t seem to be an expected behavior. The incremental backup has only 10 incremental runs, there is not that much amount of files. I hope it gets fixed soon.
I just used the full backup here and recovered it in 1 hour.