Crazy disk usage in data/rem

Help
1

For some reason we are collecting massive amounts of data in data/rem with files named like:

metrics_singlestore-leaf-02:3306_963371508_50400794.rem

We run SingleStore DB Version: 8.9.2, it started after the upgrade (we were on 8.7 beforehand)

We have 4.7GB in our database but somehow 91GB in this data/rem folder

How can I clear it out? I have tried googling for ages but can’t find any mention of it anywhere, e.g its not on https://support.singlestore.com/hc/en-us/articles/4406838603156-Understanding-Disk-Usage at all

2

Now updated to 8.9.4 after spotting this in the release notes and hoping its related?

  • Bugfix: Resolved an issue where an in-development subsystem can leak files on disk

Will I need to delete these files manually or does SingleStore clean these up for me?

3

Yep, that sorted it!

So if anyone is wondering if they’re in the same position, update your software :joy:

Would be nice if singlestore documented this somewhere as nothing anywhere mentions these files, even the line in the changelog for the bug fix just hints at some mysterious files existing but not what they are or where they are

I’m having to clean them up manually, it would be nice if SingleStore took care of it though

4

@nexuscare-lukehebb I’m glad you were able to resolve the problem by upgrading. I’ll pass your feedback along to our dev and documentation teams.

5

Good to see you take this into account.

Similarly, we regularly get release notes such as “Adds an advanced setting to skip time-intensive queries in the dashboard”.

It would be very helpful that these messages would be more specific and tell us which setting was added.

6

Thank you for sharing this information…

7

hey guys, I’m having the same issue. Thanks for the info. So after upgrading the db I can safely rm the /rem files?

8

@hanson1 I’m currently at 92% percent of disk used do to the rem folder. Do you think we can safely delete those metrics* files even before upgrading to save the space?