How to delete logs files inside data/logs to free disk storage

1

Hi,

I am having Cluster with 4 Leaf and 1 Aggregator Nodes (in Linux VM). Approx DB size in cluster in 20GB.

The logs directory in /data in all Leaf Nodes taking disk storage of more than 500GB.

I want to free up that space consumed by logs

$ du -shc /abc/memsql/<Leaf Nodes>/data/*

148G      /abc/memsql/Leaf1/data/logs

212G      /abc/memsql/Leaf2/data/logs

148G      /abc/memsql/Leaf3/data/logs

1.6G      /abc/memsql/Leaf4/data/logs

Leaf4 logs was deleted manually to free up some space, as Leaf4 went down due to full disk storage.But I didn’t see any DB issue till now.

Ref Link: Stating Not to delete/modify data dir.

Let me know how to free up disk storage without breaking SS cluster.

1 Like
2

Hi,

Its hard to give advice with just this information, but this kb article has a lot of detail on how S2DB uses disk:

How many databases do you have and how many partitions on each? Lowering snapshot_trigger_size and and running a manual snapshot is one quick way to lower disk space usage. Another approach is to create your databases with fewer partitions, though this can only be done a CREATE DATABASE time right now.

3

Hi @adam,
Thanks for the response.

Most of DB’s having 64 partitions. Total DB’s are 23.

Lowering snapshot_trigger_size and and running a manual snapshot is one quick way to lower disk space usage.

Any good article on this wiil help.

Another approach is to create your databases with fewer partitions, though this can only be done a CREATE DATABASE time right now.

Can you tell Ideal Partitions for DB size of 50GB each?

Snap:

image
image1032×489 42.6 KB

4

How many leaf nodes in the cluster?

If you expect the dbs to always be reasonably small (as in, never grow to TBs of data) you can likely you just use 2-3 partitions per leaf (or even less) depending on your query patterns.

If you want to reduce the snapshot_trigger_size you could drop it to 512mb or 256 mb via:

for example:

sdb-admin update-config --all --set-global --key "snapshot_trigger_size" --value "536870912"

-Adam

5

Hi @adam ,

4 Leaf Nodes (Stated in first post)

If you want to reduce the snapshot_trigger_size you could drop it to 512mb or 256 mb via:If you want to reduce the snapshot_trigger_size you could drop it to 512mb or 256 mb via:

ok, but after reducing snapshot_trigger_size:

  1. Will it take automatically take snapshot of DB’s?
  2. I have previous snapshots present dir: data/snapshots consist of 1.5GB in each leaf node, may be it will be replaced with new snapshot. But How will I know snapshot is taken successfully after reducing snapshot_trigger_size ??
  3. Once snapshot is confirmed can I remove files in log dir: data/logs to free up 500+ GB disk storage?

Lowering snapshot_trigger_size and and running a manual snapshot is one quick way to lower disk space usage.

I guess manual snapshot will do the job as well,but it’s very timeconsuming, it took 40 mins to take snapshot of single DB still did not get completed.

Is there any method to make manual snapshot of entire cluster in single terminal command?

1 Like