System doesnt use new added Leaf

Help
1

I install a cluster on version 7.6.4 with 1 master agg, 1 child agg and 2 leaves. after load some data I remove child agg and add leaf instead of it.
I rebalanced the database. the rebalancing end quickly and status OK.

REBALANCE PARTITIONS ON DWH2
Query OK, 1 row affected (79 ms)

But system doesnt use new leaf. and I create a new db and create the tables on this db. I load the data to new database. again system doesnt use the new leaf.

when I look the leaves, I see there is no pair host on new leaf.

[mskunt@datawh03 ~]$ sdb-admin show-leaves
✓ Successfully ran ‘memsqlctl show-leaves’
±--------------±-----±-------------------±--------------±----------±-------±-------------------±-------------------------------+
| Host | Port | Availability Group | Pair Host | Pair Port | State | Opened Connections | Average Roundtrip Latency (ms) |
±--------------±-----±-------------------±--------------±----------±-------±-------------------±-------------------------------+
| 10.100.xx.222 | 3306 | 1 | 10.100.xx.224 | 3306 | online | 109 | 0.461 |
| 10.100.xx.224 | 3306 | 2 | 10.100.xx.222 | 3306 | online | 109 | 0.468 |
| 10.100.xx.221 | 3306 | 1 | null | null | online | 5 | 0.396 |
±--------------±-----±-------------------±--------------±----------±-------±-------------------±--------------------------

image
image1193×328 20.7 KB

3 Likes
2

I find the problem. The problem is ha need even number of nodes. I have 3 nodes so one of them is oout of game :smiley:

If I want to use 3 leaves, I need to disable ha. :frowning:

Who knows what is the price of license for addional node ?

3

I’m glad you found a solution, @mskunt! For questions about pricing, send mail to team@singlestore.com and they can help you.

4

Hello @mskunt,

You can use 3 nodes with HA. You don’t need an even number of nodes if you use HA in “load_balanced” mode instead of “paired” mode. See the second section in this article: Enabling High Availability · SingleStore Documentation

2 Likes
5

Hi Prerak
thanks for great solution.

6

Hi @prerak
I change mode to load balanced and add a new leaf. after that rebalanced the db. after these my node data distribution be like below. 2 node has same size, but 3rd has 2 time more data. is it normal ?
thanks

image
image1390×438 17.5 KB

7

Hi @mskunt,

In my experience, disk usage is not a good measure of whether all nodes have equal number of partitions. Singlestore disk usage is somewhat unpredictable because of preallocated files, etc.

I would ask you try this command, and if that produces an empty set, then you are fully rebalanced: EXPLAIN REBALANCE ALL DATABASES

Another useful command is: SHOW CLUSTER STATUS

If you want to see how many partitions are on each leaf node, try this:

select node_id, host, count(*) from information_schema.MV_CLUSTER_STATUS 
where node_id in (select distinct node_id from information_schema.LEAVES)
group by 1;

If you find that equal(ish) number of partitions, then that is another confirmation.

Lastly, if all looks good, you can possibly release some disk space by issuing SNAPSHOT DATABASE <db_name> for all of your databases.

8

thanks for great information.
“EXPLAIN REBALANCE ALL DATABASES” give empty set
and then run the query u sent. its result is :

image

should it be same partition count on all nodes? or this status is normal.

and this is SHOW LEAVES command result:

image
image1696×126 6.2 KB

thanks

9

I think in this scenario, this is normal. This is how Singlestore is deciding to balance the partitions.

10

thanks for reply prerak.
so the time is loading lots of data to singlestore :slightly_smiling_face: