chen
January 29, 2021, 1:52pm
1
Hi Team,
We have deployed MEMSQL version 7.1.3 on AKS with 2 aggregators and 8 nodes and plan to scale down the cluster to 4 nodes at a set time in the week.
Deployment details:
Thanks and Regards
adam
February 1, 2021, 11:38pm
2
Hi Chen,
I think we need a bit more information to debug.
To start with can I see show status extended output from one of the leaves with higher memory use and one of the leaves with lower memory use?
-Adam
chen
February 2, 2021, 2:40pm
3
Hi Adam,
Leaf with higher memory:
chen
February 2, 2021, 2:42pm
4
Leaf with lower memory use:
| Aborted_clients | 84920 |
adam
February 3, 2021, 3:33pm
5
Hi Chen,
It looks like the plancache is the main difference in memory use.
On the leaf with higher memory use:
| Alloc_unit_images | 10014.628 (-2340.152) MB |
On the leaf with lower memory use:
| Alloc_unit_images | 971.872 (+55.763) MB |
This is memory for query plans stored in the plancache. Its likely the new nodes haven’t compiled as many query shapes as of yet. Does your application generate a lot of unique query shapes (does it vary filters or join patterns from query to query)?
Some things to look into:
Check on the types of queries you see in the plancache (select * from information_schema.plancache). Do you see anything unexpected? Singlestore will cache a single plan if you vary the value of constants (say where c = 1 and where c = 2), but otherwise we run the optimizer and generate new plans.
plan_expiration_minutes is the knob that controls how aggressively plans are removed from the plancache (if not re-used in plan_expiration_minutes its evicted). You could try lowering this knob (its 12 hours by default I believe). If you want to flush the plancache entirely you can set it to 0 and wait for a bit and then set it back to a good value.
Singlestore 7.3 uses less memory for plans, you would probably save 10 GB+ just by upgrading.
-Adam
