I have a problem but with 7.3.12. I moved all the rowstore tables into a columnstore (was not the idea but I dont know what else to try)
And still everything is randomly failing due to memory usage, event a simple create table command.
I run this and I’m using only 7gb of memory (my cluster has 124 gb)
select DATABASE_NAME ,table_name, sum(memory_use)/(1024 1024 1024) m
from information_schema.table_statistics
group by table_name,DATABASE_NAME
order by 3 desc
I not sure what’s causing the memory consumption. The only different thing is that we created 50 databases with rowstore tables, but all of them are empty… Does it allocate memory as well?
| Aborted_clients | 290411 |
|---|---|
| Aborted_connects | 119820 |
| Active_dedicated_admin_connections | 0 |
| Aggregator_id | 1 |
| Alloc_analyze | 326.750 MB |
| Alloc_client_connection | 3.625 MB |
| Alloc_compiled_unit_sections | 23.383 (+4.602) MB |
| Alloc_databases_list_entry | 1.625 MB |
| Alloc_deleted_version | 164.250 MB |
| Alloc_global_func | 0.125 MB |
| Alloc_hash_buckets | 329.311 MB |
| Alloc_internal_key_node | 66.000 MB |
| Alloc_plan_cache | 0.625 MB |
| Alloc_protocol_packet | 47.375 (-1.625) MB |
| Alloc_replication | 1.625 MB |
| Alloc_sharding_partitions | 0.625 MB |
| Alloc_skiplist_tower | 384.500 (+0.875) MB |
| Alloc_table_autostats | 21.173 MB |
| Alloc_table_memory | 1672.233 (+1.750) MB |
| Alloc_table_metadata_cache | 1.500 MB |
| Alloc_table_primary | 426.500 (+0.375) MB |
| Alloc_thread_stacks | 194.000 MB |
| Alloc_unit_ifn_thunks | 1.711 (+0.204) MB |
| Alloc_unit_images | 4.846 (+0.916) MB |
| Alloc_variable | 280.500 (+0.500) MB |
| Alloc_variable_allocated | 265.6 MB |
| Alloc_variable_bucket_104 | allocs:880 alloc_MB:0.1 buffer_MB:0.1 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_1168 | allocs:52 alloc_MB:0.1 buffer_MB:0.1 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_11896 | allocs:21 alloc_MB:0.2 buffer_MB:0.2 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_128 | allocs:1779588 alloc_MB:217.2 buffer_MB:224.5 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_130960 | allocs:0 alloc_MB:0.0 buffer_MB:1.9 cached_buffer_MB:1.9 |
| Alloc_variable_bucket_14544 | allocs:4 alloc_MB:0.1 buffer_MB:0.4 cached_buffer_MB:0.2 |
| Alloc_variable_bucket_1480 | allocs:91 alloc_MB:0.1 buffer_MB:0.2 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_16 | allocs:41999 alloc_MB:0.6 buffer_MB:0.8 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_160 | allocs:1087 alloc_MB:0.2 buffer_MB:0.2 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_1832 | allocs:21 alloc_MB:0.0 buffer_MB:0.1 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_18696 | allocs:15 alloc_MB:0.3 buffer_MB:2.1 cached_buffer_MB:1.8 |
| Alloc_variable_bucket_200 | allocs:42 alloc_MB:0.0 buffer_MB:0.1 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_21816 | allocs:0 alloc_MB:0.0 buffer_MB:0.1 cached_buffer_MB:0.1 |
| Alloc_variable_bucket_2288 | allocs:21 alloc_MB:0.0 buffer_MB:0.1 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_24 | allocs:1809226 alloc_MB:41.4 buffer_MB:41.9 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_248 | allocs:118 alloc_MB:0.0 buffer_MB:0.1 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_26184 | allocs:0 alloc_MB:0.0 buffer_MB:0.1 cached_buffer_MB:0.1 |
| Alloc_variable_bucket_2832 | allocs:49 alloc_MB:0.1 buffer_MB:0.2 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_312 | allocs:1969 alloc_MB:0.6 buffer_MB:0.6 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_32 | allocs:17209 alloc_MB:0.5 buffer_MB:0.6 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_32728 | allocs:0 alloc_MB:0.0 buffer_MB:0.1 cached_buffer_MB:0.1 |
| Alloc_variable_bucket_3528 | allocs:11 alloc_MB:0.0 buffer_MB:0.1 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_384 | allocs:55 alloc_MB:0.0 buffer_MB:0.1 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_40 | allocs:1902 alloc_MB:0.1 buffer_MB:0.1 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_43648 | allocs:0 alloc_MB:0.0 buffer_MB:0.1 cached_buffer_MB:0.1 |
| Alloc_variable_bucket_4504 | allocs:12 alloc_MB:0.1 buffer_MB:0.1 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_48 | allocs:676 alloc_MB:0.0 buffer_MB:0.1 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_480 | allocs:22 alloc_MB:0.0 buffer_MB:0.1 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_56 | allocs:1562 alloc_MB:0.1 buffer_MB:0.1 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_5680 | allocs:30 alloc_MB:0.2 buffer_MB:0.2 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_600 | allocs:237 alloc_MB:0.1 buffer_MB:0.2 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_6224 | allocs:42 alloc_MB:0.2 buffer_MB:0.4 cached_buffer_MB:0.1 |
| Alloc_variable_bucket_64 | allocs:1622 alloc_MB:0.1 buffer_MB:0.1 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_65472 | allocs:0 alloc_MB:0.0 buffer_MB:0.1 cached_buffer_MB:0.1 |
| Alloc_variable_bucket_72 | allocs:3601 alloc_MB:0.2 buffer_MB:0.4 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_7264 | allocs:69 alloc_MB:0.5 buffer_MB:0.5 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_752 | allocs:42 alloc_MB:0.0 buffer_MB:0.1 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_80 | allocs:1516 alloc_MB:0.1 buffer_MB:0.1 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_88 | allocs:563 alloc_MB:0.0 buffer_MB:0.1 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_9344 | allocs:230 alloc_MB:2.0 buffer_MB:2.1 cached_buffer_MB:0.0 |
| Alloc_variable_bucket_936 | allocs:60 alloc_MB:0.1 buffer_MB:0.1 cached_buffer_MB:0.0 |
| Alloc_variable_cached_buffers | 4.6 MB |
| Alloc_warnings | 4.875 MB |
| Auto_attach_remaining_seconds | 0 |
| Average_garbage_collection_duration | 34 ms |
| Buffer_manager_cached_memory | 2046.2 (+6.5) MB |
| Buffer_manager_memory | 3762.9 (+6.6) MB |
| Buffer_manager_unrecycled_memory | 6.1 MB |
| Bytes_received | 5455422937201 |
| Bytes_sent | 3489911125073 |
| Columnstore_ingest_management_active_queries | 0 |
| Columnstore_ingest_management_estimated_memory | 0.000 MB |
| Columnstore_ingest_management_estimated_segments_to_flush | 0 |
| Columnstore_ingest_management_max_concurrency | 0 |
| Columnstore_ingest_management_queued_queries | 0 |
| Connections | 978906 |
| Context_switch_misses | 4307537 |
| Context_switches | 5098251 |
| Disk_space_reserved_for_secondary_index | 0 |
| Execution_time_of_reads | 1065018166 ms |
| Execution_time_of_write | 1828144188 ms |
| Failed_read_queries | 4409 |
| Failed_write_queries | 38434 |
| Free_io_pool_memory | 0.2 MB |
| GCed_versions_last_sweep | 0 |
| Idle_queue | 0 |
| Inflight_async_compilations | 0 |
| Ingest_errors_disk_space_use | 2762284 Bytes |
| License_capacity | 4 units |
| License_expiration | 0 |
| License_key | d44f474d16c842b3ab38554cb5dc37a5 |
| License_type | free |
| License_version | 4 |
| Linux_resident_memory | 6565.442 (+104.227) MB |
| Linux_resident_shared_memory | 14.481 (-4.636) MB |
| Malloc_active_memory | 639.214 (-0.140) MB |
| Malloc_transaction_cached_memory | 203.891 MB |
| Max_used_connections | 193 |
| Maximum_cluster_capacity | 4 units |
| Prepared_stmt_count | 15 |
| Queries | 915800993 |
| Query_compilation_failures | 0 |
| Query_compilations | 3138 |
| Questions | 915800993 |
| Ready_queue | 0 |
| Row_lock_wait_time | 0 ms |
| Rows_affected_by_writes | 13177963267 |
| Rows_returned_by_reads | 19002654302 |
| Seconds_until_expiration | -1 |
| Ssl_accept_renegotiates | 0 |
| Ssl_accepts | 0 |
| Ssl_callback_cache_hits | 0 |
| Ssl_cipher | |
| Ssl_cipher_list | |
| Ssl_client_connects | 0 |
| Ssl_connect_renegotiates | 0 |
| Ssl_ctx_verify_depth | 18446744073709551615 |
| Ssl_ctx_verify_mode | 0 |
| Ssl_default_timeout | 0 |
| Ssl_finished_accepts | 0 |
| Ssl_finished_connects | 0 |
| Ssl_session_cache_hits | 0 |
| Ssl_session_cache_misses | 0 |
| Ssl_session_cache_mode | SERVER |
| Ssl_session_cache_overflows | 0 |
| Ssl_session_cache_size | 20480 |
| Ssl_session_cache_timeouts | 0 |
| Ssl_sessions_reused | 0 |
| Ssl_used_session_cache_entries | 0 |
| Ssl_verify_depth | 0 |
| Ssl_verify_mode | 0 |
| Ssl_version | |
| Successful_read_queries | 290015106 |
| Successful_write_queries | 622224525 |
| Threads_background | 1 |
| Threads_cached | 187 |
| Threads_connected | 380 |
| Threads_created | 193 |
| Threads_idle | 374 |
| Threads_running | 1 |
| Threads_shutdown | 0 |
| Threads_waiting_for_disk_space | 0 |
| Total_blobs_processed_for_fsync | 0 |
| Total_blobs_submitted_for_fsync | 0 |
| Total_dedicated_admin_connections | 0 |
| Total_io_pool_memory | 13.4 MB |
| Total_server_memory | 4989.9 (-0.1) MB |
| Transaction_buffer_wait_time | 0 ms |
| Transaction_log_flush_wait_time | 656 ms |
| Used_cluster_capacity | 4 units |
| Used_instance_license_units | 0 |
| Workload_management_active_connections | 0 |
| Workload_management_active_queries | 0 |
| Workload_management_active_threads | 0 |
| Workload_management_queued_queries | 0 |
Memory used by MemSQL (20280.25 Mb) has reached the ‘maximum_memory’ setting (20480 Mb) on this node. Possible causes include (1) available query execution memory has been used up for table memory (in use table memory: 12998.38 Mb) and (2) the query is large and complex and requires more query executi
Callstack:
and sometimes the error is in use table memory: 0 Mb