OOM Crashes in SingleStore on OCP

Help
1

Hi everyone, I’m currently running a SingleStore deployment on OpenShift (OCP) with the following setup:
6 Aggregators:

  • Total memory per pod: 192G
  • Maximum_memory: 182G
    20 Leaves
  • Total memory per pod: 256G
  • Maximum_memory: 235G
    I’ve also configured resource pools to control memory usage across different projects
    However, I’m facing a recurring issue: pods are still getting killed due to OOM at the system level, not by the internal SingleStore limits.
    The query consuming too much memory is not being killed by SingleStore, and instead causes the entire pod to crash. This then triggers a lengthy recovery process: rejoining the cluster, rebalancing partitions, etc. — which severely impacts uptime and stability.

So, my questions are
Why doesn’t SingleStore terminate the runaway query once it exceeds the configured memory limits (like in the resource pool or maximum_memory)?
What could be causing the memory leak or runaway memory usage that bypasses these internal safeguards?
How can I better manage or mitigate this behavior to ensure the cluster stays healthy and stable without relying on Kubernetes OOM killing?

Thanks in advance

2

I see you’ve already opened a support ticket so you’re getting help from our support team.

For the benefit of others, here’s an AI-generated response. This isn’t fully verified – so use your judgement when following these guidelines.

Bottom Line

Kubernetes/OS-level OOM kills happen because some real memory allocations are outside SingleStore’s direct control or accounting. Bridging the gap requires conservative tuning—leave a substantial headroom between pod memory limits and SingleStore’s own max limits, tightly assign and audit all resource pools, enable disk spilling, and proactively monitor Linux-level and process memory use. If issues persist, working with deeper log analysis and possibly escalating with SingleStore support can shed more precise light on the most memory-intensive workloads and help drive improvements in future versions.

More Details

Thank you for your detailed description and thoughtful questions. Your situation—pods running SingleStore on OpenShift being OOM-killed at the system level, even with resource pools and maximum_memory configured—is a complex but increasingly common issue for containerized SingleStore deployments. Let’s break down your questions, explain technical causes, and suggest concrete next steps for a more stable environment.

1. Why doesn’t SingleStore terminate the runaway query once it exceeds the configured memory limits (resource pool or maximum_memory)?

  • Resource Pools & Query Memory Tracking: Ideally, with resource pools and memory limits configured, SingleStore should kill queries that exceed those limits. However, there are known limitations: not all memory allocations by queries are tracked by the resource pool or the internal memory accounting subsystems. For instance, only specific buffer allocations—like Alloc_query_execution buffers and buffer_manager_cached_memory—are considered for enforcement. Other memory usages (table-scan buffers, metadata caches, thread stacks, OS page cache, etc.) are not always “charged” against the query or pool. This can result in queries (especially complex ones, or those using rowstore heavily) consuming memory outside of what’s limited and being missed by the resource governor

  • maximum_memory Limit: SingleStore is supposed to stop allocating memory and terminate queries if node memory use exceeds maximum_memory. However, there can be a lag between going over the limit, detecting it, and actual termination of the offending query. Additionally, background threads, caches, or allocations outside core query tracking (like in native extensions, certain background processes, or thread stack growth) can push actual system usage beyond the threshold before SingleStore has a chance to kill the query.

  • Kubernetes CGroup Limits: Kubernetes (and OpenShift) enforce strict CGroup memory limits at the container/pod level. If memory used by the process—whether or not it is all “counted” by SingleStore’s own tracking—exceeds this, the kernel OOM Killer will forcibly terminate the process, bypassing SingleStore’s own logic.

  • Aggregator vs. Leaf Enforcement: Resource pools typically apply on Leaves, not Aggregators, so a memory-hungry query on an Aggregator might not get killed by the pool’s limit and can trigger a pod OOM. This is a known gap in enforcement.

2. What could be causing the memory leak or runaway usage that bypasses these safeguards?

  • Non-tracked Memory Allocations: Large components of the overall process memory may accrue from sources not tracked by query resource accounting (e.g., planner cache, table cache, background tasks, thread stacks, OS/cache interactions, native code, external libraries, or failed/spilled queries). These can grow, especially under high concurrency or certain workloads.
  • Memory Fragmentation or Retention: Long-lived sessions, unclosed transactions, or failure to evict cached items (such as planner cache entries or table caches) can cause persistent elevated memory utilization, even after the workload subsides.
  • Kubernetes/Container Overhead: The process might be close to its CGroup memory limit even if SingleStore’s internal counters show a healthy buffer, because other allocations (by system libraries, runtime, thread stacks, etc.) are invisible to SingleStore. When this gap becomes too large, the pod can be OOM-killed unexpectedly.
  • Pipelines or Sidecars: If running pipelines or using exporter processes, these can consume memory in the pod outside of SingleStore’s own accounting, leading to unexpected container OOM events.

3. How can I better manage or mitigate this to ensure the cluster stays healthy without relying on Kubernetes OOM killing?

Recommendations:

  • Raise Container Memory Limits / Lower Maximum_memory:
    • The “safest” practice is to leave a buffer between the container’s memory limit and SingleStore’s maximum_memory setting to account for non-tracked allocations. The recommendation is to set maximum_memory to ~80-85% of the pod/container’s total memory (sometimes even less in edge cases), especially since swap is not available in Kubernetes environments.
  • Tune Resource Pools / Enforce at Global Level:
    • Ensure all users (including root or apps using direct connections) are assigned to a resource pool with defined memory limits, and consider enforcing a default pool at the global level to avoid “gaps” where some sessions bypass resource governance.
  • Enable and Monitor Disk Spilling:
    • Disk spilling allows large queries (GROUP BY, SORT/ORDER BY, Window functions, Joins) to offload intermediate results to disk instead of RAM, preventing spikes in memory usage. Confirm that disk spilling (engine variable enable_spilling) is ON, and monitor that spill directories use fast storage (SSD recommended)

5

.

  • Tune or Audit Caches:
    • Set explicit limits on planner cache and compiled query cache using max_planner_cache_memory and max_compiled_query_cache_memory.
    • Periodically monitor and flush caches using SHOW PLANS WHERE active=0, FLUSH PLANS, and audit table memory (e.g., SHOW LEAF MEMORY).
  • Monitor Linux/Pod Overhead:
    • Use system tools (e.g., smem, pmap) inside your container to confirm real RSS (resident set size) and heap usage versus what SingleStore tracks.
  • Audit and Limit External Activity:
    • If running export pipelines, sidecars, or processes alongside SingleStore in the same pod, move them out or restrict their memory budgets.
  • Advanced: Adjust Kubernetes OOM Score:
    • Set oom_score_adj for non-SingleStore containers in the pod or node to reduce the chances of SingleStore being the first process killed (advanced topic; use with care).
  • System-Level Tuning:
    • Ensure system values (like vm.min_free_kbytes) are tuned per SingleStore documentation to reduce spurious allocation failures.

Investigative Actions:

  • Cluster/Pod Reports:
    • Continue uploading cluster reports, tracelogs, and (critically) dmesg outputs after OOM events to SingleStore support for confirmation on actual killing process and investigation of memory patterns.
  • Resource Usage Queries:
    • Regularly check:
      • SHOW LEAF MEMORY
      • SHOW STATUS EXTENDED
      • SHOW RESOURCE POOLS
      • SHOW PLANS
    • This identifies high-memory components and potential leaks.

Limitations & Open Issues

  • Not All Allocations Covered: There are acknowledged gaps in SingleStore’s ability to track/or terminate all memory-intensive sessions, especially with highly complex queries or when part of the memory use is out of resource pool visibility.
  • Kubernetes-level Enforcement is Final: The kernel OOM Killer acts on real RSS usage—anything (including non-MySQL system allocations in the pod) that pushes a pod’s usage past CGroup limits will cause an immediate kill, regardless of SingleStore’s awareness.
  • Improvements in Newer Releases: Some improvements in memory accounting and cache controls have shipped in recent versions (8.9/9.x); check release notes and consider upgrades if running an older engine, and especially review the Kubernetes Operator release notes for recent memory-related improvements

8

.

Related References