Before I answer your question, I noticed the cluster size that you referenced is larger than the free tier allocation. If you are a paying MemSQL customer, you are eligible for Enterprise Support through the MemSQL support team, which is where you can file these types of issues and errors that you encounter in the product.
See our Support FAQ for more details.
To your specific question:
It may be difficult to pinpoint exactly what happened without monitoring data, but the error likely indicates that there is a query that was compiling that overused the allocated compilation memory.
In MemSQL 6.5, we added the
max_compilation_memory_mb engine variables. The former is the timeout for compiling a query. If the timeout is reached, an error is returned. The latter is the maximum amount of memory used to compile a query. If this maximum is reached, an error is returned, which is similar to the error you saw.
What was the query you were running that returned the error? We recommend running EXPLAIN on that query, and a PROFILE, if possible.
Additionally, you could also check the memsql.log to determine what other issues were occurring during the time of the error. The memsql.log lives in the tracelogs directory of your nodes, and among other errorr and warnings, it will print queries that are being compiled on the first time of your cluster. We recommend you look in the logs during the time of the issue to determine if it provides any clues. The logs will be in the following directory, assuming a standard installation
: /var/lib/memsql/<node id>/tracelogs/memsql.log
Alternatively, we have some monitoring tables in information_schema that may provide some history of events in the cluster.