Cluster-in-a-box - minimal_disk_space error

Hello,
I am launching cluster-in-a-box using the below docker-compose on my mac.
Cluster starts successfully, but when trying to create a database I get the following error:
“CREATE PARTITION DATABASE for operation CREATE DATABASE failed with error 1883:Leaf Error (127.0.0.1:3307): Available disk space is below the value of ‘minimal_disk_space’ global variable (100 MB). This query cannot be executed.”

I tried to mount my local volume to one of the volumes in the container but it didn’t resolve the problem.
To which path do i need to mount? or other way to resolve this?

My docker-compose:

version: ‘2’

services:
memsql:
image: memsql/cluster-in-a-box:centos-7.1.9-3f0cdd2c4f-2.0.2-1.7.1
ports:
- 3306:3306
- 8080:8080
environment:
LICENSE_KEY: ${MEMSQL_LICENSE_KEY}
START_AFTER_INIT: ‘Y’

Container volumes after launch:
bash-4.2$ df -lh
Filesystem Size Used Avail Use% Mounted on
overlay 59G 53G 2.5G 96% /
tmpfs 64M 0 64M 0% /dev
tmpfs 996M 0 996M 0% /sys/fs/cgroup
grpcfuse 234G 143G 77G 66% /dev/shm
/dev/vda1 59G 53G 2.5G 96% /etc/hosts
tmpfs 996M 0 996M 0% /proc/acpi
tmpfs 996M 0 996M 0% /sys/firmware

thanks.

Did you try to mount a local volume onto /var/lib/memsql? In theory that should work.

Specifically to /var/lib/memsql throws an error:
“memsqlctl cannot find lock file /var/lib/memsql/nodes.hcl.lock”

Mounting to /var/lib does work but doesn’t solve the problem

Wow, that’s a bug. I just fixed it - should be out shortly in a new release of the cluster in a box image. Once out i’ll ping here. Thanks for the report and your patience.

@ohad please re-pull and use the image memsql/cluster-in-a-box:latest. You will be able to mount /var/lib/memsql as a volume without hitting the same problem you hit before. Of course, make sure that the user in the container has permissions to write to the volume.

Thanks again for the issue report!

Thanks for handling this @carl . However i still can’t create a database.

The mount has worked well this time as you can see - having 67G available disk:
Filesystem Size Used Avail Use% Mounted on
osxfs 234G 152G 67G 70% /var/lib/memsq

but when running “create database unittest;” I get the following errors:

From sequel pro:

CREATE PARTITION DATABASE for operation CREATE DATABASE failed with error 1883:Leaf Error (127.0.0.1:3307): Available disk space is below the value of ‘minimal_disk_space’ global variable (100 MB). This query cannot be executed.

From docker logs:
ERROR: Thread 115045: CreateFileAndAllocateSpace: Failed to allocate space for file logs/unittest_4_staging: 28 (No space left on device).

Any further advice?

Thanks.

Just to confirm you mounted to /var/lib/memsql rather than /var/lib/memsq (notice trailing l).

ughh, ignore my previous reply… I did mount the wrong one - memsq without the trailing l.

So now trying to mount the correct one with the following docker-compose:
version: ‘2’

services:
memsql:
image: memsql/cluster-in-a-box:latest
ports:
- 3306:3306
- 8080:8080
environment:
LICENSE_KEY: ${MEMSQL_LICENSE_KEY}
START_AFTER_INIT: ‘Y’
volumes:
- /tmp:/var/lib/memsql:rw

and i get this error message:

Starting tests_memsql_1 … done
Attaching to tests_memsql_1
memsql_1 | memsqlctl cannot find lock file /var/lib/memsql/nodes.hcl.lock. To fix this, please run
memsql_1 | · touch /var/lib/memsql/nodes.hcl.lock
memsql_1 | · chmod 640 /var/lib/memsql/nodes.hcl.lock
memsql_1 | · chown memsql:memsql /var/lib/memsql/nodes.hcl.lock (memsql:memsql are the owner and group of the node metadata file at /var/lib/memsql/nodes.hcl)
memsql_1 | Traceback (most recent call last):
memsql_1 | File “/startup”, line 95, in
memsql_1 | initialize_cluster = len(ctl(“list-nodes”)[“nodes”]) == 0
memsql_1 | File “/startup”, line 18, in ctl
memsql_1 | subprocess.check_output([“memsqlctl”, “-yj”] + list(args)))
memsql_1 | File “/usr/lib64/python2.7/subprocess.py”, line 575, in check_output
memsql_1 | raise CalledProcessError(retcode, cmd, output=output)
memsql_1 | subprocess.CalledProcessError: Command ‘[‘memsqlctl’, ‘-yj’, ‘list-nodes’]’ returned non-zero exit status 1
tests_memsql_1 exited with code 1

That looks like the same error as before - the issue may be that I uploaded the fix over the same tag in the docker hub, so you probably need to force re-pull the docker image to get the change.

I tried tried the latest and to be sure i also tried it’s equivalent one centos-7.1.11-6c108deb15-2.0.4-1.8.1 in both i get the following error:

2020-11-06 03:45:09.186018 Initializing MemSQL Cluster in a Box
2020-11-06 03:45:09.186161 Creating…
2020-11-06 03:45:10.259266 Done.
2020-11-06 03:45:10.259347 Configuring…
2020-11-06 03:45:11.331046 Done.
2020-11-06 03:45:11.331368 Bootstrapping…
Latest errors from MemSQL tracelog:

: Failed to connect to MemSQL: process exited: exit status 255

Initialization failed, please fix any issues and re-initialize the container.
Traceback (most recent call last):
File “/startup”, line 101, in

You may need to remove this container before continuing:
docker rm (CONTAINER_NAME)

init_cluster()
File “/startup”, line 45, in init_cluster
ctl(“start-node”, “–all”)
File “/startup”, line 18, in ctl
subprocess.check_output([“memsqlctl”, “-yj”] + list(args)))
File “/usr/lib64/python2.7/subprocess.py”, line 575, in check_output
raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command ‘[‘memsqlctl’, ‘-yj’, ‘start-node’, ‘–all’]’ returned non-zero exit status 1

I am not sure what the issue is, however I noticed you are using /tmp for each run - this directory may contain artifacts from previous runs which may be partially corrupted/deleted. Please use an empty directory dedicated to this container, and in the event of an error please clear the directory before restart.

I have successfully run the new CIAB image using /tmp as well as an empty directory without issue.

The source code for the image and associated startup scripts can be found here in the event you want to debug: GitHub - memsql/deployment-docker: This repository contains our official deployment Docker images for various products.

Hello Carl,

I’ve installed Cluster-in-a-box using Tarball and I getting the same error, however there is enough space on volume.

Hi joseph! This issue has been escalated internally and we’re working on getting the support you need. We appreciate your patience & hope to have an expert in contact with you soon.

I also ran into this issue and resolved it. Using Docker Desktop on Mac, there is a setting under Resources for Storage. The default is too small.

I also made other changes that may have contributed to the solution:

  • Upgraded to the latest version of Docker Desktop;

  • Switched from cluster-in-box to the singlestore-dev-image;

  • Added an additional parameter:

    -e SINGLESTORE_SET_GLOBAL_DEFAULT_PARTITIONS_PER_LEAF=1

I suspect only the first item (changing the Storage resource limit) was necessary.

1 Like