Node startup fails with 7.0 using ZFS

Hi,
I’m trying to set up a MemSQL 7.0 cluster (single host). The installation of 6.8 worked without problems, but upgrading to 7.0 or installing this version from scratch fails.
Installation stops with “Failed to start Leaf node on 127.0.0.1 (1/1)”.

I analyzed the log files for this node:

13285804 2020-01-16 10:02:01.350   INFO: Thread 115121: CreateOrOpen: memsql: Creating root file roots/memsql_root
13286189 2020-01-16 10:02:01.351  ERROR: open('roots/memsql_staging', O_CREAT | O_EXCL | O_DIRECT | O_DSYNC | O_WRONLY, S_IRUSR | S_IWUSR) failed. Error Invalid argument (22)
13286213 2020-01-16 10:02:01.351  ERROR: Thread 115121: CreateFileAndAllocateSpace: Failed to create file `roots/memsql_staging`: 22 (Invalid argument).
13286474 2020-01-16 10:02:01.351  ERROR: Thread 115121: CreateOrOpen: Could not create root file `roots/memsql_root`.
13368967 2020-01-16 10:02:01.434  ERROR: Drop of database `memsql` failed to drop table `$$remote$$` after a failed create database
13470655 2020-01-16 10:02:01.535   FAIL: Failed to recover 'memsql' database (error 3)

So it seems to be a file handling related problem. We run an LXC 3.0.3 environment with Ubuntu 5.0.0-37.40~18.04.1-generic 5.0.21 and the ZFS file system.
ZFS maybe doesn’t support the way the root file is created (O_DIRECT?). Is there a way to run MemSQL 7.0 on ZFS?

1 Like

Yeah, 7.0 uses a new replication/durability design that uses O_DIRECT. I’ll inquire if there is some way to work around it, but I don’t know of any of the top of my head.