SingleStore operator not creating cluster

I’m following the docs here to spin up a cluster to evaluate on AWS EKS: Create the Object Definition Files

And was able to create the operator with kubectl create -f sdb-operator.yaml
But when I run kubectl create -f sdb-cluster.yaml, the operator never creates the cluster pods. Checking the operator logs, it doesn’t look like it’s doing anything. This is all that’s in the log even after applying the cluster file:

2022/09/20 14:55:56 deleg.go:121	{cmd}	Go Version: go1.18.2	
2022/09/20 14:55:56 deleg.go:121	{cmd}	Go OS/Arch: linux/amd64	
2022/09/20 14:55:56 deleg.go:121	{cmd}	Operator Version: 3.0.33	
2022/09/20 14:55:56 deleg.go:121	{cmd}	Commit Hash: db8f5aff	
2022/09/20 14:55:56 deleg.go:121	{cmd}	Build Time: 2022-09-08T14:43:05Z	
2022/09/20 14:55:56 deleg.go:121	{cmd}	Options:	
2022/09/20 14:55:56 deleg.go:121	{cmd}	--cores-per-unit: 8.000000	
2022/09/20 14:55:56 deleg.go:121	{cmd}	--memory-per-unit: 32.000000	
2022/09/20 14:55:56 deleg.go:121	{cmd}	--overpack-factor: 0.000000	
2022/09/20 14:55:56 deleg.go:121	{cmd}	--extra-cidrs: []	
2022/09/20 14:55:56 deleg.go:121	{cmd}	--external-dns-domain-name: {false }	
2022/09/20 14:55:56 deleg.go:121	{cmd}	--external-dns-ttl: {false 0}	
2022/09/20 14:55:56 deleg.go:121	{cmd}	--ssl-secret-name: 	
2022/09/20 14:55:56 deleg.go:121	{cmd}	--merge-service-annotations: true	
2022/09/20 14:55:56 deleg.go:121	{cmd}	--backup-default-deadline-seconds: 3600	
2022/09/20 14:55:56 deleg.go:121	{cmd}	--backup-incremental-default-deadline-seconds: 3600	
2022/09/20 14:55:56 deleg.go:121	{cmd}	--cluster-id: singlestore-reporting-uat	
2022/09/20 14:55:56 deleg.go:121	{cmd}	--fs-group-id: 5555	
I0920 14:55:57.699056       1 request.go:665] Waited for 1.041794107s due to client-side throttling, not priority and fairness, request: GET:https://172.20.0.1:443/apis/batch/v1beta1?timeout=6m0s
2022/09/20 14:55:58 deleg.go:121	{controller-runtime.metrics}	Metrics server is starting to listen	addr: "0.0.0.0:9090"  
2022/09/20 14:55:58 deleg.go:121	{cmd}	starting manager	
2022/09/20 14:55:58 logr.go:249	Starting server	addr: "[::]:9090"  path: "/metrics"  kind: "metrics"  
I0920 14:55:58.464460       1 leaderelection.go:248] attempting to acquire leader lease default/memsql-operator-lock-singlestore-reporting-uat...
I0920 14:55:58.478257       1 leaderelection.go:258] successfully acquired lease default/memsql-operator-lock-singlestore-reporting-uat
2022/09/20 14:55:58 logr.go:249	{events}	Normal	object: "{Kind:ConfigMap Namespace:default Name:memsql-operator-lock-singlestore-reporting-uat UID:bff9f3a3-61c8-4349-a924-9664fafbdc95 APIVersion:v1 ResourceVersion:247815745 FieldPath:}"  reason: "LeaderElection"  message: "sdb-operator-7cc785cb69-lr8q2_b5e7e991-597c-4712-9f18-96a6d0db31f3 became leader"  
2022/09/20 14:55:58 logr.go:249	{events}	Normal	object: "{Kind:Lease Namespace:default Name:memsql-operator-lock-singlestore-reporting-uat UID:e47d54f4-c608-4423-9944-f1a1e8724210 APIVersion:coordination.k8s.io/v1 ResourceVersion:247815746 FieldPath:}"  reason: "LeaderElection"  message: "sdb-operator-7cc785cb69-lr8q2_b5e7e991-597c-4712-9f18-96a6d0db31f3 became leader"  
2022/09/20 14:55:58 logr.go:249	{controller.memsqlcluster}	Starting EventSource	reconciler group: "memsql.com"  reconciler kind: "MemsqlCluster"  source: "kind source: *v1alpha1.MemsqlCluster"  
2022/09/20 14:55:58 logr.go:249	{controller.memsqlcluster}	Starting EventSource	reconciler group: "memsql.com"  reconciler kind: "MemsqlCluster"  source: "kind source: *v1alpha1.MemsqlCluster"  
2022/09/20 14:55:58 logr.go:249	{controller.memsqlcluster}	Starting EventSource	reconciler group: "memsql.com"  reconciler kind: "MemsqlCluster"  source: "kind source: *v1.StatefulSet"  
2022/09/20 14:55:58 logr.go:249	{controller.memsqlcluster}	Starting EventSource	reconciler group: "memsql.com"  reconciler kind: "MemsqlCluster"  source: "kind source: *v1.Service"  
2022/09/20 14:55:58 logr.go:249	{controller.memsqlcluster}	Starting EventSource	reconciler group: "memsql.com"  reconciler kind: "MemsqlCluster"  source: "kind source: *v1.Secret"  
2022/09/20 14:55:58 logr.go:249	{controller.memsqlcluster}	Starting Controller	reconciler group: "memsql.com"  reconciler kind: "MemsqlCluster"  
2022/09/20 14:55:58 logr.go:249	{controller.memsqlcluster}	Starting workers	reconciler kind: "MemsqlCluster"  reconciler group: "memsql.com"  worker count: "1"

This is my sdb-cluster.yaml:

apiVersion: memsql.com/v1alpha1
kind: MemsqlCluster
metadata:
  name: sdb-cluster
spec:
  license: {license}
  adminHashedPassword: "*{hashed_password}"
  nodeImage:
    repository: memsql/node
    tag: alma-7.8.14-8b65b22706

  redundancyLevel: 1

  serviceSpec:
    objectMetaOverrides:
      labels:
        custom: label
      annotations:
        custom: annotations

  aggregatorSpec:
    count: 2
    height: 0.5
    storageGB: 256
    storageClass: ssd

    objectMetaOverrides:
      annotations:
        optional: annotation
      labels:
        optional: label

  leafSpec:
    count: 2
    height: 0.5
    storageGB: 1024
    storageClass: ssd

    objectMetaOverrides:
      annotations:
        optional: annotation
      labels:
        optional: label

And the sdb-operator.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sdb-operator
  labels:
    app.kubernetes.io/component: operator
spec:
  replicas: 1
  selector:
    matchLabels:
      name: sdb-operator
  template:
    metadata:
      labels:
        name: sdb-operator
    spec:
      serviceAccountName: sdb-operator
      containers:
        - name: sdb-operator
          image: memsql/operator:3.0.32-almalinux-db8f5aff
          imagePullPolicy: Always
          args: [
            # Cause the operator to merge rather than replace annotations on services
            "--merge-service-annotations",
            # Allow the process inside the container to have read/write access to the `/var/lib/memsql` volume.
            "--fs-group-id", "5555",
            "--cluster-id", "singlestore-reporting-uat"
          ]
          env:
            - name: WATCH_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: OPERATOR_NAME
              value: "sdb-operator"

Welcome aboard kryo! :wave:

Sorry to hear this issue is happening. Are you running on the managed or self-hosted service? What is your version number?

Thanks Maria.

I’m trying to set up a self-hosted cluster for evaluation in our kubernetes cluster.
Using the latest docker images available from dockerhub

SingleStore: memsql/node:alma-7.8.14-8b65b22706
Operator: memsql/operator:3.0.32-almalinux-db8f5aff

1 Like

Thanks for the info kryo. Could you try 2 things for me?

  1. Can you try to preload images here on the local container repo?
    Docker Container Repository (Optional)

  2. Can you confirm that in your attempts you followed the order outlined here?
    Deploy a SingleStoreDB Cluster

Please keep us updated. :rocket: