I’m deploying SingleStore on GKE using the Kubernetes operator, but leaf nodes are stuck in a bootstrap loop and won’t join the cluster. The master aggregator starts successfully, but leaves fail to join and keep showing “This node is not part of the cluster” errors.
all details are below. I’ve spent two days on this, any pointer would be appreciated. thanks.
- SingleStore Operator: 4.14.0-1bcf743a (latest)
- SingleStore Node: alma-8.9.34-3946084f80
related info ;
LoadLicenseLegacy: Cannot open license key file at ‘/opt/singlestoredb-server-8.9.34-3946084f80/lib/license’
ERROR 2277 (HY000): This node is not part of the cluster
Aborting due to query failure: ‘SELECT HOST FROM INFORMATION_SCHEMA.AGGREGATORS WHERE ROLE = ‘Leader’’
sdb-rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: sdb-operator
namespace: singlestore
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: sdb-operator
namespace: singlestore
rules:
- apiGroups:
- ""
resources:
- pods
- services
- endpoints
- persistentvolumeclaims
- events
- configmaps
- secrets
verbs:
- '*'
- apiGroups:
- policy
resources:
- poddisruptionbudgets
verbs:
- '*'
- apiGroups:
- batch
resources:
- cronjobs
verbs:
- '*'
- apiGroups:
- ""
resources:
- namespaces
verbs:
- get
- apiGroups:
- apps
- extensions
resources:
- deployments
- daemonsets
- replicasets
- statefulsets
- statefulsets/status
verbs:
- '*'
- apiGroups:
- memsql.com
resources:
- '*'
verbs:
- '*'
- apiGroups:
- networking.k8s.io
resources:
- networkpolicies
verbs:
- '*'
- apiGroups:
- ""
resources:
- serviceaccounts
verbs:
- get
- watch
- list
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: sdb-operator
namespace: singlestore
subjects:
- kind: ServiceAccount
name: sdb-operator
roleRef:
kind: Role
name: sdb-operator
apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: sdb-operator
rules:
- apiGroups:
- storage.k8s.io
resources:
- storageclasses
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- persistentvolumes
- nodes
verbs:
- get
- list
- watch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: sdb-operator
subjects:
- kind: ServiceAccount
name: sdb-operator
namespace: singlestore
roleRef:
kind: ClusterRole
name: sdb-operator
apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: backup
namespace: singlestore
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: backup
namespace: singlestore
rules:
- apiGroups: ["batch"]
resources: ["jobs"]
verbs: ["get", "list"]
- apiGroups:
- ""
resources:
- pods
verbs:
- get
- list
- apiGroups:
- memsql.com
resources:
- memsqlclusters
verbs:
- get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: backup
namespace: singlestore
subjects:
- kind: ServiceAccount
name: backup
roleRef:
kind: Role
name: backup
apiGroup: rbac.authorization.k8s.io
sdb-operator.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: sdb-operator
namespace: singlestore
labels:
app.kubernetes.io/component: operator
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: sdb-operator
template:
metadata:
labels:
app.kubernetes.io/name: sdb-operator
spec:
serviceAccountName: sdb-operator
containers:
- name: sdb-operator
image: singlestore/operator:4.73.0-52dc82f0
imagePullPolicy: Always
args: [
# Cause the operator to merge rather than replace annotations on services
"--merge-service-annotations",
# Allow the process inside the container to have read/write access to the `/var/lib/memsql` volume.
"--fs-group-id", "5555",
"--cluster-id", "sdb-cluster" ]
env:
- name: WATCH_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: OPERATOR_NAME
value: "sdb-operator"
sdb-cluster-crd.yaml
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: memsqlclusters.memsql.com
spec:
group: memsql.com
names:
kind: MemsqlCluster
listKind: MemsqlClusterList
plural: memsqlclusters
singular: memsqlcluster
shortNames:
- singlestore
- singlestoredb
- memsql
scope: Namespaced
versions:
- name: v1alpha1
served: true
storage: true
schema:
openAPIV3Schema:
description: Schema for the SingleStore Cluster
properties:
apiVersion:
description: 'APIVersion defines the versioned schema of this representation
of an object. Servers should convert recognized schemas to the latest
internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
type: string
kind:
description: 'Kind is a string value representing the REST resource this
object represents. Servers may infer this from the endpoint the client
submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
type: string
metadata:
type: object
spec:
description: Spec defines the desired state of Cluster
type: object
x-kubernetes-preserve-unknown-fields: true
status:
description: Status defines the observed state of Cluster
type: object
x-kubernetes-preserve-unknown-fields: true
type: object
subresources:
status: {}
additionalPrinterColumns:
- name: Aggregators
type: integer
description: Number of Aggregators
jsonPath: .status.expectedAggregators
- name: Leaves
type: integer
description: Number of Leaf Nodes (per availability group)
jsonPath: .status.expectedLeaves
- name: Redundancy Level
type: integer
description: Redundancy level of the Cluster
jsonPath: .spec.redundancyLevel
- name: Age
type: date
jsonPath: .metadata.creationTimestamp
sdb-cluster.yaml
apiVersion: memsql.com/v1alpha1
kind: MemsqlCluster
metadata:
name: sdb-cluster
namespace: singlestore
spec:
license: "xxxxx"
adminHashedPassword: "*xxx"
nodeImage:
repository: singlestore/node
tag: "alma-8.9.34-3946084f80"
redundancyLevel: 1
serviceSpec:
objectMetaOverrides:
labels:
custom: label
annotations:
custom: annotations
aggregatorSpec:
count: 1
height: 0.5
storageGB: 256
storageClass: standard
objectMetaOverrides:
annotations:
optional: annotation
labels:
optional: label
leafSpec:
count: 1
height: 0.5
storageGB: 1024
storageClass: standard
objectMetaOverrides:
annotations:
optional: annotation
labels:
optional: label