How to setup a usable rqlite cluster with 3 node microk8 multi-architecture cluster
I’ve setup a 3 node microk8 cluster, two arm64 (raspberry’s) and a amd64. As expected found a lot of issues due too a lot of images out there not being built for a multi-architecture cluster. Anyway I guess thats the topic for another future article :)
So I’ve decided to use rqlite as my distributed db in m y home cluster to store my custom made IoT application. The reason why I choose rqlite is because I basically wanted something as light-weighted as sqlite and to be distributed ( mainly due to availability because my raspberry pi’s sometimes become unreachable after some days running). So rqlite seemed as good as any :)
The issue I found with rqlite was that it wasn’t prepared to be run inside a kubernetes cluster. So things as pod scheduling is actually an issue. Luckily with some help from Philip O’Toole (thanks mate :) )was able to have the right mechanism into overcoming this (you can follow the discussion here). The second issue I had was when I wanted to use microk8 storage add-on. To do this I needed some way of synchronising my statefulset files between the different nodes so if pod1 would change from node1 to node2 all is data would still be available on the new node.
So I started by creating a docker configuration file for rqlite image and stateful set configuration. Afterwards built a storage sync application to run as a deamon set that would basically rsync statefull sets (marked with a syncronize-nodes label) volumes with all nodes. Finally needed to build a recover rqlite application that was able to update peers.json file in order to allow the consensus cluster to recover when the majority of rqlite pods go down or get rescheduled.
