Cluster Capers
One layer done, another ahead of me ...
I was pretty frustrated when I wrote Old Man Yells At Cloud. Pausing to write this, I’ve been at it for fourteen hours and I have both a working GPU and a Proxmox cluster to show for the effort.
I am exhausted …
And tomorrow I get to start on the stuff I should have been doing October 27th … Kubernetes clustering.
Attention Conservation Notice:
Progress, just like crawling up Everest, using two kitchen forks for hiking poles. I think I’m just venting here, but I’m too tired to be certain.
Success:
And there you have it. One workstation with GPU, a one liter PC serving in a testing role, and a Raspberry Pi5 is the graphical head for this thing, as well as the “witness”. It’s not a Proxmox node, but it’s voting in the cluster elections so there are an odd number of nodes. This will never make a difference since they all use the same five port switch for connectivity, but it’s a skill I’ll need when we have two racks at opposite ends of a datacenter.
Kubernetes:
The whole idea with this thing was to get hip with the following:
MindsDB & Letta in this environment.
Chroma vector database.
Postgres and their vector extensions.
Local Ollama instance for OpenAI API services.
Kubernetes.
Some sort of high availability doctrine.
At this point I think I know that MindsDB and Letta are both singular - they’re not meant to run in a managed HA config. That doesn’t mean no HA, that means HA happens at load balancers and databases, with a layer in between where those two services get coddled by some stuff I cook up to keep them running.
Having force fed myself a bunch of Proxmox-fu yesterday, today I get to do the same with Kubernetes and a focus on Postgres.
Conclusion:
I lost a lot of the last two weeks to a fraudulently returned (and broken) piece of equipment that landed on my doorstep.
But there’s another issue that’s revealed itself, and it has to do with my work habits.
I am on the autism spectrum. Part of that is I will fiddle endlessly with things, like this Proxmox clustering, at the expense of something in the next layer that’s much higher on my priority list. I guess I’m trying to … climb up to it? And if there is something that will take a while to run, that is something that will keep me in place while it completes. Not productive.
Not only do I have this “vertical” approach when building stuff, if there are two lanes, I’ll get stuck in one of ‘em. This isn’t the “circumscribed interests” that are so common among autists. I do have those, but that’s not what is happening here, it’s just … once I get into a groove I hate to be interrupted. See prior accounting for fourteen hours of grinding on Proxmox for an example of this.
Startup, Inc. now has my absolute favorite - we set up Jira last week. Somewhere, out there, Libby Shaw is laughing herself silly at this line. This really helps - it’s ONE place I can turn my focus, and the diversity of effort comes from plowing through the various Kanban boards.
The hardware suck has definitely been sucky but a Very Good Thing happened last week. There are some things that need doing, and someone agreed to tend to them, with only the potential of some pay if we get our next round of funding.
And in amongst all this, I’m supposed to be getting Claude Camp done by month end …
Good read 👏