Large Language Models hallucinate - they can write in a plausible fashion, but their mastery of facts is poor. My first time out with ChatGPT I was researching an inter-war cruiser, thatās a 1920 - 1940 mid-sized surface combatant with 6ā to 8ā guns. It came back spouting nonsense about the Iowa class, a battleship with 16ā guns. Since I have a lot of naval history knowledge, I immediately LOLād. A kid writing a report for school would likely miss the obvious errors.
I keep running into this with programming. Claude Code can do basic stuff, but I fed it an ugly problem in Coding With Claude and it wandered all over the place. I made some progress by hand, which I mentioned in Maltego File Grooming. AI contributed to that, but not as much as my nearly forty years of Unix administration practice did.
The solution to this is the same as itās always been for everything Iāve done since 2009 - quality curation, build systems that facilitate the capture of cognitive surplus in a way that fits existing patterns of use, and assume that the user will do little of what you envisioned to be normal use cases.
So letās delve into how this is going to work ā¦
Attention Conservation Notice:
GraphCraft wonkery herein, lots of my mumbling about design factors for sense making systems rooted in social networks. Iāve been told I need to describe my thinking in a single linear document, this is me taking a swing at that task.
Foundations:
There are a lot of LLMs out there, I just paid for coding focused Claude Pro, then noticed that Jetbrains, making of the PyCharm IDE I favor, now has a coding only LLM called Mellum. LLMs predict what comes next based on what has come before, which we interpret as displays of knowledge. I hope Mellum has some actual knowledge, like maybe itāll stop trying to import Python libraries that donāt actually exist.
Using task specific training data to make a small, focused LLM is a good thing, but what happens if your tasks are more general in nature?
You can employ Retrieval Augmented Generation, where you combine an LLM with a set of curated documents. This adds weight to which things go together, but thatās contingent on how well written the linear human language documents being used are.
You can employ GraphRAG, which uses not just curated information, but also a network that shows how the curated things connect to each other. The direction Iām going here is dictated by prior experience - the ArangoDB offer of an LLM+Knowledge Graph is compelling.
Feeding Behavior:
I have my own curated things I can put into the ArangoDB system. The data from Disinfodrome still exists, itās just the servers that were liquidated.
I am a big fan of Inoreader, the app/web RSS reader. This is a lone curatorās tool, but itās got some team functions Iāve never properly explored. I recommend it often, most recently in Ignore Memes & Viral ANYTHING yesterday, which contains links to many prior posts about it.
Substack: Your World was my first public mention of the Substack API. If youāre subscribed to accounts that do substantive work, you can ingest that, and then youāve got an AI focused on your specific interests.
It will require quite a bit more work, but the thing that really excites me is employing a social network as the knowledge graph. Getting people to curate data is tricky. There are people like me, who just do it, but itās usually for their own purposes. Building a team to do it for an ongoing service, like we did with Progressive Congress News, is a slog. But getting people to talk about their interests and post relevant articles on a social network? Itās hard to get them to stop doing that ā¦
The xAI offering is supposed to do this, but it sucks, because the human to bot ratio is terrible and itās a right wing cesspool, implicitly disconnected from objective reality. I have a recording of Twitter political activity from 2019 - 2021, and it would be good for understanding flows of disinformation, but itās not generally applicable to sense making.
But if the users (you) control the social network (Pixelfed) that is a multifaceted source of quality information, accounts that are participating become part of the feed and we get things like:
Social network validation of new accounts; do the trusted sources follow them?
Text and images, and even video clips will be made available.
Links to content deemed relevant by the account owner.
Responses contain text, image, video, sentiment, social network context.
So if youāve got a social network, and you gamify it in a transparent fashion, such that the users understand the methods and motivations, you can train a combined Knowledge Graph + LLM. Then you make that AI available to your players and it becomes the collective memory of their social network ā¦
This would have worked for Progressive Congress News, it is what weāre doing with Shall We Play A Game?, and it will bring value if we succeed in Reviving Disinfodrome.
Iāve reached out to Substack in an effort to get a sense of what theyāre willing to tolerate from a third party service. They donāt answer, so theyāve either got their own internal AI project already, or theyāre too busy building and maintaining this system to put much thought into how the API might be used. I donāt think they can stop individuals from using a system like this, but Iād prefer to monetize it by directly using the platform subscription system ā¦
Conclusion:
Three times in my career Iāve set up a help desk as part of my work, leading to three year engagements managing each of them. Three times in my career Iāve set up data mining services, also leading to multiple year engagements. One of these was a double header - help desk supporting data mining. Working on this sort of thing puts me right into my career happy place after a long period of ā¦ unpleasantness.
That being said, itās time for me to get back into PyCharm and get cracking on that LLM + Knowledge Graph stuff ā¦