Natural Knowledge Graphs
Improving LLM accuracy with social networks.
Large Language Models hallucinate - they can write in a plausible fashion, but their mastery of facts is poor. My first time out with ChatGPT I was researching an inter-war cruiser, that’s a 1920 - 1940 mid-sized surface combatant with 6” to 8” guns. It came back spouting nonsense about the Iowa class, a battleship with 16” guns. Since I have a lot of naval history knowledge, I immediately LOL’d. A kid writing a report for school would likely miss the obvious errors.
I keep running into this with programming. Claude Code can do basic stuff, but I fed it an ugly problem in Coding With Claude and it wandered all over the place. I made some progress by hand, which I mentioned in Maltego File Grooming. AI contributed to that, but not as much as my nearly forty years of Unix administration practice did.
The solution to this is the same as it’s always been for everything I’ve done since 2009 - quality curation, build systems that facilitate the capture of cognitive surplus in a way that fits existing patterns of use, and assume that the user will do little of what you envisioned to be normal use cases.
So let’s delve into how this is going to work …
Attention Conservation Notice:
GraphCraft wonkery herein, lots of my mumbling about design factors for sense making systems rooted in social networks. I’ve been told I need to describe my thinking in a single linear document, this is me taking a swing at that task.
Foundations:
There are a lot of LLMs out there, I just paid for coding focused Claude Pro, then noticed that Jetbrains, making of the PyCharm IDE I favor, now has a coding only LLM called Mellum. LLMs predict what comes next based on what has come before, which we interpret as displays of knowledge. I hope Mellum has some actual knowledge, like maybe it’ll stop trying to import Python libraries that don’t actually exist.
Using task specific training data to make a small, focused LLM is a good thing, but what happens if your tasks are more general in nature?
You can employ Retrieval Augmented Generation, where you combine an LLM with a set of curated documents. This adds weight to which things go together, but that’s contingent on how well written the linear human language documents being used are.
You can employ GraphRAG, which uses not just curated information, but also a network that shows how the curated things connect to each other. The direction I’m going here is dictated by prior experience - the ArangoDB offer of an LLM+Knowledge Graph is compelling.
Feeding Behavior:
I have my own curated things I can put into the ArangoDB system. The data from Disinfodrome still exists, it’s just the servers that were liquidated.
I am a big fan of Inoreader, the app/web RSS reader. This is a lone curator’s tool, but it’s got some team functions I’ve never properly explored. I recommend it often, most recently in Ignore Memes & Viral ANYTHING yesterday, which contains links to many prior posts about it.
Substack: Your World was my first public mention of the Substack API. If you’re subscribed to accounts that do substantive work, you can ingest that, and then you’ve got an AI focused on your specific interests.
It will require quite a bit more work, but the thing that really excites me is employing a social network as the knowledge graph. Getting people to curate data is tricky. There are people like me, who just do it, but it’s usually for their own purposes. Building a team to do it for an ongoing service, like we did with Progressive Congress News, is a slog. But getting people to talk about their interests and post relevant articles on a social network? It’s hard to get them to stop doing that …
The xAI offering is supposed to do this, but it sucks, because the human to bot ratio is terrible and it’s a right wing cesspool, implicitly disconnected from objective reality. I have a recording of Twitter political activity from 2019 - 2021, and it would be good for understanding flows of disinformation, but it’s not generally applicable to sense making.
But if the users (you) control the social network (Pixelfed) that is a multifaceted source of quality information, accounts that are participating become part of the feed and we get things like:
Social network validation of new accounts; do the trusted sources follow them?
Text and images, and even video clips will be made available.
Links to content deemed relevant by the account owner.
Responses contain text, image, video, sentiment, social network context.
So if you’ve got a social network, and you gamify it in a transparent fashion, such that the users understand the methods and motivations, you can train a combined Knowledge Graph + LLM. Then you make that AI available to your players and it becomes the collective memory of their social network …
This would have worked for Progressive Congress News, it is what we’re doing with Shall We Play A Game?, and it will bring value if we succeed in Reviving Disinfodrome.
I’ve reached out to Substack in an effort to get a sense of what they’re willing to tolerate from a third party service. They don’t answer, so they’ve either got their own internal AI project already, or they’re too busy building and maintaining this system to put much thought into how the API might be used. I don’t think they can stop individuals from using a system like this, but I’d prefer to monetize it by directly using the platform subscription system …
Conclusion:
Three times in my career I’ve set up a help desk as part of my work, leading to three year engagements managing each of them. Three times in my career I’ve set up data mining services, also leading to multiple year engagements. One of these was a double header - help desk supporting data mining. Working on this sort of thing puts me right into my career happy place after a long period of … unpleasantness.
That being said, it’s time for me to get back into PyCharm and get cracking on that LLM + Knowledge Graph stuff …