Trust is a pretty fundamental part of our existence as humans. I trust my wife, my parents, my friends, family and colleagues. I trust people in positions of authority, people who’ve spent an inordinate amount of time and effort learning something so I don’t have to.
There are so many things in our day to day life that rely on trust. Not just the small interpersonal stuff, but the electrician who wired your house with the implicit trust things won’t set on fire. Or the plumber who fit your gas powered boiler and the implicit trust you aren’t going to have a gas leak and your house will explode (bad things often end in fire don’t they).
This system of trust works because if said electrician or plumber didn’t do a good job, they wouldn’t still be in business. If they said your house wasn’t going to set on fire, and then it did, that trust is eroded. And you’d like to think they wouldn’t continue being a tradesperson for much longer. Or imagine a company started life promising they would never spy on you, and then did (hey Facebook), that trust gets eroded.
I think it’s part of the reason I’m skeptical of LLM’s being pushed into every part of our lives. This relationship starts to erode. If I ask an LLM a question, it hallucinates and gives me an innacurate response, I lose trust in it. It’s been proven time and time again that this as an in-built flaw in the way the current iteration of LLM’s work. They are statistical models predicting the next most likely word (token) based on the data they have stolen been trained on.
And my issue isn’t the fact it gets things wrong, humans get things wrong all the time. I’ve made so many mistakes in my career that have had real consequences (systems being offline, businesses losing money) it’s almost laughable. The issue, is what happens after the mistake was made.
Imagine you hired a junior/less experienced person into whatever role you’re currently in. They report to you, and you’re responsible for their output. If they made a mistake, the first time you’d probably (hopefully?) politely correct them so that they know better next time. Said junior would then probably (hopefully?) take that on board and imporove their performance for next time. One of the very first managers I ever had used to say if you ask me once you’ll get a nice answer, twice and I’ll be slightly less nice and the third time you’re clearly not learning or taking things on board.
Unless I’ve missed some big headline, this is not the case with the current iteration of LLM’s. It might hallucinate a response, and you move on with your day. Tomorrow, for the same question, maybe it will answer correctly. And then maybe it continues to answer correctly for a few weeks before the hallucination comes back again. It’s a fundamental flaw in the system, and if you had an employee who randomly and without warning went off the rails and did something crazy I would assume you wouldn’t want them to be your employee for much longer.
I’m not talking in the abstract here, I’ve spent the last several months building some internal tooling at Datadog to help people better plan high quality conference talks. Not something that generates talks for people, but something that can guide them through the things they need to think about to create a good talk. At the core of the service is an LLM, that has access to a set of tools, and a pretty strict system prompt that uses words like ALWAYS and NEVER. And even with that in place, periodically it still completely ignores the instructions and goes off and does something different.
For this service, it’s pretty inconsequential. But as AI starts to be forced pushed into every single part of our lives I can see more issues arrising. If an LLM is making medical decisions, is is always going to make the right ones? And yes, I hear you saying but a human gets it wrong too. And I’ll say it again, the difference is that a human learns and moves forward. A statistical model does not.
James