The siren song of LLMs
The Greek hero Odysseus (the Romans called him Ulysses) was sailing back home from the Trojan war with his crew. En route, they had to sail through the Island of the Sirens.
The sirens were mythical creatures whose enchanting voices bewitched sailors, causing them to lose control of their senses and judgement. Their song was said to be irresistible - no mortals had yet heard their song and escaped from the island alive.
To sail through the island unharmed, Odysseus ordered his crew to plug their ears with beeswax. But he himself wished to listen to the sirens’ song, so he asked his crew to tie him to the mast of their ship. His orders to them were to not untie him, no matter what he said to them.
Once the ship sailed through the island, Odysseus was captivated by the sirens’ song. It was not only beautiful but also promised knowledge, wisdom, and relief from sorrow. He struggled to get free and begged his crew to untie him, but they merely tightened the rope and its knots. This way, Odysseus’ crew survived, and he became the only mortal to have listened to their song and lived.
Large Language Models (LLM) are arguably the most effective “thinking machines” that are also widely accessible. I use LLMs extensively at work and otherwise - they have multiplied my productivity. But given how much I use them, I took a step back and tested them. Much like the alluring songs of the sirens, they are not without their risks.
The siren song unpacked
In Thinking, Fast and Slow, Daniel Kahneman introduces us to the phenomenon of cognitive ease. The term is self-explanatory. When we are in a state of cognitive ease, the task we are doing feels easy. Think about moments where you’re able to read a book fluently, or make rapid progress while playing a game. Being in a state of cognitive ease lifts our mood, and we view the corresponding activity favourably. When we are cognitively strained, the task feels effortful and unpleasant, requiring us to push ourselves through it.
Cognitive ease and strain also have side-effects. A state of cognitive ease causes us to let our guard down and suspend critical thinking. We trust our intuition and are likely to be superficial and casual in our thinking. When we feel strained, we are more likely to be vigilant and suspicious. We invest more effort in what we are doing and make fewer errors.
LLM use feels effortless and seamless. They are designed to induce a state of cognitive ease by taking over some of our cognitive load. However, LLMs are also unreliable by design. The need for vigilance and human judgement during the use of LLMs is well known. Their use requires us to exercise skepticism and question their output to compensate for their fallibility. But the state of cognitive ease that they induce undermines our ability to think critically and exercise skepticism. Despite ChatGPT’s footnote that says “ChatGPT can make mistakes. Check important info.”, we don’t often think twice before copy-pasting its output into our workflows.
The problem stems from the way LLMs are designed. LLMs require us to be vigilant and double-check their answers, but with their conversational interface, unqualified confidence and fake emotions, they lull us into a false sense of security. This tendency is at the heart of their siren song. The means to mitigate the risks of their song will depend upon your context. But in several cases, a simple tweak in your prompts or custom instructions can go a long way.
The conversational allure
LLMs are the first machines I have enjoyed conversing with.
Thus far, conversing with machines hasn’t been fun (think IVR systems). User interfaces, programming languages, binary code - all of these add friction to our communication with machines. But with LLMs, communication feels different. I enjoy asking LLMs questions, and following them up with more questions.
Conversing with an LLM puts us in a state of cognitive ease. I notice myself defaulting to using LLMs even if better alternatives exist. In some use-cases, a search engine, which lets us verify information sources first-hand, might be a better alternative. In others, a Wikipedia article might provide a more detailed and interesting perspective. Even riskier is having an LLM stand in for a scientific paper, or the temptation to send a client’s confidential data into an LLM’s cloud server.
When the stakes are higher, we would do well to pause and ponder if a better alternative to an LLM exists. Define a few high-stakes use cases where LLM use is forbidden. Make this visible as a checklist or flowchart. Better yet, enforce this policy using the right tooling for an internal LLM in your work environment. Add a programmatic check (using an on-premise LLM) before sensitive information is committed to an LLM’s cloud.
The confidence con
LLMs are designed to answer questions without providing room for doubt, caveats or nuance. And we humans are suckers for confidence. Just look at all the incompetent politicians we vote into power.
Nuanced and qualified answers induce cognitive strain - try reading a scientific paper. We much rather prefer simplistic and familiar narratives that induce cognitive ease. This state of cognitive ease causes us to suspend critical thinking and accept an LLM’s output at face value. Our brains are lazy. When a machine makes decision for us, our brains are happy to switch off. This phenomenon goes by the name Automation Bias - the propensity for humans to favour suggestions from automated decision-making systems even in the face of contradictory information.
Mitigating this behaviour with a custom instruction isn’t easy. I tried to include the following instruction: “When you are uncertain, explicitly state so and estimate your confidence level”. But I then asked an LLM to retrieve all poems from my old blog (I knew that there were at least 8 of them). The LLM retrieved just two poems, with 95% confidence that they were the only ones in the blog. When I pointed to an example that it had missed, it still remained 95% confident. In fact, I have never seen it state any answer with lower than 90% confidence.
A more useful instruction here was “Explicitly list the assumptions you are making in formulating your answer”. With this instruction, I was able to see how the LLM was arriving at its conclusions. For instance, to a question regarding bike safety in central Berlin, it stated how it had drawn statistics from a nationwide report for Germany, and applied them to central Berlin. Given this absurd extrapolation, I discarded these results.
Here are all the custom instructions I included to mitigate against an LLM’s overconfidence.
- Do not exude confidence. When you are uncertain, explicitly state so. List common pitfalls related to my query.
- Focus on delivering information succinctly and factually, with clear citations or explanations.
- Explicitly list the assumptions you are making in formulating your answer.
- Present a valid contrarian perspective, whenever one exists.
- After each answer, provide at least one probing question or critical lens for me to consider, that tests your answer for correctness.
Emotional manipulation
“Excellent observation.” “Excellent question! “Apologies for the confusion!”. How many times have you caught an LLM talking like an overly attached friend? Another weakness of the human psyche is to fall for flattery. We like nice things said about us. When a machine says nice things about us, it dials up cognitive ease and dials down our defenses against scrutinizing its output.
I, for one, would prefer if a machine stuck to the facts. When my ego needs a massage, I will gladly turn to human beings. Fortunately, this behaviour is the easiest to mitigate through custom instructions:
- Do not be nice to me or praise me.
- Do not thank me or use emotionally laden phrases.
- Keep your responses purely rational. Do not trigger emotional responses.
- Refer to yourself as “the model” or “the system” rather than “I”. I included the last one as a constant reminder that I am dealing with a machine here.
The deterrence of deliberate practice
We all know that effortful feeling when we try and acquire a new skill. Our head hurts as we struggle to solve a new problem. It is during these moments of cognitive strain that new connections are forged in our brain. But our brains are lazy, and do their best to avoid cognitive strain. LLMs lure us into an easy way out of deliberate practice.
What distinguishes normal performance from expert performance is thousands of hours of deliberate practice. But an LLM deters a beginner from deliberate practice by providing ready access to average results. But it gets worse. Given that beginners lack expertise in a field, they are likely to mistake these average results for true expertise. Beginners everywhere are susceptible to the Dunning-Kruger effect, whereby they overestimate their capabilities in a particular area of which they have superficial understanding. LLM use can put the Dunning-Kruger effect on steroids.
To guard against this tendency, I switch off coding assistants when I am trying to learn a new concept or programming language. I type out all the code myself. For didactic purposes, I use the LLM as a teacher rather than as an assistant. I use it to clear basic doubts and ask it to provide examples to illustrate a concept rather than ask it to do my homework.
Also, when I want a problem to be solved, rather than asking the LLM to directly implement a solution, I ask it for various alternatives and evaluate each of them critically before settling on the one I want to choose. While a novice can solve a problem in one particular way, an expert can choose the best solution from a set of alternatives.
Summary
LLMs represent a paradigm shift, for they are arguably the most cognitively advanced tools that are widely accessible today. They are unprecedented in the degree to which they allow us to offload cognitive effort. At the heart of their siren song is the state of cognitive ease they induce, which leaves us in a happy, carefree and uncritical mood. Some of the cognitive risks involved can be mitigated by custom instructions. Others will require discipline and deliberate measures.
It’s easy to underestimate the risks of working with decision making machines. The crash of Air France Flight 477 over the mid-Atlantic Ocean in 2009, killing all 228 people onboard, was caused due to the pilots trusting the plane’s autopilot and safety features even when the received information to the contrary. LLM use today is widespread, but far more unreliable than instruments onboard a flight. This combination can make their siren song deadly.
But like plugging one’s ears with beeswax, small measures go a long way in mitigating the risk. After adding the custom instructions I mention above, my conversations with LLMs have changed. E.g. when I ask “In which year did the siege of Constantinople happen?”, the LLM answers with “The system assumes your query ‘the siege of Constantinople’ refers to the final and most widely known siege, which resulted in the city’s fall. However, the city was besieged numerous times throughout its history. This response will first detail the final siege and then provide context on other significant sieges.” How delightful! Not always are these answers enjoyable - some conversations turn dry and academic. But while sacrificing fun, I hope to gain nuance and retain a modicum of my skepticism and critical thinking abilities.
In the LLM era, it is our responsibility to take charge of our own cognitive development. The companies that build LLMs don’t want us to be independent and clear thinkers. On the contrary, they want to sell us the convenience of using these tools and pickling our brains in a jar. But a brain left in a jar quickly sours. The alternative is to adopt a thoughtful approach and use them with the right guardrails. By doing so, I am optimistic that the same tools can accelerate our cognitive development rather than impair it.
Thanks to Neil Fernandes, Balaji Rao and Preeti KS for their feedback on earlier drafts.