Life is pain, Highness. Anyone who says differently is selling something.
— Dread Pirate Roberts
I can often be found saying that “The way you frame a problem shapes the solutions you can see.” I truly can’t say if I read it or rediscovered it on my own, but it has become something I believe wholeheartedly, and it’s most certainly true.
The reason I’m telling you this is because it’s of particular relevance with LLMs. You can achieve a result in a myriad of ways: ask 100 people to use AI to do a particular task and you will find yourself with 100 different ways of doing the same thing (some better than others). This is why the internet is chock-full of tips and one weird tricks to do X, because, truly, they are.
Here’s the thing though: most approaches work for reasons entirely different to what people assume is happening. Context quality has a linear relation with output quality. When it comes to LLMs, this is as true as gravity, a law of the universe that will remain in effect as long as predicting the next token is the main goal of LLM pre-training.
Then, what is context?
Everything. Even the things that don’t look like context are context. Every single thing you give the model, directly or indirectly: the harness, the thing you said, and the thing you did not say. All of it is context. Every single thing that gets passed for a given completion is the entirety of the model’s world for that task, and you better believe it has a real, measurable effect on what probabilistic path it sees ahead.
This is why it’s so important to see everything as a context problem. Failing to realize it means not working the most powerful leverage we have over a model: purposefully crafting its interpretation of the world in which the task takes place.
But what about newer models? They are super smart and can figure it all out themselves. Just ask and you shall receive!
I’m glad you asked. Current (no matter when you read this) SOTA models are truly astonishing pieces of technology. They have been post-trained into countless desirable behaviors, including developing as much of their own context for a task as possible, trying to compensate for vague or low-intent user requests. No doubt. But even these trillion-parameter models cannot escape the laws of nature: they are fundamentally constrained by the context provided to them. Success and failure are more a measure of the context than of any given model.
I’m still not convinced. It all sounds just too convenient to be true.
I hear you. I too would not believe such a claim without evidence, so let me provide some. Let’s start with the most obvious one, and go deeper from there.
A prompt is context; that one is self-evident. Mention you are in the mood for Italian food before asking for a restaurant recommendation and you will get Italian places to eat. Say you are writing an email to your boss and you will receive a professional-shaped response; mention you are friends and it changes accordingly, because of the contextual implications.
Ask your favorite AI for a summarization of a state law and you will receive one, but would you bet your reputation on it? Pass the model a file containing the entire law and the task implicitly transforms into “parse this document and extract from it.” The document becomes the world of the task, and a far more trustworthy result comes out the other side.
A harness is context. The place from which the model is run shapes what the model understands itself capable of doing, because the harness provides instructions, tools, and response formats. A model with no internet access will not reach for the internet, even if a search is the best possible path to a good answer, because the internet is outside its world. And a coding harness is not the best place for a philosophical discussion: you can have one, sure, but the environment suggests a different kind of personality and response, merely by implication. The same way you (hopefully) don’t go to Wendy’s to have an existential discussion at the drive-thru.
Even the order of the context is context. LLMs are autoregressive, so the way we feed them context has a real, tangible effect. Give a model a large text to review and placing the task definition before the text produces a different result than placing it after. The same is true of every part of the context. Ignore this and you will have a hard time getting the most performance out of them.
And then there is the biggest one, the one that gets almost no acknowledgement: the thing not said.
The Scales of Context
Context has two halves, explicit and implicit, and the implicit is often far larger than we realize. Picture a classic scale (Lady Justice style). You can only ever put weight on the explicit plate, and the model can only ever put weight on the implicit one. Different tasks require different ratios, but the important part is this: the less you put explicitly, the more the model will put implicitly.
Imagine you ask: “Plan and create a Mario Kart clone that runs in the browser, don’t stop until you are done.” It will do it. It will take all night, but it will do it. The code will run. Come morning you will be playing “Morio Kert” (models are very sensitive to copyright, you see), and you will be astonished that it works. Flabbergasted, even. But stop for a moment and ask yourself: how did everything come to be? Why is the item box the way it is? The track? The environment? Where did each of those decisions come from? Was it the model?
It’s complicated.
Remember the scale. The less you put on the explicit plate, the more the model has to put on the implicit one. But where does the model get its stuff? From pre-training, where it ingested all of the internet (or as much as the companies could steal obtain), including countless code repositories where Mario Kart-shaped things live, including literal Mario Kart clones. And from post-training, where the unwieldy pre-trained model gets tamed: taught to produce completions that look like coherent code, taught the patterns for decomposing a big task into a series of small ones. So when you ask for a Mario Kart clone, the model is already primed to produce it. The fact that it did should be no surprise at this point.
Just don’t go asking for things that aren’t well represented in the training data, because you will get less impressive results. That is, unless you put a lot into the explicit plate of the scale, of course.
Once you see it, everything starts to fall into place. Every prompt tip, every one weird trick and workflow that ever worked, worked because it changed the context.
So What?
The reason I keep insisting that everything is context is that there is a perverse incentive to keep obscuring how these models work.
Certain individuals and companies benefit enormously whenever we buy the stories wholesale: AI going rogue, agents taking action “on their own,” the one course that will finally unlock next-level AI productivity, the magic Coding Agent plugin that will fix all of your issues. Not a single day goes by without someone actively benefiting from you not understanding the nature of the beast.
And nobody benefits more than the labs and the AI rockstars. All you need is loops that loop the loop, and you are the supervisor, the manager, whatever the title of the week is. Delegate everything to the model, then pile up agents at the end to “verify” the work was done well. Never mind that it will need a big refactor in six months; you just start the cycle anew. Code is cheap now, so who cares! The ones who sell you the solution that creates the problem will happily tell you the solution is to repeat the cycle.
Anyone who says differently is selling something.
Why does the magic sell? Because using AI intentionally takes effort, and effort doesn’t feel like magic. Watching a model chew on a problem, then swooping in to fix the thing it broke midway, feels like sci-fi. Meanwhile, thoughtful code is boring. Its execution is unsurprising. It requires planning and intention, which is precisely the thing that gets in the way of magic.
What Now?
Now that you know everything is context, including the things that don’t look like it, experiment with that mindset. Iterate with intention and watch what happens. Change one thing, deliberately, and watch again. Don’t just create context, craft it, craft the world the model sees. You have way more control than you realize. In fact, you have all of it.
Whatever issue you’re having with AI (the LLM kind), I guarantee you, is a context problem. It always is.
Despite what the labs want you to believe, models are just technology. An awesome technology, so awesome that at times it seems like it must be more, like it cannot be just ones and zeroes. And yet, that is all they are. That is not a bad thing. They are awesome because of it: a marvel, a wonder, a tool.
So don’t let them take control away from you, and don’t sit around waiting for the next model to be smarter than the current one. I’ll go out on a limb here: a lot of what surprises us about today’s models was achievable before. More effort, yes, but doable. A newer model is not a bad thing, but you don’t always need the latest shiny toy. Most of us haven’t tapped the true potential of the models we already have.
I feel like I’m starting to reach old-man-yells-at-cloud energy, so I better go before I make a fool of myself.