Monday, November 9, 2026

We Gave Simulated Xenobots Inheritance. They Got Faster. Then They Plateaued

Biobot Tierra — Inheritance and Adaptation

We gave a population of simulated bots a way to pass traits to their offspring. The faster ones prospered, and their descendants took over.

Then, across tens of millions of simulation steps, this evolutionary drama turned fairly uneventful.

That plateau is the most interesting result so far from Biobot Tierra, an artificial-life experiment at Mind37. We set out to see how replicating biobot machines might accumulate useful changes across generations. What we found was a system very good at spending its initial inheritance, and with little evidence that it earned anything new.

A body that builds a body

The inspiration is the idea of xenobots from the incredible research of Michael Levin et al. out of Tufts University. Xenobots are tiny living constructs made from frog embryo cells, which swim using microscopic beating hairs called cilia. In 2021, researchers showed that swimming xenobots could sweep loose cells into piles that matured into new moving bodies, a process called kinematic self-replication. A body assembles another body from the material around it. That is an extraordinary thing to watch.

But a successful shape does not survive the handoff. The C-shaped, Pac-Man-like xenobots in those experiments replicated better than spheres, yet their offspring still rounded up into spheres. The parent's engineered geometry was lost between generations.

Why anyone wants living machines

Xenobots are interesting because they can do jobs that are hard for metal and plastic robots. They are built from ordinary cells, so they need no foreign materials, run on their own stored energy, and break down harmlessly when they die after a few days or weeks. Researchers have proposed using bots like these to:

collect pollutants, such as microplastics, from water
deliver drugs to specific places in the body
sense and record their surroundings
repair tissue, possibly using bots grown from a patient's own cells

The early lab demonstrations point in those directions. Xenobots have pushed particles into piles, and one version carried a protein that permanently changes color when exposed to a particular wavelength of light, a simple record of what it had been exposed to. In 2023, a related system called anthrobots, grown from human airway cells, helped a scratched layer of human neurons grow back across the gap in a dish.

These are early results in dishes, not products. They do share a property that is important for everything that follows: none of these bots pass their traits on.

That missing inheritance is consequential though. Darwinian evolution needs three things: variation, differences in how many offspring individuals leave, and heredity. Without heredity there is no natural selection. With Biobot Tierra we ask what would happen if something useful did carry over.

The dish

Our bots live in a small digital dish: a 128 × 128 lattice of loose cells. They move, push cells into piles, and after a fixed lifespan fall apart, returning their material to circulation. A pile that is large enough, compact, and left alone for a few simulated seconds hatches into a new bot.

Each bot carries five traits: mouth shape, size, speed, turning bias, and a color marker. Each trait is a whole number from 0 to 255, decoded into the body. Speed, for example, maps a gene g onto a velocity:

v = 40 + 60 · g / 255   (in 1/256 of a lattice site per step)

so the slowest bot covers about 3 sites per simulated second and the fastest about 8. The marker does nothing at all, which makes it useful. It lets us follow lineages, and because it has no effect, how far it drifts shows how much change chance alone produces.

Nobody stands above the dish handing out offspring to the winners. The movement and hatching rules are programmed, but reproduction follows from what bots actually do. A bot has to take part in the world for its traits to get another turn.

We built four versions of that world, differing only in how offspring inherit:

Control: offspring hatch as default spheres with random behavior, as in the lab.
Mold: offspring copy the shape of the bot that built most of their pile.
Imprint: offspring copy the behavior of the last bot to work on their pile.
Carry-over: offspring take the average traits of the cells their body is made from.

All four mechanisms are invented. None has been seen in xenobots. We are asking what their consequences would be in a model, not claiming to have found them in living tissue.

Whatever is inherited can also mutate. Each copied trait gets a small random nudge, then is clipped to the allowed range:

z_offspring = clip( z_parent + ε, 0, 255 ),   ε ≈ Normal(0, m²)

where m, the mutation size, was 4, 16 or 48. The nudge is built from four random bytes added together. By the central limit theorem their sum is already close to a bell curve, which keeps the model in exact integer arithmetic.

The first study

We ran 176 dishes for 200,000 steps each: 35.2 million simulation steps. Every dish started with eight founders and 40,000 loose cells. Every birth and death went into the record, and each dish was measured against its own founders. The model is exact and deterministic, so any run can be replayed birth for birth.

The basic measure of inheritance is heritability: fit a straight line through offspring traits plotted against parent traits, and take its slope.

b = Cov(z_offspring, z_parent) / Var(z_parent)

A slope of 1 is a perfect copy; 0 means nothing is passed on. Because our bots have a single parent, this slope estimates the heritability h² directly. The control is the sanity check: its slopes came out at 0.00–0.02, as they should.

Three lessons emerged.

Copying shape worked as copying, not as improvement. Mold passed shape down almost perfectly (a heritability above 0.8), but shape barely affected how many piles a bot built in this simulation. Faithful copies added only drift and harmful mutations, and mold dishes did worse than the control. That says as much about the model as about inheritance: our coarse lattice does not reproduce the advantage C-shaped bodies have in the lab. We searched 16 arena settings for one that would restore it. The best looked promising, then failed on fresh random seeds. It had been noise.

Blending removed variation, not information. Carry-over offspring closely tracked the average traits of their cells: the slope against the pile's average was 0.90–1.01 per trait at the smallest mutation size. But averages of mixed cells are all much alike. That is the familiar rule that averaging shrinks spread.

Among the founders, a trait's standard deviation was about 68. Among pile averages it was 7–15. Since what selection can do scales with variance, not standard deviation, that is a loss of roughly 20- to 90-fold (68²/15² ≈ 20, 68²/7² ≈ 94). The system kept transmitting information while erasing the differences that selection needs. Fleeming Jenkin raised this objection to Darwin in 1867.

Copying behavior was the promising case. Faster bots worked more piles, so their behavior was copied more often, and the population sped up. Most of that rise came in the first generation of offspring.

And then the line went flat.

Investigating the flat line

So what was the possible explanation of the plateau?

The first dishes had a problem: populations fell to about two bots in every version, including the control. We traced the material. A dead bot's cells formed a compact pile that hatched again as a larger bot, so more and more of the dish ended up in fewer and fewer bodies.

We changed the research version so a newborn took a fixed amount of material and left the rest loose. With 100,000 cells and longer lives, the dish held about 39 bots steadily.

The plateau survived.

We also went back to the first study's records to see what drove the early rise, using the Price equation. It splits the change in a trait's average from one generation to the next into two exact parts:

Δz̄ = Cov(w, z) / w̄  +  E(w · Δz) / w̄
       selection         transmission

Here z is a parent's trait, w is how many offspring it left, w̄ is the average of w, and Δz is how far its offspring's traits differ from its own. The first term is selection: does the trait go with having more offspring? The second is transmission: do offspring drift away from their parents, through mutation or imperfect copying?

For the founders' offspring under imprint (mutation size 16), the numbers were:

Δz̄ = +37.2 ± 15.5   =   selection +39.4 ± 14.6   +   transmission −2.2 ± 3.4

Almost all of the rise was selection: faster founders left more offspring. For every later group of births, the total change was −0.0 ± 0.5. It was essentially zero.

Who is the parent?

The next question sounds bureaucratic until it changes the result: who, exactly, is the parent?

The bot that builds a pile is often not the last one to touch it. In about two-thirds of births they were different bots, and under the original imprint rule it was the last toucher whose behavior was copied. Building the next generation and being copied into it were two separate accomplishments.

When offspring copied the builder instead, the response was stronger: newborn speed rose about 75 points above the founders (on a 0–255 scale), against about 45 under the last-toucher rule. Even our accounting of what caused the change depended on which bot we called the parent. Both terms of the Price equation are defined parent by parent. Read the same births through the builder instead of the last toucher, and the change shows up mostly as transmission rather than selection. The math is exact; the ancestry you feed it is a choice.

A real advantage, and a limited one

Was the change real? We put eight late-run bots into a fresh dish with eight bots resembling the founders. The late bots claimed about 60% of the offspring, and 58% in a rerun.

With no advantage, the share would be 0.5. The rerun's share of 0.585, with a standard error of 0.022, sits almost four standard errors above that:

t = (0.585 − 0.500) / 0.022 ≈ 3.9

We could also name the cause. When the founder-like bots were given the late bots' speed, the late bots' share fell to 0.46 ± 0.04, back to chance.

But the adapted bots had not made the dish more productive. A dish of adapted bots alone produced about as many offspring as a dish of founder-like bots alone. They had become better at claiming a share of reproduction without increasing the total.

The population had evolved a competitive advantage. The collective output had barely moved.

Turning off mutation

Then we removed mutation entirely.

The initial improvement still happened, about as large as with mutation, and in every dish one founder's speed took over. The population got to its eventual performance using only the variation it started with.

Selection was doing its job: useful starting variation spread. What we could not find was a continuing supply of new improvements built on top of it.

Classical quantitative genetics gives a simple way to see why. The breeder's equation says the response to selection in one generation is

R = h² · S

the heritability times the selection differential S, which is how much the parents who reproduce differ from the population average. S in turn is proportional to how much variation there is. Once one founder's speed has taken over, there is almost no variation left, so S, and with it R, falls toward zero. Mutation adds variation back, but random nudges to an already-favored speed rarely improve it.

At the largest mutation size, mutation also pushed speeds into the clipped ends of the 0–255 range, and selection and mutation settled into a balance instead of a steady climb.

Trying to break the conclusion

We fixed the tests, thresholds and conditions in advance, then ran 16 fresh random seeds, with eight more for replication. Every condition was paired seed by seed with a control dish, and a test passed only at t ≥ 2.6. That bar corresponds to about a one-sided p < 0.05 after a Bonferroni correction for four tests: dividing the 5% false-alarm budget among them.

The competitive advantage passed. The tests for continuing improvement did not. The slope of newborn speed from the second to the fifth tenth of the run, compared with the control, came out at +2.7 ± 3.4 and +2.8 ± 3.0 for the two copying rules. Both are smaller than their own uncertainty.

We then mapped mutation rates, cell supplies and a cost for speed across another 360 dishes: 72 million more simulation steps, before counting the competition trials. The cost shortened a bot's lifespan in proportion to its speed gene:

lifespan = L · (1 − c · g / 255)

where L is its normal lifespan and c is 0, 0.25, 0.5 or 0.75, so at the highest cost the fastest bots lived a quarter as long.

In every condition with a reasonable population and no more than a modest cost for speed, the early shift appeared, at about the same size. A cost of 0.5 halved it, and 0.75 reversed it. Sustained change passed in only one of 40 conditions. With eight seeds per condition, a test with no real effect still clears t ≥ 2.6 about 1.8% of the time, so about 0.7 false passes are expected across 40 conditions. A single success is about what luck delivers, not evidence of a general effect.

We also tried a back-of-the-envelope predictor, the product N × s × h: population size, the strength of selection on speed, and heritability. It tracked the cost of speed closely (correlation 0.95) but not mutation (−0.25). Mutation inflates the spread of speeds, and s is measured in units of that spread, so it hides how hard selection is really pushing.

The bots adapted. Over the roughly 75 generations we studied, we found no convincing evidence that improvement kept accumulating.

Why a plateau is the interesting result

I find that more compelling than an endlessly rising graph would have been.

It forces us to be precise about what an adaptive system is. A population can reproduce, inherit, undergo selection and become measurably better at competing, all while making no further progress once its initial advantages have spread. A rising curve can be the sorting of variation that was there from the start. Showing continuing innovation takes more evidence than that.

It also points to better questions. What happens when the environment changes? When one useful trait opens access to another? When improving a trait keeps paying off instead of running into a limit? When building, inheriting and reproducing are more tightly linked, or less?

Biobot Tierra makes those questions concrete. We can change a rule, replay a history, inspect every birth, and find out that the explanation we liked was wrong. Several times in this study, we did.

What this does not tell us

So there are limits to what we found. The bots are simple 2D agents on a lattice. Their inheritance mechanisms are hypothetical. Piles hatch in seconds rather than days, populations top out around 39 bots, and runs end after about 75 generations. A plateau here cannot show that real xenobots would stop evolving, or that an evolving biological system would be safe. Any biological work in this direction would need its own scientific and ethical assessment.

What we have is a working experiment at the boundary between replication and continuing adaptation, and a sharper question: what lets an adaptive system keep finding useful changes after its first advantages have spread?

We gave the bots inheritance. They made good use of it.

Now we want to understand what would give them somewhere else to go.

Explore Biobot Tierra · Read the methods and results

Saturday, April 25, 2026

The Cache Is the Thought — What KV Caching Reveals About How AI Actually Works

Architecture · Inference · Memory

The Cache Is the Thought

KV caching is usually described as an optimization. It is actually something more fundamental. The mechanism by which large language models hold a thought in their head.

Technical Essay — 10 min read

• • •

There is a moment, when you look closely enough at the details of large language models, when a seemingly minor implementation detail suddenly reveals itself as something structural. A load-bearing idea that everything else depends on. For me, that moment came with reading some of the recent ideas surrounding KV caching. What looked at first like a straightforward performance optimization turned out to be the core mechanism that makes modern AI inference not just fast, but possible. Understanding it changes how you think about what these systems are actually doing.

To see why, start with the mathematics of attention. The operation at the heart of every transformer model deployed today.

The Quadratic Problem

Attention(Q,K,V) = softmax( QKᵀ / \sqrtd k)V The attention operation

In their raw mathematical form, transformers are elegant but brutally expensive. Generating each new token requires attending over every previous token in the sequence, which means computation grows quadratically with sequence length. A sequence twice as long is four times as expensive. This is a direct consequence of what attention is doing. The mechanism works by computing relationships between every pair of tokens, and the cost of that operation scales with the number of pairs.

The result, without any mitigation, is that generating text token by token requires recomputing the same keys K and values V for every prior token at every step. The context grows. The recomputation grows with it. Something that might take milliseconds for a short sequence becomes seconds for a long one and for the context windows that modern applications demand, it becomes entirely impractical.

KV caching addresses this with a conceptually simple observation: if the keys and values for previous tokens haven't changed, why recompute them? Store them once. At each new step, compute only the query q_t for the current token, and compare it against what's already cached. The incremental cost of generation collapses from recomputing the entire past to attending over a stored representation of it.

That shift, from recomputation to reuse, is what takes inference from something theoretical to something deployable. But the deeper implications of the shift are less obvious than they first appear.

For details on the attention mechanism of K, Q, and V matrices see this article I wrote previously.

The Bottleneck Nobody Was Watching

The memory cost of a KV cache scales with the number of layers, the number of attention heads, the head dimension, and the sequence length. That last term, sequence length, is where the problem lives. It looks innocent in a formula. In practice it dominates everything.

The Scaling Reality

At long context lengths, the KV cache routinely exceeds the size of the model weights themselves. A system you might describe as "running a 70B parameter model" is, in memory terms, often doing something closer to managing an enormous dynamic tensor that dwarfs the static parameters. The model weights are the smaller problem.

This is where something important happened quietly in the AI scaling narrative, and it hasn't fully registered in how most people talk about these systems. For years, the dominant frame was compute: how many FLOPs does training require, how many does inference require, how do we reduce them. That frame is not wrong, but it is increasingly incomplete. In a large number of real inference scenarios, compute is no longer the bottleneck. Memory is specifically, the cost of moving the KV cache tensors around fast enough to keep the compute units fed.

This is a meaningful shift. Compute follows Moore's law reasonably well. Memory bandwidth is harder. Data movement: getting tensors from where they're stored to where they're needed, at the speeds modern accelerators demand is a problem with different scaling properties than arithmetic. A system optimized purely for FLOPs can still be slow because it spends most of its time waiting for data to arrive.

We used to think in terms of model size and compute. We should be thinking in terms of memory: how we move it, compress it, and decide what to keep.

Understanding this reorients how you think about the architecture. The question is not just how many parameters a model has or how many operations inference requires. It is where does the KV cache live at each moment in generation, how much of it fits on fast memory, and how expensive is it to retrieve the parts that don't?

The Geometry of Compression

The natural response to a memory problem is compression. And indeed, significant research effort has gone into compressing KV caches: quantizing the tensors, applying transform coding, reducing precision from FP16 to INT8 or lower. These approaches work, but they face a mathematical constraint that is easy to miss and important to understand.

Attention operates on inner products. The softmax that produces attention weights depends on the dot products between queries and keys: q · k. If compression distorts the geometry of the key and value tensors: if it changes the relative distances and angles between vectors then those dot products change, the softmax distribution changes, and everything downstream degrades. The outputs of the model shift in ways that may be subtle or severe depending on how aggressively you've compressed.

This is the same constraint that appears in classical dimensionality reduction problems. The Johnson-Lindenstrauss lemma, random projections, and related results all address the same core question: how do you reduce the size of a representation while preserving the pairwise relationships that matter? KV compression is that problem, applied to the geometry that attention depends on. You can compress, but you have to compress in ways that respect the inner product structure. Naive quantization that ignores this tends to fail in practice even when it looks acceptable on simple benchmarks.

What compression must preserve

The relative geometry of key vectors; specifically, the dot products q·k that determine which tokens attend to which. Distort these and the attention distribution changes.

What this means in practice

Aggressive quantization requires careful calibration. The safe compression budget is smaller than it looks. Geometry-aware methods consistently outperform naive approaches at the same bit depth.

What Gets Remembered

Compression reduces the cost of storing the cache. But there is a more radical question lurking underneath it: do you need to store all of it at all?

In any sufficiently long sequence, not every token matters equally to what comes next. Some tokens receive consistently high attention weights from many subsequent tokens. They are load-bearing elements of the context, anchors that the model returns to repeatedly. Others receive weights so close to zero that they are, for practical purposes, invisible. Attending over them produces essentially the same result as not attending over them. They are terms in a sum that contribute nothing meaningful to the output.

Selective eviction, dropping low-attention tokens from the cache rather than storing them indefinitely, is the architectural response to this observation. It is mathematically equivalent to approximating the full attention distribution with a sparse subset: keeping the entries that matter and discarding the ones that don't. The approximation is generally very good because the entries being discarded were contributing almost nothing.

What makes this genuinely interesting, rather than merely practical, is what it implies about the nature of context in these systems. If you can drop most of a long sequence and lose almost nothing, then the model is not using context the way a human reader uses a document: scanning it linearly, building a cumulative understanding of every sentence. It is using context more selectively, more dynamically, attending to a relatively small set of salient anchors while the rest fades into irrelevance.

The KV cache is a working memory and like all working memories, what it forgets matters as much as what it keeps.

This opens the door to something more ambitious: learned eviction policies, where the model itself develops a sense of what is worth remembering. Rather than applying a fixed rule, drop tokens below an attention threshold, the system learns to anticipate which tokens will remain relevant as generation continues. That is a very different relationship to memory than the one most people imagine when they think about how language models process text.

A Subtle Numerical Point

One of the less discussed properties of KV caching is that it is not, in a strict sense, numerically neutral. Because modern inference operates in finite precision, typically FP16 or lower, the act of caching tensors and retrieving them later can produce slightly different results than recomputing the same values fresh at each step. Floating point arithmetic is not associative. The order and grouping of operations matters, and caching changes both.

At first this seems like a defect. A source of numerical noise introduced by an optimization. But the right way to think about it is as a reminder that these systems are dynamic numerical processes, not purely symbolic ones. They are not executing a fixed logical procedure that produces a deterministic output. They are performing approximate arithmetic at scale, and the approximations matter. In some cases, caching actually improves numerical stability by avoiding the accumulation of rounding errors that would occur across repeated recomputation.

The practical implication is modest: reproducibility at the bit level requires careful attention to whether and how caching is implemented. But the conceptual implication is worth sitting with. The outputs of these models are sensitive, in small ways, to the history of how their computation was organized. The past, in a very concrete numerical sense, shapes the present.

Memory as Architecture

Pulling back from the technical details, there is a framing shift that KV caching makes almost inevitable once you take it seriously. The standard way to describe a language model is in terms of its weights: the parameters learned during training that encode the model's knowledge about language, facts, and reasoning patterns. Those weights are static. They don't change during inference. They are, in a loose sense, the model's long-term memory.

The KV cache is something different. It holds the immediate context, i.e. the specific sequence the model is currently processing, the active thread of the conversation or document. It is populated at runtime and discarded when inference ends. It is, in the same loose sense, the model's working memory: what it is currently holding in mind as it generates.

This distinction between long-term knowledge encoded in weights and short-term context held in cache maps onto something recognizable from cognitive science. And once you have that framing, a lot of recent engineering trends in AI infrastructure begin to look less like isolated optimizations and more like different strategies for managing a memory system.

Hierarchical memory systems that tier the KV cache across GPU, CPU, and SSD storage are solving the same problem as the brain's distinction between fast working memory and slower long-term retrieval. Cache reuse across multiple requests, so storing the representation of a long system prompt so it doesn't need to be recomputed for every user query is a form of memoization that treats shared context as persistent memory. Multi-agent systems that share KV state across instances are, in effect, building a shared working memory distributed across multiple reasoning processes.

The Emerging Picture

Model weights encode what a system knows. The KV cache encodes what it is currently thinking about. The next wave of AI infrastructure improvements will largely be about the relationship between those two things, i.e. how to make working memory larger, cheaper, faster, and smarter about what it retains.

What this suggests is that the next significant advances in AI capability will not come only from larger models or better training data. The axes that have dominated the conversation for the past several years. They will come substantially from how memory is managed: how it is compressed, how it is moved, how it is prioritized, and ultimately how models are taught to reason about their own context rather than passively consuming it.

KV caching began as a simple answer to a quadratic scaling problem. It has become, without anyone quite announcing it, the foundation on which practical AI cognition is built. The weights are what a model knows. The cache is what it is thinking. And thinking, it turns out, is mostly a memory management problem.

The next frontier in AI is not more parameters. It is smarter memory.

Friday, April 24, 2026

The Last Justification — Why Leaders May Be Running Out of Reasons to Exist

Philosophy · Economics · Artificial Intelligence

The Last Justification

Leaders have always claimed their authority from the management of scarcity. What happens when scarcity ends?

• • •

In 1930, John Maynard Keynes predicted that by 2030 the economic problem, the human struggle for subsistence, would be solved. He was writing at the depths of the Great Depression, which made the forecast either visionary or crazy depending on your viewpoint. Now in 2026, the people building artificial intelligence are implying essentially the same prediction on a faster timeline, and they have more reason to believe it than Keynes did.

That convergence deserves more attention than it is getting. Because if the economic problem is actually being solved. If AI delivers anything close to the post-scarcity conditions its most serious architects believe it will of infinite abundance, then something else dissolves alongside scarcity that nobody is talking about. The entire philosophical and practical foundation on which human leadership has rested for ten thousand years.

Leaders, in every civilization that has ever existed, have drawn their deepest authority from one source: someone has to manage the scarcity. Not divine right. Not democratic mandate. Not military necessity. Those are the decorative justifications, the ones that get written into constitutions and carved into monuments. Underneath all of them, holding the structure up, is the one argument that was always hardest to refute. Resources are finite. People compete for them. Without coordination and control, the result is violence. Therefore: leaders.

What happens when that argument stops being true? What if the future doesn't need leaders?

This might have seemed radically theoretical a generation ago. But the possibility of artificial general intelligence (AGI); systems that can perform most economically valuable cognitive work at near-zero marginal cost threatens to dissolve the scarcity foundations on which every human hierarchy from the pharaohs to the Fortune 500 has been constructed. To understand how radical this might be, it helps to trace the argument from its roots.

Scarcity as the Engine of Hierarchy

Before agriculture, humans lived in small bands of twenty to one hundred fifty people. Leadership in these groups was fluid, situational, and constantly checked by group consensus. The best hunter led the hunt. The most experienced elder arbitrated disputes. When the situation changed, so did the leader. There was no permanent hierarchy because there was, relatively speaking, no permanent scarcity. Needs were simple. Nature provided enough, albeit not abundantly, but enough.

The transformation came with the agricultural revolution around ten thousand years ago. Stored grain created something new in human experience: the possibility of accumulated surplus. And with surplus came its shadow twin, artificial scarcity; the condition in which resources exist in sufficient quantity but are distributed unequally through ownership and control.

The first leader who said 'this is mine' did not solve scarcity. He manufactured it.

Rousseau (paraphrased), Discourse on Inequality, 1755

Jean-Jacques Rousseau saw this with unusual clarity in 1755. His Discourse on Inequality argued that human beings in their natural state were essentially content. Needs were simple, and there was no systematic domination. The fall came when someone enclosed a piece of land and declared ownership, and others accepted the fiction. From that moment, leaders emerged not to solve natural scarcity but to protect and perpetuate artificial scarcity, i.e. to maintain the conditions that justified their own authority.

This is a darker reading of leadership than most civics textbooks provide. But it is a consistent one. Almost every function we associate with leaders, such as adjudicating property disputes, organizing armies, taxing and redistributing, managing information asymmetries traces directly to the management of scarce resources or the conflict that scarcity generates.

A Prediction Nobody Remembers

1930 Keynes writes his forecast

In 1930, at the very depth of the Great Depression, the economist John Maynard Keynes published a short essay that almost nobody read at the time and almost nobody remembers now. It was called Economic Possibilities for our Grandchildren, and it contained one of the most audacious forecasts in the history of economic thought.

While the world economy was collapsing around him, Keynes told readers not to confuse a cyclical crisis for the long-run trend. Step back far enough, he argued, and the trajectory was unambiguous: productive capacity per person had been compounding for generations. Within one hundred years, which puts us at 2030, the "economic problem," meaning the struggle for subsistence, would be solved. Not alleviated. Solved. He predicted living standards in advanced economies would be four to eight times higher than in 1930. On the material side of his ledger, he was roughly correct.

But the radical heart of the essay was what he thought would follow from this. He predicted the working week would shrink to perhaps fifteen hours. Not because of idleness, but because fifteen hours of labor would genuinely be sufficient to sustain abundant lives. The rest would be leisure. And here his anxiety surfaces in the prose, because he was not celebratory about this. He worried that humans had been so thoroughly shaped by the discipline of scarcity that they would not know what to do with themselves in its absence.

The Keynes Problem

Keynes predicted material abundance with remarkable accuracy. What he underestimated was how powerfully existing hierarchies would capture and redirect that abundance and how deeply the psychology of scarcity was embedded in human identity. Productivity gains arrived on schedule. The fifteen-hour work week did not.

He described what he called "purposiveness" which is the orientation toward future goals over present experience that scarcity conditions produce as a useful neurosis. Civilization needed it to pass through the scarcity phase. But it would need to be shed once that phase ended. Fail to shed it, and people would generate artificial purposes, artificial hierarchies, artificial scarcities simply to maintain the familiar structure of striving.

That failure to shed it is largely what happened. The abundance arrived, and was immediately captured by expanding desires, by concentrated ownership, by the managerial and political class that Keynes assumed would become unnecessary. Rather than the state withering away, institutions found new rationales for their authority. The leaders did not dissolve into leisure alongside everyone else. They found new scarcities to manage.

The Anarchist Tradition Had It Right

Keynes was not alone in this structural diagnosis, though he arrived at it from a very different direction. A generation earlier, the Russian anarchist Peter Kropotkin had argued in Mutual Aid that cooperation, not competition, was the dominant force in both nature and human history and that in conditions of genuine shared abundance, hierarchical leadership would become not just unnecessary but actively parasitic.

Kropotkin's crucial move was the same as Rousseau's, stated more bluntly: scarcity is largely manufactured by the ownership structures that leaders enforce, not a natural condition requiring management. Remove the leaders, redistribute the abundance, and the entire philosophical justification for authority collapses. What remains is voluntary coordination. People organizing themselves around shared tasks without anyone claiming the permanent right to direct others.

Marx made the same argument through different machinery. His endpoint, full communism, is explicitly a post-scarcity condition. In that terminal state, he wrote, the state withers away because there is nothing left for it to manage. Leaders and governments are, in his framework, instruments for managing class conflict over scarce resources. Eliminate scarcity through technological development and collective ownership, and the entire apparatus becomes superfluous.

That no Marxist state ever reached this destination is a separate conversation. What matters here is the theoretical logic, which is precise: scarcity is the engine, hierarchy is the machine the engine drives, and without the engine the machine has no purpose.

What Remains Without Scarcity

Before accepting this conclusion too quickly, it is worth asking what would remain of leadership if scarcity were genuinely eliminated. The honest answer is: something, but less than we might suppose.

Pure coordination problems exist independent of scarcity. Even in conditions of perfect material abundance, groups face the challenge of synchronizing action: deciding which direction to go, when to act, how to sequence collective effort. These are not about managing competing claims on scarce resources. They are about the mathematics of group decision-making: any time more than two people have potentially different preferences, some mechanism is needed to aggregate those preferences into a single action.

But notice what that mechanism looks like in the absence of scarcity pressure. It looks less like a king, a CEO, or a general, and more like a protocol. A shared norm. A voting procedure. The coordination residue of post-scarcity society might not deserve the name "leadership" at all. It might simply be coordination, which is a very different thing.

Leadership as we know it may be a scarcity-adapted institution that has outlived the conditions that made it necessary.

The anthropologist David Graeber, in his final book The Dawn of Everything, documented societies that maintained deliberate mechanisms to prevent permanent leadership from emerging such as seasonal leadership, ritual humiliation of chiefs, voluntary dispersal when hierarchy became too rigid. His argument was that hierarchy is a choice, not an inevitability, and that many human societies have at various points chosen otherwise. The story that we have always needed leaders is itself, he suggested, a story that leaders tell.

The Rupture That Changes Everything

This is the point at which a ninety-year-old essay about economic possibilities becomes urgently contemporary.

Keynes assumed abundance would arrive gradually as continued industrial productivity compounding steadily across generations. What he did not anticipate was a discontinuous jump: a technology that does not incrementally improve labor productivity, but potentially replaces the need for labor across entire categories of cognitive work, almost simultaneously, within a compressed timeframe.

1930

Keynes publishes his forecast. Predicts the economic problem solved by 2030 via gradual compounding of industrial productivity.

2022–2024

Large language models demonstrate capability across most categories of cognitive work. The gradual path becomes a potential cliff edge.

2025–2026

AI lab leaders begin publishing explicit post-scarcity forecasts. Amodei's Machines of Loving Grace predicts compression of a century of scientific progress into a decade.

2030

Keynes' original target date. The destination may arrive approximately on schedule, but via a road he could not have imagined.

If you take seriously what people like Dario Amodei are actually arguing and not the hedged public statements but the internal conviction, the claim is that we are within years of systems capable of performing most economically valuable cognitive work at near-zero marginal cost. Amodei has written explicitly about compressing a century of scientific and economic progress into roughly a decade. Sam Altman has gestured at civilizational transformation on similar timescales.

If that trajectory is even directionally correct, Keynes was not wrong. He was describing a destination that turns out to be reachable by a faster, stranger road than he imagined. His 2030 prediction may prove accurate almost to the year, while being entirely wrong about the mechanism.

This is not entirely theoretical. The early tremors of this structural shift are already visible inside organizations operating today. Middle management layers; whose primary function has always been information relay, coordination, and resource allocation across hierarchical tiers are being cut across industries at a pace that would have seemed implausible a decade ago. Teams are shrinking. Reporting structures are flattening. The organizational pyramid that defined the modern corporation for a century is losing floors.

More telling is what is happening to specialization. For most of human history, specialized knowledge was a primary source of individual power and, by extension, a justification for hierarchy. The lawyer, the engineer, the financial analyst, the data scientist occupied positions of authority partly because they possessed capabilities others simply did not have. AI is eroding that boundary rapidly. Skills that once required years of training to acquire can now be approximated in minutes by someone with no formal background and the right tool. When specialization stops being scarce, the hierarchies built around controlling access to specialized knowledge lose their rationale alongside it.

What is happening inside companies is a preview of a much larger structural question. Organizations are not shedding management layers because they have decided hierarchy is philosophically unjustified. They are shedding them because those layers are no longer functionally necessary. The economic logic that created them has changed. That same logic, operating at civilizational scale, is what Keynes was pointing at and what the most serious AI forecasters believe is now accelerating toward its conclusion.

The Question of Capture

But this is where the argument reaches its sharpest and most uncomfortable edge. Abundance delivered by artificial intelligence does not automatically mean distributed abundance. The critical question and the one that Rousseau would recognize immediately is who owns and controls the systems that produce it.

If a small number of companies or individuals control the infrastructure that generates post-scarcity conditions, then scarcity-based hierarchy has not been eliminated. It has been concentrated to a degree without historical precedent. Rather than a million leaders managing localized scarcity, you have a handful of individuals managing the systems that produce everything. The entire apparatus of leadership; its justification, its function, its claim on obedience collapses upward into a single point of control.

This is not a hypothetical concern. The same dynamic that turned agricultural surplus into feudalism, and industrial surplus into plutocracy, is already visible in the ownership structure of the systems being built. The logic is identical. Scarcity is not eliminated; it is repackaged. Access to abundance becomes the new scarce resource, and the leaders of that access inherit all the authority that scarcity has always generated.

The Recurring Pattern

Agricultural surplus → feudal hierarchy. Industrial surplus → plutocratic hierarchy. AI surplus → ? The technology changes. The capture dynamic has proven remarkably stable across all three transitions. The question is whether this one is different enough to break it.

Altman himself has implicitly acknowledged this with his advocacy for Universal Basic Income and OpenAI's original nonprofit structure. A recognition that without deliberate redistribution mechanisms, the abundance his systems might produce would simply reconcentrate at the top. The leaders would not wither away. They would become more powerful, not less, precisely because they control the systems that have made all other scarcity manageable.

An Ending Without a Conclusion

History has tested the scarcity-leadership connection repeatedly without ever fully breaking it. Every technology that promised to dissolve hierarchy: the printing press, the steam engine, electrification, the internet instead generated new hierarchies organized around control of the technology itself. The same dynamic may play out with AI.

But there is something qualitatively different about a technology that can replace cognitive labor across essentially all domains. Previous technologies replaced physical labor in specific sectors while creating new categories of cognitive work. The demand for human participation in the economy shifted but did not disappear. If AI eliminates the demand for cognitive labor as comprehensively as previous technologies eliminated the demand for physical labor, without creating new categories of work to absorb the displaced, then the economic basis for most leadership simply ceases to exist.

Keynes thought this moment would come gradually, giving human psychology and institutions time to adapt. He was wrong about the pace. The institutions built on scarcity logic are already struggling to adapt to a world that is merely becoming more automated but let alone one that reaches genuine post-scarcity conditions.

The question is whether we get a post-scarcity world at all as the traditional need for leadership fades. Every previous surplus in human history has been captured by feudal lords, by industrialists, by financiers and redirected. The realistic risk is not the dissolution of hierarchy but its opposite: a small group of people controlling systems that can produce everything, governing a world that finally has no material reason to accept being governed.

Keynes was right about the destination. The only question is whether a handful of people will control the road to get there.

Sunday, May 11, 2025

Polanyi’s Paradox: Why We Know More Than We Can Tell

Polanyi’s Paradox is the idea that much of what we know cannot be clearly expressed in words or formulas, and it is something I have thought about a lot, especially in the context of engineering and AI. Despite what some might assume, software engineering is not just about generating code. It involves layers of intuition, architectural decisions, project context, unwritten standards, and rules of thumb that experienced engineers learn over time. This is why, when engineers struggle to explain exactly how tools like Cursor or Windsurf fit into their workflow, it is not because the tools aren’t valuable, because they are. And it's not because engineers lack communication skills, it's because these tools support only part of what software engineers do. Much of the rest relies on experience and intuition that is hard to articulate. In this post, I want to explore how Polanyi’s Paradox connects to software engineering, what it tells us about the current state of AI, and what it might take to build AI systems that truly grasp the kind of knowledge humans use every day that is tacit.

But first, let's look back at the historical origins of this idea.

Historical Perspective

Michael Polanyi, a Hungarian-British polymath (1891–1976), introduced what later came to be known as Polanyi’s Paradox in 1966. Polanyi started his career as a distinguished physical chemist before turning to philosophy, bringing a scientist’s insight into how knowledge actually works in practice. In his book The Tacit Dimension, he famously wrote “we can know more than we can tell”, referring to the tacit knowledge that underlies many human skills. For example, Polanyi pointed out that we recognize familiar faces effortlessly without being able to list the exact features or rules that distinguish each face; a nod to Gestalt perception and intuition.

He illustrated the gap between knowledge and articulation with everyday instances: “The skill of a driver cannot be replaced by a thorough schooling in the theory of the motorcar,” he noted, just as one’s intimate know-how of using one’s body is entirely different from a textbook understanding of physiology. In short, there is always a residue of know-how that we carry in our minds and muscles that we cannot fully put into words.

In a time when many thinkers believed all knowledge could be neatly defined and broken into rules, Michael Polanyi pushed back with a simple but powerful idea: we know more than we can tell. He argued that intuition, experience, and personal judgment are essential parts of how we understand the world. For Polanyi, knowing isn’t just about facts and logic; it’s also about the unspoken, hard-to-explain feel we develop through doing and observing.

Relevance to Software Engineering

Although formulated 60 years ago, Polanyi’s Paradox is alive and well in modern software engineering. Despite the field’s basis in logic and code, much of what good software developers and teams do resides in tacit knowledge rather than explicit rules. In fact, studies suggest that only the “tip of the iceberg” of knowledge in a software organization is explicit and documented, perhaps ~20–30%, while the vast majority (70–80%) is unwritten, experience-based tacit knowledge.

In software teams, explicit knowledge (documents, code repositories, formal processes) is just the visible tip. The bulk of “what we know”, such as the skills, experience, intuition, team culture lies beneath the surface, undocumented but crucial. This tacit layer includes things like unwritten best practices, intuitive design sense, and the many gotchas one learns only through hard-earned experience.

This means that a software team’s most critical understanding often lives in people’s heads and habits rather than in manuals or comment blocks. Seasoned engineers accumulate a deep reservoir of intuition about architecture, code quality, and problem-solving approaches that is hard to articulate formally. They just "know" certain things which is a direct reflection of Polanyi’s Paradox in the tech world.

Polanyi’s Paradox in Software Engineering

In day-to-day software development, Polanyi’s Paradox shows up in numerous ways. For example:

User Interface Design: Good UI/UX designers often rely on an instinctive feel for what is intuitive to users. There is no complete rulebook for “easy to use.” Much of it comes from empathizing with users and leveraging subtle design sensibilities developed over time. A designer might not be able to fully explain why one layout “just works” better than another, because they are drawing on tacit knowledge of human behavior and aesthetics learned through many projects and feedback cycles. This know-how stands in contrast to explicit guidelines (like style guides or usability heuristics). Those help, but the "artistry" goes beyond what’s written down.

Debugging: Tracking down a complex bug is as much an art as a science. A veteran developer can often zero in on the likely cause of a problem quickly, guided by a “gut feeling” from past debugging experiences. This skill of knowing where to look in thousands of lines of code or which log hint to pursue is typically acquired through experience and not easily codified or taught. Two different engineers might solve the same bug via very different thought processes, neither of which is fully documented anywhere. Debugging knowledge lives in their heads as pattern recognition: “Ah, I’ve seen something like this before and it might be due to X.” Such intuitions are hard to write into a step-by-step troubleshooting guide.

Knowledge Transfer in Teams: When a new developer joins a software team, they often go through an onboarding period that involves shadowing others, pair programming, and code review. These practices recognize that a lot of the team’s knowledge is “tribal knowledge”, accumulated wisdom about the codebase and conventions that isn’t in the official docs. From knowing the historical reasons why module Y was built a certain way, to understanding which team member to ask about database quirks, these are things newcomers learn through social interaction.

Much of what senior engineers pass on to juniors is tacit: it’s storytelling, mentorship, and shared experience. If a key developer suddenly leaves, they take a trove of tacit knowledge with them, and it’s often non-trivial to fill that gap. Attempts to capture everything in exhaustive documentation often fall short, because you can’t foresee or codify every relevant detail. This is in spite of the best efforts of Program Managers and tools like Jira, Confluence, etc. Indeed, trying to convert all tacit knowledge into explicit form is “fundamentally flawed due to the very nature of tacit knowledge, which is inherently personal, context-dependent, and difficult to articulate.”

These examples show that software engineering is not just about formal specifications and algorithms; it’s a human craft. The most effective development teams leverage tacit understanding in architecture decisions, code readability, and anticipating user needs. And when this tacit element is missing, say for example, a team only follows rigid checklists without any personal intuition then the results are often mediocre. Polanyi’s Paradox explains why certain programming expertise can’t simply be “transferred” by reading a book.

As a consequence, strategies like code reviews, pair programming, and apprenticeship-style learning are crucial in tech because they help share the unspoken wisdom. Conversely, over-reliance on documentation has limits: no matter how many pages of design docs you write, there will always be nuance that new engineers must pick up by working with others. In short, a great deal of “what developers do” cannot be fully captured in code comments or process manuals. They know more than they can tell.

The Future of Software Engineeing

So all of this should make software engineers feel pretty good about their current job security, because their job is more than just code generation and anyone who has used AI tools knows that even though they can speed up their producitivity tremendously, these tools fall woefully short in many areas primarily because of this tacit knowledge engineers have built up. This is despite some CEOs making announcements that they are replacing engineers with AI. These annoucements are generally made by people who have never built and maintained a large code product or if they have it's been decades since they did so. Furthermore, announcements where they say AI is writing X% of their code are highly suspect because:

It's unclear what they are counting in the X%. Is it agents writing big chunks of code based on prompts? As opposed to how much of the X% is just code completion. Because, for example, as you type something into Google Docs and it suggests the next word or phrase, do you really think that AI wrote X% of that doc because it's doing autocomplete?
They fail to recognized the tacit knowledge that goes into code architecture.

However, this is the current state for software engineers. Thinking that this will be the future and won't change is just engaging in "copium", which I see a lot of on LinkedIn. For example, take a look at the replies to the CEO of Zapier on LinkedIn here where he states that AI won't replace jobs. There's a lot of wishful thinking in the comments that things aren't going to change that much. But this is the current state and people are not internalizing the exponential change that is happening in AI.

Additionally, the Polanyi's Paradox of tacit knowledge that is part of software engineering isn't a formidable problem to solve for AI agents and doesn't even require a huge technological breakthrough. So how do you solve the fact that an AI agent in a program like Cursor or Claude Desktop doesn't have all of an engineer's tacit knowledge? Well you give access to all that knowledge to the AI. So access to Slack conversations, entire organization repos, Microsoft Teams meetings, project management notes on customers, stand up meeting recordings, Jira, Confluence, whatever the organization is using to capture that history. If that is done, then you have an AI agent or agents that has more organization and project knowledge than any engineer that works there.

The requirements to do that are larger context windows as extended information, agents that can operate effectively over that context memory such that agents can remember over sessions, and access all of that data. Infinite memory agents are within reach now. There are many possible solutions to this now including MemGPT from Letta along with the frontier AI companies extending their context windows. The biggest obstacle will be helping organizations streamline all of their data and communications to be made available to AI agents. However, organizations are going to be very motivated to do this, because those that do make their data and commuications available are going to have a huge competitive advantage because of their accelerated pace of software delivery and better organizational decision making.

So this is not a big leap to make agents that have this extended memory ability and being able to maintain intent over these tasks. The biggest impediment will be organizations being able to adapt their data and internal processes. Does this mean that all software engineering jobs will be eliminated? No. Engineers who can orchestrate, manage, secure, and generally work with AI agents will be in demand. So those who are using AI as part of their current process and learning as much as they can about taking advantage of AI and looking to innovate around AI Engineering will be fine.

Unfortunately, there is the mindset of many engineers who don't bring anything more to the table other than their ability to pull a story off of a Jira backlog that was written by a project manager and complete that story and who don't think in terms of innovation or design at all. They work in a bureaucratic mindset of just task completion and generating just the code for that task and not in a mindset of generating ideas and innovating. This type of engineer who doesn't innovate and specifically doesn't innovate using AI will be gone and will not be able to find a job like that again in engineering. In other words:

If you are an engineer who is used to thinking like a robot, you will be replaced with a robot.

But I don't think that most engineers can be blamed for thinking of their job in terms of being automatons writing code. Many engineers ever since getting out of school have been conditioned through endless cycles of sprints, planning meetings, story points, and retrospectives to think this way. Also, in many organizations, there are layers of project managers and customer success people between them and the end users, which reinforces the idea that their end goal is code generation and not building a product for a user. In addition, in larger organizations, they may only work on a small functionality of the end product. But if they are going to survive, they are going to need to bring their human-ness to the job, their uniqueness based on their personal experiences and innovate and be creative.

Relevance to Building Better AI

So given the Polanyi's Paradox on engineering, what have been the implications for AI?

Polanyi’s Paradox has long been recognized as a core challenge in artificial intelligence. Let's go back to the beginning. In the early decades of AI (mid-20th century through 1980s), programmers found that many tasks humans find trivial were extremely difficult to specify to a computer, precisely because of tacit knowledge. As Polanyi himself noted, tasks like driving a car in traffic or recognizing a face rely on a vast array of subtle cues and intuitions that we execute without conscious thought. We can perform them, but we can’t easily enumerate how we do it. This posed a problem: how do you write a program to do something you can’t fully explain to yourself? Early AI could handle problems that had clear rules (like solving equations or playing chess to some extent), but it struggled with open-ended, real-world tasks. Polanyi’s paradox was identified as a major obstacle for AI and automation. Unless we have a complete, explicit recipe for a task, getting a machine to do it is extremely difficult.

A classic example is commonsense reasoning: even a toddler understands that a wobbling stack of blocks might fall, or that a snowman in the road isn’t a threat, but such “common sense” eluded AI programs because nobody could fully encode all the needed if-then rules. As one economist summarized, the tasks easiest to automate are those that “follow explicit, codifiable procedures,” whereas the ones requiring “flexibility, judgment and common sense, skills that we understand only tacitly, have proved most vexing to automate.” Indeed, for decades computers remained “less sophisticated than preschool-age children” at tasks like understanding natural language or navigating an unpredictable environment. This is Polanyi’s Paradox writ large in technology: machines hit a wall when facing the tacit dimension of human know-how.

The past decade, however, has seen much progress in AI, largely by finding ways around Polanyi’s Paradox rather than directly solving it. Since we can’t easily tell the machine what we know implicitly, we let the machine learn from experience just as humans do. In other words, “contemporary AI seeks to overcome Polanyi’s paradox by building machines that learn from human examples, thus inferring the rules that we tacitly apply but do not explicitly understand.” Instead of programmers hand-coding every rule, we feed AI systems with massive amounts of data (images, recordings, text, demonstrations) and let them figure out the patterns.

This is the approach of machine learning, and especially deep learning. For instance, rather than attempting to enumerate all the visual features that distinguish a pedestrian from a shadow on the road, engineers train a self-driving car’s vision system on millions of driving images and let it learn the concept of “pedestrian” by example. The AI essentially internalizes a web of correlations and features, a form of tacit knowledge, from the data.

AI can now capture tacit patterns by finding regularities in unstructured inputs. A striking success of this approach was in the game of Go: expert Go players often say they rely on intuition to judge a good move (they “know” a move is strong without being able to verbalize why). Early AI couldn’t crack Go by brute-force logic alone. But a deep learning system (AlphaGo) was able to absorb millions of human moves and even play games against itself, eventually acquiring an almost intuitive style of play that surprised human experts. In effect, AlphaGo learned the tacit principles of Go that even top players couldn’t articulate, and it went on to defeat the world champion.

Similar stories abound: large language models (LLMs) were trained on billions of sentences of human text and as a result, they learned the hard-to-pin-down rules of language, style, and meaning without anyone programming those rules in directly. These models can answer questions, write essays, or carry a conversation in a very human-like way, not because they were explicitly told how to do it, but because they statistically absorbed how humans communicate. In other words, they captured a slice of our tacit linguistic knowledge.

Interestingly, many of today’s AI successes still depend on how well we can communicate our goals to the system; the unfortunately named idea that we are all familiar with of prompt engineering. Because machines lack the shared human context we often take for granted, we’re required to make our implicit understanding more explicit than we typically would with another person. When working with a coworker, you might simply say, “You know what I mean,” and count on shared experience to fill in the blanks. With AI, that luxury doesn’t exist. Instead, we must carefully spell out our expectations, constraints, and context. Getting the AI to produce a useful output often involves trial and error, tweaking inputs, and drawing on our own unspoken know-how about what might steer it in the right direction. This interaction process is, in itself, a reflection of our tacit knowledge; how we choose to phrase a task, what examples we give, and which assumptions we clarify all stem from our intuition about how the system behaves. In this way, human users must surface their internalized expertise just to bridge the gap between what we understand intuitively and what the machine needs to be told explicitly.

The human is leveraging personal know-how about the AI’s behavior to coax the desired result. For example, when trying to generate a specialized algorithm or solve a domain-specific problem, a developer might instinctively shape their request to match what they know the AI handles well; perhaps by framing the problem in terms that are common in training data or avoiding ambiguous phrasing. They may not consciously think about it, but they draw on a kind of internal playbook developed through past interactions: what tends to work, what confuses the model, and how to steer it toward quality output. This back-and-forth of adjusting inputs, interpreting results, and refining expectations—reflects a nuanced, intuitive understanding that the developer might not even fully articulate. It’s a collaborative process that takes some experience by the engineer with working with the AI where the machine brings breadth of data and pattern recognition, while the human brings situational judgment and contextual insight. In this way, navigating AI systems becomes another expression of Polanyi’s Paradox: we rely on what we know but can’t entirely explain to get useful results from a machine that requires us to make that knowledge as explicit as possible.

Polanyi’s Paradox as a Cautionary Tale

Finally, Polanyi’s Paradox and the quest to encode (or exploit) tacit knowledge in AI bring to mind several cautionary tales and analogies. One is the legend of Frankenstein. In Mary Shelley’s novel, a scientist creates a sentient being but cannot control it or imbue it with the values and understanding needed to integrate into society. The creature, “detached from human values, brought chaos rather than order,” as one commentary puts it. This resonates with modern fears that AI systems, if created without sufficient foresight, might behave in ways misaligned with human intentions; not out of malice, but because we never explicitly told them the full story of right, wrong, and context. Polanyi’s Paradox suggests we might not even know how to fully tell them these things, since so much of our common sense and ethics is tacit.

Another example if you have watched enough Disney then you have seen Mickey Mouse as The Sorcerer’s Apprentice in Fantasia as a young apprentice who animates a broom to do his cleaning for him. Lacking the master’s wisdom, he cannot stop the broom as it relentlessly carries out his literal instructions, eventually flooding the place. This is a perfect illustration of the failure of specification. The apprentice knew just enough magic to issue a command, but not enough to specify boundaries or handle contingencies. In AI terms, he had no “off-switch” or nuance in his instructions. The story underscores the risk of viewing powerful AI tools as magical solutions without understanding their limits. If we deploy AI with the mindset of that apprentice and expect it to perfectly do our bidding when we ourselves haven’t nailed down exactly what we want. We shouldn’t be surprised when we get unintended results.

Already, we’ve seen real examples of this: an AI instructed to maximize user engagement on a platform might learn that spreading sensationalist content achieves the goal, to society’s detriment; or a chatbot given free rein to converse might start generating troubling outputs if the designers didn’t anticipate certain prompts. As one observer quipped, if you’re “not careful, like a dreaming Mickey, you too will suffer the consequences of unintended hallucinations” from AI. In plainer terms, the irony of Polanyi’s Paradox in AI is that we are building machines far more literal-minded than ourselves to handle tasks that we ourselves only understand implicitly.

Conclusion

Polanyi’s Paradox remains a guiding concept in understanding the limits of formal knowledge in both human and machine domains. Historically, it reminds us that even at the height of scientific rationalism, that "knowing" something is often inexpressible. In software engineering, it encourages the recognition that programming is not purely formal and that expertise can’t be entirely captured in textbooks and that engineers must adapt. And in AI, Polanyi’s Paradox is both a challenge and a driving force. A challenge because AI must grapple with tasks that we as humans struggle to fully describe, and a driving force behind the rise of machine learning to let computers learn the unspeakable parts of human skill.

As we integrate AI more into our lives, Polanyi’s insight also serves as a warning: we should be mindful of what we haven’t been able to tell our machines. Balancing explicit knowledge with tacit understanding, and combining human intuition with machine computation, will be an important element of buiding stronger AI. After all, paraphrasing Polanyi, we (and our AI) know more than we can tell.