There is a kind of knowledge that is not understanding. You know how to get a loan approved without knowing how the approval algorithm works. You know which posts to publish to reach your audience without knowing how the distribution system ranks them. You know how to keep your account in good standing without knowing what, precisely, constitutes a violation.
This is the dominant cognitive mode of contemporary life — a quiet, practical knowledge of how to move within systems whose actual logic remains opaque.
We did not arrive here by accident or through negligence. The systems became opaque because they became capable. The recommendation algorithm that a human editor could explain has lower ceilings than the one trained on billions of behavioral signals that no individual can hold in mind.
The loan approval model that follows a legible checklist makes worse predictions than the one whose decision process cannot be rendered in plain language. We traded comprehensibility for performance, and we made that trade so incrementally that most people did not notice it happening.
The systems are not hiding their logic from us. They simply no longer have logic in a form that language can carry.
What this produces is a particular kind of relationship between people and institutions. The institution — the bank, the platform, the employer, the algorithm — makes consequential decisions about your life by processes you cannot access.
You can appeal, in some cases. But appeal requires a theory of the error, and theories of the error require understanding the process, and the process is not available to you. The bureaucratic opacity of the twentieth century has been replaced by something more fundamental. It is not that the logic is hidden. It is that the logic does not exist in a form that can be communicated.
People adapt to this, as people adapt to most things. The adaptation takes the form of folk knowledge — informal, socially transmitted heuristics about how to behave within systems that cannot be directly understood. Use these words in your application. Avoid these topics on the platform. Post at these times.
This folk knowledge is real and often effective, which is part of what makes the situation stable. People are not helpless within opaque systems. They are just operating on inference and imitation rather than comprehension.
The concern is what this does over time to the relationship between citizens and the systems that govern their lives. Democratic accountability rests on a chain that runs from policy to decision to visible outcome to public judgment. That chain requires that the decisions be, at some level, legible.
When decisions are made by processes that cannot be rendered legible — not won't be, but genuinely cannot be — that chain breaks. Not through malice. Through complexity.
We built these systems. We continue to build more of them. The case for doing so is real — they are often better, by measurable criteria, than the comprehensible alternatives.
But "better" is always relative to a specified objective, and the objectives we specified did not include "remains accountable to the people it affects." That omission is now structural. Correcting it will require deciding, as a matter of deliberate choice, what we are willing to give up in performance terms to get comprehensibility back. We have not yet had that conversation seriously.