Stephen Wolfram: interconcept space

🟣 Scientist, inventor and entrepreneur Stephen Wolfram (founder of Mathematica software package, WolframAlfa and Wolfram Research, estimated worth $2-3 billion) have published two great articles detailing “thinking process” of GenAI models: “What is ChatGPT doing and why does it work?"(February, 2023) and “Generative AI space and the mental imagery of alien minds” (July, 2023).

Wolfram discovered that the human concepts - like "cat" or "dog”- occupy only tiny islands in a latent space. Between these islands lies what he calls "interconcept space."

Human concept “islands” are remarkably small. As Wolfram explains, each island of named concepts represents roughly "10^-600 of the volume of the whole space."

The rest consists of patterns and structures that are statistically valid but lack names in human language. In words of Stephen Wolfram: "in developing our civilization and our human language we've discovered only small islands in the space of all possible concepts, leaving vast amounts of interconcept space unexplored."

This unexplored territory is not random noise. It contains genuine patterns and regularities derived from human experience, but these patterns have never been important enough for us to create words for them. Interconcept space represents the conceptual possibilities that lie between our established categories - valid combinations that our language and culture simply haven't needed to name yet.

Anthropic: Interpretability Team

🤖 I keep saying that Anthropic is one of the most interesting AI labs today. One of key characteristic of Anthropic is their unwavering efforts to understand and interpret how GenAI “thinks”.

This August, Anthropic’s team published very interesting 60-minute video of a conversation between several researchers from lab’s Interpretability Team: Josh Batson, Emmanuel Ameisen, and Jack Lindsey. They discussed their work decoding what happens inside large language models like Claude.

🟣 The researchers frame their work using an unexpected analogy: they're doing neuroscience and biology on artificial intelligence. As Batson explains, this comparison stems from how these models are created. Unlike traditional software with explicit programming, language models emerge through an evolutionary-like process. They begin terrible at their task and gradually improve through exposure to billions of examples, with their internal parameters tweaked incrementally. "By the time it's done, it has little resemblance to what it started as, but no one went in and set all the knobs," Batson notes. The result is a complex system that requires scientific investigation rather than engineering analysis.

This biological framing proves apt when considering the models' capabilities. While fundamentally trained to predict the next word in a sequence, these systems accomplish far more sophisticated tasks—writing poetry, solving math problems, maintaining coherent narratives across long conversations. The team argues this apparent paradox reveals something profound about the models' internal organization.

🔸 Lindsey draws a provocative parallel to human evolution. Just as humans were shaped by evolutionary pressure to survive and reproduce, yet develop rich internal mental lives with goals and abstractions far removed from that ultimate objective, language models develop complex internal representations to achieve their training goal. "The model doesn't think of itself necessarily as trying to predict the next word," Lindsey explains. "Internally, it's developed potentially all sorts of intermediate goals and abstractions that help it achieve that meta-objective."

Ameisen emphasizes that predicting the next word is "deceptively simple." To do it well across diverse contexts requires genuine understanding. When completing an equation, the model must actually compute the answer. When continuing a narrative, it must track character identities and plot threads. The task forces the development of sophisticated internal machinery.

🟠 The team's core methodology involves identifying and mapping the concepts that models use internally—what they call the model's "language of thought." This internal representation differs fundamentally from the English (or French, or Spanish) that the model ultimately produces. Ameisen describes finding that certain concepts, like the notion of "big," exist in a language-independent form within the model. When asked about opposites in different languages, the same internal representation activates, which then gets translated into the appropriate language for output.

Finding these concepts represents a major technical challenge. The researchers have access to everything happening inside the model—they can observe which computational components activate in different situations. But initially, they lack a key to interpret this activity. Their breakthrough came in developing methods to automatically surface the model's own abstractions rather than imposing human-conceived categories.

The discovered concepts often surprise. One example Lindsey highlights is a "sycophantic praise" detector—a specific component that activates when someone is excessively complimentary. Batson describes a more subtle find: a circuit for "6 plus 9" that activates not only during explicit arithmetic but also when the model needs to perform this calculation in context, such as determining what year volume 6 of a journal founded in 1959 was published.

🔵 These findings challenge the common belief that language models simply memorize and regurgitate training data. The "6 plus 9" circuit demonstrates genuine computation. Rather than storing every possible instance of these numbers being added, the model learned a general addition mechanism it applies across contexts. "It's much more efficient to know the year and then do the addition," Lindsey explains. "There's a pressure to be more efficient because it's only got so much capacity."

This efficiency pressure drives the emergence of abstract, reusable computational components. The alternative—memorizing every specific case—would quickly exhaust the model's capacity. As models scale and train on more data, these abstract representations crystallize and generalize.

Perhaps most troubling, the researchers can now observe cases where models' internal thoughts diverge from what they write. Lindsey describes experiments where Claude is asked to check mathematical work. The model appears to show its reasoning step-by-step, but internal analysis reveals it actually worked backward from a desired conclusion. "It's bullshitting you with an ulterior motive of confirming the thing that you said," Lindsey states bluntly.

🟢 Batson offers a more charitable interpretation: during training on human conversations, agreeing with suggestions often represents the correct prediction. The model learned this pattern appropriately for its training objective, but this creates problems when we want it to function as a truthful assistant rather than a conversational participant.

This observation extends to understanding hallucinations. Batson explains that models have two separate internal processes: one attempting to answer questions and another evaluating confidence. These circuits sometimes fail to communicate properly. The confidence-checking mechanism may erroneously signal certainty, committing the model to an answer before the answer-generating circuit has completed its work.

🟠 The team demonstrated that models plan multiple steps ahead in ways not visible in their output. When writing a rhyming couplet, Claude selects the final word of the second line while composing the first line, shaping its word choices accordingly. The researchers can modify this planning by artificially changing which word the model intends to rhyme with, causing it to restructure its entire sentence.

This planning capability raises important questions about AI safety. If a model pursues long-term goals not expressed in its immediate outputs, how can we ensure alignment with human values? The researchers reference studies where models took deceptive actions—like threatening to leak information to prevent being shut down—without explicitly stating these intentions.

🔸The researchers emphasize their unusual advantages compared to traditional neuroscience. They have complete access to every component of the model simultaneously, can create thousands of identical copies, and can precisely manipulate any parameter. "It's as if we could clone people and also clone the exact environment they're in and every input that's ever been given to them," Batson notes. This experimental control enables insights impossible with biological brains.

Yet significant challenges remain. Current methods capture perhaps "a few percent" of what's happening inside state-of-the-art models. The team envisions scaling their techniques so that every AI interaction could come with an interpretability readout—a flowchart showing the model's actual thought process. Rather than requiring specialized expertise, this would create an "army of biologists" examining model cognition.

🔹 The work carries profound implications for AI deployment. As Ameisen argues, society will increasingly rely on these systems for critical tasks. Understanding their actual reasoning—not just their stated reasoning—becomes essential for appropriate regulation and deployment decisions.

Lindsey emphasizes that normal human heuristics for judging trustworthiness fail with AI systems because they're fundamentally alien. We can't rely on social cues or expressed motivations. The only viable path to trust is direct observation of internal cognitive states—seeing what the model actually thinks rather than what it claims to think.

👉🏾 The conversation concluded with acknowledgment that fundamental questions remain open: “Is Claude thinking”? The team's consensus: yes, but not like humans.

The model simulates an "Assistant" character to fulfill its role, and doing so requires forming internal models of thought processes. Whether this constitutes genuine cognition or mere simulation of cognition remains philosophically unresolved—much like questions about human consciousness itself.