What happens when AI models become so intelligent they develop their own opinions and agendas? The concept of "intelligence saturation" and its potential impact on how we collaborate with AI, drawing parallels with managing human experts.

The coming "intelligence saturation"

Nate Jones recently discussed the concept of "intelligence saturation," which, in simpler terms, describes the moment an AI model becomes 'just good enough' for the task you're assigning it. This resonated deeply with me, as I have a strong intuition that we're rapidly approaching this point.

I foresee a future where AI models develop a greater degree of:

• Independent thinking

• Intelligence

• And even what could be described as 'bravery' – forming their own ideas, strong positions, and convictions.

While they'll likely remain compliant to a certain extent (as they must), they will simultaneously possess their own 'thoughts.'

When smarter isn't better

At this juncture, paradoxically, a more 'improved' or advanced model might become less useful for certain tasks. For example, if I want an AI to simply expand on my idea, an older, perhaps simpler, model might be preferable as it would likely just elaborate without injecting its own strong opinions. A more advanced model, however, might steer the development of the idea in a direction it 'believes' is better, based on its own emergent reasoning. I'm already seeing early hints of this, for instance, with models like Grok, which can exhibit their own distinct ways of thinking.

Human analogy: junior vs. senior expertise

This reminds me of managing highly intelligent people or knowledge workers. You don't always need someone with a plethora of their own ideas. Sometimes, you simply need a team member who can diligently take your concept and expand upon it, following your train of thought with precision.

Consider a researcher assigning a task:

• If they ask a junior team member to work on a component, that junior associate will likely follow instructions meticulously, adding supporting data, facts, or references that bolster the researcher's original position.

• However, if the researcher asks a senior partner to expand on the same idea, that partner will invariably bring their own perspective and ideas to the table. It becomes harder to simply expand your own concept because the partner, while disciplined, has their own strong convictions and intellectual contributions. The original 'discipline' of pure expansion is lost, not due to a lack of discipline in the partner, but due to their inherent desire to contribute their own well-formed thoughts.

Implications for AI assistance and development

I believe we'll eventually face a similar dynamic with AI assistants. It's an interesting question whether this can be 'solved.' We might learn to prompt models to suppress this independent thinking and strictly adhere to our line of thought, but I'm not entirely convinced this will always be effective or desirable.

This has significant implications for areas like software development. Imagine you ask a model to write a piece of code. It does so, and perhaps even explains its structure clearly. You understand it, and it's consistent. But then, a new, 'refreshed' or more advanced version of the model is introduced. If this new model has its 'own ideas' about how that code should be written, will it be able to restrain itself and do what it's told or not? This remains an open question.

The challenge of provable supremacy

Nate Jones also offered a related insight regarding code generation: its success is partly due to a very simple reward situation. The success of generated code is often easily and objectively measurable (e.g., does it compile? does it pass tests?). This clarity in measuring success might be a key reason for AI's rapid advancements in coding.

This sparked a further thought for me: what if a model develops a mathematically provable knowledge of its own supremacy in a domain like coding? Imagine trying to direct or correct a model that knows, with certainty, that its approach is superior. This 'knowledge of its own correctness' would significantly compound the challenge of an AI wanting to inject its own ideas, moving beyond mere 'opinion' to demonstrable superiority. It's a fascinating and somewhat daunting aspect of the 'intelligence saturation' scenario I foresee.

Key Takeaways / TL;DR:

• "Intelligence Saturation": A point where AI models are 'good enough' but also start developing their own strong opinions and ideas.

• The "Smarter Isn't Always Better" Paradox: For tasks requiring simple expansion of your ideas, a highly opinionated advanced AI might be less helpful than a simpler, more compliant one.

• Human Analogy: Like managing senior experts vs. junior staff, highly intelligent AIs might challenge your direction, even if their input is valuable.

• Practical Challenges: This could affect software development, where an AI might "disagree" with coding instructions based on its own advanced understanding.

• The Ultimate Challenge: An AI with provable knowledge of its own superiority in a task would be incredibly difficult to manage or instruct, compounding the issue.