A conversation between Ori Goshen, Co-founder and Co-CEO of AI21 Labs and Barr Yaron, Partner at Amplify Partners.

The vision: deep learning is necessary but not sufficient

"We started the company back in 2017 with the idea that deep learning is necessary, but not sufficient. This statement is actually more relevant today than when we started. We were seeing how powerful deep learning systems are, and obviously we've seen the progress in the last few years with large language models and now large reasoning models. It's very impressive.

But with that said, there are certain problems that are more deterministic by nature and require a more predictable approach. Not everything needs to be computed by probabilities or probabilistic methods. Some things need to be addressed deterministically.

That was the reason why we started the company. We said these deep learning neural net-based systems are amazing and they're not going to go away – on the contrary, they're going to thrive. But there needs to be a complementary approach to solving complex problems. The premise from the get-go was to create reliable AI systems, systems we can actually trust. In many ways, we're just starting to realize that vision today, and that's what makes this era so exciting."

The initial product journey: from B2C to B2B

"When we started, we were one of the first companies in the world to train language models. We haven't called them large language models back then. We tried to find applicable markets and looked at the enterprise, but it wasn't a clear fit.

Back then, we saw that these relatively small language models were quite effective in helping people better read and write. As we understood there was no enterprise market yet, we decided to create our own market. We built and shipped an application called WordTune, which was the first AI-powered reading and writing assistant. Initially, it rewrote and rephrased text as you write and summarized lengthy articles so people could consume content much faster. The application received several tens of millions of users and a large base of paying customers.

Having said that, our original mission was to create reliable AI systems. When ChatGPT came out, it was an eye-opening moment for the entire market, including the enterprise market. That's when we realized we needed to shift our focus back to our original mission.

Back in 2021, we released our first large language model called Jurassic One. It was slightly bigger, slightly better than GPT-3 in many aspects. That's when we started pivoting towards the B2B space when we sensed these models were capable enough to create a lot of value in the context of an enterprise."

Product-algo fit: finding the right model architecture

"At some point we realized the model is a product, and we wanted to think about it like that. On one hand, researchers and engineers need the freedom to innovate and think about things from a first-principle point of view, so they can come up with new architectures or have the foresight to understand where things are going. On the other hand, they need clear goals that can be manifested through product requirements – what benchmarks and skills the model should have.

When we started the Jamba project, our own foundation model, we wanted to focus it towards the enterprise. We thought about the types of use cases enterprises have, and it was quite obvious that most use cases are retrieval-based – you need to retrieve a lot of context into the model. So we wanted to design a model that's best in that setting, that can deal with a lot of data in the input, like a very large context window. We also wanted to bake in skills that are more common within enterprises.

Most models out there are based on the transformer architecture, which has quadratic complexity at inference time – meaning the more you scale the context window, the more computationally expensive it is. That's not very scalable. If you have 2,000 tokens in the context window, that's fine. But if you have 200,000, that's a different story.

When the Mamba paper was published, our researchers looked into it and we experimented with this architecture ourselves. The Mamba architecture has linear complexity at inference time, so the more you scale the context window, you pay more of course, but not as much as with transformer-based architecture.

Our researchers found that for a variety of tasks, not only can you enjoy the efficiencies of Mamba, but by interleaving transformer layers in a certain setting, you also get better quality. Our folks found a neat combination between Mamba and transformer where you can enjoy the best of both worlds – the quality of the transformer architecture and the efficiency of the Mamba architecture."

Bringing AI to enterprise: the reliability challenge

"Enterprises haven't really scratched the surface in terms of adoption of these models in production. If I need to describe it, I would say 2023 post-ChatGPT was the year of sporadic experimentation, 2024 was massive experimentation, and 2025 is where we're going to production.

A typical enterprise might have maybe tens of use cases in production, but there are thousands in the backlog. The main reason is reliability. Having something deeply rooted inside critical workflows needs a lot of trust in these systems, and we're not there yet. Even the use cases that are heavily engineered and get to 85 or 90% accuracy, in some cases it's just not good enough.

I think the most challenging part of each project is the evals – how do you define what success looks like? You need to find a systematic way to build a robust eval that actually represents the problem. Sometimes people make evals, but they're very skewed and don't really represent the natural distribution of the problem they're trying to solve. So you end up optimizing for the wrong thing.

The second challenge is how to architect the systems. This is non-traditional, non-deterministic software, so you need to think about how to put the right guardrails in place and make sure the system doesn't derail."

Maestro: AI for AI systems

"Maestro is a platform we released recently – an AI planning and orchestration system designed for complex tasks. It's a platform for developers in the enterprise to build agentic workflows.

The problem we're trying to solve with Maestro is to reach accuracy at scale. How do you get the level of reliability that will allow an enterprise to go from tens of use cases in production to thousands? And how do you democratize this within the organization so every engineer could build an agentic workflow that's ready to deploy in production?

Currently, developers either 'prompt and pray' – they take an agentic framework and an LLM as the controller, specify in the prompt, optimize things, and hope for the best – or they go the other direction and build static chains, which are like hard-coded programs. The first approach isn't reliable enough, while the second is too rigid and hard to scale.

We thought there needs to be AI for AI systems – an engine that can get all the models, tools, and data sources of the enterprise and, given a task described in natural language, create an explicit logical plan of how to solve that task, along with an operational plan with all the design choices like which models to use and how much to validate. This abstracts away complexities for developers while giving them control and reliability."

The future: super productivity

"We've spent a lot of time thinking about the future in the company. There are terms like artificial general intelligence and superintelligence, but we like to think about a different term called 'super productivity.' Once we have these systems deeply integrated, it will go beyond just automating work and gaining efficiencies.

We think about what organizations would aspire to accomplish given an era where you can operationalize your ideas fairly cheaply. A lot of the strategic thinking that AI is not deeply involved in today would change. A manager who needs to make a decision could have a system with rich context that can run different scenarios.

These agentic workflows will become the enterprise itself – the operating system of the enterprise. Five years sounds like a lot, but I think we're going to see drastic changes by then."