Technological change typically occurs in three phases: basic research, scale-up, and industrial application, each with a different degree of methodological diversity?high, low, and medium, respectively. Historically, breakthroughs such as the steam engine and the Haber-Bosch process exemplify these phases and have had a profound impact on society. A similar pattern can be observed in the development of modern artificial intelligence (AI).
In the scale-up phase of AI, large language models (LLMs) have emerged as the most prominent example. While LLMs can be seen as highly sophisticated knowledge representation techniques, they have not fundamentally advanced AI itself. The upscaling phase of AI was dominated by the transformer architecture. More recently, other architectures, such as state-space models and recurrent neural networks, have also been scaled up. For example, Long Short-Term Memory (LSTM) networks have been scaled up to xLSTM, which in many cases outperform transformers.
We are now transitioning into the third phase: industrial AI. In this phase, we are adapting AI methods to real-world applications in robotics, life and earth sciences, engineering, or large-scale simulations that can be dramatically accelerated by AI methods. As we continue to develop these industrial AI methods, we expect to see an increase in methodological diversity, allowing us to overcome what has been called the "bitter lesson" of scaling up.
Forschungs-