Dive into the fascinating world of Physics-Informed Machine Learning (PIML) with Anton Rey, AI Lead Engineer of Cactai AI Lab at Cactus. This article provides a comprehensive introduction to PIML and its current state, highlighting some exciting scientific and industrial applications. Anton explores how the fusion of physical laws with advanced machine learning techniques is revolutionizing predictive modeling and problem-solving across various domains. We then relate PIML to various machine learning paradigms, addressing the unique challenges it faces. Finally, we look at the promising prospects of future PIML-based technologies and scientific machine learning, and see how it is setting new standards for accuracy, efficiency, and interpretability in the field of artificial intelligence.
When people think about the future of artificial intelligence (AI) and machine learning (ML), the concept that often comes to mind is Artificial General Intelligence (AGI) – an AI capable of performing any intellectual task that a human can do. This vision, popularized by science fiction and ongoing debates among tech visionaries, suggests a world where machines possess human-like consciousness and high cognitive abilities. The revolution brought about by large language models (LLMs) has accelerated our journey toward this future, demonstrating unprecedented capabilities in language understanding and generation. However, while AGI remains an elusive aspiration, another transformative breakthrough in AI is emerging with immediate and practical implications: physics-informed machine learning (PIML).
Imagine a humanoid robot attempting to create a new cake recipe without any prior cooking experience. Using whatever substances it has at its disposal, the robot would rely solely on a data-driven, experimental approach: mixing various amounts of ingredients and altering baking times, and even incorporating non-edible substances it finds nearby. A group of (rather unfortunate) human participants would then taste each result, providing feedback to help adjust and improve the recipe. This approach mirrors a purely data-driven machine learning method, optimizing the model by exploring a wide range of possibilities to find the best parameters.
Imagine instead that this robot is first trained on the basics of human taste: understanding which substances are edible, how different flavors interact, and how certain combinations affect texture and taste. For instance, it learns that chocolate pairs well with vanilla, a pinch of salt enhances sweetness, and specific ingredient combinations create distinct textures and flavors. This knowledge would streamline recipe adjustments, enabling the cook to make more informed decisions and create a delicious cake in fewer attempts.
Analogously, PIML constructs machine learning models by integrating physics through conservation principles and differential equations, ensuring that the models maintain physical consistency. PIML has already proven to be a valuable approach, complementing traditional numerical methods for certain problems by reducing time to solution, requiring less data, and enhancing result interpretability. This makes PIML a promising method for addressing the next level of complexity in physics simulations, such as those involving multiscale and multiphysics phenomena. Once mature, the implications of this discipline on society could be simply groundbreaking, as we will see next.
The current state of PIML
Popular applications
One paradigmatic example of a PIML application is climate and weather forecasting: by applying machine learning techniques, we can enhance traditional numerical methods that require significant computational resources, improving accuracy and reliability, even with limited or noisy data. This approach aims to blend complex atmospheric processes derived from data into existing mathematical models characterized by their multiphysics and often chaotic behavior, leading to more efficient and robust forecasting models.
The field is indeed starting to gain significant attention. Major industry players, alongside many research institutions, are investing heavily in this area. A popular example is Alphabet’s DeepMind. Their latest version of AlphaFold achieves state-of-the-art performance in inferring protein structures from amino acid chains, a problem of huge relevance for wide-ranging healthcare applications. It uses graph neural networks trained on a vast dataset while incorporating physical constraints based on energy minimization principles. This technology is also advancing into other groundbreaking related areas, such as molecular dynamics and computational biochemistry, with direct application in healthcare and sustainability. Additionally, similar techniques are being used to tackle problems of high relevance, such as developing plastic-eating enzymes, creating new malaria vaccines, and studying antibiotic resistance. Moreover, DeepMind has worked on ML-based control systems applied to fusion plasma physics: they created a closed-loop magnetic control system for fusion plasma stabilization, which could become a key technology for achieving sustainable fusion energy. Regarding materials science, they have applied PIML principles to create millions of new synthetic materials, some of which could be the foundation of future transformative technologies.
NVIDIA is also making significant investments in this technology. As a tech giant and hardware provider for AI computing, they also invest heavily in domain-specific AI software stacks operating on their hardware. For PIML, they have released the Modulus platform, which is currently being actively used by the research community to accelerate innovation on fields like extreme weather prediction and carbon capture technologies.
Model-based engineering
The industrial sector stands to benefit immensely from the adoption of PIML. One of the most profound impacts could be on engineered product development. Traditional model-based engineering processes can be time-consuming and costly, requiring multiple iterations and extensive testing. PIML approaches to enable the integration of machine learning tools into traditional engineering workflows regarding design, simulation, sensitivity analysis, predictive control and optimization, advocating for fully-differentiable model-based engineering design, where models can in principle be optimized end-to-end. This effectively tightens the interaction between experimental data, surrogate models, and product requirements, paving the way for the next generation of systems engineering, driven by end-to-end sensitivity analysis and the incorporation of machine learning into model-based engineering workflows. This means faster prototyping, improved design accuracy, and significant reductions in time to market, enabling companies to develop higher-quality products at lower costs while managing greater complexity in their products than ever before. Commercial software players like Collimator, Mathworks or Ansys are already offering this kind of PIML-based integration, enabling the next generation of Computer-Assisted Engineering tools.
Supervised, reinforcement, physics-informed, and domain-specific learning
Supervised-learning (SL) based on human-generated data has recently enabled Generative AI systems (GenAI) to excel at generative tasks requiring complex understanding of natural language, images, video, music. These systems are trained by imitation, excelling at interpolation, but they lack extrapolation capabilities. This is not necessarily a problem though: there is enough unexplored territory in the space of all possible movies, books, songs, pictures and drawings already spanned by all the man-made content created so far, that sheer interpolation within this training data will often yield novel and useful (even artistic and inspiring) content.
On the contrary, some Reinforcement Learning (RL) use cases have demonstrated superhuman performance in several domains by incorporating self-play instead of using pre-generated data. Deep RL systems employ reward measures to navigate through high-dimensional environment and action spaces, generating their own data in the process. They have the potential to explore previously unseen areas in the underlying optimization landscape, showcasing novelty and creativity that can surpass human domain experts. However, RL approaches are often time-consuming due to their sequential processing nature, their delayed reward characteristic, and the typically vast multidimensional landscape involved.
Embedding a domain-specific model in any form (rules of a game, geometric constraints, physical laws, etc) into the model (SL or RL), acts as a regularizer, helping the system learn specific patterns over noise, typically reducing the optimization landscape to a more manageable size. Indeed, at first sight, PIML can be viewed as just another way of integrating domain-specific knowledge into ML models: since the dawn of ML, practitioners pondered model design laying somewhere in a spectrum spanned by two extremes – (1) one learned purely data-driven end-to-end, with no domain-specific knowledge and (2) the other where the model is fully parameterized with domain-specific non-learnable hardcoded parameters. The correct blend of these two extremes is far from straightforward. Favoring the first approach increases the risk of learning noise instead of useful patterns, potentially requiring prohibitive amounts of data. Leaning into the second increases the risk of incorporating too many biases that could spoil the capacity of the model to learn from actual data, thus compromising its actual usefulness. This decision is strongly dependent on the problem to be tackled, and it is no different for PIML: how to embed physics in the modeling phases, including data curation, model architecture, loss/reward functions, and optimization procedures, is where the current research is focusing. This is typically a time-consuming iterative process for which there is no general systematic solution, as every physics application poses its own set of challenges.
The potential of PIML
There are big technical challenges, but the potential for successfully integrating ML with the physics domain is enormous, considering the following observations:
- As the most foundational empirical scientific discipline, established physical laws are barely subject to human bias: they are the result of millennia of evidence-based scientific endeavour.
- Physical and mathematical models underpin modern society, deeply rooted in science and technology. In particular, established physics theories are the root of core technologies related to computing, communications, energy, materials, healthcare, transportation, manufacturing… Also, physics-inspired mathematical models (although more subject to human bias than physics-rooted ones) are often conceptual cornerstones in other non-physical scientific disciplines such as biology, finance, economics, and sociology.
- The advancements in science and technology are key to solving some of the most pressing problems facing humanity, especially those related to pollution, healthcare, sustainability, climate change, or energy and water scarcity.
- The advancements in science and technology are key to bringing humanity to the next level: regarding life extension, life augmentation, and making life interplanetary, the limit is simply in our imagination.
Automation possibilities in model-based engineering
Looking at the development of AlphaGo, a Go-playing program by DeepMind, we can see a clear two-stage progression. In the first stage, AlphaGo learned by imitating human expert players: analyzing numerous games played by top human players, the neural network was trained to mimic their strategies. This approach produced a competent Go-playing program, but it was limited to the level of the best human players used for training.
After that first version, DeepMind then devised a method for AlphaGo to surpass human capabilities through self-play, as just mentioned. In the controlled environment of the Go game, with a straightforward reward function—winning the game—AlphaGo could play millions of games, practically without human intervention, and refine its strategies based on the outcomes. This allowed AlphaGo to exceed human performance, achieving remarkable success in 40 days.
Thinking back to model-based engineering, we could think of creating ML models fed with big amounts of data that collect zillions of decisions from experienced engineers (similarly to what happens in software assistants and LLMs trained with big amounts of source code). However, PIML approaches applied to model-based engineering suggest a more elegant and appealing approach, in which practically no human decisions from engineers would be needed for the training. Specifically, PIML principles could help create future reinforcement-learning schemas operating on fully-differentiable, data-realistic and physics-respecting surrogate models, where every parameter of the model, in any stage, is subject to optimization based on their sensitivity relations with the optimization parameters. This effectively opens the door for unprecedented levels of automation in all stages of model-based engineering projects.
Scientific Machine Learning
PIML holds significant potential in discovering new governing
laws across various domains. PIML can uncover new partial differential equations (PDEs)
that describe non-linear behavior, identify non-linear dynamics and interactions, reveal new conservation laws or symmetry principles, or find new internal representations, governing equations and coordinate systems in which a difficult problem becomes tractable. Additionally, PIML is adept at discovering governing equations for anomalous transport phenomena, multi-scale interactions, and emergent behaviors in systems with many interacting components, potentially uncovering the principles of phase transitions and non-equilibrium dynamics, and identifying the laws governing stochastic processes and chaotic systems. This goes beyond problems with clear physical semantics; PIML also holds great potential for discovering new physics-inspired mathematical models in fields beyond physics. This includes systems characterized by their sheer complexity: for the health-care domain, we could aim to decode the brain, the immune system, or cancer; for socioeconomics, we could address the complexity underlying social inequality, wealth distribution, logistics, and optimal resource allocation problems at unprecedented scales.
This opens up a broader Scientific Machine Learning (SciML) perspective, related to what some call the fourth paradigm of scientific discovery, a blend of the previous three: [1] the empirical (direct observation and experimentation), [2] theoretical (development of models and theories), and [3] computational (use of simulations on computer models with decreased complexity). Indeed, SciML embraces computing (by simulation, statistical analysis and optimization) to discover new theories (governing equations or surrogate models) from empirical data (real noisy measurements and synthetic data from models), in tight iterative loops subject to automation.
AGI and beyond
Going back to intelligence, insights from PIML could actually be crucial in improving the current state of the art LLMs and even helping to realize AGI systems. While LLMs have demonstrated impressive understanding capabilities from sheer statistical patterns in natural language data, these patterns alone are unlikely to be sufficient for reaching human-level intelligence, especially if no accurate physical model of the world was used in their training. Human-level cognition relies not only on recognizing patterns but also on understanding and interacting with the physical world, making predictions and planning – something that acquires even more relevance in robotics and embedded AI systems. This will likely be realized thanks to the construction of robust physical models, partly data-driven and partly grounded in natural phenomena anchored in scientific facts. Insights from emerging PIML and SciML approaches will likely bridge this gap by incorporating this scientific knowledge into conversational and human-facing machine learning models, allowing them to develop a deeper and more scientifically-grounded understanding of the world, which will help fight misinformation and clear the path to AGI and beyond.
Our current plans in Cactus
In Cactus, we’re exploring PIML to work with partners on inverse problems related to partial differential equations modeling wave-like phenomena. This includes applications like cancer diagnosis using ultrasound in medical imaging, and model reconstruction and sensing in geophysics and electromagnetism with seismic and electromagnetic waves. We are also open to partnerships to collaborate on other ambitious PIML-related projects.
Conclusion
I expect that PIML/SciML disciples are set to trigger the next industrial and scientific revolutions by fostering unprecedented levels of innovation and productivity. This revolution has actually started, but it is rather silent: it is just catching momentum, and most people are not aware of it, contrary to the case of the recent LLM revolution where everyone could directly interact with chatbots. The field has already made significant advancements in fields like computational biochemistry, fusion plasmas, and climate modeling among many others. It will likely continue by permeating society in terms of unprecedented access to new technologies, cheaper goods, energy, services and experiences, potentially having a key role in the resolution of humanity’s most pressing problems.
As we stand on the brink of this new era, it is crucial to invest in and prioritize further research in PIML-based and SciML technologies, navigating through hype cycles and real opportunities, and properly coordinating between different actors –academia, industry, funding institutions, and governments– in order to avoid wasting time and spending the limited resources wisely.
