Digitala Vetenskapliga Arkivet

Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Scientific Machine Learning for Forward and Inverse Problems: Physics-Informed Neural Networks and Machine Learning Algorithms with Applications to Dynamical Systems
KTH, School of Electrical Engineering and Computer Science (EECS), Computer Science, Computational Science and Technology (CST).ORCID iD: 0000-0003-4132-3175
2025 (English)Doctoral thesis, comprehensive summary (Other academic)
Abstract [en]

Scientific Machine Learning (SciML) is a promising field that combines data-driven models with physical laws and principles. A novel example is the application of Artificial Neural Networks (ANNs) to solve Ordinary Differential Equations (ODEs) and Partial Differential Equations (PDEs). One of the most recent approaches in this area is Physics-Informed Neural Networks (PINNs), which encode the governing physical equations directly into the neural network architecture. PINNs can solve both forward and inverse problems, learning the solution to differential equations and inferring unknown parameters or even functional forms. Therefore, they are particularly effective when partially known equations or incomplete models describe real-world systems. 

Differential equations enable a mathematical formulation for various fundamental physical laws. ODEs and PDEs are used to model the behavior of complex and dynamical systems in many fields of science. However, many real-world problems are either too complex to solve exactly or involve equations that are not fully known. In these cases, we rely on numerical methods to approximate solutions. While these methods can be very accurate, they often are computationally expensive, especially for large, nonlinear, or high-dimensional problems. Therefore, exploring alternative approaches like SciML to find more efficient and scalable solutions is fundamental.

This thesis presents a series of applications of SciML methods in identifying and solving real-world systems. First, we demonstrate using PINNs combined with symbolic regression to recover governing equations from sparse observational data, focusing on cellulose degradation within power transformers. PINNs are then applied to solve forward problems, specifically the 1D and 2D heat diffusion equations, which model thermal distribution in transformers. Moreover, we also develop an approach for optimal sensor placement using PINNs that improves data collection efficiency. A third case study examines how dimensionality reduction techniques, such as Principal Component Analysis (PCA), can be applied to explain and visualize high-dimensional data, where each observation comprises a large number of variables that describe physical systems. Using datasets on Cellulose Nanofibrils (CNFs) of various materials and concentrations, Machine Learning (ML) techniques are employed to characterize and interpret the system behavior. 

The second part of this thesis focuses on improving the scalability and robustness of PINNs. We propose a pretraining strategy that optimizes the initial weights, reducing stochasticity variability to address training instability and high computational costs in higher-dimensional problems arising from solving multi-dimensional or parametric PDEs. Moreover, we introduce an extension of PINNs, referred to as $PINN, which includes Bayesian probability within a domain decomposition framework. This formulation enhances performance, particularly in handling noisy data and multi-scale problems.

Abstract [sv]

Scientific Machine Learning (SciML) är ett lovande område som kombinerar datadrivna modeller med fysiska lagar och principer. Ett nytt exempel är tillämpningen av artificiella neurala nätverk (ANN) för att lösa ordinära differentialekvationer (ODE) och partiella differentialekvationer (PDE). Ett av de senaste tillvägagångssätten inom detta område är PINN (Physics-Informed Neural Networks), som kodar de styrande fysikaliska ekvationerna direkt i neuronnätets arkitektur. PINN kan lösa både framåtriktade och inversa problem, lära sig lösningen på differentialekvationer och härleda okända parametrar eller till och med funktions former. Därför är de särskilt effektiva när delvis kända ekvationer eller ofullständiga modeller beskriver verkliga system.

Differentialekvationer möjliggör en matematisk formulering av grundläggande fysiska lagar. ODE:er och PDE:er används för att modellera beteendet hos komplexa och dynamiska system inom många vetenskapliga områden. I verkligheten är många problem antingen för komplexa för att kunna lösas exakt eller innehåller ekvationer som inte är helt kända. I dessa fall förlitar vi oss på numeriska metoder för att approximera lösningar. Trots att dessa metoder kan vara mycket exakta är de ofta beräkningsmässigt dyra, särskilt för stora, olinjära eller mångdimensionella problem. Det är därför viktigt att utforska alternativa metoder som SciML för att hitta effektivare och mer skalbara lösningar.

I denna avhandling presenteras en serie tillämpningar av SciML-metoder för att identifiera och lösa verkliga system. Först demonstrerar vi hur PINN kombinerat med symbolisk regression kan användas för att återskapa styrande ekvationer från gles observationsdata, med fokus på cellulosanedbrytning i krafttransformatorer. PINNs används sedan för att lösa framåtriktade problem, särskilt 1D- och 2D-värmediffusionsekvationerna, som modellerar termisk distribution i transformatorer. Dessutom utvecklar vi ett tillvägagångssätt för optimal sensorplacering med hjälp av PINN som förbättrar datainsamlingseffektiviteten. I ett tredje användnings område undersöks hur tekniker för dimensionsreduktion, såsom Principal Component Analysis (PCA), kan tillämpas för att förklara och visualisera mångdimensionell data, där varje observation består av ett stort antal variabler som beskriver fysiska system. Med hjälp av dataset om cellulosa nanofibrillärer CNF) av olika material och koncentrationer används maskininlärningstekniker (ML) för att karakterisera och tolka systemets beteende.

Den andra delen av avhandlingen fokuserar på att förbättra skalbarheten och robustheten hos PINN. Vi föreslår en strategi för förträning som optimerar de initiala vikterna, vilket minskar stokasticitetsvariabiliteten för att hantera träningsinstabilitet och höga beräkningskostnader i problem med fler dimensioner som uppstår vid lösning av mångdimensionella eller parametriska PDE:er. Dessutom introducerar vi en tillägg för PINN, kallad $PINN, som inkluderar Bayesiansk sannolikhet inom ett ramverk för domänkomposition. Denna formulering förbättrar prestandan, särskilt vid hantering av brusiga data och flerskaliga problem.

Place, publisher, year, edition, pages
Stockholm: KTH Royal Institute of Technology, 2025. , p. xii, 124
Series
TRITA-EECS-AVL ; 2025:44
Keywords [en]
Scientific Machine Learning, Physics-Informed Neural Networks, System Identification, Inverse Problem, Forward Problem
National Category
Computer Sciences
Research subject
Computer Science
Identifiers
URN: urn:nbn:se:kth:diva-363009ISBN: 978-91-8106-263-2 (print)OAI: oai:DiVA.org:kth-363009DiVA, id: diva2:1955902
Public defence
2025-05-26, https://kth-se.zoom.us/j/66482272586, Kollegiesalen, Brinellvägen 6, Stockholm, 13:00 (English)
Opponent
Supervisors
Funder
Vinnova, 2021-03748
Note

QC 20250505

Available from: 2025-05-05 Created: 2025-05-02 Last updated: 2025-05-05Bibliographically approved
List of papers
1. Discovering Partially Known Ordinary Differential Equations: a Case Study on the Chemical Kinetics of Cellulose Degradation
Open this publication in new window or tab >>Discovering Partially Known Ordinary Differential Equations: a Case Study on the Chemical Kinetics of Cellulose Degradation
Show others...
(English)Manuscript (preprint) (Other academic)
Abstract [en]

The degree of polymerization (DP) is one of the methods for estimating the aging of polymer-based insulation systems, such as cellulose insulation in power components. The main degradation mechanisms in polymers are hydrolysis, pyrolysis, and oxidation. These mechanisms combined cause a reduction of the DP. However, the data availability for these types of problems is usually scarce. This study analyzes insulation aging using cellulose degradation data from power transformers. The aging problem for the cellulose immersed in mineral oil inside power transformers is modeled with ordinary differential equations (ODEs). We recover the governing equations of the degradation system using Physics-Informed Neural Networks (PINNs) and symbolic regression. We apply PINNs to discover the Arrhenius equation's unknown parameters in the Ekenstam ODE describing cellulose contamination content and the material aging process related to temperature for synthetic data and real DP values. A modification of the Ekenstam ODE is given by Emsley's system of ODEs, where the rate constant expressed by the Arrhenius equation decreases in time with the new formulation. We use PINNs and symbolic regression to recover the functional form of one of the ODEs of the system and to identify an unknown parameter.

National Category
Artificial Intelligence
Identifiers
urn:nbn:se:kth:diva-362870 (URN)
Funder
Vinnova, 2021-03748Vinnova, 2023-00241
Note

QC 20250430

Available from: 2025-04-28 Created: 2025-04-28 Last updated: 2025-05-02Bibliographically approved
2. Physics-informed neural networks for modelling power transformer’s dynamic thermal behaviour
Open this publication in new window or tab >>Physics-informed neural networks for modelling power transformer’s dynamic thermal behaviour
Show others...
2022 (English)In: Electric power systems research, ISSN 0378-7796, E-ISSN 1873-2046, Vol. 211, p. 108447-108447, article id 108447Article in journal (Refereed) Published
Abstract [en]

This paper focuses on the thermal modelling of power transformers using physics-informed neural networks (PINNs). PINNs are neural networks trained to consider the physical laws provided by the general nonlinear partial differential equations (PDEs). The PDE considered for the study of power transformer’s thermal behaviour is the heat diffusion equation provided with boundary conditions given by the ambient temperature at the bottom and the top-oil temperature at the top. The model is one dimensional along the transformer height. The top-oil temperature and the transformer’s temperature distribution are estimated using field measurements of ambient temperature, top-oil temperature and the load factor. The measurements from a real transformer provide more realistic solution, but also an additional challenge. The Finite Volume Method (FVM) is used to calculate the solution of the equation and further to benchmark the predictions obtained by PINNs. The results obtained by PINNs for estimating the top-oil temperature and the transformer’s thermal distribution show high accuracy and almost exactly mimic FVM solution.

Place, publisher, year, edition, pages
Elsevier, 2022
Keywords
PINNs, Power transformers, Thermal modelling
National Category
Engineering and Technology Other Electrical Engineering, Electronic Engineering, Information Engineering
Research subject
Electrical Engineering
Identifiers
urn:nbn:se:kth:diva-315639 (URN)10.1016/j.epsr.2022.108447 (DOI)000836904300022 ()2-s2.0-85134327084 (Scopus ID)
Funder
Vinnova, 2021-03748SweGRIDS - Swedish Centre for Smart Grids and Energy Storage, CPC19
Note

QC 20220912

Available from: 2022-07-14 Created: 2022-07-14 Last updated: 2025-05-02Bibliographically approved
3. Physics-Informed Neural Networks for prediction of transformer's temperature distribution
Open this publication in new window or tab >>Physics-Informed Neural Networks for prediction of transformer's temperature distribution
Show others...
2022 (English)In: 2022 21ST IEEE INTERNATIONAL CONFERENCE ON MACHINE LEARNING AND APPLICATIONS, ICMLA / [ed] Wani, MA Kantardzic, M Palade, V Neagu, D Yang, L Chan, KY, Institute of Electrical and Electronics Engineers (IEEE) , 2022, p. 1579-1586Conference paper, Published paper (Refereed)
Abstract [en]

Physics-Informed Neural Networks (PINNs) are a novel approach to the integration of physical models into Neural Networks when solving supervised learning problems. PINNs have shown potential in mapping spatio-temporal input and the solution of a partial differential equation (PDE). However, despite their advantages for many applications, they often fail to train when target PDEs contain high frequencies or multiscale features. Thermal modelling of power transformers is fundamental for improving their efficiency and extending their lifetime. In this work, we investigate the performance of different PINN architectures applied to a 1D heat diffusion equation with a specific heat source representing the heat distribution inside a transformer. Measurements, which include the top-oil temperature, the ambient temperature and the load factor are taken from a transformer in service. We demonstrate the limitations of PINNs, propose possible remedies, and provide an overall assessment of the potential of using PINNs for transformer thermal modelling.

Place, publisher, year, edition, pages
Institute of Electrical and Electronics Engineers (IEEE), 2022
Keywords
PINN, thermal modelling, power transformer
National Category
Computer Sciences
Identifiers
urn:nbn:se:kth:diva-328419 (URN)10.1109/ICMLA55696.2022.00215 (DOI)000980994900236 ()2-s2.0-85152214174 (Scopus ID)
Conference
21st IEEE International Conference on Machine Learning and Applications (IEEE ICMLA), DEC 12-14, 2022, Nassau, BAHAMAS
Note

QC 20230613

Available from: 2023-06-13 Created: 2023-06-13 Last updated: 2025-05-02Bibliographically approved
4. Optimal Sensor Placement in Power Transformers Using Physics-Informed Neural Networks
Open this publication in new window or tab >>Optimal Sensor Placement in Power Transformers Using Physics-Informed Neural Networks
Show others...
(English)Manuscript (preprint) (Other academic)
Abstract [en]

Our work aims at simulating and predicting the temperature conditions inside a power transformer using Physics-Informed Neural Networks (PINNs). The predictions obtained are then used to determine the optimal placement for temperature sensors inside the transformer under the constraint of a limited number of sensors, enabling efficient performance monitoring. The method consists of combining PINNs with Mixed Integer Optimization Programming to obtain the optimal temperature reconstruction inside the transformer. First, we extend our PINN model for the thermal modeling of power transformers to solve the heat diffusion equation from 1D to 2D space. Finally, we construct an optimal sensor placement model inside the transformer that can be applied to problems in 1D and 2D.

Keywords
physics-informed neural networks, optimal sensor placement, power components, convex optimization, thermal modelling
National Category
Artificial Intelligence
Identifiers
urn:nbn:se:kth:diva-362871 (URN)
Funder
Vinnova, 2021-03748Vinnova, 2023-00241
Note

QC 20250430

Available from: 2025-04-28 Created: 2025-04-28 Last updated: 2025-05-02Bibliographically approved
5. Data-Driven vs Traditional Approaches to Power Transformer’s Top-Oil Temperature Estimation
Open this publication in new window or tab >>Data-Driven vs Traditional Approaches to Power Transformer’s Top-Oil Temperature Estimation
Show others...
(English)Manuscript (preprint) (Other academic)
Abstract [en]

Power transformers are subjected to electrical currents and temperature fluctuations that, if not properly controlled, can lead to major deterioration of their insulation system. Therefore, monitoring the temperature of a power transformer is fundamental to ensure a long-term operational life. Models presented in the IEC 60076-7 and IEEE standards, for example, monitor the temperature by calculating the top-oil and the hot-spot temperatures. However, these models are not very accurate and rely on the power transformers' properties. This paper focuses on finding an alternative method to predict the top-oil temperatures given previous measurements. Given the large quantities of data available, machine learning methods for time series forecasting are analyzed and compared to the real measurements and the corresponding prediction of the IEC standard. The methods tested are Artificial Neural Networks (ANNs), Time-series Dense Encoder (TiDE), and Temporal Convolutional Networks (TCN) using different combinations of historical measurements. Each of these methods outperformed the IEC 60076-7 model and they are extended to estimate the temperature rise over ambient. To enhance prediction reliability, we explore the application of quantile regression to construct prediction intervals for the expected top-oil temperature ranges. The best-performing model successfully estimates conditional quantiles that provide sufficient coverage.

Keywords
power transformers, heat distribution, time-series predictions, neural networks
National Category
Artificial Intelligence
Identifiers
urn:nbn:se:kth:diva-362873 (URN)
Funder
Vinnova, 2023-00241Vinnova, 2021-03748
Note

QC 20250430

Available from: 2025-04-28 Created: 2025-04-28 Last updated: 2025-05-02Bibliographically approved
6. Time Series Predictions Based on PCA and LSTM Networks: A Framework for Predicting Brownian Rotary Diffusion of Cellulose Nanofibrils
Open this publication in new window or tab >>Time Series Predictions Based on PCA and LSTM Networks: A Framework for Predicting Brownian Rotary Diffusion of Cellulose Nanofibrils
Show others...
2024 (English)In: Computational Science – ICCS 2024 - 24th International Conference, 2024, Proceedings, Springer Nature , 2024, p. 209-223Conference paper, Published paper (Refereed)
Abstract [en]

As the quest for more sustainable and environmentally friendly materials has increased in the last decades, cellulose nanofibrils (CNFs), abundant in nature, have proven their capabilities as building blocks to create strong and stiff filaments. Experiments have been conducted to characterize CNFs with a rheo-optical flow-stop technique to study the Brownian dynamics through the CNFs’ birefringence decay after stop. This paper aims to predict the initial relaxation of birefringence using Principal Component Analysis (PCA) and Long Short-Term Memory (LSTM) networks. By reducing the dimensionality of the data frame features, we can plot the principal components (PCs) that retain most of the information and treat them as time series. We employ LSTM by training with the data before the flow stops and predicting the behavior afterward. Consequently, we reconstruct the data frames from the obtained predictions and compare them to the original data.

Place, publisher, year, edition, pages
Springer Nature, 2024
Keywords
Cellulose Nanofibrils, Long Short-Term Memory, Principal Component Analysis, Time Series
National Category
Computer Engineering
Identifiers
urn:nbn:se:kth:diva-351761 (URN)10.1007/978-3-031-63749-0_15 (DOI)001279316700015 ()2-s2.0-85199666172 (Scopus ID)
Conference
24th International Conference on Computational Science, ICCS 2024, Malaga, Spain, Jul 2 2024 - Jul 4 2024
Note

Part of ISBN 9783031637483

QC 20240813

Available from: 2024-08-13 Created: 2024-08-13 Last updated: 2025-05-02Bibliographically approved
7. Automatic learning analysis of flow-induced birefringence in cellulose nanofibrils
Open this publication in new window or tab >>Automatic learning analysis of flow-induced birefringence in cellulose nanofibrils
Show others...
2025 (English)In: Journal of Computational Science, ISSN 1877-7503, E-ISSN 1877-7511, Vol. 85, article id 102536Article in journal (Refereed) Published
Abstract [en]

Cellulose Nanofibrils (CNFs), highly present in nature, can be used as building blocks for future sustainable materials, including strong and stiff filaments. A rheo-optical flow-stop technique is used to conduct experiments to characterize the CNFs by studying Brownian dynamics through the CNFs' birefringence decay after stop. As the experiments produce large quantities of data, we reduce their dimensionality using Principal Component Analysis (PCA) and exploit the possibility of visualizing the reduced data in two ways. First, we plot the principal components (PCs) as time series, and by training LSTM networks assigned for each PC time series with the data before the flow stop, we predict the behavior after the flow stop (Bragone et al., 2024). Second, we plot the first PCs against each other to create clusters that give information about the different CNF materials and concentrations. Our approach aims at classifying the CNF materials to varying concentrations by applying unsupervised machine learning algorithms, such as k-means and Gaussian Mixture Models (GMMs). Finally, we analyze the Autocorrelation Function (ACF) and the Partial Autocorrelation Function (PACF) of the first principal component, detecting seasonality in lower concentrations.

Place, publisher, year, edition, pages
Elsevier BV, 2025
Keywords
Cellulose nanofibrils, Principal component analysis, Long short-term memory, k-means, Gaussian mixture models
National Category
Probability Theory and Statistics
Identifiers
urn:nbn:se:kth:diva-360732 (URN)10.1016/j.jocs.2025.102536 (DOI)001425378400001 ()2-s2.0-85217011665 (Scopus ID)
Note

QC 20250303

Available from: 2025-03-03 Created: 2025-03-03 Last updated: 2025-05-02Bibliographically approved
8. MILP initialization for solving parabolic PDEs with PINNs
Open this publication in new window or tab >>MILP initialization for solving parabolic PDEs with PINNs
(English)Manuscript (preprint) (Other academic)
Abstract [en]

Physics-Informed Neural Networks (PINNs) are a powerful deep learning method capable of providing solutions and parameter estimations of physical systems. Given the complexity of their neural network structure, the convergence speed is still limited compared to numerical methods, mainly when used in applications that model realistic systems. The network initialization follows a random distribution of the initial weights, as in the case of traditional neural networks, which could lead to severe model convergence bottlenecks. To overcome this problem, we follow current studies that deal with optimal initial weights in traditional neural networks. In this paper, we use a convex optimization model to improve the initialization of the weights in PINNs and accelerate convergence. We investigate two optimization models as a first training step, defined as pre-training, one involving only the boundaries and one including physics. The optimization is focused on the first layer of the neural network part of the PINN model, while the other weights are randomly initialized. We test the methods using a practical application of the heat diffusion equation to model the temperature distribution of power transformers. The PINN model with boundary pre-training is the fastest converging method at the current stage.

National Category
Artificial Intelligence
Identifiers
urn:nbn:se:kth:diva-362872 (URN)
Funder
Vinnova, 2023-00241Vinnova, 2021-03748
Note

QC 20250430

Available from: 2025-04-28 Created: 2025-04-28 Last updated: 2025-05-02Bibliographically approved
9. $PINN - a Domain Decomposition Method for Bayesian Physics-Informed Neural Networks
Open this publication in new window or tab >>$PINN - a Domain Decomposition Method for Bayesian Physics-Informed Neural Networks
Show others...
(English)Manuscript (preprint) (Other academic)
Abstract [en]

Physics-Informed Neural Networks (PINNs) are a novel computational approach for solving partial differential equations (PDEs) with noisy and sparse initial and boundary data. Although, efficient quantification of epistemic and aleatoric uncertainties in big multi-scale problems remains challenging. We propose $PINN a novel method of computing global uncertainty in PDEs using a Bayesian framework, by combining local Bayesian Physics-Informed Neural Networks (BPINN) with domain decomposition. The solution continuity across subdomains is obtained by imposing the flux continuity across the interface of neighboring subdomains. To demonstrate the effectiveness of $PINN, we conduct a series of computational experiments on PDEs in 1D and 2D spatial domains. Although we have adopted conservative PINNs (cPINNs), the method can be seamlessly extended to other domain decomposition techniques. The results infer that the proposed method recovers the global uncertainty by computing the local uncertainty exactly more efficiently as the uncertainty in each subdomain can be computed concurrently. The robustness of $PINN is verified by adding uncorrelated random noise to the training data up to 15% and testing for different domain sizes.

National Category
Artificial Intelligence
Identifiers
urn:nbn:se:kth:diva-363003 (URN)
Funder
Vinnova, 2023-00241Vinnova, 2021-03748
Note

QC 20250430

Available from: 2025-04-30 Created: 2025-04-30 Last updated: 2025-05-02Bibliographically approved

Open Access in DiVA

fulltext(11643 kB)999 downloads
File information
File name FULLTEXT01.pdfFile size 11643 kBChecksum SHA-512
42298963bb01291249379d108e33236629472073c7b514c737a6e7f4fa175505c79ddeb3f602071dc5f24c3cfe224a08982a67ee2e688355de1c9c06f36f1003
Type fulltextMimetype application/pdf

Search in DiVA

By author/editor
Bragone, Federica
By organisation
Computational Science and Technology (CST)
Computer Sciences

Search outside of DiVA

GoogleGoogle Scholar
Total: 1001 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

isbn
urn-nbn

Altmetric score

isbn
urn-nbn
Total: 1287 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf