Some neural network theories and their applications in PDE modelling

Abstract

In this thesis, efforts are devoted to analyse different neural network models for PDE modelling. PDEs have been an important tool in many scientific disciplines for modelling dynamics in the systems of interests. In the era of big data and artificial intelligence, a data-driven approach to uncover the hidden PDEs in dynamical system is emerging and has become popular. However, these neural network models are subject to certain numerical instability and vulnerable to noise contamination in the data.

Here, we shall analyze the theories and study the limitations of two neural network models specially designed for PDE modelling, namely, PDE-Nets and Physics-informed Neural Networks (PINNs). New regularization methods and activation functions are proposed to improve their stabilities and accuracies in both solution findings and coefficient estimations as well as the prediction of long-term system behaviors.

PDE-Net is one type of residual neural networks devoted to discover PDEs from data but suffers from certain numerical instability because of the application of numerical differentiation filters all over its layers. Our key contribution to overcome this is to propose a new regularization method specially designed for PDE-Nets, called Differential Spectral Normalization (DSN), whose performance is quite good as demonstrated in our theoretical error analysis as well as results from the numerical experiments conducted both on a linear 2D PDE and a non-linear 1D PDE.

Besides PDE-Nets, another major and common neural network model is Physics-informed Neural Networks (PINNs), which although have successes in a number of applications, are vulnerable to high-frequency details or heavy noise contamination present in the observed data. In light of this, we shall study the Lipschitz constants of PINNs and propose a new activation function, namely Smooth Maxmin Activation (SMA), which allows PINNs to constrain their Lipschitz constants while maintaining the universal approximation property. Com paring among different combinations of activation functions and regularization methods, our experimental results show that SMA can improve the accuracy of solution findings as well as coefficient estimations.

Lastly, we have also analyzed the Neural Tangent Kernel (NTK) of PINNs and proposed the first general NTK theory of PINNs for linear first-order and second-order PDEs with no restrictions on the number of layers and dimensions. We have shown that the NTK of PINNs converges to a fixed limiting kernel during training in the infinite width limit. With this, we then discuss how the NTK can be applied to the adaptive balancing of different loss function terms in the PINN loss function using our newly proposed Adaptive Weighting of Loss Function (AWLF). At the end, we shall exhibit several numerical experiments demonstrating the promising results of our AWLF.

In summary, to improve the numerical stability and robostness of PDE-Nets and PINNs, a number of new regularization methods and activation functions have been proposed. For PDE-Nets, a specially designed regularization method, called Differentiation Spectral Normalization (DSN) have been proposed. For PINNs, a novel activation function, namely, Smooth Maxmin Activation (SMA) has been designed and the first general Neural Tangent Kernel (NTK) theory of PINNs have been obtained along with a new application, our Adaptive Weighting of Loss Function (AWLF). Experimental results are exhibited to support our proposed methods.