Abstract

Training machine learning tools such as neural networks require the availability of sizable data, which can be difficult for engineering and scientific applications where experiments or simulations are expensive. In this work, a novel multi-fidelity physics-constrained neural network is proposed to reduce the required amount of training data, where physical knowledge is applied to constrain neural networks, and multi-fidelity networks are constructed to improve training efficiency. A low-cost low-fidelity physics-constrained neural network is used as the baseline model, whereas a limited amount of data from a high-fidelity physics-constrained neural network is used to train a second neural network to predict the difference between the two models. The proposed framework is demonstrated with two-dimensional heat transfer, phase transition, and dendritic growth problems, which are fundamental in materials modeling. Physics is described by partial differential equations. With the same set of training data, the prediction error of physics-constrained neural network can be one order of magnitude lower than that of the classical artificial neural network without physical constraints. The accuracy of the prediction is comparable to those from direct numerical solutions of equations.

References

1.
Cang
,
R.
,
Xu
,
Y.
,
Chen
,
S.
,
Liu
,
Y.
,
Jiao
,
Y.
, and
Ren
,
M. Y.
,
2016
, “
Microstructure Representation and Reconstruction of Heterogeneous Materials Via Deep Belief Network for Computational Material Design
,”
ASME J. Mech. Des.
,
139
(
7
), p.
071404
. 10.1115/1.4036649
2.
Yang
,
Z.
,
Li
,
X.
,
Brinson
,
L. C.
,
Choudhary
,
A. N.
,
Chen
,
W.
, and
Agrawal
,
A.
,
2018
, “
Microstructural Materials Design Via Deep Adversarial Learning Methodology
,”
ASME J. Mech. Des.
,
140
(
11
), p.
111416
. 10.1115/1.4041371
3.
Pan
,
S. J.
, and
Yang
,
Q.
,
2010
, “
A Survey on Transfer Learning
,”
IEEE Trans. Knowl. Data Eng.
,
22
(
10
), pp.
1345
1359
. 10.1109/TKDE.2009.191
4.
Lee
,
S.-I.
,
Chatalbashev
,
V.
,
Vickrey
,
D.
, and
Koller
,
D.
,
2007
, “
Learning a Meta-Level Prior for Feature Relevance From Multiple Related Tasks
,”
Proceedings of the 24th International Conference on Machine Learning
,
Corvalis, OR
,
June 20–24
, pp.
489
496
.
5.
Yao
,
Y.
, and
Doretto
,
G.
,
2010
, “
Boosting for Transfer Learning With Multiple Sources
,”
2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition
,
San Francisco, CA
,
June 13–18
, pp.
1855
1862
.
6.
Raina
,
R.
,
Battle
,
A.
,
Lee
,
H.
,
Packer
,
B.
, and
Ng
,
A. Y.
,
2007
, “
Self-Taught Learning: Transfer Learning From Unlabeled Data
,”
Proceedings of the 24th international conference on Machine learning
,
Corvalis, OR
,
June 20–24
,
ACM Press
,
New York
, pp.
759
766
.
7.
Dai
,
W.
,
Xue
,
G.-R.
,
Yang
,
Q.
, and
Yu
,
Y.
,
2007
, “
Co-Clustering Based Classification for Out-of-Domain Documents
,”
Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining
,
San Jose, CA
,
Aug. 12–15
,
ACM Press
,
New York
, pp.
210
219
.
8.
Lawrence
,
N. D.
, and
Platt
,
J. C.
,
2004
, “
Learning to Learn With the Informative Vector Machine
,”
Proceedings of the Twenty-First International Conference on Machine Learning
,
Banff, Alberta, Canada
,
July 4–6
,
ACM Press
,
New York
, p.
65
.
9.
Bonilla
,
E.
,
Chai
,
K. M.
, and
Williams
,
C.
,
2007
, “
Multi-Task Gaussian Process Prediction
,”
Twenty-First Annual Conference on Neural Information Processing Systems
,
Vancouver, BC, Canada
,
Dec. 3–5
, pp.
153
160
.
10.
Tercan
,
H.
,
Guajardo
,
A.
,
Heinisch
,
J.
,
Thiele
,
T.
,
Hopmann
,
C.
, and
Meisen
,
T.
,
2018
, “
Transfer-Learning: Bridging the Gap Between Real and Simulation Data for Machine Learning in Injection Molding
,”
Proc. CIRP
,
72
, pp.
185
190
. 10.1016/j.procir.2018.03.087
11.
Karpatne
,
A.
,
Atluri
,
G.
,
Faghmous
,
J. H.
,
Steinbach
,
M.
,
Banerjee
,
A.
,
Ganguly
,
A.
,
Shekhar
,
S.
,
Samatova
,
N.
, and
Kumar
,
V.
,
2017
, “
Theory-Guided Data Science: A New Paradigm for Scientific Discovery From Data
,”
IEEE Trans. Knowl. Data Eng.
,
29
(
10
), pp.
2318
2331
. 10.1109/TKDE.2017.2720168
12.
Tran
,
A.
,
Furlan
,
J. M.
,
Pagalthivarthi
,
K. V.
,
Visintainer
,
R. J.
,
Wildey
,
T.
, and
Wang
,
Y.
,
2019
, “
WearGP: A Computationally Efficient Machine Learning Framework for Local Erosive Wear Predictions Via Nodal Gaussian Processes
,”
Wear
,
422–423
(
Mar.
), pp.
9
26
. 10.1016/j.wear.2018.12.081
13.
Li-Zhi
,
L.
, and
Hou-Duo
,
Q.
,
1999
, “
A Neural Network for the Linear Complementarity Problem
,”
Math. Comput. Model.
,
29
(
3
), pp.
9
18
. 10.1016/S0895-7177(99)00026-6
14.
Xia
,
Y.
,
Leung
,
H.
, and
Wang
,
J.
,
2002
, “
A Projection Neural Network and Its Application to Constrained Optimization Problems
,”
IEEE Trans. Circuits Syst. I Fundam. Theory Appl.
,
49
(
4
), pp.
447
458
. 10.1109/81.995659
15.
Thompson
,
M. L.
, and
Kramer
,
M. A.
,
1994
, “
Modeling Chemical Processes Using Prior Knowledge and Neural Networks
,”
AIChE J.
,
40
(
8
), pp.
1328
1340
. 10.1002/aic.690400806
16.
Watson
,
P. M.
,
Gupta
,
K. C.
, and
Mahajan
,
R. L.
,
1998
, “
Development of Knowledge Based Artificial Neural Network Models for Microwave Components
,”
1998 IEEE MTT-S International Microwave Symposium Digest (Cat. No.98CH36192)
, Vol.
1
, pp.
9
12
.
17.
Wang
,
F.
,
1997
, “
Knowledge-Based Neural Models for Microwave Design
,”
IEEE Trans. Microw. Theory Tech.
,
45
(
12 part 2
), pp.
2333
2343
. 10.1109/22.643839
18.
Nagarajan
,
H. P. N.
,
Dimassi
,
S.
,
Haapala
,
K. R.
,
Gary Wang
,
G.
,
Bakrani-Balani
,
S.
,
Coatanéa
,
E.
,
Hamedi
,
A.
,
Jafarian
,
H.
, and
Mokhtarian
,
H.
,
2018
, “
Knowledge-Based Design of Artificial Neural Network Topology for Additive Manufacturing Process Modeling: A New Approach and Case Study for Fused Deposition Modeling
,”
ASME J. Mech. Des.
,
141
(
2
), p.
021705
. 10.1115/1.4042084
19.
Tresp
,
V.
,
Hollatz
,
J.
, and
Ahmad
,
S.
,
1993
, “Network Structuring and Training Using Rule-Based Knowledge,”
Advances in Neural Information Processing Systems
, vol.
5
(NIPS 1992),
Morgan-Kaufmann
,
San Francisco, CA
, pp.
871
878
.
20.
Towell
,
G. G.
, and
Shavlik
,
J. W.
,
1994
, “
Knowledge-Based Artificial Neural Networks
,”
Artif. Intell.
,
70
(
1–2
), pp.
119
165
. 10.1016/0004-3702(94)90105-8
21.
Ramuhalli
,
P.
,
Udpa
,
L.
, and
Udpa
,
S. S.
,
2005
, “
Finite-Element Neural Networks for Solving Differential Equations
,”
IEEE Trans. Neural Networks
,
16
(
6
), pp.
1381
1392
. 10.1109/TNN.2005.857945
22.
Xu
,
C.
,
Wang
,
C.
,
Ji
,
F.
, and
Yuan
,
X.
,
2012
, “
Finite-Element Neural Network-Based Solving 3-D Differential Equations in MFL
,”
IEEE Trans. Magn.
,
48
(
12
), pp.
4747
4756
. 10.1109/TMAG.2012.2207732
23.
Han
,
F.
, and
Huang
,
D. S.
,
2008
, “
A New Constrained Learning Algorithm for Function Approximation by Encoding a Priori Information Into Feedforward Neural Networks
,”
Neural Comput. Appl.
,
17
(
5–6
), pp.
433
439
. 10.1007/s00521-007-0135-5
24.
Lauer
,
F.
, and
Bloch
,
G.
,
2008
, “
Incorporating Prior Knowledge in Support Vector Regression
,”
Mach. Learn.
,
70
(
1
), pp.
89
118
. 10.1007/s10994-007-5035-5
25.
Dissanayake
,
M. W. M. G.
, and
Phan-Thien
,
N.
,
1994
, “
Neural-Network-Based Approximations for Solving Partial Differential Equations
,”
Commun. Numer. Methods Eng.
,
10
(
3
), pp.
195
201
. 10.1002/cnm.1640100303
26.
Raissi
,
M.
,
Perdikaris
,
P.
, and
Karniadakis
,
G. E.
,
2018
, “
Physics-Informed Neural Networks: A Deep Learning Framework for Solving Forward and Inverse Problems Involving Nonlinear Partial Differential Equations
,”
J. Comput. Phys.
,
378
(
Feb.
), pp.
686
707
. 10.1016/j.jcp.2018.10.045
27.
de Cursi
,
J. E. S.
, and
Koscianski
,
A.
,
2007
,
Advances and Innovations in Systems, Computing Sciences and Software Engineering
,
Springer
, pp.
567
572
.
28.
Shirvany
,
Y.
,
Hayati
,
M.
, and
Moradian
,
R.
,
2009
, “
Multilayer Perceptron Neural Networks With Novel Unsupervised Training Method for Numerical Solution of the Partial Differential Equations
,”
Appl. Soft Comput. J.
,
9
(
1
), pp.
20
29
. 10.1016/j.asoc.2008.02.003
29.
Lagaris
,
I. E.
,
Likas
,
A.
, and
Fotiadis
,
D. I.
,
1998
, “
Artificial Neural Networks for Solving Ordinary and Partial Differential Equations
,”
IEEE Trans. Neural Networks
,
9
(
5
), pp.
987
1000
. 10.1109/72.712178
30.
Beidokhti
,
R. S.
, and
Malek
,
A.
,
2009
, “
Solving Initial-Boundary Value Problems for Systems of Partial Differential Equations Using Neural Networks and Optimization Techniques
,”
J. Franklin Inst.
,
346
(
9
), pp.
898
913
. 10.1016/j.jfranklin.2009.05.003
31.
Lagaris
,
I. E.
,
Likas
,
A. C.
, and
Papageorgiou
,
D. G.
,
2000
, “
Neural-Network Methods for Boundary Value Problems With Irregular Boundaries
,”
IEEE Trans. Neural Networks
,
11
(
5
), pp.
1041
1049
. 10.1109/72.870037
32.
McFall
,
K. S.
, and
Mahan
,
J. R.
,
2009
, “
Artificial Neural Network Method for Solution of Boundary Value Problems With Exact Satisfaction of Arbitrary Boundary Conditions
,”
IEEE Trans. Neural Networks
,
20
(
8
), pp.
1221
1233
. 10.1109/TNN.2009.2020735
33.
Malek
,
A.
, and
Shekari Beidokhti
,
R.
,
2006
, “
Numerical Solution for High Order Differential Equations Using a Hybrid Neural Network-Optimization Method
,”
Appl. Math. Comput.
,
183
(
1
), pp.
260
271
. 10.1016/j.amc.2006.05.068
34.
Bellamine
,
F.
,
Almansoori
,
A.
, and
Elkamel
,
A.
,
2015
, “
Modeling of Complex Dynamic Systems Using Differential Neural Networks With the Incorporation of a Priori Knowledge
,”
Appl. Math. Comput.
,
266
(
Sept.
), pp.
515
526
. 10.1016/j.amc.2015.05.122
35.
Ferrari
,
S.
, and
Jensenius
,
M.
,
2008
, “
A Constrained Optimization Approach to Preserving Prior Knowledge During Incremental Training
,”
IEEE Trans. Neural Networks
,
19
(
6
), pp.
996
1009
. 10.1109/TNN.2007.915108
36.
Di Muro
,
G.
, and
Ferrari
,
S.
,
2008
, “
A Constrained-Optimization Approach to Training Neural Networks for Smooth Function Approximation and System Identification
,”
Proceedings of the International Joint Conference Neural Networks
, pp.
2353
2359
.
37.
Rudd
,
K.
,
Di Muro
,
G.
, and
Ferrari
,
S.
,
2014
, “
A Constrained Backpropagation Approach for the Adaptive Solution of Partial Differential Equations
,”
IEEE Trans. Neural Networks Learn. Syst.
,
25
(
3
), pp.
571
584
. 10.1109/TNNLS.2013.2277601
38.
Kennedy
,
M.
, and
O’Hagan
,
A.
,
2000
, “
Predicting the Output From a Complex Computer Code When Fast Approximations Are Available
,”
Biometrika
,
87
(
1
), pp.
1
13
. 10.1093/biomet/87.1.1
39.
Fernández-Godino
,
M. G.
,
Park
,
C.
,
Kim
,
N.-H.
, and
Haftka
,
R. T.
,
2016
, arXiv preprint arXiv:1609.07196.
40.
Peherstorfer
,
B.
,
Willcox
,
K.
, and
Gunzburger
,
M.
,
2018
, “
Survey of Multifidelity Methods in Uncertainty Propagation, Inference, and Optimization
,”
SIAM Rev.
,
60
(
3
), pp.
550
591
. 10.1137/16M1082469
41.
Wang
,
X.
,
Liu
,
Y.
,
Sun
,
W.
,
Song
,
X.
, and
Zhang
,
J.
,
2018
, “
Multidisciplinary and Multifidelity Design Optimization of Electric Vehicle Battery Thermal Management System
,”
ASME J. Mech. Des.
,
140
(
9
), p.
094501
. 10.1115/1.4040484
42.
Zhou
,
Q.
,
Wang
,
Y.
,
Choi
,
S. K.
,
Jiang
,
P.
,
Shao
,
X.
, and
Hu
,
J.
,
2017
, “
A Sequential Multi-Fidelity Metamodeling Approach for Data Regression
,”
Knowl. Based Syst.
,
134
, pp.
199
212
. 10.1016/j.knosys.2017.07.033
43.
Zhou
,
Q.
,
Wang
,
Y.
,
Choi
,
S. K.
,
Jiang
,
P.
,
Shao
,
X.
,
Hu
,
J.
, and
Shu
,
L.
,
2018
, “
A Robust Optimization Approach Based on Multi-Fidelity Metamodel
,”
Struct. Multidiscip. Optim.
,
57
(
2
), pp.
775
797
. 10.1007/s00158-017-1783-4
44.
Abadi
,
M.
,
Agarwal
,
A.
,
Barham
,
P.
,
Brevdo
,
E.
,
Chen
,
Z.
,
Citro
,
C.
,
and Corrado
,
G. S.
,
Davis
,
A.
,
Dean
,
J.
,
Devin
,
M.
,
Ghemawat
,
S.
,
Goodfellow
,
I.
,
Harp
,
A.
,
Irving
,
G.
,
Isard
,
M.
,
Jia
,
Y.
,
Jozefowicz
,
R.
,
Kaiser
,
L.
,
Kudlur
,
M.
,
Levenberg
,
J.
,
Mane
,
D.
,
Monga
,
R.
,
Moore
,
S.
,
Murray
,
D.
,
Olah
,
C.
,
Schuster
,
M.
,
Shlens
,
J.
,
Steiner
,
B.
,
Sutskever
,
I.
,
Talwar
,
K.
,
Tucker
,
P.
,
Vanhoucke
,
V.
,
Vasudevan
,
V.
,
Viegas
,
F.
,
Vinyals
,
O.
,
Warden
,
P.
,
Wattenberg
,
M.
,
Wicke
,
M.
,
Yu
,
Y.
, and
Zheng
,
X.
,
2016
, “
TensorFlow: A System for Large-Scale Machine Learning
,”
12th USENIX Symposium on Operating Systems Design and Implementation
,
Savannah, GA
,
Nov. 2–4
.
45.
Baydin
,
A. G.
,
Pearlmutter
,
B. A.
,
Radul
,
A. A.
, and
Siskind
,
J. M.
,
2018
, “
Automatic Differentiation in Machine Learning: A Survey
,”
J. Mach. Learn. Res.
,
18
(
153
), pp.
1
43
.jmlr.org/papers/v18/17-468.html
46.
Lee
,
D.
, and
Myung
,
K.
,
2017
, “
Read My Lips, Login to the Virtual World
,”
2017 IEEE International Conference on Consumer Electronics
,
Las Vegas, NV
,
Jan. 8–10
, pp.
434
435
.
47.
Tran
,
A.
,
Liu
,
D.
,
Tran
,
H.
, and
Wang
,
Y.
,
2019
, “
Quantifying Uncertainty in the Process-Structure Relationship for Al–Cu Solidification
,”
Model. Simul. Mater. Sci. Eng.
,
27
(
6
), p.
064005
. 10.1088/1361-651X/ab2690
48.
Tran
,
A.
,
Sun
,
J.
,
Furlan
,
J. M.
,
Pagalthivarthi
,
K. V.
,
Visintainer
,
R. J.
, and
Wang
,
Y.
,
2019
, “
PBO-2GP-3B: A Batch Parallel Known/Unknown Constrained Bayesian Optimization With Feasibility Classification and Its Applications in Computational Fluid Dynamics
,”
Comput. Methods Appl. Mech. Eng.
,
347
(
Apr.
), pp.
827
852
. 10.1016/j.cma.2018.12.033
You do not currently have access to this content.