Abstract

We study optimal control problems where the dynamic system evolves according to a linear stochastic differential equation with (a) multiplicative and additive white noise, (b) a mixture of white noise, and (c) white noise and colored noise. The drift rate and the diffusion matrix of the linear dynamic system depend on a continuous-time Markov chain with finite state space that is (I) partially observed and (II) completely observed. Using the results of Wonham filter theory, we reduce the partially observed problems to one with complete observation. We solve the control optimal problems explicitly with the help of dynamic programing technique, and three applications are presented to illustrate our theoretical results.

Issue Section:
Research Papers
Topics:
Chain, Differential equations, Dynamic systems, Noise (Sound), Optimal control, Filters, White noise

References

1.
Elliott
, R. J.,
Moore
,
J. B.
, and Aggoun, L.,
1994
,
Hidden Markov Models: Estimation and Control
,
Springer-Verlag
,
New York
.
2.
Escobedo-Trujillo
,
B.
, and
Garrido-Meléndez
,
J.
,
2021
, “
Stochastic LQR Optimal Control With White and Colored Noise: Dynamic Programming Technique
,”
Rev. Mexicana de Ingeniería Química
,
20
(
2
), pp.
1113
–
1129
.10.24275/rmiq/Sim2353
3.
Jung
,
P.
,
Neiman
,
A.
,
Afghan
,
M. K. N.
,
Nadkarni
,
S.
, and
Ullah
,
G.
,
2005
, “
Thermal Activation by Power-Limited Coloured Noise
,”
New J. Phys.
,
7
, pp.
17
–
17
.10.1088/1367-2630/7/1/017
4.
Arnold
,
L.
,
2013
,
Stochastic Differential Equations: Theory and Applications
,
Dover Publications
,
New York
.
5.
Napolitano
,
M.
,
2011
,
Aircraft Dynamics: From Modeling to Simulation
,
Wiley
, Hoboken, NJ.
6.
Song
,
J.
,
Niu
,
Y.
,
Lam
,
H.-K.
, and
Zou
,
Y.
,
2020
, “
Asynchronous Sliding Mode Control of Singularly Perturbed semi-Markovian Jump Systems: Application to an Operational Amplifier Circuit
,”
Automatica
,
118
, p.
109026
.10.1016/j.automatica.2020.109026
7.
Narendra
,
K. S.
, and
Tripathi
,
S. S.
,
1973
, “
Identification and Optimization of Aircraft Dynamics
,”
J. Aircr.
,
10
(
4
), pp.
193
–
199
.10.2514/3.44364
8.
De Farias
,
D.
,
Geromel
,
J.
,
Do Val
,
J.
, and
Costa
,
O.
,
2000
, “
Output Feedback Control of Markov Jump Linear Systems in Continuous-Time
,”
IEEE Trans. Autom. Control
,
45
(
5
), pp.
944
–
949
.10.1109/9.855557
9.
Zhai
,
D.
,
An
,
L.
,
Li
,
J.
, and
Zhang
,
Q.
,
2016
, “
Fault Detection for Stochastic Parameter-Varying Markovian Jump Systems With Application to Networked Control Systems
,”
Appl. Math. Modell.
,
40
(
3
), pp.
2368
–
2383
.10.1016/j.apm.2015.09.059
10.
Kumar
,
A.
, and
Jain
,
T.
,
2019
, “
Linear Quadratic Optimal Control Design: A Novel Approach Based on Krotov Conditions
,”
Math. Probl. Eng.
,
2019
, pp.
1
–
17
.10.1155/2019/9490512
11.
Prasad
,
L. B.
,
Tyagi
,
B.
, and
Gupta
,
H. O.
,
2011
, “
Optimal Control of Nonlinear Inverted Pendulum Dynamical System With Disturbance Input Using PID Controller & LQR
,”
IEEE International Conference on Control System, Computing and Engineering
, Penang, Malaysia, Nov. 25–27, pp.
540
–
545
.10.1109/ICCSCE.2011.6190585
12.
Chen
,
S.
,
Li
,
X.
, and
Zhou
,
X. Y.
,
1998
, “
Stochastic Linear Quadratic Regulators With Indefinite Control Weight Costs
,”
SIAM J. Control Optim.
,
36
(
5
), pp.
1685
–
1702
.10.1137/S0363012996310478
13.
Kalman
,
R.
,
1960
, “
Contribution to the Theory of Optimal Control
,”
Bol. Soc. Mat. Mexicana
,
5
, pp.
102
–
119
.https://www.springer.com/journal/40590/
14.
Lewis
,
F.
,
Vrabie
,
D.
, and
Syrmos
,
V.
,
1986
,
Optimal Control
,
Wiley
, Hoboken, NJ
.
15.
Wonham
,
W.
,
1968
, “
On a Matrix Riccati Equation of Stochastic Control
,”
SIAM J. Control Optim.
,
6
(
4
), pp.
681
–
697
.10.1137/0306044
16.
Tang
,
H.
,
Zhao
,
R.
,
Tang
,
S.
, and
Zeng
,
Z.
,
2012
, “
Linear Quadratic Optimal Control of a Single-Phase Grid-Connected Inverter With an LCL Filter
,”
IEEE International Symposium on Industrial Electronics
, Hangzhou, May 28–31, pp.
372
–
376
.10.1109/ISIE.2012.6237114
17.
Cheng
,
J.
,
Bo
,
W.
,
Park
,
J.
, and
Wei
,
K.
,
2017
, “
Sampled-Data Reliable Control for T-S Fuzzy semi-Markovian Jump System and Its Application to Single-Link Robot Arm Model
,”
IET Control Theory Appl.
,
11
(
12
), pp.
1904
–
1912
.10.1049/iet-cta.2016.1462
18.
Hernández-Osorio
,
M.
,
Ochoa-Velasco
,
C.
,
García-Alvarado
,
M.
,
Escobedo-Morales
,
A.
, and
Ruiz-López
,
I.
,
Benemérita Universidad Autónoma de Puebla
2019
, “
Sequential Synthesis of PID Controllers Based on LQR Method
,”
Rev. Mexicana de Ingeniería Química
,
19
(
2
), pp.
913
–
928
.10.24275/rmiq/Sim814
19.
Rojas
,
J. H. C.
,
Rodriguez
,
R.
,
López
,
J. A. Q.
, and
Perdomo
,
K. L. R.
,
2016
, “
LQR Hybrid Approach Control of a Robotic Arm Two Degrees of Freedom
,”
Int. J. Appl. Eng. Res.
,
11
(
17
), pp.
9221
–
9228
.http://www.ripublication.com/Volume/ijaerv11n17.htm
20.
Razmjooy
,
N.
,
Ramezani
,
M.
, and
Namadchian
,
A.
,
2016
, “
A New LQR Optimal Control for a Single-Link Flexible Joint Robot Manipulator Based on Grey Wolf Optimizer
,”
Majlesi J. Electr. Eng.
,
10
(
3
), pp.
53
–
60
.https://journals.iau.ir/article_696240.html
21.
López
,
J. D.
,
Espinosa
,
J. J.
, and
Agudelo
,
J. R.
,
2011
, “
LQR Control for Speed and Torque of Internal Combustion Engines
,”
IFAC Proc. Vol.
,
44
(
1
), pp.
2230
–
2235
.10.3182/20110828-6-IT-1002.02176
22.
Sam
,
Y. M.
,
Ghani
,
M. R. H. A.
, and
Ahmad
,
N.
,
2000
, “
LQR Controller for Active Car Suspension
,”
TENCON
Proceedings, Intelligent Systems and Technologies for New Millennium,
Kuala Lumpur, Malaysia, Sept. 24–27, pp.
441
–
444
.10.1109/TENCON.2000.893707
23.
Seekhao
,
P.
,
Tungpimolrut
,
K.
, and
Parnichkun
,
M.
,
2020
, “
Development and Control of a Bicycle Robot Based on Steering and Pendulum Balancing
,”
Mechatronics
,
69
, p.
102386
.10.1016/j.mechatronics.2020.102386
24.
Ahmad
,
F.
,
Kumar
,
P.
,
Bhandari
,
A.
, and
Patil
,
P. P.
,
2020
, “
Simulation of the Quadcopter Dynamics With LQR Based Control
,”
Mater. Today: Proc.
,
24
, pp.
326
–
332
.10.1016/j.matpr.2020.04.282
25.
Mao
,
X.
, and
Yuan
,
C.
,
2006
,
Stochastic Differential Equations With Markovian Switching
,
World Scientific Publishing Co
.,
UK
.
26.
Sethi
,
S.
,
Zhang
,
H.
, and
Zhang
,
Q.
,
2005
,
Average-Cost Control of Stochastic Manufacturing Systems
,
Springer
,
New York
.
27.
Athans
,
M.
,
Castanon
,
D.
,
Dunn
,
K.
,
Greene
,
C.
,
Lee
,
W.
,
Sandell
,
N.
, and
Willsky
,
A.
,
1977
, “
The Stochastic Control of the F-8C Aircraft Using a Multiple Model Adaptive Control (MMAC) method-Part I: Equilibrium Flight
,”
IEEE Trans. Autom. Control
,
22
(
5
), pp.
768
–
780
.10.1109/TAC.1977.1101599
28.
Ghosh
,
M. K.
,
Arapostathis
,
A.
, and
Marcus
,
S. I.
,
1993
, “
Optimal Control of Switching Diffusions With Applications to Flexible Manufacturing Systems
,”
SIAM J. Control Optim.
,
31
(
5
), pp.
1183
–
1204
.10.1137/0331056
29.
Ghosh
,
M. K.
,
Arapostathis
,
A.
, and
Marcus
,
S. I.
,
1997
, “
Ergodic Control of Switching Diffusions
,”
SIAM J. Control Optim.
,
35
(
6
), pp.
1952
–
1988
.10.1137/S0363012996299302
30.
Lin
,
Z.
,
Liu
,
J.
,
Wu
,
Q.
, and
Niu
,
Y.
,
2016
, “
Mixed H 2 / H ∞ Pitch Control of Wind Turbine With a Markovian Jump Model
,”
Int. J. Control
,
91
(
1
), pp.
156
–
169
.10.1080/00207179.2016.1272716
31.
Stein
,
G.
,
Hartmann
,
G.
, and
Hendrick
,
R.
,
1977
, “
Adaptive Control Laws for f-8 Flight Test
,”
IEEE Trans. Autom. Control
,
22
(
5
), pp.
758
–
767
.10.1109/TAC.1977.1101605
32.
Yin
,
G.
, and
Zhou
,
X.
,
2004
, “
Markowitz's Mean-Variance Portfolio Selection With Regime Switching: From Discrete-Time Models to Their Continuous-Time Limits
,”
IEEE Trans. Automat. Control
,
49
(
3
), pp.
349
–
360
.10.1109/TAC.2004.824479
33.
Yin
,
G.
, and
Zhu
,
C.
,
2007
, “
On the Notion of Weak Stability and Related Issues of Hybrid Diffusion Systems
,”
Nonlinear Anal.: Hybrid Syst.
,
1
(
2
), pp.
173
–
187
.10.1016/j.nahs.2006.03.003
34.
Pola
,
G.
,
Bujorianu
,
M. L.
,
Lygeros
,
J.
, and
Di Benedetto
,
M. D.
,
2003
, “
Stochastic Hybrid Models: An Overview
,”
IFAC Proc. Vol.
,
36
(
6
), pp.
45
–
50
.10.1016/S1474-6670(17)36405-4
35.
Blom
,
H. A. P.
,
2003
, “
Stochastic Hybrid Processes With Hybrid Jumps
,”
IFAC Proc. Vol.
,
36
(
6
), pp.
319
–
324
.10.1016/S1474-6670(17)36451-0
36.
Mao
,
X.
,
Yin
,
G. G.
, and
Yuan
,
C.
,
2007
, “
Stabilization and Destabilization of Hybrid Systems of Stochastic Differential Equations
,”
Automatica
,
43
(
2
), pp.
264
–
273
.10.1016/j.automatica.2006.09.006
37.
Bercu
,
B.
,
Dufour
,
F.
, and
Yin
,
G. G.
,
2009
, “
Almost Sure Stabilization for Feedback Controls of Regime-Switching Linear Systems With a Hidden Markov Chain
,”
IEEE Trans. Autom. Control
,
54
(
9
), pp.
2114
–
2125
.10.1109/TAC.2009.2026938
38.
Caines
,
P.
, and
Zhang
,
J.-F.
,
1992
, “
Adaptive Control for Jump Parameter Systems Via Nonlinear Filtering
,”
Proceedings of the 31st IEEE Conference on Decision and Control
, Tucson, AZ, Dec. 16–18, pp.
699
–
704
.10.1109/CDC.1992.371637
39.
Tran
,
K.
,
2021
, “
Optimal Exploitation for Hybrid Systems of Renewable Resources Under Partial Observation
,”
Nonlinear Anal.: Hybrid Syst.
,
40
(
5
), p.
101013
.10.1016/j.nahs.2021.101013
40.
Tran
,
K.
, and
Yin
,
G.
,
2014
, “
Stochastic Competitive Lotka–Volterra Ecosystems Under Partial Observation: Feedback Controls for Permanence and Extinction
,”
J. Franklin Inst.
,
351
(
8
), pp.
4039
–
4064
.10.1016/j.jfranklin.2014.04.015
41.
Stadtmann
,
F.
, and
Costa
,
O. L. V.
,
2018
, “
Exponential Hidden Markov Models for H ∞ Control of Jumping Systems
,”
IEEE Control Syst. Lett.
,
2
(
4
), pp.
845
–
850
.10.1109/LCSYS.2018.2847741
42.
Richter
,
J. H.
, and
Lunze
,
J.
,
2007
, “
Markov-Parameter-Based Control Reconfiguration by Matching the i/o-Behaviour of the Plant
,” European Control Conference (
ECC
), Kos, Greece, July 2–5, pp.
2942
–
2949
.10.23919/ECC.2007.7068757
43.
Wonham
,
W. M.
,
1964
, “
Some Applications of Stochastic Differential Equations to Optimal Nonlinear Filtering
,”
J. Soc. Ind. Appl. Math. Ser. A Control
,
2
(
3
), pp.
347
–
369
.10.1137/0302028
44.
Yin
,
G.
, and
Zhang
,
Q.
,
2006
,
Discrete-Time Markov Chains: Two-Time-Scale Methods and Applications
,
Stochastic Modelling and Applied Probability Series
,
Springer, New York
.
45.
de Paula Carvalho
,
L.
,
Rosa
,
T. E.
,
Jayawardhana
,
B.
, and
do Valle Costa
,
O. L.
,
2020
, “
Fault Compensation Controller for Markovian Jump Linear Systems
,”
IFAC-PapersOnLine
,
53
(
2
), pp.
4103
–
4108
.10.1016/j.ifacol.2020.12.2441
46.
Zhu
,
C.
, and
Yin
,
G.
,
2007
, “
Asymptotic Properties of Hybrid Diffusion Systems
,”
SIAM J. Control Optim.
,
46
(
4
), pp.
1155
–
1179
.10.1137/060649343
47.
Baloui Jamkhaneh
,
E.
,
2011
, “
The Effect of Different Noise Perturbations on the Parameter of the RC Circuit Using Stochastic Differential Equation (SDE)
,”
World Appl. Sci. J.
,
13
, pp.
2198
–
2202
.https://www.idosi.org/wasj/wasj13(10)2011.htm
48.
Durrett
,
R.
,
1996
,
Stochastic Calculus: A Practical Introduction
,
Taylor & Francis
,
New York
.
49.
Pham
,
H.
,
2009
,
Continuous-Time Stochastic Control and Optimization With Financial Applications
, Vol.
61
,
Springer
,
New York
.
50.
Hernández-Lerma
,
O.
,
1994
,
Lectures on Continuos-Time Markov Control Processes
, Aportaciones Matemáticas,
Sociedad Matemática Mexicana
,
México
.
51.
Klebaner
,
F. C.
,
2005
,
Introduction to Stochastic Calculus With Applications
,
Imperial College Press
,
London
.
52.
Bauerle
,
N.
, and
Rieder
,
U.
,
2004
, “
Portfolio Optimization With Markov-Modulated Stock Prices and Interest Rates
,”
IEEE Trans. Autom. Control
,
49
(
3
), pp.
442
–
447
.10.1109/TAC.2004.824471
53.
Schäl
,
M.
,
1975
, “
Conditions for Optimality and for the Limit of n-Stage Optimal Policies to Be Optimal
,”
Z. Wahrs. Verw. Gerb.
,
32
(
4
), pp.
179
–
196
.10.1007/BF00532612
54.
Friedman
,
A.
,
2012
,
Stochastic Differential Equations and Applications
,
Dover Publications
,
New York
.
55.
Morimoto
,
H.
,
2010
,
Stochastic Control and Mathematical Modeling: Applications in Economics
, England Series Encyclopedia of Mathematics and its Applications,
Cambridge University Press
, Cambridge, UK.
Copyright © 2023 by ASME
You do not currently have access to this content.