First-hitting3lace2ptimization3roblems for7wo-dimensional

'egenerate'iffusion3rocesses

MARIO LEFEBVRE

Department of Mathematics and Industrial Engineering,

Polytechnique Montréal,

2500, chemin de Polytechnique, Montréal (Québec) H3T 1J4,

CANADA

Abstract: Optimal control problems for degenerate two-dimensional diffusion processes are considered. The

processes could serve as models for wear processes. The objective is to make the controlled process leave the

continuation region through a given part of the boundary. Explicit and exact solutions are obtained for important

processes such as the geometric Brownian motion and the Ornstein-Uhlenbeck process.

Key-Words: Wear processes, Kolmogorov backward equation, homing problem, dynamic programming, special

functions, geometric Brownian motion, Ornstein-Uhlenbeck process.

5HFHLYHG$SULO5HYLVHG2FWREHU$FFHSWHG1RYHPEHU3XEOLVKHG'HFHPEHU

1 Introduction

Let (X(t), Y (t)) be the controlled two-dimensional

process defined by

dX(t) = f[X(t), Y (t)]dt, (1)

dY(t) = b[Y(t)]u(t)dt+m[Y(t)]dt

+{v[Y(t)]}1/2 dW(t),(2)

where {W(t), t ≥0}is a standard Brownian motion.

The functions m(·)∈Rand v(·)>0are such

that the uncontrolled process (X0(t), Y0(t)) obtained

by setting u(t)≡0in Eq. (2) is a (degenerate)

two-dimensional diffusion process.

Assume that (X(0), Y (0)) = (x, y)∈C⊂R2.

We define the first-passage time

τ(x, y) = inf{t > 0 : (X(t), Y (t)) ∈D⊂R2},

(3)

where D=Cc; that is, Dis the complement of Cin

R2.

If the function f(·,·)in Eq. (1) is such that it

is always positive in the region C, then the above

process could be an appropriate model for the wear

X(t)of a certain device at time t. Indeed, in reality,

wear should increase with time. We assume that

the wear depends on a variable Y(t)that evolves

according to a diffusion process; [1].

The aim is to find the control u∗(t)that minimizes

the expected value of the cost function

J(x, y) = τ(x,y)

2q0u2(t)dt+K[X(τ), Y (τ)],

(4)

where q0is a positive constant and K(·,·)is the

terminal cost function.

This type of problem, when the final time is a

first-passage time, is called an LQG homing problem.

Such problems have been treated extensively by the

author; see, for instance, [2] and, [3]. Other papers

on this topic are, [4], [5] and,[6]. In general, the

optimizer seeks to minimize or maximize the time

spent by the controlled process in the continuation

region C. Here, we are interested in the place where

the controlled process will leave C.

The current paper is more realistic than other

papers published on the optimal control of wear

processes because the controlled process is defined in

such a way that wear is strictly increasing with time,

and the final time is a random variable, rather than

being fixed.

To solve our problem, we shall use dynamic

programming. We define the value function

F(x, y) = inf

u(t),0≤t<τ (x)E[J(x, y)].(5)

We find that the optimal control can be expressed as

follows:

u∗(x, y) = −b(y)

Fy(x, y),(6)

where Fy(x, y) = ∂F (x, y)/∂y.

Remark. We wrote u(t)for the control variable in

Eq. (2), as most authors do. Actually, it would be

more accurate to write u[X(t), Y (t)].

WSEAS TRANSACTIONS on SYSTEMS and CONTROL

DOI: 10.37394/23203.2024.19.37

Mario Lefebvre

E-ISSN: 2224-2856

353

Volume 19, 2024

Next, assume that there exists a positive constant

θsuch that

v(y) = θ b2(y)/q0.(7)

Then, we can show ([7] or [8]) that the function

Φ(x, y) := e−F(x,y)/θ(8)

satisfies the linear partial differential equation (PDE)

2v(y)Φyy +m(y)Φy+f(x, y)Φx= 0,(9)

and is subject to the boundary condition

Φ(x, y) = e−K(x,y)/θif (x, y)∈D. (10)

Moreover, we can write that

Φ(x, y) = Ee−K[X0(T),Y0(T)]/θ,(11)

where T(= T(x, y)) is the same as τ(x, y), but for

the uncontrolled process (X0(t), Y0(t)). Hence, if the

relation in Eq. (7) holds, it is possible to determine

the optimal control by computing a mathematical

expectation for the uncontrolled process.

In the next section, the function F(x, y)will

be computed explicitly for important diffusion

processes, such as the Ornstein-Uhlenbeck process,

from which the optimal control follows at once my

making use of Eq. (6).

2 Optimal&ontrol3roblems

Case 1. Suppose first that Eq. (2) is given by

dY(t) = b0u(t)dt−αY (t)dt+σdW(t),(12)

where b0,αand σare positive constants. Then, if

u(t)≡0,{Y(t), t ≥0}is an Ornstein-Uhlenbeck

process, which is one of the most important diffusion

processes. Notice that Eq. (7) is satisfied by taking

θ=σ2q0/b2

Let

τ(x, y) = inf{t > 0 : Y(t)−X(t) = k1or k2},

(13)

where x≥0,y > 0and 0≤k1< y −x < k2.

Suppose that the function f[X(t), Y (t)] in Eq. (1)

is given by

f[X(t), Y (t)] = αY (t).(14)

Moreover, we choose

K[X(τ), Y (τ)] = 1if Y(τ)−X(τ) = k1,

0if Y(τ)−X(τ) = k2.

(15)

That is, the aim is to make Y(t)−X(t)leave

the interval (k1, k2)at k2. Since Y(0) = yis

assumed to be positive, Y(t)is always positive in the

continuation region C:= {(x, y)∈R2: 0 ≤k1<

y−x < k2}. It follows that X(t)will be strictly

increasing in C.

To obtain the function Φ(x, y)defined in Eq. (8),

we must solve the PDE

2σ2Φyy −αy Φy+αy Φx= 0,(16)

subject to the boundary conditions

Φ(x, y) = e−1/θif y−x=k1,

1if y−x=k2.(17)

Let z:= y−x. We shall try to find a solution of

the form

Φ(x, y) = Ψ(z).(18)

This is an application of the method of similarity

solutions, and zis called the similarity variable.

Equation (16) simplifies to the ordinary

differential equation (ODE)

2σ2Ψ′′(z) = 0.(19)

Hence, we may write that

Ψ(z) = c1z+c0.(20)

The solution that satisfies the boundary conditions

Ψ(k1) = e−1/θand Ψ(k2) = 1 is

Ψ(z) = (z−k2)e−1/θ−z+k1

k1−k2

for k1≤z≤k2.

(21)

We can now state the following proposition.

Proposition 2.1. The value function in the problem

considered above is given by

F(x, y) =

−θln (y−x)e−1/θ−1−k2e−1/θ+k1

k1−k2(22)

for k1≤y−x≤k2. Furthermore, from Eq. (6), the

optimal control is

u∗(x, y) =

−b0

θe−1/θ−1

k2e−1/θ−k1−(y−x)e−1/θ−1(23)

for k1< y −x < k2, where θ=σ2q0/b2

WSEAS TRANSACTIONS on SYSTEMS and CONTROL

DOI: 10.37394/23203.2024.19.37

Mario Lefebvre

E-ISSN: 2224-2856

354

Volume 19, 2024

In the special case when k1= 0 and σ=q0=

b0=k2= 1, the value function is given by

F(x, y) = −ln 1−e−1(y−x) + e−1(24)

and the optimal control becomes

u∗(x, y) = 1−e−1

e−1+ (y−x) (1 −e−1)(25)

for 0< y −x < 1. This function is shown in

Figure 1 when x= 0. We see that the optimal control

decreases as yincreases from 0 to 1, which is logical

since the optimizer wants the controlled process to

leave the continuation region Cthrough the straight

line y−x= 1.

Figure 1: Function u∗(0, y)in Case 1 for y∈[0,1]

when k1= 0 and σ=q0=b0=k2= 1.

Case 2. Suppose next that

dX(t) = f0X(t)/Y(t)dt, (26)

dY(t) = b0Y1/2(t)u(t)dt+σ Y 1/2(t)dW(t),

(27)

where f0= 0, and b0and σare positive constants.

This time the uncontrolled process {Y(t), t ≥0}is

a limit case of a Cox-Ingersoll-Ross process, which

is used in financial mathematics as a model for the

evolution of interest rates, or a particular squared

Bessel process (if σ= 2). Equation (7) is satisfied

with θ=σ2q0/b2

0, as in Case 1.

We define the first-passage time

τ(x, y) = inf{t > 0 : X(t)/Y(t) = k1or k2},

(28)

where x > 0,y > 0and 0< k1< x/y < k2. Then

X(t)will increase with time if f0>0, as it should in

the case of a wear process.

Remark. If X(t)represents the wear of a device

at time t, then f0must be positive, whereas f0is

negative if X(t)is rather the remaining lifetime of

the device.

The terminal cost function is given by

K[X(τ), Y (τ)] = 1if X(τ)/Y(τ) = k1,

0if X(τ)/Y(τ) = k2.

(29)

The function Φ(x, y)is a solution of the PDE

2σ2yΦyy +f0

yΦx= 0,(30)

subject to the boundary condition

Φ(x, y) = e−1/θif x/y=k1,

1if x/y=k2.(31)

We assume that

Φ(x, y) = Ψ(z),(32)

where the similarity variable is z:= x/y.

Equation (30) is then reduced to the ODE

2σ2z2Ψ′′(z) + σ2zΨ′(z) + f0zΨ′(z) = 0 (33)

and the boundary conditions are

Ψ(z) = e−1/θif z=k1,

1if z=k2.(34)

Since z > 0in the continuation region C, we may

write that

2σ2zΨ′′(z)+(σ2+f0)Ψ′(z) = 0.(35)

κ:= σ2+f0= 0,(36)

then the solution is the same as in Case 1. When κis

different from zero, we find that the general solution

of Eq. (35) is

Ψ(z) = c1+c2z∆,(37)

where

∆ := −2f0

σ2+ 1.(38)

Proposition 2.2. The value function in Case 2 is given

F(x, y) = −θln[Ψ(x/y)],(39)

where the function Ψ(·)is defined in Eq. (37) and

the constants c1and c2are determined by using the

boundary conditions in Eq. (34).

Moreover, the optimal control is

u∗(x, y) = −b0√y

Fy(x, y)(40)

for k1< x/y < k2.

WSEAS TRANSACTIONS on SYSTEMS and CONTROL

DOI: 10.37394/23203.2024.19.37

Mario Lefebvre

E-ISSN: 2224-2856

355

Volume 19, 2024

Assume that b0=q0=σ= 1, so that θ= 1, and

f0= 1 as well. Then, κ= 2 and ∆ = −3. If k1= 1

and k2= 2, we find that

F(x, y) = ln(7) −ln 8−e−1+8(e−1−1)y3

x3

(41)

for 1≤x/y≤2. Furthermore, the optimal control is

u∗(x, y) = 24y5/2 (1 −e−1)

(x3−8y3)e−1+ 8(y3−x3).(42)

The function F(x, y)is presented in Figure 2 in

terms of x/y∈[1,2], and the optimal control u∗(1, y)

is displayed in Figure 3 for y∈[0.5,1].

Figure 2: Value function F(x, y)in Case 2 for x/y∈

[1,2] when q0=b0=f0=σ=k1= 1 and k2= 2.

Figure 3: Optimal control u∗(1, y)in Case 2 for y∈

[0.5,1] when q0=b0=f0=σ=k1= 1 and

k2= 2.

Case 3. Finally, we consider the process defined by

dX(t) = f0Y2(t)dt, (43)

dY(t) = b0Y(t)u(t)dt+µY (t)dt+σY (t)dW(t),

(44)

where f0= 0,b0and σare positive constants, and

µ∈R. We assume that (X(0), Y (0)) = (x, y), with

xand ypositive. The relation in Eq. (7) is satisfied if

θ=σ2q0/b2

0, as in the previous cases.

In the above case, the uncontrolled process

{Y(t), t ≥0}is a geometric Brownian motion, which

is very important in financial mathematics. Since this

process is always positive (when Y(0) >0), the

variable X(t)will increase with tif u(t)≡0.

The first-passage time τ(x, y)is defined by

τ(x, y) = inf{t > 0 : X(t)/Y2(t) = k1or k2},

(45)

where 0< k1< x/y2< k2, and the terminal cost

function is

K[X(τ), Y (τ)] = 1if X(τ)/Y2(τ) = k1,

0if X(τ)/Y2(τ) = k2.

(46)

For this model, we can generalize the cost function

defined in Eq. (4) to

J(x, y) = τ(x,y)

01

2q0u2(t) + λdt

+K[X(τ), Y (τ)],(47)

where λ∈R. Then, the aim is also to make the

controlled process leave the continuation region Cas

rapidly as possible (if λ > 0) or remain in Cas long

as possible (if λ < 0).

The function Φ(x, y)defined in Eq. (8) is now

given by

Φ(x, y) = Eexp −λT +K[X(T), Y (T)]

θ.

(48)

It is a solution of the PDE

2σ2y2Φyy +µy Φy+f0y2Φx=βΦ,(49)

where β:= λ/θ, and the boundary conditions are

Φ(x, y) = e−1/θif x/y2=k1,

1if x/y2=k2.(50)

Let z:= x/y2and define Ψ(z) = Φ(x, y), as

above. The function Ψsatisfies the ODE

2σ2z2Ψ′′(z) + [(3σ2−2µ)z+f0]Ψ′(z) = βΨ(z).

(51)

The boundary conditions are the same as in the

previous cases:

Ψ(z) = e−1/θif z=k1,

1if z=k2.(52)

The general solution of Eq. (51) is in terms of the

Kummer functions M(·,·,·)and U(·,·,·) (9],

WSEAS TRANSACTIONS on SYSTEMS and CONTROL

DOI: 10.37394/23203.2024.19.37

Mario Lefebvre

E-ISSN: 2224-2856

356

Volume 19, 2024

Ψ(z) = z−γ−4µ

8σ2+1

×c1Mγ−4µ

8σ2+1

4,γ

4σ2+ 1,f0

2σ2z

+c2Uγ−4µ

8σ2+1

4,γ

4σ2+ 1,f0

2σ2z,(53)

where

γ:= 2σ4+ 4(2β−µ)σ2+ 4µ2.(54)

Proposition 2.3. We may write that the value function

in Case 3 is

F(x, y) = −θln[Ψ(x/y2)],(55)

where Ψ(·)is defined in Eq. (53). The constants c1

and c2are deduced from the boundary conditions in

Eq. (52). Furthermore, the optimal control is given

u∗(x, y) = −b0y

Fy(x, y)(56)

for k1< x/y2< k2.

Let us take b0=q0=f0=µ= 1,σ=√2and

λ= 8. Then θ= 2,β= 4 and the value function

F(x, y)can be expressed as elementary functions:

Ψ(z) = c1(1 + 4z) + c2z e1/(4z).(57)

Let k1= 1 and k2= 2. We find that

c1=2e−5/8 −1

10e−1/8 −9(58)

and

c2=e−1/4 (5 −9e−1/2)

10e−1/8 −9.(59)

Proceeding as above, we can obtain the value function

F(x, y)and the optimal control u∗(x, y)explicitly.

Now, when λ= 0 (as in Cases 1 and 2), the

function Ψ(z)(denoted by Ψ0(z)) becomes

Ψ0(z) = d1+d2Ei1−1

4z,(60)

where Ei1(z)is an exponential integral function

defined by

Ei1(z) = ∞

e−wz

wdw. (61)

The constants d1and d2for which the boundary

conditions are satisfied are

d1=Ei1(−1/8)e−1/2 −Ei1(−1/4)

Ei1(−1/8) −Ei1(−1/4) (62)

and

d2=1−e−1/2

Ei1(−1/8) −Ei1(−1/4).(63)

From Ψ0(z), we can now compute the corresponding

value function F0(x, y)and the optimal control

u∗

0(x, y).

The value functions F(x, y)and F0(x, y)and the

optimal controls u∗(x, y)and u∗

0(x, y)are shown

respectively in Figure 4 and Figure 5.

Figure 4: Functions F(x, y)(solid line) and F0(x, y)

in Case 3 for x/y2∈[1,2] when b0=q0=f0=

µ= 1,σ=√2and λ= 8.

Figure 5: Functions u∗(1, y)(solid line) and u∗

0(1, y)

in Case 3 for y∈[(√2)−1,1] when b0=q0=f0=

µ= 1,σ=√2and λ= 8.

If we replace µ= 1 by µ=−1, we find that

the general solution of Eq. (51) is given in terms of

modified Bessel functions:

Ψ(z) = e1/(8z)c1I√5/2[1/(8z)]

+c2K√5/2[1/(8z)].(64)

The value function F(x, y)and the optimal control

u∗(x, y)are presented respectively in Fig.6 & Fig.7,

together with the corresponding functions when

µ= 1. We see that although the value functions are

WSEAS TRANSACTIONS on SYSTEMS and CONTROL

DOI: 10.37394/23203.2024.19.37

Mario Lefebvre

E-ISSN: 2224-2856

357

Volume 19, 2024

rather similar, the optimal controls are quite different.

Figure 6: Functions F(x, y)when µ= 1 (solid line)

and µ=−1in Case 3 for x/y2∈[1,2] when b0=

q0=f0= 1,σ=√2and λ= 8.

Figure 7: Functions u∗(1, y)when µ= 1 (solid line)

and µ=−1in Case 3 for y∈[(√2)−1,1] when

b0=q0=f0= 1,σ=√2and λ= 8.

3 Conclusion

In this paper, we obtained explicit and exact solutions

to optimal control problems for two-dimensional

diffusion processes that could be used as models for

the wear (or the remaining lifetime) of a device.

These problems are particular LQG homing problems

which are very difficult to solve, especially in two or

more dimensions.

Using a result due to Whittle, it was possible

to transform the control problems into purely

probabilistic problems. Indeed, when the relation

in Eq. (7) holds, it is possible to reduce the

non-linear PDE satisfied by the value function to

a linear PDE. This linear PDE is in fact the

Kolmogorov backward equation satisfied by a certain

mathematical expectation for the corresponding

uncontrolled process.

Solving the Kolmogorov backward equation,

subject to the appropriate boundary conditions, is in

itself a difficult problem. Here, the linear PDE was

solved explicitly in three important cases by making

use of the method of similarity solutions.

When the relation in Eq. (7) does not hold, we can

try other methods to obtain at least an approximate

expression for the value function. We can also

compute a numerical solution in any particular case.

Another possibility is to calculate bounds for the

value function and the corresponding optimal control,

as was done in,[10].

We could try to solve particular problems when

there is more than one explanatory variable Y(t).

Finally, we could consider discrete-time versions of

the problem treated in this paper.

Acknowledgements. The author would like to

express his gratitude to the reviewers of this paper for

their constructive comments.

References:

[1] R. Rishel, Controlled wear process: modeling

optimal control, IEEE Transactions on

Automatic Control, Vol. 36, No. 9, 1991,

pp. 1100–1102. https://doi.org/10.1109/9.83548

[2] M. Lefebvre and P. Pazhoheshfar, An optimal

control problem for the maintenance of a

machine, International Journal of Systems

Science, Vol. 53, No. 15, 2022, pp. 3364–3373.

https://doi.org/10.1080/00207721.2022.2083258

[3] M. Lefebvre, First-passage times and

optimal control of integrated jump-diffusion

processes, Fractal and Fractional,

Vol. 7, No. 2, 2023, Article 152.

https://doi.org/10.3390/fractalfract7020152

[4] J. Kuhn, The risk-sensitive homing

problem, Journal of Applied Probability,

Vol. 22, No. 4, 1985, pp. 796–803.

https://doi.org/10.2307/3213947

[5] C. Makasu, Risk-sensitive control for a

class of homing problems, Automatica,

Vol. 45, No. 10, 2009, pp. 2454–2455.

https://doi.org/10.1016/j.automatica.2009.06.015

[6] M. Kounta and N.J. Dawson, Linear quadratic

Gaussian homing for Markov processes

with regime switching and applications

to controlled population growth/decay,

Methodology and Computing in Applied

Probability, Vol. 23, 2021, pp. 1155–1172.

https://doi.org/10.1007/s11009-020-09800-2

[7] P. Whittle, Optimization over Time, Vol. I,

Wiley, Chichester, 1982.

WSEAS TRANSACTIONS on SYSTEMS and CONTROL

DOI: 10.37394/23203.2024.19.37

Mario Lefebvre

E-ISSN: 2224-2856

358

Volume 19, 2024

[8] P. Whittle, Risk-Sensitive Optimal Control,

Wiley, Chichester, 1990.

[9] M. Abramowitz and I.A. Stegun, Handbook

of Mathematical Functions with Formulas,

Graphs, and Mathematical Tables, Dover, New

York, 1965.

[10] C. Makasu, Homing problems with control in

the diffusion coefficient, IEEE Transactions on

Automatic Control, Vol. 67, No. 7, 2022,

pp. 3770–3772. https://doi.org/10.1109

/TAC.2022.3157077

Contribution of Individual Authors to the

Creation of a Scientific Article (Ghostwriting

Policy)

The author contributed in the present research, at all

stages from the formulation of the problem to the

final findings and solution.

Sources of Funding for Research Presented in a

Scientific Article or Scientific Article Itself

No funding was received for conducting this study.

Conflict of Interest

The author has no conflict of interest to declare that

is relevant to the content of this article.

Creative Commons Attribution License 4.0

(Attribution 4.0 International, CC BY 4.0)

This article is published under the terms of the

Creative Commons Attribution License 4.0

https://creativecommons.org/licenses/by/4.0/deed.en

_US

WSEAS TRANSACTIONS on SYSTEMS and CONTROL

DOI: 10.37394/23203.2024.19.37

Mario Lefebvre

E-ISSN: 2224-2856

359

Volume 19, 2024