The Sigmoid Neural Network Activation Function and its Connections
to Airy’s and the Nield-Kuznetsov Functions
M.H. HAMDAN
Department of Mathematics and Statistics
University of New Brunswick
100 Tucker Park Road, Saint John, New Brunswick, E2L 4L5
CANADA
D.C. ROACH
Department of Engineering
University of New Brunswick
100 Tucker Park Road, Saint John, New Brunswick, E2L 4L5
CANADA
Abstract: - Analysis and solution of Airy’s inhomogeneous equation, when its forcing function is the sigmoid
neural network activation function, are provided in this work. Relationship between the Nield-Kuznetsov, the
Scorer, the sigmoid, the polylogarithm and Airy’s functions are established. Solutions to initial and boundary
value problems, when the sigmoid function is involved, are obtained. Computations were carried out using
Wolfram Alpha.
Key-Words: - Sigmoid Logistic Function; Airy’s Equation; Nield-Kuznetsov Functions.
Received: May 17, 2021. Revised: February 26, 2022. Accepted: March 21, 2022. Published: April 27, 2022.
1 Introduction
A function that continues to receive considerable
attention in neural networks and deep learning,
as it serves as an activation function, is the
sigmoid logistic function, (cf. [1-6] and the
references therein). Its characteristic S-shaped
curve, [3], maps the number line onto a finite-
length subinterval, such as (0,1), and its most
common form is given by:
(1)
Recently, this smoothly-increasing function
has made it to the porous media literature, where
it has been used in modelling variations in
permeability across a porous layer, [5].
The S-shaped graph of the sigmoid function
makes it appealing in the study of transition
layer, and Roach and Hamdan, [5], provided a
modification of and used it in the modelling
of Poiseuille flow through a Brinkman porous
layer of variable permeability, wherein they
created a continuously varying permeability
between relatively constant permeability
regions. An advantage of the Roach and Hamdan
approach is that it treats the flow domain as one
region with variable permeability, while
replicating flow in layered media.
PROOF
DOI: 10.37394/232020.2022.2.13
M. H. Hamdan, D. C. Roach
E-ISSN: 2732-9941
108
Volume 2, 2022
Depending on the choice of permeability
variations in the transition layer, Brinkman’s
equation can sometimes be reduced to Airy’s
inhomogeneous ordinary differential equation
(ODE), [7]. Airy’s ODE, [8-10], has received
considerable attention in the literature, and
general approaches to solutions of Airy’s
inhomogeenoeus ODE have been introduced,
(cf. [11-19] and the references therein). In
addition to its importance in mathematical
physics, solutions to the inhomogeneous Airy’s
ODE when its forcing function is a general
function of the independent variable, give rise to
new functions that are important in the
advancement of our mathematical library of
functions.
Of particular interest to the current work is the
solution to Airy’s inhomogeneous ODE when its
forcing function is the sigmoid logistic function,
, defined in (1), above. The objective is to
find the general solution to the resulting
inhomogeneous ODE, then use the general
solution to obtain solutions to initial and
boundary value problems. It will be shown that
this choice of forcing function leads to the
establishment of connections between the
sigmoid function, Airy’s functions, the Nield-
Kuznetsov functions, Scorer functions, and
polylogarithmic functions.
2 Properties of the Sigmoid Logistic
Function
Some properties of the sigmoid function, ,
are listed in what follows, (some of these
properties can be found in von Seggern, [3], and
Weisstein, [6]). In what follows, “prime”
notation denotes ordinary differentiation with
respect to the argument.
Property 1: Domain of is the set of real
numbers, and its range is the
interval . Graph of S(x) is shown in Fig. 1
for . This graph was obtained using
Wolfram Alpha.
Property 2: has horizontal asymptotes at
with
and
(2)
Property 3: The first two derivatives of are
given by
S(x)
x
Fig. 1 Graph of S(x) for
(3)
(4)
Property 4: Values at zero of and its first
two derivatives are
Property 5: Properties 3 and 4 imply that is
increasing on its domain and has a point of
inflection at .
Property 6: represents solution to
Bernoulli’s ODE of the form
(5)
where , with initial value
.
Property 7: Indefinite integral of is given
by
(6)
where C is a constant and log stands for the
natural logarithm.
PROOF
DOI: 10.37394/232020.2022.2.13
M. H. Hamdan, D. C. Roach
E-ISSN: 2732-9941
109
Volume 2, 2022
Property 8: Definite integral of over the
interval is given by
(7)
Property 9: Higher derivatives of can be
written as polynomials in . This might be of
convenience in obtaining higher deivatives of
.
(8)
) (9)
Equation (28) suggests that is related to the
solution of an ODE of the form:
(10)
Property 10: Higher derivatives of can be
written in terms of the first derivative, .
This might also be of convenience in obtaining
higher derivatives of . With the first
derivative given by (8), the second derivative can
be written as:
] (11)
Equation (11) suggests that is related to the
solution of an ODE of the form:
(12)
Property 11: possesses the following
Maclaurin series representation, [6]:
(13)
where is an Euler polynomial.
3 Solution to Airy’s ODE with
Sigmoid Forcing Function
Airy’s inhomogeneous ODE with as its
forcing function takes the form:
(14)
The general solution to (14) can be expressed
in the form
(15)
where and are the linearly
independent Airy’s functions of the first and
second kind, whose non-zero Wronskian is given
by, [9,10]:
(16)
and is the particular solution, expressible as,
[11]:
(17)
The integrals on the right of (17) are evaluated
using integration by parts, with the help of (6), to
yield:
(18)
(19)
Using (18) and (19) in (17), yields
(20)
Using (16), equation (20) can be written as:
(21)
where
(21)
is the dilogarithm function, [20,21].
Using the value
in (21), and
subsequent use in (15), the general solution to (14)
can be written as:
(22)
The above analysis furnishes the proof to the
following Theorem.
PROOF
DOI: 10.37394/232020.2022.2.13
M. H. Hamdan, D. C. Roach
E-ISSN: 2732-9941
110
Volume 2, 2022
Theorem 1.
The particular solution to Airy’s inhomogeneous
ODE with the sigmoid logistic function as its
forcing function is given by
and its general solution is given by
4 Connections with the Scorer
Functions and the Nield-Kuznetsov
Functions
Based on the analysis provided in Hamdan and
Kamel [11], the particular solution (17) can also be
expressed in the form
(23)
where the integral function, , is the Nield-
Kuznetsov function of the second kind that is defined
by the following equivalent forms, [11]:
(24)
(25)
and the integral function is the Nield-
Kuznetsov function of the first kind that is
defined by, [7,11]:
(26)
Hamdan and Kamel, [11], showed that
(27)
where and are the Scorer functions,
[22], that represent particular solutions to the
Airy’s inhomogeneous equation when the
forcing functions are
.
Now, equations (22) to (27) render the following
relationships between , , , ,
, , and :
(28)
(29)
Using (13), equation (41) can be written in terms of
the Scorer functions, as:
(30)
The above analysis furnishes the proof to the
following Theorem.
Theorem 2.
The Nield-Kuznetsov functions of the first and
second kinds are related to dilogarithm, sigmoid,
Airy’s, and Scorer’s functions through the
equations
5 Solution to Initial Value Problem
Consider the initial value problem composed of
solving equation (14) subject to the initial conditions
(31)
(32)
PROOF
DOI: 10.37394/232020.2022.2.13
M. H. Hamdan, D. C. Roach
E-ISSN: 2732-9941
111
Volume 2, 2022
where and are known constants.
From (15) and (21), the general solution is written as:
(33)
Evaluating (33) at , and using (31) yields
(34)
Differentiating (33) once results in:
(35)
Evaluating (35) at , and using (32) yields
(36)
Equations (34) and (36) provide the following
solution for and :
(37)
(38)
where
(39)
(40)
(41)
(42)
wherein is the Gamma function.
The following solution to the initial value problem
is thus obtained:
(43)
For the sake of illustration, consider the case of
and in the initial conditions (31) and
(32). Equations (37) and (38) render the following
values for the arbitrary constants:
(44)
(45)
Solution (43) then gives:
(46)
Graph of this solution is shown in Fig. 2, below.
Fig. 2. Solution to the Initial Value Problem
and
Computation and graphing of (46) was carried
out on Wolfram Alpha.
6 Solution to Boundary Value Problem
Consider the two-point boundary value problem
composed of solving equation (14) subject to the
following conditions on interval :
(47)
(48)
where and are known constants.
Using conditions (47) and (48) in the general
solution (2) yields:
(49)
(50)
Solutions to (49) and (50) are given by
(51)
PROOF
DOI: 10.37394/232020.2022.2.13
M. H. Hamdan, D. C. Roach
E-ISSN: 2732-9941
112
Volume 2, 2022
(52)
Using (51) and (52) in (22) gives the
following solution to the posed boundary value
problem:
(53)
For the sake of illustration, consider the
values in boundary
conditions (47) and (48). Expressions (51) and
(52) then render the following values for the
arbitrary constants:
(54)
(55)
and solution (53) then takes the form:
(56)
Graph of (56) is shown in Fig. 3 below.
7 Conclusion
The problem of solving the inhomogeneous
Airy’s equation (14), when its forcing function is
the sigmoid logistic function, was provided in
this work. Relationships between the Nield-
Kuznetsov functions of the first and second
kinds, Airy’s functions, the Scorer functions, the
sigmoid logistic function and the dilogarithm
function have been established and given in
Theorems 1 and 2.
Fig. 3. Solution to the Initial Value Problem
Computation and graphing of (56) was
carried out using Wolfram Alpha.
General solution to the posed problem was cast
in terms of Euler’s dilogarithm function, as given
in Theorem 1. General formulations of an initial
value problem and a two-point boundary value
problem were given and solutions were obtained
for particular values of the initial and boundary
conditions. Solutions and graphs have been
carried out using Wolfram Alpha.
References:
[1] Skapura, D.M., Building Neural Networks.
ACM Press, New York, NY, 1996.
[2] Haykin, S., Neural Networks: A
Comprehensive Foundation, 2/e. Prentice-Hall,
Inc., 1999.
[3] von Seggern, D., CRC Standard Curves and
Surfaces with Mathematics, 2nd ed. Boca Raton,
FL: CRC Press, 2007.
[4] Wood, T., Sigmoid Function. Retrieved
March 8, 2022. https://deepai.org/machine-
learning-glossary-and-terms/sigmoid-function.
[5] Roach, D.C. and Hamdan, M.H., On the Sigmoid
Function as a Variable Permeability Model for
Brinkman Equation, Trans. on Applied and
Theoretical Mechanics, WSEAS, Vol. 17, 2022, pp.
29-38.
[6] Weisstein, Eric W., Sigmoid Function. From
MathWorld--A Wolfram Web Resource, 2022,
https://mathworld.wolfram.com/SigmoidFuncti
on.html
PROOF
DOI: 10.37394/232020.2022.2.13
M. H. Hamdan, D. C. Roach
E-ISSN: 2732-9941
113
Volume 2, 2022
[7] Nield, D.A. and Kuznetsov, A.V., The effect
of a transition layer between a fluid and a porous
medium: shear flow in a channel, Transp Porous
Med, Vol. 78, 2009, pp. 477-487.
[8] Airy, G.B., On the Intensity of Light in the
Neighbourhood of a Caustic, Trans. Cambridge
Phil. Soc., Vol. 6, 1838, pp. 379-401.
[9] Abramowitz, M. and Stegun, I.A., Handbook
of Mathematical Functions, Dover, New York,
1984.
[10] Vallée, O. and Soares, M., Airy functions
and applications to Physics. World Scientific,
London, 2004.
[11] Hamdan, M.H. and Kamel, M.T., On the
Ni(x) Integral Function and its Application to the
Airy's Non-homogeneous Equation, Applied
Math. Comput. Vol. 21(17), 2011, pp. 7349-
7360.
[12] Hamdan, M.H., Alzahrani, S.M., Abu Zaytoon,
M.S. and Jayyousi Dajani, S., Inhomogeneous Airy’s
and Generalized Airy’s Equations with Initial and
Boundary Conditions, Int. J. Circuits, Systems and
Signal Processing, Vol. 15, 2021, pp. 1486-1496.
[13] Hamdan, M.H., Jayyousi Dajani, S.and Abu
Zaytoon, M,S., Nield-Kuznetsov Functions: Current
Advances and New Results, Int. J. Circuits, Systems
and Signal Processing, Vol. 15, 2021, pp. 1506-
1520.
[14] Hamdan, M.H., Jayyousi Dajani, S., and
Abu Zaytoon, M.S., Higher Derivatives and
Polynomials of the Standard Nield-Kuznetsov
Function of the First Kind, Int. J. Circuits,
Systems and Signal Processing, Vol. 15, 2021,
pp. 1737-1743.
[15] Miller, J. C. P. and Mursi, Z., Notes on the
solution of the equation ,
Quarterly J. Mech. Appl. Math., Vol. 3, 1950, pp.
113-118.
[16] Alzahrani, S.M., Gadoura, I. and Hamdan,
M.H., Ascending Series Solution to Airy's
Inhomogeneous Boundary Value Problem, Int. J.
Open Problems Compt. Math., Vol. 9(1), 2016,
pp. 1-11.
[17] S.M. Alzahrani, I. Gadoura, and M.H.
Hamdan, Initial and boundary value problems
involving the inhomogeneous Weber equation
and the Nield-Kuznetsov parametric function,
The International Journal of Engineering and
Science, Vol.6 No.3, 2017, pp. 9-24.
[18] Abu Zaytoon, M. S., Alderson, T. L. and
Hamdan, M. H., Flow through a Variable
Permeability Brinkman Porous Core, J. Appl.
Mathematics and physics, Vol. 4, 2016, pp. 766– 778.
[19] Jayyousi Dajani, S. and Hamdan, M.H.,
Airy’s Inhomogeneous Equation with Special
Forcing Function, İSTANBUL International
Modern Scientific Research Congress –II,
Istanbul, TURKEY, Proceedings, ISBN: 978-
625-7898-59-1, IKSAD Publishing House, Dec.
23-25, 2021, pp. 1367-1375.
[20] L.C. Maximon, The Dilogarithm Function
for Complex Argument, Proc. R. Soc. Lond. A,
Vol. 459, 2003, pp. 2807–2819.
[21] Lewin, L., ed., Structural Properties of
Polylogarithms, Mathematical Surveys and
Monographs, Vol. 37, 1991, Providence, RI:
Amer. Math. Soc. ISBN 978-0-8218-1634-9.
[22] Scorer, R.S., Numerical Evaluation of
Integrals of the Form
and
the Tabulation of the Function
, Quarterly J. Mech. Appl. Math., Vol. 3,
1950, pp. 107-112.
Contribution of individual authors
Both authors contributed to literature review,
problem formulation and solution, and manuscript
preparation.
Sources of funding
This work has not received any financial support
from any source.
Creative Commons Attribution
License 4.0 (Attribution 4.0
International , CC BY 4.0)
This article is published under the terms of the
Creative Commons Attribution License 4.0
https://creativecommons.org/licenses/by/4.0/deed.en
_US
PROOF
DOI: 10.37394/232020.2022.2.13
M. H. Hamdan, D. C. Roach
E-ISSN: 2732-9941
114
Volume 2, 2022
