CC BY
2Stochastic Differential Games and Queueing Models To Innovation and Patenting
Andre A. Keller1
Research Group for Learning, Innovation and Knowledge in Organizations, Universite de Haute Alsace, Mulhouse, France e-mail address: andre.keller@uha.fr
Abstract. Dynamic differential games have been widely applied to the timing of product and device innovations. Uncertainty is also inherent in the process of technological innovation: R&D expenditures will be engaged in an unforeseeable environment and possibly lead to innovations after a random time interval. Reinganum [Reinganum, 1982] enumerates such uncertainties and risks: feasibility, delays in the process, imitation by rivals. Uncertainties generally affect the fundamentals of the standard differential game problem: discounted profit functional, differential state equations of the system, initial states. Two ways of resolution may be taken [Dockner, 2000]: firstly, stochastic differential games with Wiener process and secondly differential games with deterministic stages between random jumps (Poisson driven probabilities) of the modes. The player will then maximize the expected flows of his discounted profits subject to the stochastic state constraints of the system. In this context, the state evolution is described by a stochastic differential equation SDE (the Ito equation or the Kolmogorov forward equation KFE). According to the Dasguspta and Stiglitz’s model [Dasguspta, 1980], R&D efforts exert direct and induced influences (through accumulated knowledge) about the chances of success of innovations. The incentive to innovate and the R&D competition can be supplemented by a competition around a patent. This presentation is focused on such essential economic and managerial problems (R&D investments by firms, innovation process , and patent protection) with uncertainties using stochastic differential games [Friedman, 2004], [Yeung, 2006], [Kythe, 2003], modeling with It SDEs [Allen, 2007] and queueing models [Gross, 1998]. The computations are carried out using the software Mathematica 5.1 and other specialized packages [Wolfram, 2003], [Kythe, 2003].
1 Universite de Haute Alsace, Campus de la Fonderie, rue de la Fonderie, 68093 Mulhouse Cedex France.
Keywords: Stochastic differential game, uncertainty, Markov-Nash equilibrium,
technological innovation JEL Classification: C73, O31, Q55 - Mathematics Subject Classification . Primary 60K25, 91A23; Secondary 60H15, 90B22, 91A80.
1. Uncertainties and basic stochastic processes
Definition 1. (stochastic process). Given a filtered probability space (Q, F, P) with filtration 2 {Tt,t > 0} satisfying the conditions of right continuity and completion, a stochastic process is a collection of random variables {xt}teT defined on (Q, F, P) with values in R". The process may be represented by the function [0ksendal, 2003]
(t, w) ^ x(t, w) : T x Q ^ R".
We will thus have the number of random events that occur in [0,t].
Definition 2. (random va,ria,ble, path). For each t G T fixed, the state of the process is given. We then have a random variable (RV)defined by
w ^ xt(w), w G Q.
For each experiment w G Q, we have a path of xt defined by
t ^ xt(w), t G T.
Definition 3. (stationary process). A process xt is stationary if
xtl,.. ., xtn and xtl+s,..., xtn+s have the same joint distribution for all n and s.3
1.1. Brownian motion
Definition 4 (one-dimensional Brownian motion, or Wiener process [Friedman, 2004]. A stochastic process {zt}t>o, is a Brownian motion satisfying the conditions:
(i) zo = 0, (ii) the process has stationary independent increments ztk — ztk-1 (1 < k < n), and (iii) if 0 < s < t, zt — zs is normally distributed with E[zt — zs] =
(t — s)^ and E[{zt — zs}2] = (t — s)<r2, where ^ is the drift and a2 - the variance.
According to the first condition any zt that starts at z0 can be redefined as zt — z0. The second condition tells that the random increment ztn+1 — ztn is independent of the previous one ztn-1 — ztn-2, for all n. Increments are stationary when zt — zt-s has the same distribution for any t and s constants. With the third condition, the RV has
2 In algebra, a filtration of a group is ordinarily a sequence Gn(n G N) of subgroups such that
Gn+i C Gn. A filtration is often used to represent the change of the sets of measurable events in
terms of information quantity. A filtered c-algebra is an increasing sequence of Borel c-algebras
with Ft CF for each t and ti < t2 ^ Fti C Ft^.
3 A weaker concept states that the first two moments are the same for all t and that the covariance between xti and xt2 depends on the time interval ti — t2 [Ross, 1996].
the following probability density function (PDF) f(z) = (V^Tro^t)-1 exp[—4(£^)2]. Since the variance linearly increases in time, the Wiener process is non stationary. Stochastic differential equations (SDEs) frequently introduce uncertainty through a simple Brownian motion and are defined by
dxt = ndt + adzt,
where the constant i is the drift rate, a2 the variance rate of xt (a denotes the diffusion rate), dt a short time interval, and dzt the increment of the Brownian motion. The Figure 1 (a) shows three different realizations B\, B2, B3 of the Brownian process. The deterministic part is clearly governed by the ordinary differential equation (ODE) yt = i which solution is linear in time.4
Fig. 1 : Standard Brownian motions
Example (total factor productivity). Let us consider a differential representation for the production technology [Walde, 2006]. With an AK technology, we have Yt = F(At,K) = AtK, where Yt and At denote continuous functions of time t (a more convenient notation), where A states for the total factor productivity (TFP), Y, the
4 The calculations use the packages Statistics* ContinuousDistributions, StochasticEquations‘ EulerSimulate of Mathematica 5.1 and Itovns3 [Kendall, 1993].The primitive of EulerSimulate is EulerSimulate [drift, diffusion, {x, x0}, {duration, nsteps}]. It returns a list of simulated values for the corresponding Ito process. More generally , a system of Ito processes can also be simulated by specifying the drift vector and the matrix of diffusion. The Mathematica package ItosLemma implements Ito’s lemma for stochastic multidimensional calculus, computing stochastic derivatives and Ito—Taylor series. The primitive Itomake[x[t], ^,a], where ^ is the drift and a the diffusion, stores the rule x[t + dt] = x[t\ + ^dt + adBi.
output and K, a fixed amount of global productive factors. Suppose TFP grows at a deterministic rate g with At = gAt dAt = gAtdt. We have the differential
dF (At ,K)= FAdAt + Fk dK.
We easily deduce the growth of Yt as Yt = gKAt. The evolution in time is
Yt = gK J Asds.
A more realistic situation consists in the introduction of the uncertainties that may affect the TFP. Let suppose that At will be driven by a Brownian motion with drift such as dAt = gdt + adzt, where g and a are constants. Solving the SDE, we have At = A0 + gt + azt. The time evolution of Yt is given by
Yt = AoK + gKt + aKzt.
In this example5, the evolution of the output consists of two parts: a deterministic trend and a stochastic deviation component from the trend. However, since Yt may be negative, we have to look for another specification.
A RV may also evolve according to a geometric Brownian, such as
dxt
----= iiat + aazt dxt = /j,xtat + axtdzt.
xt
The Figure 1 (b) shows three different realizations b\,b2, b3 of such a geometric Brownian process. The deterministic part is governed by the ODE Yt = aYt, which solution is clearly exponential in time.
Definition 5. (one-dimensional Ito processes [0ksendal, 2003]). Let Bt be a one-dimensional Brownian motion on the probability space (Q, F, P). A stochastic Ito integral is a stochastic process xt of the form:
xt = xo + us(u)ds + Vs(u)dBs,
Jo Jo
that
P{/0 vs(^)2ds < to for all t > 0} = 1
and P{ f |ws(w)|ds < to for all t > 0} = 1.
J o
5 Walde [W(ilde, 2006] also gives another specification where the drift rate is AK and the diffusion rate aK. The solution takes the form
Yt = Yo + AKt + aKzt.
Theorem 1 (the one-dimensional Ito formula [0ksendal, 2003]). Let xt be an Ito process given by dxt = udt + vdBt. Let a twice continuously differentiable function g(t,x) G C2([0, to) x R) on [0, to) x R. Then yt = g(t,xt) is again an Ito process, and
dyt = ^(t,xt)dt + ^(t,xt)dxt + xt){dxtf,
where (dxt)2 = (dxt).(dxt) is calculated according to the multiplication rules dt.dt = dt.dBt = dBtdt = 0, dBt.dBt = dt.
Proof.
See 0ksendal [0ksendal, 2003], p.46 (with slightly different notations).
Example (Total factor productivity (continued) [Walde, 2006]). Let TFP follow a geometric 6 Brownian motion
dAt
—— = gat + aazt,
At
where g and a are constants. Hence, we have
ft dAs
-r-=gt + azt, z0 = 0. (1)
0 As
To evaluate the integral on the LHS, the Ito formula is used for the logarithmic function g(t,x) = ln x, x > 0. We have
dg(t, x) = gtdt + gxdx + ^gxx(dxf,
where dx = gxdt + axdz. Since (dx)2 = g2x2(dt)2 + a2x2(dz)2 + 2gadtdz = a2x2dt, we deduce
dg(t, x) = gtdt + gxdx + —gxxa2x2dt.
We then obtain
d\nAt = —d,At + =
dAt 1 2 a 2 ,n 1 2 ,n
= -, / so cr Atat = —-------a at.
At 2 A(t)2 4 At 2
Hence, = din At + 5a2dt. The expression of the integral is then
ft dAs 1 At 1 2 -r-1 = In -r- + -v t.
Jo As Ao 2
The process rather describes the rate of change of a RV, than the random variable itself.
6
From (1), we then deduce the time evolution of the TFP: At = Aoexp[(<? — ^a2)t+ +vzt\.
1.2. Poisson Process
The occurrence of discrete events at times t0,ti,t2,... (e.g. innovations) are often modeled as a Poisson process. For a Poisson process, the time intervals Ati = ti —t0, At2 = t2 — ti,... between successive events are independent variables drawn from an exponential distributed population. The parameterized PDF is given by f (x; A) = Ae-Aa: for some positive constant A. Suppose a system that starts in state 0 at initial time t0. It will change to state 1 at some time t = T, where T is drawn from an exponential distribution. The probability that the system will be in state 1 at time ti is given by the integral
/■ ti
Pi(ti) = Ae-Xt dt = 1 — e-Ati.
J 0
The probability of the system still being in state 0 is the complement P0(ti) = e-Ati. The absolute rate of change of being in state 1 is = \e~xt. We deduce an exponential transition with rate A such as
^ = AP0. (2)
More generally, for any number of states, a system of differential equations such as
(2) will describe the probabilities of being in each state. Since the transition time from the state Pn to Pn+i is exponential for all n, a Poisson process will be deduced. Schematically, it is illustrated by the chain of the Figure 2, where Pj is the probability
Fig. 2: Poisson process at constant rate
of the jth state when j events have occurred. The initial conditions are such that P0(0) = 1, Pj(0) = 0 for all j > 0. We have to determine Pn(t). Since the transitions are exponentially distributed, we have the system
^ = -AP0, ^ = APo - APi,..., ^ = APn_i - APn. dt dt dt
Given the initial condition P0(0) = 1, the solution of the first equation is P0(t) = e-At. Substituting this result into the second equation, we have the ODE
— + APi = \e~xt. dt
Solving the ODE, we obtain 7
Pi(t) = (At)e—Xt.
Continuing by substitution, we have
Mt) = Pn{t) = fe-Xt.
2! n!
In this simple counting Poisson process, the probability Pn(t) then expresses that exactly n events have occurred at time t. The expected number of occurrences by time t is
___ °° ( \ j.\n ° ( \-f-\n — i
E[n,t] = J2nPn(t) = = e~Xt(Xt) J2 (n-i)\ =xt■
n = 0 n=i n=i
Definition 6 (Poisson process). A stochastic process qt is a Poisson process with arrival rate 8 A if: (i) q0 = 0, (ii) the process has independent increments, and (iii) the increments qT — qt (or jumps) in any time interval t — t is Poisson distributed with mean A(t — t), say qT — qt ^ Poi(A(T — t)).
The probability that the process increases n times between t and t > t is given
by
P{<Zr -qt = n} = n = 0, 1, ....
n!
The SDE of a standard Poisson process is
dxt = adt + bdqt, where the increment dqt is driven by
f0 w.p. 1 — Adt, qt [1 w.p. Adt,
The Figure 3 illustrates two situations: in figure (a) jumps have the same amplitude, in the second (b) jump amplitudes are random. If no jump occurs (dq = 0), the variable will follow a linear growth xt = x0 + at.When a jump occurs, xt increases by b. The Figure 3 (b) shows an extension of the Poisson process where the amplitude of the jumps bt are governed by some distribution, such as bt ^ N(^, a2).
7 The solution of the generalized ODE (in usual notations) with variable parameters x + a(t) = b(t) is given by
/»-f
f? a(u)du
x(t) = e- ^ aWds{c + J Jl a(u)dub(s)ds}
' 1
where C is a constant of integration. The solution for the process is achieved by setting x = P1, a(t) = A and b(t) = Xe-Xt.
8 A high arrival rate means that the process jumps more often.
J" ^r
Jo-
T<j ] X0+^I t Jcr
arrivals — " arrivals
(a) Constant jump amplitude
(b) Random jump amplitude
Fig. 3 : Poisson processes with constant and random jump amplitude
Lemma 1 (Change of Variable Formula CVF). Let xt be a Poisson stochastic process given by
dxt = a(t, xt, qt)dt + b(t, xt, qt)dqt.
Let a twice continuously differentiable function F(t, x) G C2([0, to) x R) on [0, to) x R, the differential is
dF (t,xt) = (Ft + Fxa(.))dt + {F (t,xt- + b(.)) — F (t— ,xt)}dqt,
where t— denotes a date that precedes a jump at time t.
Example (Total factor productivity (continued) [Walde, 2006]). Let TFP follow a geometric Poisson process
dAt
—— = gelt + adqt,
At
where g and a are constants. Hence, applying the Ito’s lemma for Poisson processes, we obtain the solution 9
At = A0 exp[gt + (qt — q0)ln(1 +a)].
The TFP will then follow a deterministic exponential trend and have a stochastic deviation, given by (qt — q0) ln(1 + a), a > 0.
9 The proof of lemma 1.9 is shown in appendix A. The detailed calculations for this example are given in the same appendix.
1.3. Queueing models
Queueing models are a typical application of exponential transitions and Poisson processes. Some events occur at some constant rate A and are treated at a constant rate /i. Let us consider a M/M/1 queue, where the first M states for memoryless arrivals (i.e. inter-arrivals times of occurring events are often modeled as exponentially distributed variables), the second M is the same for departures and the 1 states a single server. The chain may by represented schematically by the Figure 4.
Fig. 4: Queues with a single service M/M/1
The system of dynamic equations is:
= —APq + A4P-i = APt — ^P- — A4-P + A4p2i • • • i
dPn
^ = APn_i — A Pn — [iPn + AtPn+i-
The steady state probabilities, once the system has stabilized at the equilibrium, are characterized by the probability that exactly n events occur and by the expected number of presently waiting eventsi0
Pn = (1 - -)(-)" and E[n] = f>Pn =
i i n=0 1 — A/i
2. Stochastic control problem and differential games
There are two particular ways to introduce uncertainty in the differential games. The first way is based on piecewise deterministic processes, where the system switches
10 In the steady state, the derivatives vanish and we deduce the geometric series Pi = -Po,Pl = (-)2Po,...,P„ = (-),1Po,...
which converges only if the rate of arrivals A is less than the rate v of processing. Under this condition
we have
P0{1 + (-) + (-)2+ + ...} = Po(l - -r1 = 1.
V V V V
The expression of Pn will then follow from Po = 1 — (A/v).
from one deterministic mode to another, at random jump times. The second way consists of introducing continuous stochastic noise processesii.
2.1. Optimal control under uncertainty
The uncertainty may take the form of a Brownian motion (also Wiener process). This process will generally influence the evolution of the state variable. This evolution will be described by an SDE of the form [Dockner, 2000]:
dxt = f (t, xt, ut)dt + a(t, xt, ut)dwt, x0 given, (3)
where xt denotes the n-dimensional vector of the states and ut - an m-dimensional vector of controls. A k-dimensional Wiener process wt is a continuous-time stochastic process, such as w : [0,T) x S ^ Rfc, where S denotes the set of points £ of possible realizations of the RV. The functions are such that f : Q = {(t,x,u)\t G [0,T),x G X,u G U(t,x)} ^ Rn and a : Q ^ Rnxfci2. A solution xt of the SDE (3) must satisfy the following integral equation
xt = / f (s,xs ,us)ds + a(s,xs,us)dws
00
for all £ of w(s, £) in a set of probability 1i3 footnote Any solution to the SDE is a stochastic process depending on the realizations of £ G S. The correct notation is rather x(t, £) than x(t) or xt, showing that the value of xt cannot be known, without knowing the realization of £ (see [Dockner, 2000], p.228)..
Lemma 2 (Ito’s lemma). Suppose that xt solves the SDE (3). Let G : [0,T) x X ^ R be a function with continuous partial derivatives Gt,Gx,Gxx. The function g(t) = G(t, xt) will satisfy the SDE:
dg(t) = {Gt(t, xt) + Gx(t, xt)f (t, xt, ut) +
ut).a(t, xt, u
+ Gx(t, xt)a(t, xt, ut)dwt.
+ T^tr[Gxx(t, xt)a(t, xt, ut).a(t, xt, ut)']}dt+
11 Memoryless Poisson models of patent race are associated with Dasgupta and Stiglitz [Dasguspta, 1980], Lee and Wilde [Lee, 1980], Loury [Loury, 1979], Reinganum [Reinganum, 2004], [?]. The probability to innovate and to obtain a patent depends on the current R&D investment. Reinganum [Reinganum, 1982] and Yeung and Petrosyan [Yeung, 2006] consider non cooperative and cooperative games.
12 An entry Gij of this n x k matrix evaluates the direct impact of the jth component of the k-dimensional Wiener process on the evolution of the ith component of the n-dimensional state vector.
13 The second integral is such as lim^^o £i-—! a(t{, xti, utl ){wtl+1 — w(tl}, where 0 = t1 < t2 < • •• <ti = t and 8 = max{|t;+1 — t\\, 1 < l < L — 1}
The stochastic control problem is given by
/■T
E«(.)[/ F(t,xt,ut))e rt dt + e rTS(xt)],
0
F (t,xt,ut ))e Tt 'U ' " rT ‘
0
s.t. (4)
dxt = f (t,xt,ut)dt + a(t,xt,ut)dwt, x0 given,ut G U(t,xt).
The following optimality conditions are based on the Bellman equation (HJB).
Theorem 2 (Optimality conditions). Let a function be defined as V : [0,T) x X ^ R with continuous partial derivatives Vt, Vx and Vxx. Assume that V satisfies the HJB equation:
rV(t, xt) — Vt(t, xt) = max {F(t, xt, u) + Vx(t, xt)f (t, xt, u)
+ 7^tr[Vxx(t,xt)a(t,xt,ut).a(t,xt,uty]\ut G U(t,xt)}, (5)
for all (t, xt) G [0, T) x X. Let $(t, xt) be the set of controls maximizing the RHS of (5) and ut be a feasible control path with state trajectory xt s.t. ut G $(t, xt) holds a.s. for a.a. t G [0,T):
(i) if T < to and if the boundary condition V(T, x) = S(x) holds for all x G X then u(.) is an optimal control path;
(ii) if T=to and if either V is bounded and r > 0; or V is bounded below with limt^TO e-rtEU(.) V(t, xt) < 0 holds, then u(.) is a catching up optimal control path.
Proof. See Dockner et al. [Dockner, 2000], p.229-30, Yeung and Petrosyan [Yeung, 2006], p.16 with different notations.
2.2. Differential games with random process
Let us consider a N-players game. The control variable by the ith player is denoted by ut at time t for i G{1, 2,...,N}. The vector of controls by the opponents of player i will be u— = {uj, u^,..., uit-i, u\+1,..., uN}. The controls are subject to the constraints u\ G Ul(t,xt,u-'1) C Rm*, where the xt G X are the state variables of the system. The state equation for the game will then be given (by omitting the time index of arguments)
dxt = f (t, x, u1 , ..., uN)dt + a(t, x,ui,..., uN)dwt,
where wt is a k-dimensional Wiener process. The functions f and a are both defined on Q = {(t,x,ut,u-i)\t G [0,T),x G X, ul G Ut(t,x,u~t)} with values in Rn and Rnxk respectively. The objective of each player is to maximize the expectation of his discounted flow of payoffs
(u( )) = EU() {{ Fl(t,xt,ul,u~l)e~ritdt + e-ritSi(xT )\x0 given},
where F* is a real-valued utility function defined on S* a real-valued scrap value
function defined on X, and r* the discount rate.
2.3. Piecewise deterministic control problem
Let us consider an autonomous problem defined over an unbounded time interval [0, to). The evolution of the system may be deterministic, except at certain jump times given by the finite set {T\,T2,... ,TM}. At each of these dates, the system switches from one mode to another. The following description is inspired from Dock-ner et al. [Dockner, 2000]. Let X C R" denote the state space and xt G X the state at time t. The set of controls, when the current mode is h G M, is given by U(h,xt) C Rm.
The motion is described by the differential equation Xt = f(h,xt,ut) where f (h,.,.) maps fi(h) = {(x, u)|x G X,u G U(h, x)} into R". The instantaneous payoffs of the player consist of F(h,xt,ut), a real-valued function defined on 0(h), and the lump sum payoff Shk (xt), when a jump occurs from mode h to mode k, (k = h). The payoffs are discounted at the constant rate r > 0. The motion of the system mode is a continuous-time stochastic process h : [0, to) x S ^ M, where the set S of points £ represents realizations of some random variable. Thus, the event, that the mode is h at time t, is {£ G S|ht(£) = h} and its probability is denoted by P{ht(£) = h}. The probability that the system switches from mode h to mode k during the time interval (t, t + At] is proportional to the length of A. We have
limA^o^-lP{^+A = k\ht = h} = qhk{xt, ut)}, k ^ h,
where qhk : fi(h) ^ R+. The risk neutral player seeks to maximize the expectation of the discounted payoff flow, conditional on the initial state and mode. Initially, we
state
J(u( -)) — E«( . )[ / F(hti xt,ut)e dt+
' Jo
^53 ShT_ hTl (xti )e~rTl lxo,ho given],
lEN
where Ti denotes the Ith jump and hTx_, the mode immediately before the switch. We also have
J(u(.)) = E„( ) [ {F(ht,xt,ut)+
Jo
+ qhtk(xuut)Shtk(xt)}e~rtdt l xo,ho given]. (6)
k=ht
Definition 7 (feasible path). Given S a set of points £ G S representing possible realizations of a random variable, the control path u : [0, to) x S ^ Rm is feasible
for the stochastic control problem if the following conditions are satisfied: (i) it is non-anticipating, (ii) the constraints xt G X and u(t) G U(ht,xt) are a.s. verified, (iii) the process (h(.),x(.)) and the integral in (4) are well defined. The control path is optimal, if it is feasible and if J(U(.)) > J(u(.)) for all feasible paths.
Dockner et al. [Dockner, 2000] p.206-7, deduce the fundamental theorem.
Theorem 3 (optimal control path). Let us consider the autonomous problem 14 and suppose the existence of a bounded function V : M x X ^ R which have the following properties. The function V(h,x) is continuously differentiable in x for all h G M and is such that (omitting the time argument) the HJB equation
rV(h, x) = max {F(h, x, u) + Vx(h, x)f (h, x, u)+
+ ^ qhk(x,u)[Shk(x) + V(k,x) - V(h, x)]|u G U(h,x)}, (7)
k=h
is verified for all (h,x) G M x X. Let $(h, x) be the set of controls maximizing the RHS of (7). Then the control path ut is optimal if the following conditions are satisfied: the control u(.) is feasible and u(.) G $(h(.),x(.)) holds a.s.
Proof. See Dockner et al. [Dockner, 2000], p. 207.
2.4. Piecewise deterministic differential games
Let a N-players autonomous game be a piecewise deterministic differential game over an infinite time horizon. Denote by u\ the control value by player i and
^ —i /1 ^ i— 1 ^ i+1 „ N\
ut = (ut ,■ ■ -,ut ,ut ,---,ut )
the vector of controls of the opponents of player i. The player’s i set of feasible controls is
Ui(ht,xt,u-i) C Rm*,
when the system is in mode ht G M and state xt G X. The objective functional of the ith player is given by
Ji(uU) = E„C) [ Fi(ht, xt, ut)e-ritdt
Jo
+ 5Z Shel_ hTl xTi e-riTl 1 x0,h0 givenL
lEN l
where Sh(t hT (xti ) denotes the payoff received if a jump occurs at time Ti from hT__ to h^. Suppose as in [Dockner, 2000], that all players use a stationary Markov
14 Dockner et al. [Dockner, 2000] also consider the non-autonomous problem.
strategy of the form uit = 4,l(ht,xt) then the player i’s optimal control problem is of the form
max«i) J_i (u(.)) = E«(.)[ F4,_i (ht,xt,ut)e~ritdt
( ■ ) Jo
+ E ShT_ ,hTl (xTi )e-riTl 1 x0,h0 given],
lEN
s.t.
xt = f_i (ht,xt,uгt),
xo given, ut G Ui_i(ht, xt),
where the piecewise deterministic process h(.) is determined by the initial condition h0 and the switching rates q^-i hk(xt,ult). The functions F^ and f _i have the same pattern as
F^_i (h, x, u1) = Fl(h, x, (h, x), ■ ■ ■ , $i-1(h, x), u, $i+1 (h, x), ■ ■ ■ , $N(h, x)) ■
The functions U^_i and q^_i hk are defined by
Ui_i(h, x) = Ul(h, x, 4>1(h, x), ■ ■ ■, 4>i-1(h, x), $i+1(h, x), ■ ■ ■, $N(h, x)),
qi _i,hk(x, ui) = q (x, $^(h, x), ■ ■ ■, $г-^(h, x), uг, 0l+1(h, x), ■ ■ ■, $N (h, x)) ■
Ddefinition 8 (stationary Markov-Nash equilibrium). A N-tuple of functions $i : M x X ^ Rmi, i = 1, ■ ■ ■ ,N is a stationary Markov-Nash equilibrium of the game
r(ho, xo) if an optimal control path u( ) exists for each player i. If ($1, ■ ■ ■ , $N) is a
stationary Markov-Nash for all games T(h, x), then it is sub-game perfect.
Theorem 4 (stationary Markov-Nash equilibrium). Let us consider a given N-tuple of functions $i : M x X ^ Rmi, i = 1, ■ ■ ■ ,N and assume that the piecewise deterministic process defined by state motion
xt = f (ht, xt, 4>l(ht, xt), ■ ■ ■, $N(ht, xt)) and the switching rates
qhk(xt, 4>l(ht, xt), ■ ■ ■, $N(ht, xt))
is well defined for all initial conditions (ho,xo) = (h, x) G M x X. Suppose the existence of N bounded functions V1 : M x X ^ R,i = 1, ■ ■ ■ ,N such that Vi(h, x) is continuously differentiable in x and such that the HJB equations
riVi(h, x) = max {F^_i (h,x,ul) + VX(h, x)f _i (h, x, u1)
+ E _i,hk(h, ui)(Sihk(x) + Vi(k, x) - Vi(h, x))| u G UI_(h, x)}, (8)
k=h
are satisfied for all i = 1,N and all (h, x) G M x X. Denote by $*(h, x) the set of all ul G U^(h,x) which maximize the RHS of (8). If (h,x) G $*(h, x) and all (h,x) G M x X then (j)1, . ) is a stationary Markov-Nash equilibrium. The
equilibrium is sub-game perfect.
Proof. See Dockner et al.[Dockner, 2000], p.212.
3. A stochastic game of R&D competition
The following game is initially due to Reinganum [Reinganum, 1982] and has been detailed by Dockner et al. [Dockner, 2000]. In this game N firms have competing R&D projects. The dynamic game supposes that: (i) no firm knows in advance the
Fig. 5 : A stochastic innovation game
amount of R&D that must be invested, (ii) R&D activities are costly but contribute to higher accumulation of common know-how and then have positive externalities
(iii) one successful innovation may be achieved by using different paths. Resources in R&D positively influence the probability of successful innovations. Once a firm has won the competition, it acquires a monopolistic position. The problem is illustrated in the figure 5.
Game presentation
The time Ti to complete a project is a RV, whose probability distribution is Fit = P{t < t}, where for convenience Fit is taken similar to Fi (t). The RVs Ti are stochastically independent, since knowledge is supposed to have no spillover between firms. Denote by t = min {t^ i = 1, 2,..., N} the date of an innovation. According to independency, we may read
N
P{t < t} = 1 -H(1 - Fit).
i=1
Let uit > 0 be the rate of R&D effort. The rate of the distribution Fi is assumed to be proportional to the R&D efforts
Fit = Auit (1 — Fit), Fi0 = 0,x > ° (9)
where 1 — Fi is the survival probability and Fit(1 — Fit)-1 is the hazard rate. Assume that the present value of the innovator’s net benefits Pj is constant and greater than the ones of the other competitors PF. The costs of R&D efforts are quadratic in the
investment rate. The game is played over a finite horizon T. This stochastic game
belongs to the class of piecewise deterministic games. The system is in mode 0 before the innovation is happens. Ones an innovation by firm i has occurred, the system switches from one mode to another mode i,i = 1, 2,...,N [Dockner, 2000].
Game analysis
The expected discounted profit to be maximized by the ith player is
f {pjFit ^(1 — Fjt) + pF E Fjt n(1 — Fkt) —
0 j=i j=i k=j
e-rt N
-----7T~uit ri(l - Fjt)}dt. (10)
j=1
This expression consists of three terms, which weights are the probabilities: firstly the firm i’s value of net payoff Pj if the firm becomes the first to innovate, secondly the firm i’s value of payoff PF if this firm looses the competition and, thirdly, the discounted value of the cost of R&D efforts. Substituting the RHS of (9) into (10) the payoff expression is simplified as follows
rT e-rt N.
{AP/Mjj + APp 'y Ujt — uit }II (1 — Fjt)dt. (11)
0 j=i j=1
Let introduce the state transformation
— ln(1 — Fit) = Azit ^ 1 — Fit = e XZit.
The state variable zit denotes the firm i’s accumulated know-how via the R&D efforts. A differentiation w.r.t. time yields i?i(1 — Fi)-1 = XZ:i. Hence, we have Z-i = uit, zi0 = 0. The corresponding payoff is deduced from (11)
fT e-rt N'
Jl= {XPruu + APp Ujt----— uft} x exp[—A Zjt\dt.
0 j=i 2 j=1
Let yt be equal to exp—X^N=1 Zjt]. The differentiation w.r.t. time yields
N
yt = —Ayt^2ujt, yo = 1. (12)
j=1
The game is then transformed to the following exponential game
(T e~rt
J = I {XPiUu -\- XPp y ^ Ujt ~ uit}ytdt, 02
0 j=i
s.t.
N
yt = —Ayt^2 uit, yo = 1. i=1
Markov—Nash equilibrium
When omitting the time argument, the current-value Hamiltonians (i = 1, 2,..., N)
are
e-rt N'
TC{t,y,Ui,Hi) = y{XPrui + A Pp -ui} ~ + y^/uj)>
j=i j=i
where /j,i,i = 1, 2,...,N are the current costate variables. The first order conditions (FOCs) are
dW,y u„K) = 0 ^ ^ = n _
m*, > <13>
• _ oH(t,y,ui,pi) m —--------------------j
uit = —btXert. (14)
We have N +1 boundary conditions15 from the state equation (12) and FOCs (13). To solve the boundary value problem let us conjecture a solution.
15 There are 2N + 1 variables namely y, ^i, ui,i = 1, 2,..., N and 2N + 1 equations.
Using the expression of ui in (13) and substituting into (12) we have yt = \2Nbtytert, y0 = 1. To hold the conjectured solution, bt must satisfy a Ricatti differential equation (RDE):
X2ert
bt =------^{(2iV - 1)6? + 26t(l - N)(Pf - PT)}, (15)
bF = -Pi .
The solution of the RDE (15) can be found by an CVF method letting g(t) = -1/bt.
j
— time time time
(a) Control variable (b) State variable (a) Costate variable
Fig. 6: Control, state, costate in an innovation game with two firms
Substituting the solution of bt into (14) yields the Markovian identical strategies ut = _ 2AP/(Pj — PF)(N — l)ert
“ (2N - 1 )P/ + {P/ + 2(N - 1 )PF} x exp[i(P/ - Pf)(N - l)A2(ert - e^)]'
The Figure 6 shows the control variable, the state and the costate variables, when the game is reduced to two competitors (N = 2).
4. A stochastic game of patent race
In some market models N incumbents and one entrant aim to invent a new process or a new product, and patent their innovations. In the model of [Dasguspta, 1980], [Reinganum, 1982], [Tirole, 1990]the patent race is to develop a cost reducing production process for an existing product, in the model of Gayle [Gayle, 2001] the firm are patenting a new product. This model is in line with the one of Loury [Loury, 1979] and its developments by Lee and Wilde [Lee, 1980] where a stochastic relationship is assumed between R&D investments and the time at which an innovation will occur.
Description of the game
An output market is composed of N +1 firms, which are attempting to produce a new product and taking simultaneously a patent16. N of these firms are incumbents and one firm is a potential entrant. The rate of investment in R&D is xi,i = 1,...,N for an incumbent, and z for the entrant. The date of success for an innovation is supposed to depend only on the R&D investment rate17 such as Ti(xi). The probability that the firm i succeeds at or before the date t is
P{Ti(xi) < t} = 1 — exp[-h(xi)t], t € [0, to),
where h(.) denotes the hazard function. Let this function be twice differentiable, strictly increasing, concave and satisfying the following conditions: h'(.) > 0, h"(.) <
0 for all xi, z € [0, to), and \imxiz^<xh'(.) = 0. The conditional probability that the firm i will succeed in the instant, given that it has not already succeeded is
P{Ti(xi) € (t,t + dt] \ri > t} = h(xi)dt, t € [0, to).
This result is due to the memoryless property of the exponential distribution18. Let fi represents the date at which the first firm introduces an innovation. Then fi = minj=i{f(xj)}, and
P{fi < t} = 1 — P{Tj > t}, for all j = i
= 1 — exp[E h(xj)]. j=i
Finally, let us suppose the following constant non-discounted profits: Pa the incumbent i ’s profit before any innovation occurs, Pw the incumbent i’s profit if he wins the patent race, Ple the incumbent i’s profit if he loses the patent race to the entrant, Pli the incumbent i’s profit if he loses the patent race to another incumbent, and Pe the profit of the entrant when he wins the patent race.
The expected profits of incumbents i, i = 1, 2,...,N are given by:
pW p LE W pLI
x {P/4 — Xi + h(xi)—---1- h(z)—----H h(xk)——} x e~rtdt. (16)
k=i
16 The following game of patent race is inspired from the presentation of Gayle [Gayle, 2001].
17 Reinganum [Reinganum, 1982] [?] assumes that the probability of introducing an innovation is a function of both the current of investment in R&D and the accumulated stock of technology.
18 Let the PDF of the R.V. T be f (t)dt = \e-Xtdt. We have the moment E[Tk ] = f™ tk f (t)dt. Then E[Tk] = r(fc + 1)A-fc = X-k k!, since r(k) = e-uuk-1du = (k - 1)!. Hence, E[r] = A-1.
The expected profit of the entrant is
N
VE = exp[—{h(xi) + h(xk) + h(z)}t] x {h(z)-----------------------z}xe~rtdt. (17)
•^0 k=i r
Integrating (16) and (17), we have the system for the R&D subgame
oW jjLE _ \t jjLI
Vi _ pj - xi + Hxi)^r + Kz)^~ + h(xk)^r-
r + h(xi) +J2k^iHxk) + h(z) '
ve_ Kz)^-z
r + h(xi) + Y,N=i h(xk) + h(z)
Best response functions
The reaction functions (BRFs) are then deduced from the following N +1 FOCs
dVi , dVE
—— =0, i = I,..., N and — = 0.
dxi dz
A Nash equilibrium of R&D spending must then satisfy the following conditions (omitting the nonzero denominator of the LHS):
N pW
{r + h(xi) + ^2 Hxk) + Kz)} x {h'{xi)-^----------1)-
k=i
pW pLE N pLI
- {PtA -Xi + h(xi)-^— + h(z)—r-------\-^h(xk)-y-}h'(xi) = 0, (18)
k=i
N p p
{r + h{xi) + '^h{xk) + h{z)}{ti{z)~--------1) - (h(z)—-------z)h'(z) = 0. (19)
k=i
The equation (18) is the incumbent i’s BRF with upward sloping19, given the R&D investments of his opponents. Similarly, (19) is the entrant’s BRF with upward sloping, given the R&D investments of his opponents.
19 In the case of symmetry between the BRFs of the incumbent and entrant, when N = 1, PLE = PA = 0, we have the incumbent’s BRF
F(x, z) = {h! (x) -|—h!(x)h(z)}PW — h'(x)PA — r — {/i(a?) + h(z)} + xh'(x). r
By the implicit function theorem, we deduce
r->W
dx Fz h'{z){h'{x) —-----1}
dz Fx h"(x){l + ^}PW +xh"(x)'
The derivative dx/dz is positive because of a non-negative numerator with non-negative expected profits and due to a negative denominator with the concavity of h(.).
Symmetric Nash equilibrium
Incumbent's R&D BFR's Incumbent
(b) Equilibrium's neighbourhood
Fig. 7: Symmetric Nash equilibria
A symmetry of the BRFs is achieved when N = 1,PLE = PA = 0. The BRFs are then defined by
{h'(x) -|—h!(x)h(z)}P^' — h,(x)Pfr — r — {h(x) + h(z)} + xh'(x) = 0, r
{h'(z) -|—h'(z)h(x)}PE — r — {h(x) + h(z)} + zh'(z) = 0. r
The Figure 7 (a) shows the reaction functions and the Nash equilibrium. Due to the stability condition, the incumbent’s reaction function is steeper than the entrant’s one20Both reactions functions intersect on the 45° line. The expected pre-innovation profit PA only occurs in the incumbent’s BRF. A positive PA will then shift the incumbent’s BRF to the left, as it is shown in figure Fig.(b). As a consequence to the shift, we observe with [Gayle, 2001] that the R&D spending equilibrium of the incumbent will be less than that the one of the entrant.
5. Concluding remarks
20 The stability condition is expressed by
\Vix\ > IVXz I and \VZEZ\ > \VZEX\ in the neighborhood of the equilibrium (see Figure 7 (b)).
The differential games in R&D economics give some indications and results about debates and questions, like the relation between the concentration of an industry and the intensity of R&D investments. The consequences of the market structures (socially managed market, pure monopolist with barriers to entry, competitive economy) on R&D have been studied notably by Dasguspta and Stiglitz [Dasguspta, 1980]. For example, the correlation between concentration and R&D efforts depends upon the existing degree of concentration and upon free entrance. For Loury [Loury, 1979], given a market structure, firms will rather more invest in R&D than is socially optimal. Reinganum [Reinganum, 1982] considers the optimal resource allocation in R&D, the social optimality of the game and implications of innovation policies (taxes and subsidies, patents). Noncooperation or cooperation aspects also play a great role. If costs are not too high, a firm may choose to cooperate in R&D investments with one or some other firms. Dasguspta and Stiglitz [Dasguspta, 1980], Reinganum [Reinganum, 1982], Amir, Evstigneev, Wooders [Amir, 2003] have studied such cooperations. Yeung and Petrosyan [Yeung, 2006] consider the theory of the cooperative stochastic differential games and analyze a cooperation R&D game under uncertainty over a finite planning period. This paper has introduced to the approach of stochastic games in theory and application. Two simple examples of noncooperative games in R&D have been developed. The first example of stochastic game belongs to the class of piecewise deterministic games. Then, when an innovation occurs, the system switches from one mode to another. Indeed, the winning firm is supposed to acquire immediately a monopolistic position. The second example considers a stochastic game between several incumbents of an industry and a potential entrant. In this model, the R&D spending equilibrium of the representative incumbent (the game is symmetric) is rather less than the one of the entrant. Recent papers relax or improve some strong assumptions. Sennewald [Sennewald, 2007] relaxes the strong assumption of boundedness conditions when applying the Bellman equation (bounded utility function, bounded coefficients in the differential equation). The HJB can then still be used with linearly boundedness. Let us also indicate the less recent model of endogenous growth by Aghion and Howitt [Aghion, 1992]. In this model, according to a forward-looking difference equation, the research expenses in any period depend upon the expected research investment next period.
Appendix
Ito’s lemma for Poisson Processes
Ito’s lemma for Poisson processes is not frequently presented in textbooks. This appendix shows the corresponding rule, its proof and one application to TFP.
Lemma 3 (Ito’s lemma for a Poisson process). Given a Poisson SDE
dxt = adt + bdqt, where a and b are constant. (20)
Let F(t,xt) be a continuously differentiable equation of t and x. Then, we have
dF(t, xt) = (Ft + aFx)dt + {F(t, xt + b) — F(t, xt)}dqt. (21)
Proof. By differentiating the function F(t,xt) and using equation (20), we have
dF(t, xt) = F(t + dt, xt+dt) — F(t, xt) = (22)
= F(t + dt, xt + adt + bdqt) — F(t,xt).
Adding and subtracting F(t + dt, xt + adt) in equation (22) we have
dF (t,xt) = F (t + dt,xt + adt + bdqt) — F (t + dt,xt + adt) —
—F(t, xt) + F(t + dt, xt + adt).
The last two terms of equation (23) correspond to a situation where we have no jump. Hence, with xt = at and dxt = adt, we have
F(t + dt, xt + dxt) — F(t, xt) — Ftdt + Fxdxt — = (Ft + aFx)dt.
According to equations (23) and (24), we deduce
dF(t, xt) = F(t + dt, xt + adt + bdqt) — F(t + dt, xt + adt) +
+ (Ft + aFx )dt.
(24)
(25)
The first two terms of equation (25) have a different expression according that a jump may or not occur. We have
F(t + dt, xt + adt + bdqt) — F(t + dt, xt + adt) =
JF(t,xt + b) — F(t,xt) w.p. Adt, with jump
1 0 w.p. 1 — Adt, without jump
= {F(t, xt + b) — F(t, xt)}dqt.
Then, we find the Ito’s formula for a Poisson stochastic differential equation
dF (t,xt) = (Ft + aFx)dt + {F (t,xt + b) — F (t,xt )}dqt.
Example (Total factor productivity). The stochastic differential equation is given by [Walde, 2006])
= gdt + adqt, (26)
At
where g and a are constants. The equation (26) is equivalent to
dAt = gAtdt + aAtdqt.
In this example, the parameters of the stochastic differential equation (20) are a = gAt = at and b = aAt = bt. Let us apply the Ito’s formula (21). Taking F(t,At) = ln At, we have Ft = 0 and Fa = 1/At. We also determine
ln( At + aAt) — ln At = ln(1 + a).
Then according to Ito’s formula, we have
d ln At = g + ln(1 + a)dqt. (27)
By integrating both sides of equation (27), we obtain
f ln As = gt + ln(1 + a) f qs.
00
Hence,
[ln As]0 = gt + [qs ]0,
and
ln At = ln Ao + gt + ln( 1 + a)(qt — qo). The path of the total factor productivity is then given by
At = Ao exp[gt + ln(1 + a)(qt — qo)].
References
Aghion P., Howitt P. 1992. A Model of Growth through Creative Destruction. Econometrica, 60 (2): 323-351.
Allen E. 2007. Modeling with Ito Stochastic Diffrential Equations, Dordrecht. The Netherlands. Springer.
Amir R., Evstigneev, Wooders J., Noncooperative versus Cooperative R&D with Endogeneous Spillover Rates. Games and Econ. Behav, 42: 183-207.
Dasguspta P., Stiglitz J. 1980. Uncertainty, Industrial Structure and the Speed of R&D. Bell J.Econom., 11: 1-28.
Dockner E., Jorgenson S., Long N.V.,Sorger G., 2000. Differential Games in Economics and Management Science. Cambridge. Mass., Cambridge University Press.
Friedman A., 2004 Stochastic Differential Equations and Applications. New York. Dover Publications Inc.
Gayle P.G., 2001 Market Structure and Product Innovation. Boulder (Colorado). University of Colorado; 01-15.
Gross D., Harris C.M. 199S. Fundamentals of Queueing Theory. ed. 3. New York. Wiley & Sons
Kendall W. S. 1993. Itovsn3: Doing Stochastic Calculus with Mathematica. In: H.R. Varian, Edit, Economic and Financial Modeling with Mathematica. New York. Springer-Verlag; 214-23S.
Kythe P.K., Puri P., Schaferkotter M. 2003. Partial Differential Equations and Boundary Value Problems with Mathematica. ed. 2. New York. Chapman & Hall/CRC.
Lee T., Wilde L. 19S0. Market Structure and Innovation : A Reformulation. The Quart. J. Econ., 194: 429-436.
Loury G.C. 1979. Market Structure and Innovation, The Quart. J. Econ., 93: 395-410.
Malliaris A.G., Brock W. 19S4. Stochastic Methods in Economics and Finance. Amsterdam, New York, Oxford, North Holland.
0ksendal B. 2003 Stochastic Differential Equations : An Intoduction with Applications. Berlin Heidelberg. Springer-Verlag.
Reinganum J.F., 19S2. A Dynamic Game of R and D : Patent Protection and Competitive Behavior. Econometrica, 50 (3): 671-6SS
Ross S. 1996. Stochastic Processes. ed.2. New York. John Wiley & Sons.
Sennewald K. 2007 Controlled Stochastic Differential Equations under Poisson Uncertainty and with Unbounded Utility. J.Econ.Dynam.Control, 31: 1106-1131.
Tirole J. 1990 The Theory of Industrial Organization. Cambridge. Mass. London. The MIT Press.
Walde K. 2006 Applied Intertemporal Optimization. lecture notes. University of W-rzburg. Germany. http://www.waelde.com/aio.
Wolfram S. 2003. The MATHEMATICA Book. ed. 5. Champain U. Wolfram Media Inc.
Yeung D.W.K., Petrosyan L. 2006, Cooperative Stochastic Differential Games. New York. Springer Science. Business Media Inc.
