CC BY
3Contributions to Game Theory and Management, XI, 103-112
Nash Equilibria in Mixed Stationary Strategies for m-Player Mean Payoff Games on Networks
Dmitrii Lozovanu1 and Stefan Pickl2
1 Institute of Mathematics and Computer Science of Moldova Academy of Sciences,
Academiei 5, Chisinau, MD-2028, Moldova, E-mail: lozovanu@math.md
2 Institute for Theoretical Computer Science, Mathematics and Operations Research,
Universität der Bundeswehr München, 85577 Neubiberg-München, Germany, E-mail: stefan.pickl@unibw.de
Abstract We consider a class of non-zero-sum mean payoff games on networks that extends the two-player zero-sum mean payoff game introduced by Ehrenfeucht and Mycielski. We show that for the considered class of games there exist Nash equilibria in mixed stationary strategies and propose an approach for determining the optimal strategies of the players. Keywords: mean payoff game, pure stationary strategy, mixed stationary strategy, Nash equilibria
1. Introduction
In this paper we consider a class of m-player mean payoff games on networks that generalizes the following two-player zero-sum mean payoff game introduced by Ehrenfeucht and Mycielski, 1979, and considered by Gurvich et al., 1988.
Let G = (X, E) be a finite directed graph in which every vertex x e X has at least one outgoing directed edge e = (x, y) e E. On the edge set E it is given a function c : E ^ R which assigns a cost c(e) to each edge e e E. In addition the vertex set X is divided into two disjoint subsets X^d X2 (X = Xi U X2, Xi n X2 = 0) which are regarded as position sets of the two players. The game starts at a given position x0 e X. If x0 e X1 then the move is done by the first player, otherwise it is done dy second one. Move means the
x0 xi
e0 = (x0,x1) e E. After that if x1 e X1 then the move is done by the first player, otherwise it is done by the second one and so on indefinitely. The first player has
1 t_1
the aim to maximize lim inf - V^ c(eT) while the second player has the aim
t^w t —^
t=0
1 t_1
to minimize lim sup - V^ c(eT). Ehrenfeucht and Mycielski, 1979, proved that
T=0
for this game there exists a value v(x0) such that the first player has
cl StFclt-
11_1
egy of moves that insures lim inf - V^ c(eT) > v(x0) and the second player
t^W t ^-^
T=0
1t-
has a strategy of moves that insure lim sup - ^^ c(eT) < v(x0). Furthermore
t^w t T=0
Ehrenfeucht and Mycielski, 1979 and Gurvich et al., 1988 showed that the players can achieve the value v(x0) applying the strategies of moves which do not de-t
Ehrenfeucht and Mycielski, 1979 and Zwick and Paterson,1996, called such strategies positional strategies. Gurvich et al., 1988 and Lozovanu and Pickl, 2006, called these strategies stationary strategies.
In this paper we will regard such strategies as pure stationary strategies because each move in a position of the game is chosen from the set of feasible strategies of moves by the corresponding player with the probability equal to 1 and in each position such a strategy does not change in time.
A generalization of the zero-sum mean payoff game to rn-player, where m > 2, is the following. Consider a finite directed graph G = (X, E) in which every vertex has at least one outgoing directed edge. Assume that the vertex set X is divided into m disjoint subsets Xi, X2,..., Xm (X = Xi UX2 U • • • U Xm; X, nXj = 0, i = j)
m
the edge set m functions c® : F ^ R, i = 1,2,..., m are defined that assign to each directed edge e = (x, y) e E the values c^ c;;,..., c^™ that are regarded as the
1, 2, . . . , m.
Gm given position x0 e X at the moment of time t = 0 where the piayer i e {1, 2,..., m} who is owner of the starting position x0 makes a move from x0 to a neighbor position x1 e X through the directed edge e0 = (x0, x1) e E. After that players 1,2,..., m receive the corresponding rewards c^, c^,..., c^. Then at the moment of time t = 1 the player k e {1, 2,..., m} who is owner of position x1 makes a move from x1 to a position x2 e V through the directed edge e1 = (x1, x2) e E, players 1,2,..., m receive the corresponding rewards c^, c^,..., c^, and so on, indefinitely. Such a play of the game on G produces the sequence of positions x0, x1, x2,..., xt... where each xt is the position at the moment of time t. In this game the players make moves through the directed edges in their positions in order to maximize their average rewards per move
The game formulated above in the case m = 2 and c1 = —c;; = ce, Ve e E is transformed into a two-player mean payoff game for which Nash equilibria in pure stationary strategies exist. In general a non-zero-sum mean payoff game may have no Nash equilibrium in pure stationary strategies. This fact has been shown by Gurvich et al., 1988, where an example of two-player non-zero-sum mean payoff game that has no Nash equilibrium in pure strategies is constructed. Alpern, 1991
m
m
mean payoff games for which Nash equilibria in pure stationary strategies exist is presented in Lozovanu and Pickl, 2006.
In this paper we consider the non-zero-sum mean payoff games in mixed stationary strategies. We define a mixed stationary strategy of moves in a position x e X, for the player i e{1, 2,...,m} asa probability distribution over the set of feasible
xm Nash equilibrium in mixed stationary strategies. Based on a constructive proof of this result we propose an approach for determining the optimal mixed stationary strategies of the players.
T=0
The paper is organized as follows. In Section 2 a class of average stochastic positional games that generalizes non-zero-sum mean payoff games is considered. Then in Sections 3 is shown how an average stochastic positional game can be formulated in terms of pure and mixed stationary strategies. In Section 4 some results concerned with the existence of Nash equilibria in mixed stationary strategies for average stochastic positional games are presented. Additionally an approach for determining the optimal strategies of players is proposed. In Sections 5,6, based on results from the Sections 3,4, is proven the existence of Nash equilibria in mixed stationary strategies for non-zero-sum mean payoff games and an approach for determining the optimal strategies of the players is proposed.
2. A generalization of mean payoff games to average stochastic positional games
The problem of determining Nash equilibria in mixed stationary strategies for mean payoff games leads to a special class of average stochastic games that Lozovanu and Pickl, 2015 called average stochastic positional games. Lozovanu, 2018 shown that this class of games possesses of Nash equilibria in mixed stationary strategies. Therefore in the paper we shall use the average stochastic positional games for studying the non-zero-sum mean payoff games.
m
X
- a partition X = X1 U X2 U • • • U Xm where Xj represents the position set of player i e {1, 2,..., m};
- a finite set A(x) of actions in each state x e X;
- a step reward f j(x, a) with respect to each player i e {1, 2,..., m} in each state x e X and for an arbitrary action a e A(x);
- a transition probability function p : X x A(x) x X ^ [0,1] that gives
xex
the probability transitions pX,y from an arbitrary x e X to an arbitrary y e X
for a fixed action a e A(x), where PX y = 1, ^x e X, a e A(x);
yex
x0 e X
The game starts at the moment of time t = 0 in the state x0 where the player i e {1, 2,..., m} who is the owner of the state position x0 (x0 e Xj) chooses an action a0 e A(x0) and determines the rewards f 1(x0, a0), f2(x0, a0),..., fm(x0, a0) for the 1 , 2, . . . , m y = x1 e X
according to a probability distribution {pX0,y}• At the moment of time t = 1 the player k e {1, 2,..., m} who is the owner of the state position x1 (x1 e Xk) chooses an action a1 e A(x1) and players 1,2,..., m receive the corresponding rewards f 1(x1, a1), f 2(x1, a1),..., fm(x1, a1^. Then the game passes to a state y = x2 e X according to a probability distribution {pXl,y} and so 011 indefinitely. Such a play of the game produces a sequence of states and actions x0, a0, x1, a1,..., xt, at,... that defines a stream of stage rewards f 1(xt,at),f2(xt, at),..., fm(xt,at), t = 0, 1, 2, . . .
players
WX0 =liminf -V E(f J(xT,aT)), i = 1, 2,..., m
0 t^w t z—'
T=0
E
Markov process induced by actions chosen by players in their position sets and given a starting state x0. Each player in this game has the aim to maximize his average
m=1
decision problem with given action sets A(x) for x e X, a transition probability function p : X x A(x) x X ^ [0,1] and step rewards f (x, a) = f 1(x, a) for
xex
x e X Mid a e A(x). If in an average stochastic positional game the probabilities pX,y take only values 0 Mid 1, i. e. pX,y e {0,1}, Vx, y e ^md Va e A(x), then such a game becomes a mean payoff game on the graph G = (X, E), where e = (x, y) e E if and only if there exists a e A(x) such that pX,y = 1. So, in this case the set of directed edges E(x) = {e = (x, y) e E|y e X} with the common origin in x corresponds to the set of actions A(x) in the posit ion x of the game.
In the paper we will study the average stochastic positional game when the players use pure and mixed stationary strategies of choosing the actions in the states.
3. Average stochastic positional games in pure and mixed stationary strategies
A strategy of player i e {1, 2,..., m} in a stochastic positional game is a mapping s® that provides for every state xt e X® a probability distribution over the set of actions A(xt). If these probabilities take only values 0 Mid 1, then s® is called a pure strategy, otherwise s® is called a mixed strategy. If these probabilities depend only on the state xt = x e X® (i. e. s® does not depend on t), then s® is called a stationary strategy, otherwise s® is called a non-stationary strategy.
Thus, we can identify the set of mixed stationary strategies S® of player i with the set of solutions of the system
( £ s®xj0 = 1, Vx e X®;
) oeA(x) (!)
[ s®X o > 0, Vx e X®, Va e A(x).
s®
player i e {1, 2,..., m}. So, the set of pure stationary strategies S® of player i corresponds to the set of basic solutions of system (1).
Let s= (s1, s2,..., sm) e S = S1 x S2 x • • • x Sm be a profile of stationary strategies (pure or mixed strategies) of the players. Then the elements of probability transition matrix Ps = (p®,y) in the Markov process induced by s can be calculated as follows:
pX,y = Y^ s®x,apX,y fc>r x e ^ i = 1, 2, ...,m. (2)
aeA(x)
If we denote by Qs = y) the limiting probability matrix of matrix Ps then the average payoffs per transition (s), (s),..., (s) for the players induced by s
<40(s) = E E qXo,yf j(y, ), i = 1, 2,..., m, (3)
k=1yeXfc
where
f j(y,sk )= E sk,0fj(y,a)^r y e Xk, k e{1, 2,.. ., m} (4)
aeA(y)
expresses the average reward (step reward) of player i in the state y e Xk when player k uses the strategy sk.
The functions (s), <2o(s), ..., <£0(s) on S = S1 x S2 x ••• x Sm, defined according to (10),(11), determine a game in normal form that we denote by ({Si}i=T^, {<X0(s)}j=^m )• This game corresponds to the average stochastic positional game in mixed stationary strategies that in extended form is determined by the tuple ({Xj}j=тm, {A(x)}xeX, {f j(x, a)}j=^m, p, xo)• The functions (s), <2o(s),... (s) on S = S1 x S2 x • • • x Sm, determine the game ({Sj}j=^m, 0 (s)}j=^m ) that corresponds to the average stochastic positional game in pure strategies. In the extended form this game is also determined by the tuple ({Xi}i=тm, {A(x)}xex, {fi(x,a)}i=^, P, xo)■
A stochastic positional games can be considered also for the case when the starting state is chosen randomly according to a given distribution {0x} on X. So, for a given stochastic positional game we may assume that the play starts in
the state x e X with probability #x > 0 where = 1. If the players use
xex
mixed stationary strategies then the payoff functions
(s)= E 0x<X(s), i = 1, 2, ..., m
xex
on S define a game in normal form ({Sj}i=^m, We(s)}j=^m ) that in extended form is determined by ({Xi^^, {A(x)}xex, {f j(x, a)}4=^,p, {#x}xex)• In the case 0x = 0, Vx e X \ {x0}, 0xo = 1 the considered game becomes a stochastic
xo
4. Nash equilibria for an average stochastic positional game and determining the optimal stationary strategies of the players
We present a Nash equilibria existence result and an approach for determining the optimal mixed stationary strategies of the players for the average stochastic positional game when the starting state of the game is chosen randomly according to a given distribution {0x} on the set of states X. In this case for the game in normal form ({Sj}i=^m, We(s)}j=^m ) the set of strategies Sj and the payoff functions (s), i = 1,2,..., m can be specified as follows:
Let Sj, i e {1, 2,... m} be the set of solutions of the system £ 4,a = 1, Vx e Xi;
aeA(x) (5)
4,a > 0, Vx e Xj, Va e A(x).
On S = S1 x S2 x • • • x Sm we define m payoff functions
- (S1,S2,...lSm)^^ ]T sX,a f(x,a)q*, i =1, 2,.. ., m, (6) k=i xexk oeA(x)
where qx for x G X are determined uniquely from the following system of linear equations
' m
qy - E E E sX,o qx = o,
k=i xexfc oeA(x)
<
m
qy + wy - E E E sX,o pX,y wx = ^y,
k=i xeXfc oeA(x)
for an arbitrary fixed profile s = (s1, s2,..., sm) G S.
The functions —e(s1, s2,..., sm), i = 1, 2,..., m, represent the payoff functions for the average stochastic game in normal form ({Si}i=j^, {—e(s)}j=^m ) determined by the tuple ({Xi}i=T^, {A(x)}x£x, {/j(x, a)}j=^, P, {^y}yex) where 0y for y G X are given nonnegative values such that EyeX = 1-
If 0y = 0, Vy G X \ {x0} and #X0 = 1, then we obtain an average stochastic game in normal form ({S®}j=^m, {WX0(s)}i=rm ) when the starting state x0 is fixed, i.e. —e (sT,s2,..., sm) = ^X 0 (sT,s2, ..., sm), i = 1, 2, ...,m. So, in this case the game is determined by ({Xj}j=^m, {A(x)}xeX, {/j(x, a)}j=^m, P, x0).
Lozovanu, 2018, showed that each payoff function — jj (s), i G {1, 2, ...,m} in the game ({Sj}j=^m, {— e(s)}i=rm ) quasi-monotonic (quasi-convex and quasi-concave) with respect to sj on a convex and compact set Sj for fixed s1, s2, ..., sj-1, sj+T, ..., sm. Moreover for the game ({Sj }j=^m, {—e (s)}i=Tm ) has been shown that each payoff function — e(s), i G {1,2, ...,m} is graph-continuous in the sense of Dasgupta and Maskin, 1986. Based on these properties Lozovanu, 2018, proved the following theorem.
Theorem 1. The game ({Sj}i=^m, {—e(s)}i=rm) possesses a Nash equiUbriurn s*= (s1*, s2*,..., sm*) G S which is a Nash equilibrium, in mixed, stationary strategies for the average stochastic positional game determined, by ({Xj}i=^m, {A(x)}xeX, {/j(x, a)}j=^, p, {0y}y£X)■ If ^y > 0, Vy G X, then s*= (s1 == s2*,..., sm*) is a, Nash equilibrium in mixed stationary strategies for the average stochastic positional game ({Sj}i=^m, {wy (s)}i=rm ) wth an arbitrary starting state y G X.
Thus, for an average stochastic positional game a Nash equilibrium in mixed stationary strategies can be found using the game model ({Sj}i=^m, {—e(s)}i=rm) where Sj and — e(s), i = 1, 2,..., m, are determined according to (5)-(7). This game model in the case m = 2, /(x, a) = / 1(x, a) = —/2(x, a), Vx G X, Va G A(x) corresponds to a zero-sum two-player average stochastic positional game and we can use it for determining the optimal stationary strategies of the players. Note that in this case the equilibrium may exists in pure stationary strategies and consequentely such a game model allows to determine the optimal pure stationary strategies of the players.
Vy G X;
Vy G X
(7)
For antagonistic average stochastic positional games Lozovanu and Pickl, 2016, proposed an another approach for determining Nash equilibria in pure stationary strategies. However the approach from Lozovanu and Pickl, 2016, couldn't be extended for non-zero average stochastic positional games and for the non-zero mean payoff games. Nevertheless such an approach allows to ground finite efficient iterative procedures for determining the optimal pure stationary strategies of the players.
5. Formulation of mean payoff games in mixed stationary strategies
Consider a mean payoff game determined by the tuple (G, {Xj}i=y-m, {cj}j=rm, x0) where G = (X, E) is a finite directed graph with vertex set X and edge set E, x = Xi u x2 U • • • U Xm (Xj n Xj = 0, i = j) is a partition of X that determine the corresponding position set of players, cj : E ^ R1, i = 1, 2,..., m are the real functions that determine the rewards on edges of graph G and x0 is the starting position of the game.
G
defined in a similar way as for the average stochastic positional game. We identify the set of mixed stationary strategies Sj of player i € {1, 2,..., m} with the set of solutions of the system
E = 1 Vx € Xi;
yex(x) (8)
4,y > 0, Vx € Xj, y € X(x)
where X(x) represents the set of neighbor vertices for the vertex x, i.e. X(x) = {y € X|e = (x,y) € E}.
Let s = (s1, s2,..., sm) be a profile of stationary strategies (pure or mixed strategies) of the players. This means that the moves in the mean payoff game from an arbitrary x € X to y € X induced by s are made according to probabilities of the stochastic matrix Ps = (sx-y), where
sX,y if e = (x' y) € E'x € Xi' y € X; i = 1 2'...' m; /q\ 0 if e = (x, y) € E. 1 '
s
sition matrix Ps = (sx-y) ^^d ^^e corresponding rewards , i = 1,2, ...,m on edges (x, y) € E. ^^^rfore, if Qs = (qX y) is the limiting probability matrix of Ps then the average rewards per t ransition (s), (s), . (s) for the players can be determined as follows
m
< (s) = EE ), i = 1, 2,...,m, (10)
where
M¿(y,sfc )= E sk,xc¿ y G Xk, k e{1, 2,...,m} (11)
xex(y)
expresses the average step reward of player j in the state y € Xk when player k uses the mixed stationary strategy sfc. The functions w^ (s), (s), ..., w^ (s) on S = S1 x S2 x • • • x Sm, defined according to (10), (11), determine a game in normal form that we denote by ({S®}i=i-m, {wX0 (s)}i=^m )• This game corresponds to the mean payoff game in mixed stationary strategies on G with a fixed starting position x0. So this game is determined by the tuple (G, {c®}i=rm, x0).
In a similar way as for an average stochastic game here we can consider the G
given distribution {0x} on X. So, for such a game we will assume that the play starts in the states x € X with probabilities 0x > 0 where £ = 1. If the
xex
players in such a game use mixed stationary strategies of moves in their positions then the payoff functions
(s)=E 0xwX(s), i =1, 2,..., m
x£X
on S define a game in normal form ({Si}i=^m, We(s)}j=i-m ) that is determined by the following tuple (G, {Xi}i=I-m, {0®}®=^, {#x}xex) • In the case 0x = 0, Vx € X \ {v0}, = 1 this game becomes a mean payoff with fixed starting state x0.
6. Nash equilibria in mixed stationary strategies for mean payoff games and determining the optimal strategies of the players
In this section we show how the results from the previous sections can be applied for determining Nash equilibria and the optimal mixed stationary strategies of the players for mean payoff games.
Let ({Si}i=i-m, (s)}i=i-m) be the game in normal form for the mean payoff game determined by (G, {Xi}i=^m, {c®}i=rm, {^x}xeX). We show that S® and (s) for i € {1,2,..., m} can be defined as follows:
S® represents a set of the solutions of the system ( £ s*x,„ = 1, Vx € X®;
< yex (x) (12)
I sx,y > 0, Vx € X®, y € X(x)
and
(sV,...,sm ) = ^ E E sx,y c®(x,y)qx, (13)
k=i yexfc yex(x)
where qx for x € X are determined uniquely (via sx,y) from the following system of equations
qy - £ £ £ <y qx = 0, Vy e X(x);
k=1 xeXk vGX(x) m
qy + wy- £ £ £ sX,y wx = ^y, Vy e X(x).
fc=i yex xex(y)
Here, 0y for y € X represent arbitrary fixed positive values such that = 1.
yex
Using Theorem 1 we can prove now the following result.
G
({Sj}j=rm, We(s)}j=rm) possesses a Nash equilibrium s*= (s1 , s2 ,..., sm*) € S which is a Nash equilibrium in mixed stationary strategies for the mean payoff game on G with an arbitrary starting position x0 € X.
Proof. To prove the theorem it is sufficient to show that the functions ^(s), i € {1, 2,..., m} defined according to (13), (14) represent the payoff functions for the mean payoff game determined by (G, {Xj}i=^m, {cj}i=I-m, ). This
is easy to verify because if we replace in (6) the rewards fj (x, a) for x € X and a € A(x) by rewards cX y for (x,y) € E and in (6), (7) we replace the probabilities pX,y, x € Xk, a € A(x) for the corresponding players k =1, 2,..., m by pX,y € {0,1} according to the structure of graph G then we obtain that (6), (7) are transformed into (13), (14). If after that we apply Theorem 1 then obtain the proof of the theorem.
So, the optimal mixed stationary strategies of the players in a mean payoff game can be found if we determine the optimal stationary strategies of the players for the game ({Sj}i=i-m, We(s)}j=i-m) where Sj and ^(s) for i € {1,2, ..., m} are defined according to (12) - (14). If m = 2, cx,y = ci,y = —cX,y, V(x, y) € E then we obtain a game model in normal form for the zero-sum two-player mean payoff on G
considered game model allows to determine the optimal pure stationary strategies of the players. For antagonistic mean payoff games on graphs the approach for antagonistic average stochastic positional games from Lozovanu and Pickl, 2016, can also be adapted if we take into account the mentioned transforms in the proof of Theorem 2, i.e. we should change the rewards f j(x, a) for x € X, a € A(x) by rewards cX y for (x, y) € E and replace the probabilities pX,y, x € Xk, a € A(x), k =1,2,..., m by probabilities pX,y € {0,1} according to the structure of G
7. Conclusion
The considered class of non-zero mean payoff games generalizes the zero-sum two-player mean payoff games on graphs studied by Ehrenfeucht and Mycielski, 1979, Gurvich et al., 1988, Lozovanu and Pickl, 2015 and Zwick and Paterson,1996. For zero-sum two-player mean payoff games on graphs there exist Nash equilibria in pure stationary strategies. For the case of non-zero-sum mean payoff games on networks Nash equilibria in pure stationary strategies may not exist. The results presented in the paper show that an arbitrary mean payoff game in mixed stationary strategies possesses a Nash equilibrium and such an equilibrium can be found using the game models proposed in Sections 5,6.
References
Alpern, S. (1991). Cycles in extencsive form perfect information games. Journal of Mathematical Analysis and Applications, 159, 1-17.
Dasgupta, P. and Maskin, E. (1986). The existence of an equilibrium in discontinuous economic games. The Review of Economic Studies, 53, 1-26.
Ehrenfeucht, A. and Mycielski, J. (1979). Positional strategies for mean payoff games. International Journal of Game Theory, 8, 109-113.
Gurvich, V., Karzanov, A. and Khachiyan, L. (1988). Cyclic games and an algorithm, to find minimax cicle means in directed graphs. USSR Computational Mathematics and Mathematical Physics, 28, 85-91.
Lozovanu, D. (2018). Stationary Nash, equilibria for average stochastic positional games. In: Petrosyan, L., Mazalov, V., Zenkevich, N. (eds) Static amd Dynamic Game Theory: Foundations and Applications, Springer (to appear).
Lozovanu, D. and Pickl, S. (2006). Nash, equilibria conditions for cyclic games with, m players. Electronic Notes in Discrete Mathematics, 28, 85-91.
Lozovanu, D. and Pickl, S. (2015). Optimization of Stochastic Discrete System,s and Control on, Complex Networks. Springer.
Lozovanu, D. and Pickl, S. (2016). Determining the optimal strategies for zero-sum average stochastic positional games. Electronic Notes in Discrete Mathematics, 55, 155-159.
Zwick, U. and Paterson, M. (1996). The complexity of mean payoffs on graphs. Theoretical Computer Science, 158, 343-359.
