CC BY
47
UDC 519.83 Вестник СПбГУ. Прикладная математика. Информатика... 2019. Т. 15. Вып. 1
MSC 91A15
Stochastic game of data transmission in the presence of buffers of finite capacity*
V. M. Bure, E. M. Parilina
St. Petersburg State University, 7—9, Universitetskaya nab., St. Petersburg, 199034, Russian Federation
For citation: Bure V. M., Parilina E. M. Stochastic game of data transmission in the presence of buffers of finite capacity. Vestnik of Saint Petersburg University. Applied Mathematics. Computer Science. Control Processes, 2019, vol. 15, iss. 1, pp. 39-46. https://doi.org/10.21638/11702/ spbu10.2019.103
The game-theoretic model of data transmission in a network of a given topology is presented. Two players (network nodes) tend to send as many random data packagesas possible to the final nodes through one common node. Each playerhas a finite capacity buffer for storing data packages. A system of costs for sending and storing data packages andrewards for the successful package delivery is introduced. A dynamic conflict-controlled process is modelled as a stochastic gamewith a finite set of states. The existence of the Nash equilibrium and a cooperative solution is proved. The cooperative solution is a strategy profile which maximizes the total expected payoff. The price of anarchy in the network is calculated. The price comparesthe players' payoffs in the Nash equilibrium and cooperative solution. Keywords: data transmission, slotted ALOHA, the price of anarchy, stochastic game.
Introduction. We propose a game-theoretic model of data transmission using slotted ALOHA scheme [1-3]. There are two players (network nodes) who want to transmit as many data packages of a unit capacity as possible. They may send the packages independently or in cooperation and transitions are gone through the common node. Players are not symmetric which means that data packages can appear at the nodes with different probabilities that do not vary in time. The time is assumed to be discrete. Game-theoretic models of data transmission in the networks of different topologies are also considered in the papers [4, 5]. In particular, in [4] three models of data transmission in the networks of different topologies were presented in case of complete information about the presence of packages at the other player. In [5] the authors consider a model of data transmission based on slotted ALOHA scheme in the absence of complete information on the presence of packages at the other player. In papers [4, 5] the Nash equilibria and a cooperative solution are found. Moreover, these two equilibria are compared using the price of anarchy. In the work [6], the authors provide an analysis of data transmission in ALOHA scheme networks, namely, the Nash equilibrium in one-shot game with n symmetric and asymmetric players.
Dynamic process of data transmission in a network of a given topology of slotted ALOHA scheme is modeled as a stochastic game. Unlike [5], we assume that each node (or player) has a finite capacity buffer at which the received data packages are stored before transmission to the destination node. The stochastic model of data transmission is presented in [7] and there is a relay node in data transmission scheme, which is considered as a buffer for package keeping. Contrary to this approach we assume that each player has a buffer to keep randomly appeared packages. When we find the optimal players'
* The work of E. M. Parilina is supported by Russian Science Foundation (project N 17-11-01079). © Санкт-Петербургский государственный университет, 2019
strategies, the two approaches are used — non-cooperative and cooperative. We find the Nash equilibrium and cooperative solution according to these approaches.
Model. Consider the data transmission slotted ALOHA-like scheme represented in Fig. 1. Players (Nodes) 1 and 2 tend to send data packages to the nodes r1 and r2, respectively. As one can see from the data transmission scheme, the package should go through the node of a unit capacity which is common for both players. Player i = 1,2 has a buffer of capacity ki, which means that it can possess from 0 to ki data packages of a unit capacity at each time period. At the beginning of each time period, Players 1 and 2 may receive a data package of a unit capacity with probability v1 e (0,1) and v2 e (0,1) respectively if at the beginning of the period he possesses less than ki packages. The packages independently appear at nodes.
Figure 1. Data transmission scheme
In each time period, Player i = 1,2 can transmit only one package to the destination node ri. In case both players simultaneously transit the packages, the packages are back to the Nodes. For successful package delivery, it is necessary that only one Player transits the package. If the package is delivered to the destination node, the player receives a payoff of 1 minus the costs of a package transmission which is equal to c e (0,1). The player bears the costs of d e[0,1) for one time period delay per each unit package, d << 1.
By the state of the system in time period t, we mean the pair (w1(t),w2(t)), where wi(t) e {0,1,...,hi} is the number of data packages at Player i's buffer, i = 1,2. The set of the system states at any time period t is denoted by Q, |Q| = (k1 + 1)(k2 + 1). Let the states be w(1), ..., w(m), where m =(k1 + 1)(k2 + 1).
On the basis of the given assumptions we define a stochastic game with a finite state set and finite action sets.
Stochastic game. The time is assumed to be discrete. The set of states at each time period is Q = {w = (w1,w2) ■ wi e [0,ki], i = 1,2}. At state w the set of Player i's actions is Af that is
A! = i{^ f " = 0' (1)
I {w}, if wi = 0.
In formula (1) action t means "to transmit" a package, w — "to wait".
In state w = (w1,w2), here w1 = w2 = 0, the payoff function of Player i is equal u! (a! ,a!) = 0 for any i = 1, 2, and both actions a! and a! are w. If w = (w1,w2), w1 > 0 and w2 = 0, then the payoff functions are defined by matrix
((1 - c - d(w1 - 1);0)) ( (-dw1;0) ),
where Player 1 chooses rows (row 1 correspond to action t, row 2 — to action w); Player 2 chooses a column. In case, when w = (wi,w2), wi = 0 and w2 > 0, then the players' payoff functions are defined by matrix
((0; 1 - c- dl - 1)) (0; -di2)).
If w = (wi,w2), wi > 0 and i2 > 0, then each player has two actions t and w and players' payoff functions in this state are defined by
(-c - dii; -c - di2 ) (1 - c - d(ii - 1); -di2)\ (-dii;1- c - d(i2 - 1)) (-dii; -dw2) /
Let assume that the players' strategies in the whole game are stationary which seems to be a natural assumption in data transmission modeling. The stationary strategy depends on the state and does not depend on time and the history of the stage. We suppose that the player does not know the number of packages at the other player's buffer. And we also assume that Player i uses the same strategy in any state (wi,wj), where wi e{1,...,ki - 1}, and may be the other strategy in state (wi,ij), when wi = ki. Therefore, Player i's mixed stationary strategy ni is (pf ,prr), i = 1,2, where pf e [0,1] is a probability of choosing action t in any state (wi,ij), j + i, j = 1,2, if wi = ki, ij e [0,kj] (probability of transmitting a package when the buffer of Player i is full); prnf e [0,1] is a probability of choosing action t in any state (wi,ij), j + i, j = 1,2, if wi e [1,ki - 1], ij e [0,kj] (probability of transmitting a package when the buffer of Player i is not full). It is supposed that the Player i chooses the same probabilities of transition when the buffer is not full, wi e [0,ki - 1]. One may consider another class of stationary strategies, e. g., the probabilities of transition may be different for different number of packages at the buffer. Obviously, this increases the number of pure strategies and strategy profiles and the number of calculations for finding the equilibria. The stationary strategy profile is (ni,n2) = ((Pi,pr\f), (Pf,P2f)). Denote by Si the set of stationary strategies of Player i. The set of pure stationary strategies of Player i = 1,2 is {(0,0), (0,1), (1,0), (1,1)}.
Now we define transition probabilities n(w"/w',n) to state w" e Q from state w' e Q if strategy profile n is realised. They are n(w"/l,n), which is represented in table 1, where w = (ii,ij), i = 1, 2, j = 1, 2, i + j, and n = (Vi,Vj) = ((pf ,p'rri), (pf ,prrf)). The transition probabilities for the pairs of states, which are not presented in table 1 are null.
We have defined two-person stochastic game G by a tuple
{Q, {A% }i=i,2;wen, {ut }i=i,2;wen, {Si}i=i,2, {n(w'7w',n)}w"en,w'en,^HixE2 ,S),
where 6 e (0,1) is a common discount rate.
We consider the discounted expected payoff as a payoff in stochastic game G given by
Ei(n)=no(I - 6n(n))-iui(n). (2)
In formula (2) is an initial probability distribution over the set of states, I is an identity matrix of size m, n(n) is a mx m matrix of transition probabilities n(-/-,n), whose (l, n)-th entry is a probability of transition from state to state w(n), when strategy profile n is realised.
Table 1. Transition probabilities
7т(ш"/ш', Tj) w' = (w'iyw'j)
ViVj шi = =0 < = <=1
Vi(l - Vj) шi = =0 < = 1, < = 0
(1-Vi)(l-Vj) ш'. = ш'. = 0 ш'-' = ш". = 0 г 7
(1-щ)(1 -Vj)pf Ld'i = 0, Lü'- ё [1, kj - 1] ¡У' = 0, ш". =ш'-~ 1 г ' .7 .7
Ld'i = 0, Lü'- ё [1, kj - 1] ¡У' = 0, ¡У.' = ¡У г ' .7 .7
Lü[ = 0, üj'- ё [1, kj - 1] < = 0, = üj'j+ 1
1 - Vj)p"f Ld'i = 0, Lü'- ё [1, kj - 1] = 1, ШЧ = ш'.-1 г ' .7 .7
Vi[py»j + (l-py)(l-»j)] Ld'i = 0, Lü'- ё [1, kj - 1] = 1, ш'! = ш'. г ' .7 .7
Vi(l~p"f)vj Ld'i = 0, Lü'- ё [1, kj - 1] = 1, ш'! = ш'. + 1 г ' .7 .7
ё [1 ,ki - 1], u'i е ihkj-1] = ¡У - 1, г г ' ш'! = ш'. 7 7
(1 - Vi)VjP™f(l -p"f) co'i ё [1 ,ki - 1], u'i e [1 ,kj-l] < = ^ - 1, = +1
{1-щ){1 + +Vi(l-Vj)pf(l-pf) + +(l-Vi)Vj(l-pf)pf + +(l-Vi)(l-Vj)(l-pf )(l-pf) co'i ё [1 ,ki - 1], w'j ё [1 ,kj - 1] ш" = г г' = ш'. J J
+(1 - vi)vjp"fp"f + + ViVjP™f (l-p"f) ё [1 ,ki - 1], w'j ё [1 ,kj - 1] üj'f = Lü'-, г г' ¡У.' = ш'. +1 J J
nf nf + ViV]Pi Pj ё [1 ,ki - 1], w'j ё [1 ,kj - 1] Lü'-' = ¡У + 1, г г ' ¡У' = ¡У + 1 .7 .7
(l-UiXl-u^pf ui'i = 0, LOj = kj < = 0, = ^ - 1
lü'■ = 0, LÜ'- = kj ¡У' = 0, Lü"- = ¡У ^г 7 7
1 - Vj)pfj lü'■ = 0, Lü'- = kj = 1, ¡У' = ¡У - 1 г ' .7 .7
lü'■ = 0, Lü'- = kj ¡У' = 1, ш"- = ¡У г ' .7 .7
(l-^)pf(l-pf) u'i = ki, lü'- = kj < = " = ^
(l-pf)(l-pJ/) + ^pf(l-pJ/)+ +Vj(l-p{)pfi +p{p{ = ki, u'i = kJ ш" = г г' = ш'. .7 .7
(l-ViK'il-p'i) co'i ё [1, ki - 1], lü'- = kj (У' = — 1. (У' = ixJ^ шг шг -1-' 7 7
(1-Ui)(l-Uj)(l-pf)pt ё [1, ki - 1], lü'- = kj < = = К' - ь
ViPifPj + Vi (1 - p"f) (1 - pfj ) + +vivj(l-p"f)pfj ё [1 ,ki - 1], "j = kj = ш'- + 1, г г ' ¡У' = ¡У .7 .7
(1 -Vi )p"fpfj + Vip"f ( 1 -Pj) + +(l-Vi)(l-pf)(l-pf) + ё [1 ,ki - 1], и', = kj ¡У' = ¡У, г г' ¡У' = ¡У j J
Vi( 1 - Vj)(l-p"f)pfj ё [1, ki - 1], lü'- = kj Lü'-' = Lü'- + 1, ¡У' = ¡У - 1 г г ' .7 .7
Payoff function ui(rj) defines the payoff for any state w. The values of function ui(Vi,Vj), = 1,2, i + j, are represented in table 2.
We consider two approaches (cooperative and non-cooperative) to find a solution in game G. We consider the Nash equilibrium as an optimality principle within a non-cooperative approach. Following a cooperative approach we find the cooperative solution maximizing the total players' expected payoff in game G.
Table 2. Payoff functions
U> = (u>i,u>j)
0 0 uii = u>j =0
0 p"f (1 - с + d) — dujj Lüi = 0, üJj ё [1, kj - 1]
0 pf. (1 - с + d) — dujj Uli = 0, uij = kj
p"r(l-p]r)(l+d)-cp^ -dtoi py(l-p^)(l+d)-cpy -dtOj Uli <= [1, ki - 1], Ulj € [1, kj - 1]
(1 -P"f)(l +d) -cp{ -db>i ~pf )(1 +d) -cp"f - dujj uii = ki, Ulj € [1, kj - 1]
pf(l -pf )(l + d) -cp! -duii Pfj (1 -pf)(l +d) -cp? - dujj Lüi — ^z "i ^ j — kj
Theorem. There exist the Nash equilibrium and cooperative solution in game G. Proof. Existence of the Nash equilibrium follows from Fink's results (see [8]) using the fact that the set of states in game G is finite and the set of actions of any player is also finite for any state w. The cooperative solution 'q* is an argument of a well-known dynamic program problem and it always exists. Moreover, it is a pure stationary strategy profile.
To estimate the selfishness in the network we calculate the price of anarchy [9] given
by
PoA(G) =
£ Ei(n*) i=l
(3)
min £ Ei(n)
neNE(G) i=i
where NE(G) is the set of the Nash equilibria in game G. One can notice the PoA is not defined if the sum in the denominator in (3) is null.
Simulation study. As an example we consider game G with the following parameters: c = 0.2, d = 0.03. Let Player 1 has a smaller buffer of capacity k\ = 2 contrary to Player 2's buffer of capacity k2 = 4. Both players have the same discount factor 0.99. The probabilities of package appearance at Players 1 and 2 are v\ = 0.6 and = 0.2, respectively. There are 15 states in the game. Let the game start with the state (0,0) when there are no packages at the nodes. The set of pure stationary strategies for each player is {(0,0), (0,1), (1,0), (1,1)}. Player i's mixed stationary strategy n is (pf ), i = 1,2. Transition matrix is of size 15 x 15 and it is a function of stationary strategy profile V = (Vi,V2). To find a cooperative solution we need to calculate the total players' payoff in the whole game for each pure stationary strategy profile (there are 16 such profiles) and find the maximal one. To find the Nash equilibria in stochastic game G we use Lemke— Howson algorithm [10-12]. For this we calculate the matrices of expected payoffs of Players 1 and 2 by formula (2) for each pure stationary strategy profile. Therefore, we obtain two payoff matrices A and B for Players 1 and 2, respectively:
A =
/-5.8518 -5.8518 -5.8518 -5.8518 \
47.5200 -2.7638 5.0169 -5.8518
43.7829 34.8935 -12.9591 -20.9107
47.5200 38.7075 -7.5023 -20.9767
B =
/-10.6254 15.8400 5.5014 15.8400 \
-10.6254 15.3433 3.7837 15.8400
-10.6254 -11.8717 -26.0360 -27.5207
^-10.6254 -12.3383 -21.0293 -28.2416y
Then we use these matrices in Lemke—Howson algorithm.
There are four Nash equilibria in bimatrix game with payoff matrices (A, B): a) = (0,0.0619,0,0.93819), 6 = (1,0,0,0) with payoffs 47.5200 and -10.6254 of Players 1 and 2, respectively; b) £i = (0,0.9697,0,0.0303), £2 = (0,0.2672,0,0.7328) with payoffs -5.0266 and 14.5050 of Players 1 and 2, respectively; c) £1 = (0,1,0,0), £2 = (0,0,0,1) with payoffs -5.8518 and 15.8400 of Players 1 and 2, respectively; d) = (1,0,0,0), £2 = (0,0,0,1) with payoffs -5.8518 and 15.8400 of Players 1 and 2, respectively. The "worst" Nash equilibrium, i. e. the Nash equilibrium with the smallest total players' payoff, is the second one £1 = (0,0.9697,0,0.0303), £2 = (0,0.2672,0,0.7328). The equilibria in bimatrix game corresponds to the Nash equilibria in stochastic game G in stationary strategies (n1 ,n2). Mixed strategy n1 prescribes to choose pure strategy (0,1) with probability 0.9697 and (1,1) with probability 0.0303. Mixed strategy n2 prescribes to choose pure strategy (0,1) with probability 0.2672 and (1,1) with probability 0.7328. Both players randomize between two strategies (0,1) and (1,1), but Player 1 with much higher probability 0.9697 chooses strategy (0,1) contrary to Player 2 who chooses strategy (0,1) with probability 0.2672. The total players' payoff in the Nash equilibrium is 9.4784.
The cooperative solution is (n1 ,n2), where n1 = (1,1), n2 = (0,0), with total players' payoff 36.8946. In cooperation Player 1 transmits a package in any state regardless his buffer is full or not, and Player 2 never transmits packages. The price of anarchy in game G is 3.8925. We do not discuss if the price of anarchy is respectively high. It may be interpreted taking into account the costs of coordination of players' strategies.
0.0
1.0
50
100
PoA
0.0
Figure 2. Price of anarchy as a function of probabilities v\ and V2
The sums of the players' payoffs in all Nash equilibria and cooperative solution are positive. In case of negative total players' payoff in the "worst" Nash equilibrium, the explanation of a price of anarchy is questionable. Therefore, in case of negative sum of players' payoffs in the Nash equilibrium the modification of the state payoff functions (e. g., we may add a constant to all state payoffs to make them non-negative) should be made to apply the price of anarchy.
The graph of the price of anarchy as a function of probabilities v1 and v2 is presented on Fig. 2. For simulation we use the following parameters: c = 0.2, d = 0.03, k1 = 2, k2 = 4, 5 = 0.99, and the game starts with state (0,0). As one can observe from the graph, the price of anarchy is non-monotonic function of probabilities v1 and v2. Calculations show the regions of parameters v1 and v2 for which the PoA is large and the coordination of players' strategies may increase the total payoff in more than 100 times in some cases.
Conclusion. We have constructed a model of the process of data transmission with two participants using the theory of stochastic games. The existence of the Nash equilibrium and a cooperative solution follows from well-known results based on the game model. The results are illustrated by the numerical example. The model may be extended in different directions: (i) the packages may have different priority to be sent; (ii) the probabilities of package appearance at the nodes may vary on time; (iii) players' strategies may be different, e. g., the probability to transmit a package maybe some particular function of the number of packages at the player's buffer; (iv) the mechanism of payoff transfers may be implemented to support realization of the cooperative solution.
References
1. Altman E., Barman D., El Azouzi R., Jimenez T. A game theoretic approach for delay minimization in slotted ALOHA. 2004 IEEE Intern. Conference on Communications (IEEE Cat. no. 04CH37577), 2004, vol. 7, pp. 3999-4003.
2. Marban S., van de Ven P., Borm P., Hamers H. ALOHA networks: a game-theoretic approach. Math. Meth. Oper. Res., 2013, vol. 78, iss. 2, pp. 221-242.
3. Sagduyu Y. E., Ephremides A. A game-theoretic look at simple relay channel. Wireless Networks, 2006, vol. 12, no. 5, pp. 545-560.
4. Bure V. M., Parilina E. M. Stohasticheskie modeli peredachi dannyh v setyah s razlichnymi topologiyami [Stochastic models of data transmission in networks with different topologies]. Large-Scale Systems Control, 2017, vol. 68, pp. 6-29. (In Russian)
5. Bure V. M., Parilina E. M. Igra "Mnozhestvennyj dostup" s nepolnoj informaciej [Multiple access game with imperfect information]. Mathematical Game Theory and Applications, 2017, vol. 9(4), pp. 3-17. (In Russian)
6. Inaltekin H., Wicker S. B. The analysis of Nash equilibria of the one-shot random-access game for wireless networks and the behavior of selfish nodes. IEEE/ACM Transactions on Networking, 2008, vol. 16, no. 5, pp. 1094-1107.
7. Afghah F., Razi A., Abedi A. Stochastic game theoretical model for packet forwarding in relay network. Telecommunication Systems, 2013, vol. 52, iss. 4, pp. 1877-1893.
8. Fink A. M. Equilibrium in a stochastic ra-person game. J. Sci. Hirosima Univ. Series A-I, 1964, vol. 28, pp. 89-93.
9. Koutsoupias E., Papadimitriou C. Worst-case equilibria. Proceedings of the 16th Annual Symposium on Theoretical Aspects of Computer Science, 1999, pp. 404-413.
10. Herings P. J.-J., Peeters R. Homotopy methods to compute equilibria in game theory. Econ. Theory, 2010, vol. 42, pp. 119-156.
11. Lemke C. E., Howson J. T. Equilibrium points of bimatrix games. J. Soc. Indust. Appl. Math., 1964, vol. 12, pp. 413-423.
12. Raghavan T. E. S., Filar J. A. Algorithms for stochastic games — a survey. ZOR — Methods and Models of Operations Research, 1991, vol. 35, pp. 437-472.
Received: October 18, 2018.
Accepted: December 18, 2018.
Author's information:
Vladimir M. Bure — Dr. Sci. in Physics and Mathematics, Professor; v.bure@spbu.ru
Е1епа M. Parilina — PhD in Physics and Mathematics, Associate Professor; e.parilina@spbu.ru
Стохастическая игра передачи данных с буферами конечных емкостей*
В. М. Буре, Е. М. Парилина
Санкт-Петербургский государственный университет, Российская Федерация, 199034, Санкт-Петербург, Университетская наб., 7—9
Для цитирования: Bure V. M., Parilina E. M. Stochastic game of data transmission in the presence of buffers of finite capacity // Вестник Санкт-Петербургского университета. Прикладная математика. Информатика. Процессы управления. 2019. Т. 15. Вып. 1. С. 39-46. https://doi.org/10.21638/11702/spbu10.2019.103 (In English)
В работе представлена теоретико-игровая модель передачи данных в сети заданной конфигурации. Цель двух игроков (вершин в сети) — переслать как можно больше пакетов данных в конечные вершины сети, используя одну общую вершину. Каждый игрок имеет буферы конечной емкости для хранения вершин. Предложена система издержек за пересылку и хранение пакетов, а также вознаграждений за успешные доставки пакетов. Динамический конфликтно-управляемый процесс моделируется стохастической игрой с конечным множеством состояний. Доказано существование равновесия по Нэшу и вектора кооперативных стратегий, при котором достигается максимальный суммарный выигрыш игроков. Вычислена цена анархии в сети, с помощью которой сравниваются выигрыши игроков в равновесии по Нэшу и наборе кооперативных стратегий.
Ключевые слова : передача данных, ALOHA, цена анархии, стохастическая игра. Контактная информация:
Буре Владимир Мансурович — д-р техн. наук, проф.; v.bure@spbu.ru Парилина Елена Михайловна — канд. физ.-мат. наук, доц.; e.parilina@spbu.ru
* Работа Е. М. Парилиной выполнена при финансовой поддержке Российского научного фонда (проект № 17-11-01079).
