CC BY
8
Algorithm of Data Storage Based on Combining of the Methods of Locating and Coding non-Homogeneous Data
Maksimov V. A. A. F. Mozhaisky Military Aerospace academy St. Petersburg, Russia falcon225@yandex.ru
Abstract. Proposed an approach to the designing and application of advanced spacecraft data storage systems based on heterogeneous storage devices. This approach serving for for the purpose of increasing fault tolerance of information processing systems. Using of heterogeneous storage devices and storage methods allows flexible system management depend on its condition. It allows reduce all kinds of redundancy at storage systems. The novelty of this work lies in taking into account difference of the importance of stored data.
Keywords: storage systems, heterogeneity, distribution, fault tolerance, correctness, remote sensing of the Earth, regenerating coding.
Introduction
The current trends at the development of space satellites are characterized by increasing volume of data collected during spacecraft lifecycle and by a shift at the place of processing these data from ground-based processing units to on-board processing units. This information can be divided by type (telemetry, special, navigation, etc.), or by importance for the consumer (ground control system, onboard systems, etc.).
The largest amounts of data are generated by the earth remote sensing satellites. Its characteristics are improved with every generation and as a result amount of data accumulated on board is significantly growing. However, the problem of fault tolerance onboard data storage is not specific only for remote sensing satellites. Errors occurred during the storage process at commands, software or navigation data may lead to the disasters consequences [1].
Long-term operation of the remote sensing satellites allows to reveal a number of features of the onboard equipment operation:
1) due the high performance of all kinds of spacecraft equipment, it generates a large amounts of data, which requires special approaches to their processing and storage. At the same time, parameters of FLASH-memory modules, used at the storage systems are not full-suitable for fault-tolerant storage.
2) one of the mainstream tends for last decades at Earth remote-sensing is decreasing of data delivering time. One of the solutions for this can be full or partial onboard data processing. But organization of such onboard data processing demands errors check and corrections of this operations. Besides it demands fault-tolerant storage of processing results [2].
3) the demands placed on existing and prospective Earth remote sensing satellites (ERSS) with regard of the lifecycle
extension are constantly increasing. However, the methods of reliable space information systems design for long-term effects of space factors are insufficiently developed [3].
4) base data attributes for fault-tolerant maintenance are integrity and errorless. Both of this attributes suffer from corruption and data loss due storage at the data storage system and transferring throw high speed radio links. Increasing the level of integrity and errorless of data, as an integral part of the spacecraft fault-tolerance in general, requires the introduction of various types of redundancy (hardware, software, time redundancy, etc.). But it is not always acceptable in terms of mass-dimensional or functional restrictions for the satellites systems.
One of the new approach to the onboard data storage systems (ODSS) design is construction of data storage systems from modules based on different physical principles [4, 5]. These memory modules (MRAM, FRAM, SONOM, etc.) have some differences at the specifications, which makes them attractive for use at on board satellite systems. It should be noted that the choice of one specific type of memory is difficult because of substantial differences in their characteristics (e. g., having a higher radiation tolerance, they have less available information capacity).
At the same time, it should be noted that the data stored at ODSS have different properties.
For example, system data (data required for the correct operation of the system) have noticeably much importance than results of remote sensing data. Besides the structure of remote sensing data is also heterogeneous.
In most Russian ERS satellites, after the first step of data processing it has stored encoded in a JPG file format. In accordance with international standards (ISO/IEC 10918) in the file container is JFIF, we can identify several data regions.
Thus, the damage or data blocks loss from regions with SOS, APPn, COM markers are not critical for main ERS functions. At worth case this will lead to minor decline of ERS quality results (collision of a few pixels or blocks of pixels at result image). However, even slight corruption DQT, DHI, DRI regions blocks can lead to a significant decline of ERS quality results or to impossibility of remote sensing results perception.
Now exist a significant theoretical and practical groundwork in the field of distributed data storage with the using a regenerating codes [6-8] that allow significantly improve the reliability of data storage compared to the replication method.
All of the above leads to the necessity of development methods for the designing and application of heterogeneous storage systems.
In order to resolve the emerging contradiction between necessity of reliable storage of large amounts of data on board the spacecraft on the one side and the lack of currently developed methods of designing and application of such storage at the other side, proposed an algorithm of data storage based on combining of the methods of locating and coding non-homogeneous data.
Concept of heterogeneous data STORAGE system
Concept of distributed data storage at heterogeneous data storage system (HDSS) composed in designing and application such system at ERS onboard systems. HDSS consist from set of storage units (SU). Each storage unit consists of memory controller (MC) and set of memory modules (MM). HDSS means aggregation of hardware and software parts. Hardware part means structure of HDSS. Software part means a set of processing procedures on data (data encoding, data placement). Introducing of data storage method similar to Big Data at data centers is offered instead of double or triple replication. It means that initial data divides to several data regions according importance index. Each data regions is encoding with regenerating code of certain construction. Next step is determination of data placement (assignment data block with certain importance index to a certain type of memory modules).
Data placement depends on data type and HDSS storage units condition. Before data blocks will write at the MM they should be encoding with error-correcting codes.
As writing above, at the current time exist a couple of works at this field, but there is no one system method for designing and application of fault tolerance HDSS.
Introducing in HDSS different types of memory allows to flexible data placement management and ensure fault tolerance.
Besides that development of the inter satellite communication and evolution of nano-satellite conceptions allows to considerate inter satellite storage network [9, 10].
In this case heterogeneous nature of such system can provide flexible management of data storage process witch is necessary because of low-available nodes at network.
As further development of concept it is possible to introduce at the importance index value of semantic importance of data (importance for the final consumer).
Application of HDSS for ERSS can lead to some drawbacks:
• increasing system complexity;
• necessity of development special management system.
Main target of HDSS organization is reliable and errorless
data storage.
Fault tolerance of HDSS at this work means probability of data block will not loss or contains not corrected errors.
Main advantages of distributed storage concept are:
• high technical redness of HDSS;
• maintaining of HDSS fault tolerance;
• maintaining of data availability;
• flexible management of system;
• maintaining demand efficiency of HDSS;
• technical and technology reserve for future research.
Considering ERS as information system it is obvious that it
main function determinates as issuance of reliable information about the objects of observation to the end user.
So, reliable and errorless data storage at the heterogeneous HDSS of ESR is carried out through:
1) Dividing data objects to the regions according to the importance index.
2) Application different constructions of regenerating codes for different data.
3) Data placement according to the importance index.
4) Application of error corrections codes depends of data placement and importance.
Using such complex multi-level approach can ensure high HDSS fault-tolerance.
Math formulation of problem
Technical analysis of modern storage systems and technologies condition shows that storage process characterized by couple of properties witch borders reliability of data storage at current ERS concept.
First, designing HDSS from homogeneous memory modules is hardly to realized for all kinds of memory (except NAND-FLASH) because of huge mass of result system. Besides that it is hardly to make a difference for different importance index data.
Second, storage with replication of data leads to significant increasing of data redundancy and respectively to decreasing efficient storage capacity.
Third, evaluation fault tolerance of HDSS performed for systems "at average" and not takes in to account importance index of data.
Fourth, for onboard data processing needed large volumes of reliable memory with limitless (or very large) count cycles of write and read before failure.
Current approaches to HDSS design based on replication of data and using non-reliable NAND FLASH can not ensure such opportunities regardless of the level of scaling and improvement of technological processes due to constructive limitations.
Five, using of non NAND FLASH MU will allow save energy resources and use them for data transformation for more reliable storage and reducing the level of information redundancy.
Task of HDSS design is form from array of available memory modules types and controllers such structure of HDSS that met the fitness criteria. Besides that it is necessary to choose constructions of regenerating and error correct codes and data placement for each importance index of data. Forming storage methods should met criteria of minimal information redundancy. Parameters of HDSS with such storage methods too should met the fitness criteria.
Storage methods should be formed for maximum payload of HDSS (initial storage methods). Storage methods (combination of construction of regenerating and error correction codes and data placement) should be corrected according with system condition. Storage methods may be form for each data obtaining and should lead to maximum of data reliability.
Thus task of designing and application consist of two sub-tasks. First, subtask is synthesis of HDSS structure and storage methods.
Second, subtask of HDSS management.
In the math form, the problems of constructing and using a heterogeneous storage system can be described as follows.
1. Subtask of HDSS structure synthesis and storage methods.
Given:
1. HDSS operational requirements:
(Dmax,Mmax, Emax, Vmin) ,
where
D - HDSS overall dimensions; M - HDSS mass; E - HDSS energy consumption; V - HDSS effective capacity.
2. Requirements for HDSS performance indicators:
{PElmm, PDNLmin, URmin, ^^min),
where
PELmin - requirements for data errorless probability; PDNLmin - requirements for data availability probability; URmin - requirements for HDSS productivity at read mode; UWmin - requirements for HDSS productivity at write mode;
3. Structure of data:
(K, Fk),
where
K - number of importance levels at earth remote sensing data; Fk - specific capacity of k-th data type.
4. Available set of memory chips:
MP = {mpj};
5. Available set of regenerating codes: A = {cij}, possessing properties:
ci = {ni,kt),
where
kl - is number of blocks to which the data is divided; n - number of blocks to which the data is encoded;
Red (a) = —- - information redundancy, introduced by rege-kl
nerating codes.
6. Error-correcting codes: C = {cn }, possessing properties:
cn = (nn . h :
where
nn - length of code block in bytes;
kn - numbers of information digits in error-correcting code; - degree of error-correcting.
n
RedCn =— - information redundancy, introduced by error-
kn
correcting code. Find: Structure
So = (nn, n22v, nJl I,
where
nM - number of MM j-th type in b i-th SU, storage methods Sp = (ai ,b, cn), such that:
Uw (So, Spk), Red (So,Spk )> - HDSS performance indicator;
GII = {PELk (So, ^ SPk ) > PELmink n nPDNLk (S0 St, SPk ) > PDNLmink H UR (S0 St, SPk ) >
> URmin n UW (So,St,Spk) > UWmin} - HDSS fitness criterion.
Under the constraints: HDSS should met requirements for onboard hardware: ^^ € , where:
Y4)1 =< D (So) < Dmax n M (So) < Mmax H nE (So) < Emax n Vef (So,SPk ) > Vmin > .
2. Subtask of HDDS management when solving target tasks. Given:
1) HDDS structure CXfl: So.
2) HDDS state: SS = [ssIS }.
3) New data: DN = (VN, K, FK), where VN - new data volume.
Find:
Storage methods: (Spk (t
o .Sp0 (tAF ) y.Y^ (So.Spk (tAF )))e {Gn }
%) =< PELk (So.SPk ).PDNLk (So.SPk ),UR (So.SPk ).
PSk(So,Ss,Spk (t))argmaxPSk(So,Ss,Spk(t))j, under the constraints:
Spk* (t )€ SpkD, Red (So, Ss, Sp* ) < Red (So, Ss, Sp° ).
Algorithm of data storage based on combining of the methods of locating and coding non-homogeneous data
Synthesis of the heterogeneous structure occurs under conditions of a priori uncertainty of the functional conditions HDDS. On the one hand, the impact of unfavorable space factors on the most frequently used orbits for spacecraft is well studied and can be accepted as normal operating conditions.
On the other hand, at the spacecraft lifetime cycle, the impact of various unforeseen factors is possible: the effect of a physical collision with other objects (space debris), etc.
According with math formulation of problem Algorithm should solve both subtasks of designing and management of HDDS.
Formation of requirements for HDDS is defined as one of the stages of external design. Based on the research of HDDS systems as an element of onboard aperture, analysis of interaction with other structural elements of the system, necessary technical characteristics are determined and a feasibility study is carried out. The results of the external design stage are formulated in the form of technical requirements for the onboard computing system. In most cases, the requirements for the performance HDDS can be satisfied with a variety of options for the composition and
parameters of the system. In this case, the cost of development and operation, the availability of technological back-up and the used software can become decisive when choosing system options.
At the first stage, the initial data is collected. It consists in a comprehensive analysis of the requirements for the operation conditions of the satellite, analysis of technical requirements for HDDS, as well as available memory options [11].
Also before or at the stage of system-wide design, data are collected on reliability indicators of various drive options. The data can be obtained from the manufacturer on the basis of tests carried out by them or the test data can be produced by the designer in accordance with the approved methodology.
Algorithm of data storage based on combining of the methods of locating and coding non-homogeneous data structurally consist of two parts. Each part solving corresponding subtask.
For first subtask of HDSS structure synthesis and initial storage methods it is necessary to execute the following sequence of steps.
Step 1. Based on PELmin and available for implementation constructions of error-correcting codes C = {cn} for each of memory modules - mpcj a choice is made of such constructions of error-correcting codes which provide the required probability of non-avoidable errors at k-th type data set:
PELkcj (ckcj ) ^ PELmink •
In this case, the minimum information redundancy conditions must be satisfied:
K
Ckcj *) = argmin{reJ(Ckcj)},
where
red (cfcj) - information redundancy introduced by the error-correcting code used for the k-th data type in the j-th type MM in combination with the c-th MC.
Step 2. Based on chosen types and construction of error-correcting codes for each data type and combination of MM and MC calculating effective data capacity: Vefcj = vcj- • red(cfcj-).
Step 3. For each data type forms many combinations of substructures and corresponding construction of error-correcting codes. It should be noted that when forming substructures, the assumption that within the substructure, the data is evenly dis-
tributed, i. e. b.
V
kcj
kcj
min Vkcj cj
. This assumption is possible
because of during HDDS application data placement is not even and thus probability of data loss is much less.
During forming substructure and storage methods solving following optimization problem:
{\ Sok, Ak /} = PDNLk (Sok, Ak ) ^ PDNLmink,
where
So, - substructure of HDDS.
k
This problem can be solved by a known method of borders and brunches.
Step 4. Forming of full structure of HDDS can be find by solving classic Multiple-choice knapsack problem:
1) For each data type k set of vectors enumerates from 1 to j.
2) Fmd: max £ £ Vef^
i=ljGM (Sok ) K
£ £
i=\jsN (Sok )
K
£ £ gij t=ljEN (Sok )
K
£ £ e
i=ijeN(Sok )
£
j xij, with restrictions:
■j 1J
• xij — Mmax;
xij — ^max ;
• x — F •
ij max
ij
= 1;
j€M (Sok )
i = 1... K;
Xj €{0,1};
j € N (Sok).
Problem can be solved by dynamic programming method. As result forms full HDDS structure that meet operational requirements for system and has maximum effective capacity.
Step 5. To determine the suitability HDDS structure it is necessary to carry out simulation modeling.
According to the results of modeling, the decision on the compliance of the functioning is made - Gir
If following conditions are true:
So, Spk *) : Y4 ( So, Spk ))e{Ga }
HDSS structure synthesis and initial storage methods is considered complete. In the case of failing to meet the specified requirements, make the necessary adjustments and the implementation of the synthesis algorithm is repeated.
Main of function time there are exist sufficient information reserve of HDDS capacity because of ERS systems doses not work with full payload and has regular communication sessions. Fault tolerance of HDDS can be enhanced by using this reserve.
At the same time, certain restctions on the quality of operation remains. HDDS should meet fitness criterion.
Optimal operating mode (in terms of HDDS fault tolerance) should meet HDSS optimality criterion:
Oil = {< PELk (So, Ss, SpK )= argmax PELk (So, Ss, SpK )n nPDNLk (So ^ SPK )= arg max PDNLk (So, ^ SPK ) n
nTR (So, Ss, SpK )< TRmax n TW
(So, Ss, SpK )< TWmax n nRed (So, Ss, Spk )< Red (So, Ss, Sp0 )>
In such way tasks of HDDS management may be formulated as following: it is necessary to find such data placement at HDDS structure and construction of regenerating codes that makes meet HDSS optimality criterion.
For the subtask of HDDS management when solving target tasks it is necessary to execute the following sequence of steps.
Step 1. When new data received by HDDS functional control is carried out.
According to the results of functional control, the HDDS state vector is formed: SsS = {'VS ,VuS).
Step 2. New arrived data object DN = (VN,K, FK} divided in to k regions according to data type.
Step 3. Based on the initial storage methods - Spk0 computes permissible redundancy - Red (Spk ).
Step 4. Based on the HDDS condition and permissible redundancy for regenerating codes - Red («k ) selected regenerating codes construction:
kk =
Vk
min( Vs -Vus)/ Red (ak
s
nk = Red (ak0) • kk , where
Red (ak0) - information redundancy.
Step 5. The next step is to find the placement of data blocks by structure. The task is formulated as follows: it is necessary to find the bijection bk : nk ^ s , such that: PDNLk (bk) ^ max. This task is proposed to solve the Cauchy Method. Step 5. According with chosen data placement - bk, and restrictions on the information redundancy - Red (ck ) calculating constructions of error correction codes:
(cks*) = argmaxPELk (cks ) ,
where
cks - chosen type and construction of error correction codes for k-th data type and s-th MM.
Step 7. Based on the way of storage for k-th data type Spk = (ak,bk, ck) corrected system statement Sss for further calculations of storage methods.
Step 8. Similarly, can be chosen storage methods for other data types.
Step 9. Simulation of storage system functioning in accordance with the operating model is carried out. In case of meeting the requirements:
So , Spk : Î4 (So, Spk }
data writes with calculated storage methods. In the case of failing to meet the specified requirements, it is the necessary to make specified adjustments and repeat an algorithm.
Conclusion
Thus, based on the formulation of problems of designing and application HDDS, they can be attributed to the class of combinatorial optimization problems. There are known ways to solve this type of problem. However, it is worth noting a few features.
Thus problem of searching of HDDS optimal structure and storage methods for full payload can be solved by different ways, including by a full search method.
However, this is unacceptable for the HDDS application, since the choice of the method of coding and placement of data blocks for storage must be made promptly and decided on the control device storage, which has a relatively low performance.
In this regard to solve it, a combined method for selecting the type of regenerating encoding and placing data blocks on the storage is proposed.
The choice of error correction codes based on the type of data block and type of MM on which this block assigned.
The proposed approach to providing fault tolerance of remote sensing information processing system, based on design and application HDDS allows flexible storage process management. Besides that it allows to correct HDDS properties.
This takes into account the initial importance of the remote sensing data areas, the structure and statement of HDDS as well as compliance of the operating characteristics of HDDS requirements.
References
1. Petrov A. G., Ulanova A. V., Chumakov A. I., Vasiliev A. L. The study of information loss in the circuits of flash memory in active and passive modes, when exposed to ionizing [Issledo-vaniya poteri informatsii v mikroskhemakh flesh-pamyati v ak-tivnom i passivnom rezhimakh pri ioniziruyushchem vozdeyst-vii]. Radiation resistance of electronic systems. Moscow, 2014, Vol. 17, pp. 175-176. (In Russ.)
2. Savinyh V. P., Solomatin V. A. Electro-optical system for remote sensing [Optiko-elektronnye sistemy distantsionnogo zondiro-vaniya]. Moscow, Nedra, 1996. 315 p. (In Russ.)
3. The concept of development of Russian space system of remote sensing for the period up to 2025 [Kontseptsiya razvitiya rossiyskoy kosmicheskoy sistemy distantsionnogo zondirovaniya zemli na period do 2025 goda], Moscow, Federal space agency, 2006. 72 p. (In Russ.)
4. Zaharov I. V., Kremez G. V., Maksimov V. A. Construction of distributed onboard storage systems for remote Earth-sensing satellites [Postroenie raspredelennykh zapominayushchikh ustroystv bortovykh vychislitel'nykh sistem kosmicheskikh apparatov distantsionnogo zondirovaniya zemli]. Proc, Mo-zhaisky Military Aerospace Acad, [Trudy voenno-kosmicheskoy akademii imeni A, F, Mozhayskogo], 2016, Is. 652, pp. 160-166. (In Russ.)
5. Denisov A., Demin A., Letunovskiy A. Optical Digital Systems and Complexes for Space Applications. Intellectual Technologies on Transport, 2016, no. 1, pp. 16-22.
6. Dimakis A. G., Ramchandran K., Wu Y., Suh C. A Survey on Network Codes for Distributed Storage, CSIT, 2010, pp. 12-18.
7. Wu Y., Dimakis A. G., Ramchandran K. Deterministic Regenerating Codes for Distributed Storage. Allerton Conf, Control Computing and Communication, 2007, pp. 236-242.
8. Weatherspoon H., Kubiatowicz J. Erasure coding vs. replication: A quantitative comparison. Proc, First Int, Workshop on Peer-to-Peer Systems (IPTPS 2015), 2015, pp. 65-78.
9. Molette P., Cougnet C., Saint-Aubert Ph., Young R. W., Helas D. Technical and Economical Comparison Between a Modular Geostationary Space Platform and a Cluster of Satellites. Acta Astronautica (Pergamon Press Ltd,), 1984, no. 12 (11), pp. 771-784.
10. Herz E. EO and SAR Constellation Imagery Collection Planning. Proc, 14th Int, Conf, Space Operations, SpaceOps AIAA, 2014. 214 p.
11. Haeusser G. B., Osuna A., Bosman Ch., Jahn D., Tarella G. J. ILM Library: Information Lifecycle Management Best Practices. IBM, 2016. 256 p.
Алгоритм хранения информации на основе комбинирования способов
размещения и кодирования неоднородных по важности данных
Максимов В. А. Военно-космическая академия им. А. Ф. Можайского Санкт-Петербург, Россия falcon225@yandex.ru
Аннотация. Предложен подход к построению и применению систем хранения данных перспективных космических аппаратов на основе гетерогенных запоминающих устройств с целью повышения отказоустойчивости функционирования систем обработки информации. Применение гетерогенных накопителей допускает более гибкое управление способами хранения, что позволяет снизить степень информационной избыточности всех видов в системе, а также гибко регулировать эксплуатационные параметры системы. Новизна работы заключается в учете неоднородности важности хранимых данных.
Ключевые слова: система хранения данных, гетерогенность, распределенность, отказоустойчивость, безошибочность, дистанционное зондирование Земли, восстанавливающее кодирование.
ЛИТЕРАТУРА
1. Петров А. Г., Уланова А. В., Чумаков А. И., Васильев А. Л. Исследования потери информации в микросхемах флэш-памяти в активном и пассивном режимах при ионизирующем воздействии // Радиационная стойкость электронных систем - «Стойкость-2014». - М., 2014. Вып. 17. С. 175-176.
2. Савиных В. П., Соломатин В. А. Оптико-электронные системы дистанционного зондирования. - М.: Недра, 1996. 315 с.
3. Концепция развития российской космической системы дистанционного зондирования земли на период до 2025 года. -М.: Федеральное космическое агентство, 2006. 72 с.
4. Захаров И. В., Кремез Г. В., Максимов В. А. Построение распределенных запоминающих устройств бортовых вычислительных систем космических аппаратов дистанционного зондирования земли // Тр. ВКА им. А. Ф. Можайского. 2016. Вып. 652. С. 160-166.
5. Denisov A., Demin A., Letunovskiy A. Optical Digital Systems and Complexes for Space Applications // Intellectual Technologies on Transport. 2016. № 1. P. 16-22.
6. Dimakis A. G., Ramchandran K., Wu Y., Suh C. A Survey on Network Codes for Distributed Storage // CSIT. 2010. Р. 12-18.
7. Wu Y., Dimakis A. G., Ramchandran K. Deterministic Regenerating Codes for Distributed Storage // Allerton Conf. Control Computing and Communication. 2007. P. 236-242.
8. Weatherspoon H., Kubiatowicz J. Erasure coding vs. replication: A quantitative comparison // Proc. First Int. Workshop on Peer-to-Peer Systems (IPTPS 2015). 2015. Р. 65-78.
9. Haeusser G. B., Osuna A., Bosman Ch., Jahn D., Tarella G. J. ILM Library: Information Lifecycle Management Best Practices. - IBM, 2016. 256 p.
10. Herz E. EO and SAR Constellation Imagery Collection Planning // Proc. 14-th Int. Conf. Space Operations, SpaceOps AIAA. 2014. 214 p.
11. Molette P., Cougnet C., Saint-Aubert Ph., Young R. W., Helas D. Technical and Economical Comparison Between a Modular Geostationary Space Platform and a Cluster of Satellites // Acta Astronautica (Pergamon Press Ltd.). 1984. № 12 (11). Р. 771-784.
