CC BY-NC-ND
86
MODULAR DATA CENTER: THE HOLISTIC VIEW
Maxim AMZARAKOV
Director, INO Uptime Technology
Address: 9, Chasovaya str., Moscow, 125315, Russian Federation E-mail: m.amzarakov@uptimetechnology.ru
Rafael SUKHOV
Finance Manager, INO Uptime Technology Address: 9, Chasovaya str., Moscow, 125315, Russian Federation E-mail: r.sukhov@uptimetechnology.ru
Eugene ISAEV
Professor, Head of Department for Education Stack Group, National Research University Higher School of Economics; Head of the Laboratory, P.N.Lebedev Physical Institute of the Russian Academy of Sciences Address: 20, Myasnitskaya str., Moscow, 101000, Russian Federation E-mail: eisaev@hse.ru
^ Datacenter modularity is a new term in data processing. The given article aims at making distinction between the ^ modular, mobile, modern and traditional datacenter, and reviews solutions available in the market.
The present research provides a systematic view of the currently existing data center technology solutions, as well as the major factors influencing the cost and possible engineering pitfalls and determining basic rules of calculating and evaluating their cost and further maintenance. The concept of energy efficiency is studied here, as well as its influence on the primary cost of a data center, its maintenance cost, and thus its final cost. The conditions for the modular solutions for data processing centers emergence have also been studied here. Classifying and identifying key features allows precise positioning of the applicability of existing technologies. For this purpose, the paper provides major features of the applicability limits of available technologies, while technological solutions from different vendors are evaluated for containing engineering systems. The received estimations are presented in a convenient and comparable tabular form and.
The research results are provided in the form ofa summarizing table allowing comparing the features of each solution in several aspects: form factor, complete solution, modularity, flexibility, further development in several key engineering solutions.
Key words: datacenter, modular datacenter, IT industry, information technologies.
Introduction
The «Datacenter world» today is a numerous set of products and solutions providing end-user with the computational and storage resources. Some people treat the datacenter as «High level» services such as Cloud term SaaS, which is quite different in fact. A datacenter is defined as a facility used to host
computer systems and associated components, such as telecommunications and storage systems. It generally includes redundant or backup power supplies, redundant data communications connections, environmental controls (e.g., air conditioning, fire suppression) and security devices [1]. The diversity of meaning of the term «datacenter» confuses the understanding of the topic.
Datacenters do not provide any other services except IT and telecommunication equipment hosting. But the most common view of a datacenter is «something I can connect to» or «something which does the data processing for me». The truth is somewhere in between. Both approaches to datacenters can be applied if we say that a datacenter is both an infrastructure to host computers and computers with the network it self. In order to clarify the view of the topic we define a datacenter as follows:
A data center is a highly technological facility to host computer technologies and data storage to ensure the continuity and reliability of data processing.
In other words, a datacenter is the physical layer supplying computer equipment with power and network wired connections. This notion is used in consideration for the purposes of this article.
1. What a datacenter needs
In general, the aim of a datacenter facility is to host the computing equipment, data storing and telecommunication equipment.
Basically, all requirements for datacenters come from the business. These demands are directly or indirectly linked to requirements of the IT equipment/systems.
These requirements can be ranged by their reliability, sustainability, maintainability, power consumption, data floor space etc.
In the early days of the IT the datacenter design was fixed and depended on the IT equipment for which a datacenter was designed. «Computer room» was the word to describe infrastructure supplying power and cooling for specific IT hardware. As computer industry grew, server rooms became more and more univesal and could host different types of IT hardware. The diversity of IT equipment today causes headaches for data-ceter designers. And datacenters are becoming more and more flexible.
The source of the requirements (inputs) changes the datacenter design and the design process itself. Two following examples will show you how datacenter design can vary and be sophisticated depending on the source of the requirements.
Example 1: indirect requirements
One big international company is planning to deploy the Enterprise Resource Planning (ERP) system. The company has 200K employees accessing the system. The company also intends to improve the documenting and the internal processes by implementing remote desktop environment based on the IaaS services.
This requires a number of IT equipment, data storage and telecommunication infrastructure installed in the datacenter. Designing these systems requires a certain amount of computing resources and data storage, as does designing the telecommunication (network) part the system. This kind of the requirements can be easily transformed to ones for datacenter facility infrastructure. The designer of a datacenter selects infrastructure components on the basis of a fixed set of «consumers» of datacenter capacities.
The datacenter, designed and set up in this example, is specially developed for the specific needs (inputs) of the equipment hosted in it. This is an «ideal» example of datacenter design from the designer's point of view.
This is an example of how indirect business shoul define a datacenter. The company's direct requirements are reflected in its IT system.
Example 2: direct requirements
The second example comes from direct business needs — a commercial data center.
The company is planning to build a datacenter to provide basic services like location of goods.
The requirements in this example can only be described though the number of crates (e.g. floor space) and the planned power capacity, otherwise the designer will not be able to choose appropriate cables, sockets, network cabling and etc. Also no information was provided about the required reliability of the infrastructure.
Details
The specifics of the data center allow seeing the kinds of criteria which are simpler/harder to reach.
♦ The data center today (based on the IT needs) can
be described by the folloThe floor space footage;
♦ The power density per square feet;
♦ The power efficiency;
♦ The reliability (or expected uptime);
♦ The lifetime.
The floor space and power capacity are defined straightforward in both examples. Power efficiency is more valuable for commercial DC, we assume. Reliability can be defined simply in the first example and becomes a complicated question in the second one.
Most of the customers coming into the datacenter can say: I want to have a Tier 3 [2] level of uptime for my servers and PUE less than 1.2 [3, 4, 5]! In the examples above the reliability of the last datacenter can be higher than that of the first one and much higher than that of
Healt Load 100,000 for Equipment Footprint (Watts/equipment sq. ft.)
10,000
100
10
Year of Announcement
1992 1994 1996 1998 2000 2002 2004 2006 2008 2010 2012 2014
— Communication - Extreme Density Compute Servers - 1U -5K-- Storage Arrays —I— Tape Storage
—■— Communication - High Density --X— Compute Servers - 2U+ —•— Workstations (stand-alone)
Fig. 1. Historic and projected head load by equipment type. Source: D. Floyer, «Network Go GrEEN», Wikibon, May 07, 2011 (http://wikib0n.0rg/wiki/v/Netw0rks_G0_GrEEN)
the third (XaaS seems to be more redundant). What is the reliability standard you should choose for your application? What is reliability standard the designer has to take into account in the absence of the customer whom he can ask about it in the last example?
Lifetime
Lifetime of the IT equipment is nearly three years [6]. The requirements of the new IT equipment do not change in the form-factor. But these parameters will be completely different in terms of network ports number, power consumption per unit (see fig. 1), etc. So after 3 years the data center equipment shall be renovated.
The lifetime of the facility technologies, like cooling or power distribution, is close to 10 years after the installation.
We are well aware that in this case it is practically hard to justify the investment, if the equipment has served for three years only [2].
More details. Flexibility
As you already can see, the first example becomes less and less simple in the design as we start thinking farther than three years.
Design is much more detailed in the first example. All plugs, network cords, places, spaces are predesigned, hence we have to provide IT system and IT hardware design.
In the case of unpredictable IT hardware demand, the cabling infrastructure, power distribution infrastructure of the datacenter should be flexible enough to easily adopt to new IT equipment.
Keeping in mind the «life time issue», the flexibility in capacities, in network connections, in space layout,
there tends to be an ultimate solution for the datacenter.
This can be true. Is the datacenter design flexible enough to make the final step of the datacenter growth completely different than designed initially? If it is possible, can the datacenter be modified by the capacity reduction or changed after several years of operations without the critical load interruption?
Power
The necessity to modify the datacenter design also comes from the technical characteristics of IT equipment to be placed in it. As mentioned above, the lifetimes of the IT equipment and datacenter facility infrastructure differ significantly. The latter one is three times more than the former one. At this point we expect the datacenter requirements will have to be substantially changed every three years because of the IT equipment. The regular server in the early 2Ks consumed 50W; modern server consumes at least 300W [7, 8]. The regular servers seven years ago had one or two Ethernet interfaces and most of them were connected to one port, today servers have three interfaces at least (one is for management) and most of them are connected to the datacenter network [9]. So we can say once every three or five years we face a significant change[2].
Timeframe
We spoke before about power demand, space capacities and possible change in three or five years. What if the change comes earlier? Is there a reason to have this change? The growth of IT is very rapid all over the world. This enhances the computer and automation system
growth and change, IT growth and changes, datacenter facilities grow and change, one after another. There are a lot of examples when a data center becomes outdated before it starts working, because IT requirements of the company changed too fast.
The usual time frame of the datacenter development is 6months for design and one year to buy and build it. Can one imagine how many changes IT world can face with in the one year and a half? A new iPhone is released every year with half its version new. A new Microsoft Windows appears approximately every 1,5 years [10]. This can be the planned growth you can predict. But can you predict the future?
Power efficiency
One of the data center characteristics is caused not by IT, but by the money limit. The power consumption of the IT equipment (see fig. 1) and the price per kilowatt are growing rapidly. Both facts lead to the basic answer: we have to save money!
The efficiency of the data center is described by the proportion between the power consumed by the data center and the power supplied to the IT equipment. These characteristics are called PUE and calculated by the formula:
PUE =
p
where PIT is the amount of power consumed by IT hardware.
The number theoretically can be as low as 1. The data center power efficiency is better as PUE number is lower.
Drivers
There is always a solution when the requirements are identified. Modular datacenters come to cover the above-mentioned needs [4]. There is also a specific use of such solutions, but this will be covered later in this article.
To recap above mentioned, we should say the demands which drive the datacenter technology development are the following:
♦ reliability;
♦ flexibility;
♦ deployment time;
♦ mobility;
♦ power efficiency.
2. Terms
Keeping in mind examples and requirements discussed above, we must define mobile, modular, modern and traditional types of data center.
We could say mobile and modular types are modern, while modern is not only mobile and modular.
Traditional
The traditional approach to building a data center is to construct concrete/brick solid building and bring all necessary infrastructure in. Everything is fixed and cannot (or is very expensive to) be changed.
Modern
The modern data center reflects the modern needs of flexibility, deployment time, mobility, reliability and power efficiency. Which cannot be achieved in a «concrete» building. And of course, a modern data center can be a «concrete» solid building, but very efficient as it must be today.
Mobile
The data center can be dissembled and moved to another place. Such kind of solution is highly prefabricated. And most of the times it is reliable as the consequences of «factory made product». Mobile solutions address the deployment time issue. The TCO of containerized solution can be as little as the lifetime of the IT hardware filled in the container. Should it be lifetime issue solution? Mobile can also be modular.
Modular
If necessary, flexibility can be the main driver for the modular approach of the datacenter design.
Flexibility of all the components in the data center and the flexibility of the investments are the major characteristics of the modular solutions.
A modular data center (as the word suggests) consists of modules. The modules are the infrastructure components of the datacenter. Modules give the possibility of «on-demand» growth of the datacenter. These modules can also be highly prefabricated, reducing the delivery and installation time.
3. Modular containers
The above-mentioned demands have their solution. Containerized datacenters are coming in to the market. This chapter describes five world known and real operation-proven solutions of this kind.
In this and the following chapters we intend to describe the existing modular solutions briefly. The tables describing the solution show which component of the data center is included, which is not included, and which is partly included (checked in both rows).
Microsoft
Microsoft modular datacenter's key feature is the virtual computing environment. The solution is based on integration of IT and networking equipment in one container. The container is produced and shipped to the customer from the factory where all the systems inside it are prefabricated and tested [10].
The centralized power supply with autonomous power generating and cooling supplies have to be ready on site to allow container installation.
The centralized systems can be modular in units, but not easily modular in distribution. The backbone distribution system of power and cooling has to be prebuilt on site and has the «end situation» capacity. This requires more investments at the first growth stage and restricts the change of the end state capacity and sometimes the stage of growth.
Table 1.
Microsoft MDC
0 1= a-
CO OJ 3 u S 75 3
Q. OJ OJ 0 0 ¡{г
= u_ CO z 0 ^ о
Included ✓ ✓ ✓ ✓ S
Not included ✓ V s У
SUNMD
The first well-known modular datacenter was SUN Project Black box.The modularity of this solution meant the ability to bring the computing resources to the place where they were necessary. Sun Project Black box included up to 280 servers built into a ISO-standardized shipping container. Sun Microsystems was purchased by Oracle in 2010. Now the project is named Sun MD s20 [11]. It unifies infrastructure and IT hardware monitoring and controls. The power and cooling for the IT equipment has to be supplied by site infrastructure. The solution is applicable for fast deployment where IT demands are not vast and can fit into one or a couple of containers.
Table 2.
SUN MD
ra
CO CD 3 u I 75 <D 3
= UL CO Z 0 OL 0
Included ✓ ✓ ✓ ✓ ✓
Not included ✓ V У У
ASTModular
AST started with the containerized datacenter and then came to the modular datacenter infrastructure. AST today has a set of modular products and solutions [12]. IT container has integrated uninterruptible power supply. Availability of water-cooled block container, direct and indirect cooling block container allows choosing the appropriate cooling solution for certain site and weather conditions. Power container and vestibule container complete the containerized modular data center solution.
Table 3.
AST Modular
О s a-
CO OJ 3 u S 75 3 <J
OJ OJ 0 0
= U_ CO z 0 0
Included ✓ ✓ ✓ ✓ У
Not included
Huawei CDC
20 feet or 40 feet sea containers [13]. The key feature, beside low price, is the indirect free-cooling incorporated into the container. The solution also has the «cooling container», «tambour» and power container which can be stacked to the «data container» providing the modular data center solution.
Table 4.
Huawei CDC
О s
CO OJ 3 u S 75 S _ÇJ
Oi Oi 0 0
= U_ CO z 0 a. 0
Included ✓ ✓ ✓ ✓ ✓ У
Not included ✓ ✓
HP POD
HP Performance Optimized Datacenter (POD) 20c is based on the superstructure of special design [14]. It does not use the «traditional» ISO shipping container, which restricts entrance openings, inside interior etc. The block size is the same because of transportable dimensions. High temperature chilled water close-coupled heat removal provides higher energy efficiency but requires the cooling plant on site.
Table 5.
HP POD
О s
CO OJ 3 u S 75 S
Q. OJ OJ о о
= UL CO z о а. о
Included ✓ ✓ ✓ ✓ V S
Not included
4. Modular solutions
All of above-mentioned solutions use containers of fixed size. Most of these solutions require the site infrastructure for power and cooling to connect to.
The container solution is the best choice in case of accidental or partial IT demand.
The free choice in power and cooling infrastructure, the necessity of interconnection between the «building block» using the containerized design, the fixed data-floor space and server rack dimensions restrictions are the drawbacks of containerized solutions and limit their usability as the basis of large datacenters.
Examples of that are the AST NON ISO Modular Data center and Colt Modular Datacenter.
ASTModular
AST NON ISO Modular datacenter is based on the same set of cooling, power, IT room building blocks as its containerized solution. But the outside is different. The cooling is provided by modules of different capacities and sizes. Availability of IT room block of two sizes (25' and 45') allows to assembling the data room according to the customer's needs.
Table 6.
AST Modular NON ISO
о is
со 0J 3 u S 75 3
0J 0J о о ¡{г
= U. со z о о
Included ✓ ✓ ✓ ✓ ✓ ✓ ✓
Not included
Colt
The Colt modular data center is one of the containerized solutions. Colt has the growth «block» of data-
center space and power. [15] Most of the construction blocks are produced at the factory. The customer selects the floor space block (server rooms) of his datacenter. Then Colt is able to provide the block-to-block growth of the datacenter.
Table 7.
Colt Modular DC
о s
со 0J 3 U S 75 S _ÇJ
0J 0J о о
= UL со z о а. о
Included ✓ ✓ ✓ ✓ ✓ S ✓
Not included
Conclusion
There is no «general» description of modular datacenter today. You can see from examples above many directions of modularity in technologies. The means of modularity, noted in the article of J. Rath «DCK Guide to the Modular Datacenter» [16], are the following:
♦ deployment;
♦ consumption;
♦ financing.
The modular datacenter differs from traditional at least by flexibility and modularity in the deployment process. At the same time modular datacenter does not not require to be mobile, but can also be constructed from mobile blocks.
The author's definitions of pure modular datacenter are:
Ф step by step growth of basic capacities: space, power, cooling;
Ф changeability of the capacities during the DC operations;
Ф flexibility in easy IT demand adaptation.
We include all the described above features of mobile and modular solutions into one table to make the comparison easier. Table 8 allows comparing the advantages and disadvantages of each solution. Does the solution include Power and Cooling or is this required on site? Is it possible to modular growth of the power inside the container? Further develop -ment availability and the flexibility are also shown in the table. ■
Table 8.
Modular datacenter comparison
Solution Modular Growth Flexibility
M t
o ra ra ra « a ra
s ** J3 u <B <D 3 ol o <0 «u S ol u <0 <D 3 ol ■JS g ol <D g
> CO CO OL o CO OL o CO OL o o £ o OL
Oracle Sun MD O O ft O O O O o O O O
Microsoft ft o 1 1 o o o o O o o
AST ISO • • • • • • • o • • •
Huawei CDC • • • » • • • o • • •
HP POD ft o 1 1 o o o o O o o
AST NON ISO • • • • • • • • • • •
Colt • • • • • • • • • • •
References
1. Kelley C., Cooley J. (2011) Deploying and Using Containerized / Modular Data Center Facilities. The Green Grid White Papers.
2. The Uptime Institute: http://www.uptimeinstitute.com/(accessed 30.07.2013).
3. Sukhov R.R., Amzarakov M.B., Isaev E.A. (2013) Advanced Data Center Economy. Business Informatics, no. 2 (24), pp.13-18.
4. The Green Grid Consortium: http://www.thegreengrid.org/Global/Content/white-papers/The-Green-Grid-Data-Center-Power-Efficiency-Metrics-PUE-and-DCiE (accessed 05.07.2013).
5. Uptime Technology Russia: http://www.uptimetechnology.ru/evaluation_pue.html (accessed 30.07.2013).
6. Intel: http://www.intel.com/(accessed30.07.2013)
7. Turner IV P.E., Brill K.G.(2010) Cost Model: Dollars per kW plus Dollars per Square Foot of Computer Floor. The Uptime Institute White paper.
8. Koomey J. (2007) Estimation Total Power Consumption by Servers in the U.S. and the World. Jonathan Koomey Analytics Press.
9. Minas L., Ellison B. (2009) The Problem of Power Consumption in Servers. Intel.
10.Microsoft Company: http://www.microsoft.com/(accessed3 0.07.2013) 11.Oracle: http://www.oracle.com/(accessed 30.07.2013)
12.AST Modular: http://www.astmodular.com/(accessed 30.07.2013)
13.Huawei: http://www.huawei.com/(accessed 30.07.2013)
14.Hewlett Packard Company: http://www.hp.com/(accessed 30.07.2013)
15.Colt Technology Services Group: http://www.colt.net/ (accessed 30.07.2013)
16.Rath J. (2013) DCK Guide To Modular Data Centers: Second Edition. DataCenterKnowledge Journal.
МОДУЛЬНЫЙ ЦЕНТР ОБРАБОТКИ ДАННЫХ: ЦЕЛОСТНЫЙ ВЗГЛЯД
М.Б. АМЗАРАКОВ
Директор АНО «Институт «Аптайм» Адрес: 125315, г. Москва, ул. Часовая, д. 9 E-mail: m.amzarakov@uptimetechnology.ru
Р.Р. СУХОВ
Финансовый управляющий АНО «Институт «Аптайм» Адрес: 125315, г. Москва, ул. Часовая, д. 9 E-mail: r.sukhov@uptimetechnology.ru
Е.А. ИСАЕВ
кандидат технических наук, профессор, заведующий базовой кафедрой Группы компаний «Стек», Национальный исследовательский университет «Высшая школа экономики»; заведующий лабораторией, Физический институт имени П.Н. Лебедева Российской академии наук Адрес: 101000, г. Москва, ул. Мясницкая, д. 20 E-mail: eisaev@hse.ru
Настоящая работа ставит перед собой цель систематизировать существующие в мире технологические решения в области центров обработки данных. В статье рассмотрены основные факторы, влияющие на стоимость и инженерную сложность дата-центров и определяющие базовые правила формирования и оценки стоимости их создания и последующей эксплуатации. Рассмотрено понятие энергоэффективности и еевлияние на первичную стоимость дата-центра, стоимость его эксплуатации и, как следствие, совокупную стоимость владения. Также рассмотрены предпосылки появления модульных решений для центров обработки данных. Систематизация и определение ключевых отличий и особенностей позволяют точно позиционировать применимость существующих технологий. Для этого в работе изложены ключевые особенности и определены границы применимости существующих технологий, а технологические решения разных производителей оценены на предмет укомплектованности инженерными системами. Полученные оценки приведены в удобном для сопоставления табличном виде.
Итогом работы является обобщенная таблица, позволяющая сравнить возможности каждого решения в нескольких аспектах, включая форм-фактор, законченность решения, модульность, гибкость, возможность развития по ^ нескольким ключевым инженерным решениям. ^
Ключевые слова: центр обработки данных, модульный центр обработки данных, ИТ-индустрия, информационные технологии.
Литература
1. Kelley C., Cooley J. Deploying and Using Containerized / Modular Data Center Facilities. The Green Grid White Papers, 2011.
2. The Uptime Institute: http://www.uptimeinstitute.com/(^aTa обращения 30.07.2013).
3. Sukhov R.R., Amzarakov M.B., Isaev E.A.Advanced Data Center Economy// Бизнес-информатика. 2013. № 2 (24). С.13-18.
4. The Green Grid Consortium: http://www.thegreengrid.org/Global/Content/white-papers/The-Green-Grid- Data-Center-Power-Efficiency-Metrics-PUE-and-DCiE (дата обращения 05.07.2013).
5. Uptime Technology Russia: http://www.uptimetechnology.ru/evaluation_pue.html (дата обращения 30.07.2013).
6. Intel: http://www.intel.com/(дата обращения30.07.2013)
7. Turner IV P.E., Brill K.G. Cost Model: Dollars per kW plus Dollars per Square Foot of Computer Floor. The Uptime Institute White paper, 2010.
8. Koomey J. Estimation Total Power Consumption by Servers in the U.S. and the World. Jonathan Koomey Analytics Press, 2007.
9. Minas L., Ellison B. The Problem of Power Consumption in Servers. Intel, 2009.
10. Microsoft Company: http://www.microsoft.com/(дата обращения 30.07.2013)
11. Oracle: http://www.oracle.com/(дата обращения 30.07.2013)
12. ASTModular: http://www.astmodular.com/(дата обращения 30.07.2013)
13. Huawei: http://www.huawei.com/(дата обращения 30.07.2013)
14. Hewlett Packard Company: http://www.hp.com/(дата обращения 30.07.2013)
15. Colt Technology Services Group: http://www.colt.net/ (дата обращения 30.07.2013)
16. Rath J. DCK Guide ToModular Data Centers: Second Edition. Data Center Knowledge Journal, 2013.
