Energy-Efficient Framework for Virtual Machine Consolidation in Cloud Data Centers

Kejing He＊, Zhibo Li, Dongyan Deng, Yanhua Chen

School of Computer Science and Engineering, South China University of Technology, Guangzhou 510641, China

* The corresponding author, email: kjhe@scut.edu.cn

I. INTRODUCTION

Cloud computing is an on-demand computing model that provides services to users through Internet [1]. The various services provided by cloud computing can be categorized into three kinds: SaaS, PaaS and IaaS [2]. The popularity and application of cloud computing have brought great convenience to businesses and individuals that need large-scale data storage and processing [3]. With the development of society and the growth of demand for computing power, more and more data centers have been set up with the continually expanding of their scale. According to the report of International Data Corporation (IDC), the number of data centers being built around the world will continue growing until it peaks at 8.6 million in 2017 [4]. As there are a large number of IT infrastructures in data center, the operation and maintenance of it requires an enormous amount of energy, resulting in a substantial increase of operating costs for the cloud service providers. For instance, US data centers consumed about 70 billion kilowatt-hours of electricity in 2014 [5], representing 2 percent of the country’s total energy consumption.According to the analysis of related report, the high energy consumption of data centers is mainly caused by the following two reasons.First, with the increase of computing power requirements, the scale of the data centers is constantly expanding. Second, the utilization of data center servers tends to be relatively low. According to statistics of research reports[6-7], most data centers are at a low level of resource utilization, which is approximately 30%, while the energy consumption of active cloud servers is very high thus enormous amount of energy is wasted.

Virtual machine consolidation is an effective method that can enhance resource utilization and reduce energy consumption of data center. However, since the workload in data center is dynamic, the dynamic migration of virtual machines may cause physical nodes to be overloaded, resulting in the degradation of server performance and user service level agreements violation (SLAV). Besides,the migrations of virtual machines should be controlled within a certain amount. Excessive migration operations of data center will take up too much bandwidth and affect the service of data center. Thus, we need to deal with the tradeoff between energy consumption and excellent service performance to meet SLA. In this paper, considering both energy consumption and quality of service, we propose a new energy efficiency-oriented virtual machine consolidation framework based on the original virtual machine consolidation framework in CloudSim [8]. There are two main contributions in our framework: First, in the underloaded host decision step, this paper proposes an IUD method based on the overload threshold of hosts and the average utilization of all active hosts. Second, in the migration target host selection step, this paper puts forward a MAUD policy based on the average utilization of the data center.

The rest of this paper is organized as follows. Section II introduces the related work.Section III analyzes the original virtual machine consolidation framework in CloudSim and introduces our motivation. Section IV presents the design of the new virtual machine consolidation framework based on the original framework in CloudSim. Section V introduces the methodology of the redesigned framework.In section VI, we conduct experiments on the proposed framework. Finally, some conclusions are made in section VII.

In this paper, the authors proposed a new virtual machine live migration framework for better energy efficiency based on existing research considering the power consumption and user quality of service.

II. RELATED WORK

The process of virtual machine consolidation consists of four decision-making tasks: physical node overload decision, virtual machine to migrate selection, physical node underload decision, and target node for the migrated VM selection. The task to optimize the virtual machine consolidation framework can be seen as the optimization to these four decision-making tasks, resulting in energy consumption reduction of data center and the overall energy efficiency improvement of data center.

Nathuji and Schwan [9] have applied dynamic VMC to increase the energy efficiency of data center up to 34%. Beloglazov and Buyya [10] have proposed a new virtual machine deployment strategy--Modified Best Fit Decreasing (MBFD) and results show that dynamic virtual machine deployment strategies are better than the static deployment strategies in reducing energy consumption.

Beloglazov and Buyya [11] have proposed four adaptive heuristic algorithms applying statistical analysis on the historical data of virtual machines to estimate the overload thresholds of CPU utilization. The four algorithms are as follows: the Median Absolute Deviation method (MAD), Interquartile Range method(IQR), Local Regression method (LR) and Robust Local Regression (LRR). They have also proposed three different virtual machine selection policies: Minimum Migration Time policy (MMT), Random Selection policy (RS)and the Maximum Correlation policy (MC).The simulation results show that the application of dynamic virtual machine consolidation algorithm outperforms static deployment algorithm NPA and DVFS. Besides, LR algorithm performed best among the four kinds of overloaded host detection algorithms, and the MMT policy performed best among the three kinds of virtual machine selection policies.

Cao and Dong [12] have proposed an im-proved VMC framework of cloud data center that it classifies the overload host into overload host with SLAV and without SLAV. Masoumzadeh and Hlavacs [13] have proposed a novel strategy for virtual machine selection that exploits dynamic criteria to select the virtual machine to migrate, and the result of experiments shows that the proposed method is better than previous single criterion method. Xiao et al. [14] have put forward a novel system that allocates the resources of data center according to the requirement of applications. Their method can reduce the number of servers and achieve good energy efficiency.Gondhi and Kailu [15] have proposed a faster virtual machine consolidation strategy based on prediction, which makes improvements on PABFD algorithm. Similarly, Hieu et al. [16]have also proposed a new virtual machine consolidation strategy, which improves the energy efficiency of cloud data centers by using usage prediction strategy. The experimental results show that this strategy reduces overall migration times and overall server power consumption, while it also meets service level agreements well.

In previous studies, we have found that the state division of hosts in cloud data center is too cursory: an active node is either overloaded or underloaded. We think that the classification of underloaded hosts is necessary to be more fine-grained. Meanwhile, the target host selection for VM migration in CloudSim can also be improved. Thus, we propose a new virtual machine consolidation framework to achieve better energy efficiency.

III. THE ORIGINAL FRAMEWORK

The virtual machine consolidation framework in CloudSim mainly consists of the following four steps.

(1) Host overloading detection. In this step,the overload detection algorithm is exploited to judge whether a host is overloaded or not and finally get a set of overloaded hosts.

(2) Virtual machine selection. In this step,virtual machine selection policies (such as MMT, RS, MC, MU) are used to choose one or more VMs from the set of overloaded hosts conducted by step 1 to migrate out. Repeat this process until the host is not overloaded,and then all chosen VMs should be added to the list of vmsToMigrate.

(3) Virtual machine placement. In this step, the virtual machines in the list of vmsToMigrate are sorted according to the CPU utilization. Then for each virtual machine, the PABFD policy is used to choose one target host from the set of underloaded hosts and idle hosts that can accept virtual machine. If appropriate target host exists, these virtual machines will be migrated to the target host. And thus some idle hosts may be turned on.

(4) Host underloading detection. In this step, overall underloaded hosts are denoted as OUH; the underloaded hosts that have accepted VMs are denoted as EUH and the rest of the underloaded hosts are denoted as RUH.For the underloaded hosts in the set of RUH,the host with the lowest utilization will be checked iteratively. If all the virtual machines on it can be migrated out, the host will be shut down to save energy.

However, there are some problems in the original framework of CloudSim:

(1) The state division of hosts in cloud data center is too cursory-whereas an active node can have more states rather than only overloaded or underloaded.

(2) In original framework, both the criterion and goal are the power consumption. However, the way that only considers the power consumption may lead to the load imbalance of data centers, leading to frequent virtual machine migration and the degradation of quality of services, finally impact the overall energy efficiency of data centers.

IV. THE DESIGN OF NEW FRAMEWORK

For the problems of the virtual machine consolidation framework in CloudSim, we improve it and redesign a new framework. The flow chart of the new framework is shown in figure 1.

The new virtual machine consolidation framework mainly consists of the following fi ve steps.

(1) The overload decision algorithm (ODA)is used to judge whether a host is overloaded or not and fi nally get a set of overloaded hosts.

(2) For the underloaded hosts, all of them are subdivided into three states: UH, UM,UL based on their CPU utilization. When an underloaded host is at UH state, it has a high workload and is easy to become overloaded after accepting migrated VM. An underloaded host at UL state means that it has a low workload. Lastly, UM state is the intermediate state between UH and UL. The Venn diagram of the structure of divisions of the hosts is shown in figure 2.

(3) For the overloaded hosts, one or more VMs on it should be selected to migrate out.The policies for virtual machine selection are used to choose VM to migrate out from an overloaded host. Repeat this process until the host is not overloaded and finally obtain the list of vmToMiagrate.

(4) For the VMs in the list of vmsToMigrate, we sorted them according to the CPU utilization in decreasing order. Then for each VM, the MAUD policy we proposed in this paper is used to choose one target host that can accept VM. Firstly, we find the suitable target host in the set of UM hosts. When there are none of hosts in UM state that can receive the VM, we look for target host in other underloaded state hosts and idle hosts.

(5) In this step, overall underloaded hosts are denoted as OUH. The underloaded hosts that have accepted VM are denoted as EUH,and the rest of the underloaded hosts are denoted as RUH. For the underloaded hosts in the set of RUH, we firstly choose the host with the lowest utilization in the set of UL hosts to turn off. If all the VMs on it can be migrated out, the host will be shut down to save energy.

V. METHODOLOGY OF NEW FRAMEWORK

5.1 The improved underload decision algorithm

In this paper, we present an improved underload decision algorithm based on the average utilization and the current threshold of ODA,which is abbreviated to IUD. Through the criteria of the average utilization and the current threshold of ODA, we divide the underloaded hosts into more fine-grained states: UH, UM and UL.

Fig. 2 The Venn diagram of the new framework

The present threshold of ODA is denoted as Ut, and the average utilization of the active hosts in data center is denoted as Ua. We can obtain Utand Uafrom Eq. (1) and Eq. (2). And the parameter Hirepresents the host i, n represents the number of active hosts in the data center.

We obtain the above three states of the underloaded hosts according to the criteria of Uaand Ut, which can be gotten from Eq. (3),where the parameter s is the safe parameter(10≥≥s)

Through the IUD algorithm, we will improve the framework in the CloudSim. In the step of host underloading detection, we will firstly try to find a suitable host in the set of UL hosts to turn off. However, if there is no appropriate one in the set of UL hosts, we will find a suitable host in other sets. What is more,in the step of virtual machine placement, we will firstly try to find an appropriate target host in the set of UM hosts. If the UM hosts set is empty or none of the hosts in the set is suitable to accept the virtual machine, we will find a suitable host in other sets.

Algorithm 1: MAUD Policy

5.2 The minimum average utilization diあerence policy

For the VMs that should be migrated out, we should find new target host to place them. The virtual machine consolidation framework in CloudSim has implemented a VM placement policy which was named PABFD. The PABFD policy takes the power consumption as the only criterion, which can reduce the energy consumption of the data center effectively but may lead to frequent virtual machine migration and the degradation of quality of services,and thus impact the overall energy efficiency of data centers.

Considering the load balancing problem, we propose a new VM placement policy MAUD.The MAUD policy adopts the difference between the utilization of the host after accepting the VM and the average utilization of data center as a criterion. Considering the workload situation of data center, it will reduce the number of migrations, power consumption and also the SLAV. The pseudo-code of MAUD policy is shown in Algorithm 1. Moreover, the word diffUsage means the difference between utilization of the host after accepting the VM and Ua, which can be evaluated through Eq.(4).

VI. IMPLEMENTATION AND EVALUATION

In this section, we will implement and evaluate the new virtual machine consolidation framework proposed in this paper. In our implementation, we apply Cloud Computing simulation platform-CloudSim [8], which is face to the IaaS layer of the Cloud Computing,to conduct the experiment, and the version of CloudSim is 3.0.

6.1 Experimental setup

To compare with the original framework in the CloudSim, the experimental setup in this paper imitates the experimental setup in Ref. [11].In CloudSim, we build a virtual data center with 800 heterogeneous physical nodes. Half of the physical nodes are HP ProLiant ML110 G4 servers, and the remaining are HP ProLiant ML110 G5. All physical nodes have a dual-core CPU and 4GB RAM. The frequency of each core of HP ProLiant ML110 G4 servers is 1860 MIPS while that of the HP ProLiant ML110 G5 servers is 2660 MIPS. What is more, each node is modeled as occupying 1GB/second network bandwidth. For the types of virtual machines, we use the four types of VMs shown in Ref. [11]. Initially, we allocate all of the VMs according to the resource requirements of their types. And then we will allocate VMs according to the workload data.

6.2 Workload data and the evaluated frameworks

The workload data is from CoMon project[17], which is a monitoring system to Planet-Lab. All of these utilization data are collected from more than 1000 VMs at more than 500 different places around the world. The characteristic of ten days workload can be found in Ref. [11]. To evaluate the proposed new framework with IUD algorithm and MAUD policy, we denote the original framework in CloudSim as Original, the original framework with whose underload decision algorithm replaced by IUD algorithm as IUD, the original framework with whose VM placement policy replaced by MAUD policy as MAUD, the new framework with IUD algorithm and MAUD policy combined as IUMA. The definitions of the four frameworks are shown in Table I.

For the overload decision algorithm and VM selection algorithm, we adopt the four overload decision algorithms (MAD, IQR,LR, LRR) and four VM selection algorithms(MMT, MC, MU, RS) that are integrated into original CloudSim. What is more, for each execution of ODA, we choose the best safety pa-rameter of it as the to be applied safety parameter. The best safety parameter of the MAD method, IQR method, LR method and LRR method are 2.5, 1.5, 1.2 and 1.2 respectively.Totally, there are 16 different combinations of the overload decision algorithm associated with VM selection algorithm.

Table I The definition of four frameworks

6.3 Metric

To evaluate the framework, we adopt the following metrics related to the virtual machine consolidation in this experiment.

(1) Migrations: it means the number of the VM migrations of the whole hosts in data center. This metric can judge whether an algorithm is rational or not for the reason that an algorithm with an experimental result of high Migration metric may have problems in its design.

(2) Energy: it represents the total energy consumption of all the hosts in data center. It is an important metric to evaluate the framework for the reason that the goal of virtual machine consolidation is to reduce the energy consumption.

(3) SLAV: it means the Service Level Agreement violation caused by all the hosts in data center during the simulation. From Eq. (5),we know that SLAV is mainly determined by two parameters: SLATAH and PDM. The full name of parameter SLATAH is SLAV Time per Active Host, which represents the average time for all active hosts at a state where CPU requirement is over 100%. The SLATAH value can be calculated through Eq. (6), where N stands for the number of physical nodes, TSirepresents the time of host i at the state that CPU requirement is over 100%, and Taiis the total active time of host i. What is more, the full name of PDM is Performance Degrada-tion due to Migrations, which represents the performance degradation caused due to the migrations. The PDM value can be calculated through Eq. (7), where M is the number of virtual machines, Cdjrepresents the requested CPU resource of VMjduring migration, Crjrepresents the total request CPU resource of VMj.

Fig. 3 The mean and standard deviation of Migrations for four frameworks

Fig. 4 The mean and standard deviation of Energy for four frameworks

(4) ESV: it means the overall energy efficiency of the framework, which can be obtained through the product of Energy and SLAV, as shown in Eq. (8). It is also an important metric to evaluate the framework.

6.4 Result and analysis

In this section, we will show and analyze the experimental results. From figure 3 to figure 6, we can see the comparison of mean and deviation of four frameworks on four metrics:Migrations, Energy, SLAV and ESV. For all of the four metrics, the less of the average of each of them, the better result we will achieve.We can easily draw the conclusion that the three new frameworks proposed in this paper perform better than the original framework in CloudSim.

According to the experimental results among the 16 different combinations of ODA and VM selection algorithms, the best one is LR_MMT_1.2. Thus, we present the results of LR_MMT_1.2 for four frameworks in table 2. In general, comparing with the Original framework in term of virtual machine migrations, IUD framework reduces 16.2%, MAUD framework reduces 22.5%, and IUMA framework reduces 20.6%. In term of the mean energy consumption, IUD framework reduces 0.61%, MAUD framework reduces 0.83%,and IUMA framework reduces 1.15%. In term of SLAV, IUD framework reduces 44.6%,MAUD framework reduces 55.7%, and IUMA framework reduces 58.7%. In term of ESV,IUD framework reduces 45.6%, MAUD framework reduces 56.5%, and IUMA frame-work reduces 59.8%.

VII. CONCLUSION

In this paper, we have proposed a new virtual machine live migration framework for better energy efficiency based on existing research considering the power consumption and user quality of service. There are two main contributions in this new framework: (1) In the step of underloaded host detection, by subdividing underloaded hosts into more fine-grained states according to overload upper threshold of certain host and the average utilization of all active hosts, this paper proposes a new method named IUD Algorithm; (2) In the step of migration target host selection, based on the average utilization of the data center, this paper puts forward a new algorithm named MAUD policy.

Finally, based on real workload data on PlanetLab, experiments have been done with the cloud simulation platform CloudSim. The experimental result shows that the proposed algorithm reduces both the energy consumption and SLA violation rate of data centers compared with the existing algorithms, and achieves good result for improving energy efficiency of data centers.

Compared with the original consolidation framework in CloudSim, the new framework proposed in this paper reduces the migrations of virtual machines by an average of 20.6%.What is more, energy consumption is reduced by an average of 1.15%, SLAV is reduced by an average of 58.7%, and the overall energy efficiency parameter ESV is reduced by an average 59.8%.

ACKNOWLEDGEMENTS

This work was supported by the National Natural Science Foundation of China (NSFC)(No. 61272200, 10805019), the Program for Excellent Young Teachers in Higher Education of Guangdong, China (No. Yq2013012),the Fundamental Research Funds for the Central Universities (2015ZJ010), the Special Support Program of Guangdong Province(201528004), and the Pearl River Science &Technology Star Project (201610010046). Part of the work has been published in the Proceedings of 2016 4th International Conference on Cloud Computing and Intelligence Systems(CCIS). Thank CCIS for recommending to the China Communications.

Fig. 5 The mean and standard deviation of SLAV for four frameworks

Fig. 6 The mean and standard deviation of ESV for four frameworks

Table II The result of four frameworks corresponding to LR_MMT_1.2

[1] Q. Hassan, “Demystifying cloud computing”,The Journal of Defense Software Engineering,vol. 1, 2011, pp. 16-21.

[2] M. B. Mollah, K. R. Islam, S. S. Islam “Next generation of computing through cloud computing technology”, Proc. 25th IEEE Canadian Conference Electrical & Computer Engineering (CCECE),2012, pp. 1-6.

[3] M. Haghighat, S. Zonouz, M. Abdel-Mottaleb, “CloudID: Trustworthy cloud-based and cross-enterprise biometric identification”, Expert Systems with Applications, vol. 42, no. 21, 2015,pp. 7905-7916.

[4] “IDC: Amount of World’s Data Centers to Start Declining in 2017”, Available: http://www.datacenterknowledge.com/archives/2014/11/11/idc-amount-of-worlds-data-centers-to-startdeclining-in-2017/.

[5] “Here ’s How Much Energy All US Data Centers Consume”, Available: http://www.datacenterknowledge.com/archives/2016/06/27/hereshow-much-energy-all-us-data-centers-consume/.

[6] L. A. Barroso, U. Holzle, “The case for energy-proportional computing”, Computer, vol. 40,no. 12, 2007, pp. 33-37.

[7] X. Fan, W. D. Weber, L. A. Barroso, “Power provisioning for a warehouse-sized computer”, Proc ACM SIGARCH Computer Architecture News, vol.35, no. 2, 2007, pp. 13-23.

[8] R. N. Calheiros, R. Ranjan, A. Beloglazov, C.A. De Rose, R. Buyya, “CloudSim: a toolkit for modeling and simulation of cloud computing environments and evaluation of resource provisioning algorithms”, Software: Practice and experience, vol. 41, no. 1, 2011, pp. 23-50.

[9] R. Nathuji, K. Schwan, “Virtualpower: coordinated power management in virtualized enterprise systems”, Proc. ACM SIGOPS Operating Systems Review, vol. 41, no. 6, 2007, pp. 265-278.

[10] A. Beloglazov, R. Buyya, “Energy efficient allocation of virtual machines in cloud data centers”,Proc. 10th IEEE/ACM International Conference on Cluster, Cloud and Grid Computing (CCGrid),2010, pp. 577-578.

[11] A. Beloglazov, R. Buyya, “Optimal online deterministic algorithms and adaptive heuristics for energy and performance efficient dynamic consolidation of virtual machines in cloud data centers”, Concurrency and Computation:Practice and Experience, vol. 24, no. 13, 2012, pp. 1397-1420.

[12] Z. Cao, S. Dong, “An energy-aware heuristic framework for virtual machine consolidation in Cloud computing”, The Journal of Supercomputing, vol. 69, no. 1, 2014, pp. 429-451.

[13] S. S. Masoumzadeh, H. Hlavacs, “Integrating VM selection criteria in distributed dynamic VM consolidation using Fuzzy Q-Learning”, Proc.IEEE 9th International Conference on Network and Service Management (CNSM), 2013, pp.332-338.

[14] Z. Xiao, W. Song, Q. Chen, “Dynamic resource allocation using virtual machines for cloud computing environment”, IEEE transactions on parallel and distributed systems, vol. 24, no. 6,2013, pp. 1107-1117.

[15] N. K. Gondhi, P. Kailu, “Prediction based energy efficient virtual machine consolidation in cloud computing”, Proc. IEEE Second International Conference on Advances in Computing and Communication Engineering (ICACCE), 2015, pp.437-441.

[16] N. T. Hieu, M. Di Francesco, A. Ylä-Jääski, “Virtual machine consolidation with usage prediction for energy-efficient cloud data centers”, Proc.IEEE 8th International Conference on Cloud Computing (CLOUD), 2015, pp. 750-757.

[17] K. Park, V. S. Pai, “CoMon: a mostly-scalable monitoring system for PlanetLab”, ACM SIGOPS Operating Systems Review, vol. 40, no. 1, pp. 65-74.