There seems to be a trend with the Oracle database community: The use of current commodity hardware enables a large System Global Area (SGA) and a correspondingly large database buffer cache. The SGA is a group of shared memory structures that contain data and control information for an Oracle database. The database buffer cache is a component of the SGA that holds copies of frequently used data blocks. For more information use Oracle’s documentation library. In this study by Principled Technologies each virtualized RAC node was allocated 148 GB of virtual memory. Why did we include so much memory for a virtualized Oracle RAC node?
- This acknowledges a trend in servers having more main memory (RAM), as the additional cost is relatively minimal. This additional memory is therefore available for Database Administrators (DBA) to allocate to the SGA.
- VMware’s vMotion is a foundational technology that enables the business to non-disruptively move a Virtual Machine (VM) from one server to another. As the study indicates, this can include vMotion of a virtualized production Oracle RAC node.
- The goal of the study was to demonstrate the simultaneous vMotion of multiple large virtual memory VMs which are running an OLTP workload. We proved that these vMotion operations could be done efficiently, quickly and without data loss or other instability. Since Oracle RAC is specifically designed to fail a node and fence it from the cluster in the event of the slightest hint of instability, this is an amazing accomplishment.
DBAs manage the enterprise’s most mission critical applications. Thus, DBAs want clear and convincing evidence of the benefits virtualization offers before adding another layer to the Oracle stack. In this recent study by IOUG entitled, “From Database Clouds to Big Data: 2013 IOUG Survey on Database Manageability” the 4th greatest challenge of database administrators surveyed was, “Managing a larger number of databases with the same resources.” The value of the Principled Technologies research study is that it demonstrates that Oracle DBAs can virtualize their Oracle database servers including production RAC nodes, and then easily move those VMs without any loss of availability. Oracle RAC is a cluster of many servers which access one database, a many-to-one architecture. This provides resiliency: If one server fails the surviving servers continue to provide database availability. Another interesting finding from this study is, “We have already consolidated our database and infrastructure onto a single technology platform for our critical business applications”. The study adds more color by noting, “A developing trend may be taking hold, as many IT organizations move towards database consolidation onto a shared and/or cloud environment.” Virtualization is the platform which enables both consolidation and manageability for Oracle DBAs. In terms of manageability, virtualization offers the DBA:
- The ability to proactively move database servers in preparation for planned outages: If a server, network switch, or other hardware within the infrastructure requires maintenance the virtualized database server / RAC node can be moved off of that hardware.
- Consolidation: Multiple databases can efficiently and securely share the same hardware as they are isolated to individual VMs.
- Improved Service Level Agreements (SLA): Improvements in proactive and automated reactive manageability will improve the SLAs for the business by improving database uptime.
- Strong Storage Integration: VMware integrates with EMC storage and other vendors for increased manageability, performance and availability.
There is another part of manageability picture that is equally important. That is quality of service (QoS). For example, if a given vMotion event takes too long and during that time database performance is significantly degraded then the QoS (Quality of Service) is low. The Principled Technologies study punches this up by saying, “Quick and successful virtual machine migrations are a key part of running a virtualized infrastructure.” So, if vMotion adds value in terms of manageability, then QoS must provide the following functionality:
- vMotion windows should be kept to a minimum
- The vMotion end-user performance impact should be minimal
The Principled Technologies study was designed to validate both ease of manageability and QoS. In three different tests, multiple virtualized Oracle RAC nodes under a significant OLTP workload are simultaneously moved with vMotion. Diving into the details of the architecture:
Software technology stacks includes
- Oracle 11g Release 2 (188.8.131.52)
- Oracle Enterprise Linux 6.4 x86_64 Red Hat compatible kernel
- VMware vSphere 5.1
- VMware vCenter Server 5.1
- Cisco UCS 5108 Blade Server Chassis 2 x Cisco UCS 6248UP Fabric Interconnects
- 3 x Cisco UCS B200 M3 Blade server
- Each with two Xeon E5-2680 processors
- Each with 384 GB RAM
- 3 x Cisco UCS B200 M3 Blade server
- 2 x Cisco Nexus 5548UP switches
- EMC VMAX Cloud Edition
- 2,994 GB in the Diamond level
- 18,004 GB in the Gold level
- 15,308 GB in the Silver level
Most of the above software and hardware above is self-explanatory but more detail is needed for the EMC VMAX Cloud Edition. This storage array is unique, as it enables self-service provisioning of storage based upon performance levels. The concept is simple and powerful: Create a storage catalog of pre-engineered storage levels designed to provide different levels of performance. For example, the fastest storage level is Diamond offering a greater amount of Flash drives to ensure low latency and high IOPS. Below Diamond are the Platinum, Gold, Silver and Bronze storage performance levels. Let’s look at how the Oracle RAC database was easily configured on the EMC Cloud Edition:
The picture above shows how simple it was to provision the Oracle 11gR2 database used in this study. The datafiles and tempfiles were placed on the Diamond level, as this level offered the best latency and IOPS. In the study, the goal was to place a significant workload on the virtualized Oracle RAC nodes and configure the storage to support that workload. One notable metric: The average core utilization prior to vMotion of the RAC nodes was approximately 33% with no significant I/O waits. This is important: Core utilization was used to define the workload parameters, not storage performance. As noted in the study, “In the configuration we tested, Transactions Per Second (TPS) from the hardware and software stack reached a maximum of 3,654 TPS.” In terms of storage performance, there was plenty of TPS to spare and by no means was the VMAX CE maxed out.
Everything else, i.e. the VM’s guest operating system, Oracle home, redo and undo, were placed on the Gold catalog level as these files didn’t require the same level of performance. Does this database layout seem relatively simple and straight forward? The strength of the VMAX CE is it enables a simplified database layout, as performance has been pre-engineered for the business. But there is another key consideration that some DBAs might not consider when working with storage administrators: Shared disk performance. It is entirely possible for the storage administrator to have multiple applications share the same disk meaning shared latency and IOPS. This has the potential for one application to impact another application using the same disk, and this is sometimes difficult to diagnose. The EMC VMAX CE isolates logical resources to secure the appropriate performance for each tenant. It is this isolation that prevents applications from impacting each other on the same storage array. For a DBA managing multiple mission critical Oracle database servers, this means he or she can reserve resources for CPU, RAM and now storage. This in turn provides an improved SLA to the business.
A very recent video by AAR Corp. has been posted at EMC in which Vice President of IT Operations Jim Gross discusses, “Why AAR is using the VMAX 10K for improved availability and decreased maintenance windows for the company.” The core of the VMAX CE storage array is the VMAX 10K, so the AAR video is a good example of how performance, availability and maintenance can be improved using EMC VMAX. For Oracle DBAs interested in an EMC VMAX / virtualized Oracle customer case study, Jim Gross reviews their use of VMware vSphere and Oracle on EMC VMAX.
Getting back to the Principled Technologies study, we tested the ability to use vMotion to move multiple VMs configured with Oracle Enterprise Linux (OEL), 156 GB of virtual RAM, and 16 vCPUs. For the Oracle DBA reading this article, The SGA was sized at 145 GB with 3,686 MB for the Process Global Area (PGA). The PGA is a memory area which stores user and control data for the exclusive use of a given Oracle server process. There is thus a PGA memory area for each server process. The sum of all the PGA memory is referred to as the total instance PGA. Several best practices techniques were used (e.g., huge pages) in the study to optimize database performance. In appendix A called, “System Configuration Information” the study provides great detail on most of the steps used to configure the tests, including the implementation of huge pages. The first of the three tests involved a single vMotion of one virtualized RAC node on server A to Server B.
Figure 1: Diagram of our single vMotion test, where we migrated a single VM from host A to host B
As you can see from the Figure 1 above, the final state was server A with no virtual machines and Server B with two virtual machines. The results for this test included:
Core Utilization for VMware vSphere host:
Looking at the above table we can see the vMotion of the RAC node took 130 seconds and added approximately 14% for Host A and 10% more CPU utilization for Host B. Quick definition: Bandwidth is the maximum throughput of a logical or physical communication path. During the 130 seconds of vMotion, Host A transmitted 10,618 Megabits per second (Mbps) over the network to Host B. This test showed that a single very large virtualized RAC node supporting significant workload completed in just over 2 minutes with no data loss, and CPU utilization peaking at around 14%. There was an additional load on the server during the vMotion window, but this settled back down 90 seconds after the vMotion had completed.
This test is a good example of using a rolling maintenance window to work on servers. The scenario involves moving a virtualized RAC node to a standby server so that maintenance can be performed on the production server. Once completed, the same process would be followed for the other two RAC nodes. Using the rolling maintenance window approach would mean approximately 260 seconds of vMotion, from the standby server and back to production server. Across all three servers, this totals to 780 seconds or 13 minutes of vMotion time without loss of availability of the RAC nodes to the business. Very impressive!
The next test in the study vMotioned two of the three virtual machines as shown in the Figure below.
Figure 2: Diagram of our double vMotion test, where we swapped two VMs from host B to host C
As shown in the above picture, the VM on server B was moved to server C and the VM on server C was moved to server B. Then final state was server B had the VM originally from server C and server C had the VM from server B. The results from the second test include:
Core Utilization for VMware vSphere host:
The above table shows that the simultaneous vMotion of the two VMs took 155 seconds and added approximately additional CPU utilization of 22% for Host B and 21% for Host C. Let’s use a table to look at the network throughput in Mbps.
There seems to be a trend between test one and test two in that the throughput is peaking at approximately 10,500 Mbps. More bandwidth will increase throughput and hopefully reduce the amount of time for these vMotion tests. This test showed that moving two very large virtualized RAC nodes supporting significant OLTP workloads could be completed in just over 2.5 minutes with no data loss or unavailability. CPU utilization peaked at 22% additional load and settled back down 90 seconds after the vMotion had completed.
Moving two virtualized Oracle RAC nodes would reduce a rolling window maintenance process from three steps down to two. In a two-step rolling window upgrade the following process is proposed:
- Move two VMs simultaneous to the standby servers and back to product servers which would total approximately 310 seconds
- Move the final VM to the standby server and back for 260 seconds
Using this two-step rolling window approach reduces the total vMotion time to 570 seconds or 9.5 minutes. Compare this to the three step rolling window approach of moving each virtual machine which was estimated to take 13 minutes. Moving multiple virtual machines could thus offer substantial time savings to the business.
The final test involves moving all three virtual machines as shown below.
Figure 3: Diagram of our triple vMotion test, where we migrated the VMs from one host to another
As Figure 3 shows, The VM on server C moves to server A and the VM on server A moves to server B and the VM on server B moves to server C. We have nicknamed this the “merry-go-round” scenario. The results of this test are below:
Core Utilization for VMware vSphere host:
The above table shows that simultaneous vMotion of the three VMs took 180 seconds, and added a maximum of 21% more CPU utilization across all the servers. Let’s use a table to look at the network throughput in Mbps.
The trend continues as the final triple vMotion test did not break the 10,500 Mbps ceiling. Moving all the VMs at once could apply to proactively avoiding a disaster or doing maintenance without deferring to the rolling window approach. In this case moving all three virtualized RAC nodes to three standby servers and back to production would require approximately 360 seconds of vMotion time or just 6 minutes. The advantage to the business is very little time is spent using vMotion and there is no loss of availability or data.
Managing a larger number of databases with the same resources continues to be a challenge for Oracle database administrators and for other DBAs too. Selection of a virtualization and storage platform that is open, supports any application, unifies manageability, availability and performance for the business. The EMC Cloud Edition adds a new dimension to virtualization study as it enables the DBA to provision storage and isolates logical resources meaning one database will not impact another on the storage array. Using the VMAX Cloud Edition (CE) means acceleration of database provisioning as this task is transformed into a self-service and DBA managed function. Also performance protection as the CE has pre-engineered catalog levels simplifying database layout and isolating latency and IOPS. Overall the VMAX Cloud Editions offers a faster-time-to-market and better SLAs to the business.
The cornerstone of this study was proving how VMware virtualization can offer greater agility to DBAs managing Oracle RAC databases by adding vMotion capabilities that improve application availability. Using a table we can see the agility virtualization adds to Oracle RAC databases.
As the number of simultaneous vMotion increases the number of minutes to move the RAC nodes in a rolling window scenario decreases without additional core utilization beyond 22%. This research study is critically important as it demonstrated that three significantly loaded and very large virtualized Oracle RAC nodes can be vMotioned to other servers in 180 seconds. In an emergency situation with little time to evaluate options a DBA can move all three RAC nodes in 3 minutes with no loss of availability to the business. When minutes translate into thousand or perhaps tens of thousands of dollars the ability proactively to vMotion Oracle RAC nodes justifies the investment in moving mission critical applications to a virtualized infrastructure. From the perspective of the DBA managing the mission critical database this means no actionable work on your part as the vMotion of the database is transparently done in the background. Certainly, monitoring the move and watching the database is warranted but other than taking those precautions not much is left to do.
After the original production servers have been upgraded or replaced then another scheduled vMotion can be completed taking 3 minutes for a total of 6 minutes of vMotion time. Acknowledging the trend of larger databases and having more of them to manage this research study by Principled Technologies shows VMware, Cisco, and EMC can effectively address the challenges DBAs and business have in maintaining availability, improving performance and most importantly enabling the teams responsible for the mission critical application.