I want to thank everyone for attending our session on improving virtualized Oracle performance. As promised I’m posting a blog to answer some of the questions we received as part of the session.
NUMA stands for Non-Uniform Memory Access (NUMA) is an architecture in which each CPU in a multiprocessor server has its own memory bank. A processor can access memory in its own memory bank faster than memory in another processor’s memory bank. Wikipedia has a more detailed description of NUMA for folks interested in a deep dive.
The challenge is to configure both Oracle and VMware to optimize access to local memory rather than having to find data in another processor’s memory bank. Darryl Smith describes the challenge from the database perspective here but I’ll paste for easy reading.
“vShpere is NUMA aware and will try to schedule processes to work with on the socket holding the memory the process needs. The problem with databases is that the memory you need is not known until after the process starts up and the database has determined the optimizer plan that your process will use. Generally this is not an issue as you will utilize the buffer cache that your process just populated. Given that vShpere is actually providing a virtual CPU to the OS, the next instruction that the process issues, it will likely move to a new processor/socket.”
Some of the practices to optimize NUMA performance are:
- Try to keep the database VM on the fewest number of sockets possible. In minimizing the number of processor sockets the number of times memory is referenced from non-local memory bank is correspondingly minimized.
- Use the VMware setting, “numa.vcpu.preferHT=True” forces the vCPUs to the fewest number of physical sockets. VMware has the steps to configure one virtual machine or configure all virtual machines in this article: Configure virtual machines to use hyper-threading with NUMA in VMware ESXi
- From the Oracle Databases on VMware Best Practices Guide
o Keep NUMA enabled in server hardware BIOS
- From the Oracle database configuration perspective the DBA might want to research support recommendations and patches that address NUMA. A good place to start is with the Oracle Support bulletin called, “Top 11 Things to do NOW to stabilize your RAC Cluster Environment (Doc ID 1344678.1).” Depending on the database version Oracle has patches that address NUMA performance. Oracle makes a good recommendation to make changes in a copy of production (test environment) before applying patches for disabling or enabling NUMA in the database. Checkout Oracle support article, “Oracle NUMA Usage Recommendation (Doc ID 759565.1)” for steps to enable or disable NUMA in the database.
EMC IT has had the following feedback in using these configuration guidelines for optimizing NUMA performance:
“We have seen about a 5% transaction latency reduction using this option. Be cautioned, if your database workload is pushing your vCPUs beyond 50% utilization, this setting is not for you, as your VM would benefit from using the full core, over the hyperthreaded core.”
Your mileage may vary but I hope this provides a technical overview with supporting references in configuring VMware and Oracle for optimal NUMA performance.
You might be interested in this white paper: EMC IT's Virtual Oracle Deployment Framework: Technical White Paper and Darryl Smith's Virtual Webinar on Virtualizing Oracle Databases.