Tweet this document:
Follow us on Twitter:
XtremIO Best Practices for Oracle using Advanced Format
Architecting a database on an All Flash Array (AFA) like EMC’s XtremIO is best done by reviewing practices to optimize I/O performance. One consideration is the use of Advanced Format and how it impacts the performance of the database Redo logs. Advanced Format refers to a new physical sector size of 4096 bytes (4KB) replacing original 512 byte standard. The larger 4KB physical sector size has these benefits:
A DBA might be very interested in using a 4KB physical sector size to gain the efficiencies and improved error correction but there are a few considerations to review. For example, some applications and databases do not recognize the newer 4KB physical sector size. At EMC we have extensively tested Oracle on XtremIO following recommendations by Oracle support in Support Note, “4K ASM” (1133713.1).” To address the problem of a database or application not recognizing the new 4KB physical sector size there is the option to use 512-byte (512e) emulation mode.
Emulation mode uses a 4KB physical sector with eight 512 byte logical sectors. A database expecting to update (read and write) a 512 byte sector can accomplish this by using the logical block address (LBA) to update the logical sector. This means the 4KB physical sector size is transparent to the database as it can write to the 512 byte logical sector and thus backwards compatibility is maintained.
Picture 1: 512-Byte Emulation Mode
Unfortunately, there is the possibility of misaligned 4KB operations: one 512 byte update causing two 4K physical sectors to be updated. Before exploring the impact of misaligned operations on the Oracle database we need to how writes are managed in emulation mode.
Picture 2: Writes in 512-Byte Emulation Mode
Shown above writes are managed by:
The key point of this read-modify-write process is the entire 4KB physical sector is modified. A request to modify one 512 bytes logical sector means reading 4KB into memory and writing the 4KB physical sector back to disk. For optimal efficiency it would be ideal to update multiple logical sectors belonging to one physical sector in one operation. When properly aligned writes to logical sectors are a one-to-one match to the physical sector and do not cause excessive I/O.
Misalignment is caused when incorrectly partitioning the LUN. To quote Thomas Krenn’s Wiki on Partition Alignment:
Picture 3: Negative Impact of Misalignment
Quick side note:
The solution is not to partition the LUN but present the unpartitioned device to ASM. There is an excellent blog by Bart Sjerps (Dirty Cache) called, “Fun with Linux UDEV and ASM: UDEV to create ASM disk volumes” that provides steps on using unpartitioned devices with ASM. In the blog Linux UDEV is reviewed as a solution to eliminate the misalignment:
Reading through the blog Bart points to the fact using UDEV and maintaining the ASM rules can be hard work and so he created a script called, “asm” and a RPM called, “asmdisks” to automate the use of UDEV with ASM. Highly recommended reading and look to the bottom of the blog for the link to download the RPM. Why not use ASMlib? Bart goes into detail on some of the challenges in using ASMlib in the same blog so I’m not going to list them here but rather encourage you to review the ASMlib section.
Here are a few examples of how to determine if you are using emulation mode as detailed in the Unix & Linux website under “How can I find the actual size of a flash disk?”
sgdisk –print <device>
Disk /dev/sdb: 15691776 sectors, 7.5 GiB
Logical sector size: 512 bytes
The output shows the number of sectors and the logical sector size.
Using the /sys directly:
For the sector size:
udisks outputs the information directly.
udisks –show-info <device> | grep size
Get physical block sector size:
blockdev –getpbsz <device>
Print sector size in bytes:
blockdev –getss <device>
Print device size in bytes:
blockdev –getsize64 <device>
Print the size in 512-byte sectors:
blockdev –getsz <device>
Beyond using unpartitioned devices for ASM to bypass the misalignment issue is there any other recommendations? Interestingly, the database online redo log files by default have a block size of 512 bytes. For optimal online redo log efficiency it would be ideal to change from 512 byte block size to 4096 byte block size. As of version 11.2 this can be changed by specifying the BLOCKSIZE clause to values of 512 (default), 1024, or 4096 bytes.
Picture 4:Online Redo Log Blocksize Recommendation
For example, in recent XtremIO testing we used:
ALTER DATABASE ADD LOGFILETHREAD 1 GROUP 4 (‘+REDODG’) SIZE 8G BLOCKSIZE 4096;
Before creating new online redo logs with the 4K blocksize there is a known issue with emulation mode. Emulation mode makes the 4K physical sector size transparent to the database so when creating online redo log files the database checks the sector size and finds 512 byte sectors. Unfortunately, discovering 512 byte sectors when attempting to write 4096 byte blocks results in an error like:
ORA‐01378: The logical block size (4096) of file +DATA is not compatible with the disk sector size (media sector size is 512 and host sector size is 512) 
The solution is using a hidden database parameter _DISK_SECTOR_SIZE_OVERRIDE to TRUE. This parameter overrides the sector size check performed with creating optimally sized redo log files. This parameter can be changed dynamically.
ALTER SYSTEM “_DISK_SECTOR_SIZE_OVERRIDE”=”TRUE”;
If you creating new online redo logs with the 4K blocksize then you might have to drop the original 512 byte redo log files groups.
ALTER DATABASE DROP LOGFILE GROUP 1;
Summary of 512e emulation mode best practices:
Below is a summary of the recommendations in this blog. Time for a disclaimer: I would also encourage you to review Oracle support note 1681266.1 “4K redo logs and SSD based storage” as a good place to start in determining if these recommendations are a good fit for your databases. Test these recommendations in a copy of production and validate the impact. Now that the disclaimer is over here are the steps.
4KB Native Mode
In native mode the physical sector size and the logical sector size are the same: 4 KB. If planning on using advanced format native mode the DBA will have to create 4 KB block size redo logs. Outside of the redo logs there are a few other considerations for 11gR2 and higher.
Picture 5: 4K Native Mode
Unfortunately, I haven’t the time to fully explore 4K native mode but promise a follow-up for my next blog. I did want to provide this summary table below because it highlights Oracle’s recommendation to use 4KB online redo logs for emulation mode and native mode. In native mode there is no option to use 512 byte redo logs so in a good way Oracle automatically directs the DBA into using the optimal 4KB blocksize for the redo logs.
Summary table: Supported and Preferred Modes
In the above summary table we see that emulation mode will support both 512 KB and 4 KB redo log block sizes but 4 KB is preferred. The overall recommendation is to use 4 KB block size for your redo logs.
Next Blog: Exploring the 4K native mode and insights into XtremIO testing using Advanced Format.
Table of References
 Advanced Format from Wikipedia, URL: http://en.wikipedia.org/wiki/Advanced_Format
 Thomas Krenn Wiki on Partition Alignment, URL: http://www.thomas-krenn.com/en/wiki/Partition_Alignment
 Bart Sjerps (Dirty Cache) called, “Fun with Linux UDEV and ASM: UDEV to create ASM disk volumes” URL: http://bartsjerps.wordpress.com/2014/07/01/linux-udev-create-asm-disk-volumes/#more-1534
 Oracle Support Note 1681266.1 entitled, “4K redo logs and SSD – based Storage”