How to maximize PCIe flash performance

What is Oracle ASM?
The Oracle® automatic storage management system (ASM) was developed 10 years ago to make it much easier for database administrators (DBAs) to use and tune database storage. Oracle ASM enables DBAs to:

Automatically stripe data over each RAW device to improve database storage performance
Mirror data for greater fault tolerance
Simplify the management and extension of database storage for the cloud and, with the ASM Cluster File System (ACFS), use the snapshot and replication functionality to increase availability
Add the Oracle Real Application Clusters (RAC) capability to help reduce total cost of ownership (TCO), expand scalability and increase availability, among other benefits
Easily move data from one device to another while the database is active with no performance degradation
Reduce or eliminate storage or Linux administrator time for configuring database storage
Use ASM as a Linux®/Unix operating system file system called ACFS. (I know what you are thinking. Since you need Oracle Grid up and running to mount and use ASM, how can an ACFS device be available to the operating system at system boot? The reason is that the kernel has been modified to allow this functionality. Learn more about ACFS here.)
What’s more, it’s free – comes with Oracle Grid

The drawbacks of using Oracle ASM:

DBAs now control the storage they are using. Therefore, they need to know more about the storage and how the logical unit numbers (LUNs) are being used by Oracle ASM, and how to create ASM disk groups for higher performance.
Most ASM commands are executed through SQLPlus, not through the command line. That means storage is accessed through SQLPlus and sometimes ASMCMD, isolating the storage and making it harder for Linux admins to identify storage issues.
Recover Manager (RMAN) is the only guaranteed/supported method of backing up databases on ASM.

What will be covered in this blog and what won’t
ASM is quite complex to learn and to set up properly for both performance and high availability. I won’t be going over all the commands and configurations of ASM, but I will cover how to set up an aligned Seagate Nytro WarpDrive PCIe® card and create an ASM disk to be assigned to an ASM disk group. There are many websites and books that go over all the details of Oracle ASM, and the most current book that I would recommend is “Database Cloud Storage: The Essential Guide to Oracle Automatic Storage Management.” Or visit Oracle’s docs.oracle.com website.

Setting up ASM
The following steps cover configuring an LUN for ASM. In order to use ASM, you will need to install the Oracle Grid software from otn.oracle.com. I prefer using Oracle ASMLIB when configuring ASM. Included in the box of the latest version of Oracle Linux, ASMLIB offers an easier way to configure ASM. If you are using an older version of ASM, you will need to install the RPMs for ASM from support.oracle.com.

Step 1: Create aligned partition
Refer to Part 1 of this series to create a LUN on a 1M boundary. Oracle recommends using the full disk for ASM, so just create one large aligned partition. I suggest using this command:

echo “2048,,” | sfdisk –uS /dev/sdX –force

Step 2: Create an ASM disk
Once the device has an aligned partition created on it, we can assign it to ASM by using the ASM create disk command with two input parameters – ASM disk name and the PCIe flash partitioned device name – as follows:

/etc/init.d/oracleasm createdisk ASMDISK1 /dev/sda1

To verify that the create ASM disk process was successful, and the device was marked as an ASM disk, enter the following commands:

/etc/init.d/oracleasm querydisk /dev/sda1

(the output should state: “/dev/sda is an Oracle ASM disk [OK])

/etc/init.d/oracleasm listdisks

(the output should state: ASMDISK1)

Step 3: Assign ASM disk to disk group
The ASM disk group is the primary component of ASM as well as the highest level data structure in ASM. A disk group is a container of multiple ASM disks, and it is the disk group that the database references when creating Oracle Tablespaces.

There are multiple ways to create an ASM disk group. The easiest way is to use ASM Configuration Assistant (ASMCA), which walks you through the creation process. See Oracle ASM documentation on how to use ASMCA.

Here are the steps for creating a disk group:

a: Log in to GRID using sqlplus / as sysasm.

b: Select name, path, header status, state from v$asm_disk as follows:

c: Create diskgroup DG1 external redundancy disk using this command:

create diskgroup DG1 external redundancy disk
‘/dev/oracleasm/disks/D1’;

The disk group is now ready to be used in creating an Oracle database Tablespace. To use this disk group in an Oracle database, please refer to Oracle’s database documentation at docs.oracle.com.

This post is part of a four-part series:
Part 1 describes how to align the PCIe devices.
Part 2 describes how to create RAW devices or file systems.
Part 3 describes how to use Oracle ASM when deploying PCIe flash cards.
Part 4 describes how to persist assignment to dynamically changing Nytro PCIe devices.