Activating Huge Page

 Metalink note 361468.1 describes the fundamental steps for enabling Oracle to use huge pages.

Changes have to be implemented on OS and DB side.

• Step 1: Have the memlock user limit set in /etc/security/limits.conf file. Set the value (in KB) slightly smaller than installed RAM

nr_hugepages * Hugepagesize

40960         *  2048         = 83886080

This is a soft limit which can be ideally set slightly smaller than RAM and/or slightly bigger than SGA.

• Step 2: Re-logon to the Oracle product owner account (e.g. 'oracle') and check the memlock limit (ulimit -l)
• Step 3: If you have Oracle Database 11g or later, the default database created uses the Automatic Memory Management (AMM) feature which is incompatible with HugePages. Disable AMM before proceeding. To disable, set the initialization parameters MEMORY_TARGET and MEMORY_MAX_TARGET to 0 (zero).

memory_target and memory_max_target need to be set to 0

• Step 4: Make sure that all your database instances are up (including ASM instances) as they would run on production. Use the script hugepages_settings.sh in Document 401749.1 to calculate the recommended value for the vm.nr_hugepages kernel parameter (/etc/sysctl.conf file)

Run of the script returned value 40963 (the requested 40960 was not well calculated for the size of the SGA. By using use_large_pages=ONLY this will create problem at startup as described below)

Indeed, the hugepages_settings.sh script includes # Cumulative number of pages required to handle the running shared memory segments in the calculation.

for SEG_BYTES in `ipcs -m | cut -c44-300 | awk '{print $1}' | grep "[0-9][0-9]*"`
do
    MIN_PG=`echo "$SEG_BYTES/($HPG_SZ*1024)" | bc -q`
    if [ $MIN_PG -gt 0 ]; then
        NUM_PG=`echo "$NUM_PG+$MIN_PG+1" | bc -q`
    fi
done

 

• Edit the file /etc/sysctl.conf and set the vm.nr_hugepages parameter there:

...

vm.nr_hugepages = 1496

...

• Server reboot
• Apply the following change in order to allow Oracle to use huge pages:

ALTER SYSTEM SET use_large_pages = ONLY scope=spfile;

This setting will force Oracle to check SGA size and total number of hugepages allocated by OS at startup. if whole SGA cannot be allocated in hugepages the database won't startup

 ERROR:

 Failed to allocate shared global region with large pages, unix errno = 12.

 

Aborting Instance startup.

 

ORA-27137: unable to allocate Large Pages to create a shared memory segment

ALTER SYSTEM SET use_large_pages = TRUE scope=spfile;

Starting with 11.2.0.3 Oracle will now allocate what it can of the SGA in hugepages and if it runs out, it will allocate the rest of the SGA using small pages. With this new behavior additional shared memory segments are an expected side effect. Part of the change is to ensure that each shared memory segment making up the SGA only contains sub-areas with an identical alignment requirement - hence the SGA will spread over more separate SHM segments. In this supported mixed page mode the database will exhaust the available hugepages, before switching to regular sized pages. 

• If needed, adjust the SGA size (actually Oracle checks the value of sga_max_size at startup, not sga_target) (in this case, the OS settings suggested by customer were wrong, thus I had to resize SGA in order to allocate whole SGA in hugepages; other than that, no SGA adjustment should be done, because the number of hugepages is calculated keeping already in consideration the SGA size):

ALTER SYSTEM SET sga_max_size=81604378624 scope=spfile;

ALTER SYSTEM SET sga_target=81604378624 scope=both;

If not set properly, you would get something like this:

SQL> startup

ORA-27137: unable to allocate large pages to create a shared memory segment

Linux-x86_64 Error: 12: Cannot allocate memory

Additional information: 2097152

Alert log:

Starting ORACLE instance (normal)

************************ Large Pages Information *******************

Parameter use_large_pages = ONLY

Per process system memlock (soft) limit = 80 GB

 

Large Pages unused system wide = 40960 (80 GB)

Large Pages configured system wide = 40960 (80 GB)

Large Page size = 2048 KB

 

ERROR:

  Failed to allocate shared global region with large pages, unix errno = 12.

  Aborting Instance startup.

  ORA-27137: unable to allocate Large Pages to create a shared memory segment

 

ACTION:

  Total System Global Area size is 80 GB. Prior to next instance restart:

1. Increase the number of unused large pages (page size 2048 KB)

to at least 40964 (80 GB) to allocate 100% System Global Area

with large pages.

2. Large pages are automatically locked into physical memory.

Increase the per process memlock (soft) limit to at least 80 GB to lock

100% System Global Area's large pages into physical memory

            sga_max_size needs to be set precisely and also by keeping in consideration additional pages allocated by Oracle as shared memory segments (you can check them via ipcs -m).

An optimal configuration is shown at startup as follows:

Starting ORACLE instance (normal)
************************ Large Pages Information *******************
Parameter use_large_pages = ONLY
Per process system memlock (soft) limit = 82 GB

Total Shared Global Region in Large Pages = 80 GB (100%)

Large Pages used by this instance: 40961 (80 GB)
Large Pages unused system wide = 2 (4096 KB)
Large Pages configured system wide = 40963 (80 GB)
Large Page size = 2048 KB
********************************************************************

• DB bounce