Site Awareness

What are remote mirroring and site awareness? In a remote mirror configuration, SF hosts and the storage are distributed to two or more sites. These sites are typically connected via a redundant high-capacity network (Fiber Channel or Dense Wavelength Division Multiplexing) that provides access to storage. The disk groups include disks from all sites and volumes are created with a mirror at each site.

To verify that you have the Site Awareness feature enabled, use the following command:

root@desap2ia:/root# vxlicrep |grep -i site

   Site License                        = YES

   Site License                        = YES

   Site License                        = YES

   Site Awareness                      = Enabled

   Site License                        = YES

   Site License                        = YES

Configuring Site Awareness              

1)       Assign the physical resources (hosts and disks) to a site.

2)       Configure site-based allocation on a diskgroup

3)       Configure site consistency on the diskgroup and the volumes

4)       Set mirrored volumes read policy to site-based (if necessary)

5)       Use site-based storage allocation when creating new volumes.

Assigning disk to a site:

To assign a disk to a site:

#vxdisk [-g diskgroup] settag site=sitename [disk|e:enclosure] [disk|e:enclosure]…

#vxdisk –g datadg settag site=siteA emc0_1 emc0_2

#vxdisk settag apevmx09_111d site=site09

Or

# vxdisk –g datadg settag site=siteA e:emc0

To list the site tags on a disk:

# vxdisk [-g diskgroup] listtag

root@desap2ia:/root# vxdisk -g sapz42_dg listtag

DEVICE          NAME                            VALUE                             

apevmx09_07fe   site                            site09

apevmx09_07ff   site                            site09

apevmx09_08f1   site                            site09

apevmx09_11a2   site                            site09

apevmx09_11a3   site                            site09

apevmx09_111d   site                            site09

apevmx09_111e   site                            site09

apevmx09_0262   site                            site09

apevmx09_0800   site                            site09

apevmx09_0802   site                            site09

apevmx10_01ca   site                            site10

apevmx10_07e3   site                            site10

apevmx10_07e4   site                            site10

apevmx10_10d4   site                            site10

apevmx10_0701   site                            site10

apevmx10_0702   site                            site10

apevmx10_1047   site                            site10

apevmx10_1048   site                            site10

apevmx10_1059   site                            site10

apevmx10_1070   site                            site10

apevmx10_1071   site                            site10

To remove the site tag from a disk:

#vxdisk [-g diskgroup] rmtag site=sitename [disk|e:enclosure]

Assigning host to a site

To assign a host to a site

# vxdctl set site=sitename

# vxdctl set site=siteA

To observe the site assignment of a host:

# vxdctl list|grep siteid

Siteid: siteA

To more the site name from a host:

#vxdctl [-F] unset site

-F Use the force option if imported disk groups are registered to the site.

Configuring site-based allocation on a disk group

To add site definition to a disk group configuration:

#vxdg –g diskgroup addsite sitename

For Example:

#vxdg –g datadg addsite siteA

#vxdg –g datadg addsite siteB

·          A new volume is automatically mirrored across all sites

·          The volume’s read policy is automatically set to siteread

·          Use allsites=off with vxassist make for non-default configuraiton

To remove a site from the disk group configuration:

#vxdg –g diskgroup rmsite sitename

·          No change is made to the physical resources that are assigned to that site.

·          You must use the force option (-f) if the site is detached.

Enabling site consistency in a disk group Site consistency in a disk group means that the data at each site is guaranteed to be consistent. Therefore, any disk failure in a site-consistent disk group that causes one of the volumes to be left without a complete plex causes the site to be detached. To enable fast resynchronization of volumes when the site is reattached, the volumes are created with the FastResync feature enabled and an instant snap DCO volume by default.

When you turn on site consistency in a disk group:

·          By Default, new volumes are created as site-consistent.

·          Existing volumes are not impacted.

·          Disk group configuration copy is placed across sites

To Set site consistency in a disk group:

#vxdg –g diskgroup set siteconsistent=on

To set site consistency on an existing volume:

#vxsnap –g diskgroup prepare volume

#vxvol –g diskgroup set siteconsistent=on volume

To turn off site consistency:

#vxvol –g diskgroup set siteconsistent=off volume

#vxdg –g diskgroup set siteconsistent=off

Read policy in a site-consistent setup

The siteread policy:

·          Is available with SF5.0 and later

·          Instructs each host to read data only from local plexes, provided that local plexes are available.

·          Is the default policy if site-based allocation is enabled.

To manually set the read policy to siteread, run the following command:

#vxvol –g diskgroup rdpol siteread volume

File System Size Before Extension:

root@desap2ia:/usr/sap/Z42/sapdata999# df -k .

Filesystem    1024-blocks      Free %Used    Iused %Iused Mounted on

/dev/vx/dsk/sapz42_dg/usrsapTMP    20971520  16315608   23%     5230     1% /usr/sap/Z42/sapdata999

We need to observe the following things:

1) The READPOL should be SITEREAD

2) Three Plex Should there

3) Check the length of Log Plex

root@desap2ia:/usr/sap/Z42/sapdata999# vxprint -htg sapz42_dg usrsapTMP

V  NAME         RVG/VSET/CO  KSTATE   STATE    LENGTH   READPOL   PREFPLEX UTYPE

PL NAME         VOLUME       KSTATE   STATE    LENGTH   LAYOUT    NCOL/WID MODE

SD NAME         PLEX         DISK     DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE

SV NAME         PLEX         VOLNAME  NVOLLAYR LENGTH   [COL/]OFF AM/NM    MODE

SC NAME         PLEX         CACHE    DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE

DC NAME         PARENTVOL    LOGVOL

SP NAME         SNAPVOL      DCO

EX NAME         ASSOC        VC                       PERMS    MODE     STATE

SR NAME         KSTATE

v  usrsapTMP    -            ENABLED  ACTIVE   41943040 SITEREAD  -        fsgen

pl usrsapTMP-01 usrsapTMP    ENABLED  ACTIVE   41943040 CONCAT    -        RW

sd sapz42_05-01 usrsapTMP-01 sapz42_05 0       41943040 0         apevmx09_08f1 ENA

pl usrsapTMP-02 usrsapTMP    ENABLED  ACTIVE   41943040 CONCAT    -        RW

sd sapz42_05m-01 usrsapTMP-02 sapz42_05m 0     41943040 0         apevmx10_1059 ENA

pl usrsapTMP-03 usrsapTMP    ENABLED  ACTIVE   LOGONLY  CONCAT    -        RW

sd sapz42_drl1-19 usrsapTMP-03 sapz42_drl1 26000 1040   LOG       apevmx10_10d4 ENA

Before Extending volume Remove Log Plex:

root@desap2ia:/usr/sap/Z42/sapdata999# vxassist -g sapz42_dg remove log usrsapTMP

root@desap2ia:/usr/sap/Z42/sapdata999# vxprint -htg sapz42_dg usrsapTMP         

V  NAME         RVG/VSET/CO  KSTATE   STATE    LENGTH   READPOL   PREFPLEX UTYPE

PL NAME         VOLUME       KSTATE   STATE    LENGTH   LAYOUT    NCOL/WID MODE

SD NAME         PLEX         DISK     DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE

SV NAME         PLEX         VOLNAME  NVOLLAYR LENGTH   [COL/]OFF AM/NM    MODE

SC NAME         PLEX         CACHE    DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE

DC NAME         PARENTVOL    LOGVOL

SP NAME         SNAPVOL      DCO

EX NAME         ASSOC        VC                       PERMS    MODE     STATE

SR NAME         KSTATE

v  usrsapTMP    -            ENABLED  ACTIVE   41943040 SITEREAD  -        fsgen

pl usrsapTMP-01 usrsapTMP    ENABLED  ACTIVE   41943040 CONCAT    -        RW

sd sapz42_05-01 usrsapTMP-01 sapz42_05 0       41943040 0         apevmx09_08f1 ENA

pl usrsapTMP-02 usrsapTMP    ENABLED  ACTIVE   41943040 CONCAT    -        RW

sd sapz42_05m-01 usrsapTMP-02 sapz42_05m 0     41943040 0         apevmx10_1059 ENA

          

Storage Team has provided the below Disk

apevmx09_111d auto:none -          -          online invalid

apevmx09_111e auto:none -          -          online invalid

apevmx10_1070 auto:none -          -          online invalid

apevmx10_1071 auto:none -          -          online invalid  

# vxdisksetup -i apevmx09_111d

# vxdisksetup -i apevmx09_111e

# vxdisksetup -i apevmx10_1070

# vxdisksetup -i apevmx10_1071

Checking how the site read is configured

root@desap2ia:/tmp# vxdisk -g sapz42_dg listtag

DEVICE          NAME                            VALUE                             

apevmx09_07fe   site                            site09

apevmx09_07ff   site                            site09

apevmx09_08f1   site                            site09

apevmx09_0800   site                            site09

apevmx09_0802   site                            site09

apevmx10_07e3   site                            site10

apevmx10_07e4   site                            site10

apevmx10_10d4   site                            site10

apevmx10_0701   site                            site10

apevmx10_0702   site                            site10

apevmx10_1059   site                            site10

Disk from Enclousre apevmx09 is configured for site09

Disk from Enclousre apevmx10 is configured for site10

so we need to tag the disk as per site

cb20cpsdbpre# vxdisk settag apevmx09_111d site=site09

cb20cpsdbpre# vxdisk settag apevmx09_111e site=site09

cb20cpsdbpre# vxdisk settag apevmx10_1070 site=site10

cb20cpsdbpre# vxdisk settag apevmx10_1071 site=site10

# vxdg -g usrsapTMP adddisk sapz42_06=apevmx09_111d

# vxdg -g usrsapTMP adddisk sapz42_07=apevmx09_111e

# vxdg -g usrsapTMP adddisk sapz42_06m=apevmx10_1070

# vxdg -g usrsapTMP adddisk sapz42_07m=apevmx10_1071

Calculating the space upto which we could extend the volume using the disk

# vxassist -g sapz42_dg maxgrow usrsapTMP layout=mirror-concat alloc=sapz42_06 sapz42_06m

Extending the Volume

root@desap2ia:/tmp# /etc/vx/bin/vxresize -g sapz42_dg -F vxfs usrsapTMP +40g alloc=sapz42_06 sapz42_06m

root@desap2ia:/tmp#  vxprint -htg sapz42_dg usrsapTMP

V  NAME         RVG/VSET/CO  KSTATE   STATE    LENGTH   READPOL   PREFPLEX UTYPE

PL NAME         VOLUME       KSTATE   STATE    LENGTH   LAYOUT    NCOL/WID MODE

SD NAME         PLEX         DISK     DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE

SV NAME         PLEX         VOLNAME  NVOLLAYR LENGTH   [COL/]OFF AM/NM    MODE

SC NAME         PLEX         CACHE    DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE

DC NAME         PARENTVOL    LOGVOL

SP NAME         SNAPVOL      DCO

EX NAME         ASSOC        VC                       PERMS    MODE     STATE

SR NAME         KSTATE

v  usrsapTMP    -            ENABLED  ACTIVE   125829120 SITEREAD -        fsgen

pl usrsapTMP-01 usrsapTMP    ENABLED  ACTIVE   125829120 CONCAT   -        RW

sd sapz42_05-01 usrsapTMP-01 sapz42_05 0       41943040 0         apevmx09_08f1 ENA

sd sapz42_06-01 usrsapTMP-01 sapz42_06 0       83886080 41943040  apevmx09_111d ENA

pl usrsapTMP-02 usrsapTMP    ENABLED  ACTIVE   125829120 CONCAT   -        RW

sd sapz42_05m-01 usrsapTMP-02 sapz42_05m 0     41943040 0         apevmx10_1059 ENA

sd sapz42_06m-01 usrsapTMP-02 sapz42_06m 0     83886080 41943040  apevmx10_1070 ENA

root@desap2ia:/tmp#

Adding Drl Logs Back:

To add DRL logs to an existing volume, use the following command:

# vxassist [-b] [-g diskgroup] addlog volume logtype=drl \   [nlog=n] [loglen=size]

# vxassist -b -g sapz42_dg addlog usrsapTMP logtype=drl nlog=1 loglen=1040 alloc=sapz42_drl1

root@desap2ia:/etc/vx/bin# vxassist -b -g sapz42_dg addlog usrsapTMP logtype=drl nlog=1 loglen=1040 alloc=sapz42_drl1

root@desap2ia:/etc/vx/bin# vxprint -htg sapz42_dg usrsapTMP

V  NAME         RVG/VSET/CO  KSTATE   STATE    LENGTH   READPOL   PREFPLEX UTYPE

PL NAME         VOLUME       KSTATE   STATE    LENGTH   LAYOUT    NCOL/WID MODE

SD NAME         PLEX         DISK     DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE

SV NAME         PLEX         VOLNAME  NVOLLAYR LENGTH   [COL/]OFF AM/NM    MODE

SC NAME         PLEX         CACHE    DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE

DC NAME         PARENTVOL    LOGVOL

SP NAME         SNAPVOL      DCO

EX NAME         ASSOC        VC                       PERMS    MODE     STATE

SR NAME         KSTATE

v  usrsapTMP    -            ENABLED  ACTIVE   125829120 SITEREAD -        fsgen

pl usrsapTMP-01 usrsapTMP    ENABLED  ACTIVE   125829120 CONCAT   -        RW

sd sapz42_05-01 usrsapTMP-01 sapz42_05 0       41943040 0         apevmx09_08f1 ENA

sd sapz42_06-01 usrsapTMP-01 sapz42_06 0       83886080 41943040  apevmx09_111d ENA

pl usrsapTMP-02 usrsapTMP    ENABLED  ACTIVE   125829120 CONCAT   -        RW

sd sapz42_05m-01 usrsapTMP-02 sapz42_05m 0     41943040 0         apevmx10_1059 ENA

sd sapz42_06m-01 usrsapTMP-02 sapz42_06m 0     83886080 41943040  apevmx10_1070 ENA

pl usrsapTMP-03 usrsapTMP    ENABLED  ACTIVE   LOGONLY  CONCAT    -        RW

sd sapz42_drl1-09 usrsapTMP-03 sapz42_drl1 9360 1040    LOG       apevmx10_10d4 ENA

FS Size before Extension

root@desap2ia:/usr/sap/Z42/sapdata999# df -k .

Filesystem    1024-blocks      Free %Used    Iused %Iused Mounted on

/dev/vx/dsk/sapz42_dg/usrsapTMP    20971520  16315608   23%     5230     1% /usr/sap/Z42/sapdata999

FS Size After Exetension

root@desap2ia:/tmp# df -k /usr/sap/Z42/sapdata999

Filesystem    1024-blocks      Free %Used    Iused %Iused Mounted on

/dev/vx/dsk/sapz42_dg/usrsapTMP    62914560  55627570   12%     5231     1% /usr/sap/Z42/sapdata999

What could be the issue, if we are not tagging the disk with respect to site:

Disk used without SITEREAD

root@desap2ia:/etc/vx/bin# vxassist -g sapz42_dg maxsize usrsapz42_ext_vol layout=mirror-concat alloc="sapz42_08 sapz42_08m"

VxVM vxassist INFO V-5-1-13775 Site consistent volumes use new dco log types (dcoversion=20). Creating it by default.

VxVM vxassist ERROR V-5-1-325 dm:usrsapz42_ext_vol: No disks match specification

Disk with tag SITEREAD

root@desap2ia:/etc/vx/bin# vxdg -g sapz42_dg adddisk sapz42_09m=apevmx10_1048

root@desap2ia:/etc/vx/bin# vxassist -g sapz42_dg maxgrow usrsapz42_ext_vol layout=mirror-concat alloc="sapz42_08 sapz42_08m"

Volume usrsapz42_ext_vol can be extended by 251582464 to: 725389312 (354194Mb)