What
are remote mirroring and site awareness? In a remote mirror configuration,
SF hosts and the storage are distributed to two or more sites. These sites are
typically connected via a redundant high-capacity network (Fiber Channel or
Dense Wavelength Division Multiplexing) that provides access to storage. The
disk groups include disks from all sites and volumes are created with a mirror
at each site.
To
verify that you have the Site Awareness feature enabled, use the following
command:
root@desap2ia:/root# vxlicrep |grep -i
site
Site License = YES
Site License
= YES
Site License
= YES
Site Awareness
= Enabled
Site License
= YES
Site License
= YES
Configuring Site Awareness
1)
Assign
the physical resources (hosts and disks) to a site.
2)
Configure
site-based allocation on a diskgroup
3)
Configure
site consistency on the diskgroup and the volumes
4)
Set
mirrored volumes read policy to site-based (if necessary)
5)
Use
site-based storage allocation when creating new volumes.
Assigning
disk to a site:
To assign a disk to a site:
#vxdisk
[-g diskgroup]
settag site=sitename [disk|e:enclosure] [disk|e:enclosure]…
#vxdisk –g
datadg settag site=siteA emc0_1 emc0_2
#vxdisk
settag apevmx09_111d site=site09
Or
# vxdisk –g
datadg settag site=siteA e:emc0
To
list the site tags on a disk:
# vxdisk [-g diskgroup]
listtag
root@desap2ia:/root# vxdisk -g sapz42_dg listtag
DEVICE
NAME
VALUE
apevmx09_07fe
site
site09
apevmx09_07ff
site site09
apevmx09_08f1
site
site09
apevmx09_11a2
site
site09
apevmx09_11a3
site
site09
apevmx09_111d
site
site09
apevmx09_111e
site site09
apevmx09_0262
site
site09
apevmx09_0800
site
site09
apevmx09_0802
site
site09
apevmx10_01ca
site
site10
apevmx10_07e3
site
site10
apevmx10_07e4
site
site10
apevmx10_10d4
site
site10
apevmx10_0701
site
site10
apevmx10_0702
site site10
apevmx10_1047
site
site10
apevmx10_1048
site
site10
apevmx10_1059
site
site10
apevmx10_1070
site
site10
apevmx10_1071
site
site10
To
remove the site tag from a disk:
#vxdisk [-g diskgroup]
rmtag site=sitename [disk|e:enclosure]
Assigning host to a site
To
assign a host to a site
# vxdctl set site=sitename
# vxdctl set site=siteA
To
observe the site assignment of a host:
# vxdctl list|grep siteid
Siteid: siteA
To
more the site name from a host:
#vxdctl [-F] unset site
-F Use the force option if imported disk groups are
registered to the site.
Configuring site-based allocation on a disk group
To add site definition to a disk group configuration:
#vxdg –g diskgroup
addsite sitename
For
Example:
#vxdg –g datadg addsite siteA
#vxdg –g datadg addsite siteB
·
A new
volume is automatically mirrored across all sites
·
The
volume’s read policy is automatically set to siteread
·
Use allsites=off with vxassist make for non-default
configuraiton
To
remove a site from the disk group configuration:
#vxdg –g diskgroup
rmsite sitename
·
No
change is made to the physical resources that are assigned to that site.
·
You
must use the force option (-f) if the site is detached.
Enabling
site consistency in a disk group Site consistency in a disk group means
that the data at each site is guaranteed to be consistent. Therefore, any disk
failure in a site-consistent disk group that causes one of the volumes to be
left without a complete plex causes the site to be detached. To enable fast
resynchronization of volumes when the site is reattached, the volumes are
created with the FastResync feature enabled and an instant snap DCO volume by
default.
When you turn on site consistency in a disk group:
·
By
Default, new volumes are created as site-consistent.
·
Existing
volumes are not impacted.
·
Disk
group configuration copy is placed across sites
To Set site consistency in a disk group:
#vxdg –g diskgroup
set siteconsistent=on
To set site consistency on an existing volume:
#vxsnap –g diskgroup
prepare volume
#vxvol –g diskgroup
set siteconsistent=on volume
To turn off site consistency:
#vxvol –g diskgroup
set siteconsistent=off volume
#vxdg –g diskgroup
set siteconsistent=off
Read
policy in a site-consistent setup
The siteread policy:
·
Is
available with SF5.0 and later
·
Instructs
each host to read data only from local plexes, provided that local plexes are
available.
·
Is
the default policy if site-based allocation is enabled.
To manually set the read policy to siteread, run the
following command:
#vxvol –g diskgroup
rdpol siteread volume
File
System Size Before Extension:
root@desap2ia:/usr/sap/Z42/sapdata999#
df -k .
Filesystem
1024-blocks Free %Used Iused
%Iused Mounted on
/dev/vx/dsk/sapz42_dg/usrsapTMP
20971520 16315608 23%
5230 1% /usr/sap/Z42/sapdata999
We need to observe the following
things:
1) The READPOL should be SITEREAD
2) Three Plex Should there
3) Check the length of Log Plex
root@desap2ia:/usr/sap/Z42/sapdata999#
vxprint -htg sapz42_dg usrsapTMP
V
NAME RVG/VSET/CO
KSTATE STATE LENGTH
READPOL PREFPLEX UTYPE
PL
NAME
VOLUME KSTATE
STATE LENGTH LAYOUT NCOL/WID
MODE
SD
NAME
PLEX
DISK DISKOFFS LENGTH [COL/]OFF
DEVICE MODE
SV
NAME
PLEX VOLNAME NVOLLAYR
LENGTH [COL/]OFF AM/NM MODE
SC
NAME
PLEX CACHE
DISKOFFS LENGTH [COL/]OFF DEVICE MODE
DC
NAME
PARENTVOL LOGVOL
SP
NAME
SNAPVOL DCO
EX
NAME
ASSOC
VC
PERMS MODE STATE
SR
NAME KSTATE
v
usrsapTMP
-
ENABLED ACTIVE 41943040 SITEREAD
- fsgen
pl
usrsapTMP-01 usrsapTMP
ENABLED ACTIVE 41943040 CONCAT
- RW
sd
sapz42_05-01 usrsapTMP-01 sapz42_05 0
41943040 0 apevmx09_08f1 ENA
pl
usrsapTMP-02 usrsapTMP
ENABLED ACTIVE 41943040 CONCAT
- RW
sd
sapz42_05m-01 usrsapTMP-02 sapz42_05m 0 41943040
0 apevmx10_1059 ENA
pl
usrsapTMP-03 usrsapTMP
ENABLED ACTIVE LOGONLY CONCAT
- RW
sd
sapz42_drl1-19 usrsapTMP-03 sapz42_drl1 26000 1040
LOG apevmx10_10d4 ENA
Before Extending volume Remove Log
Plex:
root@desap2ia:/usr/sap/Z42/sapdata999#
vxassist -g sapz42_dg remove log usrsapTMP
root@desap2ia:/usr/sap/Z42/sapdata999#
vxprint -htg sapz42_dg usrsapTMP
V
NAME RVG/VSET/CO
KSTATE STATE LENGTH
READPOL PREFPLEX UTYPE
PL
NAME
VOLUME KSTATE
STATE LENGTH LAYOUT NCOL/WID
MODE
SD
NAME
PLEX
DISK DISKOFFS LENGTH [COL/]OFF
DEVICE MODE
SV
NAME
PLEX VOLNAME NVOLLAYR
LENGTH [COL/]OFF AM/NM MODE
SC
NAME
PLEX CACHE
DISKOFFS LENGTH [COL/]OFF DEVICE MODE
DC
NAME
PARENTVOL LOGVOL
SP
NAME
SNAPVOL DCO
EX
NAME ASSOC
VC
PERMS MODE STATE
SR
NAME KSTATE
v
usrsapTMP
-
ENABLED ACTIVE 41943040 SITEREAD
- fsgen
pl
usrsapTMP-01 usrsapTMP ENABLED ACTIVE
41943040 CONCAT -
RW
sd
sapz42_05-01 usrsapTMP-01 sapz42_05 0
41943040 0 apevmx09_08f1 ENA
pl
usrsapTMP-02 usrsapTMP ENABLED ACTIVE
41943040 CONCAT -
RW
sd
sapz42_05m-01 usrsapTMP-02 sapz42_05m 0 41943040
0 apevmx10_1059 ENA
Storage Team has provided the below
Disk
apevmx09_111d
auto:none -
- online invalid
apevmx09_111e
auto:none -
- online invalid
apevmx10_1070
auto:none -
- online invalid
apevmx10_1071
auto:none -
- online
invalid
#
vxdisksetup -i apevmx09_111d
#
vxdisksetup -i apevmx09_111e
#
vxdisksetup -i apevmx10_1070
#
vxdisksetup -i apevmx10_1071
Checking how the site read is
configured
root@desap2ia:/tmp#
vxdisk -g sapz42_dg listtag
DEVICE
NAME
VALUE
apevmx09_07fe
site
site09
apevmx09_07ff
site
site09
apevmx09_08f1
site
site09
apevmx09_0800
site
site09
apevmx09_0802
site
site09
apevmx10_07e3
site
site10
apevmx10_07e4
site
site10
apevmx10_10d4
site
site10
apevmx10_0701
site
site10
apevmx10_0702
site
site10
apevmx10_1059
site
site10
Disk from Enclousre apevmx09 is
configured for site09
Disk from Enclousre apevmx10 is
configured for site10
so we need to tag the disk as per
site
cb20cpsdbpre#
vxdisk settag apevmx09_111d site=site09
cb20cpsdbpre#
vxdisk settag apevmx09_111e site=site09
cb20cpsdbpre#
vxdisk settag apevmx10_1070 site=site10
cb20cpsdbpre#
vxdisk settag apevmx10_1071 site=site10
#
vxdg -g usrsapTMP adddisk sapz42_06=apevmx09_111d
#
vxdg -g usrsapTMP adddisk sapz42_07=apevmx09_111e
#
vxdg -g usrsapTMP adddisk sapz42_06m=apevmx10_1070
#
vxdg -g usrsapTMP adddisk sapz42_07m=apevmx10_1071
Calculating the space upto which we
could extend the volume using the disk
#
vxassist -g sapz42_dg maxgrow usrsapTMP layout=mirror-concat alloc=sapz42_06
sapz42_06m
Extending the Volume
root@desap2ia:/tmp#
/etc/vx/bin/vxresize -g sapz42_dg -F vxfs usrsapTMP +40g alloc=sapz42_06
sapz42_06m
root@desap2ia:/tmp#
vxprint -htg sapz42_dg usrsapTMP
V
NAME RVG/VSET/CO
KSTATE STATE LENGTH
READPOL PREFPLEX UTYPE
PL
NAME
VOLUME KSTATE
STATE LENGTH LAYOUT NCOL/WID
MODE
SD
NAME PLEX
DISK DISKOFFS LENGTH [COL/]OFF
DEVICE MODE
SV
NAME
PLEX VOLNAME NVOLLAYR
LENGTH [COL/]OFF AM/NM MODE
SC
NAME
PLEX CACHE
DISKOFFS LENGTH [COL/]OFF DEVICE MODE
DC
NAME
PARENTVOL LOGVOL
SP
NAME
SNAPVOL DCO
EX
NAME
ASSOC
VC
PERMS MODE STATE
SR
NAME KSTATE
v
usrsapTMP
- ENABLED
ACTIVE 125829120 SITEREAD
- fsgen
pl
usrsapTMP-01 usrsapTMP ENABLED ACTIVE
125829120 CONCAT - RW
sd
sapz42_05-01 usrsapTMP-01 sapz42_05 0
41943040 0 apevmx09_08f1 ENA
sd
sapz42_06-01 usrsapTMP-01 sapz42_06 0
83886080 41943040 apevmx09_111d ENA
pl
usrsapTMP-02 usrsapTMP ENABLED ACTIVE 125829120
CONCAT - RW
sd
sapz42_05m-01 usrsapTMP-02 sapz42_05m 0 41943040
0 apevmx10_1059 ENA
sd
sapz42_06m-01 usrsapTMP-02 sapz42_06m 0 83886080
41943040 apevmx10_1070 ENA
root@desap2ia:/tmp#
Adding Drl Logs Back:
To add DRL logs to an existing
volume, use the following command:
#
vxassist [-b] [-g diskgroup] addlog volume logtype=drl \ [nlog=n]
[loglen=size]
#
vxassist -b -g sapz42_dg addlog usrsapTMP logtype=drl nlog=1 loglen=1040
alloc=sapz42_drl1
root@desap2ia:/etc/vx/bin#
vxassist -b -g sapz42_dg addlog usrsapTMP logtype=drl nlog=1 loglen=1040
alloc=sapz42_drl1
root@desap2ia:/etc/vx/bin#
vxprint -htg sapz42_dg usrsapTMP
V
NAME RVG/VSET/CO
KSTATE STATE LENGTH
READPOL PREFPLEX UTYPE
PL
NAME VOLUME
KSTATE STATE LENGTH
LAYOUT NCOL/WID MODE
SD
NAME
PLEX
DISK DISKOFFS LENGTH [COL/]OFF
DEVICE MODE
SV
NAME
PLEX VOLNAME NVOLLAYR
LENGTH [COL/]OFF AM/NM MODE
SC
NAME
PLEX CACHE
DISKOFFS LENGTH [COL/]OFF DEVICE MODE
DC
NAME
PARENTVOL LOGVOL
SP
NAME
SNAPVOL DCO
EX
NAME
ASSOC
VC
PERMS MODE STATE
SR
NAME KSTATE
v
usrsapTMP
-
ENABLED ACTIVE 125829120 SITEREAD
- fsgen
pl
usrsapTMP-01 usrsapTMP ENABLED ACTIVE
125829120 CONCAT - RW
sd
sapz42_05-01 usrsapTMP-01 sapz42_05 0
41943040 0 apevmx09_08f1 ENA
sd
sapz42_06-01 usrsapTMP-01 sapz42_06 0 83886080
41943040 apevmx09_111d ENA
pl
usrsapTMP-02 usrsapTMP ENABLED ACTIVE
125829120 CONCAT - RW
sd
sapz42_05m-01 usrsapTMP-02 sapz42_05m 0 41943040
0 apevmx10_1059 ENA
sd
sapz42_06m-01 usrsapTMP-02 sapz42_06m 0 83886080 41943040
apevmx10_1070 ENA
pl
usrsapTMP-03 usrsapTMP ENABLED ACTIVE
LOGONLY CONCAT
- RW
sd
sapz42_drl1-09 usrsapTMP-03 sapz42_drl1 9360 1040
LOG apevmx10_10d4 ENA
FS Size before Extension
root@desap2ia:/usr/sap/Z42/sapdata999#
df -k .
Filesystem
1024-blocks Free %Used Iused
%Iused Mounted on
/dev/vx/dsk/sapz42_dg/usrsapTMP
20971520 16315608 23%
5230 1% /usr/sap/Z42/sapdata999
FS Size After Exetension
root@desap2ia:/tmp#
df -k /usr/sap/Z42/sapdata999
Filesystem
1024-blocks Free %Used Iused
%Iused Mounted on
/dev/vx/dsk/sapz42_dg/usrsapTMP
62914560 55627570 12%
5231 1% /usr/sap/Z42/sapdata999
What could be the issue, if we are
not tagging the disk with respect to site:
Disk used without SITEREAD
root@desap2ia:/etc/vx/bin#
vxassist -g sapz42_dg maxsize usrsapz42_ext_vol layout=mirror-concat
alloc="sapz42_08 sapz42_08m"
VxVM
vxassist INFO V-5-1-13775 Site consistent volumes use new dco log types
(dcoversion=20). Creating it by default.
VxVM
vxassist ERROR V-5-1-325 dm:usrsapz42_ext_vol: No disks match specification
Disk with tag SITEREAD
root@desap2ia:/etc/vx/bin#
vxdg -g sapz42_dg adddisk sapz42_09m=apevmx10_1048
root@desap2ia:/etc/vx/bin#
vxassist -g sapz42_dg maxgrow usrsapz42_ext_vol layout=mirror-concat
alloc="sapz42_08 sapz42_08m"
Volume usrsapz42_ext_vol can be
extended by 251582464 to: 725389312 (354194Mb)