Corrections for troubleshooting I/O fencing procedures

Corrections for troubleshooting I/O fencing procedures

How vxfen driver checks for pre-existing split-brain condition

Replace this topic in the Veritas Cluster Server User's Guide for 5.0 MP3 with the following content.

The vxfen driver functions to prevent an ejected node from rejoining the cluster after the failure of the private network links and before the private network links are repaired.

For example, suppose the cluster of system 1 and system 2 is functioning normally when the private network links are broken. Also suppose system 1 is the ejected system. When system 1 restarts before the private network links are restored, its membership configuration does not show system 2; however, when it attempts to register with the coordinator disks, it discovers system 2 is registered with them. Given this conflicting information about system 2, system 1 does not join the cluster and returns an error from vxfenconfig that resembles:

vxfenconfig: ERROR: There exists the potential for a preexisting  split-brain. The coordinator disks list no nodes which are in  the current membership. However, they also list nodes which are  not in the current membership. I/O Fencing Disabled!

Also, the following information is displayed on the console:

<date> <system name> vxfen: WARNING: Potentially a preexisting <date> <system name> split-brain. <date> <system name> Dropping out of cluster. <date> <system name> Refer to user documentation for steps <date> <system name> required to clear preexisting split-brain. <date> <system name> <date> <system name> I/O Fencing DISABLED! <date> <system name> <date> <system name> gab: GAB:20032: Port b closed

However, the same error can occur when the private network links are working and both systems go down, system 1 restarts, and system 2 fails to come back up. From the view of the cluster from system 1, system 2 may still have the registrations on the coordinator disks.

<date> 07:29:25 VCS CRITICAL V-16-1-10029 VxFEN driver not configured. VCS Stopping. Manually restart VCS after configuring fencing <date> 07:30:12 VCS NOTICE V-16-1-11022 VCS engine (had) started <date> 07:30:12 VCS NOTICE V-16-1-11050 VCS engine version=4.1 <date> 07:30:12 VCS NOTICE V-16-1-11051 VCS engine join version=4.1000 <date> 07:30:12 VCS NOTICE V-16-1-11052 VCS engine pstamp=4.1 04/27/07-03:15:00 <date> 07:30:12 VCS NOTICE V-16-1-10114 Opening GAB library <date> 07:30:12 VCS NOTICE V-16-1-10619 'HAD' starting on: system2 <date> 07:30:12 VCS WARNING V-16-1-11030 HAD not ready to receive this command. Message was: 0xc31 <date> 07:30:12 VCS INFO V-16-1-10125 GAB timeout set to 15000 ms <date> 07:30:17 VCS INFO V-16-1-10077 Received new cluster membership <date> 07:30:17 VCS NOTICE V-16-1-10080 System (system2) - Membership: 0x1, Jeopardy: 0x0 <date> 07:30:17 VCS NOTICE V-16-1-10086 System system2 (Node '0') is in Regular Membership - Membership: 0x1 <date> 07:30:17 VCS NOTICE V-16-1-10322 System system2 (Node '0') changed state from CURRENT_DISCOVER_WAIT to LOCAL_BUILD <date> 07:30:18 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying... <date> 07:30:20 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying... <date> 07:30:20 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying... <date> 07:30:20 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying... <date> 07:30:20 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying... <date> 07:30:20 VCS CRITICAL V-16-1-10037 VxFEN driver not configured. Retrying... <date> 07:30:20 VCS CRITICAL V-16-1-10029 VxFEN driver not configured. VCS Stopping. Manually restart VCS after configuring fencing

To resolve actual and apparent potential split brain conditions

Depending on the split-brain condition that you encountered, do the following:

Actual potential split brain condition - system 2 is up and system 1 is ejected

Determine if system2 is up or not. If system 1 is up and running, shut it down and repair the private network links to remove the split brain condition.

Restart system 1.

Apparent potential split brain condition - system 2 is down and system 1 is ejected physically verify that system 2 is down.

Verify the systems currently registered with the coordinator disks. Use the following command:

# vxfenadm -g all -f /etc/vxfentab

The output of this command identifies the keys registered with the coordinator disks.

Clear the keys on the coordinator disks as well as the data disks using the vxfenclearpre command.

# /opt/VRTSvcs/rac/bin/vxfenclearpre.

To check whether current host is an not using Fencing Information.

system2:root:/var/VRTSvcs/log # vxfenadm -d I/O Fencing Cluster Information: ================================ VCS FEN vxfenadm ERROR V-11-2-1115 Local node is not a member of cluster!Checking whether there is an split brain condition existing or not. system2:root:/var/VRTSvcs/log # /sbin/vxfenconfig -c VCS FEN vxfenconfig NOTICE Driver will use SCSI-3 compliant disks. VCS FEN vxfenconfig ERROR V-11-2-1016 There exists the potential for a preexisting split-brain         The coordinator disks list no nodes which,         are in the current membership.  However, they,         also list nodes which are not in the,         current membership.         I/O Fencing Disabled!

Verify the systems currently registered with the coordinator disks. Use the following command:

system2:root:/var/VRTSvcs/log # vxfenadm -g all -f /etc/vxfentab Device Name: /dev/sde Total Number Of Keys: 2 key[0]:          Key Value [Numeric Format]:  66,45,45,45,45,45,45,45         Key Value [Character Format]: B------- key[1]:          Key Value [Numeric Format]:  66,45,45,45,45,45,45,45         Key Value [Character Format]: B-------Device Name: /dev/sdep Total Number Of Keys: 2 key[0]:          Key Value [Numeric Format]:  66,45,45,45,45,45,45,45         Key Value [Character Format]: B------- key[1]:          Key Value [Numeric Format]:  66,45,45,45,45,45,45,45         Key Value [Character Format]: B------- Device Name: /dev/sdbw Total Number Of Keys: 2 key[0]:          Key Value [Numeric Format]:  66,45,45,45,45,45,45,45         Key Value [Character Format]: B------- key[1]:          Key Value [Numeric Format]:  66,45,45,45,45,45,45,45         Key Value [Character Format]: B------- Device Name: /dev/sdhh Total Number Of Keys: 2 key[0]:          Key Value [Numeric Format]:  66,45,45,45,45,45,45,45         Key Value [Character Format]: B------- key[1]:          Key Value [Numeric Format]:  66,45,45,45,45,45,45,45         Key Value [Character Format]: B------- Device Name: /dev/sdd Total Number Of Keys: 2 key[0]:          Key Value [Numeric Format]:  66,45,45,45,45,45,45,45         Key Value [Character Format]: B------- key[1]:          Key Value [Numeric Format]:  66,45,45,45,45,45,45,45         Key Value [Character Format]: B------- Device Name: /dev/sdbv Total Number Of Keys: 2 key[0]:          Key Value [Numeric Format]:  66,45,45,45,45,45,45,45         Key Value [Character Format]: B------- key[1]:          Key Value [Numeric Format]:  66,45,45,45,45,45,45,45         Key Value [Character Format]: B-------Device Name: /dev/sdhg Total Number Of Keys: 2 key[0]:          Key Value [Numeric Format]:  66,45,45,45,45,45,45,45         Key Value [Character Format]: B------- key[1]:          Key Value [Numeric Format]:  66,45,45,45,45,45,45,45         Key Value [Character Format]: B------- Device Name: /dev/sdeo Total Number Of Keys: 2 key[0]:          Key Value [Numeric Format]:  66,45,45,45,45,45,45,45         Key Value [Character Format]: B------- key[1]:          Key Value [Numeric Format]:  66,45,45,45,45,45,45,45         Key Value [Character Format]: B------- Device Name: /dev/sdc Total Number Of Keys: 2 key[0]:          Key Value [Numeric Format]:  66,45,45,45,45,45,45,45         Key Value [Character Format]: B------- key[1]:          Key Value [Numeric Format]:  66,45,45,45,45,45,45,45         Key Value [Character Format]: B------- Device Name: /dev/sdhf Total Number Of Keys: 2 key[0]:          Key Value [Numeric Format]:  66,45,45,45,45,45,45,45         Key Value [Character Format]: B------- key[1]:          Key Value [Numeric Format]:  66,45,45,45,45,45,45,45         Key Value [Character Format]: B------- Device Name: /dev/sden Total Number Of Keys: 2 key[0]:          Key Value [Numeric Format]:  66,45,45,45,45,45,45,45         Key Value [Character Format]: B------- key[1]:          Key Value [Numeric Format]:  66,45,45,45,45,45,45,45         Key Value [Character Format]: B------- Device Name: /dev/sdbu Total Number Of Keys: 2 key[0]:          Key Value [Numeric Format]:  66,45,45,45,45,45,45,45         Key Value [Character Format]: B------- key[1]:          Key Value [Numeric Format]:  66,45,45,45,45,45,45,45         Key Value [Character Format]: B------- Device Name: /dev/sdb Total Number Of Keys: 2 key[0]:          Key Value [Numeric Format]:  66,45,45,45,45,45,45,45         Key Value [Character Format]: B------- key[1]:          Key Value [Numeric Format]:  66,45,45,45,45,45,45,45         Key Value [Character Format]: B------- Device Name: /dev/sdem Total Number Of Keys: 2 key[0]:          Key Value [Numeric Format]:  66,45,45,45,45,45,45,45         Key Value [Character Format]: B------- key[1]:          Key Value [Numeric Format]:  66,45,45,45,45,45,45,45         Key Value [Character Format]: B------- system2:root:/var/VRTSvcs/log #

Clear the vxfencing keys

system2:root:/ # /opt/VRTSvcs/vxfen/bin/vxfenclearpre This script recovers from a preexisting split brain condition. This script removes the SCSI-3 registrations and reservations on the coordinator disks as well as the data disks contained in shared disk groups.          ******** WARNING!!!!!!!! ******** THIS SCRIPT CAN ONLY BE USED IF THERE ARE NO OTHER ACTIVE NODES IN THE CLUSTER!  VERIFY ALL OTHER NODES ARE POWERED OFF OR INCAPABLE OF ACCESSING SHARED STORAGE. If this is not the case, data corruption will result.Do you still want to continue: [y/n] (default : n) yCleaning up the coordinator disks...Cleaning up the data disks for all shared disk groups... Successfully removed SCSI-3 persistent registration and reservations from the coordinator disks as well as the shared data disks. Reboot the server to proceed with normal cluster startup...

system2:root:/ # /sbin/vxfenconfig -c             VCS FEN vxfenconfig NOTICE Driver will use SCSI-3 compliant disks.

system2:root:/ # gabconfig -a GAB Port Memberships =============================================================== Port a gen   906801 membership 0                                Port b gen   906816 membership 0

Do normal hastart on node which is system2.

Thank you for Reading,

For Reading other article, visit to “https://sites.google.com/site/unixwikis/https://sites.google.com/site/unixwikis/”

touch: Changing the Time Stamps

The touch command changes these times and is used in the following manner:

# touch options expressions filename(s)

When touch is used without options or an expression, both times are set to the current time. The file is created if it doesn't exist.., but not overwritten if it does:

#touch note #Creates file if it doesn't exist

Various Options of Touch Command:

-a (touch -a) Update only the access time to current time

-c (touch -c) Don't create if the file doesn't exist

-d (touch -d) Set an arbitrary time for both times using free format human readable data string

-m (touch -m) Update only the modification time to current time

-r (touch -r) Update both file's times from a reference file

-t (touch -t) Set an arbitrary time for both time using stamp

If you want to check the file timestamp, you can do so with this command:

#stat file

[root@localhost indrajit]# stat abcd

  File: `abcd'

  Size: 0         Blocks: 0          IO Block: 4096   regular empty file

Device: 802h/2050d Inode: 281784      Links: 1

Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)

Access: 2013-04-15 22:40:00.000000000 -0700

Modify: 2013-04-15 22:40:00.000000000 -0700

Change: 2013-12-12 20:08:43.506978374 -0800

Now let's use the command on file abcd, but only after seen its initial time stamps:

[root@localhost indrajit]# ls -l abcd Modification Time

-rw-r--r--. 1 root root 0 Apr 15  2013 abcd

[root@localhost indrajit]# ls -lu abcd Access Time

-rw-r--r--. 1 root root 0 Apr 15  2013 abcd

With the -m (modification) option, you can alter the modificaiton time alone:

#touch abcd

#ls -l abcd

-rw-r--r--   1 root       sys              0 Dec 13 05:33 abcd

#touch -m 02281030 abcd;ls -l abcd

-rw-r--r--   1 root       sys              0 Feb 28  2013 abcd

The -a (access) option changes the access time:

#ls -lu abcd

-rw-r--r--   1 root       sys              0 Dec 13 05:33 abcd

#touch -a 01261650 abcd;ls -lu abcd

-rw-r--r--   1 root       sys              0 Jan 26  2013 abcd

-c option

If you use this option, touch won’t do anything at all if the file specified doesn't exist. Look:

#ls -l xyz

xyz not found

#touch -c xyz;ls -l xyz

xyz not found

-r option

This option might come in handy if you want to copy a timestamp from a file to another file. Like so:

#ls -l abcd abcd1

-rw-r--r--   1 root       sys              0 Feb 28  2013 abcd

-rw-r--r--   1 root       sys              0 Dec 13 05:41 abcd1

#touch -r abcd1 abcd;ls -l abcd

-rw-r--r--   1 root       sys              0 Dec 13 05:41 abcd

-d and –t options

Both (-d) and (-t) options do the same thing, which is setting the same arbitrary timestamps for access and modification times. The difference is that (-d) uses free format human readable date, this means that you can use “Sun, 29 Feb 2004 16:21:42? or “2004-02-29 16:21:42” or even “next Thursday”. This option is complex to fully describe it here. On the other hand (-t) uses a simple stamp that you are confined to use. The stamp is [[CC]YY]MMDDhhmm[.ss]. [CC] is for century and you may ignore it and ignore the seconds as well. If you ignore [CC] the command will substitute it depending on what you enter as year. If you specify the year with only two digits, then CC is 20 for years in the range (0~68) and 19 for years in (69~99).

#ls -l abcd

-rw-r--r--   1 root       sys              0 Dec 13 05:41 abcd

#touch -t 3404152240 abcd;ls -l abcd

-rw-r--r--   1 root       sys              0 Apr 15  2034 abcd

#touch -t 8804152240 abcd;ls -l abcd

-rw-r--r--   1 root       sys              0 Apr 15  1988 abcd

In the first command the file timestamps will be set to: 15th April 2034 10:40 PM. While the second command will set it to: 15th April 1988 which is in a different century. If no year is specified it will be set to the current year. Example:

#touch -t 04152240 abcd;ls -l abcd

-rw-r--r--   1 root       sys              0 Apr 15  2013 abcd

vxfs fsadm: attempt to resize......failed with errno 28

When extending the filesystem online using fsadm command its failing with below error:

# fsadm -F vxfs -b 10240M /fs_test 
vxfs fsadm: /dev/vg00/rfstest is currently 5636096 sectors - size will be increased
vxfs fsadm: attempt to resize /dev/vg00/rfstest failed with errno 28 
vxfs fsadm:  Retry the operation after freeing up some space

Error number 28 means ENOSPC from /usr/include/sys/errno.h
#define ENOSPC          28      /* No space left on device      */


Why is the ENOSPC
error while running fsadm even though there is enough free space on the filesystem?
Solution
NOTE: If a file system is full, busy or too fragmented, the resize operation may fails.

Follow the below steps to extend the filesystem online using fsadm :
Checked and found filesystem is heavily fragmented. As it can be seen from the above output that there is no free extent of 8KB or above.

# fsadm -F vxfs -D -E /fs_test 

  Directory Fragmentation Report
             Dirs        Total      Immed    Immeds   Dirs to   Blocks to
             Searched    Blocks     Dirs     to Add   Reduce    Reduce
  total          2887      1996      2131         0         3          99

  Extent Fragmentation Report
        Total    Average      Average     Total
        Files    File Blks    # Extents   Free Blks
        47977          85           5     1513016
    blocks used for indirects: 26820
    % Free blocks in extents smaller than 64 blks: 100.00
    % Free blocks in extents smaller than  8 blks: 100.00
    % blks allocated to extents 64 blks or larger: 90.52
    Free Extents By Size
           1:     215670            2:     221005            4:     213834
           8:          0           16:          0           32:          0
          64:          0          128:          0          256:          0
         512:          0         1024:          0         2048:          0
        4096:          0         8192:          0        16384:          0
       32768:          0        65536:          0       131072:          0
      262144:          0       524288:          0      1048576:          0
     2097152:          0      4194304:          0      8388608:          0
    16777216:          0     33554432:          0     67108864:          0
   134217728:          0    268435456:          0    536870912:          0
  1073741824:          0   2147483648
Ran the defragmentation on the filesystem.

# fsadm -F vxfs -d -e /fs_test 
NOTE: Defragmentation can take long time depending on the degree of fragmentation, disk speed and number of inodes in the filesystem. In this case it took about 45 min. More information is available on man 1M fsadm_vxfs command.
Fragmentation report after defragmentation of filesystem:

# fsadm -F vxfs -D -E /fs_test 

  Directory Fragmentation Report
             Dirs        Total      Immed    Immeds   Dirs to   Blocks to
             Searched    Blocks     Dirs     to Add   Reduce    Reduce
  total          2887      1879      2131         0         2           5

  Extent Fragmentation Report
        Total    Average      Average     Total
        Files    File Blks    # Extents   Free Blks
        47977          84           1     1539889
    blocks used for indirects: 48
    % Free blocks in extents smaller than 64 blks: 4.94
    % Free blocks in extents smaller than  8 blks: 0.52
    % blks allocated to extents 64 blks or larger: 91.40
    Free Extents By Size
           1:       1029            2:       1144            4:       1165
           8:       1319           16:       1253           32:       1170
          64:       1023          128:        879          256:        611
         512:        388         1024:        171         2048:         89
        4096:         36         8192:         12        16384:          4
       32768:          4        65536:          0       131072:          1
      262144:          0       524288:          0      1048576:          0
     2097152:          0      4194304:          0      8388608:          0
    16777216:          0     33554432:          0     67108864:          0
   134217728:          0    268435456:          0    536870912:          0
  1073741824:          0   2147483648:          0
Now its looks much better. Free extents are larger blocks.
Tried extending the filesystem now using fsadm command and its successful now:

# fsadm -F vxfs -b 10485760 /fs_test
UX:vxfs fsadm: INFO: V-3-25942: /dev/vg00/rfstest size increased from 5636096 sectors to 10485760 sectors

# bdf /fs_test 
Filesystem              kbytes    used   avail %used Mounted on
/dev/vg00/fstest      10485760 4097395 5990074   41% /fs_test 

How To Install and Configure EMC PowerPath License on HPUnix

Question: How do I install EMC Powerpath License?

Answer: Before you start EMC powerpath License installation, please check whether EMCPower Path software is been installed.

1. Checking EMC Powerpath Software installed or not

testhost:/root#swlist -l product EMCpower
# Initializing...
# Contacting target "testhost"...
#
# Target:  testhost:/
#
  EMCpower      HP.5.1.SP2_b113 PowerPath IA64

2. Register EMC Powerpath

Before you can use the EMC powerpath software, you should register it using the EMC Powerpath License key received when you purchased the software from EMC.

Use emcpreg tool to install EMC Powerpath license key as shown below.


# emcpreg -install
===========   EMC PowerPath Registration ===========
Do you have a new registration key or keys to enter?[n] y
Enter the registration keys(s) for your product(s),
one per line, pressing Enter after each key.
After typing all keys, press Enter again.Key (Enter if done): **emc-powerpath-license-key**
1 key(s) successfully added.
Key successfully installed.Key (Enter if done):
Key  is invalid, ignored.
Try again or press Enter if done.
1 key(s) successfully registered.


3. Verify EMC Powerpath Registration

Use EMC powermt command to check the registration as shown below.


# powermt check_registration
Key **emc-powerpath-license-key**
  Product: PowerPath
  Capabilities: All

4. Verify Multiple Paths

Once you’ve installed EMC powerpath, execute powermt display dev=all as shown below to verify whether multiple paths as displayed accordingly.

# powermt display dev=all
Pseudo name=emcpowera
CLARiiON ID=00000000000 [db-server]
Logical device ID=12121212121212 [LUN 1]
state=alive; policy=CLAROpt; priority=0; queued-IOs=0
Owner: default=SP A, current=SP A
==============================================================================
---------------- Host ---------------   - Stor -   -- I/O Path -  -- Stats ---
### HW Path                 I/O Paths    Interf.   Mode    State  Q-IOs Errors
==============================================================================
  1 QLogic Fibre Channel 2300 sdb        SP A4     active  alive      0      0
  1 QLogic Fibre Channel 2300 sdd        SP B4     active  alive      0      0
  2 QLogic Fibre Channel 2300 sdf        SP A5     active  alive      0      0
  2 QLogic Fibre Channel 2300 sdh        SP B5     active  alive      0      0