Data protector : Too many disks have failed

Today got a ticket saying backup team is getting below error from Data protector while doing ZDB/BCV/Online backup.

[Major] From: SSEA@unixmemoirs01 ""  Time: 5/10/2012 21:11:39 AM

      Volume group /dev/vgdata3 has logical volume(s) with errors.

      Too many disks have failed.

The issue will be most probably from storage side. Here issue was logical volume was extended, but the new disk details not added in the BCV scripts.

Detect tape drive after replacement HP-UX 11.31

There are two ways we can do this.

1. Complex way :)

root @ uxmemoirs01[/root]
#  ioscan -fnC tape
Class     I  H/W Path  Driver S/W State   H/W Type     Description
==================================================================
tape      0  3/0/12/1/0/4/0.36.17.255.0.0.0  stape   CLAIMED     DEVICE       HP      Ultrium 3-SCSI
                      /dev/rmt/0m             /dev/rmt/0mn            /dev/rmt/c23t0d0BEST    /dev/rmt/c23t0d0BESTn
                      /dev/rmt/0mb            /dev/rmt/0mnb           /dev/rmt/c23t0d0BESTb   /dev/rmt/c23t0d0BESTnb
tape      1  3/0/12/1/0/4/0.36.17.255.0.0.1  stape   CLAIMED     DEVICE       HP      Ultrium 3-SCSI
                      /dev/rmt/1m             /dev/rmt/1mn            /dev/rmt/c23t0d1BEST    /dev/rmt/c23t0d1BESTn
                      /dev/rmt/1mb            /dev/rmt/1mnb           /dev/rmt/c23t0d1BESTb   /dev/rmt/c23t0d1BESTnb
tape      5  3/0/14/1/0/4/0.35.17.255.0.0.1  stape   NO_HW       DEVICE       HP      Ultrium 3-SCSI
                      /dev/rmt/5m             /dev/rmt/5mn            /dev/rmt/c56t0d1BEST    /dev/rmt/c56t0d1BESTn
                      /dev/rmt/5mb            /dev/rmt/5mnb           /dev/rmt/c56t0d1BESTb   /dev/rmt/c56t0d1BESTnb
tape      6  3/0/14/1/0/4/0.35.17.255.0.0.2  stape   CLAIMED     DEVICE       HP      Ultrium 3-SCSI
                      /dev/rmt/6m             /dev/rmt/6mn            /dev/rmt/c56t0d2BEST    /dev/rmt/c56t0d2BESTn
                      /dev/rmt/6mb            /dev/rmt/6mnb           /dev/rmt/c56t0d2BESTb   /dev/rmt/c56t0d2BESTnb


Error in dmesg
class : lunpath, instance 332
Evpd inquiry page 83h/80h failed or the current page 83h/80h data do not match the previous known page 83h/80h data on LUN id 0x0 probed beneath the target path (class = tgtpath, instance = 28) The lun path is (class = lunpath, instance 332).Run 'scsimgr replace_wwid' command to validate the change

class : lunpath, instance 332
An attempt to probe existing LUN id 0x1000000000000 failed with errno of 14.

Get hardware path using Instance number

root @ uxmemiors01[/root]
# scsimgr -p get_attr -C lunpath -I 332 -a hw_path
3/0/14/1/0/4/0.0x100000e002235993.0x1000000000000

Get Lun H/W path , Lunpath H/W path, Legacy H/W path using above

root @ uxmemiors01[/root]
# ioscan -m hwpath -H 3/0/14/1/0/4/0.0x100000e002235993.0x1000000000000
Lun H/W Path      Lunpath H/W Path                 Legacy H/W Path
====================================================================
64000/0xfa00/0xa0
                  3/0/14/1/0/4/0.0x100000e002235993.0x1000000000000   3/0/14/1/0/4/0.35.17.255.0.0.1

Replace WWID using Lun H/W path

root @uxmemiors01[/root]
# scsimgr -f replace_wwid -H 64000/0xfa00/0xa0
scsimgr: Successfully validated binding of LUN paths with new LUN.

root @ uxmemiors01[/root]
# ioscan -fnC tape
Class     I  H/W Path  Driver S/W State   H/W Type     Description
==================================================================
tape      0  3/0/12/1/0/4/0.36.17.255.0.0.0  stape   CLAIMED     DEVICE       HP      Ultrium 3-SCSI
                      /dev/rmt/0m             /dev/rmt/0mn            /dev/rmt/c23t0d0BEST    /dev/rmt/c23t0d0BESTn
                      /dev/rmt/0mb            /dev/rmt/0mnb           /dev/rmt/c23t0d0BESTb   /dev/rmt/c23t0d0BESTnb
tape      1  3/0/12/1/0/4/0.36.17.255.0.0.1  stape   CLAIMED     DEVICE       HP      Ultrium 3-SCSI
                      /dev/rmt/1m             /dev/rmt/1mn            /dev/rmt/c23t0d1BEST    /dev/rmt/c23t0d1BESTn
                      /dev/rmt/1mb            /dev/rmt/1mnb           /dev/rmt/c23t0d1BESTb   /dev/rmt/c23t0d1BESTnb
tape      5  3/0/14/1/0/4/0.35.17.255.0.0.1  stape   CLAIMED     DEVICE       HP      Ultrium 3-SCSI
                      /dev/rmt/5m             /dev/rmt/5mn            /dev/rmt/c56t0d1BEST    /dev/rmt/c56t0d1BESTn
                      /dev/rmt/5mb            /dev/rmt/5mnb           /dev/rmt/c56t0d1BESTb   /dev/rmt/c56t0d1BESTnb
tape      6  3/0/14/1/0/4/0.35.17.255.0.0.2  stape   CLAIMED     DEVICE       HP      Ultrium 3-SCSI
                      /dev/rmt/6m             /dev/rmt/6mn            /dev/rmt/c56t0d2BEST    /dev/rmt/c56t0d2BESTn
                      /dev/rmt/6mb            /dev/rmt/6mnb           /dev/rmt/c56t0d2BESTb   /dev/rmt/c56t0d2BESTnb

root @ uxmemiors01[/root]



2. Simple way to accomplish the above

 Error in dmesg output

class : lunpath, instance 8
Evpd inquiry page 83h/80h failed or the current page 83h/80h data do not match the previous known page 83h/80h data on LUN id 0x0 probed beneath the target path (class = tgtpath, instance = 7) The lun path is (class = lunpath, instance 8).Run 'scsimgr replace_wwid' command to validate the change


# ioscan -fnC tape
Class     I  H/W Path  Driver S/W State   H/W Type     Description
==================================================================
tape      4  0/0/14/1/0/4/1.36.17.255.0.0.0  stape   CLAIMED     DEVICE       HP      Ultrium 3-SCSI
                      /dev/rmt/4m              /dev/rmt/4mn             /dev/rmt/c108t0d0BEST    /dev/rmt/c108t0d0BESTn
                      /dev/rmt/4mb             /dev/rmt/4mnb            /dev/rmt/c108t0d0BESTb   /dev/rmt/c108t0d0BESTnb
tape      5  0/0/14/1/0/4/1.36.17.255.0.0.1  stape   CLAIMED     DEVICE       HP      Ultrium 3-SCSI
                      /dev/rmt/5m              /dev/rmt/5mn             /dev/rmt/c108t0d1BEST    /dev/rmt/c108t0d1BESTn
                      /dev/rmt/5mb             /dev/rmt/5mnb            /dev/rmt/c108t0d1BESTb   /dev/rmt/c108t0d1BESTnb
tape      0  1/0/4/1/0/4/1.35.17.255.0.0.1   stape   NO_HW       DEVICE       HP      Ultrium 3-SCSI
                      /dev/rmt/0m             /dev/rmt/0mn            /dev/rmt/c19t0d1BEST    /dev/rmt/c19t0d1BESTn
                      /dev/rmt/0mb            /dev/rmt/0mnb           /dev/rmt/c19t0d1BESTb   /dev/rmt/c19t0d1BESTnb
tape      1  1/0/4/1/0/4/1.35.17.255.0.0.2   stape   CLAIMED     DEVICE       HP      Ultrium 3-SCSI
                      /dev/rmt/1m             /dev/rmt/1mn            /dev/rmt/c19t0d2BEST    /dev/rmt/c19t0d2BESTn
                      /dev/rmt/1mb            /dev/rmt/1mnb           /dev/rmt/c19t0d2BESTb   /dev/rmt/c19t0d2BESTnb


Replace wwid

# scsimgr -f replace_wwid -C lunpath -I 8
Binding of LUN path 1/0/4/1/0/4/1.0x100000e002235993.0x1000000000000 with new LUN validated successfully
# ioscan -fnC tape
Class     I  H/W Path  Driver S/W State   H/W Type     Description
==================================================================
tape      4  0/0/14/1/0/4/1.36.17.255.0.0.0  stape   CLAIMED     DEVICE       HP      Ultrium 3-SCSI
                      /dev/rmt/4m              /dev/rmt/4mn             /dev/rmt/c108t0d0BEST    /dev/rmt/c108t0d0BESTn
                      /dev/rmt/4mb             /dev/rmt/4mnb            /dev/rmt/c108t0d0BESTb   /dev/rmt/c108t0d0BESTnb
tape      5  0/0/14/1/0/4/1.36.17.255.0.0.1  stape   CLAIMED     DEVICE       HP      Ultrium 3-SCSI
                      /dev/rmt/5m              /dev/rmt/5mn             /dev/rmt/c108t0d1BEST    /dev/rmt/c108t0d1BESTn
                      /dev/rmt/5mb             /dev/rmt/5mnb            /dev/rmt/c108t0d1BESTb   /dev/rmt/c108t0d1BESTnb
tape      0  1/0/4/1/0/4/1.35.17.255.0.0.1   stape   CLAIMED     DEVICE       HP      Ultrium 3-SCSI
                      /dev/rmt/0m             /dev/rmt/0mn            /dev/rmt/c19t0d1BEST    /dev/rmt/c19t0d1BESTn
                      /dev/rmt/0mb            /dev/rmt/0mnb           /dev/rmt/c19t0d1BESTb   /dev/rmt/c19t0d1BESTnb
tape      1  1/0/4/1/0/4/1.35.17.255.0.0.2   stape   CLAIMED     DEVICE       HP      Ultrium 3-SCSI
                      /dev/rmt/1m             /dev/rmt/1mn            /dev/rmt/c19t0d2BEST    /dev/rmt/c19t0d2BESTn
                      /dev/rmt/1mb            /dev/rmt/1mnb           /dev/rmt/c19t0d2BESTb   /dev/rmt/c19t0d2BESTnb
#

Using kctune to change kernel paramaters

Changes that needs reboot

Normal output

# /usr/sbin/kctune ninode
Tunable  Value  Expression
ninode    8192  Default

Detailed output
# /usr/sbin/kctune -v ninode
Tunable             ninode
Description         Maximum number of HFS file system open inodes that can be in memory.
Module              ufs
Current Value       8192 [Default]
Value at Next Boot  Default (automatic)
Value at Last Boot  8192
Default Value       8192 (automatic)
Constraints         ninode >= 14
Can Change          At Next Boot Only

Changing the parameter

# /usr/sbin/kctune ninode=34816
NOTE:    The requested changes could not be applied to the currently
         running system, for the following reasons:
         - The tunable 'ninode' cannot be changed without a reboot.
       * The requested changes have been saved, and will take effect at
         next boot.
Tunable               Value  Expression
ninode   (now)         8192  Default
         (next boot)  34816  34816
# /usr/sbin/kctune ninode
Tunable               Value  Expression
ninode   (now)         8192  Default
         (next boot)  34816  34816


Changes that affects immediately without reboot

# /usr/sbin/kctune maxfiles_lim
Tunable       Value  Expression  Changes
maxfiles_lim   4096  Default     Immed

# /usr/sbin/kctune -v maxfiles_lim
Tunable             maxfiles_lim
Description         Hard maximum number of file descriptors per process
Module              fs
Current Value       4096 [Default]
Value at Next Boot  4096 [Default]
Value at Last Boot  4096
Default Value       4096
Constraints         maxfiles_lim >= 32
                    maxfiles_lim <= 1048576
                    maxfiles_lim >= maxfiles
Can Change          Immediately or at Next Boot


Changing the parameter

# /usr/sbin/kctune maxfiles_lim=63488
     ==> Update the automatic 'backup' configuration first? yes
       * The automatic 'backup' configuration has been updated.
       * Future operations will update the backup without prompting.
WARNING: The validity of the tunable values could not be completely
         verified, because the value of the tunable 'maxfiles' will not
         be known until the system is booted. The tunable values will be
         verified during boot. Please check the console messages during
         boot to see if there are any tunable value errors.
       * The requested changes have been applied to the currently
         running configuration.
Tunable                 Value  Expression  Changes
maxfiles_lim  (before)   4096  Default     Immed
              (now)     63488  63488
#