Solaris/VxVM: Rebuilding a disk group from CBR backups.

Issue:
Rebuilding a disk group from CBR backups.

Solution:

Customer has accidentally destroyed a disk group, or there was some private region corruption. This would require a disk group rebuild.

You will need to have several things on hand to successfully complete a rebuild.
A current CBR backup (/etc/vx/cbr/bk  the file ending in .cfgrec)
You must know the exact disk media names for each disk in the disk group.
The disk group must have the exact same name.
If re-initializing disks due to vtoc/private region corruption, you must know the exact length, and offset of the private region
(to avoid making the private region too big, and overwriting the beginning of the file system.

For a disk group that has been destroyed/corrupted, the first step is to check the dir /etc/vx/cbr/bk for a recent configuration backup. We are working with a disk group that is called "tdg".

-bash-3.00# ls -al

total 8
drwxr-xr-x   4 root     root         512 May  8 09:18 .
drwxr-xr-x   3 root     root         512 May  8 09:20 ..
drwxr-xr-x   2 root     root         512 May  5 11:03 rootdg.1241470818.10.sprsx4100a0-18
drwxr-xr-x   2 root     root         512 May  8 09:20 tdg.1241799525.16.sprsx4100a0-18

We have a recent config backup, so we'll use that.

-bash-3.00# cd /etc/vx/cbr/bk/tdg.1241799525.16.sprsx4100a0-18

-bash-3.00# ls -al

total 48220
drwxr-xr-x   2 root     root         512 May  8 09:20 .
drwxr-xr-x   4 root     root         512 May  8 09:18 ..
-rw-r--r--   1 root     root     24649728 May  8 09:20 1241799525.16.sprsx4100a0-18.binconfig
-rw-r--r--   1 root     root        6502 May  8 09:19 1241799525.16.sprsx4100a0-18.cfgrec
-rw-r--r--   1 root     root        1652 May  8 09:20 1241799525.16.sprsx4100a0-18.dginfo
-rw-r--r--   1 root     root        2705 May  8 09:19 1241799525.16.sprsx4100a0-18.diskinfo

Next, Let's check to see if the configuration looks right in a vxprint. We are going to use the most recent file with the .cfgrec extension.

-bash-3.00# cat 1241799525.16.sprsx4100a0-18.cfgrec | vxprint -D -

Disk group: tdg

TY NAME         ASSOC        KSTATE   LENGTH   PLOFFS   STATE    TUTIL0  PUTIL0
dg tdg          tdg          -        -        -        -        -       -

dm Disk_0       emc_clariion0_50 -    7089920  -        -        -       -
dm Disk_1       emc_clariion0_51 -    7089920  -        -        -       -

v  tvol         fsgen        ENABLED  14178304 -        ACTIVE   -       -
pl tvol-01      tvol         ENABLED  14178304 -        ACTIVE   -       -
sd Disk_0-01    tvol-01      ENABLED  7089920  0        -        -       -
sd Disk_1-01    tvol-01      ENABLED  7088384  7089920  -        -       -

If this looks like the correct layout, we then need to dump this out into a format that vxmake can use to recreate the disk group objects.

# cat 1241799525.16.sprsx4100a0-18.cfgrec | vxprint -D - -mvprsh > /tmp/tdg.out

If you have any disks that need to be re-initialized due to private region corruption, etc., do it now. Make sure that you know the private region offset, and length for that particular disk.
We can get this information from the file with the extension ".diskinfo" in the "/etc/vx/cbr/bk/<config backup>" directory.

-bash-3.00# less 1241799525.16.sprsx4100a0-18.diskinfo
DISK_ATTRIBUTE
Number_of_disk_backup=       2
UUID=DGC%5FRAID%205%5FAPM00064800091%5F60060160C68218008412F61304EBDC11
Device:    emc_clariion0_50
devicetag: emc_clariion0_50
type:      auto
hostid:    sprsx4100a0-18
disk:      name=Disk_0 id=1241470402.9.sprsx4100a0-18
group:     name=tdg id=1241799525.16.sprsx4100a0-18
info:      format=cdsdisk,privoffset=256,pubslice=2,privslice=2
flags:     online ready private autoconfig autoimport imported
pubpaths:  block=/dev/vx/dmp/emc_clariion0_50s2 char=/dev/vx/rdmp/emc_clariion0_50s2
guid:      {9aeb85c0-1dd1-11b2-a9e2-00144fe54846}
udid:      DGC%5FRAID%205%5FAPM00064800091%5F60060160C68218008412F61304EBDC11
site:      -
version:   3.1
iosize:    min=512 (bytes) max=2048 (blocks)
public:    slice=2 offset=65792 len=7089920 disk_offset=0
private:   slice=2 offset=256 len=65536 disk_offset=0
update:    time=1241799527 seqno=0.25
ssb:       actual_seqno=0.0
headers:   0 240
configs:   count=1 len=48144
logs:      count=1 len=7296
Defined regions:
config   priv 000048-000239[000192]: copy=01 offset=000000 enabled
config   priv 000256-048207[047952]: copy=01 offset=000192 enabled
log      priv 048208-055503[007296]: copy=01 offset=000000 enabled
lockrgn  priv 055504-055647[000144]: part=00 offset=000000
Multipathing information:
numpaths:   1
c0t60060160C68218008412F61304EBDC11d0s2 state=enabled   type=primary

UUID=DGC%5FRAID%205%5FAPM00064800091%5F60060160C68218006C1EF93504EBDC11
Device:    emc_clariion0_51
devicetag: emc_clariion0_51
type:      auto
hostid:    sprsx4100a0-18
disk:      name=Disk_1 id=1241470406.11.sprsx4100a0-18
group:     name=tdg id=1241799525.16.sprsx4100a0-18
info:      format=cdsdisk,privoffset=256,pubslice=2,privslice=2
flags:     online ready private autoconfig autoimport imported
pubpaths:  block=/dev/vx/dmp/emc_clariion0_51s2 char=/dev/vx/rdmp/emc_clariion0_51s2
guid:      {9d34466e-1dd1-11b2-a9e2-00144fe54846}
udid:      DGC%5FRAID%205%5FAPM00064800091%5F60060160C68218006C1EF93504EBDC11
site:      -
version:   3.1
iosize:    min=512 (bytes) max=2048 (blocks)
public:    slice=2 offset=65792 len=7089920 disk_offset=0
private:   slice=2 offset=256 len=65536 disk_offset=0
update:    time=1241799527 seqno=0.25
ssb:       actual_seqno=0.0
headers:   0 240
configs:   count=1 len=48144
logs:      count=1 len=7296
Defined regions:
config   priv 000048-000239[000192]: copy=01 offset=000000 enabled
config   priv 000256-048207[047952]: copy=01 offset=000192 enabled
log      priv 048208-055503[007296]: copy=01 offset=000000 enabled
lockrgn  priv 055504-055647[000144]: part=00 offset=000000
Multipathing information:
numpaths:   1
c0t60060160C68218006C1EF93504EBDC11d0s2 state=enabled   type=primary

From this output, we can see that both disks are CDS format. Both disks, emc_clariion0_50 and emc_clariion0_51 have the private region offset of 256 and a length of 65536.
emc_clariion0_50 was named Disk_0, and emc_clariion0_50 was named Disk_1.

Now we will initialize them with these values.

-bash-3.00# vxdisksetup -i emc_clariion0_50 privlen=65536 privoffset=256
-bash-3.00# vxdisksetup -i emc_clariion0_51 privlen=65536 privoffset=256

Now re-create the disk group with the same disks, with exactly the same disk media names as before.

# vxdg init tdg Disk_0=emc_clariion0_50 Disk_1=emc_clariion0_51

Now if all the disks have the right disk media names, and are initialized with the correct private region sizes, we can replay the /tmp/tdg.out through vxmake.
This will recreate all of the volume manager objects that were in the disk group, at precisely the same place they were before.

-bash-3.00# vxmake -g tdg -d /tmp/tdg.out

Now check vxprint layout, and start the volumes.

-bash-3.00# vxprint -htqg tdg
dg tdg          default      default  24000    1241801076.30.sprsx4100a0-18

dm Disk_0       emc_clariion0_50 auto 65536    7101888  -
dm Disk_1       emc_clariion0_51 auto 65536    7101888  -

v  tvol         -            DISABLED EMPTY    14178304 SELECT    -        fsgen
pl tvol-01      tvol         DISABLED EMPTY    14178304 CONCAT    -        RW
sd Disk_0-01    tvol-01      Disk_0   0        7089920  0         emc_clariion0_50 ENA
sd Disk_1-01    tvol-01      Disk_1   0        7088384  7089920   emc_clariion0_51 ENA

Startup the volumes

-bash-3.00# vxvol -g tdg start tvol

Assuming everything went well, you should now be able to mount file systems, and check for consistency.

Solaris / VxVM: Recovering plexes from DISABLED RECOVER state

Solution

Recovering plexes from DISABLED RECOVER state once lost connectivity has been restored to one or more disks.

#vxdisk list

T41_1 sliced - - online

T41_2 sliced - - online

- - data_dg15 data_dg failed was:T41_2

- - data_dg16 data_dg failed was:T41_1

 

# vxdisk -o alldgs list

DEVICE TYPE DISK GROUP STATUS

T41_1 sliced - (data_dg) online

T41_2 sliced - (data_dg) online

v nas04b_v - DISABLED ACTIVE 943718400 SELECT - fsgen

pl nas04b_v-01 nas04b_v DISABLED NODEVICE 943718400 CONCAT - RW

sd data_dg16-01 nas04b_v-01 data_dg16 0 943718400 0 - NDEV

Do:

#/usr/lib/vxvm/bin/vxreattach -r T41_2

#/usr/lib/vxvm/bin/vxreattach -r T41_1

Migrate from DISABLED RECOVER to ENABLED ACTIVE.

Symptom:

How to recover or start a volume that has a plex in the disabled and recover state

Solution:

The vxprint -ht output gives, among other things, the kernel state (column 4) and the state (column 5) of a plex .

Consider the following case:

# vxprint -ht -g testdg

dg testdg default default 84000 970356463.1203.alu

dm testdg01 c1t4d0s2 sliced 2179 8920560 -

dm testdg02 c1t6d0s2 sliced 2179 8920560 -

v test - DISABLED ACTIVE 17840128 fsgen - SELECT

pl test-01 test DISABLED RECOVER 17841120 CONCAT - RW

sd testdg01-01 test-01 testdg01 0 8920560 0 c1t4d0 ENA

sd testdg02-01 test-01 testdg02 0 8920560 8920560 c1t6d0 ENA

From the above output, it can be seen that the volume test has plex test-01 in the DISABLED and RECOVER state.

To recover the volume test, use the vxmend command. This operation applies only to volumes, or to plexes associated with a volume.

This will manually reset or change the state of a plex or volume.

The following is the procedure to recover/start this volume:

1. Bring plex test-01 to the DISABLED and OFFLINE state using the following command: vxmend -o force off <recover_plex>

For example,

# vxmend -g testdg -o force off test-01

The below output shows plex test-01 in the DISABLED and OFFLINE state:

# vxprint -ht -g testdg

dg testdg default default 84000 970356463.1203.alu

dm testdg01 c1t4d0s2 sliced 2179 8920560 -

dm testdg02 c1t6d0s2 sliced 2179 8920560 -

v test - DISABLED ACTIVE 17840128 fsgen - SELECT

pl test-01 test DISABLED OFFLINE 17841120 CONCAT - RW

sd testdg01-01 test-01 testdg01 0 8920560 0 c1t4d0 ENA

sd testdg02-01 test-01 testdg02 0 8920560 8920560 c1t6d0 ENA

2. Bring plex test-01 to the DISABLED and STALE state using the following command: vxmend on <recover_plex>

For example,

# vxmend -g testdg on test-01

The below output shows plex test-01 in the DISABLED and STALE state:

# vxprint -ht -g testdg

dg testdg default default 84000 970356463.1203.alu

dm testdg01 c1t4d0s2 sliced 2179 8920560 -

dm testdg02 c1t6d0s2 sliced 2179 8920560 -

v test - DISABLED ACTIVE 17840128 fsgen - SELECT

pl test-01 test DISABLED STALE 17841120 CONCAT - RW

sd testdg01-01 test-01 testdg01 0 8920560 0 c1t4d0 ENA

sd testdg02-01 test-01 testdg02 0 8920560 8920560 c1t6d0 ENA

 

3. Bring plex test-01 to the DISABLED and CLEAN state using the following command: vxmend fix clean <recover_plex>

For example,

# vxmend -g testdg fix clean test-01

The below output shows plex test-01 in the DISABLED and CLEAN state:

# vxprint -ht -g testdg

dg testdg default default 84000 970356463.1203.alu

dm testdg01 c1t4d0s2 sliced 2179 8920560 -

dm testdg02 c1t6d0s2 sliced 2179 8920560 -

v test - DISABLED ACTIVE 17840128 fsgen - SELECT

pl test-01 test DISABLED CLEAN 17841120 CONCAT - RW

sd testdg01-01 test-01 testdg01 0 8920560 0 c1t4d0 ENA

sd testdg02-01 test-01 testdg02 0 8920560 8920560 c1t6d0 ENA

4. Once plex test-01 is in the DISABLED/CLEAN state, the volume test can be started with the following command: vxvol start <volume>

For example,

# vxvol start test

It can be seen in the below output that the volume is now ENABLED and ACTIVE:

# vxprint -ht -g testdg

dg testdg default default 84000 970356463.1203.alu

dm testdg01 c1t4d0s2 sliced 2179 8920560 -

dm testdg02 c1t6d0s2 sliced 2179 8920560 -

v test - ENABLED ACTIVE 17840128 fsgen - SELECT

pl test-01 test ENABLED ACTIVE 17841120 CONCAT - RW

sd testdg01-01 test-01 testdg01 0 8920560 0 c1t4d0 ENA

sd testdg02-01 test-01 testdg02 0 8920560 8920560 c1t6d0 ENA

To summarize:

# vxmend -g testdg -o force off test-01

# vxmend -g testdg on test-01

# vxmend -g testdg fix clean test-01

# vxvol -g testdg start test

Solaris 10 & Above: Increase the performance of a UFS filesystem

Solution

1. Set the Write-Throttle High Watermark to 1/64 of total memory size
2. Set the Write-Throttle Low Watermark to 1/128 of total memory size
3. These Values must be set PERMANENTLY in /etc/system

 
For a system with 1 GB RAM,

• Add the following entries into /etc/system and reboot.
  
  
  set ufs:ufs_HW = 16777216
  set ufs:ufs_LW = 8388608

Solaris: How to identify if a particular process/application is performing RAMDOM I/O or SEQUENTIAL I/O?

Solution: truss

RAMDOM I/O
#truss –p <PID>
----------------------------SNIP--------------------------------------------

sigprocmask(SIG_BLOCK, 0xFFBE6D7C, 0x00000000)  = 0
setitimer(ITIMER_REAL, 0xFFBE6D04, 0x00000000)  = 0
sigprocmask(SIG_UNBLOCK, 0xFFBE6D7C, 0x00000000) = 0
lseek(9, 2560, SEEK_SET)                        = 2560
read(9, "\0\n\0\0\0\0\0 D\001\0\0".., 512)      = 512
write(13, "\0\v\0\0\f\0\0\001\001", 11)         = 11
write(13, "\0\v\0\0\f\0\0\001\002", 11)         = 11
read(13, "\0\v\0\0\f\0\0\001\002", 2064)        = 11
lseek(9, 2560, SEEK_SET)                        = 2560
read(9, "\0\n\0\0\0\0\0 D\001\0\0".., 512)      = 512
lseek(9, 117760, SEEK_SET)                      = 117760
read(9, "\0\n19 n\0\0\0 D19 o\0\0".., 512)      = 512
lseek(9, 117760, SEEK_SET)                      = 117760
read(9, "\0\n19 n\0\0\0 D19 o\0\0".., 512)      = 512
write(13, "\0B3\0\006\0\0\0\0\004\0".., 179)    = 179
read(13, "\092\0\006\0\0\0\0\003 ^".., 2064)    = 146
write(13, "\0 B\0\006\0\0\0\0\0\v05".., 66)     = 66
read(13, "\09E\0\006\0\0\0\0\011 i".., 2064)    = 158
pread64(406, "0602\0\00201A3E0\0 B O9D".., 8192, 0x347C0000) = 8192
pread64(406, "0602\0\002\0B7 0\0 B O9B".., 8192, 0x16E60000) = 8192
pread64(406, "0602\0\002\0B7 1\0 B O9B".., 8192, 0x16E62000) = 8192

----------------------------SNIP--------------------------------------------
If you see "lseek" system calls being made, then the system is performing a "RANDOM I/O".
In a RANDOM I/O operation, "lseek" system calls will be made either BEFORE the READ or WRITE system calls.

SEQUENTIAL I/O
#truss –p <PID>
    
Here, you would find only "READ" & "WRITE" system calls. No "lseek" system calls will be executed.

 

SEQUENTIAL I/O    =    Read, Write RANDOM I/O    =    Read, Write, lseek

Solaris 10 & Above: LOGROTATE configuration using logadm

Solaris 10 & above versions have builtin "logadm" utility meant for logfile rotation.
Options for logadm command:

-w:             Modify the configuration file

-c <count>:     Delete the oldest versions until there are no more than <count> files left.

-p <period>:    Rotate the logfiles after a specified time period

    Example:    -p 30d = 30 days

                -p 2w = 2 weeks

                -p 4m = 4 months

                -p 2y = 2 years

-s <size>:      Size limit of the file

LOGROTATE CONFIGURATION

1st Option:

# logadm –w /var/apache/logs/access_logs –s 100m

# logadm –w /oracle/bdump/alert_db.log –p 7d –c 10 

2nd Option:

# cd /etc

# vi logadm.conf 

- Add:

/var/apache/logs/access_logs –s 100m

/oracle/bdump/alert_db.log –p 7d –c 10

  
- Save & Exit.
- Restart logrotate – DONE.

Solaris 11.2: ipadm Fails With Insufficient Memory Error

When multiple network interfaces are created in the kernel zone, the ipadm command fails with the following error:

ipadm: cannot create interface net3: Insufficient memory

Workaround: Increase the capped-memory value for the kernel zone. The default value is 2 GB. For example:

# zonecfg -z kzone1

kzone1> select capped-memory

kzone1> set physical=16g

kzone1> end

kzone1> exit

Solaris 11: Identifying Ethernet MAC addresses

Solaris 11 introduced very useful command that you can use for checking and to set all data links –dladm.
Let's assume you need to find out ethernet (MAC) addresses of all network interfaces on your Solaris box.
Issuing the dladm command with option -m you will get the list of factory MAC addresses, their slot identifiers and their availability.

$ dladm show-phys -m
LINK SLOT ADDRESS INUSE CLIENT
net0 primary 8:0:27:4:7c:5e yes net0

Solaris 10 and older:This command however does not work on older releases of Solaris. In order to see ethernet address of the network interface even though it is not plumed you can use the following:

$ prtpicl -c network -v

This should include driver-name and instance to associate an block to a specific network interface and the local-mac-address to get the mac address of that entry.
Other option valid for all versions of Solaris is to use the netstat command:

$ netstat -pn