Tuesday, January 12, 2016

Solaris 11.2 – using netstat in liu of lsof

Most of the time we are in situation to find the process which is holding the same port which may cause the application issues. This post will help us whenever we are in same position. Yes, we can find the  process with using port in different methods. Let we start to search it.....
root@solaris-1# ndd /dev/tcp tcp_listen_hash
    TCP            zone IP addr         port  seqnum   backlog (q0/q/max)
----skipped------
281 ffffffff81d46440 0 ::ffff:127.0.0.1 00025 00000000 0/0/16
----skipped------
 
From the above output we can find the process of the port 25, please follow the below steps
 
root@solaris-1# mdb -k
Loading modules:[unix krtld genunix specfs dtrace cpu.generic uppc pcplusmp ufs md mpt ip hook neti sctp arp usba fctl nca lofs audiosup cpc fcip random crypto zfs logindmux ptm sppp sata nfs]
> 
> ffffffff81d46440::print tcp_t tcp_rq
tcp_rq = 0xffffffff8961f028
> 
> 0xffffffff8961f028::q2stream
ffffffff87052e10
> 
> ffffffff87052e10$ 0xffffffff874bc780::whereopen
file ffffffff893689d8
ffffffff88dc8900
> 
> ffffffff88dc8900::ps
S    PID   PPID   PGID    SID    UID      FLAGS             ADDR NAME
R    935      1    935    935      0 0x52010000 ffffffff88dc8900 sendmail
> 
root@solaris-1#
 
Yes!!! We have found the process (sendmail) which using the port 25
We can use lsof utility to find the process with using port. Although lsof will not be available on all Solaris server, we have to install the lsof package. NOTE: This utility can't be used in Non-global zones.
 
root@solaris-1# /usr/local/bin/lsof -i :25
lsof: WARNING: access /.lsof_solaris-1: No such file or directory
lsof: WARNING: created device cache file: /.lsof_solaris-1
COMMAND  PID USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
sendmail 935 root    5u  IPv4 0xffffffff81d46240      0t0  TCP localhost:smtp (LISTEN)
root@solaris-1#
 
Yes, we have found the process with using port by using lsof utility.
Now our concern would be how to check the process with using port in non-global zone level. Yes, we have the option to do that by using our customized script. Please follow the steps...
 
root@solaris-1# cat /var/tmp/check_port.sh
#!/bin/ksh
 
CODE='---------------------------------------------'
PID=$(/usr/bin/ps -ef | sed 1d | awk '{print $2}')
 
if [ $# -eq 0 ]; then
   read PORT?"Enter port you would like to know pid for: "
else
   PORT=$1
fi
 
for f in $PID
do
   /usr/proc/bin/pfiles $f 2>/dev/null | /usr/xpg4/bin/grep -q "port: $PORT"
   if [ $? -eq 0 ]; then
      echo $CODE
      echo "Port: $PORT is being used by PID:\c"
      /usr/bin/ps -ef -o pid -o args | egrep -v "grep|pfiles" | grep $f
   fi
done
exit 0
root@solaris-1#
 
Copy the script and executed as mentioned below
 
root@solaris-1# /var/tmp/check_port.sh
 
Enter port you would like to know pid for: 25
---------------------------------------------
Port: 25 is being used by PID:  935 /usr/lib/sendmail -bl -q15m
root@solaris-1#
Yes... We have found the process with using the port by executing customized scripts.
 
With netstat enhancements in Solaris 11.2 release onwards new flag "-u" has been added, which would list USER, PID and program name of the process and network port details.
$ netstat -un -P tcp                                                                                                          
TCP: IPv4
   Local Address        Remote Address      User     Pid     Command     Swind  Send-Q  Rwind  Recv-Q    State
-------------------- -------------------- -------- ------ ------------- ------- ------ ------- ------ -----------
127.0.0.1.59101          127.0.0.1.5011            root    6364   telnet         130880      0  139264      0 ESTABLISHED
127.0.0.1.5011            127.0.0.1.59101          root    4884   vntsd          139060     0  130880      0 ESTABLISHED
192.168.11.2.46829    192.168.11.3.389        root      816    nscd            49232      0  128872      0 ESTABLISHED
127.0.0.1.5010            127.0.0.1.51811          root    4884   vntsd          139060     0  130880      0 ESTABLISHED
169.254.182.77.12      169.254.182.76.24      root    4930      fmd            64096      0  128872     0 ESTABLISHED
192.168.11.2.22          192.168.11.3.49363    root    4172     sshd            61952     31  128544    0 ESTABLISHED
 
<<output truncated>>
With "-v" verbose flag netstat prints the program name with full path information including other information.
t5-2@~# netstat -nu -P tcp -v
TCP: IPv4
Local/Remote Address  Swind   Snext     Suna    Rwind   Rnext     Rack    Rto   Mss     State      User    Pid      Command
-------------------- ------- -------- -------- ------- -------- -------- ----- ----- ----------- -------- ------ --------------
127.0.0.1.59101    
127.0.0.1.5011        130880 eb88eb26 eb88eb26  139264 a9687631 a9687631  1125  8180 ESTABLISHED root       6364 /usr/bin/telnet 0 5011
127.0.0.1.5011     
127.0.0.1.59101       139060 a9687631 a9687631  130880 eb88eb26 eb88eb26  1125  8180 ESTABLISHED root      4884 /usr/lib/ldoms/vntsd -i virtual-console-concentrator@0 -p localhost -t 0
192.168.11.2.22   
192.168.11.3.44826   42112 aa006f8f aa006f4f  128544 d23c83de d23c83de   685  1248 ESTABLISHED root       4172 /usr/lib/ssh/sshd
192.168.11.2.63024
192.168.11.3.389      49232 cd6ad3cc cd6ad3cc  128872 ce3c602e ce3c602e   203  1448 CLOSE_WAIT  root           816 /usr/sbin/nscd
<<output truncated>>
To display all TCP, UDP, Domain sockets you can use "-au"
 # netstat -aun
UDP: IPv4
   Local Address        Remote Address      User    Pid      Command       State
-------------------- -------------------- -------- ------ -------------- ----------
      *.*                                 root            107 in.mpathd      Unbound
      *.*                                 root            107 in.mpathd      Unbound
      *.*                                 netadm      712      nwamd      Unbound
      *.*                                 netadm      712      nwamd      Unbound
      *.111                             daemon     3080   rpcbind       Idle
      *.*                                 daemon     3080   rpcbind       Unbound
      *.58525                         daemon     3080   rpcbind       Idle
      *.111                             daemon     3080   rpcbind       Idle
      *.*                                 daemon     3080   rpcbind       Unbound
      *.46768                         daemon     3080   rpcbind       Idle
<<output truncated>>
With this feature netstat can be alternative for lsof for looking up port related information.
If you compile and run lsof from the source in Solaris. Use lsof "-i4" flag to display open ports in Solaris to display ports associated with IPv4 .
# ./lsof -i4 |head
COMMAND     PID     USER   FD   TYPE         DEVICE SIZE/OFF NODE NAME
in.mpathd       107     root        3u   IPv4 0xc40046b332c0      0t0  TCP localhost:5999 (LISTEN)
in.mpathd       107     root        5u   IPv4 0xc400469f6080       0t0  UDP *:* (Unbound)
nwamd            712   netadm    5u   IPv4 0xc40048ea4100      0t0  UDP *:* (Unbound)
rpcbind         3080   daemon    8u   IPv4 0xc400499eac00      0t0  UDP 0.0.0.17:chargen (Idle)
rpcbind         3080   daemon    9u   IPv4 0xc400499ea700      0t0  UDP 0.0.0.17:chargen (Unbound)
rpcbind         3080   daemon   10u  IPv4 0xc400499ea200      0t0  UDP 0.0.0.17:chargen (Idle)
rpcbind         3080   daemon   11u  IPv4 0xc400499b5200      0t0  TCP *:* (LISTEN)
rpcbind         3080   daemon   12u  IPv4 0xc400499b49c0      0t0  TCP *:* (IDLE)
nfs4cbd        3155   daemon    7u    IPv4 0xc400499b3100      0t0  TCP *:* (LISTEN)
<<output truncated>>
 

Bash History: Display Date & Time For Each Command

How do I display shell command history with date and time under UNIX or Linux operating systems?
If the HISTTIMEFORMAT is set, the time stamp information associated with each history entry is written to the history file, marked with the history comment character. Defining the environment variable as follows:
$ HISTTIMEFORMAT="%d/%m/%y %T "
OR
$ echo 'export HISTTIMEFORMAT="%d/%m/%y %T "' >> ~/.bash_profile
Where,
%d - Day
%m - Month
%y - Year
%T - Time
To see history type
$ history
Sample outputs:
  • ....
  • ..
  • 986 11/03/10 04:31:36 memcached-tool 10.10.28.22:11211 stats
  • 987 11/03/10 04:31:36 w
  • 988 11/03/10 04:31:37 iostat
  • 989 11/03/10 04:31:37 top
  • 990 11/03/10 04:31:37 at
  • 991 11/03/10 04:31:38 atop
  • 992 11/03/10 04:31:40 collectl
  • 993 11/03/10 04:31:41 grep CPU /proc/cpuinfo
  • 994 11/03/10 04:31:45 vmstat 3 100
  • 995 11/03/10 04:31:55 sar -W -f /var/log/sa/sa12
  • ....
  • ..
References:
For more info type the following commands:
  • man bash
  • help history
  • man 3 strftime

OEL - Using Linux installer Yum

Generic YUM usage – lesser known options

Find a package which delivers a certain file:
  • # yum whatprovides "*/gnu/stubs-32.h"
  • glibc-devel-2.12-1.80.el6.i686 : Object files for development using standard C: libraries.
  • Repo : ol6_u3_base
  • Matched from:
  • Filename : /usr/include/gnu/stubs-32.h
List a package and download the RPM
  • # yum list dhcp
  • Loaded plugins: refresh-packagekit, security
  • Available Packages
  • dhcp.x86_64 12:4.1.1-34.P1.0.1.el6 ol6_latest
  • # yumdownloader dhcp.x86_64 --destdir /KITS/YUM
  • Loaded plugins: refresh-packagekit
  • dhcp-4.1.1-34.P1.0.1.el6.x86_64.rpm | 815 kB 00:
List older versions of a package
  • # yum --showduplicate list bind-utils
  • Loaded plugins: refresh-packagekit, security
  • Installed Packages
  • bind-utils.x86_64 32:9.8.2-0.10.rc1.el6 @anaconda-OracleLinuxServer-201206261930.x86_64/6.3
  • Available Packages
  • bind-utils.x86_64 32:9.7.0-5.P2.el6 ol6_latest
  • bind-utils.x86_64 32:9.7.0-5.P2.el6_0.1 ol6_latest
  • bind-utils.x86_64 32:9.7.3-2.el6 ol6_latest
  • bind-utils.x86_64 32:9.7.3-2.el6_1.P1.1 ol6_latest
  • ....
Run RPM to upgrade a package with multiple packages
  • # rpm -Uvh bind-9.8.2-0.17.rc1.0.2.el6_4.4.x86_64.rpm bind-libs-9.8.2-0.17.rc1.0.2.el6_4.4.x86_64.rpm \
  • bind-utils-9.8.2-0.17.rc1.0.2.el6_4.4.x86_64.rpm

Standard YUM commands:

1) Install a package
# yum install glibc-headers-2.12-1.107.el6.x86_64
2) Check, drop and install a newer version of a package
Check the current status of Oracle RDBMS SERVER preinstall package
  • # rpm -qa | grep -i preinstall
  • oracle-rdbms-server-11gR2-preinstall-1.0-6.el6.x86_64
Deinstall the 11.2 package
  • # yum remove oracle-rdbms-server-11gR2-preinstall.x86_64
  • Loaded plugins: refresh-packagekit, security
Check whether 12c preinstall package is available from our YUM repository
  • # yum list | grep preinstall
  • oracle-rdbms-server-12cR1-preinstall.x86_64
Install it
# yum install oracle-rdbms-server-12cR1-preinstall.x86_64

Error: Plugin "refresh-packagekit" can't be imported using YUM

  • # yum list
  • Plugin "refresh-packagekit" can't be imported
  • Loaded plugins: security
  • Traceback (most recent call last):
  • File "/usr/bin/yum", line 29, in <module>
  • yummain.user_main(sys.argv[1:], exit_code=True)
  • File "/usr/share/yum-cli/yummain.py", line 285, in user_main
  • errcode = main(args)
  • File "/usr/share/yum-cli/yummain.py", line 136, in main
  • result, resultmsgs = base.doCommands()
  • File "/usr/share/yum-cli/cli.py", line 434, in doCommands
  • self._getTs(needTsRemove)
  • File "/usr/lib/python2.6/site-packages/yum/depsolve.py", line 99, in _getTs
  • self._getTsInfo(remove_only)
  • File "/usr/lib/python2.6/site-packages/yum/depsolve.py", line 110, in _getTsInfo
Solution: Remove $ORACLE_HOME/lib from LD_LIBRARY_PATH
  • [root@hrac1 Desktop]# env | grep LD
  • LD_LIBRARY_PATH=/u01/app/11.2.0/grid/lib:/lib:/usr/lib
  • You have new mail in /var/spool/mail/root
  • [root@hrac1 Desktop]# export LD_LIBRARY_PATH=/usr/lib
  • [root@hrac1 Desktop]# yum list
  • Loaded plugins: refresh-packagekit, security
  • ol6_u3_base | 1.4 kB 00:00

OEL - UDEV setup in a Multipath env for RAC/ASM

Environment
OEL 6.4 with
11.2.0.4.0 RAC system

Map /dev/dm-? devices to the asm disk discovery string like /dev/asm

Display ISCSI partitions/Disks available for ASM
dmsetup ls
--tree -o device
vg_oel64-lv_swap (252:1)
 +-
(8:3)
 +-
(8:2)
vg_oel64-lv_root (252:0)
 +-
(8:2)
grac41_disk1p1
(252:6)
 +-grac41_disk1
(252:2)

+- (8:240)

+- (65:32)
grac41_disk2p1
(252:7)
 +-grac41_disk2
(252:3)

+- (65:0)

+- (65:48)
grac41_disk3p1
(252:5)
 +-grac41_disk3
(252:4)

+- (65:16)

+- (65:64)

What is  Major number 252 ?
cat /proc/devices |  egrep 'devices|252'
..
Block
devices:
252
device-mapper
-->  Driver for
Device-Mapper

Availabe partitions:
dmsetup ls
grep p1
grac41_disk1p1
(252:5)
grac41_disk2p1
(252:7)
grac41_disk3p1
(252:6)



Map /dev/dm-? devices to /dev/asmdisk_OF-disk1 ( for example /dev/dm-5 to asmdisk_OF-disk1)

Create/Add to  12-dm-permissions.rules in /etc/udev/rules.d following
entries:

ENV{DM_NAME}=="grac41_disk1p1",NAME="asmdisk_OF-disk1",
OWNER:="grid", GROUP:="asmadmin",
MODE:="660"
ENV{DM_NAME}=="grac41_disk2p1",NAME="asmdisk_OF-disk2",
OWNER:="grid", GROUP:="asmadmin",
MODE:="660"
ENV{DM_NAME}=="grac41_disk3p1",NAME="asmdisk_OF-disk3",
OWNER:="grid", GROUP:="asmadmin",
MODE:="660"
  • Change the protections for underlying sdX disks used by Device Mapper 
Note : Don’t use Name in the 99-oracle-asmdevices.rules file as we get 2 disks for a
single WWID.
  • Here we change group, owner and protection of the /dev/sdX devices created by ISCSI. 

Check WWID of ISCSI disks
#  multipath -ll
|      grep dm-
grac41_disk3
(14f504e46494c45525237556f4c352d574b4e6c2d446e736b) dm-4 OPNFILER,VIRTUAL-DISK
grac41_disk2
(14f504e46494c4552786431396d6c2d6d4950492d52517178) dm-2 OPNFILER,VIRTUAL-DISK
grac41_disk1
(14f504e46494c45523264556151442d5261336d2d566e4250) dm-3 OPNFILER,VIRTUAL-DISK

Create/Add to  99-oracle-asmdevices.rules in /etc/udev/rules.d following entries
KERNEL=="sd*", SUBSYSTEM=="block", ENV{DEVTYPE}=="disk",
ENV{ID_SERIAL}=="14f504e46494c45525237556f4c352d574b4e6c2d446e736b",
OWNER="grid", GROUP="asmadmin",
MODE="0660"
KERNEL=="sd*", SUBSYSTEM=="block", ENV{DEVTYPE}=="disk",
ENV{ID_SERIAL}=="14f504e46494c4552786431396d6c2d6d4950492d52517178",
OWNER="grid", GROUP="asmadmin",
MODE="0660"
KERNEL=="sd*", SUBSYSTEM=="block", ENV{DEVTYPE}=="disk",
ENV{ID_SERIAL}=="14f504e46494c45523264556151442d5261336d2d566e4250",
OWNER="grid", GROUP="asmadmin",
MODE="0660"
Reload UDEV ( OEL 6 stile )
udevadm control --reload-rules
start_udev
Verify ASM disk protection
ls -l /dev/asmdisk_OF*
brw-rw----. 1 grid asmadmin 252, 5 Feb 28
10:06 /dev/asmdisk_OF-disk1
brw-rw----. 1 grid asmadmin 252, 7 Feb 28
10:06 /dev/asmdisk_OF-disk2
brw-rw----. 1 grid asmadmin 252, 6 Feb 28
10:06 /dev/asmdisk_OF-disk3
Find all attached sd disks used by our multipath configuration
multipath -ll | grep sd

|- 21:0:0:0 sdt 65:48 active ready running
`- 22:0:0:0 sdu 65:64 active ready running
|- 17:0:0:0 sdp 8:240 active ready running
`- 18:0:0:0 sdr 65:16 active ready running
|- 20:0:0:0 sdq 65:0  active ready running
`- 19:0:0:0 sds 65:32 active ready running
Verify SD disk protection
ls -l /dev/sdt /dev/sdu /dev/sdp /dev/sdr /dev/sdq /dev/sds
brw-rw----. 1 grid asmadmin
8
, 240 Feb 28 10:55 /dev/sdp
brw-rw----. 1 grid asmadmin 65,   0
Feb 28 10:55 /dev/sdq
brw-rw----. 1 grid asmadmin 65,  16 Feb 28 10:55 /dev/sdr
brw-rw----. 1 grid asmadmin 65,  32 Feb 28 10:55 /dev/sds
brw-rw----. 1 grid asmadmin 65,  48 Feb 28 10:55 /dev/sdt
brw-rw----. 1 grid asmadmin 65,  64 Feb 28 10:55 /dev/sdu
- The major number changes as there are eight major numbers reserved
(
 or even more ) for SCSI devices
- Those major numbers are 8, 65, 66, 67, 68, 79, 70, and 71
Verify your ASM disks on all nodes before you use asmca to create
an OPENFILER diskgroup