CPU Memory Utilization Commands Ubuntu

CPU Commands in Linux:

top  ==> Press c
top  ==> Press shift+m

Show Processes in Descending order of Memory in Linux:
ps aux | head -1; ps aux | sort -rnk 4 | head

ps -eo pid,ppid,%mem,%cpu,cmd --sort=-%cpu | head



Show Processes in Descending order of CPU in Linux:
ps aux | head -1; ps aux | sort -rnk 3 | head

ps -eo pid,ppid,%mem,%cpu,cmd --sort=-%mem | head

Nginx Key and Certificate Creation

SSL Key and Certificate creation in Nginx:

Ubuntu@Server:/etc/nginx$ mkdir ssl
Ubuntu@Server:/etc/nginx$ cd ssl

//Below commands are explained at the end of this....

Ubuntu@Server:/etc/nginx/ssl$ sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/nginx/ssl/dmodi_nginx.key -out /etc/nginx/ssl/dmodi_nginx.crt
[sudo] password for wibmoapp:
    Generating a 2048 bit RSA private key
    ...................................+++
    .....................................+++
    writing new private key to '/etc/nginx/ssl/dmodi_nginx.key'
    -----
    You are about to be asked to enter information that will be incorporated
    into your certificate request.
    What you are about to enter is what is called a Distinguished Name or a DN.
    There are quite a few fields but you can leave some blank
    For some fields there will be a default value,
    If you enter '.', the field will be left blank.
    -----
    Country Name (2 letter code) [AU]:IN
    State or Province Name (full name) [Some-State]:Karnataka
    Locality Name (eg, city) []:Bengaluru
    Organization Name (eg, company) [Internet Widgits Pty Ltd]:Wibmo
    Organizational Unit Name (eg, section) []:Payzapp Team
    Common Name (e.g. server FQDN or YOUR name) []:www.deepakmodi.com
    Email Address []:deepak.modi@wibmo.com
    
Ubuntu@Server:/etc/nginx/ssl$ ls -lrth
total 8.0K
-rw-r----- 1 root root 1.7K Feb 27 18:53 dmodi_nginx.key
-rw-r----- 1 root root 1.5K Feb 27 18:53 dmodi_nginx.crt
Ubuntu@Server:/etc/nginx/ssl$


Now modify the nginx.conf file:
server {
        listen 80 default_server;
        listen [::]:80 default_server ipv6only=on;

        listen 443 ssl;

        root /usr/share/nginx/html;
        index index.html index.htm;

        server_name your_domain.com;
        ssl_certificate /etc/nginx/ssl/dmodi_nginx.crt;
        ssl_certificate_key /etc/nginx/ssl/dmodi_nginx.key;

        location / {
                try_files $uri $uri/ =404;
        }
}

Restart the Nginx:
service nginx restart

Now Try:
http://server_domain_or_IP
and
https://server_domain_or_IP

Command Explanation:
openssl: This is the basic command line tool for creating and managing OpenSSL certificates, keys, and other files.

req: This subcommand specifies that we want to use X.509 certificate signing request (CSR) management. The "X.509" 
    is a public key infrastructure standard that SSL and TLS adheres to for its key and certificate management. 
    We want to create a new X.509 cert, so we are using this subcommand.
    
-x509: This further modifies the previous subcommand by telling the utility that we want to make a self-signed 
    certificate instead of generating a certificate signing request, as would normally happen.
    
-nodes: This tells OpenSSL to skip the option to secure our certificate with a passphrase. We need Nginx to be 
        able to read the file, without user intervention, when the server starts up. A passphrase would prevent 
        this from happening because we would have to enter it after every restart.
        
-days 365: This option sets the length of time that the certificate will be considered valid. We set it for one year here.

-newkey rsa:2048: This specifies that we want to generate a new certificate and a new key at the same time. We did 
    not create the key that is required to sign the certificate in a previous step, so we need to create it along 
    with the certificate. The rsa:2048 portion tells it to make an RSA key that is 2048 bits long.
    
-keyout: This line tells OpenSSL where to place the generated private key file that we are creating.

-out: This tells OpenSSL where to place the certificate that we are creating.

Squid Proxy Installation and Configuration

Squid

Squid is a most popular caching and forwarding HTTP web proxy server. It is used to cache web pages from 
a web server to improve web server speed, reduce response times and reduce network bandwidth usage.

Installation of Squid in Ubuntu:
sudo apt update    --To update ubuntu.
sudo apt -y install squid
sudo systemctl start squid
sudo systemctl enable squid
sudo systemctl status squid

Squid configuration file: /etc/squid/squid.conf
Squid Access log: /var/log/squid/access.log
Squid Cache log: /var/log/squid/cache.log

Configure Squid:
vi /etc/squid/squid.conf

http_port : This is the default port for the HTTP proxy server, by default it is 3128, you may change 
            it to any other port that you want, you may also add the “transparent” tag to the end of 
            the line like http_port 8888 transparent to make Squid proxy act like a transparent proxy if you want.
            
http_access deny all : This line won’t let anybody to access the HTTP proxy server, that’s why you need to change 
            it to http_access allow all to start using your Squid proxy server.
            
visible_hostname : This directive is used to set the specific hostname to a squid server. You can give any hostname to squid.


Restart Squid:
sudo systemctl restart squid


Configure squid as an HTTP proxy using only the client IP address for authentication. To allow only one IP address to 
access the internet through your new proxy server, you will need to define new acl (access control list) in the configuration file.

vi /etc/squid/squid.conf
acl localnet src XX.XX.XX.XX
Where XX.XX.XX.XX is the IP address of client machine. This acl should be added in the beginning of the ACL’s section.
Example: acl localnet src 192.168.0.102  #Boss IP address, Some comments

You will need to restart Squid service to take the new changes into effect.
$ sudo systemctl restart squid

Open Ports in Squid Proxy. By default, only certain ports are allowed in the squid configuration, add like below: 
acl Safe_ports port XXX    --Where XXX is the port number that you wish to allow. 


Block Websites:
sudo touch /etc/squid/blacklisted_sites.acl    --Create a file
.badsite1.com    --File content
.badsite2.com    --File content
Squid will block all references to that sites including www.badsite1, subsite.badsite1.com etc. 

acl bad_urls dstdomain "/etc/squid/blacklisted_sites.acl"
http_access deny bad_urls



Block Specific Keyword with Squid
sudo touch /etc/squid/blockkeywords.lst
    facebook
    instagram
    gmail

acl blockkeywordlist url_regex "/etc/squid/blockkeywords.lst"
http_access deny blockkeywordlist

sudo systemctl restart squid

Configure Proxy in Browser now to hit Squid:
Open Firefox and go to Edit –> Preferences –> Advanced –> Network –> Settings and select “Manual proxy configuration”.

Hint: https://www.tecmint.com/install-squid-in-ubuntu/

Ubuntu System Hardware Details

Know Ubuntu System Hardware details:

OS Details:
uname -a
    Linux stg-wib-app32 3.19.0-25-generic #26~14.04.1-Ubuntu SMP Fri Jul 24 21:16:20 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux


RAM Memory Details:
free -m
                 total       used       free     shared    buffers     cached
    Mem:         32175      18520      13655          0        248       4840
    -/+ buffers/cache:      13430      18745
    Swap:        32765         64      32701

Core and CPU:
lscpu
    Architecture:          x86_64
    CPU op-mode(s):        32-bit, 64-bit
    Byte Order:            Little Endian
    CPU(s):                8
    On-line CPU(s) list:   0-7
    Thread(s) per core:    1
    Core(s) per socket:    4
    Socket(s):             2
    NUMA node(s):          1
    Vendor ID:             GenuineIntel
    CPU family:            6
    Model:                 79
    Stepping:              1
    CPU MHz:               2097.570
    BogoMIPS:              4195.14
    Hypervisor vendor:     VMware
    Virtualization type:   full
    L1d cache:             32K
    L1i cache:             32K
    L2 cache:              256K
    L3 cache:              40960K
    NUMA node0 CPU(s):     0-7


Processor Details:
cat /proc/cpuinfo

    processor       : 0
    vendor_id       : GenuineIntel
    cpu family      : 6
    model           : 79
    model name      : Intel(R) Xeon(R) CPU E5-2683 v4 @ 2.10GHz
    stepping        : 1
    microcode       : 0xb00002a
    cpu MHz         : 2097.570
    cache size      : 40960 KB
    physical id     : 0
    siblings        : 4
    core id         : 0
    cpu cores       : 4
    apicid          : 0
    initial apicid  : 0
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 20
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch ida arat epb pln pts dtherm fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 invpcid rtm rdseed adx smap xsaveopt
    bugs            :
    bogomips        : 4195.14
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 42 bits physical, 48 bits virtual
    power management:

    processor       : 1
    vendor_id       : GenuineIntel
    cpu family      : 6
    model           : 79
    model name      : Intel(R) Xeon(R) CPU E5-2683 v4 @ 2.10GHz
    stepping        : 1
    microcode       : 0xb00002a
    cpu MHz         : 2097.570
    cache size      : 40960 KB
    physical id     : 0
    siblings        : 4
    core id         : 1
    cpu cores       : 4
    apicid          : 1
    initial apicid  : 1
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 20
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch ida arat epb pln pts dtherm fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 invpcid rtm rdseed adx smap xsaveopt
    bugs            :
    bogomips        : 4195.14
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 42 bits physical, 48 bits virtual
    power management:

    processor       : 2
    vendor_id       : GenuineIntel
    cpu family      : 6
    model           : 79
    model name      : Intel(R) Xeon(R) CPU E5-2683 v4 @ 2.10GHz
    stepping        : 1
    microcode       : 0xb00002a
    cpu MHz         : 2097.570
    cache size      : 40960 KB
    physical id     : 0
    siblings        : 4
    core id         : 2
    cpu cores       : 4
    apicid          : 2
    initial apicid  : 2
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 20
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch ida arat epb pln pts dtherm fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 invpcid rtm rdseed adx smap xsaveopt
    bugs            :
    bogomips        : 4195.14
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 42 bits physical, 48 bits virtual
    power management:

    processor       : 3
    vendor_id       : GenuineIntel
    cpu family      : 6
    model           : 79
    model name      : Intel(R) Xeon(R) CPU E5-2683 v4 @ 2.10GHz
    stepping        : 1
    microcode       : 0xb00002a
    cpu MHz         : 2097.570
    cache size      : 40960 KB
    physical id     : 0
    siblings        : 4
    core id         : 3
    cpu cores       : 4
    apicid          : 3
    initial apicid  : 3
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 20
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch ida arat epb pln pts dtherm fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 invpcid rtm rdseed adx smap xsaveopt
    bugs            :
    bogomips        : 4195.14
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 42 bits physical, 48 bits virtual
    power management:

    processor       : 4
    vendor_id       : GenuineIntel
    cpu family      : 6
    model           : 79
    model name      : Intel(R) Xeon(R) CPU E5-2683 v4 @ 2.10GHz
    stepping        : 1
    microcode       : 0xb00002a
    cpu MHz         : 2097.570
    cache size      : 40960 KB
    physical id     : 1
    siblings        : 4
    core id         : 0
    cpu cores       : 4
    apicid          : 4
    initial apicid  : 4
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 20
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch ida arat epb pln pts dtherm fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 invpcid rtm rdseed adx smap xsaveopt
    bugs            :
    bogomips        : 4195.14
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 42 bits physical, 48 bits virtual
    power management:

    processor       : 5
    vendor_id       : GenuineIntel
    cpu family      : 6
    model           : 79
    model name      : Intel(R) Xeon(R) CPU E5-2683 v4 @ 2.10GHz
    stepping        : 1
    microcode       : 0xb00002a
    cpu MHz         : 2097.570
    cache size      : 40960 KB
    physical id     : 1
    siblings        : 4
    core id         : 1
    cpu cores       : 4
    apicid          : 5
    initial apicid  : 5
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 20
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch ida arat epb pln pts dtherm fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 invpcid rtm rdseed adx smap xsaveopt
    bugs            :
    bogomips        : 4195.14
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 42 bits physical, 48 bits virtual
    power management:

    processor       : 6
    vendor_id       : GenuineIntel
    cpu family      : 6
    model           : 79
    model name      : Intel(R) Xeon(R) CPU E5-2683 v4 @ 2.10GHz
    stepping        : 1
    microcode       : 0xb00002a
    cpu MHz         : 2097.570
    cache size      : 40960 KB
    physical id     : 1
    siblings        : 4
    core id         : 2
    cpu cores       : 4
    apicid          : 6
    initial apicid  : 6
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 20
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch ida arat epb pln pts dtherm fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 invpcid rtm rdseed adx smap xsaveopt
    bugs            :
    bogomips        : 4195.14
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 42 bits physical, 48 bits virtual
    power management:

    processor       : 7
    vendor_id       : GenuineIntel
    cpu family      : 6
    model           : 79
    model name      : Intel(R) Xeon(R) CPU E5-2683 v4 @ 2.10GHz
    stepping        : 1
    microcode       : 0xb00002a
    cpu MHz         : 2097.570
    cache size      : 40960 KB
    physical id     : 1
    siblings        : 4
    core id         : 3
    cpu cores       : 4
    apicid          : 7
    initial apicid  : 7
    fpu             : yes
    fpu_exception   : yes
    cpuid level     : 20
    wp              : yes
    flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch ida arat epb pln pts dtherm fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 invpcid rtm rdseed adx smap xsaveopt
    bugs            :
    bogomips        : 4195.14
    clflush size    : 64
    cache_alignment : 64
    address sizes   : 42 bits physical, 48 bits virtual
    power management:

Choose JVM / Java Version in Ubuntu

Choose JVM Version while running an application in Ubuntu. 
Assumption: There are multiple JVM versions installed in system and for few processes we want JDK1.7, for few we want JDK1.8.

Ubunut$   update-alternatives --config java
There are 2 choices for the alternative java (providing /usr/bin/java).  
 
  Selection    Path                                            Priority   Status
------------------------------------------------------------
* 0            /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java   1081      auto mode
  1            /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java   1081      manual mode
  2            /usr/lib/jvm/jdk1.8.0_161/bin/java               1         manual mode
  3            /usr/lib/jvm/jdk1.7.0_160/bin/java               1         manual mode

Press <enter> to keep the current choice[*], or type selection number:  3   Press Enter (For using JDK1.7)

Sensitive Password Entry in Console using Java

Dear Reader,
When you need an application where some Password is required to start the application and that need 
to be entered in Console (Unix OS/Windows OS) manually Eg: Starting Master HSM server for Card Encryption/Decryption, 
Starting Unix based critical applications, you don't want to show either * or text while typing password. This may 
cause a shoulder surfing.

In Java you can do this using Below API, I have tested this Windows and UNIX. Running this in Eclipse will throw Exception.
Hence Run in Command Prompt.

import java.io.Console;
public class HSMPassword {
    public static void main(String[] args) {
        char[] pass ;
        Console console = System.console();
        System.out.println("Reading Password From Console, Please Press Enter after typing your password.");
        pass = console.readPassword("");
        String slotPIN = new String(pass);
        System.out.println("Password Entered is: "+slotPIN);
    }
}
--------------Starting in Console--------------------
Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation.  All rights reserved.
E:\MyDiary>javac HSMPassword.java
E:\MyDiary>java HSMPassword
Reading Password From Console, Please Press Enter after typing your password.

Password Entered is: deepak

E:\MyDiary>java HSMPassword
Reading Password From Console, Please Press Enter after typing your password.

Password Entered is: hello
E:\MyDiary>

Unix command to know about Memory, CPU and Hardware Details

/proc/meminfo

/proc/cpuinfo

lscpu

Change Date in Ubuntu

Change Date in Ubuntu:
root@XXX:/home/dmodi# sudo date --set="2016-09-06 15:02:30"
Tue Sep 6 15:02:30 IST 2016

Unix - Grep a process and kill all at once

Dear Reader,
Today I am writing how to find/grep a process and kill in Unix.
This is solely for my reference, so even if it is not useful for you, sorry for that.

//vi killProcess.sh

#Store All processes in ProcessFile.txt
exec ps -eaf | grep APPNAME | cut -c -135

ps -eaf | grep MY_PROCESS | awk '{print $2}' > ProcessFile.txt

while read line
do
    echo "Killing Process Id $line"
    kill -9 $line
done < ProcessFile.txt

echo 'Executing command after killing processes: ps -eaf | grep APPNAME'
exec ps -eaf | grep APPNAME | cut -c -135

File Search and Recursive grep

File Search and Recursive grep:
    //This command will search in the current directory and its sub-directories for a file named FileName.sh. 
    find * -name "FileName.sh" -print
    find . -name "FileName.sh" -print

NOTE: The -print option will print out the path of any file that is found with that name. 

//This command will only list file names eG*.sh.
    find . -type f -name 'eG*.sh'

//Search file called httpd.conf in all directories:
    find / -type f -name httpd.conf

//Search file called httpd.conf in /usr/local directory:
    $ find /usr/local -type f -name httpd.conf

-----------------------------grep command--------------------------
grep command is used to search some content in a file or list of files. The input we provide is case sensitive
for grep command. There are options to overcome this.

For Non-Zip files:
    grep 'SMS' applicationNonZip.log.txt     (CASE-SENSITIVE "SMS")
    grep -i 'SMS' applicationNonZip.log.txt  (NON CASE-SENSITIVE "SMS")

For Zip files:
    zgrep 'SMS' application.log.gz
    zgrep -i 'SMS' application.log.gz

For Searching Recursively in current directory and its sub-directories:
    grep -r --include=*.sh MAIN_CLUSTER .     
Search MAIN_CLUSTER in *.sh files in current directory and sub-directories. It will print file Name and Matched Line Contents.

grep -rl SMS *
Search SMS recursively and print ONLY FILE NAMES containing SMS. It will not print Matched Line Contents.

Search String Forward:   /STRING  (Press ENTER. Press "n" for next occurance, "N" (SHIFT+n) for back occurance).  
SHIFT+g (means G, will take cursor to end of file). Press only g, you will be moved to begining of file.

Search and Replace First occurance:          :s/EG3_OLD/EG2_NEW
Search and Replace in one line occurance:    :s/EG3_OLD/EG2_NEW/g
Search and Replace in entire File all Occurance:      :%s/EG3_OLD/EG2_NEW/g

//Check Running Processes
ps -eaf | cut -c -125 | grep  _PROCESSNAME

//Setting Java Home Path in Unix
export JAVA_HOME=/usr/lib/jvm/jdk1.7.0_60
export CLASSPATH=$JAVA_HOME/lib/tools.jar:.:
export PATH=$JAVA_HOME/bin:$PATH:

//Search and Replace String in a File without opening
sed -i -- 's/OLD/NEW/g' FileName.sh

//List of Users in ubuntu
compgen -u

//List of User Groups in ubuntu
compgen -g

//Create user in ubuntu
sudo adduser lion     (It will ask for password and details, enter those...)

//To know which group current logged in user belongs to:
groups

//To know which group any user belongs to:
groups USERNAME

----------------------Setting Oracle Home in Unix------------------------
export ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_1
export ORACLE_SID=orcl
export PATH=$ORACLE_HOME/bin:$ORACLE_HOME:$PATH:.

//Login to Oracle System
sqlplus
Enter user-name: USERNAME
Enter password:  password

//Login to Oracle System as SYSDBA
sqlplus / as sysdba
SQL> startup   (Start Oracle Instance)
SQL> exit

lsnrctl start  (Start Oracle Listener)

//Setting Linesize and PageSize in Unix. Run below command in SQL Prompt.
SET linesize 300;
SET pagesize 2000;
//Setting AutoTrace On, this will give detailed plan if we execute any sql query in oracle. Run below command in SQL Prompt.
SET autotrace ON;

//Manual Index Query in Oracle
select /*+ index(TABLE_NAME index_name) */ Count(*) from TABLE_NAME where 
    time_stamp >='06-NOV-13' and time_stamp <='07-NOV-13' and card_no = 'XXXXX';

----------------------END------------------------

Truncate running nohup.out file in Unix

Truncate running nohup.out file in Unix
Sometimes because of faulty logger implementations or a process which is running for a long time in Unix
Operating System, background console output file “nohup.out” continues to grow in size and finally we need
to restart the process to either remove this or truncate this.

The below command will empty the “nohup.out” file without impacting running process:


UNIX_SHELL$   >   nohup.out


Or

UNIX_SHELL$  cat   /dev/null   >   nohup.out

JMAP, HISTO, Thread Dump, High CPU Utilization

Dear Reader,

In a production environment when we reach 100% or more than 100% CPU usage, developers and Production Support guys
have a nightmare to fix and handle the client calls. Java profiling help us to detect such CPU usage but not an 
option in production systems. Se we take thread dump, histo and use TDA or some other tools to pin point the faulty 
code implementation. This is really a panic scenario when you are handling production environment and issue comes at odd times.
This is written only for Unix environment (Ubuntu and Solaris) where these Unix command runs.

Fortunately, Java comes with some great debugging tools, We need to assemble those tools with Linux commands.

I am going to explain below items here:
    1) Introduction about Java threads and its relation to Linux LWP (Light Weight Process).
    2) Step-by-step process to take thread dump and analyze CPU utilization.
    3) "jmap" - Memory Map (dump), Command to get this.
    4) "jmap" - Histo, Command to get this.
    5) Command to see list of open files in Unix.
    6) Command to achieve the same (Resolving High CPU issue) in Sun Solaris Systems.

    

1) Introduction: A java program starts when JVM calls the main method, this creates a thread called the main thread and any 
    thread you create using java code will be derived from the main thread. The same exact behavior occurs at the Linux OS 
    level, the main thread for java means a Process for the OS and every thread you create using java the OS will create a 
    Light-weight-process or LWP. In short: Java main thread = Linux process and Java supporting threads = Linux LWP.
    LWP we give an alias name as Native ID.

    The solution requies:
        Ask Linux which LWP is eating the CPU.
        Ask Java for a Thread Dump.
        Map this LWP to a Java thread.
        Get the part of code causing the issue.    
    
2) Step-by-step process:    
        Get the PID: the very first step is to know what is the Java process ID, we will use Linux commands 
        as below. Use ONE OF THE BELOW commands (we use our grep "DAPPNAME", you can use anything like "grep java"):
        jps -v | cut -c -106 | grep DAPPNAME
        jps -mvl | cut -c -106 | grep DAPPNAME
        ps -eaf | cut -c -106 | grep DAPPNAME
        ps -ef | cut -c -106 | grep DAPPNAME
        ps -eaf | grep java
        
        Below are the sample output when you execute the command:
        dmodi@MyDirectory:~$ jps -mlv | cut -c -106 | grep DAPPNAME
        8243 org.quickserver.net.server.QuickServer -load config/DmodiServer.xml -DAPPNAME=CLIENT    
        13712 org.quickserver.net.server.QuickServer -load ./conf/DmodiDNXServer.xml -DAPPNAME=SERVER
        12229 org.quickserver.net.server.QuickServer -load ./config/DmodiPOSServer.xml -DAPPNAME=SERVER2
        
        Explanation: "jps" - Java Virtual Machine Process Status Tool, a command in Unix. "106" shows 106 
        characters we want to display in console.
        
        
        The next step is to get the CPU usage per each LWP related to the main process, we can use below commands:
        //Replace PROCESS_ID with numeric process_id you get after using "top" command in Unix, which is having more than 100% 
        CPU usage.
        
        ps -eLo pid,lwp,nlwp,ruser,pcpu,stime,etime,args | grep PROCESS_ID | cut -c -106 > ThreadsList.txt
        
  
        The newly created file ThreadsList.txt will contain below things in similar way (The headers will not be 
        shown as below):        
        PID   LWP  NLWP RUSER    %CPU STIME  ELAPSED     COMMAND
        8243  8243 3  dmodi  0.0 May13  1-19:20:18 java -Dprogram.name=run.sh -Xms64m -Xmx100m -Dsun.rmi.dgc
        8243  8244 3  dmodi  0.0 May13  1-19:20:18 java -Dprogram.name=run.sh -Xms64m -Xmx100m -Dsun.rmi.dgc
        8243  8245 3  dmodi 99.9 May13  1-19:20:18 java -Dprogram.name=run.sh -Xms64m -Xmx100m -Dsun.rmi.dgc
         
         
         To see headers too, just execute the below command (but it will display all processes, Process_Id is not entered):
         ps -eLo pid,lwp,nlwp,ruser,pcpu,stime,etime,args > ThreadsList.txt
         

         Explanation: PID is process Id.
            LWP: is Light weight processes Lists (java Threads for given ProcessId) for the above PID. These values are in Decimal.
            NLWP: is number of LWP created for the above PID.            
            We can see LWP (Thread) - 8245 is eating CPU. We need to convert this value into HEX value which will be "2035".
         
        Now take the thread dump and kill the process Id: 8243. 
        See below command:
        //Taking thread dump
   
       
        jstack -l 8243 > ThreadDump_15_May_2014_13_PM.txt
             

        //Killing process
        kill -9 8243
 
        kill -3 8243  (Can also be used in Ubuntu. If used for Solaris, It will generate Thread dump file along with killing).

        
  Open the thread dump file ThreadDump_15_May_2014_13_PM.txt and search the hexa value "2035". 
     Also if you are using TDA (Thread dump analyzer tool) to see this dump file, you can see Native-ID column. You can see the 
     decimal thread Id (LWP): 8245 too. Click the link in TDA for this ThreadId, it will display the faulty code in TDA console.. 
  
3) "jmap" - Memory Map (dump): Prints shared object memory maps or heap memory details of a given JVM process.
        dmodiUnixUser@productName79:~$ jmap -dump:file=deepak.bin 8243
        Dumping heap to /home/dmodiUnixUser/deepak.bin ...
        Heap dump file created
        dmodiUnixUser@productName79:~$ ls
        deepak.bin
        This newly created file will be big in size (of 5-10 MB around). You can't see this content using 
        "less" or "cat" command. You need tool to see this. We don't use this generally, so not mentioning here.
            
4) "jmap" - Histo: See below command:
        dmodiUnixUser@productName79:~$ jmap -histo:live 8243 > deepak.txt
        Contents of this file "deepak.txt" will have similar like above:

        num     #instances         #bytes  class name
        ----------------------------------------------
        1:         14452        2229096  <constMethodKlass>
        2:         14452        1740720  <methodKlass>
        3:          1004        1406296  <constantPoolKlass>
        4:          1336        1270504  [B
        5:         25057        1060840  <symbolKlass>
        6:           835         809368  <constantPoolCacheKlass>
        7:          1004         787096  <instanceKlassKlass>

5) List of open files in Linux:    
    lsof - list open files    
    dmodiUnixUser@productName79:~$ lsof | grep home/dmodi/productName/dist/
    COMMAND   PID  USER             FD     TYPE   DEVICE  SIZE/OFF  NODE  NAME
    java    16460  dmodiUnixUser  mem    REG    9,2     25680 8127062 home/dmodi/productName/dist/sample1.jar
    java    16460  dmodiUnixUser  mem    REG    9,2     66770 8127061 home/dmodi/productName/dist/sample2.jar

    dmodiUnixUser@productName79:~$ lsof | grep PROCESS_ID > help.txt
    dmodiUnixUser@productName79:~$ less help.txt

6) The Above mentioned few commands may not work for Sun Solaris system. Hence to track high CPU consuming Process and ThreadId, 
   "prstat" is used. The syntax is "prstat -L -p ".

 Example: prstat -L -p 22991 >> 22991.txt     
 This will generate file name 22991.txt having all the Threads and CPU Usage details for ProcessId 22991.

 The headers will be like this :
 PID  USERNAME  SIZE   RSS STATE  PRI NICE   TIME   CPU   PROCESS/LWPID
22991  root     7667M 4953M cpu18     0    0  14:47:00 99.6% java/119286
22991  root     7667M 4953M sleep    59    0   0:02:14 0.0% java/46
22991  root     7667M 4953M sleep    59    0   0:00:15 0.0% java/15
22991  root     7667M 4953M sleep    59    0   0:00:15 0.0% java/14

Here "PID corresponds to Solaris Java process ID".
      "CPU corresponds to the CPU utilization % of a particular Java Thread".
      "PROCESS/LWPID corresponds to Light Weight Process ID e.g. your native Java ThreadID belonging to ProcessID 22991".
HERE prstat says, ThreadId #119286 is the top CPU contributor with 99.6% utilization and hence the faulty code must be fixed.

Hence take thread dump immediately using command "kill -3 22991". This command will generate a Thread Dump from Java process HotSpot VM format.
Convert Thread ID #119286 which is in decimal format to HEXA, corresponds value is 0X1D1F6 (this HEXA format, see 0X prefix). 
HEXA value is "1D1F6". Now search this HEXA value in Thread Dump file, you will get the exactly faulty code stacktrace.
----------------------------------END-----------------------------------------

Compiling and Running a Java program in UNIX requiring many JAR files

Dear reader,
Here I am writing a command used to compile and run a Java Program in Unix environment without Creating a JAR file.
The compilation requires many SUPPORTED Jar files which are there in "lib" directory. Below is the directory structure:
-------------------------Directory Structure start--------------------------
[dmodi@iqaapp8 BillPayBusinessService]$ ls
lib  org
[dmodi@iqaapp8 BillPayBusinessService]$ ls lib
axis.jar  commons-discovery.jar  commons-logging-1.1.1.jar  javaee.jar  jaxrpc.jar  weblogic.jar  wsdl4j.jar
[dmodi@iqaapp8 BillPayBusinessService]$ ls org/tempuri/
holders             Service.java          ServiceSoapStub.java  PushDataTo_TMSResponsePushDataTo_TMSResult.java   
ServiceSoap.java    MySoapHeader.java     ServiceLocator.java   TestOne.java (This is the main program which will execute).
[dmodi@iqaapp8 BillPayBusinessService]$
-------------------------Directory Structure end--------------------------

-------------------------
Compilation command:
[dmodi@iqaapp8 BillPayBusinessService]$ javac -d  .  -cp .:lib/weblogic.jar:lib/axis.jar:lib/commons-discovery.jar:
lib/commons-logging-1.1.1.jar:lib/javaee.jar:lib/jaxrpc.jar:lib/wsdl4j.jar  org/tempuri/*.java

Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
[dmodi@iqaapp8 BillPayBusinessService]$
-------------------------
Running command:
[dmodi@iqaapp8 BillPayBusinessService]$ java -cp .:lib/weblogic.jar:lib/axis.jar:lib/commons-discovery.jar:
lib/commons-logging-1.1.1.jar:lib/javaee.jar:lib/jaxrpc.jar:lib/wsdl4j.jar  org/tempuri/TestOne

//OUTPUT
-------------------------

Unix command to copy files between 2 Systems

Dear Reader,
Today I am writing Unix commands to copy files and directories in Unix systems without FileZilla or 
WinScp like User Interfaces.

Suppose you want to copy a file from one system (SOURCE) to another system (DESTINATION), Login into 
DESTINATION system. Go to the location where you want to keep the copied contents.

Assumptions:
    SOURCE IP: X.X.X.113  (Type full numeric complete IP address in copy command).
    DESTINATION IP: X.X.X.50 (Type full numeric complete IP address in copy command).
    UserName for SOURCE IP: dm0945
    Password: System will ask once you enter the below copy command.    
    First login into DESTINATION IP system and move to destination directory, then type below command:
    
    scp dm0945@X.X.X.113:/home/dmodi/log/q2-2011-09-21.log  .    (Press Enter, it will ask password for X.X.X.113 system.
    The log file will be copied to current location of DESTINATION IP System. 
    
Above command will copy a file, if you want entire diectory to copy, tweak the above command as below:
    scp -r dm0945@X.X.X.113:/home/dmodi/log/q2-2011-09-21.log  . (Press Enter, it will ask password for X.X.X.113 system.
    Entire log directory will be copied to current location of DESTINATION IP System. 


---------------END---------------