Introduction to Linux Shell and Shell Scripting

In this post we will discuss about linux shells and shell scripting so before understanding shell scripting we have to get familiar with following terminologies –

• ·         Kernel
• ·         Shell
• ·         Terminal

What is Kernel?

The kernel is a computer program that is the core of a computer’s operating system, with complete control over everything in the system. It manages following resources of the Linux system –

• ·         File management
• ·         Process management
• ·         I/O management
• ·         Memory management
• ·         Device management etc.

Linus Torvalds has developed Linux OS and he is only responsible for development of linux kernel.

Complete Linux system = Kernel + GNU system utilities and libraries + other management scripts + installation scripts.

 

What is Shell?

A shell is special user program which provide an interface to user to use operating system services. Shell accept human readable commands from user and convert them into something which kernel can understand. It is a command language interpreter that execute commands read from input devices such as keyboards or from files. The shell gets started when the user logs in or start the terminal.

Shell is broadly classified into two categories –

• ·         Command Line Shell
• ·         Graphical shell

 

Command Line Shell

Shell can be accessed by user using a command line interface. A special program called Terminal in linux or Command Prompt in Windows OS is provided to type in the human readable commands such as “cat”, “ls” etc. and then it is being execute.
Working with command line shell is bit difficult for the beginners because it’s hard to memorize so many commands. It is very powerful, it allows user to store commands in a file and execute them together. This way any repetitive task can be easily automated. These files are usually called batch files (.bat files) in Windows and Shell Scripts (.sh files) in Linux systems.

 

Graphical Shells

Graphical shells provide means for manipulating programs based on graphical user interface (GUI), by allowing for operations such as opening, closing, moving and resizing windows, as well as switching focus between windows. Window OS or Linux OS can be considered as good example which provide GUI to user for interacting with program. User do not need to type in command for every actions.

There are several shells are available for Linux systems like –

 BASH (Bourne Again Shell) – It is most widely used shell in Linux systems. It is used as default login shell in Linux systems and in macOS. It can also be installed on Windows OS.

CSH (C Shell) – The C shell’s syntax and usage are very similar to the C programming language.

 KSH (Korn Shell) – The Korn Shell also was the base for the POSIX Shell standard specifications etc.

Each shell does the same job but understand different commands and provide different built in functions.

Shell Scripting

Usually shells are interactive that mean, they accept command as input from users and execute them. However some time we want to execute a bunch of commands routinely, so we have type in all commands each time in terminal.
As shell can also take commands as input from file we can write these commands in a file and can execute them in shell to avoid this repetitive work. These files are called Shell Scripts or Shell Programs. Shell scripts are similar to the batch file in MS-DOS. Each shell script is saved with .sh file extension eg. myscript.sh

A shell script have syntax just like any other programming language. If you have any prior experience with any programming language like Python, C/C++ etc. it would be very easy to get started with it.
A shell script comprises following elements –

• ·         Shell Keywords – if, else, break etc.
• ·         Shell commands – cd, ls, echo, pwd, touch etc.
• ·         Functions
• ·         Control flow – if, then, else, case and shell loops etc.

 

Why do we need shell scripts?

There are many reasons to write shell scripts –

• ·         To avoid repetitive work and automation
• ·         System admins use shell scripting for routine backups
• ·         System monitoring
• ·         Adding new functionality to the shell etc.

Advantages of shell scripts

• ·         The command and syntax are exactly the same as those directly entered in command line, so programmer do not need to switch to entirely different syntax
• ·         Writing shell scripts are much quicker
• ·         Quick start
• ·         Interactive debugging etc.

Disadvantages of shell scripts

• ·         Prone to costly errors, a single mistake can change the command which might be harmful
• ·         Slow execution speed
• ·         Design flaws within the language syntax or implementation
• ·         Not well suited for large and complex task
• ·         Provide minimal data structure unlike other scripting languages. etc

 

creating self-signed x509 certificate in HP UNIX

First of all, install openssl by using below command-

#swinstall –s /tmp/OpenSSL_A.01.00.02r.001_HP-UX_B.11.31_IA_PA.depot

To check installation: -

#swlist | grep –I openssl

Now you can create self-signed x509 certificate by following below steps: -

1.Run the following OpenSSL command to generate your private key and public certificate and Answer the questions and enter the Common Name when prompted.

# openssl req -newkey rsa:2048 -nodes -keyout key.pem -x509 -days 365 -out certificate.pem

2. Review the created certificate:

#openssl x509 -text -noout -in certificate.pem

3. Now you can verify newly created self-signed certificate :-

# openssl verify certificate.pem

Received Output like :-

certificate.pem: C = IN, ST = Himachal Pradesh, L = Shimla, O = SYSNET, OU = JSK, CN = SYSNET, emailAddress = amitesh@jsk.in depth lookup: self-signed certificate

OK

4.Combine your key and certificate in a PKCS#12 (P12) bundle: -

#openssl pkcs12 -inkey key.pem -in certificate.pem -export -out certificate.p12

5.Validate your P2 file.

#openssl pkcs12 -in certificate.p12 -noout -info

for reference
-
# openssl req -newkey rsa:2048 -nodes -keyout key.pem -x509 -days 365 -out certificate.pem
Generating a RSA private key
.................................+++++
............................................................................................................................................+++++
writing new private key to 'key.pem'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:IN
State or Province Name (full name) [Some-State]:Himachal Pradesh
Locality Name (eg, city) []:Shimla
Organization Name (eg, company) [Internet Widgits Pty Ltd]:SYSNET
Organizational Unit Name (eg, section) []:JSK
Common Name (e.g. server FQDN or YOUR name) []:SYSNET
Email Address []:amitesh@jsk.in
#
#
#
# openssl x509 -text -noout -in certificate.pem
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            92:e3:9d:6a:1f:43:0f:5a
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=IN, ST=Himachal Pradesh, L=Shimla, O=SYSNET, OU=JSK, CN=SYSNET/emailAddress=amitesh@jsk.in
        Validity
            Not Before: Mar  7 09:31:38 2020 GMT
            Not After : Mar  7 09:31:38 2021 GMT
        Subject: C=IN, ST=Himachal Pradesh, L=Shimla, O=SYSNET, OU=JSK, CN=SYSNET/emailAddress=amitesh@jsk.in
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:c8:44:4f:4d:7b:dd:73:06:3c:e2:ed:24:22:21:
                    f9:dd:1f:32:49:30:1e:40:81:7a:25:cd:12:0d:27:
                    43:8c:00:50:7b:06:a0:46:ba:55:bc:ee:8d:09:d4:
                    8d:bb:24:32:50:73:7f:72:f0:ae:19:b5:ac:28:5f:
                    52:8a:65:fc:86:84:26:83:53:16:25:56:d9:15:33:
                    b3:a6:56:40:57:88:a0:4b:83:33:43:f4:e5:89:99:
                    13:52:ac:b8:65:28:32:88:a2:84:4a:40:58:2a:88:
                    f9:c0:90:38:02:7d:17:4f:73:22:62:8c:21:77:d1:
                    28:71:f5:e5:77:e3:0a:c7:0f:44:4b:8b:68:52:0c:
                    b6:27:7c:8d:24:29:df:6d:1e:be:99:e9:00:f9:18:
                    09:0d:e0:0b:be:95:60:f6:bc:0d:28:a6:f5:00:70:
                    54:db:fb:4c:90:e8:ce:71:06:03:d7:7e:11:d1:20:
                    67:ff:66:60:ac:79:e4:53:86:6b:89:08:1e:81:04:
                    bf:69:5b:82:c0:9e:47:dd:9c:8e:ed:de:1a:8d:9d:
                    37:b2:84:4f:64:e1:3a:5e:22:32:fe:59:5e:d8:1a:
                    dd:86:f6:29:f0:cd:8f:37:ca:13:d9:c2:7c:09:c8:
                    50:42:b8:02:8d:19:3a:5e:66:27:a6:54:eb:89:74:
                    a2:29
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Key Identifier:
                1A:5B:3D:07:5C:A8:26:A0:97:FE:BA:2E:1B:10:E6:0A:5B:42:DC:C8
            X509v3 Authority Key Identifier:
                keyid:1A:5B:3D:07:5C:A8:26:A0:97:FE:BA:2E:1B:10:E6:0A:5B:42:DC:C8

            X509v3 Basic Constraints:
                CA:TRUE
    Signature Algorithm: sha256WithRSAEncryption
         65:3e:70:1e:02:47:7b:1f:3b:8e:ab:90:17:1f:d8:05:29:7e:
         72:54:92:b0:ac:0e:84:23:0b:be:c4:65:a3:d4:3c:9b:9f:65:
         cf:70:0f:ec:20:68:bd:2d:45:27:1b:e9:0b:d7:d0:9f:8f:9c:
         0f:1a:0f:f6:ec:1e:58:64:07:80:ef:81:54:b0:d8:5f:fd:37:
         b4:b8:3c:df:56:13:29:e0:5c:b1:cd:27:9d:f6:d8:71:4d:8b:
         ae:18:21:0a:d6:56:5d:c5:b5:80:ad:76:cc:23:87:9c:a4:93:
         79:82:b1:c2:d0:29:ff:b1:04:d9:81:83:4c:de:81:cc:d9:e0:
         54:7f:43:eb:3b:11:f4:58:ea:7c:9a:ca:ea:cd:a3:b6:4b:85:
         1a:20:d2:33:2c:7c:bd:10:0e:4f:9c:7d:1e:76:2e:5a:fb:c4:
         f7:66:e9:08:c2:4b:21:44:14:e6:8d:23:36:18:b8:2c:68:e2:
         c8:dc:59:b8:fd:e1:e9:3d:5a:73:6d:69:cb:43:5b:f2:92:17:
         74:86:3e:4c:f4:c7:d5:cc:89:ec:c9:f1:51:d9:6b:4c:8c:5b:
         08:34:36:a3:04:0d:e7:57:65:4f:93:7c:84:8f:6f:36:73:22:
         9d:63:80:f7:e2:ed:60:c0:08:0f:11:c0:ea:2d:3e:7d:a6:8f:
         e7:fa:73:56
#
#
## openssl verify certificate.pem
certificate.pem: C = IN, ST = Himachal Pradesh, L = Shimla, O = SYSNET, OU = JSK, CN = SYSNET, emailAddress = amitesh@jsk.in
error 18 at 0 depth lookup:self signed certificate
OK
# openssl pkcs12 -inkey key.pem -in certificate.pem -export -out certificate.p12
Enter Export Password:
Verifying - Enter Export Password:
#
#
# openssl pkcs12 -in certificate.p12 -noout -info
Enter Import Password:
MAC Iteration 2048
MAC verified OK
PKCS7 Encrypted data: pbeWithSHA1And40BitRC2-CBC, Iteration 2048
Certificate bag
PKCS7 Data
Shrouded Keybag: pbeWithSHA1And3-KeyTripleDES-CBC, Iteration 2048
#
#

Some Important Facts about LVM

Suppose you have two volume groups like vg01 and vg02 and both sizes 5 GB. And vgo1 have no logical volumes but vg02 have two logical volumes.

[root@rhelnode1 /]# vgs

  VG   #PV #LV #SN Attr   VSize  VFree

  rhel   1   2   0 wz--n- 15.63g 4.00m

  vg01   1   0   0 wz--n- 5.00g  4.99g

  vg02   2   2   0 wz--n-  5.00g 0.49g

So, you can split both vgs to utilize the space of vgo1.

[root@rhelnode1 /]# vgsplit vg01 vg02 /dev/sdb

  Existing volume group "vg02" successfully split from "vg01"

Here you have to specify device path of vg01 (/dev/sdb).

Now you can check vgo1 space splits into vg02.

[root@rhelnode1 /]# vgs

  VG   #PV #LV #SN Attr   VSize  VFree

  rhel   1   2   0 wz--n- 15.63g 4.00m

  vg02   2   2   0 wz--n-  9.99g 5.49g

----------------------------------------------------------------

Extending filesystem in Linux 7

We have a volumegroup vg01 which have free space then and while creating logical volume we had taken less space.So now we can extend logical volume accordingly.

[root@rhelnode1 /]# vgs

  VG   #PV #LV #SN Attr   VSize  VFree

  rhel   1   4   0 wz--n- 22.01g    4.00m

  vg01   1   1   0 wz--n-  2.00g 1020.00m

[root@rhelnode1 /]# df -h

Filesystem               Size  Used Avail Use% Mounted on

/dev/mapper/vg01-backup 1014M   33M  982M   4% /backup

Here we can see vg01 have 1GB free space. So we can extend logical volume named backup by using below command-

[root@rhelnode1 /]# lvextend -L 1.5G /dev/vg01/backup

  Size of logical volume vg01/backup changed from 1.00 GiB (256 extents) to 1.50 GiB (384 extents).

  Logical volume vg01/backup successfully resized.

Now run below command to extand size of logical volume-

[root@rhelnode1 /]# xfs_growfs /dev/vg01/backup

meta-data=/dev/mapper/vg01-backup isize=512    agcount=4, agsize=65536 blks

         =                       sectsz=512   attr=2, projid32bit=1

         =                       crc=1        finobt=0 spinodes=0

data     =                       bsize=4096   blocks=262144, imaxpct=25

         =                       sunit=0      swidth=0 blks

naming   =version 2              bsize=4096   ascii-ci=0 ftype=1

log      =internal               bsize=4096   blocks=2560, version=2

         =                       sectsz=512   sunit=0 blks, lazy-count=1

realtime =none                   extsz=4096   blocks=0, rtextents=0

data blocks changed from 262144 to 393216

After that, we can check our logical volume have updated space.

[root@rhelnode1 /]# df -h

Filesystem               Size  Used Avail Use% Mounted on

/dev/mapper/vg01-backup  1.5G   33M  1.5G   3% /backup

----------------------------------------------------------------

Suppose you have 2 physical volumes in a volume group. But 1 physical volume is sufficient then you can remove another physical volume-

[root@rhelnode1 /]# pvs

  PV         VG   Fmt  Attr PSize  PFree  

  /dev/sda3  rhel lvm2 a--  15.63g   4.00m

  /dev/sdb   vg02 lvm2 a--   5.00g   5.00g

  /dev/sdc   vg02 lvm2 a--   5.00g 508.00m

[root@rhelnode1 /]# vgdisplay vg02

  --- Volume group ---

  VG Name               vg02

  System ID             

  Format                lvm2

  Metadata Areas        2

  Metadata Sequence No  5

  VG Access             read/write

  VG Status             resizable

  MAX LV                0

  Cur LV                2

  Open LV               2

  Max PV                0

  Cur PV                2

  Act PV                2

  VG Size               9.99 GiB

  PE Size               4.00 MiB

  Total PE              2558

  Alloc PE / Size       1152 / 4.50 GiB

  Free  PE / Size       1406 / 5.49 GiB

  VG UUID               vovDvR-WQUV-M0sE-OAZG-emAd-Vw77-Nzdkdv

Here you can see /dev/sdb pv is totally free.then you can reduce your vg by using below command -

[root@rhelnode1 /]# vgreduce  vg02 /dev/sdb

  Removed "/dev/sdb" from volume group "vg02"

[root@rhelnode1 /]# 

Rename of Logical volumes and volume groups -

First of all, you have to change request mode of volume group by using below command :-

[root@rhelnode1 /]# lvchange -a n vg02

[root@rhelnode1 /]# lvs

  LV     VG   Attr       LSize Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert

  home   rhel -wi-ao---- 7.81g                                                    

  var    rhel -wi-ao---- 7.81g                                                    

  backup vg02 -wi------- 2.50g                                                    

  sync   vg02 -wi------- 2.00g 

[root@rhelnode1 /]# lvrename /dev/mapper/vg02/sync /dev/mapper/vg02-shared

  Renamed "sync" to "vg02-shared" in volume group "vg02"

Hint - current lvname with path  =    /dev/mapper/vg02/sync

          New lvname with path        =    /dev/mapper/vg02-shared

[root@rhelnode1 /]# vgrename /dev/mapper/vg02 /dev/mapper/vg00

  Volume group "vg02" successfully renamed to "vg00"

Hint-  Current vgname with path     =       /dev/mapper/vg02

          New vgname with path            =        /dev/mapper/vg00

[root@rhelnode1 /]# lvs

  LV          VG   Attr       LSize Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert

  home        rhel -wi-ao---- 7.81g                                                    

  var         rhel -wi-ao---- 7.81g                                                    

  backup      vg00 -wi------- 2.50g                                                    

  vg02-shared vg00 -wi------- 2.00g                                                    

[root@rhelnode1 /]# 

Now you can change request mode by using below command : -

[root@rhelnode1 /]# vgchange -a y vg00

  Volume group "vg00" successfully changed.

Now you can mount your mount point and change also /etc/fstab entries.

How to check your volume group is clustered or non-clustered?

[root@rhelnode1 /]# vgdisplay vg00 --config 'global {locking_type=0}'

  WARNING: Locking disabled. Be careful! This could corrupt your metadata.

  --- Volume group ---

    Clustered                yes

If you want convert clustered volume into non clustered volume then use below command -

[root@rhelnode1 /]# vgchange -cn vg00 --config 'global {locking_type=0}'

  WARNING: Locking disabled. Be careful! This could corrupt your metadata.

  Volume group "vg00" is changed to not clustered.

[root@rhelnode1 /]# 

Now successfully changed.

Creating File System in Linux using LVM

File System using LVM in Linux

If you want to create a file system in linux using Logical volume manager.
Then this post will help you.

First of all, create a .vhd (virtual hard disk) using your storage system, size as you required like 5 GB.
Then follow below steps to create a file system :-


1.Then login your server and find newly created disk by using below command :-

[root@rhelnode1 /]# fdisk -l

If you can not find recently created 5GB disk. Then follow below easy steps to scan your storage path like HBA or SCSI :-

A. Find you scsi hosts

[root@rhelnode1 /]# ls /sys/class/scsi_host
host0  host1  host2

B.Scan each host ports :-

[root@rhelnode1 /]# echo "- - -"> /sys/class/scsi_host/host0/scan
[root@rhelnode1 /]# echo "- - -"> /sys/class/scsi_host/host1/scan
[root@rhelnode1 /]# echo "- - -"> /sys/class/scsi_host/host2/scan

C.now run you can run again fdisk command :-

[root@rhelnode1 /]# fdisk -l 2> /dev/null | egrep '^Disk'| grep -v mapper| grep -v identifier
Disk /dev/sda: 48.3 GB, 48318382080 bytes, 94371840 sectors
Disk label type: dos
Disk /dev/sdb: 5368 MB, 5368709120 bytes, 10485760 sectors
Disk /dev/sdc: 5368 MB, 5368709120 bytes, 10485760 sectors

D. To see block device mapping:-

[root@rhelnode1 /]# lsblk

NAME            MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT

fd0               2:0    1    4K  0 disk

sda               8:0    0  127G  0 disk

├─sda1            8:1    0  700M  0 part /boot

├─sda2            8:2    0  4.9G  0 part

│ └─vg01-swap   253:1    0  4.9G  0 lvm  [SWAP]

└─sda3            8:3    0 49.3G  0 part

  ├─rhel-root   253:0    0 13.7G  0 lvm  /

  ├─rhel-home   253:2    0 11.7G  0 lvm  /home

  ├─rhel-var    253:3    0   12G  0 lvm  /var

  └─rhel-backup 253:4    0 11.8G  0 lvm  /backup

sr0              11:0    1 1024M  0 rom

[root@amiteshjsk /]#

2.Now create a physical volume 

[root@rhelnode1 /]# pvcreate /dev/sdc
  Physical volume "/dev/sdc" successfully created.

3.Now create volume group and display you vg details

[root@rhelnode1 /]# vgcreate vg02 /dev/sdc
  Volume group "vg02" successfully created
[root@rhelnode1 /]# vgdisplay vg02
  --- Volume group ---
  VG Name               vg02
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  1
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                0
  Open LV               0
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               5.00 GiB
  PE Size               4.00 MiB
  Total PE              1279
  Alloc PE / Size       0 / 0   
  Free  PE / Size       1279 / 5.00 GiB
  VG UUID               vovDvR-WQUV-M0sE-OAZG-emAd-Vw77-Nzdkdv
   
[root@rhelnode1 /]# 

4. Now you can create desired size(Max 5GB) logical volumes.

So here I want create two logical volumes (2GB +2.5GB)

[root@rhelnode1 /]# lvcreate -L 2G -n sync vg02

  Logical volume "sync" created.

[root@rhelnode1 /]# lvcreate -L 2.5G -n backup vg02

  Logical volume "backup" created.

[root@rhelnode1 /]# 

5. Now format disk using xfs filesystem

[root@rhelnode1 /]# mkfs.xfs /dev/vg02/sync

meta-data=/dev/vg02/sync         isize=512    agcount=4, agsize=131072 blks

         =                       sectsz=512   attr=2, projid32bit=1

         =                       crc=1        finobt=0, sparse=0

data     =                       bsize=4096   blocks=524288, imaxpct=25

         =                       sunit=0      swidth=0 blks

naming   =version 2              bsize=4096   ascii-ci=0 ftype=1

log      =internal log           bsize=4096   blocks=2560, version=2

         =                       sectsz=512   sunit=0 blks, lazy-count=1

realtime =none                   extsz=4096   blocks=0, rtextents=0

[root@rhelnode1 /]# mkfs.xfs /dev/vg02/backup

meta-data=/dev/vg02/backup       isize=512    agcount=4, agsize=163840 blks

         =                       sectsz=512   attr=2, projid32bit=1

         =                       crc=1        finobt=0, sparse=0

data     =                       bsize=4096   blocks=655360, imaxpct=25

         =                       sunit=0      swidth=0 blks

naming   =version 2              bsize=4096   ascii-ci=0 ftype=1

log      =internal log           bsize=4096   blocks=2560, version=2

         =                       sectsz=512   sunit=0 blks, lazy-count=1

realtime =none                   extsz=4096   blocks=0, rtextents=0

[root@rhelnode1 /]# 

6. Now you can mount your logical volumes after creating directories 

[root@rhelnode1 /]# mkdir backup sync

[root@rhelnode1 /]# mount /dev/vg02/sync /sync

[root@rhelnode1 /]# mount /dev/vg02/backup /backup

[root@rhelnode1 /]# df -h

Filesystem               Size  Used Avail Use% Mounted on

/dev/sda2                 21G  3.2G   18G  16% /

devtmpfs                 1.2G     0  1.2G   0% /dev

tmpfs                    1.3G  144K  1.3G   1% /dev/shm

tmpfs                    1.3G  9.0M  1.2G   1% /run

tmpfs                    1.3G     0  1.3G   0% /sys/fs/cgroup

/dev/mapper/rhel-var     7.9G  1.3G  6.6G  16% /var

tmpfs                    247M  8.0K  247M   1% /run/user/0

/dev/mapper/vg01-data    2.0G   33M  2.0G   2% /data

/dev/mapper/vg02-sync    2.0G   33M  2.0G   2% /sync

/dev/mapper/vg02-backup  2.5G   33M  2.5G   2% /backup

[root@rhelnode1 /]# 

7. Now make entries in /etc/fstab file for auto mount after reboot-

[root@rhelnode1 /]# vi /etc/fstab

/dev/mapper/vg01-data     /data     xfs   defaults      0 2

/dev/mapper/vg02-sync     /sync     xfs   defaults      0 2

/dev/mapper/vg02-backup   /backup   xfs   defaults      0 2

Note:-

/etc/fstab contains 6 parameters per row. Each row represents one file system details. They are as below:

1.    Volume                 /dev/mapper/vg02-data

2.    Mount point          /data

3.    File system type   xfs

4.    Options                defaults

5.    Dump                  1,0 (for backup dump in failure) 

6.    Pass                     0,1,2 (for fsck sequence top or bottom)