Sei sulla pagina 1di 184

Linux Server Administration Web Development Pentesting

Tech Tips Contact

SEARCH

Search Search

LINUX

Bash Script Step By Step,


You Will Love It
February 7, 2017 admin 3 Comments

Today we are going to talk about bash script


or shell scripting actually, they are called
shell scripts in general but we are going to
call them bash scripts because we are going to
use bash among the other Linux shells. There
Linux Server Administration Web Development Pentesting

Tech Tips Contact

Bash shebang

Setting permission

Print messages

using variables

Environment variables

User variables

Command substitution

Math calculation

if-then statement

Nested if- Statement

Numeric Comparisons

String Comparisons

File Comparisons
Linux Server Administration Web Development Pentesting

Tech Tips Contact

The key to bash script is the ability to enter


multiple Commands and deal with the results
from each command, even possibly passing the
results of one command to another. The shell
allows you to run multiple commands in a
single step.

If you want to run two commands together or


more, you can type them on the same line,
separated with a semicolon

pwd ; whoami

Actually what you have typed is a bash script!!

This simple script uses just two bash shell


commands.

The pwd command runs rst, displaying the


current working directory followed by the
output of the whoami command, showing who
is currently logged in user.

Using this technique, you can string together as


many commands as you wish, maximum
Linux Server Administration Web Development Pentesting

Tech Tips Contact

Well, thats ne but it has a problem that you


must enter the command at the command
prompt every time you want to run it ok, what
about we combine the commands into a le.
And when we need to run those commands we
run that le only. This is called the bash script

Now create an empty le using touch command


as we discussed that in a previous post about
Linux commands. The rst line we should
dene which shell we will use as we know there
are many shells on Linux bash is one of them

Bash Script Shebang


Here we will write bash script and in order to do
that we will type

#!/bin/bash

In a normal bash script line, the pound sign (#)


is used as a comment line which is not
processed by the shell. However, the rst line
Linux Server Administration Web Development Pentesting

Tech Tips Contact

And the shell commands are entered one per


line followed by enter and you can write a
comment by adding the pound sign at the
beginning of the le like this

#!/bin/bash

# This is a comment

pwd

whoami

You can use the semicolon and put multiple


commands on the same

Line if you want to, but in bash script, you can


list commands on separate lines, and this to
make it simpler to read them later. The shell
will process them any way.

Set Script Permission


Save the le, you are almost nished. All you
need to do now is to set that le to be
executable in order to be able to run it
Linux Server Administration Web Development Pentesting

Tech Tips Contact

chmod +x ./myscript

Then try run it by just type it in the shell

./myscript

And Yes it is executed

Print Messages
As we know from other posts, printing text is
done by echo command

So take your knowledge to bash script and


apply it

Now we edit our le and type this

#!/bin/bash
Linux Server Administration Web Development Pentesting

Tech Tips Contact

whoami

Look at the output

Perfect! Now we can run commands and display


text using echo command

If you dont know echo command or how to edit


a le I recommend you to view previous articles
about basic Linux commands

Using Variables
Variables allow you to store information in the
bash script for use with other commands in the
script.

Running individual commands from the bash


script is good, but this has its limitations.

There are two types of variables you can use in


your bash script
Linux Server Administration Web Development Pentesting

Tech Tips Contact

in your shell commands to process information.


You can do this by using environment variables.

We talked about environment variables on


another post you can check it.

#!/bin/bash

# display user home

echo "Home for the current user is: $HOME"

Notice that we were able to place the $HOME


system variable in the double quotation marks
in the rst string, and the shell script still know
it

What if we want to print the dollar sign itself?

echo "I have $1 in my pocket"

The script sees a dollar sign within quotes; it


Linux Server Administration Web Development Pentesting

Tech Tips Contact

By using escape characters back slash \ before


the dollar sign like this

echo "I have \$1 in my pocket"

Now the bash script will print the dollar sign as


it is

User variables
In addition to the environment variables, a bash
script allows you to set and use your own
variables in the script.

Variables dened in the shell script maintain


their values till bash script execution nished.

Like system variables, user variables can be


referenced using the dollar sign

#!/bin/bash

# testing variables
Linux Server Administration Web Development Pentesting

Tech Tips Contact

chmod +x myscript

./myscript

Command substitution
One of the best features of bash scripts is the
ability to extract information from the output of
a command and assign it to a variable so you
can use that value anywhere in your script

There are two ways to do that

The backtick character `


The $() format

Make sure when you type backtick character it


is not the single quotation mark.

You must surround the entire command line


command with two backtick characters like this

mydir=`pwd`
Linux Server Administration Web Development Pentesting

Tech Tips Contact

#!/bin/bash

mydir=$(pwd)

echo $mydir

The output of the command will be stored in


that variable called mydir.

Math calculation
You can perform basic math calculations using
$[] format

#!/bin/bash
var1=$(( 5 + 5 ))

echo $var1

var2=$(( $var1 * 2 ))

echo $var2

Just that easy


Linux Server Administration Web Development Pentesting

Tech Tips Contact

Bash script requires some sort of logic ow


control between the commands in the script.
Like if the value is greater than 5 do that else
do whatever you can imagine any login you
want.

The most basic structure of if-then statement is


like this

if command

then

commands

and here is an example

#!/bin/bash

if pwd

then

echo "It works"

fi
Linux Server Administration Web Development Pentesting

Tech Tips Contact

May be searching for a specic user in the


users /etc/passwd and if it exists it prints that
the user is present

#!/bin/bash

user=likegeeks

if grep $user /etc/passwd

then

echo "The user $user Exists"

fi

We use grep command to search for the user in


/etc/passwd le. You can check our tutorial
about basic Linux commands if you dont know
grep command.

If the user exists the bash script will print the


message.
Linux Server Administration Web Development Pentesting

Tech Tips Contact

if-then-else Statement
The if-then-else statement looks like this

if command

then

commands

else

commands

if the rst command runs and returns with a


zero which means success it will not hit the
commands after the else statement, otherwise,
if the if statement returns non-zero which
means the statement condition not correct, At
this case the shell will hit the commands after
else statement.

#!/bin/bash

user=anotherUser
Linux Server Administration Web Development Pentesting

Tech Tips Contact

echo "The user $user doesnt exist"

fi

We are doing good till now, keep moving

Now, what if we have more else statements,


like if user correct print this else if it is some
other print this else print another thing?

Well that is easy also we can achieve by


nesting if statements like this

if command1

then

commands

elif command2

then

commands
Linux Server Administration Web Development Pentesting

Tech Tips Contact

commands after it else if none of those return


zero it will execute the last commands only.

#!/bin/bash
user=anotherUser
if grep $user /etc/passwd

then

echo "The user $user Exists"

elif ls /home

echo "The user doesnt exist but anyway there is a direc

fi

You can imagine any scenario here may be if


the user doesnt exist create it using the
useradd command or do anything else.

Numeric Comparisons
You can perform numeric comparison between
two numeric values using numeric comparison
checks as on this list

n1 -eq n2 Checks if n1 is equal to n2


Linux Server Administration Web Development Pentesting

Tech Tips Contact

n1 -le n2 Checks if n1 is less than or equal to n2

n1 -lt n2 Checks if n1 is less than n2

n1 -ne n2 Checks if n1 is not equal to n2

As an example, we will try one of them and the


rest is the same

Note that the comparison statement is in


square brackets as shown.

#!/bin/bash

val1=6

if [ $val1 -gt 5 ]

then

echo "The test value $value1 is greater than 5"

else

echo "The test value $value1 is not greater than 5"

fi

The val1 is greater than 5 so it will run the rst


Linux Server Administration Web Development Pentesting

Tech Tips Contact

The comparison functions you can use to


evaluate two string values are

str1 = str2 Checks if str1 is the same as string


str2

str1 != str2 Checks if str1 is not the same as


str2

str1 < str2 Checks if str1 is less than str2

str1 > str2 Checks if str1 is greater than str2

-n str1 Checks if str1 has a length greater than


zero

-z str1 Checks if str1 has a length of zero

We can apply string comparison on our


example

#!/bin/bash

user ="likegeeks"

if [$user = $USER]

then
Linux Server Administration Web Development Pentesting

Tech Tips Contact

One tricky note about the greater than and


less than for string comparisons MUST be
escaped with the back slash because by just
using the greater-than symbol itself in the
script, no errors are generated, but the results
are wrong. The script interpreted the
greater-than symbol as an output redirection.
So you should do it like that

#!/bin/bash

val1=text

val2="another text"

if [ $val1 \> "$val2" ]

then
echo "$val1 is greater than $val2"

else

echo "$val1 is less than $val2"

fi
Linux Server Administration Web Development Pentesting

Tech Tips Contact

Just little x is to wrap the $vals with double


quotation, forcing it to stay as one string like
this

#!/bin/bash
val1=text

val2="another text"

if [ $val1 \> "$val2" ]

then

echo "$val1 is greater than $val2"

else

echo "$val1 is less than $val2"

fi

Last tricky note about greater than and less


than for string comparisons is when working
with uppercase and lowercase letters. The
sort command handles uppercase letters
opposite to the way the test conditions
consider them
Linux Server Administration Web Development Pentesting

Tech Tips Contact

then

echo "$val1 is greater than $val2"

else

echo "$val1 is less than $val2"

fi

sort myfile

likegeeks

Likegeeks

Capitalized letters are treated as less than


lowercase letters in test comparisons. However,
the sort command does exactly the opposite.

String test comparisons use standard ASCII


Linux Server Administration Web Development Pentesting

Tech Tips Contact

dened for the system locale language


settings.

File Comparisons
This is the best and most powerful and most
used comparison in bash scripting there are
many le comparisons that you can do in bash
script

-d le Checks if le exists and is a directory

-e le Checks if le exists

-f le Checks if le exists and is a le

-r le Checks if le exists and is readable

-s le Checks if le exists and is not empty

-w le Checks if le exists and is writable

-x le Checks if le exists and is executable

le1 -nt le2 Checks if le1 is newer than le2


Linux Server Administration Web Development Pentesting

Tech Tips Contact

-G le Checks if le exists and the default group


is the same as the current user

As they imply, you will never forget them.

Lets pick one of them and take it as an


example

#!/bin/bash

mydir=/home/likegeeks

if [ -d $mydir ]

then

echo "The $mydir directory exists"

cd $ mydir

ls

else

echo "The $mydir directory does not exist"

fi
Linux Server Administration Web Development Pentesting

Tech Tips Contact

There are some other advanced if-then features


but lets make it on another post.

Thats for now. I hope you enjoy it and keep


practicing more and more.

Wait for another tutorial about bash scripting so


stay tuned.

LINUX

Bash Scripting The


Awesome Guide Part2
February 9, 2017 admin 4 Comments

In the previous post, we talked about how to


Linux Server Administration Web Development Pentesting

Tech Tips Contact

demonstrates for loop, while in bash scripts

we will discuss the following:

for command

Iterating over simple values

Iterating over complex values

Reading values from a command

The eld separator

Iterating over directory les

for Command C-Style

The while Command

Nesting Loops

Looping on File Data

Controlling the Loop

The break command


Linux Server Administration Web Development Pentesting

Tech Tips Contact

for Command
The bash shell provides the for command to
allow you to create a loop that iterates through
a series of values. This is the basic format of
the bash shell for command

for var in list

do

commands

done

In each iteration, the variable var contains the


current value in the list. The rst iteration uses
the rst item in the list; the second iteration
contains the second item, and so on until the
end of the list items

Iterating over simple


values
The most basic use of the for command in bash
Linux Server Administration Web Development Pentesting

Tech Tips Contact

do

echo The $var item

done

As you can see from the output the $var


variable is changed on every loop cycle till the
last item on the list.

Notice that the $var variable retained its value


and allowed us to change the value and use it
outside of the for command loop, like any
variable.

Iterating over complex


values
Your list maybe contains some comma or two
words but you want to deal with them as one
item on the list.
Linux Server Administration Web Development Pentesting

Tech Tips Contact

echo "This is: $var"

done

We play nice till now, always we do. Just keep


reading and practicing.

Reading values from a


command
Another way to a list is to use the output of a
command. You use command substitution to
execute any command that produces output.

#!/bin/bash

file="myfile"

for var in $(cat $file)

do

echo " $var"

done
Linux Server Administration Web Development Pentesting

Tech Tips Contact

Notice that our le contains one word per line,


not separated by spaces.

The for command still iterates through the


output of the cat command one line at a time,
assuming that each line has one word.
However, this doesnt solve the problem of
having spaces in data.

If you list that contains words with spaces in it,


the for command still takes each word as a
separate value. Theres a reason for this, which
we look at now.

The eld separator


The cause of this problem is the special
environment variable IFS, called the internal
eld separator. By default, the bash shell
considers the following characters as eld
Linux Server Administration Web Development Pentesting

Tech Tips Contact

the data, it assumes that youre starting a new


data eld in the list.

To solve this problem, you can temporarily


change the IFS environment variable values in
your bash script suppose that you want to
separate by new lines so it will be like this

IFS=$'\n'

So after you add this to your bash script it will


ignore spaces and tabs and consider new lines
as a separator.

#!/bin/bash

file="/etc/passwd"

IFS=$'\n'

for var in $(cat $file)

do

echo " $var"

done

You got it. Bash scripting is easy just little


attention
Linux Server Administration Web Development Pentesting

Tech Tips Contact

In this case, the separator is colon like the case


of /etc/passwd le which contains the users
information you can assign it like this

IFS=:

How bash scripting is awesome?

Iterating over directory


les
One of the most common things when using for
loop in bash scripting is iterate over les in a
directory and deal with them.

For example, we want to list the le inside


/home directory so the code will be like this

#!/bin/bash

for file in /home/likegeeks/*

do

if [ -d "$file" ]
Linux Server Administration Web Development Pentesting

Tech Tips Contact

echo "$file is a file"

fi

done

From the previous post, you should know the if


statement and how to dierentiate between
les and folders, so if you dont know I
recommend you to review it bash script step
by step.

Here we use wildcard character which is


asterisk * and this is called in bash scripting le
globbing which is a process of producing
lenames automatically that matches the
wildcard character in our case asterisk means
Linux Server Administration Web Development Pentesting

Tech Tips Contact

Sure enough, it lists all les and directories in


that folder

for Command C-Style


If you know c language you may found that the
for loop here is weird because you are familiar
with this syntax

for (i = 0; i < 10; i++)

printf(number is %d\n, i);

The bash scripting also supports a version of


the for loop that looks similar to the C-style for
loop with little dierence heres the syntax.

for (( variable assignment ; condition ; iteration


process ))

So it looks like this


Linux Server Administration Web Development Pentesting

Tech Tips Contact

for (( i=1; i <= 10; i++ ))

do

echo "number is $i"

done

And this is the output

The while Command


The for loop is not the only way of looping in
bash scripting. The while command allows you
to dene a command to test and then loop
through a set of commands as long as the
dened test command returns a zero exit status
which means success. It tests the test
command at the start of each iteration. When
the test command returns a nonzero exit status
Linux Server Administration Web Development Pentesting

Tech Tips Contact

while test command

do

other commands

done

and here is an example

#!/bin/bash

var1=5

while [ $var1 -gt 0 ]

do

echo $var1

var1=$[ $var1 - 1 ]

done

The script is simple; it starts with while


command to check if var1 is greater than zero
then the loop will run and the var1 value will be
decreased every time by 1 and on ever loop
iteration it will print the value of var1, Once the
var1 value is zero the loop will exit.
Linux Server Administration Web Development Pentesting

Tech Tips Contact

If we dont decrease the value of var1 it will be


the same value and the loop will be innite.

Nesting Loops
A loop statement can use any other type of
command within the loop, including other loop
commands. This is called a nested loop.

Heres an example of nested loops.

#!/bin/bash

for (( a = 1; a <= 3; a++ ))

do

echo "Start $a:"

for (( b = 1; b <= 3; b++ ))


do

echo " Inner loop: $b"

done

done

As you can see from the results the outer loop


hits rst then goes into the inner loop and
completes it and go back to the outer loop and
Linux Server Administration Web Development Pentesting

Tech Tips Contact

Looping on File Data


This is the most common usage for the for
loopin bash scripting.

We can iterate over le content for example


/etc/passwd le and see what we will get

#!/bin/bash

IFS=$'\n'

for entry in $(cat /etc/passwd)

do

echo "Values in $entry "

IFS=:

for value in $entry

do

echo " $value"

done

done
Linux Server Administration Web Development Pentesting

Tech Tips Contact

You can apply this idea when you have CSV or


any comma separated values le or whatever.
The idea is the same; you just will change the
separator to t your needs.

Controlling the Loop


Maybe after the loop starts you want to stop at
a specic value, will you wait until the loop is
nished? Of course not, there are two
commands help us in this.

break command
Linux Server Administration Web Development Pentesting

Tech Tips Contact

any type of loop, including while and until


loops.

#!/bin/bash

for var1 in 1 2 3 4 5 6 7 8 9 10

do

if [ $var1 -eq 5 ]

then

break

fi

echo "Number: $var1"

done

The for loop should normally have iterated


through all the values in the list.

However, the shell executed the break


command, which stopped the for loop
Linux Server Administration Web Development Pentesting

Tech Tips Contact

do

if [ $var1 -eq 5 ]

then
break
fi

echo "Iteration: $var1"

var1=$(( $var1 + 1 ))

done

The while loop terminated when the if-then


condition was met, executing the break
command.

The continue command


The continue command is a way to prematurely
stop processing commands inside of a loop but
not terminate the loop completely.
Linux Server Administration Web Development Pentesting

Tech Tips Contact

do

if [ $var1 -gt 5 ] && [ $var1 -lt 10 ]

then

continue
fi

echo "Iteration number: $var1"

done

When the conditions of the if-then statement


are met (the value is greater than 5 and less
than 10), the shell executes the continue
command, which skips the rest of the
commands in the loop, but keeps the loop
going.

Processing the Output of


Linux Server Administration Web Development Pentesting

Tech Tips Contact

the done command.

So instead of displaying the results on screen,


the shell redirects the results of the for
command to the le or whatever

#!/bin/bash

for (( a = 1; a < 10; a++ ))

do

echo "Number is $a"

done > myfile.txt

echo "finished."

The shell creates the le myle.txt and


redirects the output of the for command to the
le. And if we check that le we will nd our
loop output inside it.

Lets employ out bash scripting knowledge in


something useful.
Linux Server Administration Web Development Pentesting

Tech Tips Contact

available on your system for you to use, just


scan all the folders in the PATH environment
variable. We discussed for loop and if
statements and le separator so our toolset is
ready. Lets combine them together and make
something pretty

#!/bin/bash

IFS=:

for folder in $PATH

do

echo "$folder:"

for file in $folder/*

do

if [ -x $file ]

then

echo " $file"

fi

done

done

This is just awesome. We were able to get all


the executables on the system that we can run
Linux Server Administration Web Development Pentesting

Tech Tips Contact

Now nothing stops you except your


imagination. And this is the beauty of bash
scripting

This for now, I hope you enjoy the article and


learn a new thing or at least review your
knowledge if you forget it. My last word is
keepreading and practicing.

Thanks
Linux Server Administration Web Development Pentesting

Tech Tips Contact

LINUX

Linux Bash Scripting The


Awesome Guide Part3
February 12, 2017 admin 2 Comments

So far youve seen how to write Linux bash


scripts that interact with data, variables, and
les and how to control the ow of the bash
script Today we will continue our series about
Linux bash scripting.

I recommend you to review the previous posts


if you want to know what we are talking about.

bash scripting part1

bash scripting part2

Our main points are


Linux Server Administration Web Development Pentesting

Tech Tips Contact

Grabbing all parameters

Shift command

Bash scripting options

Separating options from parameters

Processing options with values

Standardizing Options

Getting User Input

Reading password

Reading from a le

Today we will know how to retrieve input from


the user and deal with that input so our script
becomes more interactive.

The most basic method of passing data to your


shell script is to use command line parameters.
Linux Server Administration Web Development Pentesting

Tech Tips Contact

This example passes two command line


parameters (10 and 20) to the script. So how to
read those parameters in our bash script?

Reading parameters
The bash shell assigns special variables, called
positional parameters, to all of the command
line parameters entered

$0 being the scripts name


$1 being the rst parameter
$2 being the second parameter, and so on,
up to $9 for the ninth parameter

Heres a simple example how to use command


line parameter in a shell script

#!/bin/bash

echo $0

echo $1

echo $2

echo $3
Linux Server Administration Web Development Pentesting

Tech Tips Contact

If you need to enter more command line


parameters, each parameter must be
separated by a space

Here is another example of how we can use two


parameters and calculate the sum of them

#!/bin/bash

total=$[ $1 + $2 ]

echo The first parameter is $1.

echo The second parameter is $2.

echo The sum is $total.

The parameters is not restricted to numbers it


could be strings like this

#!/bin/bash
Linux Server Administration Web Development Pentesting

Tech Tips Contact

And the result is as expected.

But what if our parameter contains a space and


we want to pass it as one value? I guess you
know the answer from the previous posts. The
answer is to use quotations.

If your script needs more than nine parameters,


you must use braces around the variable
number, such as ${10}.

Testing parameters
If the script is run without the parameters and
your code expecting it, youll get an error
message from your script.

So Always check your parameters to make sure


that they exist

#!/bin/bash

if [ -n "$1" ]
Linux Server Administration Web Development Pentesting

Tech Tips Contact

fi

Counting parameters
You can count how many parameters were
entered on the command line. The bash shell
provides a special variable for this purpose.

The special $# variable contains the number of


the command line.

#!/bin/bash

echo There were $# parameters passed.

./myscript 1 2 3 4 5

How awesome is Linux bash scripting? this


variable also provides a geeky way of getting
Linux Server Administration Web Development Pentesting

Tech Tips Contact

echo The last parameter was ${!#}

Grabbing all parameters


In some situations, you want to grab all the
parameters provided.

The $* and $@ variables in Linux bash scripting


provides you all of your parameters. Both of
these variables include all the command line
parameters within a single variable. So you
dont have to grab them by $# variable and
iterate over them, just one step

The $* variable takes all the parameters


supplied on the command line as a single word.

The $@ variable takes all the parameters as


separate words in the same string, It allows you
to iterate through them

This code shows the dierence between them


Linux Server Administration Web Development Pentesting

Tech Tips Contact

Both variables produce the same output but if


you want to know the dierence look at the
following example

#!/bin/bash

count=1

for param in "$*"

do

echo "\$* Parameter #$count = $param"

count=$(( $count + 1 ))

done

count=1

for param in "$@"

do

echo "\$@ Parameter #$count = $param"

count=$(( $count + 1 ))

done

Now you see the dierence.


Linux Server Administration Web Development Pentesting

Tech Tips Contact

$* variable treated all the parameters as a


single parameter, while the $@ variable treated
each parameter separately. So you can use any
one of those variables according to your needs

Shift command
The shift command has some little risk in Linux
bash scripting it literally shifts the command
line parameters in their relative positions.

When you use the shift command, it moves


each parameter variable one position to the left
by default. So, the value for variable $3 is
moved to $2, the value for variable $2 is moved
to $1, and the value for variable $1 is discarded
(note that the value for variable $0, the script
name, remains unchanged).

This is another great way to iterate through


parameters

#!/bin/bash

count=1
Linux Server Administration Web Development Pentesting

Tech Tips Contact

shift

done

Here the script performs a while loop, checking


the length of the rst parameters value. When
The rst parameters length is zero, the loop
ends. After testing the rst parameter, the shift
command is used to shift all the parameters
one position.

Careful when using shift command because


when a parameter is shifted its value is
removed and cannot be recovered.

Bash scripting options


Options are single letters preceded by a dash
that alters the behavior of a command.

#!/bin/bash

echo
Linux Server Administration Web Development Pentesting

Tech Tips Contact

-b) echo "Found the -b option" ;;

-c) echo "Found the -c option" ;;

*) echo "$1 is not an option" ;;

esac

shift

done

$ ./myscipt a b c d

The case statement checks each parameter for


valid options. When one is found, the
appropriate commands are run in the case
statement.

Separating options from


parameters
Linux Server Administration Web Development Pentesting

Tech Tips Contact

the options are nished and when the normal


parameters start.

This special character is the double dash ().


The shell uses the double dash to indicate the
end of the options list. After seeing the double
dash, your script can safely process the
remaining command line parameters as
parameters and not options

#!/bin/bash

while [ -n "$1" ]

do

case "$1" in

-a) echo "Found the -a option" ;;

-b) echo "Found the -b option";;

-c) echo "Found the -c option" ;;

--) shift

break ;;

*) echo "$1 is not an option";;

esac

shift
Linux Server Administration Web Development Pentesting

Tech Tips Contact

echo "Parameter #$count: $param"

count=$(( $count + 1 ))

done

This bash script uses the break command to


break out of the while loop when it encounters
the double dash.

As you can see from the result when the script


reaches the double dash, it stops processing
options and assumes that any remaining
parameters are command line parameters. I
love Linux bash scripting.

Processing options with


values
When you dig deep onto Linux bash scripting
Sometimes you need options with additional
Linux Server Administration Web Development Pentesting

Tech Tips Contact

command line option requires an additional


parameter and be able to process it.

#!/bin/bash

while [ -n "$1" ]

do

case "$1" in

-a) echo "Found the -a option";;

-b) param="$2"

echo "Found the -b option, with parameter value $param"

shift ;;

-c) echo "Found the -c option";;

--) shift

break ;;

*) echo "$1 is not an option";;

esac

shift

done

count=1

for param in "$@"

do
Linux Server Administration Web Development Pentesting

Tech Tips Contact

./myscript -a -b test1 -d

In this example, the case statement denes


three options that it processes.

The -b option also requires an additional


parameter value. Because the parameter being
processed is $1, you know that the additional
parameter value is located in $2 (because all
the parameters are shifted after they are
processed). Just extract the parameter value
from the $2 variable. Of course, because we
used two parameter spots for this option, you
also need to set the shift command to shift one
additional position.

Well, that works well but there are limitations.


For example, this doesnt work if you try to
combine multiple options in one parameter like
this
Linux Server Administration Web Development Pentesting

Tech Tips Contact

processing options that can help you in this


situation

Standardizing Options
When you start your Linux bash scripting, its
completely up to you to choose which letter
options you select to use and how you select to
use them.

However, a few letter options have achieved a


somewhat standard meaning in the Linux
world.

And here is the list of the common options

-a Shows all objects

-c Produces a count

-d Species a directory

-e Expands an object

-f Species a le to read data from


Linux Server Administration Web Development Pentesting

Tech Tips Contact

-l Produces a long format version of the


output

-n Uses a non-interactive (batch) mode

-o Species an output le to redirect all


output to

-q Runs in quiet mode

-r Processes directories and les


recursively

-s Runs in silent mode

-v Produces verbose output

-x Excludes an object

-y Answers yes to all questions

If you work with Linux Youll probably recognize


most of these option meanings.

Using the same meaning for your options helps


Linux Server Administration Web Development Pentesting

Tech Tips Contact

Providing command line options and


parameters is a great way to get data from
your bash script users, but sometimes your
script needs to be more interactive.

Sometimes you need data from the user while


the bash scripting is running.

The bash shell provides the read command just


for this purpose.

The read command accepts input either from


standard input (the keyboard) or from another
le descriptor. After receiving the input, the
read command places the data into a variable
and here is an example.

#!/bin/bash

echo -n "Enter your name: "

read name

echo "Hello $name, welcome to my program."

Notice that the echo command that generated


the prompt uses the n option. This prevents
the newline character at the end of the string,
Linux Server Administration Web Development Pentesting

Tech Tips Contact

You can specify multiple variables like this

#!/bin/bash
read -p "Enter your name: " first last

echo "Your data for $last, $first"

If you dont specify any variable for read


command the read command places any data it
receives in the special environment variable
REPLY.

#!/bin/bash

read -p "Enter your name: "

echo Hello $REPLY, welcome to my program

If your bash script must go on regardless of


Linux Server Administration Web Development Pentesting

Tech Tips Contact

#!/bin/bash

if read -t 5 -p "Enter your name: " name

then

echo "Hello $name, welcome to my script"


else

echo "Sorry, too slow! "

fi

If you do not enter data for ve seconds the


script will execute the else clause and print
sorry message

Reading password
In Linux bash scripting Sometimes you dont
want that input to display on the screen like
entering a password.

The -s option prevents the data entered in the


read command from being displayed on the
screen.
Linux Server Administration Web Development Pentesting

Tech Tips Contact

read -s -p "Enter your password: " pass

echo "Is your password really $pass? "

Reading from a le
The read command reads a single line of text
from the le on each call.

When no more lines are left in the le, the read


command just stop.

Now if you want to get all le data you can pipe


the result of cat command of the le to while
command that contains read command (I know
the cat command smells like noob but I want
beginners and gurus got my point).

#!/bin/bash

count=1

cat myfile | while read line

do
Linux Server Administration Web Development Pentesting

Tech Tips Contact

We just pass the le content to while loop and


iterate over every line and print the line
number and the content and each time you
increase the count by one simple enough huh?

I hope you nd this post interesting and Im


going to make more posts about Linux bash
scripting

Thanks.
Linux Server Administration Web Development Pentesting

Tech Tips Contact

On the previous post weve talked about


parameters and options in detail and today
we will talk about something is very important
in shell scripting which is input & output &
redirection.

So far, youve seen two methods for displaying


the output from your shell scripts

Displaying output on the screen


Redirecting output to a le

Sometimes you need to display some data on


the screen and other data in a le so you need
to know how Linux handles input and output so
you can get your shell script output to the right
place

Our main points are:

Standard le descriptors

STDIN (Standard Input)

STDOUT (Standard Output)


Linux Server Administration Web Development Pentesting

Tech Tips Contact

Redirecting output in shell scripts

Redirecting Input in shell scripts

Creating your own redirection

Creating input le descriptors

Closing le descriptors

Listing open le descriptors

Suppressing command output

Standard le descriptors
Everything is a le in Linux and that includes
input and output and Linux identies each le
using the le descriptor.

Each process is allowed to have up to nine open


le descriptors at a time. The bash shell
reserves the rst three le descriptors 0, 1, 2

0 STDIN Standard input


Linux Server Administration Web Development Pentesting

Tech Tips Contact

input and output from your shell script.

You need to fully understand those three


because those are like the backbones of your
shell scripting, so we are going to describe
every one of them in detail.

STDIN
This stands for the standard input to the shell.
For a terminal interface, the standard input is
the keyboard.

When you use the input redirect symbol (<) in


shell scripting, Linux replaces the standard
input le descriptor with the le referenced. It
reads the le and sends the data just as if it
were typed on the keyboard No magic.

Many bash commands accept input from STDIN


If no les are specied on the command line
like cat command.

When you enter the cat command on the


command line without anything, it accepts
Linux Server Administration Web Development Pentesting

Tech Tips Contact

This stands for the standard output for the


shell. The standard output is the screen.

Most bash commands direct their output to the


STDOUT le descriptor by default which is the
screen.

You can also append data to a le. You do this


using the >> symbol.

So if we have a le contains data we can


append another data to it using this symbol like
this

pwd >> myfile

The output generated by pwd is appended to


myle without deleting the existed content.

Fine but if you try to redirect something and


that command run into a problem
Linux Server Administration Web Development Pentesting

Tech Tips Contact

Here there is no le called xle on my PC and it


generates error and the shell doesnt redirect
the error message to the output redirection le
but the error message appeared on the screen
and here is the third type of le descriptors

STDERR
This le descriptor standard error output for the
shell

By default, the STDERR le descriptor points to


the same place as the STDOUT le descriptor
thats why when an error occurs you see the
error on the screen.

So you need to redirect the errors to maybe log


le or any else instead of printing it on the
screen

Redirecting errors
As we see the STDERR le descriptor is set to
the value 2. We can redirect the errors by
Linux Server Administration Web Development Pentesting

Tech Tips Contact

cat ./myfile

As you see the error now is in the le and


nothing on the screen

Redirecting errors and


normal output
In shell scripting, if you want to redirect both
errors and the normal output, you need to
precede each with the appropriate le
descriptor for the data you want to redirect like
this

ls l myfile xfile anotherfile 2>


errorcontent 1> correctcontent
Linux Server Administration Web Development Pentesting

Tech Tips Contact

using the 2> symbol.

If you want, you can redirect both STDERR and


STDOUT output to the same output le use &>
symbol like this

ls l myfile xfile anotherfile &>


content

All error and standard output are redirected to


le named content.

Redirecting Output in
Scripts
There are two methods for redirecting output in
shell scripting

Temporarily redirection
Linux Server Administration Web Development Pentesting

Tech Tips Contact

You can redirect an individual output line to


STDERR. You just need to use the output
redirection symbol to redirect the output to the
STDERR le descriptor and you must precede
the le descriptor number with an ampersand
(&) like this

#!/bin/bash

echo "This is an error" >&2

echo "This is normal output"

So if we run it we will see both lines printed


normally because as we know STDERR output
to STDOUT if you redirect STDERR when
running the script we should do it like this

./myscript 2> myfile


Linux Server Administration Web Development Pentesting

Tech Tips Contact

Permanent redirections
If you have lots of data that youre redirecting
in your script, it would be hard to redirect every
echo statement. Instead, you can redirect to a
specic le descriptor for the duration of the
script by using the exec command.

#!/bin/bash

exec 1>outfile

echo "This is a test of redirecting all output"

echo "from a shell script to another file."

echo "without having to redirect every line"

If we look at the le called outle we will see


the output of echo lines.

You can also redirect the STDOUT in the middle


Linux Server Administration Web Development Pentesting

Tech Tips Contact

echo "now redirecting all output to another location"

exec 1>myfile

echo "This should go to the myfile file"


echo "and this should go to the myerror file"

The exec command redirects any output going


to STDERR to the le myerror. Then, the script
uses the echo statement to display a few lines
to STDOUT which is the screen.

After that, the exec command is used again to


redirect STDOUT to the myle le and nally,
we redirect the error from within the echo
statement to go to STDERR which in this case is
myerror le.

Now you have all the ability to redirect all of


your output to whatever you want Excellent!
Linux Server Administration Web Development Pentesting

Tech Tips Contact

exec command allows you to redirect STDIN


from a le.

exec 0< myfile

This command tell the shell to take the input


from the le called myle instead of STDIN and
here is an example

#!/bin/bash

exec 0< testfile

count=1

while read line

do

echo "Line #$count: $line"

count=$(( $count + 1 ))

done

Shell scripting is easy.


Linux Server Administration Web Development Pentesting

Tech Tips Contact

le instead of the keyboard.

Some Linux system administrators use this


technique to read the log les for processing
and we will discuss more ways to read the log
on the upcoming posts more professionally.

Creating Your Own


Redirection
When you redirect input and output in your
shell script, youre not limited to the three
default le descriptors. As I mentioned that you
could have up to nine open le descriptors in
the shell. The other six le descriptors from 3
through 8 and are available for you to use as
either input or output redirection. You can
assign any of these le descriptors to a le and
then use them in your shell scripts

You can assign a le descriptor for output by


using the exec command and heres an
example how to do that
Linux Server Administration Web Development Pentesting

Tech Tips Contact

echo "And this should be back on the screen"

Creating input le
descriptors
You can redirect input le descriptors in shell
scripting exactly the same way as output le
descriptors. Save the STDIN le descriptor
location to another le descriptor before
redirecting it to a le.

When youre nished reading the le, you can


restore STDIN to its original location

#!/bin/bash

exec 6<&0

exec 0< myfile

count=1

while read line

do
Linux Server Administration Web Development Pentesting

Tech Tips Contact

read -p "Are you done now? " answer

case $answer in

y) echo "Goodbye";;

n) echo "Sorry, this is the end.";;

esac

In this example, le descriptor 6 is used to hold


the location for STDIN. The shell script then
redirects STDIN to a le. All the input for the
read command comes from the redirected
STDIN, which is now the input le.

After all the lines have been read, the shell


script returns STDIN to its original location by
redirecting it to le descriptor 6. And the shell
script makes sure that STDIN is back to normal
by using another read command and now it is
Linux Server Administration Web Development Pentesting

Tech Tips Contact

descriptors when the script exits. There are


situations you need to manually close a le
descriptor before the end of the script. To close
a le descriptor, redirect it to the special
symbol &- like this

exec 3>&-

#!/bin/bash

exec 3> myfile

echo "This is a test line of data" >&3

exec 3>&-

echo "This won't work" >&3

As you can see it gives error bad le descriptor


because it is no longer exist

Note: careful in shell scripting when closing le


descriptors. If you open the same output le
later on in your shell script, the shell replaces
Linux Server Administration Web Development Pentesting

Tech Tips Contact

descriptors
The lsof command lists all the open le
descriptors on the entire Linux system

On many Linux systems like Fedora, the lsof


command is located in the /usr/sbin.

This command is very useful actually it displays


information about every le currently open on
the Linux system. This includes all the
processes running on background, as well as
any user accounts logged into the system.

This command has a lot of options so I think I


will make a special post about it later but lets
take the important parameters we need

-p, allows you to specify a process ID

-d, allows you to specify the le descriptor


numbers to display

To get the current PID of the process, you can


use the special environment variable $$, which
Linux Server Administration Web Development Pentesting

Tech Tips Contact

lsof -a -p $$ -d 0,1,2

The le type associated with STDIN, STDOUT,


and STDERR is character mode. Because the
STDIN, STDOUT, and STDERR le descriptors all
point to the terminal, the name of the output
le is the device name of the terminal. All three
standard les are available for both reading and
writing.

Now, lets look at the results of the lsof


command from inside a script thats opened a
couple of alternative le descriptors

#!/bin/bash

exec 3> myfile1

exec 6> myfile2

exec 7< myfile3

lsof -a -p $$ -d 0,1,2,3,6,7
Linux Server Administration Web Development Pentesting

Tech Tips Contact

two for output (3 and 6) and one for input (7).

And you can see the pathname for the les


used in the le descriptors.

Suppressing Command
Output
Sometimes you dont want to see any output
this often occurs if youre running a script as a
background process (we will discuss how to
make you shell script run in the background in
the next posts)

We redirect the output to the hole which is


/dev/null

For example, we can suppress errors like this

ls -al badfile anotherfile 2> /dev/null

And this idea is also used when you want to


truncate a le without deleting it completely

cat /dev/null > myfile


Linux Server Administration Web Development Pentesting

Tech Tips Contact

I hope you enjoy it; the next post will be how to


control our running script and how to run your
shell script in the background without
interruption and how to pause them while they
are running and some other cool stu, Stay
tuned.

Thanks

LINUX

Linux Bash Scripting The


Awesome Guide Part5
February 15, 2017 admin 0 Comments

On the last post, weve talked about input and


Linux Server Administration Web Development Pentesting

Tech Tips Contact

command line interface in real-time mode. This


isnt the only way to run Linux bash scripts in
Linux.

Our main points are:

Linux signals

Stop a process

Pause a process

Trapping signals

Trapping the script exit

Modifying or removing a trap

Running scripts in background mode

Running Scripts without a Hang-Up

Viewing jobs

Restarting stopped jobs


Linux Server Administration Web Development Pentesting

Tech Tips Contact

Starting scripts with a new shell

There are various control methods include


sending signals to your script, modifying a
scripts priority, and switching the run mode
while a script is running. This post describes the
dierent ways you can control your Linux bash
scripts.

Linux signals
There are more than 30 Linux signals that can
be generated by the system and applications
and this is the most common Linux system
signals that youll run across in your Linux bash
script writing

Signal Value Description

1 SIGHUP Hangs up the


process

2 SIGINT Interrupts the


process
Linux Server Administration Web Development Pentesting

Tech Tips Contact

15 SIGTERM Terminates the


process if possible

17 SIGSTOP Unconditionally
stops, but doesnt terminate, the process

18 SIGTSTP Stops or pauses the


process, but doesnt terminate

19 SIGCONT Continues a stopped


process

If the bash shell receives a SIGHUP signal, such


as when you leave an interactive shell, it exits.
Before it exits, it passes the SIGHUP signal to
any processes started by the shell, including
any running shell scripts

With a SIGINT signal, the shell is just


interrupted. The Linux kernel stops giving the
shell processing time on the CPU. When this
happens, the shell passes the SIGINT signal to
any processes started by the shell to notify
them.
Linux Server Administration Web Development Pentesting

Tech Tips Contact

Generating signals
The bash shell allows you to generate two basic
Linux signals using key combinations on the
keyboard. This feature comes in handy if you
need to stop or pause a running bash script

Stop a process
The Ctrl+C key combination generates a SIGINT
signal and sends it to any processes currently
running in the shell which simply stops the
current process running in the shell.

$ sleep 100

Ctrl+C

Pause a process
The Ctrl+Z key combination generates a
Linux Server Administration Web Development Pentesting

Tech Tips Contact

$ sleep 100

Ctrl+Z

The number in the square brackets is the job


number assigned by the shell. The shell refers
to each process running in the shell as a job
which is unique. It assigns the rst started
process job number 1, the second job number
2, and so on

If you have a stopped job assigned to your shell


the bash warns you if you try to exit the shell.

You can view the stopped jobs using the ps


command

ps l
Linux Server Administration Web Development Pentesting

Tech Tips Contact

In the S column (process state), shows the


stopped jobs state as T. This indicates the
command is either being traced or is stopped

If you want to terminate a stopped job you can


kill its process by using kill command I
recommend you to review the basic Linux
commands if you need more info about kill
command

kill processID

Trapping signals
The trap command allows you to specify which
Linux signals your shell script can watch for and
intercept from the shell. If the script receives a
signal listed in the trap command, it prevents it
from being processed by the shell and instead
handles it locally.

So instead of allowing your Linux bash script to


leave signals ungoverned, you can use trap
command to do that.
Linux Server Administration Web Development Pentesting

Tech Tips Contact

#!/bin/bash

trap "echo ' Trapped Ctrl-C'" SIGINT

echo This is a test script

count=1
while [ $count -le 10 ]

do

echo "Loop #$count"

sleep 1

count=$(( $count + 1 ))

done

The trap command used in this example


displays a simple text message each time it
detects the SIGINT signal when hitting Ctrl+C.

Each time the Ctrl+C key combination was


Linux Server Administration Web Development Pentesting

Tech Tips Contact

You can trap them when the shell script exits;


just add the EXIT signal to the trap command

#!/bin/bash

trap "echo Goodbye..." EXIT

count=1

while [ $count -le 5 ]

do

echo "Loop #$count"

sleep 1

count=$(( $count + 1 ))

done

When the Linux bash script gets exit, the trap is


triggered and the shell executes the echo
command specied
Linux Server Administration Web Development Pentesting

Tech Tips Contact

trap
You can reissue the trap command with new
options

#!/bin/bash

trap "echo 'Ctrl-C is trapped.'" SIGINT

count=1

while [ $count -le 5 ]

do

echo "Loop #$count"

sleep 1

count=$(( $count + 1 ))

done

trap "echo ' I modified the trap!'" SIGINT

count=1

while [ $count -le 5 ]

do

echo "Second Loop #$count"

sleep 1

count=$(( $count + 1 ))

done
Linux Server Administration Web Development Pentesting

Tech Tips Contact

After the signal trap is modied, the bash script


manages the signal or signals dierently.

You can also remove a set trap. Simply add two


dashes after the trap command

#!/bin/bash

trap "echo 'Ctrl-C is trapped.'" SIGINT

count=1

while [ $count -le 5 ]

do

echo "Loop #$count"

sleep 1
count=$(( $count + 1 ))

done

trap -- SIGINT

echo "I just removed the trap"

count=1

while [ $count -le 5 ]

do
Linux Server Administration Web Development Pentesting

Tech Tips Contact

If a signal is received before the trap is


removed, the script processes it per the original
trap command

$ ./myscript

Crtl+C

The rst Ctrl+C were used to attempt to


terminate the script. Because the signal was
received before the trap was removed, the
script executed the echo command specied in
the trap. After the script executed the trap
removal, then Ctrl+C could terminate the bash
script

Running Linux bash


scripts in background
Linux Server Administration Web Development Pentesting

Tech Tips Contact

cant do anything else in your terminal session.


Fortunately, theres a simple solution to that
problem.

If you see the output of the ps command you


will see all the running processes in the
background and not tied to the terminal.

We can do the same just place ampersand


symbol after the command

#!/bin/bash

count=1

while [ $count -le 10 ]

do
sleep 1

count=$(( $count + 1 ))

done

$ ./myscipt &
Linux Server Administration Web Development Pentesting

Tech Tips Contact

done.

Notice that while the background process is


running, it still uses your terminal monitor for
STDOUT and STDERR messages so if the error
occurs you will see the error message and
normal output also.

If the terminal session exit, the background


process also exit

So what if you want to continue running even if


you close the terminal?

Running Scripts without a


Hang-Up
You can run your Linux bash scripts in the
background process even if you exit the
Linux Server Administration Web Development Pentesting

Tech Tips Contact

blocking any SIGHUP signals that are sent to


the process. This prevents the process from
exiting when you exit your terminal.

$ nohup ./myscript &

The nohup command disassociates the process


from the terminal, the process loses the
STDOUT and STDERR output links. To
accommodate any output generated by the
command, the nohup command automatically
redirects STDOUT and STDERR messages to a
le, called nohup.out

Note when running multiple commands from


the same directory, because all the output is
sent to the same nohup.out le

Viewing jobs
The jobs command allows you to view the
Linux Server Administration Web Development Pentesting

Tech Tips Contact

do

echo "Loop #$count"

sleep 10
count=$(( $count + 1 ))
done

Then run it

$ ./myscript

Then stop it using the Ctrl+Z

Run the same bash script but in background


using the ampersand symbol and to make life a
little easier, Im going to make the output of
that script is redirected to a le so it doesnt
appear on the screen

$ ./myscript > outfile &


Linux Server Administration Web Development Pentesting

Tech Tips Contact

The jobs command shows both the stopped and


the running jobs

jobs l

-l parameter to view the process ID

Restarting stopped jobs


To restart a job in background mode, use the bg
command

$ ./myscript

Ten press Ctrl+z

Now it is stopped

$ bg

As you can see it is now running in background


Linux Server Administration Web Development Pentesting

Tech Tips Contact

To restart a job in foreground mode, use the fg


command.

$ fg 1

Scheduling a job
The Linux system provides a couple of ways to
run a bash script at a preselected time: the at
command and the cron table

The at command

This is the format of the command

at [-f lename] time

The at command recognizes lots of dierent


time formats

A standard hour and minute, such as 10:15


An AM/PM indicator, such as 10:15PM
A specic named time, such as now, noon,
midnight
Linux Server Administration Web Development Pentesting

Tech Tips Contact

MM/DD/YY, or DD.MM.YY
A text date, such as Jul 4 or Dec 25, with or
without the year
Now + 25 minutes
10:15PM tomorrow
10:15 + 7 days

We dont want to dig deep into the at command


but for now, just make it simple and we will
discuss it in detail in future posts.

$ at -f ./myscript now

The M parameter is to send output to e-mail if


the system has e-mail and if not this will
suppress the output of the at command

To list the pending jobs use atq command

$ atq
Linux Server Administration Web Development Pentesting

Tech Tips Contact

you can use the atrm command to remove a


pending job by specifying the job number

$ atrm 18

Scheduling scripts
Using the at command to schedule a script to
run at a preset time is great, but what if you
need that script to run at the same time every
day or once a week or once a month.

The Linux system uses the crontab command to


allow you to schedule jobs that need to run
regularly.

The crontab program runs in the background


and checks special tables, called cron tables,
for jobs that are scheduled to run

To list an existing cron table, use the -l


Linux Server Administration Web Development Pentesting

Tech Tips Contact

minute,Hour, dayofmonth, month, and


dayofweek

So if you want to run a command at 10:30 on


every day, you would use this cron table entry

30 10 * * * command

The wildcard character (*) used in the


dayofmonth, month, and dayofweek elds
indicates that cron will execute the command
every day of every month at 10:30.

To specify a command to run at 4:30 PM every


Monday, you would use the following

30 16 * * 1 command

The day of the week start from 0 to 6 where 0 is


Sunday and 6 is Saturday

Heres another example: to execute a


command at 12 noon on the rst day of every
month, you would use the following format
Linux Server Administration Web Development Pentesting

Tech Tips Contact

the cron in great detail in future posts.

To add entries to your cron table, use the -e


parameter like this

crontab e

Then type your command like the following

30 10 * * * /home/likegeeks/Desktop
/myscript

This will schedule our script to run at 10:30


every day

Note sometimes you see error says Resource


temporarily unavailable.

All you have to do is this

$ rm -f /var/run/crond.pid

You should be root user

Just that simple!

You can use one of the pre-congured cron


Linux Server Administration Web Development Pentesting

Tech Tips Contact

/etc/cron.daily

/etc/cron.weekly

/etc/cron.monthly

Just put your bash script le on any of those


directories and it will run periodically.

Starting scripts with a


new shell
Remember from the previous posts weve
talked about startup les I recommend you to
review the previous posts to get the point.

$HOME/.bash_prole

$HOME/.bash_login

$HOME/.prole

Just place any scripts you want to run at login


time in the rst le listed.

Ok but what about running our bash script


Linux Server Administration Web Development Pentesting

Tech Tips Contact

execute that command

This for now, I hope you nd the post useful.

Thanks.

LINUX

Bash Scripting The


Awesome Guide Part6 Bash
Functions
February 17, 2017 admin 2 Comments

Before we talk about bash functions lets


discuss this situation. When writing bash
scripts, youll nd yourself that you are using
Linux Server Administration Web Development Pentesting

Tech Tips Contact

refer to that block of code anywhere in your


bash script without having to rewrite it.

The bash shell provides a feature allowing you


to do just that called Functions.

Bash functions are blocks of script code that


you assign a name to and reuse anywhere in
your code. Anytime you need to use that block
of code in your script, you simply use the
function name you assigned it.

We are going to talk about how to create your


own bash functions and how to use them in
other shell scripts.

Our main points are:

Creating a function

Using functions

Using the return command

Using function output


Linux Server Administration Web Development Pentesting

Tech Tips Contact

Recursive function

Creating libraries

Use bash functions from command line

Creating a function
You can create a function like this

functionName {

Or like this

functionName() {

The parenthesis on the second way is used to


pass values to the function from outside of it so
these values can be used inside the function.

Using functions
#!/bin/bash
Linux Server Administration Web Development Pentesting

Tech Tips Contact

while [ $count -le 3 ]

do

myfunc

count=$(( $count + 1 ))

done

echo "This is the end of the loop"

myfunc

echo "End of the script"

Here weve created a function called myfunc


and in order to call it, we just type itsname.

The function can be called many times as you


want.

Notice: If you attempt to use a function before


its dened, youll get an error message

#!/bin/bash
Linux Server Administration Web Development Pentesting

Tech Tips Contact

count=$(( $count + 1 ))

done

echo "This is the end of the loop"

function myfunc {

echo "This is an example of using a function"

echo "End of the script"

Another notice: bash function name must be


unique, or youll have a problem. If you redene
a function, the new denition overrides the
original function denition without any errors

#!/bin/bash

function myfunc {

echo "The first function definition"

myfunc

function myfunc {
Linux Server Administration Web Development Pentesting

Tech Tips Contact

As you see the second function denition takes


control from the rst one without any error so
take care when dening functions.

Using the return


command
The return command allows you to specify a
single integer value to dene the function exit
status.

There are two ways of using return command;


the rst way is like this

#!/bin/bash

function myfunc {

read -p "Enter a value: " value

echo "adding value"

return $(( $value + 10 ))

}
Linux Server Administration Web Development Pentesting

Tech Tips Contact

The myfunc function adds 10 to the value


contained in the $value variable provided by
the user input. It then returns the result using
the return command, which the script displays
using the $? Variable.

If you execute any other commands before


retrieving the value of the function, using the
$? variable, the return value from the function
is lost. Remember that the $? Variable returns
the exit status of the last executed command.

You cannot use this return value technique if


you need to return either larger integer values
or a string value.

Using function output


The second way of returning a value from a
bash function is to capture the output of a
command to a shell variable; you can also
capture the output of a function to a shell
Linux Server Administration Web Development Pentesting

Tech Tips Contact

function myfunc {

read -p "Enter a value: " value

echo $(( $value + 10 ))

}
result=$( myfunc)

echo "The value is $result"

Passing parameters to a
function
We can deal with bash functions like small
snippets that we can reuse and thats ok but we
need to make the function like an engine, we
give it something and it returns another thing
based on what we gave.

Functions can use the standard parameter


environment variables to represent any
parameters passed to the function on the
command line. For example, the name of the
Linux Server Administration Web Development Pentesting

Tech Tips Contact

parameters passed to the function. I


recommend you to review the previous posts to
empower your knowledge about them onLinux
bash scripting.

We pass parameters to functions on the same


command line as the function, like this

myfunc $val1 10 20

The following example shows you how to


retrieve the parameter values using the
parameter environment variables

#!/bin/bash

function addnum {

if [ $# -eq 0 ] || [ $# -gt 2 ]

then

echo -1

elif [ $# -eq 1 ]

then

echo $(( $1 + $1 ))

else
Linux Server Administration Web Development Pentesting

Tech Tips Contact

value=$(addnum 10 15)

echo $value

echo -n "Adding one number: "

value=$(addnum 10)

echo $value

echo -n "Adding no numbers: "

value=$(addnum)

echo $value

echo -n "Adding three numbers: "

value=$(addnum 10 15 20)

echo $value

The addnum function checks the number of


parameters passed to it by the script. If there
are no parameters, or if there are more than
two parameters, addnum returns a value of -1.
If theres one parameter, addnum adds the
parameter to itself for the result. If there are
Linux Server Administration Web Development Pentesting

Tech Tips Contact

values from the command line of the script. The


following example fails

#!/bin/bash
function myfunc {
echo $(( $1 + $2 ))

if [ $# -eq 2 ]

then

value=$( myfunc)

echo "The result is $value"

else

echo "Usage: myfunc a b"

fi

Instead, if you want to use those values in your


bash function, you have to manually pass them
when you call the function like this
Linux Server Administration Web Development Pentesting

Tech Tips Contact

if [ $# -eq 2 ]

then

value=$(myfunc $1 $2)

echo "The result is $value"

else

echo "Usage: myfunc a b"

fi

Now they are available for the function to use,


just like any other parameter

Handling variables in
bash functions
Every variable we use has a scope, the scope is
where the variable is visible

Variables dened inside functions can have a


dierent scope than regular variables.
Linux Server Administration Web Development Pentesting

Tech Tips Contact

Local

Global variables
Global variables are variables that are visible
and valid anywhere in the bash script. If you
dene a global variable in the main section of a
script, you can retrieve its value inside a
function.

The same, if you dene a global variable inside


a function, you can retrieve its value in the
main section of the script.

By default, any variables you dene in the


script are global variables. Variables dened
outside of a function can be accessed inside the
function without problems

#!/bin/bash

function myfunc {

value=$(( $value + 10 ))

read -p "Enter a value: " value

myfunc
Linux Server Administration Web Development Pentesting

Tech Tips Contact

When the variable is assigned a new value


inside the function, that new value is still valid
when the script references the variable as the
above example the variable $value is changed
inside the function.

So how to overcome something like this; Use


local variables

Local variables
Any variables that the bash function uses
internally can be declared as local variables. To
do that, just use the local keyword in front of
the variable like this

local temp=$(( $value + 5 ))

If a variable with the same name appears


outside the function in the script, the shell
keeps the two variable values separate. Now
you can easily keep your function variables
separate from your script variables

#!/bin/bash
Linux Server Administration Web Development Pentesting

Tech Tips Contact

temp=4

myfunc

echo "The temp from outside is $temp"

Now when you use the $temp variable inside


the myfunc function, it doesnt aect the value
assigned to the $temp variable in the main
script.

Passing arrays to
functions
The art of passing an array variable to a bash
function can be confusing. If you try to pass the
array variable as a single parameter, it doesnt
work

#!/bin/bash

function myfunc {

echo "The parameters are: $@"


Linux Server Administration Web Development Pentesting

Tech Tips Contact

echo "The original array is: ${myarray[*]}"

myfunc $myarray

If you try using the array variable as a function


parameter, the function only picks up the rst
value of the array variable

To solve this problem, you must disassemble


the array variable into its individual values and
use the values as function parameters. Inside
the function, you can reassemble all the
parameters into a new array variable like this.

#!/bin/bash

function myfunc {

local newarray

newarray=("$@")

echo "The new array value is: ${newarray[*]}"

myarray=(1 2 3 4 5)
Linux Server Administration Web Development Pentesting

Tech Tips Contact

The function rebuilds the array variable from


the command line parameters

Recursive function
This feature enables the function to call itself
from within the function itself

The classic example of a recursive function is


calculating factorials. A factorial of a number is
the value of the preceding numbers multiplied
with the number. Thus, to nd the factorial of 5,
youd perform the following equation. Thus the
factorial of 5 is:

5! = 1 * 2 * 3 * 4 * 5

Using recursion, the equation is reduced down


to the following format

x! = x * (x-1)!

So to write factorial function using bash


scripting it will be like this
Linux Server Administration Web Development Pentesting

Tech Tips Contact

echo 1

else

local temp=$(( $1 - 1 ))

local result=$(factorial $temp)

echo $(( $result * $1 ))

fi

read -p "Enter value: " value

result=$(factorial $value)

echo "The factorial of $value is: $result"

Using recursive bash functions is so easy!

Creating libraries
Now we know how to write functions and how
to call them but what if you want to use these
bash functions or blocks of code on
dierent bash script les without copying
and pasting it over your les.
Linux Server Administration Web Development Pentesting

Tech Tips Contact

The key to using function libraries is the source


command. You use the source command to run
the library le script inside of your shell script.
This makes the functions available to the script,
without it, the function will not be visible in the
scope of the bash script

The source command has a shortcut alias,


called the dot operator. To source a le in a
shell script, you just need to add the following
line:

. ./myscript

Lets assume that we have a le called myfuncs


and contains the following

function addnum {

echo $(( $1 + $2 ))

Now we will use it inside another bash script le


like this

#!/bin/bash
Linux Server Administration Web Development Pentesting

Tech Tips Contact

Awesome!! Weve used the bash functions


inside myfuncs le inside our bash script le
but what if we want to use those functions from
our bash shell directly?

Use bash functions from


command line
Well, that is easy if you read the previous post
which was about signals and jobs you will
have idea that we can source our functions le
in .bashrc le and hence we can use the
functions directly from the bash shell. Cool

Edit .bashrc le and add this line

. /home/likegeeks/Desktop/myfuncs

Just make sure you type the correct path and


now from the shell when we type the following

$ addnum 10 20
Linux Server Administration Web Development Pentesting

Tech Tips Contact

functions are automatically available for any


bash scripts without sourcing wow!! That really
cool right

Note: you may need to logout and login to use


the bash functions from the shell

Another note: if you make your function name


like the any of the built-in commands you will
overwrite the default command so you should
take care of that.

With these examples, I nish my post today


hope you like it

Thank you.
Linux Server Administration Web Development Pentesting

Tech Tips Contact

LINUX

31+ Examples For Sed Linux


Command In Text
Manipulation
February 19, 2017 admin 2 Comments

On the previous post weve talked about bash


functions and how to use it from the command
line and weve seen some other cool stu I
recommend you to review it, Today we will talk
about a very useful tool for string manipulation
called sed, sed Linux command is one of the
most common tools that people use to work
with text les like log les, conguration les,
and other text les. If you perform any type of
data manipulation in your bash scripts, you
want to become familiar with the sed and gawk
tools in this post we are going to focus on sed
Linux command and see its ability to
Linux Server Administration Web Development Pentesting

Tech Tips Contact

extract some text, delete, modify or whatever.


The Linux system provides some common tools
for doing just that one of those tools is sed.

We will discuss the 31+ examples with pictures


to show the output of everyone.

Our main points are:

Understand sed command

Using multiple sed commands in the


command line

Reading commands from a le

Substituting ags

Replacing characters

Limiting sed

Deleting lines

Inserting and appending text


Linux Server Administration Web Development Pentesting

Tech Tips Contact

Reading data from a le

Useful examples

Understand sed Linux


command
The sed command is called a stream editor, it is
an interactive text editor, such as nano, you
interactively use keyboard commands to insert,
delete, or replace text in the data. SedLinux
command edits a stream of data based on a set
of rules you provide and this is the format of
the sed command

$ sed options file

By default, the sed Linux command applies the


specied commands to the STDIN. This allows
you to pipe data directly to the sed editor for
processing like this

$ echo "This is a test" | sed 's/test


/another test/'
Linux Server Administration Web Development Pentesting

Tech Tips Contact

between the forward slashes. In this example,


the words another test were substituted for the
word test so the result will be like this

Thats the power of using the sed Linux


command.

The above example was a very basic example


to demonstrate the tool. We can use sedLinux
command to manipulate les as well.

This is our le

$ sed 's/test/another test' ./myfile

Youll start seeing results before the sed editor


completes processing the entire le because
sed returns the data instantaneously awesome!
Linux Server Administration Web Development Pentesting

Tech Tips Contact

text le itself. It only sends the modied text to


STDOUT. If you look at the text le, it still
contains the original data. You can overwrite
the le with the new content very easy if you
follow our previous posts we talk about
redirections

Using multiple sed Linux


commands in the
command line
To execute more than one command from the
sed command line, just use the -e option like
this

$ sed -e 's/This/That/; s/test/another


test/' ./myfile
Linux Server Administration Web Development Pentesting

Tech Tips Contact

Also, you can use single quotation to separate


commands like this

$ sed -e '
> s/This/That/

> s/test/another test/' myfile

The same result no big deal.

Reading commands from


a le
If you have lots of sed commands you want to
process, it is often easier to just store them in a
separate le. Use the -f option to specify the le
in the sed command like this

$ cat mycommands
Linux Server Administration Web Development Pentesting

Tech Tips Contact

Substituting ags
Look at the following example carefully

$ cat myfile

$ sed 's/test/another test/' myfile

The substitute command works ne in replacing


text in multiple lines, but it replaces only the
rst occurrence in each line. To get the
substitute command to work on dierent
occurrences of the text, you must use a
Linux Server Administration Web Development Pentesting

Tech Tips Contact

There are four types of substitutions

A number, indicating the pattern occurrence


for which new text should be substituted
g, indicating that new text should be
substituted for all occurrences of the existing
text
p, indicating that the contents of the original
line should be printed
w le, which means to write the results of the
substitution to a le

The rst type of substitution, you can specify


which occurrence of the matching pattern the
sed Linux command should substitute new text
for

$ sed 's/test/another test/2' myfile

As a result of specifying a 2 as the substitution


ag, the sed Linux command replaces the
Linux Server Administration Web Development Pentesting

Tech Tips Contact

text

$ sed 's/test/another test/g' myfile

The p substitution ag prints a line that


contains a matching pattern in the substitute
command which used with -n option suppresses
output from the sed command so it produces
output only for lines that have been modied
by the substitute command

$ cat myfile

$ sed -n 's/test/another test/p' myfile

The w substitution ag produces the same


Linux Server Administration Web Development Pentesting

Tech Tips Contact

The output of the sed command appears in


STDOUT, but only the lines that include the
matching pattern are stored in output le.

Replacing characters
Suppose that you want to substitute the C shell
for the bash shell in the /etc/passwd le, youd
have to do this

$ sed 's/\/bin\/bash/\/bin\/csh/'
/etc/passwd

That looks confusing for some people because


the forward slash is used as the string
delimiter, you must use a backslash to escape
it.

Luckily there is another way to achieve that.


The sed Linux command allows you to select a
dierent character for the string delimiter in the
Linux Server Administration Web Development Pentesting

Tech Tips Contact

The exclamation point is used for the string


delimiter. It is now easier to read.

Limiting sed
The commands you use in the sed command
apply to all lines of the text data. If you want to
apply a command only to a specic line or a
group of lines, there are two forms

A numeric range of lines


A text pattern that lters out a line

The address you specify in the command can


be a single line number or a range of lines
specied by a starting line number, a comma,
and an ending line number

$ sed '2s/test/another test/' myfile


Linux Server Administration Web Development Pentesting

Tech Tips Contact

Also, we can start from a line to the end of the


le

$ sed '2,$s/test/another test/' myfile

You can specify a text pattern to lter lines for


the sed command. The pattern is written like
this

$ sed '/likegeeks/s/bash/csh/'
/etc/passwd

The command was applied only to the line with


the matching text pattern.
Linux Server Administration Web Development Pentesting

Tech Tips Contact

Deleting lines
If you need to delete specic lines of text in a
text stream, you can use the delete command

Be careful with the delete command, because if


you forget to include an addressing scheme, all
the lines are deleted from the stream

$ sed '3d' myfile

Here we delete the third line only from myle

$ sed '2,3d' myfile

Here we delete a range of lines the second and


Linux Server Administration Web Development Pentesting

Tech Tips Contact

Here we delete from the third line to the end of


the le

Note that the sed command doesnt touch the


original le. Any lines you delete are only gone
from the output only.

$ sed '/test 1/d' myfile

Here we use pattern to delete the line if


matched on the rst line

You can also delete a range of lines using two


text patterns like the following

$ sed '/second/,/fourth/d' myfile


Linux Server Administration Web Development Pentesting

Tech Tips Contact

The rst to the third line deleted.

Inserting and appending


text
The sed Linux command allows you to insert
and append text lines to the data stream using
the following commands

The insert command (i) adds a new line


before the specied line
The append command (a) adds a new line
after the specied line

You must specify the line to insert or append


the line to insert on a separate line by itself So
you cant use these commands on a single
command line.

$ echo "Another test" | sed 'i\First


test '
Linux Server Administration Web Development Pentesting

Tech Tips Contact

While in append mode the text appears after


the data stream text.

This works well for adding text before or after


the text in the data stream, but what about
adding text in the middle?

To insert or append data inside the data stream


lines, you must specify where the sed
command where you want the data to appear.

You can match either a numeric line number or


a text pattern, and of course, you cannot use a
range of addresses.

$ sed '2i\This is the inserted line.'


myfile
Linux Server Administration Web Development Pentesting

Tech Tips Contact

And the appending goes the same way but look


at the position of the appended text

$ sed '2a\This is the appended line.'


myfile

The dierence is it places the new text line


after the specied line number.

Modifying lines
The change command allows you to change the
contents of an entire line of text in the data
stream. All you have to do is to specify the line
that you want to change.

$ sed '3c\This is a modified line.'


Linux Server Administration Web Development Pentesting

Tech Tips Contact

You can also use a text pattern or a regular


expression and all lines match that pattern will
be modied.

$ sed '/This is/c This is a changed


line of text.' myfile

Transforming characters
The transform command (y) works on a single
character like this.

$ sed 'y/123/567/' myfile


Linux Server Administration Web Development Pentesting

Tech Tips Contact

You cant limit the transformation to a specic


occurrence of the character.

Printing line numbers


The equal sign command prints the current line
number for the line within the data stream.

$ sed '=' myfile

The sed editor prints the line number before


the actual line of text nothing fancy here.

However by using n combined with the equal


sign the sed command display the line number
that contains the matching text pattern only.

$ sed -n '/test/=' myfile


Linux Server Administration Web Development Pentesting

Tech Tips Contact

Weve seen how to insert and append data to


the data stream. Now we will read data from a
le.

The read command (r) allows you to insert data


contained in a separate le.

You can only specify a single line number or


text pattern address. The sed Linux command
inserts the text from the le after the address
specied.

$ cat newfile

$ sed '3r newfile' myfile

The le called newle content is just inserted


after the third line as expected.
Linux Server Administration Web Development Pentesting

Tech Tips Contact

Cool right?

Useful examples
We can use the read command is to use it in
conjunction with a delete command to replace
a placeholder in a le with data from another
le. Suppose we have the following le called
newle

The word DATA in the le is a placeholder for a


real content which is stored on another le
called data. We will replace it with the actual
content

$ Sed '/DATA>/ {
Linux Server Administration Web Development Pentesting

Tech Tips Contact

Now the placeholder text is replaced with the


data le content. Super cool!!

This is just a very small intro about sed


command. Actually, sed Linux command is
another world by itself.

We can spend weeks to cover sed and its uses


but you can use your mind with our previous
posts to produce something great. As I said
before the only limitation is your imagination.

I hope you enjoy whatve introduced today


about the string manipulation using sed Linux
command.

This is just a beginning for sed command I will


Linux Server Administration Web Development Pentesting

Tech Tips Contact

LINUX

30 Examples For Awk


Command In Text
Processing
February 21, 2017 admin 3 Comments

On the previous post weve talked about sed


Linux command and weve seen many
examples of using it in text processing and how
it is good in this, nobody can deny that sed is
very handy tool but it has some limitations,
sometimes you need a more advanced tool for
manipulating data, one that provides a more
programming-like environment giving you more
control to modify data in a le more robust. This
Linux Server Administration Web Development Pentesting

Tech Tips Contact

than the sed editor by providing a programming


language instead of just editor commands.
Within the awk programming language, you
can do the following

Dene variables to store data.


Use arithmetic and string operators to
operate on data.
Use structured programming concepts and
control ow, such as if-then statements and
loops, to add logic to your text processing.
Generate formatted reports

Actually generating formatted reports comes


very handy when working with log les contain
hundreds or maybe millions of lines and output
a readable report that you can benet from.

our main points are:

command options

Reading the program script from the


command line
Linux Server Administration Web Development Pentesting

Tech Tips Contact

Running scripts before processing data

Running scripts after processing data

Built-in variables

Data variables

User dened variables

Structured Commands

Formatted Printing

Built-In Functions

User Dened Functions

awk command options


The awk command has a basic format as
follows

$ awk options program file

And those are some of the options for awk


Linux Server Administration Web Development Pentesting

Tech Tips Contact

-f le species a le name to read the


program from

-v var=value Denes a variable and default


value used in the awk command

mf N species the maximum number of


elds to process in the data le

mr N Species the maximum record size in


the data le

-W keyword Species the compatibility mode


or warning level for awk

The real power of awk is in the program script.


You can write scripts to read the data within a
text line and then manipulate and display the
data to create any type of output report.

Reading the program


script from the command
line
awk program script is dened by opening and
Linux Server Administration Web Development Pentesting

Tech Tips Contact

like this

$ awk '{print "Welcome to awk command


tutorial"}'

If you run this command nothing will happen!!


And that because no lename was dened in
the command line.

The awk command retrieves data from STDIN.


When you run the program, it just waits for text
to come in via STDIN

If you type a line of text and press the Enter,


the awk command runs the text through the
program script. Just like the sed editor, the awk
command executes the program script on each
line of text available in the data stream.
Because the program script is set to display a
xed text string, you get the same text output.

$ awk '{print "Welcome to awk command


tutorial "}'
Linux Server Administration Web Development Pentesting

Tech Tips Contact

string we provide.

To terminate the program we have to send


End-of-File (EOF) character. The Ctrl+D key
combination generates an EOF character in
bash. Maybe you disappointed by this example
but wait for the awesomeness.

Using data eld variables


One of the primary features of awk is its ability
to manipulate data in the text le. It does this
by automatically assigning a variable to each
data element in a line. By default, awk assigns
the following variables to each data eld it
detects in the line of text

$0 represents the entire line of text.


$1 represents the rst data eld in the line of
text.
$2 represents the second data eld in the line
of text.
$n represents the nth data eld in the line of
text.
Linux Server Administration Web Development Pentesting

Tech Tips Contact

Look at the following le and see how awk deal


with it

$ awk '{print $1}' myfile

This command uses the $1 eld variable to


display only the rst data eld for each line of
text.

Sometimes the separator in some les is not


space or a tab but something else. You can
specify it using F option

$ awk -F: '{print $1}' /etc/passwd


Linux Server Administration Web Development Pentesting

Tech Tips Contact

This command displays the rst data eld in


the passwd le. Because the /etc/passwd le
uses a colon to separate the data elds.

Using multiple commands


Any programming language wouldnt be very
useful if you could only execute one command.

The awk programming language allows you to


combine commands into a normal program.

To use multiple commands on the command


line, just place a semicolon between each
command

$ echo "My name is Tom" | awk


'{$4="Adam"; print $0}'

The rst command assigns a value to the $4


eld variable. The second command then prints
Linux Server Administration Web Development Pentesting

Tech Tips Contact

As with sed command, the awk command


allows you to store your scripts in a le and
refer to them in the command line with the f
option

Our le contains this script

{print $1 " has a home directory at "


$6}

$ awk -F: -f testfile /etc/passwd

Here we print the username which is the rst


eld $1 and the home path which is the sixth
eld $6 from /etc/passwd and we specify the
le that contains that script which is called
myscipt with -f option and surely the separator
Linux Server Administration Web Development Pentesting

Tech Tips Contact

This is our le

text = " has a home directory at "

print $1 text $6

$ awk -F: -f testfile /etc/passwd

Here we dene a variable that holds a text


string used in the print command.

Running scripts before


processing data
Sometimes, you may need to run a script
Linux Server Administration Web Development Pentesting

Tech Tips Contact

It forces awk to execute the script specied


after the BEGIN keyword and before awk reads
the data

$ awk 'BEGIN {print "Hello World!"}'

Lets apply it to something we can see the


result

$ awk 'BEGIN {print "The File Contents:"}

{print $0}' myfile

Now after awk command executes the BEGIN


script, it uses the second script to process any
le data. Be careful when doing this; both of
the scripts are still considered one text string
on the awk command line. You need to place
your single quotation marks accordingly.

Running scripts after


Linux Server Administration Web Development Pentesting

Tech Tips Contact

$ awk 'BEGIN {print "The File Contents:"}

{print $0}

END {print "End of File"}' myfile

When the awk command is nished printing the


le contents, it executes the commands in the
END script. This is useful to use to add the
footer as anexample.

We can put all these elements together into a


nice little script le

BEGIN {

print "The latest list of users and shells"

print " UserName \t HomePath"

print "-------- \t -------"

FS=":"

}
Linux Server Administration Web Development Pentesting

Tech Tips Contact

print "The end"

This script uses the BEGIN script to create a


header section for the report. It also denes the
le separator FS and prints the footer at the
end. Then we use the le

$ awk -f myscript /etc/passwd

This gives you a small taste of the power


available when you use simple awk scripts.

Built-in variables
The awk command uses built-in variables to
reference specic features within the program
data
Linux Server Administration Web Development Pentesting

Tech Tips Contact

But those are not the only variables, there are


more built-in variables

The following list is some of the built-in


variables that awk command use:

FIELDWIDTHS A space-separated list of


numbers dening the exact width (in spaces) of
each data eld

FS Input eld separator character

RS Input record separator character

OFS Output eld separator character

ORS Output record separator character

By default, awk sets the OFS variable to a


space, By setting the OFS variable, you can use
any string to separate data elds in the output

$ awk 'BEGIN{FS=":"; OFS="-"} {print


$1,$6,$7}' /etc/passwd
Linux Server Administration Web Development Pentesting

Tech Tips Contact

The FIELDWIDTHS variable allows you to read


records without using a eld separator
character.

In some situations, instead of using a eld


separator, data is placed in specic columns
within the record. In these instances, you must
set the FIELDWIDTHS variable to match the
layout of the data in the records

After you set the FIELDWIDTHS variable, awk


ignores the FS and calculates data elds based
on the provided eld width sizes

Suppose we have this content

1235.9652147.91

927-8.365217.27

36257.8157492.5

$ awk 'BEGIN{FIELDWIDTHS="3 5 2
5"}{print $1,$2,$3,$4}'
Linux Server Administration Web Development Pentesting

Tech Tips Contact

denes four data elds, and awk command


parses the data record accordingly. The string
of numbers in each record is split based on the
dened eld width values

The RS and ORS variables dene how your awk


command handles records in the data. By
default, awk sets the RS and ORS variables to
the newline character which means that each
new line of text in the input data stream is a
new record

Sometimes, you run into situations where data


elds are spread across multiple lines in the
data stream

Like an address and phone number, each on a


separate line

Person Name

123 High Street

(222) 466-1234
Linux Server Administration Web Development Pentesting

Tech Tips Contact

and RS variable values, awk reads each line as


a separate record and interprets each space in
the record as a eld separator. This is not what
you want

To solve this problem, you need to set the FS


variable to the newline character. This indicates
that each line in the data is a separate eld and
all the data on a line belongs to the data eld
and set the RS variable to an empty string. The
awk command interprets each blank line as a
record separator

$ awk 'BEGIN{FS="\n"; RS=""} {print


$1,$3}' addresses

Awesome! The awk command interpreted each


line in the le as a data eld and the blank lines
as record separators.
Linux Server Administration Web Development Pentesting

Tech Tips Contact

extract information from the shell environment

ARGC The number of command line


parameters present

ARGIND The index in ARGV of the current


le being processed

ARGV An array of command line parameters

ENVIRON An associative array of the current


shell environment variables and their values

ERRNO The system error if an error occurs


when reading or closing input les

FILENAME The lename of the data le used


for input to the awk command

FNR The current record number in the data


le

IGNORECASE If set to a non-zero value


ignores the case of characters in strings used in
the awk command
Linux Server Administration Web Development Pentesting

Tech Tips Contact

You should recognize a few of these variables


from previous posts about shell scripting
series

The ARGC and ARGV variables allow you to


retrieve the number of command line
parameters

This can be a little tricky because awk


command doesnt count the script as part of
the command line parameters

$ awk 'BEGIN{print ARGC,ARGV[1]}'


myfile

The ENVIRON variable uses an associative array


to retrieve shell environment variables like this.

$ awk '

BEGIN{

print ENVIRON["HOME"]

print ENVIRON["PATH"]
Linux Server Administration Web Development Pentesting

Tech Tips Contact

You can use shell variables without ENVIRON


variables like this

$echo | awk -v home=$HOME '{print "My


home is " home}'

The NF variable allows you to specify the last


data eld in the record without having to know
its position

$ awk 'BEGIN{FS=":"; OFS=":"} {print


$1,$NF}' /etc/passwd
Linux Server Administration Web Development Pentesting

Tech Tips Contact

The FNR and NR variables are similar to each


other but slightly dierent. The FNR variable
contains the number of records processed in
the current data le. The NR variable contains
the total number of records processed.

Lets take a look at those two examples to


illustrate the dierence

$ awk 'BEGIN{FS=","}{print
$1,"FNR="FNR}' myfile myfile

In this example, the awk command denes two


input les. (It species the same input le
twice.) The script prints the rst data eld value
and the current value of the FNR variable.
Notice that the FNR value was reset to 1 when
the awk command processed the second data
Linux Server Administration Web Development Pentesting

Tech Tips Contact

BEGIN {FS=","}

{print $1,"FNR="FNR,"NR="NR}

END{print "There were",NR,"records processed"}'

The FNR variable value was reset when awk


processed the second data le, but the NR
variable maintained its count into the second
data le.

User dened variables


Like any other programming language, awk
allows you to dene your own variables for use
within the script.

awk user-dened variable name can be any


number of letters, digits, and underscores, but
Linux Server Administration Web Development Pentesting

Tech Tips Contact

BEGIN{

test="This is a test"

print test

}'

Structured Commands
The awk programming language supports the
standard if-then-else format of the if statement.
You must dene a condition for the if statement
to evaluate, enclosed in parentheses.

Here is an example

testle contains the following

10

15
Linux Server Administration Web Development Pentesting

Tech Tips Contact

$ awk '{if ($1 > 20) print $1}'


testfile

Just that simple

You can execute multiple statements in the if


statement, you must enclose them with braces

$ awk '{

if ($1 > 20)

x = $1 * 2

print x

}' testfile
Linux Server Administration Web Development Pentesting

Tech Tips Contact

x = $1 * 2

print x

} else
{

x = $1 / 2

print x

}}' testfile

You can use the else clause on a single line, but


you must use a semicolon after the if statement
Linux Server Administration Web Development Pentesting

Tech Tips Contact

The while loop allows you to iterate over a set


of data, checking a condition that stops the
iteration

cat myfile

124 127 130

112 142 135

175 158 245

$ awk '{

total = 0

i = 1

while (i < 4)
{

total += $i

i++

avg = total / 3

print "Average:",avg

}' testfile
Linux Server Administration Web Development Pentesting

Tech Tips Contact

The while statement iterates through the data


elds in the record, adding each value to the
total variable and incrementing the counter
variable i.

When the counter value is equal to 4, the while


condition becomes FALSE, and the loop
terminates, dropping through to the next
statement in the script. That statement
calculates the average and prints the average.

The awk programming language supports using


the break and continue statements in while
loops, allowing you to jump out of the middle of
the loop

$ awk '{

total = 0

i = 1

while (i < 4)

{
Linux Server Administration Web Development Pentesting

Tech Tips Contact

avg = total / 2

print "The average of the first two elements is:",avg

}' testfile

The for loop


The for loop is a common method used in many
programming languages for looping.

The awk programming language supports for


loops

$ awk '{

total = 0

for (i = 1; i < 4; i++)

{
Linux Server Administration Web Development Pentesting

Tech Tips Contact

}' testfile

By dening the iteration counter in the for loop,


you dont have to worry about incrementing it
yourself as you did when using the while
statement.

Formatted Printing
The printf command in awk allows you to
specify detailed instructions on how to display
data.

It species exactly how the formatted output


should appear, using both text elements and
format speciers. A format specier is a special
code that indicates what type of variable is
displayed and how to display it. The awk
command uses each format specier as a
Linux Server Administration Web Development Pentesting

Tech Tips Contact

The format speciers use the following format

%[modier]control-letter

This list is the format speciers you can use


with printf:

c Displays a number as an ASCII


character

d Displays an integer value

i Displays an integer value (same as d)

e Displays a number in scientic


notation

f Displays a oating-point value

g Displays either scientic notation or


oating point, whichever is shorter

o Displays an octal value

s Displays a text string


Linux Server Administration Web Development Pentesting

Tech Tips Contact

}'

Here as an example, we display a large value


using scientic notation %e.

We are not going to try every format specier.


You know the concept

Built-In Functions
The awk programming language provides quite
a few built-in functions that perform
mathematical, string, and time functions. You
can utilize these functions in your awk scripts

Mathematical functions
If you love math, those are some of the
mathematical functions you can use with awk

cos(x) The cosine of x, with x specied in


Linux Server Administration Web Development Pentesting

Tech Tips Contact

toward 0

log(x) The natural logarithm of x

rand() A random oating point value larger


than 0 and less than 1

sin(x) The sine of x, with x specied in


radians

sqrt(x) The square root of x

and they can be used normally

$ awk 'BEGIN{x=exp(5); print x}'

String functions
There are many string functions you can check
the list but we will examine one of them as an
example and the rest is the same

$ awk 'BEGIN{x = "likegeeks"; print


Linux Server Administration Web Development Pentesting

Tech Tips Contact

The function toupper convert case to upper


case for the string passed.

User Dened Functions


You can create your own functions for use in
awk scripts, just dene them and use them

$ awk '

function myprint()

printf "The user %s has home path at %s\n", $1,$6

BEGIN{FS=":"}

{
myprint()

}' /etc/passwd
Linux Server Administration Web Development Pentesting

Tech Tips Contact

Here we dene a function called myprint then


we use it in out script to print output using
printf built-in function.

With this example, I nish my post today hope


you like it.

Thank you

Admin

https://likegeeks.com

RELATED ARTICLES

LINUX LINUX LINUX

Potrebbero piacerti anche