Lecture 8

Introduction to Python
Lecture 8
 Summary

• Python exceptions
• Processes and Threads
• Programming with Threads
 Python Exceptions

In Python, there are two distinguishable kinds of errors: syntax errors and
exceptions.

Syntax errors are usually called parsing errors.

Exceptions are events that can modify the flow of control through a program.

Python triggers exceptions automatically on errors, after that they can be

intercepted and processed by the program, or can be thrown up to the calling
methods.

Exceptions allow jumping directly to a routine/handler which can process the

unexpected occurred event
 Exception roles

• Error handling

• Event notification

• Special case handling

• Termination actions

• Unusual flow control

 Statements processing exceptions

try/except
Catch and recover from exceptions raised by Python, or by you.

try/finally
Perform cleanup actions, whether exceptions occur or not.

raise
Trigger an exception manually in your code.

assert
Conditionally trigger an exception in your code.

with/as
Implement context managers in Python 2.6 and 3.0 (optional in 2.5)
 The general format of the try statement

try:
<statements> # Run this main action first
except <name1>:
<statements> # Run if name1 is raised during try block
except (name2, name3):
<statements> # Run if any of these exceptions occur
except <name4> as <data>:
<statements> # Run if name4 is raised, and get instance raised
except:
<statements> # Run for all (other) exceptions raised
else:
<statements> # Run if no exception was raised during try block
 try statement clauses

Clause Comments

except: Catch all (or all other) exception types

except name: Catch a specific exception only

except name as value: Catch the listed exception and its instance

except (name1, name2): Catch any of the listed exceptions

except (name1, name2) as value: Catch any listed exception and its instance

else: Run if no exceptions are raised

finally: Always perform this block

 Catching an exception

def getAtIndex(str, idx):
 Output:

return str[idx]





print(getAtIndex("python", 7))
print("Statement after catching
exception...")
Traceback (most recent call last):
File "exception-1.py", line 5, in <module>
print(getAtIndex("python", 7))
File "exception-1.py", line 2, in getAtIndex
return str[idx]
IndexError: string index out of range

Notice that the statement after the

function call is never executed

Catching the exception

def getAtIndex(str, idx):
 Output:
return str[idx]


 Exception occurred
try:
 Statement after catching exception...
print(getAtIndex("python", 7))

except IndexError:

print("Exception occurred”)

print("Statement after catching After handling the exception, the

exception...") execution of the program is resumed
 Raising an exception

def getAtIndex(str, idx):
 Output:

if idx > len(str):

raise IndexError
 Exception occurred
return str[idx]
 Statement after catching exception...


try:

print(getAtIndex("python", 7))

except IndexError:

print("Exception occurred”)

print("Statement after catching exception...")

One can raise exceptions using the assert directive too:

def assert_fct(x):
 Output:

assert x != 0

print(x)
 Traceback (most recent call last):

 File "exception-5.py", line 5, in <module>
assert_fct(0) assert_fct(0)
File "exception-5.py", line 2, in assert_fct
assert x != 0
AssertionError
 Creating a custom exception

class CustomException(Exception):
 Output:

def __str__(self):

return "CustomException representation..."
 Exception occurred CustomException

 representation...


def getAtIndex(str, idx):

if idx > len(str):

raise CustomException

return str[idx]



try:

print(getAtIndex("python", 7))

except CustomException as e:

print("Exception occurred {}".format(e))
 Custom exceptions

The recommended mode for creating custom exceptions is by using OOP, using
classes and classes hierarchies.

The most important advantages of using class-based exceptions are:

• They can be organized in categories (adding future categories will not require
changes in the try block)

• They have state information attached

• They support inheritance

 Termination Actions

Let’s suppose we need to deal with a file, open it, do some processing, then close it

class MyException(Exception):
 Output:

pass


 finally statement reached...
def process_file(file):
 File "exception-4.py", line 9, in
raise MyException
 <module>

 process_file(f)
try:
 File "exception-4.py", line 5, in
f = open("../lab3/morse.txt", "r")
 process_file
process_file(f)
 raise MyException
print("after file processing...")
 __main__.MyException
finally:

f.close()

print("finally statement reached...")

The try/finally construction assures that the statements

included in the finally block are executed regardless of
what happens in the try block

This is extremely useful when we involve various resources in our program and we
have to release them regardless of what events could occur during the execution of
the program flow
 Unified try/except/finally

class MyException(Exception):
 Output:

def __str__(self):

return "MyException"
 Exception occurred MyException

 finally statement reached...
def process_file(file):

raise MyException



try:

f = open("../lab3/morse.txt", "r")

process_file(f)

print("after file processing...")

except MyException as me:

print("Exception occurred {}".format(me))

finally:

f.close()

print("finally statement reached...")

In this case, the exception raised in process_file() function is caught and processed in
the exception handler defined in the except block, then the code defined in finally
section is executed. In this way we assure ourselves that we release all the resources that
we used during the program (file handlers, connections to databases, sockets, etc…)
 C vs. Python error handling

C style Python style

doStuff() def doStuff():
{ doFirstThing()
if (doFirstThing() == ERROR) doNextThing()
return ERROR; ...
if (doNextThing() == ERROR) doLastThing()
return ERROR;
... if __name__ == '__main__':
return doLastThing(); try:
} doStuff()
except:
main() badEnding()
{ else:
if (doStuff() == ERROR) goodEnding()
badEnding();
else
goodEnding();
}
 Processes and Threads

Processes and Threads

 Processes context switching

How multiple processes share a CPU

(image from http://www.w3ii.com/en-US/operating_system/os_process_scheduling.html)

2 states for processes:

• Run
• Sleep PCB - Program Context Block
Relation between threads and processes

[image taken from http://www.javamex.com/tutorials/threads/how_threads_work.shtml]

 Python Threads

A Thread or a Thread of Execution is defined in computer science as the

smallest unit that can be scheduled in an operating system.

There are two different kind of threads:

• Kernel threads
• User-space Threads or user threads

(image from http://www.python-course.eu/threads.php)

 Threads

Every process has at least one thread, i.e. the process itself. A process
can start multiple threads. The operating system executes these
threads like parallel "processes". On a single processor machine, this
parallelism is achieved by thread scheduling or timeslicing.

Advantages of Threading:

• Multithreaded programs can run faster on computer systems with

multiple CPUs, because theses threads can be executed truly
concurrent.
• A program can remain responsive to input. This is true both on single
and on multiple CPU
• Threads of a process can share the memory of global variables. If a
global variable is changed in one thread, this change is valid for all
threads. A thread can have local variables.
 Threads in Python

Python has a Global Interpreter Lock (GIL)

It imposes various restrictions on threads, one of the most important
is that it cannot utilize multiple CPUs

Consider the following piece of code:

def count(n):
while n > 0:
n -= 1

Two scenarios:

• Run it twice in series: Some performance results on a Dual-Core

MacBook?
count(100000000)
count(100000000) Sequential : 24.6s
Threaded : 45.5s (1.8X slower!)
• Now, run it in parallel in two threads
t1 = Thread(target=count,args=(100000000,)) And if disabled one of the CPU cores, why
t1.start()
t2 = Thread(target=count,args=(100000000,)) does the threaded performance get better?
t2.start()
t1.join(); t2.join() Threaded : 38.0s
GIL in Python

Image taken from: https://callhub.io/understanding-python-gil/

 Python Threads

Thread-Specific State

• Each thread has its own interpreter specific

data structure (PyThreadState)

Current stack frame (for Python code)

Current recursion depth
Thread ID
Some per-thread exception information
Optional tracing/profiling/debugging hooks

• It's a small C structure (<100 bytes)

The interpreter has a global variable that simply points to the

ThreadState structure of the currently running thread
 Python Threads

• Only one Python thread can execute in the

interpreter at once

• There is a "global interpreter lock" that carefully

controls thread execution

• The GIL ensures that sure each thread get

exclusive access to the interpreter internals
when it's running (and that call-outs to C
extensions play nice)
 Using _thread package
import _thread
 Starting main program...

 Ending main program...
def print_chars(thread_name, char):
 Thread1
for i in range(5):
 Thread1 *
print("{} {}".format(thread_name, i * char))
 Thread1 **
print("{} successfully ended".format(thread_name))
 Thread1 ***

 Thread1 ****
try:
 Thread1 successfully ended
print("Starting main program...")
 Thread2
_thread.start_new_thread(print_chars, ("Thread1 ", "*"))
 Thread2 -
_thread.start_new_thread(print_chars, ("Thread2 ", "-"))
 Thread2 --
_thread.start_new_thread(print_chars, ("Thread3 ", "="))
 Thread3
print("Ending main program...")
 Thread3 =
except:
 Thread3 ==
print("Threads are unable to start")
 Thread3 ===

 Thread3 ====
while 1:
 Thread3 successfully ended
pass Thread2 ---
Thread2 ----
Thread2 successfully ended

_thread is a deprecated module, acting at low-level.

_thread.start_new_thread method was used here to start a new thread. One can notice
here that we have a while loop in the main program, looping forever. This is to avoid the
situation in which the main program starts the three threads, but it exits before the threads
finish their execution, and the buffers are not flushed (the output is buffered by default)

See the program output and notice that the threads output is displayed after the main
program end its execution.
 First threading program

import threading
 Starting thread Thread1


 Thread1
class ThreadExample(threading.Thread):
 Thread1 *
def __init__(self, name, char):
 Thread1 **
threading.Thread.__init__(self)
 Thread1 ***
self.name = name
 Thread1 ****
self.char = char
 Ending thread Thread1

 Starting thread Thread2
def run(self):
 Thread2
print("Starting thread {} ".format(self.name))
 Thread2 -
print_chars(self.name, self.char)
 Thread2 --
print("Ending thread {} ".format(self.name))
 Thread2 ---

 Thread2 ----
def print_chars(thread_name, char):
 Ending thread Thread2
for i in range(5):
 Starting thread Thread3
print("{} {}".format(thread_name, i * char))
 Exiting the main program...

 Thread3

 Thread3 =
if __name__ == "__main__":
 Thread3 ==
thread_one = ThreadExample("Thread1", "*")
 Thread3 ===
thread_two = ThreadExample("Thread2", "-")
 Thread3 ====
thread_three = ThreadExample("Thread3", "=")
 Ending thread Thread3


thread_one.start()

thread_two.start()

thread_three.start()



print("Exiting the main program...")

? What happened here?

 Using join()
import threading
 Starting thread Thread1

 Thread1
class ThreadExample(threading.Thread):
 Thread1 *
def __init__(self, name, char):
 Thread1 **
threading.Thread.__init__(self)
 Thread1 ***
self.name = name
 Thread1 ****
self.char = char
 Ending thread Thread1

 Starting thread Thread2
def run(self):
 Thread2
print("Starting thread {} ".format(self.name))
 Thread2 -
print_chars(self.name, self.char)
 Thread2 --
print("Ending thread {} ".format(self.name))
 Thread2 ---

 Thread2 ----
def print_chars(thread_name, char):
 Ending thread Thread2
for i in range(5):
 Starting thread Thread3
print("{} {}".format(thread_name, i * char))
 Thread3

 Thread3 =

 Thread3 ==
if __name__ == "__main__":
 Thread3 ===
thread_one = ThreadExample("Thread1", "*")
 Thread3 ====
thread_two = ThreadExample("Thread2", "-")
 Ending thread Thread3
thread_three = ThreadExample("Thread3", "=")
 Exiting the main program...


thread_one.start()

thread_two.start()

thread_three.start()
 Note that now the main


thread_one.join()
 program ends AFTER all the
thread_two.join()
 threads finish
thread_three.join()



print("Exiting the main program...") !
 Determining current thread
import threading
 Starting thread Thread37
import random
 Thread37

 Thread37 *
class ThreadExample(threading.Thread):
 Thread37 **
def __init__(self, name = None, char = '+'):

Thread37 ***
threading.Thread.__init__(self)

if name is None:
 Thread37 ****
self.name = 'Thread' + str(random.randint(1, 100))
 Ending thread Thread37
else:
 Starting thread Thread4
self.name = name
 Thread4
self.char = char
 Thread4 -

 Thread4 --
def run(self):
 Thread4 ---
print("Starting thread {} ".format(self.name))
 Thread4 ----
print_chars(threading.current_thread().getName(), self.char)
 Ending thread Thread4
print("Ending thread {} ".format(self.name))

Starting thread Thread84


def print_chars(thread_name, char):
 Thread84
for i in range(5):
 Thread84 =
print("{} {}".format(thread_name, i * char))
 Thread84 ==

 Thread84 ===

 Thread84 ====
if __name__ == "__main__":
 Ending thread Thread84
thread_one = ThreadExample(char = "*")
 Exiting the main program...
thread_two = ThreadExample(char = "-")

thread_three = ThreadExample(char = "=")



thread_one.start()

thread_two.start()

thread_three.start()



thread_one.join()

thread_two.join()

thread_three.join()



print("Exiting the main program...")
 Thread synchronization

Thread synchronization is defined as a mechanism which ensures that two or more

concurrent processes or threads do not simultaneously execute some particular
program segment known as critical section.
 Synchronizing threads

import threading

We want to increment the variable

 count from two distinct threads.
count = 0


 For this, we define a function that
def increment_count(times):
 increments that variable and pass
global count

for i in range(1, times):
 it to the Thread constructor.
count += 1



return count


 But, wait! What happens with the



 output? We expect 2000000 but
if __name__ == "__main__":
 every time the result is different.


t1 = threading.Thread(target=increment_count, args=[1000001])

t2 = threading.Thread(target=increment_count, args=[1000001])



Not really what we wanted, right?
t1.start()

t2.start()



t1.join()

t2.join()



Output:
print(count)

1285489
Process finished with exit code 0

count = count + 1 1357161

Process finished with exit code 0
read count from memory
1249069
read count from memory again
Process finished with exit code 0
increment the value by 1
 Synchronizing threads

import threading



This time we add a lock around
count = 0
 the critical section represented by


def increment_count(times):
 the counter increment
global count

for i in range(1, times):
lock.acquire()

count += 1

lock.release()

return count
 This time, the result is always the

 same and the correct one.




if __name__ == "__main__":



lock = threading.Lock()



t1 = threading.Thread(target=increment_count, args=[1000001])

t2 = threading.Thread(target=increment_count, args=[1000001])



t1.start()

t2.start()



t1.join()
 Output:
t2.join()



print(count)
 2000000
Process finished with exit code 0

2000000
Process finished with exit code 0

2000000
Process finished with exit code 0
 Synchronizing threads

import threading

import time

We want to print some info about

 the running threads, first the info
class myThread (threading.Thread):

def __init__(self, threadID, name, counter):
 about one thread, then the info
threading.Thread.__init__(self)
 about a second thread.
self.threadID = threadID

self.name = name

self.counter = counter

def run(self):

print ("Starting " + self.name)

print_time(self.name, self.counter, 3)
 But, wait! What happens with the

 output? The print result is
def print_time(threadName, delay, counter):

while counter:
 scrambled between threads…
time.sleep(delay)

print ("%s: %s" % (threadName, time.ctime(time.time())))

counter -= 1
 Not really what we wanted, right?


threads = []



# Create new threads

thread1 = myThread(1, "Thread-1", 1)

thread2 = myThread(2, "Thread-2", 2)
 Output:


# Start new Threads

thread1.start()
 Starting Thread-1
thread2.start()
 Starting Thread-2


# Add threads to thread list
 Thread-1: Thu Nov 24 18:20:49 2016
threads.append(thread1)
 Thread-2: Thu Nov 24 18:20:50 2016
threads.append(thread2)



Thread-1: Thu Nov 24 18:20:50 2016
# Wait for all threads to complete
 Thread-1: Thu Nov 24 18:20:51 2016
for t in threads:
 Thread-2: Thu Nov 24 18:20:52 2016
t.join()

print ("Exiting Main Thread") Thread-2: Thu Nov 24 18:20:54 2016
Exiting Main Thread
 Synchronizing threads
import threading

import time


 This time one can see that the
class myThread (threading.Thread):

def __init__(self, threadID, name, counter):
 output looks much better.
threading.Thread.__init__(self)

self.threadID = threadID

self.name = name

self.counter = counter

def run(self):

print ("Starting " + self.name)

# Get lock to synchronize threads

threadLock.acquire()

print_time(self.name, self.counter, 3)

# Free lock to release next thread

critical section
threadLock.release()



def print_time(threadName, delay, counter):

while counter:

time.sleep(delay)

print ("%s: %s" % (threadName, time.ctime(time.time())))

counter -= 1



threadLock = threading.Lock()

threads = []



# Create new threads

thread1 = myThread(1, "Thread-1", 1)

thread2 = myThread(2, "Thread-2", 2)
 Output:


# Start new Threads

thread1.start()
 Starting Thread-1
thread2.start()
 Starting Thread-2


# Add threads to thread list

Thread-1: Thu Nov 24 18:26:59 2016
threads.append(thread1)
 Thread-1: Thu Nov 24 18:27:00 2016
threads.append(thread2)
 Thread-1: Thu Nov 24 18:27:01 2016


# Wait for all threads to complete
 Thread-2: Thu Nov 24 18:27:03 2016
for t in threads:
 Thread-2: Thu Nov 24 18:27:05 2016
t.join()
 Thread-2: Thu Nov 24 18:27:07 2016
print ("Exiting Main Thread")
Exiting Main Thread
 Queue data structure

A queue is an abstract data structure, open at both ends. One of the

ends is always used to insert data in the structure, while the other end is
used to extract data fro the structure.

The data insertion operation is called enqueueing, while the extraction

of the data from the queue is called dequeueing.
 Multithread Queues in Python

The Queue module provides a FIFO implementation suitable for multi-

threaded programming. It can be used to pass messages or other data
between producer and consumer threads safely. Locking is handled for the
caller, so it is simple to have as many threads as you want working with the
same Queue instance.

• get(): The get() removes and returns an item from the queue.

• put(): The put() adds item to a queue.

• qsize() : The qsize() returns the number of items that are currently in the
queue.

• empty(): The empty() returns True if queue is empty; otherwise, False.

• full(): the full() returns True if queue is full; otherwise, False.

 Multithread Queues

import queue
 threadList = ["Thread-1", "Thread-2", "Thread-3"]


import threading
 nameList = ["One", "Two", "Three", "Four", "Five"]

import time
 queueLock = threading.Lock()


 workQueue = queue.Queue(10)

exitFlag = 0
 threads = []


 threadID = 1

class myThread (threading.Thread):
 


 # Create new threads

def __init__(self, threadID, name, q):
 for tName in threadList:

threading.Thread.__init__(self)
 thread = myThread(threadID, tName, workQueue)

self.threadID = threadID
 thread.start()

self.name = name
 threads.append(thread)

self.q = q
 threadID += 1


 

def run(self):
 # Fill the queue

print ("Starting " + self.name)
 queueLock.acquire()

process_data(self.name, self.q)
 for word in nameList:

print ("Exiting " + self.name)
 workQueue.put(word)


 queueLock.release()

def process_data(threadName, q):
 

while not exitFlag:
 # Wait for queue to empty

queueLock.acquire()
 while not workQueue.empty():

if not workQueue.empty():
 pass

data = q.get()
 

queueLock.release()
 # Notify threads it's time to exit

print ("%s processing %s" % (threadName, data))
 exitFlag = 1

else:
 

queueLock.release()
 # Wait for all threads to complete

time.sleep(1) for t in threads:

t.join()

print ("Exiting Main Thread")

Starting Thread-1
Starting Thread-2
Starting Thread-3
Thread-1 processing One
Thread-2 processing Two
Thread-3 processing Three
Thread-1 processing Four
Thread-3 processing Five
Exiting Thread-1
Exiting Thread-2
Exiting Thread-3
Exiting Main Thread
 References

Mark Lutz - Learning Python, O’Reilly

Queue – A thread-safe FIFO implementationhttps://pymotw.com/2/

Queue/

Python 3 - Multithreaded programming https://

www.tutorialspoint.com/python3/python_multithreading.htm

Inside the Python GIL - http://www.dabeaz.com/python/GIL.pdf

Operating System Tutorial - http://www.w3ii.com/en-US/

operating_system/os_process_scheduling.html