Operating Systems and System

Programming
MIDORI SUGAYA
SHIBAURA INSTITUTE OF TECHNOLOGY
INFORMATION SCIENCE AND ENGINEERING.
1

2013/09/30
Course Structure
Place
5142 (5 Build. 1F, PC Training Room)
Course Schedule
Totally 14 sessions
First two sessions
Learn basic idea and concept of operating systems
Learn basic command for Linux
Install Linux for your computer (Using virtual machine)
The remaining sessions
Learn detailed mechanism of operating systems and
system call programming
Course Materials
http://www.sic.shibaura-it.ac.jp/~doly/os/
http://www.dlab.ise.shibaura-it.ac.jp/lectures.html
2

2013/09/30
Course Schedule
Week Topic
1-2 (OS) Introduction and Overview, Operating Systems Structure
(Includes what is system call)
(SysPro) Administrative, Basic Commands
3-5 (OS) Process Management Scheduling, Process and Threads,
Process Synchronization Deadlocks
(SysPro) Process, Thread, Synchronization, C and deep C
6-8 (OS) Memory management : Main memory and virtual memory,
File-System Interface,
(SysPro) Memory, File operations,
9-11 (OS) Process Communication
(SysPro) (1) Signal (2) Socket
12-13 (OS) Security, Protection
(SysPro) Security
3

2013/09/29
For Computer Science Students
In first two session
I think it is necessary to explain the summary of OS before the
system call programing.
For computer science students
Project1: Recompile kernel and install it.
Please recompile your kernel from kernel-source.
Download the original kernel
Do the lspci command to check your device
Project2: Research the sequence of system call in Linux
How to trap kernel mode
How to use the service of kernel
How to return user mode
Project3: Adding a System Call to the Linux Kernel, p.74
Getting Started
Building a New Kernel
Extending Kernel Source
Using the System Call From a User Program

4

2013/09/29
Course Materials
Required Text
Operating System Concepts, 8 and 9
th
edition
Silberschatz, Galvin, & Gagne
! Multiple Cheap Options exist!
Amazon/eBay
Operating Systems; Principles & Practice
Anderson & Dahlin
Main differences
More focus on typical modern kernels
and hardware
Less focus on historical systems
Stronger coverage of file systems
Weaker coverage of security
6

2013/09/30
Materials
7


Understanding UNIX/LINUX Programming: A Guide to Theory and
Practice, Bruce Molay, Prentice Hall; 1st version (2002/11/25), 11,200


2013/09/29
Basic Command
Login
Linux 1.2.13 (maya) (ttyp1)
maya login: doly
Password: *******
Logout
$exit
Read manual
$ man

8

2013/09/29
Todays Contents
Operating systems introduction
2013/09/29
9

Home ICT
Computing Environment
Health Care
Energy Saving
Public facilities
Electric generating facilities
Data Center
Super Computer
Cloud
Internet
Smart Phone
Display
Touch Panel
Network Communications
Smart Grid
Very Large Scale Computer
Embedded Systems
Programming
Education
Sensor Network
Robot
Home ICT
Expected Computing Environment
Health Care
Energy Saving
Public facilities
Electric generating facilities
Data Center
Super Computer
Cloud
Internet
Smart Phone
Display
Touch Panel
Network Communications
Smart Grid
Very Large Scale Computer
Embedded Systems
Programming
Education
Sensor Network
Robot
Behavior of user
Control Robots
Smart Grid
Advanced
Control through
the use of data
Home ICT
Expected Computing Environment
Health Care
Energy Saving
Public facilities
Electric generating facilities
Data Center
Super Computer
Cloud
Internet
Smart Phone
Display
Touch Panel
Network Communications
Smart Grid
Very Large Scale Computer
Embedded Systems
Programming
Education
Sensor Network
Robot
Behavior of user
Control Robots
Smart Grid
Advanced
Control through
the use of data

Year 2000-Ubiquitous
Year 2010-Smart GridCPS (Cyber Physical Computing)
basic knowledge of system architecture (operating systems) is always required

What operating systems Do
For Users, For Applications, etc..
Abstract view of the components of a computer systems
1
3

2013/09/29
What Operating Systems Do
Computer = HW + OS + Apps + Users
Provide abstraction of hardware to use its services
OS serves as interface between HW and (Apps & Users)
OS provides service for Apps & Users
Possible to write program independent from machine
architecture
Manages resources
Like a middle manager
OS provides the way to share the limited resources in computer
systems
Ordering ! Scheduling
Avoid resource abuse or monopolize ! Protection
1
4

2013/09/29
Computer-System Organization
What are all the parts, and how do they fit together?
1
5

2013/09/29
Computer-System Operation
Bootstrap program
Initialization of the system (from CPU registers to device
controllers).
Load operating system and start execution
$ dmesg
Interrupt
From hardware
Trigger an interrupt at any time by sending a signal to CPU, by
way of system bus
$ cat /proc/interrupts
From software
May trigger an interrupt by executing a special operation called
a system call
1
6

2013/09/29
Storage Structure
Main memory (RAM)
Programs must be loaded into RAM to run
Instructions and data fetched from RAM into registers
RAM is volatile
Medium size and speed
$ free
Other electronic (volatile) memory is faster, smaller, and more
expensive per bit:
Registers
CPU cache
$ cat /proc/cpuinfo (cache size)
Non-volatile memory (permanent storage) is slower, larger, and
less expensive per bit:
Solid-state disk
Magnetic disks
Optical disks
$ df -h
1
8

2013/09/29
Storage-device hierarchy
hierarchy
1
9

2013/09/29
I/O Structure
Typical operation involves I/O requests, direct memory access (DMA),
and interrupt handling.
How modern computer system works
$ cat /proc/dma
2
0

2013/09/29
Computer-System Architecture
Different Operating Systems for Different Kinds of Computer
Environments
Single-Processor Systems
One main CPU which manages the computer and runs user
apps.
Other specialized processors ( disk controllers, GPUs, etc. ) do
not run user apps.
Multiprocessor Systems
Increased throughput - Faster execution, but not 100% linear
speedup.
Economy of scale - They can share peripherals, disks, memory,
shared among processors.
Increased reliability If functions can be distributed properly
among several processers, a failure of a CPU will not halt the
system, only slows system, doesn't crash it.

2
1

2013/09/29
Multi-Processor Systems
Two types of multiprocessor systems in use today.
Asymmetric multiprocessing architecture
Each processor is assigned a specific task
Symmetric multiprocessing (SMP) architecture
Each processor performs all tasks within the operating systems
A dual core design with two cores places on the same machine.
2
2

2013/09/29
Clustered Systems
Another type of multiple CPU systems - Independent systems
Generally accepted definition:
shared common storage and connected by a high-speed LAN,
working together.
Clustering is usually used to provide high-availability service;
That is, service will continue even if one or more systems in the
cluster fail.
General structure of a clustered system
2
3

2013/09/29
Operating System Structure
Multiprogram
A single user can not keep either the CPU or the I/O devices bus at
all times.
The idea :
OS keeps several jobs in memory simultaneously
The job may have to wait for some task, such as an I/O
operation, to complete
Time-sharing (or multi-tasking)
Logical extension of multiprogramming
OS requires:
Memory management
Process management
Job scheduling
Resource allocation strategies
Swap space / virtual memory in physical memory
Interrupt handling
File system management
Protection and security
Inter-process communications
2
4

2013/09/29
Operating-System Operations
Modern operating systems are interrupt driven.
Events are always signaled by the occurrence of an interrupt or
trap (or an exception).
Trap: software generated interrupt caused by an error (invalid
memory access, etc..) or by a specific request from a user
program.
For each type of interrupt, interrupt handler determine what type
of action should be taken.
OS and users share the hardware and software resources.
Need make sure that an error in a user program could cause
problems.
E.g. if a process gets stuck in an infinite loop, this loop could
prevent the correct operation of many other programs, or even
operating system itself.
2
5

2013/09/29
Dual-Mode and Multimode Operation
User mode
when executing harmless code in user applications
Kernel mode( a.k.a. system mode, supervisor mode, privileged
mode )
when executing potentially dangerous code in the system
kernel.
Certain machine instructions ( privileged instructions ) can only
be executed in kernel mode.
Kernel mode can only be entered by making system calls. User
code cannot flip the mode switch.
Modern computers support dual-mode operation in hardware, and
therefore most modern OSes support dual-mode operation.

2
6

2013/09/29
Transition from user to kernel mode
User Kernel
User mode
Kernel mode (supervisor mode, system mode, privileged mode)
With the mode, we are able to distinguish between a task that is executed
on behalf of the OS or the user.
Hardware allows privileged instructions to be executed only in kernel mode.
Check $ time ./a.out
System calls provide the means for a user program to ask the operating system
on the user programs behalf.
2
7

2013/09/29
Process Management
A program in execution is a process. A program by itself is not a process.
passive entity, file stored on disk
A process
Active entity, a single-threaded process has one program counter specifying the
next instruction to execute.
A process needs certain resources to accomplish its task.
CPU time, memory, files, and I/O devices
These resources are either given to the process when it is created or allocated to it
while it is running.
Check $ top
All these processes can potentially execute concurrently by multiplexing the
CPU among then on a single CPU, for example.
An OS is responsible for the following tasks with regards to process
management:
Creating and deleting both user and system processes
Ensuring that each process receives its necessary resources, without interfering
with other processes.
Suspending and resuming processes
Process synchronization and communication
Deadlock handling
In system call programming section, we will develop a process.
2
8

2013/09/29
Memory Management
Main Memory
A repository of quickly accessible data shared by the CPU and I/O
devices.
The main memory is generally the only large storage device that
CPU is able to address and access directly access.
To improve both the utilization of the CPU and the speed of the
computers response to its users, general-purpose computers must
keep several programs in memory, creating a need for memory
management.
An OS is responsible for the following tasks with regards to memory
management:
Keeping track of which blocks of memory are currently in use, and
by which processes.
Determining which blocks of code and data to move into and out
of memory, and when.
Allocating and deallocating memory as needed. ( E.g. new,
malloc )

2
9

2013/09/29
Storage Management
To make the computer system convenient for users, the OS provides
a uniform, logical view of information storage.
The OS abstracts from the physical properties of its storage
devices to define a logical storage unit, the file. The OS maps
files onto physical media and access these files via the storage
devices.
OS maps files onto physical media and access these files via the
storage devices
An OS is responsible for the following tasks with regards tofile-
system management:
Creating and deleting files and directories
Supporting primitives for manipulating files and directories.
( open, flush, etc. )
Mapping files onto secondary storage.
Backing up files onto stable permanent storage media.
3
0

2013/09/29
File-System Management
Computers can store information on several different types of
physical media
Magnetic disk
Optical disk
Magnetic tape
Solid state disk etc..
Each medium is controlled by a device, that also has its own
unique characteristic.
Access speed, capacity, data-transfer rate, access method
3
1

2013/09/29
Directory Tree
Unix organize file abstraction as tree
provides directories and files. In this organization, user can find
unique file, use the command that move to the different tree node,
and read/write the contents of file.

Root level
/etc, /home, /binetc
User data /home/user
Operation command
ls list
cd change directory
pwd print working directory

3
2

/
etc home bin dev var usr
tmp spool
mail fax
2013/09/30
Caching
Caching is an important principle of computer systems.

Because cache has limited size, cache management is an
important design problem.
Careful selection of the cache size and of a replacement policy
can result in greatly increased performance.
3
3

Slower Storage
cache
2013/09/29
Performance of various level of storage
Caching
In a hierarchical storage structure, the same data may appear in
different levels of the storage system.
Data read in from disk follows a migration path from the hard
drive to main memory, then to the CPU cache, and finally to the
registers before it can be used, while data being written follows
the reverse path.
2013/09/29
3
4

I/O Systems
The I/O subsystem consists of several components:
A memory-management component that includes buffering,
caching, and spooling.
A general device-driver interface.
Drivers for specific hardware devices.
$ cd /dev/
$ ls l /dev/
2013/09/29
3
5

Protection and Security
Protection
involves ensuring that no process access or interfere with
resources to which they are not entitled, either by design or by
accident. ( E.g. "protection faults" when pointer variables are
misused. )
Security
involves protecting the system from deliberate attacks, either
from legitimate users of the system attempting to gain
unauthorized access and privileges, or external attackers
attempting to access or damage the system.

2013/09/29
3
6

Computing Environment
Mobile Computing
Computing on small handheld devices such as smart phones or
tablets. ( As opposed to laptops, which still fall under traditional
computing. )
May take advantage of additional built-in sensors, such as GPS,
tilt, compass, and inertial movement.
Typically connect to the Internet using wireless networking ( IEEE
802.11 ) or cellular telephone technology.
Limited in storage capacity, memory capacity, and computing
power relative to a PC.
Generally uses slower processors, that consume less battery
power and produce less heat.
The two dominant OSes today are Google Android and Apple
iOS.
2013/09/29
3
7

Distributed Systems
A collection of physically separate, consists of multiple, possibly
heterogeneous, computers connected together via a network and
cooperating in some way, form, or fashion.
Networks may range from small tight LANs to broad reaching WANs.
WAN = Wide Area Network, such as an international corporation
MAN =Metropolitan Area Network, covering a region the size of a
city for example.
LAN =Local Area Network, typical of a home, business, single-site
corporation, or university campus.
PAN = Personal Area Network, such as the bluetooth connection
between your PC, phone, headset, car, etc.
Network access speeds, throughputs, reliabilities, are all important
issues.
OS view of the network may range from just a special form of file
access to complex well-coordinated network operating systems.
Shared resources may include files, CPU cycles, RAM, printers, and
other resources.

2013/09/29
3
8

Client-Server Computing
A defined server provides services ( HW or SW ) to other systems
which serve as clients. ( Technically clients and servers are
processes, not HW, and may co-exist on the same physical
computer. )
A process may act as both client and server of either the same
or different resources.
Served resources
may include disk space, CPU cycles, time of day, IP name
information, graphical displays ( X Servers ), or other resources.

2013/09/29
3
9

General structure of a client-server system
Peer-to-Peer Computing
There is no clear "leader" or overall organization.
Any computer or process on the network may provide services
to any other which requests it.
May employ a central "directory" server for looking up the location
of resources, or may use peer-to-peer searching to find resources.
E.g. Skype uses a central server to locate a desired peer, and
then further communication is peer to peer.

2013/09/29
4
0

Peer-to-peer system with no centralized service
Cloud Computing
Delivers computing, storage, and applications as a service over a network.
May contain thousands of physical computers, millions of virtual ones,
and petabytes of total storage.

2013/09/29
4
1

Cloud Computing
Types of cloud computing:
Public cloud - Available to anyone willing to pay for the service.
Private cloud - Run by a company for internal use only.
Hybrid cloud - A cloud with both public and private
components.
Software as a Service - SaaS - Applications such as word
processors available via the Internet
Platform as a Service - PaaS - A software stack available for
application use, such as a database server
Infrastructure as a Service - IaaS - Servers or storage available
on the Internet, such as backup servers, photo storage, or file
storage.
Service providers may provide more than one type of service

2013/09/29
4
2

Real-time Embedded Systems
Embedded into devices
such as automobiles, climate control systems, process control,
and even toasters and refrigerators.
May involve specialized chips, or generic CPUs applied to a
particular task. ( Consider the current price of 80286 or even
8086 or 8088 chips, which are still plenty powerful enough for
simple electronic devices such as kids toys. )
Process control devices require real-time ( interrupt driven )
OSes. Response time can be critical for many such devices.
2013/09/29
4
4

Open-Source Operating Systems
Open-Source software
is published ( sometimes sold ) with the source code, so that
anyone can see and optionally modify the code.
is often developed and maintained by a small army of loosely
connected often unpaid programmers, each working towards
the common good.
Critics argue that open-source SW can be buggy
but proponents counter that bugs are found and fixed quickly,
since there are so many pairs of eyes inspecting all the code.
Open-source operating systems are a good resource
for studying OS development, since students can examine the
source code and even change it and re-compile the changes.
2013/09/29
4
5

Linux
Developed by Linus Torvalds in Finland in 1991 as the first full
operating system developed by GNU.
Many different distributions of Linux have evolved from Linus's
original,
including RedHat, SUSE, Fedora, Debian, Slackware, and
Ubuntu, each geared toward a different group of end-users and
operating environments.
To run Linux on a Windows system using VMware, follow these steps:
Download the free "VMware Player" tool from
http://www.vmware.com/download/player and install it on your
system
Choose a Linux version from among hundreds of virtual machine
images at http://www.vmware.com/appliances
Boot the virtual machine within VMware Player.

2013/09/29
4
6

Utility
The free software movement is gaining rapidly in popularity, leading
to thousands of ongoing projects involving untold numbers of
programmers.
Sites such as http://freshmeat.net/ and http://distrowatch.com/
provide portals to many of these projects.

2013/09/29
4
7

BASIC COMMAND
2013/09/30
4
8

Basic file operations
Start terminal
$ xterm
Read the contents of file
$ cat shopping-list
Milk
Apple
Jam
$ more longfile
Copy file
$ cp shopping-list last-week.list
Remove file
$ rm old.data shopping.june2012
Move file (change the name)
$ mv prog1.c first_program.c
4
9

2013/09/30
Basic directory operations
Create and delete directory
$ mkdir /home/doly
$ mv first_program.c mycode
Lets try
Move a file to a different directory
Create a file under your home directory
5
0

2013/09/30
File Attribute
Control file access through correct use of file and directory
permissions and ownership
$ id
uid=505(doly) gid=20(staff) groups=20(staff),
$ ls l file.txta
rwx rwx rwx
User, Group, Other
r: read, w: write, x: execute
$ touch test
$ ls -l test
-rw-r--r-- 1 doly staff 0 3 21 11:17 test
5
1

2013/09/30
File operations
Change ownership
Permit the execution privilege of a file for other
$ chmod a+x test
ls -l test
-rwxr-xr-x 1 doly staff 0 3 21 11:17 test
Grant execution privilege for all users of test file
$ chmod g+w,g-r test
ls -l test
-rwx-wxr-x 1 doly staff 0 3 21 11:17 test



5
2

2013/09/30
KERNEL DATA STRUCTURE
2013/09/30
5
3

Kernel Data Structures
Lists
2013/09/30
5
4

Singly linked list
Doubly linked list
Circularly linked list
Kernel Data Structures
Hash Function and Maps
Bitmaps
A string of 1s and 0s used to keep track of the boolean state of a
collection of objects, such as the free state of blocks on a disk or
pages in memory..
2013/09/30
5
5