Distributed Systems Principles and Paradigms

Christoph Dorn
Distributed Systems Group, Vienna University of Technology c.dorn@infosys.tuwien.ac.at http://www.infosys.tuwien.ac.at/staff/dorn Slides adapted from Maarten van Steen, VU Amsterdam, steen@cs.vu.nl

Chapter 11: Distributed File Systems

Contents
Chapter 01: Introduction 02: Architectures 03: Processes 04: Communication 05: Naming 06: Synchronization 07: Consistency & Replication 08: Fault Tolerance 09: Security 10: Distributed Object-Based Systems 11: Distributed File Systems 12: Distributed Web-Based Systems 13: Distributed Coordination-Based Systems
DS WS 2013 2 / 14

Distributed File Systems

General goal Try to make a le system transparently available to remote clients.
1. File moved to client Client Server Client Server Old file New file Requests from client to access remote file 2. Accesses are done on client

File stays on server

3. When client is done, file is returned to

Remote access model

Upload/download model

DS WS 2013

3 / 14

Example: NFS Architecture

NFS NFS is implemented using the Virtual File System abstraction, which is now used for lots of different operating systems.
Client System call layer Server System call layer

Virtual file system (VFS) layer Local file system interface

Virtual file system (VFS) layer Local file system interface

NFS client

NFS server

RPC client stub

RPC server stub

Network

DS WS 2013

4 / 14

Example: NFS Architecture

Essence VFS provides standard le system interface, and allows to hide difference between accessing local or remote le system. Question Is NFS actually a le system?

DS WS 2013

5 / 14

NFS File Operations

Oper. Create Create Link Symlink Mkdir Mknod Rename Remove Rmdir Open Close Lookup Readdir Readlink Getattr Setattr Read Write
DS WS 2013

v3 Yes No Yes Yes Yes Yes Yes Yes Yes No No Yes Yes Yes Yes Yes Yes Yes

v4 No Yes Yes No No No Yes Yes No Yes Yes Yes Yes Yes Yes Yes Yes Yes

Description Create a regular le Create a nonregular le Create a hard link to a le Create a symbolic link to a le Create a subdirectory Create a special le Change the name of a le Remove a le from a le system Remove an empty subdirectory Open a le Close a le Look up a le by means of a name Read the entries in a directory Read the path name in a symbolic link Get the attribute values for a le Set one or more le-attribute values Read the data contained in a le Write data to a le
6 / 14

Cluster-Based File Systems

Observation With very large data collections, following a simple client-server approach is not going to work for speeding up le accesses, apply striping techniques by which les can be fetched in parallel.
File block of file a File block of file e

a a a b

b b

c c c d

d d

e e e

Whole-file distribution

a c d e

a c d

b c e

a d

b e

DS WS 2013

File-striped system 7 / 14

Example: Google File System

file name, chunk index GFS client contact address Instructions Chunk ID, range Chunk server Chunk data Linux file system Linux file system Linux file system Chunk server Chunk server Chunk-server state Master

The Google solution Divide les in large 64 MB chunks, and distribute/replicate chunks across many servers: The master maintains only a (le name, chunk server) table in main memory minimal I/O Files are replicated using a primary-backup scheme; the master is kept out of the loop
DS WS 2013 8 / 14

File sharing semantics

Problem When dealing with distributed le systems, we need to take into account the ordering of concurrent read/write operations and expected semantics (i.e., consistency).
Original file Single machine a b Process A a b c 3. Read gets "ab" Client machine #2 Process B a b Process B a b Client machine #1 a b Process A a b c 1. Read "ab"

2. Write "c"

File server

1. Write "c"

2. Read gets "abc" (a)

DS WS 2013

9 / 14

(b)

File sharing semantics

Semantics
UNIX semantics: a read operation returns the effect of the

last write operation can only be implemented for remote access models in which there is only a single copy of the le Transaction semantics: the le system supports transactions on a single le issue is how to allow concurrent access to a physically distributed le Session semantics: the effects of read and write operations are seen only by the client that has opened (a local copy) of the le what happens when a le is closed (only one client may actually win)

DS WS 2013

10 / 14

Example: File sharing in Coda

Essence Coda assumes transactional semantics, but without the full-edged capabilities of real transactions. Note: Transactional issues reappear in the form of this ordering could have taken place.
Session S A Client Open(RD) Server Open(WR) Client Session S B Time File f Close File f Invalidate Close

DS WS 2013

11 / 14

Consistency and replication

Observation In modern distributed le systems, client-side caching is the preferred technique for attaining performance; server-side replication is done for fault tolerance. Observation Clients are allowed to keep (large parts of) a le, and will be notied when control is withdrawn servers are now generally stateful
1. Client asks for file Client 2. Server delegates file Server Old file Local copy 3. Server recalls delegation Updated file 4. Client sends returns file

DS WS 2013

12 / 14

Example: Client-side caching in Coda

Session S A Client A Open(RD) Invalidate (callback break) File f Open(WR) Close Session S B Session S B Close Open(RD) File f OK (no file transfer) Close Session SA

Server

File f

Open(WR) Client B

Close Time

Note By making use of transactional semantics, it becomes possible to further improve performance.
DS WS 2013 13 / 14

Example: Server-side replication in Coda

Server S1 Server S3

Client A

Server S2

Broken network

Client B

Main issue Ensure that concurrent updates are detected: Each client has an Accessible Volume Storage Group (AVSG): is a subset of the actual VSG. Version vector CVVi (f )[j ] = k Si knows that Sj has seen version k of f . Example: A updates f S1 = S2 = [+1, +1, +0]; B updates f S3 = [+0, +0, +1].
DS WS 2013 14 / 14