A New Model of Abstraction for Operating System Design
Gregor Kiczales, Marvin Theimer, and Brent Welch
( gregor,theimer,welch} @parc.xerox.com
Xerox PARC
3333 Coyote Hill Road
Palo Alto, CA 94304
Abstract
The historica h i s of operating systems work has been
the concept of black-box abstraction. But, we believe that
this traditional notion of abstractionfails to support emerg-
ing practice in the operating system community where,
more and more, we seem to want to give clients access to
previously internal aspects of the implementation. We
present a new model of abstraction, dual-abstraction, and
show how to think about existing work under that model. We
also discuss some issuesf o r future work that the new model
makes evident.
1 Introduction
We are taught, from the early days of our education as
software engineers, that the concept of abstraction is funda-
mental to all the systems we build. While it is difficult to
define this concept precisely, we are all familiar with the
basic notion that we should design “clean, elegant interfaces
to a system that hide the implementationdetails” from the
client. The underlying idea is one of reuse: by making the
abstraction interface general, and isolating the client from
the details of the implementation,we hope to make it possi-
ble for a large number of users to use the same abstraction
without caring about the implementation.This principle has
been at the very heart of the notion of operating systems,
where the original goals were to provide a library of useful
base functionality and manage the interaction among vari-
ous users contending for the resources of the machine.
While some would be reluctant to come right out and say
it, it is evident that much of the OS community no longer
believes in this model of abstraction. We have recognized
that not only is it not possible to hide the implementation
from the client--the performance characteristics of the
implementation will always show through--but also it isn’t
a good idea to prevent the client from having access to those
details--for performance reasons they would often like to
adjust them.
346
0-8186-3015-9/92 $03.00 0 1992 IEEE
The intuition behind much of the current work in the OS
community can be summarized as follows:
The application programmer, and not the operat-
ing system implementor, is in the best position to
determine just how the operating system should be
implemented--it is only with an understanding of the
behavior of the application that an informed judge-
ment can be made about how to make the perfor-
mance trade-offs in the OS. While a clever OS
implementor might be able to provide a good
generic implementation, there will inevitably be
applications whose “usage patterns” are such that a
different implementation will be desired.
The task is to find a way to “open the OS imple-
mentation up” so that clients can selectively replace
or adjust those parts of the implementation that don’t
meet their perfonnance needs.
h writing this paper, we have three goals: First, to show
that current work in the OS community is, in fact, driven by
this intuition that black-box abstraction is no longer viable.
Second, to present an initial sketch of a new model of
abstraction, and show that it is useful to think of existing
work in these terms. Third, to see what that sketch has to say
about longer term work, and what the real difficulties will
be in making operating systems that do a radically better job
of being adjustable by their clients.
2 OperatingSystems Already Provide Custom-
ized Implementations
That the operating system community has long realized
that implementationscannot be opaque black boxes is evi-
dent in a variety of work. One of the primary examples is
virtual memory management.
Most applications see virtual memory as a traditional
operating system abstraction whose implementation is
opaque and works adequately. However, some applications
exhibit patterns of page access that interact badly with the
page replacement strategies used by generic implementa-
tions. The response of operating system designers has been
to add parameters or operations to the VM interface that
allow applications to declare their access patterns for
selected regions of memory. An example is the madvise
system call provided in Berkeley UNIX.
Adding these parameters is based on the recognition that
a single, fixed implementation won’t satisfy all prospective
users. Unfortunately, it itself suffers from the same prob-
lem--there is no reason to believe that n fixed implementa-
tions (for any reasonably small value of n) will satisfy all
prospective users. The response of some systems, such as
Mach, has been to go to a more general approach in which
the VM implementation is partially supplied by the applica-
tion in the form of an external pager process[l]. The exter-
nal pager implements the protocol between the VM system
and the backing storage for a memory object. In this case
the pager runs as a separate process in its own address
space, not as part of the operating system kernel.
These examples illustrate two basic approaches to oper-
ating system custornization: selection from a set of kernel-
resident algorithms, or the use of upcall to invoke a user-
level implementation of a protocol used by the kernel. The
first approach is less flexible, but more efficient because the
customizations run in the kernel’s protection domain. The
user-level approach provides wide flexibility, but at the cost
of increased communication overhead and additional code
in the kernel to protect against errant server processes. Both
of these approaches reflect the safety requirements of the
operating system in which customizations by different
applications should not interfere with one another. This ten-
sion between customization and safety is an important point
about which we will have more to say later.
3 A Dual-Abstraction Approach to Design
While we claim that the traditional approach of interfaces
backed by opaque implementations is insufficient to meet
all demands placed upon it, it is important to remember the
benefits that have been gained by that approach. Most appli-
cations are perfectly happy with the performance they get
from the generic implementation of a functional abstraction
and are in fact simpler for not having to deal with the details
of its implementation. But, some clients simply cannot
make due with the generic implementation. The problem
with the traditional model of abstraction--in which imple-
mentation details are not only hidden from, but also entirely
off limitsto the client--is that it has nothing to say about the
second situation.
In practical terms, the result is that in order to support a
small set of special-purpose applications one must currently
supply an entirely new implementation of the functionality.
That is, an application that is unhappy with 10% of the
implementation must do 100% of the work of reimplement-
341
ing the desired functionality. It is to avoid this drastic reim-
plementation overhead that operating systems research has
increasingly focused on how to expose the generic imple-
mentation of an abstraction in order to allow its customiza-
tion.
The primary contribution of this paper is to provide a
framework in which most of the benefits of the traditional
approach to systems design can be maintained while mak-
ing implementations “open” in a principled manner to those
applications that need it. In particular, we propose a “dual
abstraction” approach in which the behavior of the system
is provided in a traditional way by the first, or base-level
abstraction, and the second, or meta-level, interface makes
it possible to customize the implementation of that func-
tionality. The advantage of this approach is that clients of
the abstraction that are happy with its generic implementa-
tion can simply use the base-level interface: it is no more
complex than it would have been under the traditional
story. But when the performance of the generic implemen-
tation becomes unacceptable to a particular client, that cli-
ent will be able to delve into the implementation in a
principled way, using the meta-level interface.
This approach has not been derived “from thin air.” In
fact, we have found that similar issues exist in the domain
of high-level programming languages. In particular, in the
context of object-oriented programming languages, we
have found that it is possible to open the implementation in
a principled way using a technology we call metaobjectpro-
tocols [2]. The dual-abstraction framework is an outgrowth
of the work on metaobject protocols.
We have developed two general principles which form a
basis for this approach: scope control and incrementality.
Scope control means that customizations of an implementa-
tion should only affect the clients that desire it; other clients
should still be able to use the generic implementation or
their own customizations thereof. Incrementality means
that customizations should (ideally) require effort propor-
tional to the amount of change desired rather than requiring
a complete or near-complete rewrite of the generic imple-
mentation.
In our previous work on metaobject protocols, we
achieved scope control and incrementality with a straight-
forward application of object-oriented techniques. The
class of an object controls its behavior, providing scope
control. A new behavior can be added incrementally by
1. In fact, we believe that under this approach base-level
interfaces should be simpler than under the traditional
approach. This is because all implementationcustomi-
zation issues, some of which currently appear in our
abstractions, will disappear into the meta-level inter-
face.
 defining a subclass and overriding appropriate methods.
However, in the operating system arena it takes more
than this simple application of object-oriented techniques.
Scope control translates primarily into the following two
issues: how to reconcile conflicts between multiple applica-
tion-specific modules as well as the need to prevent any one
application from imposing its decisions on others against
their will. Typically these issues are addressed by pulling
the application-specific functions out of the operating sys-
tem kernel and putting them into their own address space
(e.g., an external pager process) or into the application’s
address space (e.g., a thread package). The operating sys-
tem remains in charge of mediating access to physical
resources (e.g., page frames and processors). However,
communication between address spaces is expensive and
hence the cost of interaction between the application-spe-
cific parts of a system and the parts that meditate between
applications has dominated the design of systems so far
131141.
The use of separate protection domains to achieve scope
control can confict with the goal of incrementality because,
in practice, large pieces of functionality are moved into sep-
arate address spaces. For example, Mach clients can either
supply their own external pager or use the one in the Mach
kernel; but they cannot take the generic kernel VM imple-
mentation and make incremental changes to it. Providing
the ability to incrementally modify part or all of an operat-
ing system facility implies that the entire implementation
must be placed in suitable protected domains for safety
sake. But the overhead of communication among the vari-
ous protected parts may neutralize the very performance
gains that the customization seeks to gain. The challenge is
to m e up with new designs for the traditional operating
system abstractions that are open to incremental customiza-
tion.
The contribution of the dual-abstraction approach is to
provide a principled way for separating use of functionality
from control over implementation. Clients must obey a
meta-level interface to the abstraction that addresses both
their desire for custonization as well as system-wide con-
cerns about protection and scope control among multiple
clients. Three examples are discussed below: scheduler
activations [5], the X-kernel network protocols [6],and file
system protocols.
Scheduler activations augment the traditional notion of a
thread with a feedback channel to the application. This
allows the application to react to the kernel’s scheduling
decisions that result from blocking I/O, page faults, and pre-
emptive scheduling. In response, the application can make
its best decision about which of its own threads should be
scheduled on the remainiig scheduler activations. This is an
example of a basic kernel abstraction, threads, that has been
348
redesigned to work more effectively with an application-
specific thread implementation. In this example, the dual-
abstraction framework would distinguish between the base
functionality, threads, and the meta-level functionality of
scheduling control.
The X-kernel is designed to support a wide variety of net-
work protocols. The kernel provides a rich set of protocol
building blocks such as client-server binding, “blast” proto-
cols for high throughput, retransmission for reliable &ins-
port, and selection among addressing formats. The actual
protocols are configured by specifying a graph that
describes the interconnections of the building blocks. The
graphical specification is essentially a rneta-level interface
about the protocol implementations.
File system design is another area where it can be bene-
ficial to make a distinction between base and meta-level
functionality.The basic file system interface is one that pro-
vides openclose-read-write access to files. Implementation
issues such as buffering, cache consistency[7], server repli-
cation, and disconnected operation [8], are refinements that
most applications do not want to have to deal with. How-
ever, some sophisticated clients may want control over
these features. A meta-level interface that is cleanly Sepa-
rated from the base file system interface can provide control
to these clients without disturbing the clients of the basic
interface.
4 Future Work
The dual abstraction model presented in the previous sec-
tion provides a natural framework for thinking about imple-
mentations which ‘‘open themselves up” to their clients. It
is consistent with the familiar principle that behavior and
implementation should be separated--the base level abstrac-
tion provides the functionality, the meta-level abstraction
can be used to adjust the implementation of that functional-
ity. A conversation between the human designerfimplemen-
tor and client of an operating system might also reflect this
split into dual abstractions. Much of the time they would
just talk about the functionality that would be provided. At
other times they would rely on their expertise to “go meta”
and talk about crucial performance issues. Implicit in this
approach is the fact that considerable experience with sys-
tem implementations is required before a good meta-level
interface into an implementation can be devised.
One problem area with dual abstractions is that there are
some concepts that are easily expressed at the rneta-level
but are not simply manifest at the base-level. These con-
cepts are “global” concepts that affect all parts of a base-
level implementation. Examples include concurrency con-
trol, automatic storage management, and exception han-
dling. It is easy to specify the meta-level concept of how
exception events should be handled in a system. However,
the implementation of exception handling in a system typi-
cally permeates almost every module of the system. Imag-
ine changing fmm a policy in which callers of modules
must handle exceptions to a policy in which the modules
themselves are responsible for handling exceptions; it is
almost certain that one would need to make massive
changes throughout the implementation of the entire sys-
tem. With concurrency control, it is common to provide
locking primitives that become no-ops in a single-threaded
environment. However, this approach doesn’t help trans-
f m a single-threaded implementation in to a multi-
threaded implementation. Similarly, a program written with
explicit free calls can easily be converted to a run with a gar-
bage collector, while the converse is not at all easy. These
examples show that even though a concept may be easy to
express at the meta-level it may still be difficult to realize in
the base level implementation.
5 Concluding Remarks
We have presented a new framework for thinking about
how to open up operating system implementations in a pM-
cipled manner. Our framework is at a conceptually high-
level: dual abstractions replace the traditional notion of
black box abstractions. The contribution of this approach is
to provide a framework for organizing systems in a princi-
pled way. As part of this new framework, we suggested two
general design principles: scope control and incrementality.
We have shown how existing work in the community can be
productively considered in this new framework.
One source of related ideas we have not yet mentioned is
the work on reflective operating systems [9]. This work has
the same intellectual roots [lo] as the metaobject protocol
work which led us to our dual abstraction framework. In the
next few years, we expect a productive synthesis of the
reflective operating system work with the ideas we have
sketched in this paper.
As a 6nal point, the discussion in this paper provides an
interesting perspective on the current nature of software. It
suggests in fact that software is anything but “soft.” While
it is true that it is more malleable than physical artifacts, the
difficulty of providing implementations that are tunable by
their clients makes it clear that we don’t yet know how to
take real advantage of that malleability.
349
6 References
M.Young,A. Tevanian, R.Rashid, D. Golub, J.Eppinger.
J. CHew, W.Bolosky, D. Black and R. B m n , ” The
Duality of Memory and Communication io the
Implementationof a Multiprocessor operating System”,
Proc. 11th Symp. on Operating System Prin., operating
Systems Review 21.5 (Nov. 1987). 63-76.
G.Kiczales. J. des Rivieres, D. G.Bobrow, “The Art of the
Metaobject Rotocol,” MlT Ress 1991.
D. Clark, The Structuringof Systems Using Upcalls, Proc.
of the 10th Symp. on Operating System Prin., Operating
Systems Review 19,s (Dec. 1985). 171-180.
B. N.Bershad,T. E. Anderson, E. D. Lazowska and H. M.
Levy, User-LevelInterprocessCommunicationfor Shared
Memory Mdtiprocessors,ACM Transactions on
Computer Systems, 9,2 (May 1991). 175-198.
T Anderson, B. Bershad, E. L a z o w h H. Levy,
“Scheduler Activations: Effective Kernel Support for the
User-Level Management of Parallelism. Proc. of the 13th
Symp. on Operating System Prin., Operating Systems
Review, Oct. 1991,95-109.
L.Peterson, N. Hutchinson, S. O’Mally, M. Abbot6 “RPC
in the x-Kernel: Evaluating New Design Techniques”,
Proc. of the 12th Symp. on Operating System Prin.,
Operating Systems Review, Dec. 1989,91-101.
M. Nelson, B. Welch and J. Ousterhout, Caching in the
SpriteNetwork File System,ACM Transactions Computer
Systems 6.1 (Feb. 1988), 134-154.
James Kistler and M. Satyanarayanan,“Disconnected
Operation in the Coda File System.”, Proc. 13th Symp. on
Operating System Prin., Operating Systems Review 25.5
(October 19!91),213-225.
Y, Yokote, A. Mitsuzawa, N. Fujinami, M. Tokoro,
“Reflective Object Management in the Muse Operating
System”, Proc. of the 2nd International Workshop on
Object-Orientation in Operating Systems (Oct 1991), 16-
23.
Brian C. Smith “Reflectionand Semantics in a Procedural
Language”, PhD Thesis, MIT, January 1982.