Data Development with Microsoft ® technologies

ET
.N

Volume 4 / Issue 3

Microsoft Talks Data!

Link up with LINQ
ADO.NET Data
Services
Microsoft Sync
Framework
SQL Server 2008
ODBC Rocks!

www.code-magazine.com
US $ 5.95 Can $ 8.95
TABLE OF CONTENTS

Features
5 Welcome Letter 71 The Data Dude Meets Team Build
Gert Drapers
6 An Entity Data Model for Relational Data Part I:
Defining the Entity Data Model 76 XML Tools in Visual Studio 2008
Michael Pizzo Stan Kitsis

10 An Entity Data Model for Relational Data Part II: 80 ODBC Rocks!
Mapping an Entity Data Model to a Relational Chris Lee
Store
Michael Pizzo 84 Tesla: Democratizing the Cloud
Brian Beckman and Erik Meijer
16 Programming Against the ADO.NET Entity
Framework
Departments
Shyam Pather

22 Rich Query for DataSet – An Introduction to

LINQ to DataSet 50 CoDe Compilers
Andrew Conrad
57 Advertisers Index
26 LINQ to Relational Data: Who’s Who?
Elisa Flasko

32 Browsing Windows Live Expo with LINQ to XML

David Schach

38 ADO.NET Data Services

Mike Flasko and Elisa Flasko

42 Caching with SQL Server Compact and the

Microsoft Sync Framework for ADO.NET
Steve Lasker

52 Introducing the Microsoft Sync Framework:

Next Generation Synchronization Framework
Moe Khosravy

56 What’s New in SQL Server 2008?

Anthony Carrabino

58 Programming SQL Server 2008

Vaughn Washington

63 Use SQL CLR 2.0—Advancing CLR Integration in

SQL Server 2008
José Blakeley and Christian Kleinerman

68 Visual Studio 2008: RAD Gets RADer

Jonathan Wells

US subscriptions are US $29.99 for one year. Subscriptions outside the US pay US $44.99. Payments should be made in US dollars drawn on a US
bank. American Express, MasterCard, Visa, and Discover credit cards are accepted. Bill me option is available only for US subscriptions. Back issues
are available. For subscription information, email subscriptions@code-magazine.com or contact customer service at 832-717-4445 ext 10.
Subscribe online at www.code-magazine.com
CoDe Component Developer Magazine (ISSN # 1547-5166) is published bimonthly by EPS Software Corporation, 6605 Cypresswood Drive., Suite
300, Spring, TX 77379. POSTMASTER: Send address changes to CoDe Component Developer Magazine, 6605 Cypresswood Drive., Suite 300,
Spring, TX 77379.

4 Table of Contents www.code-magazine.com


The Data Programmability Team
Dear Reader,
It’s a great time to be working with data.
As we take stock of how our customers use
information today, we see data being pulled
from the Web, our desktops, departmental,
workgroup and enterprise servers, and then
surfacing in all kinds of different form fac-
tors. The sheer quantity of data available in
systems is already mind-boggling and grow-
ing at a tremendous rate. At Microsoft, espe-
cially in the Data Programmability team, we
are ridiculously excited to be working at the
center of this storm. The authors of these
articles will show you new innovations and
best practices across the spectrum. These
experts will show you new techniques and
components for handling the myriad data
shapes, locations, and sources that develop-
ers deal with on a regular basis. This edition
of CoDe Focus offers us the opportunity
to tell the story of our vision for the data
platform as it stands today and in the near
future. Of course it doesn’t stop there. Your
thoughts, critical feedback, and participa-
tion in Community Technology Previews
will help us all realize that vision.
“Data everywhere, usable by anyone at any
time.”—That’s our grand mission: provide
the best programming patterns, infrastruc-
ture, and tools for developing data-driven
applications for Windows, .NET, and Mi-
crosoft SQL Server.
www.code-magazine.com
I invite you to take a walk with us across
the techniques, tools, and components in
this issue. We’ll be looking at our native
data access technologies where we con-
tinue to innovate in raw speed and power
to ensure the fattest and fastest pipe to da-
tabases via ODBC. We’ll show you some
of the innovation we have been doing in
XML Tools to support building applica-
tions that manipulate and store XML and
work with schemas via XSD. We’ll dive
into the .NET Framework where we have
a bucket full of great stuff. Language In-
tegrated Query (LINQ) offers a huge step
forward in writing understandable and
approachable queries of all sorts in pro-
gramming languages. We’ll tour LINQ so-
lutions for XML, SQL, Entities, and Data-
Set. Data access APIs are not a complete
panacea, of course, and we’ve got a big
down payment for model-driven develop-
ment with data. We are super excited to
announce the release of the ADO.NET
Entity Framework and the foundational
Entity Data Model, which drives a set of
core data programming experiences now
and into the future. We are pleased to
show the Entity Designer that provides an
exciting developer experience for creating
conceptual models in the EDM that can
be used directly with the ADO.NET Entity
Framework.
ONLINE QUICK ID 0712012
As Web application technologies evolve,
the work we are doing evolves as well; this
includes areas we are currently working in
such as ADO.NET Data Services and tier
splitting.
Two trends stand out as harbingers of radi-
cal change: the proliferation of mobile com-
puting devices and near-ubiquitous wireless
network connectivity. Together they open
up a world of possibilities for new applica-
tions. The one-two combo we’ll show you
with the new release of SQL Server Com-
pact and Microsoft Sync Framework can
unshackle your applications from the desk-
top office and get your customers dancing
in the aisles.
To get involved, try stuff out, and to learn
more, come visit us on the Web at http://
msdn.com/data and then join the conversa-
tions starting from http://blogs.msdn.com/
data.
So let’s take that stroll—if you’re ready!
Sincerely,
Samuel Druker
General Manager, Data Programmability
Microsoft Corporation
http://msdn.com/data
Welcome 5
ONLINE QUICK ID 0712022
Michael Pizzo
Michael.Pizzo@microsoft.com
Michael Pizzo has worked for
over 17 years in the design
and delivery of data access
solutions and APIs at Microsoft.
Michael first got involved in data
access as a Program Manager
for Microsoft Excel in 1987,
integrating Microsoft’s flagship
spreadsheet product with
relational data. This led to his
involvement in the design and
delivery of ODBC, along with the
ODBC-based Microsoft Query
Tool shipped with Microsoft
Office. During the design of
ODBC, Michael was active in the
standards organizations, sitting
as Chair for the SQL Access
Group, working with X/Open
on the CAE specification for
“Data Management: SQL-Call
Level Interface (CLI)”, serving
as Microsoft’s representative
to the ANSI X3H2 Database
Committee, and as an elected
ANSI representative to the
ISO committee meetings that
defined and adopted Part 3 of
the ANSI/ISO SQL specification
for a call-Level Interface
(SQL/CLI). Following ODBC,
Michael was a key designer
and driver of Microsoft’s OLE
DB API for componentized
data access within a COM
environment, and later owned
the design and delivery of ADO.
NET version 1.0. He is currently
a Principle Architect in the
Data Programmability Team at
Microsoft, contributing to the
architecture and design of the
next version of ADO.NET and core
building block for Microsoft’s
exciting new data platform; The
ADO.NET Entity Framework.
6 An Entity Data Model for Relational Data Part I: Defining the Entity Data Model
An Entity Data Model for
Relational Data Part I:
Defining the Entity Data
Model
Microsoft’s Entity Data Model allows you to define an application-
oriented view of your data consistent with how you reason about
that data.
Part I of this article describes the Entity Data Model and how it enables you to
represent real-world concepts in a way that makes relationships between related
pieces of data more explicit and easier to query, navigate, and consume than
through the traditional relational database model. Part II of the article discusses
how Microsoft’s ADO.NET Entity Framework provides a flexible mapping of an
application-oriented conceptual schema in terms of the Entity Data Model to existing
relational database schemas. Shyam Pather’s article, “Programming Against
the ADO.NET Entity Framework” completes the picture by describing the actual
programming model and API exposed by the framework.
T
he world around us is infinitely complex. Dr. Chen compares the relational and entity-relation-
Describing that world in a way we can rea- ship models, saying that the entity-relationship model
son about requires us to break it into simpler adopts a “…more natural view that the real world
components. We can see a pattern emerge in how consists of entities and relationships” where the rela-
we describe the physical world when we think tional model “…can achieve a high degree of data in-
about how we communicate concepts to our chil- dependence, but it may lose some important seman-
dren; there are “things” (“Tessa”, “Kara”, a blan- tic information about the real world.” For example,
ket) and relationships between the three tables ("Person", "Ob-
things (“The blanket belongs to ject", and "Ownership") are not
Kara.”) Fast Facts enough to understand the mod-
el. You must know that columns
ADO.NET Entity Framework
In his groundbreaking 1976 within the "Ownership" table
paper introducing the entity- lets developers build contain the primary key fields of
relationship model, Dr. Peter applications that the object being owned and the
Chen defines a way of modeling access data by programming owner—the semantic meaning of
real-world concepts by break- against a conceptual those columns is not captured by
ing complex information into application model instead the model itself.
things (entities) and associa- of programming
tions between things (relation- Microsoft’s new ADO.NET En-
directly against a relational
ships). tity Framework is an implemen-
storage schema. tation of Dr. Chen’s entity-rela-
Contrast this with the rela- tionship model that maps rela-
tional model employed by rela- tional database schemas to an
tional databases today. The relational model, first Entity Data Model. This article describes key aspects
described by Edgar Codd in 1969, emphasizes rela- of the Entity Data Model along with features of the
tions, or tables of data, not relationships. The re- ADO.NET Entity Framework, which allow mapping
lational model is built around data normalization, of that Entity Data Model to a relational store.
which simplifies data storage and maintenance by
minimizing the duplication of information in or-
der to enforce data consistency. For example, to Components of the Entity Data Model
describe the fact that the blanket belongs to Kara,
you might define three tables: Person, Object, and Following the entity-relationship model, the Entity
Ownership. Data Model is composed of entities and relationships.
www.code-magazine.com
What Is an Entity?
An entity is an instance of an EntityType (for ex-
ample, a person or a blanket). The EntityType de-
scribes the Properties that define the structure of
the entity (for example, name, birthdate, hair col-
or and eye color). In order to be an entity, there
must be a set of Key properties that uniquely iden-
tify the instance from other instances of the same
EntityType within an EntitySet (for example, so-
cial security number).
Inheritance
EntityTypes may extend other EntityTypes
through inheritance. For example, a Salesperson
EntityType may extend an Employee EntityType
by adding properties for Region, Quota, and
Commission, while an Engineer may extend the
same Employee EntityType by adding properties
indicating the ProductTeam she is a member of.
The Entity Data Model does not support multiple
inheritance (a Salesperson cannot also be an En-
gineer, within the same model).
Inheritance typically implies substitutability (ei-
ther a SalesPerson or an Engineer can be supplied
anywhere an Employee is requested) as well as
polymorphism (a request for Employees can re-
turn both SalesPersons and Engineers).
ComplexTypes
Related properties may be grouped together
as a single composite property. For example,
StreetAddress, City, Region, and ZipCode may be
grouped together into a single "Address" property.
The structure of that composite property is de-
fined through a ComplexType that can be used by
multiple EntityTypes, as well as other Complex-
Types, within a schema (for example, Employ-
ees may have a "HomeAddress" property, while
Orders may have a "ShipTo" address). Complex-
Types differ from EntityTypes in that they do
not have independent identifiers; an instance
of a complex type is addressed by an instance
of an entity plus the name of the property on
that instance to which the complex type is de-
fined.
Relationships
Relationships define interesting associations be-
tween entities (for example, "Ownership"). Rela-
tionships are described by an AssociationType,
which defines the types of entities that make up
the association (for example, “ManagerEmployee”
is made up of two “Employee” EntityTypes), their
Roles (“Manager” and “Employee”) and Cardinal-
ity (each Employee has at most one Manager, while
a Manager can have one or more Employees). Re-
lationships may be one to one (for example, a mar-
riage), one to many (for example, Manager to Em-
ployees), or many to many (for example, students
to classes).
www.code-magazine.com
Compositional Relationships
Compositional relationships are a special type of
relationship in which one entity within the rela-
tionship contains the related entity (or entities).
For example, OrderLines may be contained within
an Order. In a compositional relationship, the con-
tained entity (“OrderLine”) must be related to ex-
actly one containing entity (“Order”). Thus:
• A contained entity cannot be associated with
more than one containing entity through the
compositional relationship. It is possible for
it to be associated with other entities through
non-compositional relationships (an Order-
Line cannot be associated with more than
one Order, but may be associated with prod-
ucts, suppliers, etc.).
• Any instance of a contained entity must be
related to an instance of a containing entity
(an OrderLine must have an Order).
• Deleting the containing entity (Order) deletes
all contained entities (OrderLines).
• Additionally, an entity can be the contained
entity in at most one compositional relation-
ship.
While the initial version of the Microsoft Entity
Data Model does not directly support composi-
tional relationships, most of the characteristics of
a compositional relationship (other than the re-
striction that an entity can be the contained en-
tity in at most one compositional relationship) can
be modeled through identifying relationships. In
an identifying relationship, the key field(s) of the
containing entity make up part of the key for the
contained entity, and referential integrity is used
to ensure the contained entity has a non-null con-
taining entity, and is deleted if the containing en-
tity is deleted.
Relationships with Payloads
Relationships with payloads (often called associa-
tion entities) are used to add additional informa-
tion to a relationship—for example, an "Employ-
ment" relationship between a Company and a Per-
son may include HireDate, Salary, and Level. Al-
though the initial version of the Entity Data Model
does not directly support relationships with pay-
loads, the same information can be represented by
defining an intermediate EntityType with the ad-
ditional information that has one to one relation-
ships with the other two EntityTypes (for example,
an “Employment” EntityType with HireDate, Sal-
ary, and Level, and relationships to both Company
and Person).
N-ary Relationships
Similarly, although the first version of the Entity
Data Model only supports binary relationships
(relationships with exactly two ends), n-ary re-
lationships (relationships that may have more
than two ends) can be represented by defining an
intermediate EntityType with more than two bi-
An Entity Data Model for Relational Data Part I: Defining the Entity Data Model 7
 Conclusion
By modeling data Microsoft’s Entity Data model defines an entity-
in terms of instances and relationship model for dealing with data. By mod-
eling data in terms of instances and relationships,
relationships, services such as services such as querying, reporting, synchroniz-
querying, reporting, ing, and programmability against an object model
can be defined in terms of that entity model. Part II
synchronizing, of this article describes how the Entity Data Model
and programmability against is used by the Microsoft ADO.NET Entity Frame-
work to define an application-oriented schema
an object model that can be flexibly mapped to a variety of rela-
can be defined in terms of that tional schema representations.

entity model. Michael Pizzo

nary relationships. (For example, an EntityType

“Game” with relationships to home team, visiting
team, referee, and the location. In this case, you
may want to add other properties, such as start
time and duration of the game, and final score.)

While defining an intermediate EntityType cap-

tures the content of relationships with payloads
and n-ary relationships, doing so loses some of
the semantic meaning of the model (“Employ-
ment” isn’t really an EntityType; it exists only to
describe the relationship between two entities).
Microsoft is looking to add both association en-
tities and n-ary relationships, as well as compo-
sitional relationships, to future versions of the
Entity Data Model to more fully represent the
semantic meaning.

Containers
So far I've described how entity and relationship
types are defined. Applications interact with enti-
ties and relationships through an instance of an
EDM schema, defined by named sets of entity and
relationship instances.

EntitySets

Instances of entities live within a named EntitySet.

A single instance of an entity can belong to only
one EntitySet. An EntitySet is the equivalent of a
relational table.

RelationshipSets

Just as entities live within a named EntitySet, re-

lationship instances live within a RelationshipSet.
RelationshipSets hold the relationship instanc-
es of a particular type between entity instances
within two specific EntitySets. RelationshipSets
are loosely analogous to join tables in relational
schemas.

EntityContainers

EntitySets and RelationshipSets are defined within

an EntityContainer. An EntityContainer can have
multiple EntitySets of the same EntityType.

8 An Entity Data Model for Relational Data Part I: Defining the Entity Data Model www.code-magazine.com
ONLINE QUICK ID 0712032
Michael Pizzo
Michael.Pizzo@microsoft.com
Michael Pizzo has worked for
over 17 years in the design
and delivery of data access
solutions and APIs at Microsoft.
Michael first got involved in data
access as a Program Manager
for Microsoft Excel in 1987,
integrating Microsoft’s flagship
spreadsheet product with
relational data. This led to his
involvement in the design and
delivery of ODBC, along with the
ODBC-based Microsoft Query
Tool shipped with Microsoft
Office. During the design of
ODBC, Michael was active in the
standards organizations, sitting
as Chair for the SQL Access
Group, working with X/Open
on the CAE specification for
“Data Management: SQL-Call
Level Interface (CLI)”, serving
as Microsoft’s representative
to the ANSI X3H2 Database
Committee, and as an elected
ANSI representative to the
ISO committee meetings that
defined and adopted Part 3 of
the ANSI/ISO SQL specification
for a call-Level Interface
(SQL/CLI). Following ODBC,
Michael was a key designer
and driver of Microsoft’s OLE
DB API for componentized
data access within a COM
environment, and later owned
the design and delivery of ADO.
NET version 1.0. He is currently
a Principle Architect in the
Data Programmability Team at
Microsoft, contributing to the
architecture and design of the
next version of ADO.NET and core
building block for Microsoft’s
exciting new data platform; The
ADO.NET Entity Framework.
10 An Entity Data Model for Relational Data Part II: Mapping an Entity Data Model to a Relational Store
An Entity Data Model for
Relational Data Part II:
Mapping an Entity Data
Model to a Relational Store
The ADO.NET Entity Framework allows you to define an
application-oriented view of your data consistent with how you
reason about that data, and map that conceptual view to existing
relational schemas.
Part I of this article described the Entity Data Model and how it enables you to
model real-world concepts in a more natural way. Part II of the article describes how
that Entity Data Model is used within the ADO.NET Entity Framework to define an
application-oriented conceptual view of your data, and how that view can be flexibly
mapped to existing relational schemas. Shyam Pather’s article, “Programming
Against the ADO.NET Entity Framework” completes the picture by describing the
actual programming model and API used by developers to work with data using the
ADO.NET Entity Framework.
T
he ADO.NET Entity Framework supports and updates against the conceptual model such that
flexible mapping of an application-oriented the query passed to the underlying ADO.NET Data
conceptual view of the data, defined in terms Provider is in terms of common relational operators
of the Entity Data Model, to against the actual storage sche-
existing relational tables. The Fast Facts ma. All query evaluation is done
Entity Framework includes an within the store, and returned
EntityClient that maps between Microsoft’s new results are assembled to expose
an Entity Data Model and a re- ADO.NET Entity Framework is possibly polymorphic, hierarchi-
lational schema, an Object Ser- an implementation of cal data readers.
vices layer that allows applica- Dr. Chen’s entity-relationship
tions to query and return results model that maps relational The ability to generate these views
in terms of strongly-typed Com- database schemas to an on the client, based on metadata
mon Language Runtime (CLR) supplied at runtime, provides a
Entity Data Model.
data objects, and a Metadata loose coupling between the ap-
Runtime that exposes metadata plication and storage schema, al-
and mapping information to both the EntityClient lowing each to evolve independently over time with-
and Object Services layer. out having to define application-specific views on the
server or recompiling the application to account for
changes in the database schema. Alternatively, gener-
EntityClient ated client views may optionally be compiled into the
application as a performance optimization.
The EntityClient is an ADO.NET Data Provider that
allows applications to work in terms of a conceptual
Entity Data Model. The EntityClient accepts queries Object Services
defined in terms of a canonical Entity SQL (ESQL)
grammar; a version of SQL extended to provide first- While applications can consume their conceptual
class operators around types, collections, and rela- Entity Data Model through existing ADO.NET
tionships defined in the Entity Data Model. Data Provider APIs using the EntityClient, it is of-
ten convenient to query and return results in terms
The EntityClient generates client-side query and up- of strongly-typed CLR data objects. The ADO.NET
date views to represent the mapping between the Entity Framework includes an Object Services layer
conceptual Entity Data Model and the relational stor- that allows applications to query and return results
age schema. These views are used to expand queries in terms of CLR data objects using either Entity SQL
www.code-magazine.com
or Language Integrated (LINQ) queries. Results
may optionally be identity managed (so the same
entity returned through multiple queries within the
same context return the same object instance) and
change-tracked (so changes made to the in-memory
object graph can be saved back to the database).

Metadata
In order to generate mapping views, the Entity
Framework relies on three pieces of metadata:

• A description of the actual storage schema, de-

fined in terms of a Schema Storage Definition
Language (SSDL).
• A description of the application’s conceptual
view, defined in terms of a Conceptual Schema
Definition Language (CSDL).
• A description of the mapping between the
Conceptual Model and the Storage model, de-
fined in terms of a Mapping Schema Language
(MSL).

SSDL, CSDL, and MSL are XML grammars, gener-

ally provided as XML files or resources, but which
can be loaded from other sources at runtime such as
a metadata repository.
Mapping Entities
The Entity Framework supports mapping a single
EntityType to one or more relational tables.
For example, Figure 1 shows a simple relational
schema (a simplified version of the “Northwind”
sample database shipped with Microsoft SQL
Server). Listing 1 shows the SSDL describing the
Customers table within the Northwind database.
Within Listing 1, the EntityType element defines
the Customer EntityType, the Property elements de-
fine its members, the Key element defines the fields
that uniquely identify a customer instance, and the
EntitySet element defines an EntitySet named “Cus-
tomers” containing Customer EntityTypes within
the “dbo” EntityContainer.
Figure 1: Simple relational schema from the Northwind Sample Database that ships with Microsoft SQL
Server.
Listing 2 describes an equiva-
lent Conceptual view of Cus-
tomers that mirrors the SSDL
view one for one.
Listing 3 shows the trivial map-
ping between the Customers Ta-
ble described in Listing 1 and the
Customers EntitySet described in
Listing 2.
Figure 2: Employee entity split between two tables.
Mapping a Single Entity
to Multiple Tables
Suppose, however, that the information defined in
an EntityType actually exists in multiple tables in
the database (for example, suppose the information

Listing 1: SSDL for Customers table

<Schema Namespace="NorthwindModel.Store" Alias="Self" <Property Name="CompanyName" Type="nvarchar"
xmlns="http://schemas.microsoft.com/ado/2006/04/edm/ssdl"> Nullable="false" MaxLength="40" />
<Property Name="ContactName" Type="nvarchar" MaxLength="30" />
<EntityContainer Name="dbo"> <Property Name="ContactTitle" Type="nvarchar" MaxLength="30" />
<EntitySet Name="Customers" <Property Name="Address" Type="nvarchar" MaxLength="60" />
EntityType="NorthwindModel.Store.Customer" /> <Property Name="City" Type="nvarchar" MaxLength="15" />
</EntityContainer> <Property Name="Region" Type="nvarchar" MaxLength="15" />
<Property Name="PostalCode" Type="nvarchar" MaxLength="10" />
<EntityType Name="Customer"> <Property Name="Country" Type="nvarchar" MaxLength="15" />
<Key> <Property Name="Phone" Type="nvarchar" MaxLength="24" />
<PropertyRef Name="CustomerID" /> <Property Name="Fax" Type="nvarchar" MaxLength="24" />
</Key> </EntityType>
<Property Name="CustomerID" Type="nchar"
Nullable="false" MaxLength="5" /> </Schema>

www.code-magazine.com An Entity Data Model for Relational Data Part II: Mapping an Entity Data Model to a Relational Store 11
 Listing 2: CSDL for Customers EntitySet
<Schema Namespace="NorthwindModel" Alias="Self"
xmlns="http://schemas.microsoft.com/ado/2006/04/edm">
<EntityContainer Name="NorthwindEntities">
<EntitySet Name="Customers"
EntityType="NorthwindModel.Customer" />
</EntityContainer>
<EntityType Name="Customer">
<Key>
<PropertyRef Name="CustomerID" />
</Key>
<Property Name="CustomerID" Type="String"
Nullable="false" MaxLength="5" FixedLength="true" />
Listing 3: Trivial MSL for Customers table to Customers EntitySet
<Mapping Space="C-S"
xmlns="urn:schemas-microsoft-com:windows:storage:mapping:CS">
<EntityContainerMapping StorageEntityContainer="dbo"
CdmEntityContainer="NorthwindEntities">
<EntitySetMapping Name="Customers" StoreEntitySet="Customers"
TypeName="NorthwindModel.Customer">
<ScalarProperty Name="CustomerID" ColumnName="CustomerID" />
<ScalarProperty Name="CompanyName" ColumnName="CompanyName" />
<ScalarProperty Name="ContactName" ColumnName="ContactName" />
<ScalarProperty Name="Title" ColumnName="ContactTitle" />
from the Employees table shown in Figure 1 was
instead split into tables; one containing common
address book information (FirstName, LastName,
Extension, ReportsTo, etc.) and one containing
personal information such as Birthdate, Home Ad-
dress, etc., as shown in Figure 2. In this case your
conceptual model (CSDL) would remain the same,
but your MSL would map the Employee EntityType
to two separate tables, as shown in Listing 4.
Mapping Inheritance
The Relational Data Model does not directly sup-
port the concept of inheritance. Inheritance is
commonly represented within a database in one
Listing 4: MSL mapping Employee entity to two relational tables
<EntitySetMapping Name="Employees"
TypeName="NorthwindModel.Employee">
<MappingFragment StoreEntitySet="Employees_AddressBook">
<ScalarProperty Name="EmployeeID" ColumnName="EmployeeID" />
<ScalarProperty Name="LastName" ColumnName="LastName" />
<ScalarProperty Name="FirstName" ColumnName="FirstName" />
<ScalarProperty Name="Title" ColumnName="Title" />
<ScalarProperty Name="TitleOfCourtesy"
ColumnName="TitleOfCourtesy" />
<ScalarProperty Name="Extension" ColumnName="Extension" />
<ScalarProperty Name="Photo" ColumnName="Photo" />
<ScalarProperty Name="PhotoPath" ColumnName="PhotoPath" />
</MappingFragment>
12 An Entity Data Model for Relational Data Part II: Mapping an Entity Data Model to a Relational Store
<Property Name="CompanyName" Type="String"
Nullable="false" MaxLength="40" />
<Property Name="ContactName" Type="String" MaxLength="30" />
<Property Name="Title" Type="String" MaxLength="30" />
<Property Name="Address" Type="String" MaxLength="60" />
<Property Name="City" Type="String" MaxLength="15" />
<Property Name="Region" Type="String" MaxLength="15" />
<Property Name="PostalCode" Type="String" MaxLength="10" />
<Property Name="Country" Type="String" MaxLength="15" />
<Property Name="Phone" Type="String" MaxLength="24" />
<Property Name="Fax" Type="String" MaxLength="24" />
</EntityType>
</Schema>
<ScalarProperty Name="Address" ColumnName="Address" />
<ScalarProperty Name="City" ColumnName="City" />
<ScalarProperty Name="Region" ColumnName="Region" />
<ScalarProperty Name="PostalCode" ColumnName="PostalCode" />
<ScalarProperty Name="Country" ColumnName="Country" />
<ScalarProperty Name="Phone" ColumnName="Phone" />
<ScalarProperty Name="Fax" ColumnName="Fax" />
</EntitySetMapping>
</EntityContainerMapping>
</Mapping>
of three different ways: mapping of an entire in-
heritance hierarchy to a single table, mapping of
each type in an inheritance hierarchy to a different
table, or a hybrid approach wherein the informa-
tion common across types is in a single table with
additional tables containing the added columns for
each derived type. The ADO.NET Entity Frame-
work supports mapping to all three of these inheri-
tance models.
Table per Hierarchy (TPH)
The Table per Hierarchy (TPH) mapping strategy
uses a single (sparse) table containing the data for
all types within the hierarchy. A discriminator col-
umn is used to specify the subtype of a particular
<MappingFragment StoreEntitySet="Employees_Personal" >
<ScalarProperty Name="EmployeeID" ColumnName="EmployeeID" />
<ScalarProperty Name="BirthDate" ColumnName="BirthDate" />
<ScalarProperty Name="HireDate" ColumnName="HireDate" />
<ScalarProperty Name="Address" ColumnName="Address" />
<ScalarProperty Name="City" ColumnName="City" />
<ScalarProperty Name="Region" ColumnName="Region" />
<ScalarProperty Name="PostalCode" ColumnName="PostalCode" />
<ScalarProperty Name="Country" ColumnName="Country" />
<ScalarProperty Name="HomePhone" ColumnName="HomePhone" />
<ScalarProperty Name="Notes" ColumnName="Notes" />
</MappingFragment>
</EntitySetMapping>
www.code-magazine.com
row in the table, and columns not relevant for that
type are unused. This strategy optimizes for queries
across different subtypes, particularly in the case of
polymorphic results.
Table per Concrete Type (TPT)
The Table per Concrete Type (TPT) strategy uses dif-
ferent tables for each type in the hierarchy. Queries
against the base type unions the common fields from
the different tables. This strategy optimizes queries
scoped to a particular subtype at the expense of
queries against the base type.
The following MSL fragment shows how the Ad-
dress complex property is mapped within an Enti-
tySetMapping.
<ComplexProperty Name="Address">
<ScalarProperty Name="StreetAddress"
ColumnName="Address" />
<ScalarProperty Name="City"
ColumnName="City" />
<ScalarProperty Name="Region"
ColumnName="Region" />
<ScalarProperty Name="PostalCode"
ColumnName="PostalCode" />
<ScalarProperty Name="Country"

Table per Subclass (TPC) </ComplexProperty>

ColumnName="Country" />
The Relational
Data Model does
The Table per subclass (TPC) strategy uses a single Mapping Relationships not directly support
table for the base properties across the type hierar-
chy, with separate tables for the additional proper- Relational databases use foreign keys to enforce ref- the concept of
ties defined in each subtype. This strategy is optimal erential integrity between related tables. While these inheritance.
for queries across the type hierarchy that project out integrity constraints are not a consumable part of
common fields from the base type by not requiring the data model (i.e., you can’t navigate them), they Inheritance
joins to the subtype tables. do imply relationships between tables. is commonly
The ADO.NET Entity Framework allows you to represented within
Mapping Complex Types map foreign key properties, whether or not they are a database in one
defined with a foreign key constraint, to navigable
Oftentimes there are groups of related properties relationships through Associations.
of three
that we’d like to see represented by a single com- different ways;
posite object. For example, looking at the North-
wind schema in Figure 1 we see that Customers,
The following fragment shows how an association
between Customers and Orders is defined within
the ADO.NET
Employees, and Orders all have address, city, region, the store schema definition (SSDL). Entity Framework
country, and postal code properties that represent supports mapping
addresses. It would be convenient if we could define <Association Name="FK_Orders_Customers">
a single Address type that contained this informa- <End Role="Customers" to all three of these
tion, and expose it as a single property anywhere we Type="NorthwindModel.Store.Customers" inheritance
needed to represent an address. Multiplicity="0..1" />
<End Role="Orders" models.
The ADO.NET Entity Framework supports Com- Type="NorthwindModel.Store.Orders"
plexTypes for doing just that. The snippet below Multiplicity="*" />
shows the definition of a complex type in CSDL. </Association>

<ComplexType Name="Address"> If the association in the store is based on a refer-

<Property Name="StreetAddress" ential constraint, that constraint can be called out
Type="String" MaxLength="60" /> within the Association element. For example, the
<Property Name="City" following constraint specifies that all orders must
Type="String" MaxLength="15" /> have a customer.
<Property Name="Region"
Type="String" MaxLength="15" /> <ReferentialConstraint>
<Property Name="PostalCode" <Principal Role="Customers">
Type="String" MaxLength="10" /> <PropertyRef Name="CustomerID" />
<Property Name="Country" </Principal>
Type="String" MaxLength="15" /> <Dependent Role="Orders">
</ComplexType> <PropertyRef Name="CustomerID" />
</Dependent>
The next CSDL fragment shows the use of the Com- </ReferentialConstraint>
plexType within an EntityType definition. When
binding to objects, the resulting CLR type would Associations can be defined in the conceptual sche-
have a single property called Address, of type Ad- ma (CSDL).
dress, containing the StreetAddress, City, Region,
PostalCode, and Country fields. <Association Name="FK_Orders_Customers">
<End Role="Customers"
<Property Name="Address" Type="NorthwindModel.Customers"
Type="Northwind.Address" Multiplicity="0..1" />
Nullable="false" <End Role="Orders"
/> Type="NorthwindModel.Orders"

www.code-magazine.com An Entity Data Model for Relational Data Part II: Mapping an Entity Data Model to a Relational Store 13
 Multiplicity="*" />
</Association>
In addition, since an EntityContainer can contain
multiple EntitySets of the same type, an Assoca-
tionSet is defined to specify the sets of entities be-
ing related. AssociationSets can be directly que-
ried, just as any other EntitySet.
<EntityContainer Name="NorthwindEntities">
<AssociationSet Name="FK_Orders_Customers"
Association=
"NorthwindModel.FK_Orders_Customers">
<End Role="Customers" EntitySet="Customers" />
<End Role="Orders" EntitySet="Orders" />
</AssociationSet>
</EntityContainer>
Once associations have been defined in the
CSDL, NavigationProperties can be defined on
the related Entities to navigate from one end of
the relationship to the other. For data classes
generated from the conceptual model, navigation
properties appear as any other properties and can
be used to navigate the object graph.
For example, rather than “CustomerID” appear-
ing as a string property on an Order, an Order
may contain a “Customer” property that allows
you to navigate to the actual related Customer
instance.
<NavigationProperty Name="Customers"
Relationship="NorthwindModel.FK_Orders_Customers"
FromRole="Orders"
ToRole="Customers"
/>
Navigation properties can be defined on both ends
of a relationship; where the target end has a car-
dinality greater than one, the navigation property
is a collection of related entities (for example, a
Customer may have a collection of Orders).
<NavigationProperty Name="Orders"
Relationship="NorthwindModel.FK_Orders_Customers"
FromRole="Customers"
ToRole="Orders"
/>
Not only do relationships surface as navigation
properties in generated objects, but they can be used
within queries to navigate the conceptual model.
The following query shows navigating relationships
and complex properties to select the OrderID and
City for orders placed within Washington.
SELECT o.OrderID, o.Customers.Address.City
FROM Orders AS o
WHERE o.Customers.Address.Region = 'WA'
The syntax of the query is Entity SQL (eSQL)
which is described in greater detail in Shyam
Pather’s article on programming against the En-
tity Framework. For the purposes of this example,
notice how natural it is to navigate relationships
14 An Entity Data Model for Relational Data Part II: Mapping an Entity Data Model to a Relational Store
defined within the model without having to build
complex joins in your query.
Mapping Many:Many Relationships
Many-to-many relationships (for example, a Stu-
dent may be enrolled in multiple Courses, and
each Course may have multiple students) are
generally mapped in relational databases through
an intermediate "join" table. Where that table
contains no information other than the primary
keys of the related entities, the Entity Framework
allows you to navigate directly from one end to
the other without going through the intermedi-
ate table. In the case that the intermediate table
has additional information that describes the re-
lationship (for example, number of absences and
grade) you can model the relationship as an entity
(StudentEnrollment) with relationships to each of
the other entities (Student and Course).
Mapping Functions and Stored
Procedures
Databases typically include built-in and user-
defined functions, as well as stored procedures,
for executing logic within the store. The ADO.
NET Entity Framework allows you to call stored
procedures and functions defined within your
storage schema (SSDL) as follows. For example,
the following definition describes a function to
return all sales within a given category for a given
year.
<Function Name="SalesByCategory" Aggregate="false"
BuiltIn="false" NiladicFunction="false"
IsComposable="false"
ParameterTypeSemantics=
"AllowImplicitConversion"
Schema="dbo">
<Parameter Name="CategoryName"
Type="nvarchar" Mode="in" />
<Parameter Name="OrdYear"
Type="nvarchar" Mode="in" />
</Function>
Using Stored Procedures for Change
Processing
Changes to data are often controlled through
stored procedures to insert, update, or delete
items. The ADO.NET Entity Framework supports
the option to specify a stored procedure to be
used in place of a generated DML statement in
order to insert, delete, or update changes in the
database.
Stored procedures to be used for updating must
first be specified in the storage schema definition
(SSDL) just as any other function or stored pro-
cedure. For example, a stored procedure to add
an order detail might look like the following:
www.code-magazine.com
<Function Name="AddOrderDetail"
IsComposable="false">
<Parameter Name="OrderID" Type="int"/>
<Parameter Name="ProductID" Type="int"/>
<Parameter Name="UnitPrice" Type="money" />
<Parameter Name="Quantity" Type="smallint" />
<Parameter Name="Discount" Type="real" />
</Function>
Once the stored procedure is declared, it can be
specified in a ModificationFunctionMapping ele-
ment within the EntitySet definition in the mapping
(MSL).
<InsertFunction
FunctionName=
"NorthwindModel.Store.AddOrderDetail">
<ScalarProperty ParameterName="OrderID"
Name="OrderID"/>
<ScalarProperty ParameterName="ProductID"
Name="ProductID"/>
<ScalarProperty ParameterName="UnitPrice"
Name="UnitPrice"/>
<ScalarProperty ParameterName="Quantity"
Name="Quantity"/>
<ScalarProperty ParameterName="Discount"
</InsertFunction>
Name="Discount"/>
Note that you must use all stored procedures or
all generated update views for processing changes
within an object graph. You cannot use a stored
procedure to insert an entity and generated update
views to update or delete within the same EntitySet,
or any EntitySet with a relationship to the EntitySet
updated through stored procedures.
Advanced Mapping
So far I’ve shown you examples of declarative
mappings from which the ADO.NET Entity Frame-
work generates client views for querying and up-
dating a relational schema in terms of a conceptual
application-oriented data model. There are times,
however, when you may require more control over
the views used to map your conceptual model to
the store.
Explicit Mapping
The ADO.NET Entity Framework lets you override
the automatic generation of query views by explic-
itly specifying a QueryView in your mapping. Views
are expressed in terms of Entity SQL. As a simple
example, the following mapping definition (MSL)
fragment maps the Customers EntitySet to an Entity
SQL query that returns only those customers from
Italy.
<EntitySetMapping Name="Customers">
<QueryView>
SELECT VALUE NorthwindModel.Customer(
c.CustomerID, c.CompanyName, c.ContactName,
c.ContactTitle, c.Address, c.City,
www.code-magazine.com
c.Region, c.PostalCode, c.Country,
c.Phone, c.Fax)
FROM dbo.Customers as c
WHERE c.Country = "Italy"
</QueryView>
</EntitySetMapping>
When using explicit mapping you must also specify
stored procedures, as described previously, if you
wish to update the data retrieved through these
views.
Mapping to Store Queries
In addition to allowing you to specify custom Entity In addition to
SQL query views in your mapping, the ADO.NET allowing
Entity Framework lets you define a storage table in
terms of a native store query using a DefiningQuery. you to specify
This allows you to leverage the full expressivity of custom Entity SQL
the store’s native query language, and apply map-
ping to the target as a virtual storage table. As with
query views in your
explicit query views, if you use defining queries you mapping,
must also specify stored procedures if you wish to
update the data retrieved through these virtual stor-
the ADO.NET Entity
age tables. Framework lets you
The following storage schema definition (SSDL)
define a storage
fragment shows the use of a DefiningQuery to spec- table in terms of a
ify a virtual table of Customers whose titles contain
the word “Sales”. Note that the query is written in
native store query
terms of the store’s native query language (in this using a
case, TSQL).
DefiningQuery.
<EntitySet Name="Customers"
EntityType=
"NorthwindModel.Store.Customers">
<DefiningQuery>
SELECT c.CustomerID, c.CompanyName,
c.ContactName, c.ContactTitle,
c.Address, c.City, c.Region,
c.PostalCode, c.Country, c.Phone, c.Fax
FROM Customers as c
WHERE c.ContactTitle LIKE '%Sales%'
</DefiningQuery>
</EntitySet>
Conclusion
Using the ADO.NET Entity Framework, developers
are able to program against an application-centric
Entity Data Model. Within the Entity Framework,
the EntityClient supports flexible declarative map-
ping of that conceptual model to existing relational
storage schemas exposed through extended ADO.
NET Data Providers. Developers may choose to
use the traditional ADO.NET Data Provider API by
writing to EntityClient directly, or query and return
results in terms of strongly-typed data objects using
Object Services.
Michael Pizzo
An Entity Data Model for Relational Data Part II: Mapping an Entity Data Model to a Relational Store 15
ONLINE QUICK ID 0712042

Programming Against the

Shyam Pather
spather@microsoft.com
Shyam Pather is a Senior
Development Lead on the
ADO.NET Entity Framework
The ADO.NET Entity Framework raises the level of abstraction at
which developers work with data.
Rather than coding against rows and columns, the ADO.NET Entity Framework
allows you to define a higher-level Entity Data Model over your relational data,
and then program in terms of this model. You get to deal with your data in the
shapes that make sense for your application and those shapes are expressed in a
richer vocabulary that include concepts like inheritance, complex types, and explicit
relationships.

Data Programmability Team at
Microsoft, currently focused on
building the first release of the
ADO.NET Entity Framework.
Shyam began his career at
Microsoft in the Windows
Networking team, working first
on network driver infrastructure
and then on the first two releases
of Universal Plug and Play in
Windows. Shyam joined the
SQL Server team to work on an
incubation project that eventually
became SQL Server Notification
Services. After shipping two
releases of that product, Shyam
started in his current role on
ADO.NET. His team delivers key
parts of the object-relational
mapping technology on which
the ADO.NET Entity Framework
is based.
I
n his article, An Entity Data Model for Rela- SQL (E-SQL), which augments traditional SQL
tional Data (parts I & II), Mike Pizzo explains with constructs necessary for querying in terms of
the details of the Entity Data Model and how the higher-level modeling concepts in Entity Data
to map a model to a database. In this article, I dis- Models (inheritance, complex types, and explicit
cuss the capabilities the ADO.NET Entity Frame- relationships).
work provides for program-
ming against a model after it Fast Facts Above the EntityClient layer,
is defined. The material in this the ADO.NET Entity Frame-
article is intended as an over- The ADO.NET Entity work includes an Object Ser-
view that covers the breadth of Framework allows you to vices layer, which provides a
features available to develop- program in terms of a high- programming model in terms
ers. Each of these features is a level conceptual model of of strongly-typed objects. At the
topic unto itself and is covered your data. This eliminates Object Services layer, entities
in more detail in the MSDN
much of the need for custom are represented as objects—in-
documentation that accom- stances of data classes that are
data access layers.
panies the ADO.NET Entity mapped to the entity types in
Framework. the model. The Object Services
layer supports querying with both Entity SQL and
Language Integrated Query (LINQ). Query results
Overview of the ADO.NET Entity are manifested as objects whose properties can be
Framework read and set by application code. Object Services
keeps track of changes made to the objects and can
propagate these back to the database. The ADO.
The ADO.NET Entity Framework is built in a lay- NET Entity Framework’s mapping engine performs
ered architecture on top of existing ADO.NET 2.0 the translation between changes to the entity objects
data providers. Figure 1 illustrates this layering. and operations on the underlying database tables.

The ADO.NET 2.0 Data Providers offer a common
programming model for accessing disparate data
sources. Whether you are programming against a
Microsoft SQL Server database or an Oracle data-
base, you use the same fundamental programming
abstractions: a connection object, a command ob-
ject, and a data reader object.
The Object
The ADO.NET Entity Framework introduces a
Services layer new data provider called EntityClient; this is the
supports querying first layer above the ADO.NET 2.0 data providers.
EntityClient offers the same programming abstrac-
with both Entity tions as the other data providers—connections,
SQL and Language commands, and data readers—but adds mapping ca-
pabilities that translate queries expressed in terms
Integrated of the model into the equivalent queries in terms of
Query (LINQ). the tables in the databases. To complement these
capabilities, the ADO.NET Entity Framework in-
troduces an extended query language called Entity
Developers can choose to program against either
the EntityClient layer or the Object Services layer.
This choice provides flexibility for different applica-
tion scenarios. In cases where the application does
not require objects, the EntityClient layer provides a
lower-level entry point that offers the benefits of en-
tities without the overhead of object materialization.
When you require the convenience and change-
tracking capabilities of objects, you can choose to
work at the Object Services layer. In either case,
an underlying ADO.NET data provider performs
database operations and compensates for the dif-
ferences between database systems. Thus, the ADO.
NET Entity Framework exposes exactly the same
programming model and query syntax, regardless of
the particular DBMS in which the data is stored.
In this article, I’ll approach the ADO.NET Entity
Framework top-down: First I’ll show some code

16 Programming Against the ADO.NET Entity Framework www.code-magazine.com

snippets that illustrate programming at the Object
Services layer. Then, I’ll delve into some examples
that show EntityClient in action.

A Sample Entity Data Model

For the purpose of illustration in this article, I’ll use
the sample data model shown in Figure 2. This mod-
el represents various types of athletic events, like
bike races, running events, and triathlons. There are
two base types in this model: Event, which encap-
sulates the attributes common to all athletic events,
and Participant, which represents a participant in
one or more events. The model declares a many-to-
many relationship between Events and Participants
called Event_Participant.

Both Event and Participant have sub-types. Event

is the parent for sub-types BikeEvent, RunEvent,
SwimEvent, and TriathlonEvent. Each of these sub-
types augments the base type with new properties.
Participant has a sub-type called Volunteer: This
is used to represent participants who volunteer, as
opposed to compete, in the events. Note that Vol- Figure 1: The ADO.NET Entity Framework is built in a layered architecture above the existing ADO.NET
unteer does not add any new fields to its base type, 2.0 data providers.
Participant.

The model also declares an entity container called

TriEventsEntities, which contains entity sets for
Events and Participants and an association set for In cases where the
the Event_Participant association. Figure 2 does
not show the entity container, but it is a fundamen- application does not require
tal part of the model. objects, the EntityClient layer
The ADO.NET Entity Framework supports sever-
provides a lower-level entry point
al ways to map a model like this one to relational that offers the benefits
tables. I happened to use a Table-per-Type (TPT)
mapping (a table for each base type and a table for
of entities without the overhead of
each sub-type that stores just the additional proper- object materialization.
ties declared by the sub-type). For the purposes of
this article, the mapping strategy is irrelevant: the

Figure 2: This sample data model describes athletic events and their participants.

www.code-magazine.com Programming Against the ADO.NET Entity Framework 17


Composing Queries
The ObjectQuery<T> class
supports query composition:
You can compose any
ObjectQuery<T> with additional
query operators to form a new
query. You do this via query-
builder methods that correspond
to the various query operators.
For example, the
ObjectQuery<T> class contains
a Where() method that takes
a predicate. It composes this
predicate with the original query
to form a new, more restrictive
query. ObjectQuery<T>’s query
builder methods support both
LINQ query specification (via
lambda expressions) and Entity
SQL fragments.
18 Programming Against the ADO.NET Entity Framework
ADO.NET Entity Framework supports the same
programming patterns regardless of how you map
the model.
Programming with Object Services
To program against the model with Object Services,
you need data classes that correspond to the types
declared in the model. The ADO.NET Entity Frame-
work includes tools that can generate these classes
from the model definition. Alternatively, you can
code these classes by hand (doing so requires you
to implement a specific interface mandated by the
ADO.NET Entity Framework). I’ve based the code
snippets shown in this article on classes generated by
the ADO.NET Entity Framework’s tools.
Here is a code snippet that enumerates entities in
the model:
using (TriEventsEntities entities =
new TriEventsEntities())
{ second part of the connection string specifies which
// Enumerate entities
foreach (Event e in entities.Events)
{
Console.WriteLine("Event Name: {0}",
e.EventName);
} uses to connect to the database. The ADO.NET En-
} tity Framework uses the three parts of the connection
The preceding snippet is all that’s required to connect
to the model and enumerate the Event entities within
it. Dissecting this code highlights several important
aspects of working with the ADO.NET Entity Frame-
work.
The first line of code creates an instance of the
TriEventsEntities class. The ADO.NET Entity Frame-
work tools generated this class and it corresponds to
the entity container declared in the model. It inherits
from the Object Services base class, ObjectContext,
and you can loosely refer to it as the “object context”
for the model. You can think of this class as a starting
point for interaction with the model–It encapsulates
both a connection to the database and the in-mem-
ory state required to do change tracking and identity
resolution for the objects that result from queries. It is
instantiated within a using block, which ensures that
it is disposed of properly at the end of the code.
Within the using block, the foreach loop iterates over
objects in the Events property in the object context.
This property represents the contents of the Events
entity set (the generated object context class has a
property for each entity set in the container). Iteration
over the Events property sends a query for all Events
to the database. The ADO.NET Entity Framework
materializes the results of this query into objects of
the generated Events class. Within the foreach loop,
the code simply prints the value of one of the entity’s
properties. You could easily imagine more sophisti-
cated processing logic here.
At this point, you may be wondering how the ADO.
NET Entity Framework establishes the connection
to the database because the code snippet does not
include any connection information. The default
constructor generated on the object context class,
TriEventsEntities, looks for a named connection
string in the App.config file with the same name.
The ADO.NET Entity Framework tools conveniently
create the TriEventsEntities connection string in the
App.config file when generating the class. This con-
nection string will look something like the following:
metadata=.\TriEvents.csdl|.\TriEvents.ssdl|.
\TriEvents.msl;provider=System.Data.SqlClient;
provider connection string="Data
Source=.\SqlExpress;
Initial Catalog=TriEvents;
Integrated Security=True;
MultipleActiveResultSets=true"
You can think of this connection string as consisting
of three parts. The first part specifies a list of meta-
data files; these files define the model, the database
schema, and the mapping between the two in a for-
mat defined by the ADO.NET Entity Framework. The
ADO.NET data provider to use to connect to the da-
tabase. This example specifies the SqlClient data pro-
vider. The final part of the connection string is really
another connection string: the one the data provider
string to establish a connection to a model: it uses the
metadata files to load the model and mapping infor-
mation, it uses the data provider name to instantiate
the data provider, and it uses the provider’s connec-
tion string to connect to the database.
Navigating Relationships
The Entity Data Model contains an explicit relation-
ship between Events and Participants. The Event and
Participant types have navigation properties that tra-
verse this relationship and allow you to find entities
on one end of the relationship when you are holding
a reference to the other end. For example, given an
Event object, I can use the Participants navigation
property to find the related Participant entities. The
following code snippet shows how (in this and sub-
sequent code snippets, the code shown goes within
the using block declared in the previous code snip-
pet; the using block is omitted here for brevity):
// Navigate relationships explicitly
foreach (Event e in entities.Events)
{
Console.WriteLine("Event Name: {0}",
e.EventName);
e.Participants.Load();
foreach (Participant p in e.Participants)
{
Console.WriteLine("\tParticipant:{0}",
p.ParticipantName);
}
}
www.code-magazine.com
This code navigates through the events and prints
the related participants for each one. As in the previ-
ous example, the foreach loop that iterates over the
Events results in a database query for all events. By
default, when issuing such a query, the ADO.NET En-
tity Framework does not pull in related entities (do-
ing so would add overhead that is unnecessary, unless
you are going to access those related entities). There-
fore, to obtain the related Participant objects, the code
makes a call to the Load() method on the Participants
collection on each Event entity. This issues a separate
query to the database to retrieve the participants asso-
ciated with each event. Thereafter, you can enumerate
the participants with a foreach loop.
If you know you will need to pull in related entities,
you can take advantage of an Object Services feature
called Relationship Span and avoid having to issue
separate queries. The following code snippet illus-
trates how to retrieve Event entities and the related
Participants at the same time:
// Relationship span
foreach (Event e in
entities.Events.Include("Participants"))
{ an opportunity to build the
Console.WriteLine("Event Name: {0}",
e.EventName);
foreach (Participant p in e.Participants)
{ For more information on Entity SQL, see the MSDN
Console.WriteLine("\tParticipant:{0}",
p.ParticipantName);
} As an alternative to Entity SQL, some developers may
} choose to express queries with LINQ. LINQ offers
In the outer foreach loop, the code calls the Include()
method to specify which related entities the query
should retrieve. The Include() method takes a text
representation of a path across one or more naviga-
tion properties (here you reference the “Participants”
navigation property). In models that have several lev-
els of relationships (for example, Categories related to
Products related to Suppliers), you can use the Include
method to traverse more than one level. Because the
code called the Include() method in the outer enu-
meration, it is not necessary to call the Load() method
before enumerating the Participants on each Event.
Issuing Queries
The previous snippets showed examples of enumerat-
ing the top-level properties in the object context class
to query for all entities in a particular entity set. In
your application, you will likely want to issue other
types of queries as well. The ADO.NET Entity Frame-
work’s Object Services layer supports this via the
ObjectQuery<T> class.
The following snippet shows ObjectQuery<T> used
to query for all event entities that are of the sub-type,
RunEvent. Here, the query is expressed using Entity
SQL.
// Object Query
ObjectQuery<Event> runEventsQuery =
www.code-magazine.com
entities.CreateQuery<Event>(@" Data Programmability Team
SELECT VALUE e
FROM TriEventsEntities.Events AS e
WHERE e IS OF (ONLY TriEventsModel.RunEvent)");
foreach (Event e in runEventsQuery)
{
Console.WriteLine("Event Name: {0}",
e.EventName);
}
The code calls the CreateQuery<T>() method on the
object context class to obtain an ObjectQuery<T> ob-
ject. CreateQuery<T>() takes a string representation
of the query. This example illustrates the use of the
IS OF operator in Entity SQL, which filters entities
based on type. Anil Nori
When you execute the query, the ADO.NET Entity “I am excited to work on
Framework translates the Entity SQL query represen- the next phase of the data
tation into a query in terms of the database schema. platform strategy. With our
To do this, the ADO.NET Entity Framework consults store (database) presence on
the mapping information in the mapping files refer- the servers, desktops, devices,
enced in the connection string. The Object Services mid-tier and cloud, and with
layer materializes results of the query as objects of the data development technologies
Event class. on these tiers, we really have
Entity SQL provides powerful query capabilities. Be- end-to-end data platform. I am
cause it is a text-based language, it lends itself well to quite excited about the future.”
scenarios in which you construct queries dynamically.
reference documentation. With over 20 years of experience
in building complex database
and eBusiness systems –Anil,
Distinguished Engineer on the
the advantages of strong compile-time checking and Data Programmability team, is as
IntelliSense support in Visual Studio. The ADO.NET excited as ever about architecting
Entity Framework supports LINQ over its Object Ser- and building Microsoft’s data
vices layer, as illustrated in the following snippet: platform.
// LINQ Query Previously, he was at Oracle as
var longEventsQuery = a database server architect for
from e in entities.Events Oracle8 and Oracle8i database
where e.StartDate != e.EndDate releases. At Oracle, he drove the
select e; efforts in Oracle object-relational
and extensible technology,
foreach (Event e in longEventsQuery) Internet and multi-media
{ DBMS development, and XML
Console.WriteLine("Event Name: {0}", technology. Nori also worked
e.EventName); as a Database Architect for DEC
} database products, where he
was involved in the development
Here I use LINQ to formulate a query for events of centralized and distributed
whose end dates are different than their start dates. DBMS products. In the space of
Notice that I expressed the query in native C# lan- applications, Nori co-founded
guage constructs and it is not enclosed in quotes. Asera Inc., which built eBusiness
Had this query contained a syntactic error, the platform, tools, and solutions
compiler would have caught it at compile time. For in Order Management and
more information on LINQ, see Elisa Flasko’s ar- Supply Chain. Anil is actively
ticle LINQ to Relational Data, also in this issue of involved in the database research
CoDe Focus. community. He has published
papers at ICDE, SIGMOD, and
As with the Entity SQL example, the ADO.NET VLDB conferences.
Entity Framework translates this query into the ap-
propriate form for sending to the database. The da-
tabase evaluates the query and the Object Services
layer materializes the results as objects.
Programming Against the ADO.NET Entity Framework 19
 Performing Inserts, Updates, and
Deletes
So far, all the examples have been read operations.
The ADO.NET Entity Framework’s Object Services
layer also supports the inserting, updating, and de-
leting entities. The following code snippet shows
how you can create a new RunEvent entity, along
with two related Participants:
// Insert new entities and relationships
RunEvent runEvent = new RunEvent();
runEvent.EventName = "Seattle Marathon";
runEvent.StartLocation = "Memorial Stadium";
runEvent.StartDate = new DateTime(2007, 11, 25);
runEvent.EndDate = new DateTime(2007, 11, 25);
runEvent.Distance = 26.2M;
runEvent.ElevationGain = 4000;
entities.AddToEvents(runEvent);
Participant p1 = new Participant();
p1.ParticipantName = "Colin";
entities.AddToParticipants(p1);
Volunteer v1 = new Volunteer();
v1.ParticipantName = "Carl";
entities.AddToParticipants(v1);
runEvent.Participants.Add(p1);
runEvent.Participants.Add(v1);
entities.SaveChanges();
Simply create a new instance of the RunEvent class
and set the properties. In this model, the database
server generates the EventID property, so it is not
necessary to set it here.
After creating the RunEvent object, add it to the list
of objects that the object context is tracking by call-
ing the AddToEvents() method. The generated object
context class has methods for adding objects to each
entity set, named AddTo[EntitySetName](). By calling
AddToEvents(), you make the object context aware
of the new Event entity for the purposes of change
tracking and communicating with the database.
In addition to creating the RunEvent object, the
code also creates two Participant objects in much the
same way. One of the participants is a volunteer, so
it creates an instance of the Volunteer derived class.
For both objects, the code calls the context’s Add-
ToParticipants() method to include them in the set of
objects the context is tracking.
Finally, the code adds the participant objects to the
Participants collection on the RunEvent object. This
establishes the relationship between the event and
the participants.
At this point the new objects and their relationships
only exist in memory. To persist them, you call the
SaveChanges() method on the context. This method
propagates all changes it has tracked to the database.
The ADO.NET Entity Framework’s mapping engine
20 Programming Against the ADO.NET Entity Framework
determines the appropriate database tables to modify
as a result of the operations you performed with the
objects. In this example, the operations add rows to
the tables that store information about events and
participants, as well as the link table used to relate
the two.
Though this example just showed inserting new ob-
jects, you can perform updates and deletes in much
the same way. For updates, the code simply modifies
the properties of previously read objects. The object
context will keep track of the changes and persist
them when you call SaveChanges(). For deleting en-
tities, the object context provides a DeleteObject()
method, which you can use to mark an object as
deleted. Again, when you call SaveChanges(), the
ADO.NET Entity Framework propagates the delete
operation to the database.
Programming with EntityClient
By design, programming with EntityClient follows
the same patterns familiar to ADO.NET 2.0 devel-
opers. You use a connection object to establish a
connection to a model, a command object to specify
a query, and a data reader object to retrieve query
results. EntityClient represents the results as data re-
cords, shaped according to the Entity Data Model.
Unlike Object Services, the EntityClient layer sup-
ports only Entity SQL as a query language. There is
no LINQ support at this layer because there are no
strongly-typed objects over which queries could be
expressed. Also, EntityClient does not support write
operations. To perform inserts, updates, or deletes,
you must use the Object Services layer.
Despite these restrictions, EntityClient is a remark-
ably useful and efficient way to program against en-
tities. This section looks at some code examples that
illustrate EntityClient features.
Basic Querying with EntityClient
The following code snippet shows EntityClient used
to retrieve all Event entities of type RunEvent (no-
tice that this is the same Entity SQL query used in
an earlier Object Services example—EntityClient
supports exactly the same Entity SQL syntax):
using (EntityConnection conn = new
EntityConnection("name=TriEventsEntities"))
{
conn.Open();
EntityCommand cmd = conn.CreateCommand();
cmd.CommandText = @"
SELECT VALUE e
FROM TriEventsEntities.Events AS e
WHERE e IS OF (ONLY TriEventsModel.RunEvent)";
// Use data reader to get values
using (DbDataReader reader =
cmd.ExecuteReader(
CommandBehavior.SequentialAccess))
{
www.code-magazine.com
 while (reader.Read())
{ {
Console.WriteLine(
"Event Name: {0}",
(string)reader["EventName"]);
} EntityType recordType =
} extRecord.DataRecordInfo.RecordType.EdmType
} as EntityType;
The first line creates a new EntityConnection within
a using block (this ensures the connection is disposed
properly). The EntityConnection constructor takes a
connection string. Here I’ve used the same named
connection string used under the hood in the pre-
ceding Object Services examples. Alternatively, you
could have specified any other connection string, as
long as it consisted of the three parts described ear-
lier.
After creating the connection object, the code calls
the Open() method to open the connection. Next it
creates an EntityCommand object associated with
this connection by calling the connection’s Create-
Command() method. Then it sets the command text
to an Entity SQL string.
Finally, it calls ExecuteReader() to execute the query
and obtain a data reader to iterate over the results.
This part of the code should look identical to results
processing code you may have written against ADO.
NET 2.0 providers. It uses a while loop to read each
record and prints the value of one of its fields.
EntityClient uses the same mapping engine as the
rest of the ADO.NET Entity Framework for perform-
ing query translation. The engine translates the query
written here into database terms and reassembles the
results into the shapes specified by the Entity Data
Model. Because EntityClient does not materialize
objects for the results, the overhead incurred is lower
than in the Object Services layer.
Accessing Metadata from EntityClient
While the results that come back from EntityCli-
ent queries may appear similar to those returned by
other data providers, there is one important differ-
ence: EntityClient records come with rich metadata
that describes the entity types that define the record
structure. You can use this to your advantage when
writing code that processes the result data.
Notice in the previous example that you had to
hardcode the name of the field (“EventName”) that
you wanted to access in the result record. If you
wanted to build a more dynamic system that printed
all result fields without knowing them in advance,
you could use metadata to do so, as the following
example shows:
// Use metadata to drive result retrieval
using (DbDataReader reader =
cmd.ExecuteReader(
CommandBehavior.SequentialAccess))
{
www.code-magazine.com
while (reader.Read())
IExtendedDataRecord extRecord =
reader as IExtendedDataRecord;
foreach (EdmMember member in
recordType.Members)
{
if (member is EdmProperty)
{
Console.WriteLine("{0}: {1}",
member.Name,
reader[member.Name]);
}
}
}
}
The code in this snippet could go in place of the re-
sults processing code in the previous snippet. Within
the while loop used to iterate over the result records,
you cast the reader to the IExtendedDataRecord in-
terface. This is a new interface, introduced in the
ADO.NET Entity Framework, and provides access
to the metadata associated with the results. From
the IExtendedDataRecord you can navigate to the
EdmType property to determine the entity type for
each record. The ADO.NET Entity Framework rep-
resents the entity type with an instance of Entity-
Type, a class in the ADO.NET Entity Framework’s
Metadata system (documented fully in MSDN).
Given the EntityType object, the code iterates over
the type’s members and prints the name and value
of each one. You can obtain the values by indexing
into the data record using the member name. The
if-statement that checks whether the member is of
type EdmProperty is necessary to filter out naviga-
tion properties, which do not appear in the result
records.
In a real application you could use the metadata
returned with EntityClient records to build dynamic
UI or to provide more context for the meaning of
the results. Though I did not show it in this article,
the same metadata is also available at the Object
Services layer.
Wrapping Up
This article provides just a brief overview of the
features in the ADO.NET Entity Framework’s pro-
gramming surface. The ADO.NET Entity Frame-
work’s documentation and samples describe the
features covered here (and several more I did not
cover) in greater detail. The ADO.NET Entity
Framework team encourages you to try the Beta 2
release and share your feedback.
Shyam Pather
Programming Against the ADO.NET Entity Framework 21
ONLINE QUICK ID 0712052
Andrew Conrad
aconrad@microsoft.com
http://blogs.msdn.com/
aconrad/
Andrew Conrad is a Lead
Software Design Engineer
with the SQL Server group at
Microsoft Corporation. He was
one of the core designers of
the LINQ over DataSet and
LINQ to Entities technologies
and has been working with the
LINQ project since close to its
inception.
His current projects include
codename Project “Astoria”
(REST over Entities) and the
codename Project “Jasper”
incubation project (ADO.NET for
Dynamic languages). For more
information on these projects
see: http://msdn2.microsoft.
com/en-us/data/bb419139.
aspx.
22 Rich Query for DataSet – An Introduction to LINQ to DataSet
Rich Query for DataSet —
An Introduction to LINQ to
DataSet
For years developers have been asking for query over data contained
in a DataSet in a way that supports the expressiveness needed by
today’s data-centric .NET applications. As part of the .NET framework 3.5,
Microsoft® will introduce support for a technology called Language Integrated Query
(LINQ), and with this introduction, an implementation of LINQ to DataSet.
T
he ADO.NET programming model gave us the the DumpResults method to execute queries and ex-
ability to explicitly cache data in an in-memory amine the results.
data structure called the DataSet. The DataSet
uses the relational paradigm of tables and columns
to store and represent the cached data. In addition, LINQ to DataSet—What Every Developer
the DataSet also exposes a number of services (i.e., Needs to Know
metadata, constraints, change
tracking, etc.) that makes work- Fast Facts LINQ queries target
ing with data from a relational IEnumerable<T> sources. Unfor-
database an easy and straightfor- LINQ to DataSet queries tunately, the DataSet’s DataTable
ward task. can be arbitrarily complex and the DataTable’s DataRow-
and can contain anything that Collection do not implement
The one glaring weakness of can be expressed IEnumerable<DataRow> so these
the DataSet is not exposing rich in the host language types cannot be used as sources for
query capabilities, the kind of LINQ query expressions. To work
(C# or Visual Basic).
support developers have grown around this issue, an extension
accustomed to when accessing method called AsEnumerable was
relational data. Because of this limitation, developers added to the DataTable type. This extension method
have had to live with DataSet’s limited query mecha- takes a source DataTable and wraps it in an object of
nism (via simple string expressions for sorting or filter- the type EnumerableRowCollection which implements
ing) or have had to build their own custom rich query IEnumerable<DataRow>. This allows DataTables to be
implementations on top of DataSet. This has lead to a source for LINQ query expressions.
a lot of extra work and custom application code that
must be maintained by developers. With the exception of some DataSet-specific LINQ
query operators and features that I will discuss later
As part of .NET Framework 3.5, Microsoft introduces in this article, DataSet’s LINQ implementation does
support for a technology called Language Integrated not differ from LINQ to Objects. In fact, the DataT-
Query (LINQ). LINQ exposes a common query ab- able.AsEnumerable method turns a given DataTable
straction integrated into the .NET programming lan- into a sequence of objects where the element type is
guages (C# and Visual Basic) for querying all kinds DataRow. To illustrate this point, the following code
of data including objects, XML, and relational data. shows two, equivalent, LINQ query expressions both
Hence, supporting LINQ to DataSets seemed like a using the standard LINQ to Objects implementation
natural fit. of the Where query operator to apply the predicate to
the IEnumerable<DataRow> source.
All code snippets in this article will run against a sam-
ple instance of DataSet created with the code in List- var query1 = from park in parksDataTable.AsEnumerable()
ing 1. This code creates a DataSet with three tables: where (string) park["Country"] == "Japan"
National Parks, Countries, and Continents. The Data- select park;
Set also contains a DataRelation from Continents to
Countries. After creating the DataTables and DataSet, List<DataRow> list =
data is added to the sample DataSet instance. new List<DataRow>(
parksDataTable.Rows.Cast<DataRow>());
In Listing 2 you’ll see a small utility method for dump-
ing the query results to the console if required. All of var queryLinqtoObjects = from park in list
the samples in this article will run against the sample where (string)park["Country"] == "Japan"
DataSet mentioned earlier, making it possible to use select park;
www.code-magazine.com
 Listing 1: Code to create sample DataSet
DataSet ds = new DataSet("Parks"); new object[] { 3, "Asia" });

DataTable parksDataTable = ds.Tables.Add("National Parks"); countriesDataTable.Rows.Add(

parksDataTable.Columns.Add("ID", typeof(int)); new object[] { 1, "Canada", 31612000, 1});
parksDataTable.Columns.Add("Name", typeof(string)); countriesDataTable.Rows.Add(
parksDataTable.Columns.Add("YearEstablished",typeof(int)); new object[] { 2, "USA", 302249000, 1 });
parksDataTable.Columns.Add("Country", typeof(string)); countriesDataTable.Rows.Add(
parksDataTable.Columns.Add("Rating", typeof(int)); new object[] { 3, "Argentina", 39921833, 2 });
countriesDataTable.Rows.Add(
DataTable countriesDataTable = ds.Tables.Add("Countries"); new object[] { 4, "Japan", 127433000, 3 });
countriesDataTable.Columns.Add("ID", typeof(int)); countriesDataTable.Rows.Add(
countriesDataTable.Columns.Add("Name", typeof(string)); new object[] { 5, "South Korea", 49024737, 3 });
countriesDataTable.Columns.Add("Population", typeof(long));
countriesDataTable.Columns.Add("ContinentID", typeof(int)); parksDataTable.Rows.Add(
new object[] { 1, "Jasper", 1907, "Canada", 8});
DataTable continentsDataTable = ds.Tables.Add("Continents"); parksDataTable.Rows.Add(
continentsDataTable.Columns.Add("ID", typeof(int)); new object[] { 2, "Yoho", 1886, "Canada" , 7});
continentsDataTable.Columns.Add("Name", typeof(string)); parksDataTable.Rows.Add(
new object[] { 3, "North Cascade", 1968, "USA", 9 });
DataRelation ContinentCountryDataRelation = ds.Relations.Add( parksDataTable.Rows.Add(
continentsDataTable.Columns["ID"], new object[] { 4, "Lago Puelo", 1971, "Argentina", 8 });
countriesDataTable.Columns["ContinentID"]); parksDataTable.Rows.Add(
new object[] { 5, "Bandai-Asahi", 1950, "Japan", null });
continentsDataTable.Rows.Add( parksDataTable.Rows.Add(
new object[] { 1, "North America" }); new object[] { 6, "Saikai", 1955, "Japan", 6 });
continentsDataTable.Rows.Add( parksDataTable.Rows.Add(
new object[] { 2, "South America" }); new object[] { 7, "Jirisan", 1967, "South Korea", 8 });
continentsDataTable.Rows.Add(

Listing 2: Sample method to display LINQ query results

void DumpResults<T>(IEnumerable<T> source) {
{ foreach(T t in source)
StringBuilder sb = new StringBuilder(); {
if (typeof(T) == typeof(DataRow)) foreach(PropertyInfo pi in typeof(T).GetProperties())
{ {
foreach(DataRow dr in source.Cast<DataRow>()) object o = pi.GetValue(t, null);
{ sb.Append(o + " ");
foreach(object o in dr.ItemArray) }
sb.Append((o == DBNull.Value ? "DBNull" : o) + " "); sb.Append('\n');
sb.Append('\n'); }
} }
} Console.WriteLine(sb.ToString());
else }

When writing LINQ queries against DataSet, one
needs to be aware that the element type of the source
is specifically DataRow. The following code snippet
demonstrates why this is true.
var query2 = from park in parksDataTable.AsEnumerable()
where !park.IsNull("Rating") &&
(int) park["Rating"] > 8
select park;
null values as the DBNull constant which cannot be
cast to any CLR value type; this makes writing LINQ
queries tricky since the developer must remember to
guard all indexer calls with a call to the DataRow.Is-
Null method to verify that a column value is not equal
to DBNull, before accessing the value and unboxing
it to the desired type. If this guard is not put in place,
a runtime exception will occur when the value is DB-
Null and the code tries to unbox the value.

There are actually two interesting issues here. First,
when accessing the column values of a given DataRow,
one must call the DataRow indexer. Since the column
values can be any type, the indexer return type is object
instead of the specific type of column. In other words,
the DataRow column value accessor is weakly typed.
This allows you to use a single method to retrieve col-
umn values independent of type. On the downside, it
makes writing LINQ to DataSet queries awkward and
error prone, given that the user must remember to cast
to the desired type. Second, the DataSet represents
To make writing LINQ to DataSet queries easier and
less error prone, a generic extension method called
Field<T> was added to the DataRow class. Essentially
the Field<T> method works the same as the DataRow
indexer but allows the user to specify the static type for
the return value of the method as the generic param-
eter. More importantly, if the user specifies a nullable
type as the generic parameter, the field method will
convert any DBNull values to null. Using the field
method, here is the equivalent LINQ query from the
prior example.

www.code-magazine.com Rich Query for DataSet – An Introduction to LINQ to DataSet 23

 var query3 = from park in parksDataTable.AsEnumerable()
where park.Field<int?>("Rating") > 8
select park;
As a general rule, all access to DataRow column values
should be done through the Field<T> method when
developing LINQ to DataSet query expressions. This
makes the query expression much less error prone and
generally provides much cleaner code.
Since DataSet is an entirely in-memory cache, all
LINQ queries against it are translated directly into
.NET IL (intermediate language). This is different from
LINQ technologies such as LINQ to SQL or LINQ to
Entities which require the LINQ query to be translated
to the target source’s query language, and are therefore
limited by what can be translated. As a result, LINQ
to DataSet queries can be arbitrarily complex and can
contain anything that can be expressed in the host lan-
guage. For example, here is a LINQ to DataSet query
that could not be easily translated to SQL.
Func<long, bool> checkPopulation = delegate(long i)
{ return i < 100000000;};
var query4 = from country in
countriesDataTable.AsEnumerable()
where country.Field<string>("Name").Length > 3 &&
!country.HasErrors &&
checkPopulation(
country.Field<long>("Population"))
select country;
When working with LINQ to DataSet, you should
also keep in mind how the LINQ set query operators
(Union, Distinct, Intersection, Except) work. These
operators all have two overloads, one which takes
an IEqualityComparer<T> and one that does not.
When calling one of the set query operators where
the source(s) is IEnumerable<DataRow>, the set op-
erator implementation will call DataRow.Equals() to
compare elements when an IEqualityComparer<Dat
aRow> comparer is not passed in. For developers fa-
miliar with the well known relational semantics of set
operators, the results will be quite unexpected since
the DataRow.Equals method will compare DataRow
references and not the underlying values.
To support more traditional relational semantics LINQ
to DataSet includes a type called DataRowComparer
which includes a default DataRow comparer for com-
paring two DataRows.
var source1 = from park in parksDataTable.AsEnumerable()
where park.Field<int?>("Rating") < 8
select park;
var source2 = from park in parksDataTable.AsEnumerable()
where park.Field<int>("YearEstablished") > 1960
select park;
var unionResults = source1.Union(source2,
DataRowComparer.Default);
This comparer compares the specific row values and
column types to decide if two rows are equal. It does
24 Rich Query for DataSet – An Introduction to LINQ to DataSet
not require that other metadata (i.e., column names)
match since in most cases these may differ.
Shaping the Query Results—Beyond
DataRows
LINQ query developers often want to shape their que-
ry results to meet specific application needs. The sim-
plest example of this is using the LINQ Select query
operator to project the results of a query expression.
var query5 = from park in parksDataTable.AsEnumerable()
orderby park.Field<int>("YearEstablished")
select new
{
ParkName = park.Field<string>("Name"),
Established =
park.Field<int>("YearEstablished")
};
When projecting the results of a LINQ to DataSet que-
ry, the fact that the values must be projected into a type
other than DataRow can be limiting. This must be done
because DataRows cannot be created that do not tie to
an existing DataTable. This means that the DataRow
column values are copied to the new instance based on
the specified selector and hence any changes to the new
instance are not reflected in the original DataRow.
Another detail to note—when defining the anonymous
type in the projection, a name must be specified for
each member because the value is being selected via a
method and not via a Property or Field member. This
is not really a huge issue, but a minor usability issue
when projecting from DataRows.
As a workaround you can create DataRows via the
DataTable’s DataRow factory method (DataRowCol-
lection.Add) in the projection selector expression, or
you can project the entire DataRow into a member of
a new type instance.
var query6 = from park in parksDataTable.AsEnumerable()
orderby park.Field<int>("YearEstablished")
select new
{
ParkName = park.Field<string>("Name"),
Established =
park.Field<int>("YearEstablished"),
DataRow = park
};
You might also find it interesting to group results
based on a specific key.
var query7 = from park in parksDataTable.AsEnumerable()
group park by park.Field<string>("Country") into g
select new {Country = g.Key, Count = g.Count()};
Note that the key selector for the Group By query opera-
tor uses the Field<T> method instead of the DataRow
indexer. Although using the DataRow indexer could
work, there is an even more sinister problem if one for-
gets to not unbox the return value of the indexer. Name-
ly, the compiler won’t complain if one doesn’t include
www.code-magazine.com
the proper cast expression. In some cases, this will mean
that object references, instead of the underlying values,
will be used to do comparisons between key values,
most likely not the desired result. As stated before, al-
ways use the DataRow.Field<T> method to avoid these
issues when developing LINQ to DataSet queries.
As most developers experienced in SQL program-
ming know, the Join operator is an invaluable tool
for querying relational databases where the data has
been normalized into multiple tables. Interestingly,
DataSet already has the ability to do joins via Da-
taRelation objects created by the developer to relate
multiple DataTables. This makes it entirely possible
that when using LINQ to DataSet one will not re-
quire the use of the Join query operator as much as
they would with the other LINQ technologies. The
following sample demonstrates a Join between two
DataTables without a DataRelation and the use of an
existing DataRelation.
var query8 = from park in parksDataTable.AsEnumerable()
join country in countriesDataTable.AsEnumerable()
on park.Field<string>("Country") equals
country.Field<string>("Name")
select new
{
ParkName = park.Field<string>("Name"),
Country = country.Field<string>("Name"),
Continent = country.
GetParentRow(
ContinentCountryDataRelation)
.Field <string>("Name")
};
Developers can choose to use either the LINQ Join
query operator or DataRelations. When using a Da-
taRelation, the membership of the join is pre-calculat-
ed and kept up to date based on changes to the related
DataTables. When using the LINQ Join query opera-
tor the developer does not have to maintain DataSet
DataRelations; although one may require DataRela-
tions for other functionality like constraints.
DataSet Specific LINQ Operators
In addition to the standard LINQ query operators,
LINQ to DataSet includes two special LINQ query op-
erators called CopyToDataTable and AsLinqDataView
designed specifically to work with DataTables and
IEnumerable<DataRow> sources. When using LINQ
to query a DataTable, the results of the query are often
returned as an IEnumerable<DataRow> instance. This
raises two potentially interesting issues for developers.
First, even though the individual rows in the query re-
sult are live (the individual elements in the sequence are
the same DataRows as those from the original source
DataTable(s)) the results of the query are not live. For
example, after modifying one of the DataRows within
the set of query results so that it no longer qualifies as
part of the query results set, it will remain in the set of
results until the query is run again. Second, much of
the .NET infrastructure (i.e., binding, serialization, etc.)
has been built to work with DataTables and DataViews
instead of IEnumerable<DataRow> sources. Hence, it
becomes desirable in some cases to take the results on
www.code-magazine.com
a LINQ to DataSet query and turn it into a DataView
or load it into a DataTable.
The AsLinqDataView query operator takes a LINQ to
DataSet query and turns it into a DataView instance. Al-
though similar to any standard DataSet DataView, this
instance requires you to specify the predicate and order
by selectors in the LINQ to DataSet query as lambda
expressions instead of the string-based query language
currently supported when creating DataViews.
DataView dataView = (from park in
parksDataTable.AsEnumerable()
where park.Field<int?>("YearEstablished") < 1960 Since LINQ to
orderby park.Field<string>("Country")
select park).AsDataView();
DataSet
supports all
DataRowView[] drv = dataView.FindRows("Canada"); standard LINQ
Note that when calling the AsLinqDataView method, operators and
the implementation will create a DataView index expressions,
based on the criteria in the query. This index, main-
tained on changes to the underlying source DataRows there is actually a
or changes to the DataRowView instances, is based on significant amount
the key selector(s) specified in the OrderBy clause so
that any lookups via the DataView happen via the un- of functionality
derlying index and will generally perform much better supported which is
than calling the Where query operator.
well beyond most
If you do not require live data and do not want the of the capabilities
overhead associated with maintaining it, you are able
to copy the results of a LINQ to DataSet query expres- of typical query
sion into a new or existing DataTable via the CopyTo- mechanisms.
DataTable query operator.
DataTable newParksTable = (from park in
parksDataTable.AsEnumerable()
where park.Field<int?>("YearEstablished") < 1960
orderby park.Field<int?>("YearEstablished")
select park).CopyToDataTable();
When using this operator, the DataRow column values
are copied into new DataRows, meaning that the data
is no longer live with respect to the original source. It
is up to the developer to merge back any changes to
the original source DataTable.
Conclusion
You’ll encounter many pleasures when using the LINQ
technology, such as discovering where you can use it to
solve common coding problems, often in significantly
fewer lines of code and in a much more maintainable
manner. Since LINQ to DataSet supports all standard
LINQ operators and expressions, it supports a signifi-
cant amount of functionality, well beyond most of the
capabilities of typical query mechanisms. Hence, one
should be open to experimenting with LINQ to DataS-
et to solve programming problems beyond those solved
by simple filtering, ordering, and projecting operations
demonstrated by the samples in this article.
Andrew Conrad
Rich Query for DataSet – An Introduction to LINQ to DataSet 25
ONLINE QUICK ID 0712062
Elisa Flasko
Elisa.Flasko@microsoft.com
Elisa is a Program Manager with
the Data Programmability team at
Microsoft focused on a number
of Data Platform Development
technologies including ADO.
NET, XML and SQL Native Client
products. Prior to joining Data
Programmability, Elisa worked as
a Technical Presenter Business
Development Manager for
Microsoft, travelling across North
America. Elisa has previously
worked in both large and small
companies, in positions ranging
from sales to quality assurance
and from software development
to program management.
Many aspects
of LINQ to SQL have
been architected
with simplicity
and developer
productivity
in mind.
APIs have been
designed to “just
work” for common
application
scenarios.
26 LINQ to Relational Data: Who’s Who?
LINQ to Relational Data:
Who’s Who?
With the combined launch of Visual Studio 2008, SQL Server
2008, and Windows Server 2008, Microsoft is introducing five
implementations of .NET Language Integrated Query (LINQ).
Of these five implementations, two specifically target access to relational databases:
LINQ to SQL and LINQ to Entities.
M
icrosoft Language Integrated Query With this knowledge, you can see that many as-
(LINQ) offers developers a new way to pects of LINQ to SQL have been architected with
query data using strongly-typed queries simplicity and developer productivity in mind.
and strongly-typed results, common across a APIs have been designed to “just work” for com-
number of disparate data types including rela- mon application scenarios. Examples of this design
tional databases, .NET objects, and XML. By us- include the ability to replace unfriendly database
ing strongly-typed queries and results, LINQ im- naming conventions with friendly names, map
proves developer productivity with the benefits of SQL schema objects directly to classes in the ap-
IntelliSense and compile-time error checking. plication [a table or view maps to a single class; a
column maps to a property on
LINQ to SQL, released with the the associated class], implic-
Visual Studio 2008 RTM, is de- Fast Facts itly load data that has been re-
signed to provide strongly-typed With the introduction quested but has not previously
LINQ access for rapidly devel- of Language Integrated Query been loaded into memory, and
oped applications across the in both C# and Visual Basic use common naming conven-
Microsoft SQL Server family of tions and partial methods to
and the release of Visual
databases. provide custom business or up-
Studio 2008,
date logic.
LINQ to Entities, released in an Microsoft is introducing FIVE
update to Visual Studio 2008 in different implementations of Partial methods, a new feature
the first half of 2008, is designed LINQ: LINQ to Objects, of C# and Visual Basic in Visu-
to provide strongly-typed LINQ LINQ to SQL, LINQ to Entities, al Studio 2008, allow one part
access for enterprise-grade ap- LINQ to XML, and LINQ to of a partial class to define and
plications across Microsoft SQL call methods that are invoked,
DataSet.
Server and third-party databases. if implemented in another part
of the class, otherwise the en-
tire method call is optimized away during compi-
What Is LINQ to SQL? lation. By using common naming conventions in
conjunction with these new partial methods and
LINQ to SQL is an object-relational mapping partial classes, introduced in Visual Studio 2005,
(ORM) implementation that allows the direct 1-1 LINQ to SQL allows application developers to
mapping of a Microsoft SQL Server database to provide custom business logic when using gener-
.NET classes, and query of the resulting objects ated code. Using partial classes allows developers
using LINQ. More specifically, LINQ to SQL has the flexibility to add methods, non-persistent mem-
been developed to target the rapid development bers, etc., to the generated LINQ to SQL object
scenario against Microsoft SQL Server. classes. These partial methods can add logic for
insert, update, and delete by simply implementing
Figure 1 & Figure 2 combined with the code snip- the associated partial method. Similarly, develop-
pet below demonstrate a simple LINQ to SQL ers can use the same concepts to implement partial
scenario. Figure 1 shows the LINQ to SQL map- methods that hook up eventing in the most com-
ping, and Figure 2 shows the associated database mon scenarios, for example OnValidate, OnStatus-
diagram, using the Northwind database. This code Changing or OnStatusChanged.
snippet shows a simple LINQ query against the
Northwind database. Microsoft developed LINQ to SQL with a mini-
mally intrusive object model. Developers can
DataContext db = new DataContext(); choose not to make use of generated code and in-
stead create their own classes, which do not need
var customers = from c in db.Customers to be derived from any specific base class, meaning
where c.City == "London" that you can create classes that inherit from your
select c; own base class.
www.code-magazine.com
Northwind Database Diagram
Inheritance, an important feature of object-oriented
programming, does not translate directly into the
relational database. Given this, the ability to map
in inheritance is very important. LINQ to SQL sup-
ports one of the most common database inheritance
mappings, where multiple classes in a hierarchy are
mapped to a single table, view, stored procedure,
or table valued function using a discriminator col-
umn to determine the specific type of each row/in-
stance.
As with any application framework, developers
must also have the ability to optimize the solution
to best fit their scenario. LINQ to SQL offers a
number of opportunities to optimize, including us-
ing load options to control database trips and com-
piled queries to amortize the overhead inherent in
SQL generation.
By default, LINQ to SQL enables deferred loading.
This means that if, for example, I query for my Cus-
tomer data using the Northwind model in Figure
2, I do not automatically pull the associated Order
information into memory. However, if I try to ac-
cess the associated Order information via a naviga-
tion property from a Customer instance, the associ-
ated Order information is automatically pulled into
memory for me in a second database round trip. In
the following code snippet the Order information
is not loaded into memory until I access it from the
second foreach statement.
www.code-magazine.com
Figure 1: Database diagram for the Northwind database.
Conceptual Data
Programming with
ADO.NET
var customers = from c in db.Customers Relational databases are often
where c.Orders.Count > 5 designed and normalized, not to
select c; make programming against them
easier, but to ensure performance,
data consistency, and concurrency.
foreach(var row in customers) Seldom do developers work
{ directly with data in the form that
Console.WriteLine("Customer ID = " + row.CustomerID);
is returned from a database. With
the upcoming introduction of
foreach(var order in row.Orders)
the ADO.NET Entity Framework,
Console.WriteLine("Order ID = " + order.OrderID);
Microsoft is beginning down a
}
path that will abstract away the
complications of dealing with data
In the above example, a separate query is executed at the logical database layer.
to retrieve the Orders for each Customer. If you
know in advance that you need to retrieve the or-
ders for all customers, you can use LoadOptions The ADO.NET Entity Framework,
to request that the associated Orders be retrieved brings to life the entity-relationship
along with the Customers, in a single request. model that many have used for
thirty years to capture and diagram
conceptual models of their data,
Also by default, LINQ to SQL enables ObjectTrack-
prior to mapping this information
ing, which controls the automatic change tracking
into a final relational model.
and identity management of objects retrieved from
The ADO.NET Entity Framework
the database. In some scenarios, specifically where implements the entity-relationship
you are accessing the data in a read-only manner, model in the form of the Entity
you may wish to disable ObjectTracking as a perfor- Data Model (EDM), maintaining the
mance optimization. principle concepts of entities and
relationships as first class data
Compiled queries offer another opportunity to types, and allowing developers
further optimize query performance. In many ap- to program against the same
plications you might have code that repeatedly conceptual model or business
executes the same query, possibly with different concepts that were used in design.
LINQ to Relational Data: Who’s Who? 27
 LINQ to SQL Mapping Diagram
Figure 2: LINQ to SQL mapping diagram
for a simple scenario using the Northwind
database and the associated database
diagram. Notice the use of an intermediary
table to map the many-to-many relation-
ship between Employees and Territories.
argument values. By default, LINQ to SQL parses
the language expression each time to build the cor-
responding SQL statement, regardless of whether
that expression has been seen previously. Compiled
queries allow LINQ to SQL to avoid reparsing the
expression and regenerating the SQL statement for
each repeated query.
DataContext db = new DataContext();
var customers =
CompiledQuery.Compile(
(DataContext context, string filterCountry ) =>
from c in db.Customers
where c.Orders.Count > 5
select c;
foreach(var row in customers(db, "USA"))
{ wind database and LINQ to SQL you could have
Console.WriteLine(row);
} and a DiscontinuedProducts class that inherits from
foreach(var row in customers(db, "Spain"))
{ clude additional information, for example stating
Console.WriteLine(row);
} to SQL supports Table per Hierarchy (TPH) inheri-
The above code snippet shows an example of a
simple compiled query, executed twice with varying
parameters.
When Do I Use LINQ to SQL?
The primary scenario for using LINQ to SQL is in
applications with a rapid development cycle and a
simple one-to-one object to relational mapping. In
other words, you want the object model to be struc-
tured similarly to the existing structure of your data-
base; you can use LINQ to SQL to map a subset of
28 LINQ to Relational Data: Who’s Who?
tables directly to classes, with the required columns
from each table represented as properties on the
corresponding class. Usually in these scenarios, the
database has not been heavily normalized.
As an example, consider a simple retail application
that uses the Northwind database. As you look at
the Northwind database you can see a simple archi-
tecture that maps easily to a simple object model.
Using the LINQ to SQL Designer you can select the
subset of tables that best fit your
application, rename tables or
properties to make them friendli-
er, and create an object relational
mapping to develop against. Fig-
ure 1 & Figure 2 show the LINQ
to SQL mapping and associated
database diagram for a subset of
tables from the Northwind data-
base. If you look more closely at
the object mapping you can see
that foreign keys from the data-
base are represented in the object
model as relationships between classes and allow
you to navigate from one object to another.
In looking at the many-to-many relationship be-
tween Employees and Territories in the diagram,
and by digging further into the associated relation-
ship properties, you can see that LINQ to SQL does
not directly support many-to-many relationships.
Rather, LINQ to SQL uses an intermediary class
named EmployeeTerritory with a one-to-many rela-
tionship to Employees and to Territories.
The LINQ to SQL Designer also allows additional
functionality for you to expose stored procedures
and/or table valued functions as strongly typed
methods on the generated DataContext, and map
inserts, updates, and deletes to stored procedures if
you choose not to use dynamic SQL.
The above example does not show a mapping for
any type of inheritance, although using the North-
chosen to use inheritance to create a Products class
Products. The DiscontinuedProducts class may in-
that the product has been discontinued, etc. LINQ
tance and would therefore map this two-class hier-
archy directly to the existing Northwind database as
the single Products table within, using the discrimi-
nator column “Discontinued”.
What Is LINQ to Entities?
LINQ to Entities provides LINQ access to data ex-
posed through the ADO.NET Entity Framework from
Microsoft SQL Server or other third-party databases.
The ADO.NET Entity Framework is a platform,
implementing the Entity Data Model (EDM),
www.code-magazine.com
Northwind Database with Modified
Employees Tables
Figure 3: Database diagram for a modified Northwind
Database. The Employees table has been vertically partitioned:
Employees_Personal and Employees_AddressBook.
which provides a higher level of abstraction when
developing against databases. For further discus-
sion of the Entity Framework and EDM, please see
the sidebar called Conceptual Data Programming
with ADO.NET in this article, or An Entity Data
Model for Relational Data by Michael Pizzo or
Programming Against the ADO.NET Entity Frame-
work by Shyam Pather, also found in this issue of
CoDe Focus.
More than a simple ORM, the ADO.NET Entity
Framework and LINQ to Entities allow develop-
ers to work against a conceptual or object model
with a very flexible mapping and the ability to ac-
commodate a high degree of divergence from the
underlying store.
Figure 3 & Figure 4 below show a simple LINQ to
Entities scenario, using a slightly more flexible map-
ping than seen in LINQ to SQL. Figure 3 shows the
database diagram including the changes that you
could make to the Northwind database, splitting
Employee information between two tables, to dem-
onstrate two common flexible mapping concepts,
and Figure 4 shows the corresponding conceptual
model. In these figures you can see that the object
model is not mapped directly, one-to-one, to the
database. The code snippet below shows a simple
LINQ query against this database.
var customers = from c in db.Customers
where c.Orders.Count > 5
select c;
www.code-magazine.com
Microsoft designed the ADO.NET Entity Frame-
work, and in turn LINQ to Entities, to enable flex-
ible and more complex mappings, ideal in the en-
terprise type scenario, allowing the database and
applications to evolve separately. When a change
is made in the database schema, the application is
insulated from the change by the Entity Framework,
and you don’t have to rewrite portions of the ap-
plication, but rather to simply update the mapping
files to accommodate the database change.
Similar to LINQ to SQL, LINQ to Entities uses
partial classes and partial methods to allow cus-
tomer update and business logic to be easily added
to generated code. LINQ to Entities also provides
the ability to declaratively call stored procedures
and use generated Update views when persisting
objects.
Three common mapping scenarios differentiate
LINQ to Entities from LINQ to SQL. LINQ to Enti-
ties provides the ability (through more flexible map-
pings) to map multiple tables or
views to a single entity or class,
to directly map many-to-many
relationships, and to map addi- LINQ to Entities and the Entity
tional types of inheritance. Framework allow you to map Table
Although you can map many-
per Hierarchy [inheritance] as well
to-many relationships in both as Table per Concrete Type…
LINQ to SQL and LINQ to
Entities, LINQ to Entities al-
and Table per Subclass.
lows you to directly map many-
LINQ to Relational Data: Who’s Who? 29
 AdventureWorks Products
Figure 4: LINQ to Entities mapping
diagram corresponding to the modified
Northwind database. Notice the directly
mapped many-to-many relationship
between Employees and Territories
without an intermediary table and the
Employees_Personal and Employees_
AddressBook tables are mapped into a
single entity.
to-many relationships with no intermediary class,
while LINQ to SQL requires that you create an in-
termediary class that maps one-to-many to each of
the classes that are party to the many-to-many rela-
tionship.
As discussed earlier in this article, LINQ to SQL lets
you map one of the most common inheritance sce-
narios, Table per Hierarchy. LINQ to Entities and the
ADO.NET Entity Framework allow you to map Table
per Hierarchy, similarly to LINQ to SQL, as well as
Table per Concrete Type, a separate table for each
class or type in the hierarchy, or Table per Subclass, a
hybrid approach using a shared table for information
about the base type and separate tables for informa-
tion about the derived types.
Two features of LINQ to Entities and the ADO.NET
Entity Framework that set these technologies apart is
the ability to create Entity SQL views and Defining
Queries. Entity SQL views allow you to define the
mapping between your entity model and the store
schema in terms of arbitrary Entity SQL queries. The
Defining Query feature allows you to expose a tabular
view of any native store query as a table in your stor-
age schema.
30 LINQ to Relational Data: Who’s Who?
Due to the explicit nature of LINQ to Entities, devel-
opers also have the ability to optimize the solution to
best fit their scenario. For a moment let me revisit my
previous example of LINQ to SQL implicit loading.
When I queried for Customer data, Order information
was not automatically pulled into memory, but rather
was only pulled into memory only when the Order
information was accessed. In LINQ to Entities, you
have full control over the number of database round
trips by explicitly specifying when to load such infor-
mation from the database. Navigating to associated
information that has not yet been retrieved from the
database will not cause an additional database trip.
You can further optimize LINQ to Entities by dis-
abling change tracking when working in a read-only
scenario.
When do I Use LINQ to Entities?
The primary scenario targeted by LINQ to Entities is
a flexible and more complex mapping scenario, often
seen in the enterprise, where the application is access-
ing data stored in Microsoft SQL Server or other-third
party databases.
www.code-magazine.com
In other words, the database in these scenarios con-
tains a physical data structure that could be significant-
ly different from what you expect your object model to
look like. Often in these scenarios, the database is not
owned or controlled by the application developer(s),
but rather owned by a DBA or other third party, pos-
sibly preventing application developers from making
any changes to the database and requiring them to
adapt quickly to database changes that they may not
have been aware of.

As an example, consider a simple HR application that Figure 5: Database

uses an existing company database. The database cur-
rently exists as our modified Northwind example in
Figure 3. In looking at the database, you can see that
the architecture does not lend itself directly to how
you may think of the business objects; specifically
you do not want to deal with two separate Employee
classes where each class contains half of the informa-
tion that the application will be accessing. You also do
not want to think about an EmployeeTerritory object
that does not add value beyond what is already pro-
vided in the Employee and Territory objects.
For this example I will need to create a conceptual
model that exists with a single Employee object type,
and a direct relationship between Employees and Ter-
ritories to allow ease of navigation.
When I created the Entity Data Model using the new
EDM Designer, I begin by walking through a simple
wizard to create my base mapping. In this case I chose
to automatically generate my initial mapping from an
existing database, and then I edited the model using
the drag and drop interface. I selected the subset of
tables that best fit my application, renamed tables and
properties to make them friendlier, and made a few
other changes such as combining the two Employees
tables into a single Employee Entity. Figure 4 shows a
diagram of the Entity Data Model as previewed in the
EDM Designer.
Figure 6: Entity Data Model diagram demonstrating a common
mapping of inheritance from the database in Figure 5.
The BikeEvent class, RunEvent
class, SwimEvent class,
and the TriathlonEvent class
inherit from the Event class.
diagram demonstrat-
ing Table per Subclass
hierarchy. Shared Event
table includes informa-
tion that is common
to the BikeEvent, RunEvent, SwimEvent and TriathlonEvent
tables. BikeEvent, RunEvent, SwimEvent and TriathlonEvent
tables include information specific to each entity.
Looking closely at the EDM you can see that the rela-
tionship between Employees and Territories no longer
uses an intermediary class to model the many-to-many
relationship, but rather models it directly, shown by
the *‡* symbol connecting two entities. You can also
see that I have modeled a single Employees entity in-
cluding all of the properties contained in the Employ-
ees_Personal and Employees_AddressBook tables in
the database.
In this example, you do not see a mapping for any
type of inheritance. As mentioned previously, LINQ
to Entities gives you the ability to map two types of
inheritance, in addition to the Table per Hierarchy
used in LINQ to SQL. Figure 5 and Figure 6 show an
example of Table per Subclass inheritance and how
it might be mapped to an Entity Data Model. This
EDM shows the RunEvent class, SwimEvent class,
BikeEvent class, and the TriathlonEvent class all in-
herit from the Event class.
Elisa Flasko

www.code-magazine.com LINQ to Relational Data: Who’s Who? 31

ONLINE QUICK ID 0712072
David Schach
davidsch@microsoft.com
David Schach is a principal
developer at Microsoft and was
the developer lead responsible for
LINQ to XML and XML language
integration in Visual Basic 9.0.
Prior to that, he worked on XML
and object to relational mapping
and XML features in SQL Server.
He has been involved with XML,
XSLT, and XPath and other XML
standards since 1998. In addition
to implementing XPath 1.0 and
XSLT 1.0, he was also developer
lead for msxml.
32 Browsing Windows Live Expo with LINQ to XML
Browsing Windows Live Expo
with LINQ to XML
LINQ to XML, which makes query a first class construct in C# and
Visual Basic, is the new XML API in the .NET Framework 3.5.
With the introduction of Language Integrated Query (LINQ), Microsoft is introducing
LINQ implementations that work over objects, data, and XML. LINQ to XML
improves on System.Xml in the .NET Framework 2.0 by being both simpler to use and
more efficient. Microsoft developed this new API because the W3C-based DOM API
does not integrate well into the LINQ programming model.
W
indows Live Expo, an online ser- C#
vice for buying and selling merchan-
dise, provides a set of Web services var d = XDocument.Load(
that enable programmatic access to the ser- @"http://expo.live.com/API/" +
vice’s listings, and returns results as XML. "Classifieds_GetCategories.ashx?appKey=" +
In this article, I’ll show you how to build a "-1690295249159343350");
Web page that searches and displays items from
the Windows Live Expo listings. To make the Visual Basic
page a little more interesting, I’ll show you how
to integrate the Virtual Earth map control and Dim d = XDocument.Load("http://expo.live.com/" _
use it to display the locations of the items found. + "API/Classifieds_GetCategories.ashx?appKey=" _
Figure 1 shows the sample application in final + "-1690295249159343350")
form.
We load XML using the static
Because Windows Live ser-
Fast Facts Load method, available on both
vices produce and consume LINQ to XML is a new lighter XDocument and XElement.
XML data, they are ideal can- weight and higher performant The example above uses XDoc-
didates to showcase the power XML API in the Microsoft ument.Load, however, it could
of LINQ to XML. Let me show .NET Framework 3.5. It also have used XElement.Load
you how easily you can build supports all of the standard because the XML is both a valid
an application using LINQ to XML axes including Elements, XDocument and XElement.
XML.
Attributes, Descendants
The XML that was returned
and Ancestors. These
has a root element named cat-
Loading XML and from methods return IEnumerable egories and a list of category
collections so that they
the Web integrate well with LINQ
elements with ID and name
attributes. The application will
queries. Visual Basic adds use the id attribute to query
You need an application ID XML language extensions to Windows Live Expo for items
in order to use the Windows LINQ to XML. With XMLAXIS in each category.
Live Expo API, which you can properties you can access
get by simply signing up on XML elements and attributes
the Live Expo site. Once you by name using a special
have an application ID, use syntax. With XML Literals you
the GetCategories function
can create XML documents,
to obtain a list of all category
names and their associated
elements, and fragments
IDs. Your application needs within using XML syntax.
these IDs to search the online
listings.
<e:categories
To make the Web page easier to use, I’ll DataBind xmlns:e="http://e.live.com/ns/2006/1.0">
the categories and IDs to an ASP.NET Drop- <e:category e:id="2" e:name="Autos">
DownList control. The code snippet below <e:category e:id="4" e:name="Boats">
shows how to obtain the categories in XML for- <e:category e:id="3" e:name="Cars and Trucks">
mat. <e:category e:id="5" e:name="Motorcycles">
www.code-magazine.com
Figure 1: Sample application browsing Windows Live Expo data and mapping results using Microsoft Virtual Earth. LINQ to XML
ties everything together.
You can write a LINQ query using the Descendants
method to get the category names and IDs. LINQ to
XML supports all of the standard XML axes such as El-
ements, Attributes, Descendants, and Ancestors. Here
I used the Descendants method because category ele-
ments can be nested within other category elements.
Below is the complete query.
C#
XNamespace expo =
"http://expo.live.com/ns/2006/1.0";
var categories = from c in d.Descendants(expo +
"category")
select new
{ XNamespace. You can initialize an XNamespace
Name = (string)c.Attribute(expo +
"name"),
Value = (string)c.Attribute(expo +
"id")
};
CategoryList.DataSource = categories;
CategoryList.DataBind();
The expression:
d.Descendants(expo + "category")
returns an IEnumerable of XElements that match
the name. For each element returned, the select
www.code-magazine.com
clause creates an object with two members, Name
and Value. Because the type is not named and de-
clared, it is an anonymous type created by the com- XML namespaces
piler. have
Assigning the query result to the CategoryList’s Da- traditionally
taSource property and calling DataBind displays been a source
the results in the page’s DropDownList control.
of confusion.
XML namespaces have traditionally been a source One of the goals
of confusion. One of the goals of LINQ to XML
was to simplify XML namespace handling. LINQ of LINQ to XML
to XML solves the problem with two new classes was to simplify
called XNamespace and XName. You can cre-
ate an XNamespace using the static method XML namespace
XNamespace.Get. However, for convenience the handling.
class also has an implicit conversion from string to
variable by assigning the namespace uri string to
the variable.
XNamespace expo =
"http://expo.live.com/ns/2006/1.0";
In LINQ to XML, all names are represented by the
XName class, a fully qualified name. That is, it al-
ways has both a namespace and a local name. To
get an XName object requires an XNamespace ob-
ject because local names are always allocated with
respect to some namespace. Here’s an example of
getting a name from the expo namespace.
expo.GetName("category")
Browsing Windows Live Expo with LINQ to XML 33
 However, you won’t see this code in the sample
application. For convenience, LINQ to XML over-
loads the + operator. You can get a fully qualified
name using the simpler code shown.
expo + "category"
Now creating and using XML names and namespac-
es is really simple.
As part of the .NET
Framework 3.5, You may have noticed the cast to string when getting
the attributes. Calling the Attribute method returns
all of the features an XAttribute object, but I want the attribute’s value.
of the LINQ to XML The XAttribute class has a Value property; however, it
API are available in is rarely used. Instead, the XAttribute and XElement
classes define a number of explicit conversions for
both C# and Visual converting objects to any of the common CLR types.
Basic. However, In this case I want a string but I could have also cast
to double, decimal or datetime if those value types
Visual Basic 9.0 were stored in the attribute.
includes additional
language features Visual Basic Xml Properties
to make XML As part of the .NET Framework 3.5, all of the features
processing even of the LINQ to XML API are available in both C#
simpler. and Visual Basic. However, Visual Basic 9.0 includes
additional language features to make XML process-
ing even simpler. One of those features is XML prop-
erties. Now, you can directly query XML in Visual
Basic without using an API.
An XML property lets you treat a sub element or
Reference Links attribute as if it were an ordinary CLR property.
For LINQ to XML Syntactically, XML element properties differ from
normal properties by enclosing the name in angle
http://msdn2.microsoft.com/
brackets. XML attribute properties are prefixed with
en-us/library/system.xml.
an ‘@’ character. Below you can see the same LINQ
linq(VS.90).aspx
query in Visual Basic using element and attribute
http://msdn2.microsoft.com/ properties to get the category names and ids.
en-us/library/bb387098(VS.90).
aspx Visual Basic
For Visual Basic XML literals Dim categories =
From c In d...<expo:category> _
http://msdn2.microsoft.com/ Select Name = c.@expo:name, _
en-us/library/bb384833(VS.90). Value = c.@expo:id
aspx
http://msdn2.microsoft.com/ CategoryList.DataSource = categories
CategoryList.DataBind()
en-us/library/bb384563(VS.90).
aspx
http://msdn2.microsoft.com/ What happened to the calls to Descendants and Ele-
en-us/library/bb384769(VS.90). ments? In addition to special syntax for element and
aspx attribute names there is also special syntax for com-
mon XML query axes. Table 1 shows how LINQ to
XML methods map to Visual Basic property syntax.
Tool for XML to Schema
Inference
LINQ to XML has both an Element method and an
http://msdn2.microsoft.com/ Elements method. The former returns a single element
en-us/vbasic/bb840042.aspx while the latter returns a collection. However, the
XML element property in Visual Basic maps to the El-
ements method not the Element method. That means
For information about LINQ
XML element properties are always a collection.
http://msdn2.microsoft.
com/en-us/netframework/ In Visual Basic you do not have to use the XML
aa904594.aspx property syntax but it is usually more convenient and
34 Browsing Windows Live Expo with LINQ to XML
compact than using the API directly. Sometimes you
have to use the API because there are many methods
in LINQ to XML that do not have a special syntax in
Visual Basic such as Ancestors.
In the example, the element and attribute names are
qualified. In C# you saw how to create fully quali-
fied names using an XNamespace object along with
implicit string conversions. Visual Basic goes one step
further and extends the imports statement to support
XML namespaces in addition to CLR namespaces.
You do not have to define any XNamespace objects
in the code. Just add an imports statement for each
XML namespace you need in the code. After that you
can use the prefix anywhere in the source file. Below
you can see how to define the expo and geo prefix in
Visual Basic.
Imports <xmlns:expo="http://expo.live.com...">
Imports <xmlns:geo="http://www.w3.org/2003...">
Querying for the Items
Now that you have the categories, I’ll show you how
to write the code to search for the items on Windows
Live Expo. I’ll use the ListingsByCategoryKeywordLo-
cation method to do that. This method takes a cat-
egory, a set of keywords, a location, either latitude
and longitude or a postal code, a maximum distance
to include in the search, and a maximum number of
results. You’ll get this information from the user from
the controls on the page.
With the parameterized URL constructed you can load
the XML data from Live Expo. The data comes back
as RSS with some Windows Live Expo extension ele-
ments. In the code I’ll DataBind these items to an ASP.
NET ListView control. To get the items, the query uses
the Elements method and explicitly drills into the RSS
data from the root element. For each item, the query
creates an anonymous type with the title, published
date, URL to the item’s Web page, location, and price.
Again, each element’s value is retrieved by explicitly
casting the element to the desired CLR type. The result
is then bound to an ASP.NET ListView to display on
the Web page.
C#
var d = XDocument.Load(BuildListingUrl());
var items = from item in
d.Elements("rss").Elements("channel").
Elements("item")
select new
{
Title = (string)item.Element("title"),
Published = (DateTime)item.Element("pubDate"),
Url = (string)item.Element("link"),
Location = (string)item.Element(expo +
"location").Element(expo + "city"),
Price = (decimal)item.Element(expo + "price")
};
ListView1.DataSource = items;
ListView1.DataBind();
www.code-magazine.com
The Visual Basic code is nearly the same. Just as in
C#, the element values are cast to the desired type.
However, in Visual Basic there is an explicit call to
the Value method because the XML element prop-
erty is mapped to the Elements method. This method
returns a collection of XElements. While XElement
defines explicit conversions to CLR types, the col-
lection does not. To solve this problem, Visual Basic
defined a Value extension method on the collection.
This method gets the first item from the collection
and returns its value. If the collection is empty, it re-
turns Nothing.
feed that conforms to the GeoRSS specification. I’ll
use LINQ to XML to create the GeoRSS XML.
One of the reasons why Microsoft created a new XML
API was because the W3C-based XmlDocument and
XMLElement in System.Xml cannot be functionally
created. The System.XML API uses a factory-based
pattern instead which, unfortunately, requires a fac-
tory context for creating the elements and it needs
multiple statements for constructing the XML. With
LINQ you need to do everything within the context
of a single LINQ expression.

Visual Basic The constructor for XElement, shown below, takes

a variable number of arguments.
Dim d = XDocument.Load(BuildListingUrl())
XElement(XName name, params object[] content);
Dim items =
From item In d.<rss>.<channel>.<item> _ Because each constructor can take as its argument the
Select _
constructor for its content, you can create the entire
Title = item.<title>.Value, _
Published = CDate(item.<pubDate>.Value), _
XML document or fragment within a single expression.
Url = item.<link>.Value, _
Location = _ In the LINQ query, each item from the Windows
item.<expo:location>.<expo:city>.Value, _ Live Expo data is converted to a GeoRSS item. The
Price = CDec(item.<expo:price>.Value) transformation is simple and summarized in Table 2.

ListView1.DataSource = items Listing 1 shows the C# code for the transformation.

ListView1.DataBind()
With the items converted, the Virtual Earth map

XML IntelliSense
control requests the full GeoRSS XML by embed-
ding the items inside of the RSS and channel ele-
One of the
ments. The server returns data to the map control reasons why
Visual Basic makes it even easier to use XML proper-
ties because it supports XSD-driven IntelliSense. If
via a separate aspx page. Microsoft created
you add an XSD file to the project, IntelliSense will C# a new XML API
pop up after typing a ‘<’ or ‘@’ in an expression of
var geoRss =
was because
type XElement or XDocument as shown in Figure 3.
If you don’t have a schema for your XML data, that’s new XElement("rss", the W3C-based
new XAttribute("version", "2.0"),
not a problem. Just load some sample XML into the
new XElement("channel", XmlDocument and
XML Editor in Visual Studio and infer the schema.
Save the inferred schema to your project directory
new XElement("title",
"Expo Live Result Locations"), XMLElement in
and then add the file to your project. new XElement("link", cmd),
geoItems
System.Xml
) cannot be
Creating GeoRSS for the Virtual Earth );
functionally
Map Control created.
This code correctly creates the GeoRSS XML but the
XML may not serialize exactly the way you want it
Until now the sample app has consumed XML
data from the Windows Live Expo service. Now I
want to generate some XML. The sample applica- C# Visual Basic
tion uses a Virtual Earth map control to display the d.Descendants d...<
locations of the items returned from Windows Live d.Elements d.<
Expo. After the map control is created on the Web d.Attribute d.@
page, it requests the pushpin locations from the
server. The server returns the locations in an RSS Table 1: LINQ to XML methods and XML properties.

Windows Live Expo Data GeoRSS

<item> <item>
<title> <title>
<description> <description>
<expo:location><expo:latitude> <geo:lat>
<expo:location><expo:longitude> <geo:long>
Table 2: Transforming Windows Live Expo data to GeoRSS data.

www.code-magazine.com Browsing Windows Live Expo with LINQ to XML 35

 Listing 1: Windows Live Expo data transformation to GeoRSS items and the Virtual Earth map control displayed pushpins
showing each item’s location. When I counted the
var d = XDocument.Load(BuildListingUrl()); pushpins, however, there weren’t enough.
var items = from item in
The problem is that the position information for each
d.Elements("rss").Elements("channel").Elements("item")
item is the latitude and longitude of the item’s postal
select new
{ code. Two items with the same postal have the same
Title = (string)item.Element("title"), position so the map control only displays one pushpin.
Published = (DateTime)item.Element("pubDate"), Rather than generate multiple overlapping pushpins,
Url = (string)item.Element("link"), I need to generate one pushpin and concatenate the
Location = (string)item.Element(expo + titles and descriptions from each of the items at the
location").Element(expo + "city"), same location. I can use LINQ grouping to solve this
Price = (decimal)item.Element(expo + "price") problem.
};
Grouping has two parts: an expression for the objects
ListView1.DataSource = items; to put in the group and an expression for the key to
ListView1.DataBind partition the objects. In the query, I want to put the
RSS items in the group and I want to partition on
to look. While LINQ to XML names are always fully unique latitude and longitude values. Getting the items
qualified, they do not have a prefix. If you care about for the group is easy. That is just the item variable from
the prefix assigned to a name then you need to create a the “from” clause. The key has to be a single expres-
namespace attribute that defines the prefix. Without a sion but I have two values. To solve this I’ll use a new
namespace declaration, the serializer may create a name anonymous type with two members, latitude and lon-
with a default namespace or in some cases, with a syn- gitude. I’ll then assign a name for this group so that I
thesized prefix. In this case, I want the prefix for the geo can access it in the select clause.
namespace to be “geo”. To ensure the prefix is generated
correctly. I’ll add the following code to the rss element. C#

new XAttribute(XNamespace.Xmlns + "geo", var positions =

geo.NamespaceName), from item in
d.Elements("rss").
Elements("channel").
Using Grouping Elements("item")
group item by
At this point I thought I was done with the application. new
The ASP.NET ListView control displayed the items {

Figure 2: XML IntelliSense is available in Visual Basic when you add an XSD schema to the project.

36 Browsing Windows Live Expo with LINQ to XML www.code-magazine.com

Figure 3: Inspecting the result of grouping by position. Each result has a key and a collection of items. Each item in the group has the
same key value

latitude = (string)item. <geo:long><%= _

Element(expo + "location").
Element(expo + "latitude"),
longitude = (string)item.
Element(expo + "location").
Element(expo + "longitude")
} into g
select new { position = g.Key, data = g };
The select clause then returns the group key with
the latitude and longitude and the items at that loca-
tion. See Figure 3 to inspect the group query result
in Visual Studio. The rest of the code to merge the
titles and descriptions is in the sample application.
Visual Basic XML Literals
Complementary to XML properties, are XML liter-
als. With XML literals you can create XML docu-
ments, elements, and fragments within a Visual Ba-
sic program using XML syntax. The code snippet
below shows code similar to the C# example above
to create the GeoRSS items in Visual Basic.
Visual Basic
Dim geoItems = _
From item In d.<rss>.<channel>.<item> _
Select _
<item>
<title><%= item.<title>.Value %></title>
<description><%= _
item.<description>.Value %>
</description>
<geo:lat><%= item.<expo:location>. _
<expo:latitude>.Value %></geo:lat>
item.<expo:location>. <expo:longitude>.Value _
%></geo:long>
</item>
The XML literal compiles to calls to LINQ to XML
API so the type of the geoItems variable is XElement.
In addition to XML 1.0 syntax, XML literals support
embedded Visual Basic expressions. You use embedded
expressions to programmatically create content within
an XML literal. They use an ASP.NET-like syntax and
can contain any valid Visual Basic expression.
Because the XML literal is processed and understood
by the Visual Basic compiler, you get additional sup-
port within the IDE when using XML literals. For ex-
ample, the IDE automatically inserts the end element
when starting a new element and will auto correct the
end element name when the start element name is
changed. In addition, features such as outlining, re-
naming, and find all references all work as expected
with XML literals.
Conclusion
XML is everywhere on the Web and applications need
to be able to easily produce and consume XML. While
this article barely scratches the surface of the LINQ to
XML APIs, you can see how easy it is to integrate
data from Web services using the .NET Framework
3.5. LINQ to XML vastly simplifies querying and con-
structing XML. Visual Basic takes XML programming
one level higher with XML properties, XML literals,
and XSD-driven IntelliSense.
David Schach

www.code-magazine.com Browsing Windows Live Expo with LINQ to XML 37

ONLINE QUICK ID 0712082
Mike Flasko
Mike.Flasko@microsoft.com
Mike is a Program Manager (PM)
in the SQL Data Programmability
group at Microsoft. He is
currently focused on the ADO.
NET Data Services project. Prior
to working with ADO.NET Data
Services, Mike was a PM on the
Windows Network Developer
Platform team at Microsoft,
where he was responsible for
the System.NET namespace
(.NET Framework), the Winsock
API, and the Winsock Kernel API.
Prior to his roles as a Program
Manager, Mike worked as a .NET
Developer Technology Evangelist.
38 ADO.NET Data Services
ADO.NET Data Services
Separation of presentation and data has long been considered
a best practice in the development of Web applications. Driven
by the need for low friction deployment and a richer user experience, the types and
architectures of Web applications are evolving dramatically. With the introduction
and growth of AJAX-based applications and Rich Interactive Applications (RIA)
using technologies such as Microsoft® Silverlight™, separation of presentation and
data is no longer just a best practice, it is required.
T
hese architectural changes have been accom- Primary interaction with an ADO.NET Data Ser-
panied by a number of new classes of appli- vice occurs in terms of HTTP verbs such as GET
cations. More specifically, two classes of ap- and POST, and the data exchanged in these inter-
plications have become prevalent in today’s Web actions is represented using simple formats such
landscape: AJAX-based Web applications and as XML and JSON.
RIA applications using technologies such as Micro-
soft’s Silverlight™. AJAX-based Web applications In both the AJAX application and RIA application
serve pages containing presentation and behavior, models noted above, data is retrieved via an HTTP
using JavaScript to represent page behavior and request to a middle tier server. Currently, a number
then turn back and fetch data separately, using of Web service technologies (Windows Commu-
XMLHTTP. Technologies such nication Foundation, ASMX,
as these typically remove the Fast Facts etc.) and associated standards
option of a server-side render- (WS-*, SOAP, etc.) exist and
ing process that mixes data and Originally started as may be used to serve data to
code. Instead, code written to an exploratory incubation clients in these scenarios. ADO.
drive the presentation aspects project by a few people NET Data Services and its
is pre-compiled and deployed within the ADO.NET REST-based access patterns are
to the client via the Web server. team under the Microsoft designed to be complementary
After reaching the client Web to these existing technologies.
Codename “Astoria”,
browser, the code calls back
to a Web server to retrieve the
ADO.NET Data Services For applications that are high-
data to display within the user has since moved out ly driven by business process
interface. from under the “incubation” and/or typically access data
title into a first class through a rich façade layer,
Adding to the classes of appli- member of the ADO.NET Microsoft has a strong offer-
cations mentioned, “mashup”- suite of data access ing in WCF. For applications
style applications have been that are driven more by “pure”
technologies for
introduced, which leverage data (ex. NetFlix movie queue,
the presentation/data divide
next-generation Web stock ticker, etc.) or for data/
by aggregating data from mul- applications. business logic hybrid applica-
tiple sources to provide an en- tions, ADO.NET Data Ser-
hanced experience, using the vices provides a rich offering
combined data. This landscape makes it interest- by enabling the creation of reusable UI elements
ing to talk about ADO.NET Data Services (for- with a uniform URI addressing scheme. In addi-
merly known as Microsoft Project Codename “As- tion, it also enables the introduction of business
toria”), the services that applications will use, to logic in a way that is naturally integrated with
find and manipulate data on the Web, regardless the rest of the HTTP interface such as ADO.NET
of the presentation technology used or whether or Data Services query operators (paging, sorting,
not the front-end is hosted in the same location etc.) on the server.
or on the same server as the data.
Originally introduced as an incubation project
from the ADO.NET team under the Microsoft
What Are ADO.NET Data Services? Codename “Astoria” at Mix ’07, ADO.NET Data
Services has since moved out from under the
ADO.NET Data Services, provides the ability to “incubation” title and is now a production proj-
expose data as a service, available to clients from ect. As part of our move from incubation and
across the corporate intranet or across the Inter- into development as a production product, we
net. These services are then accessible over HTTP, started from scratch using feedback and lessons
using URIs to identify the pieces of information learned from our initial CTP incubation release.
or “resources” made available through a service. If you have been following ADO.NET Data Ser-
www.code-magazine.com
vices since the beginning, you may notice a few
changes to the product in an upcoming technol-
ogy preview release. Since our initial release we
have worked a lot on the architecture of the prod-
uct, designing it such that we can broaden the ca-
pabilities of the product to address the feedback
we have received to date without adding undue
complexity to the system as a whole.
ADO.NET Data Services,
provides the ability to
expose data as a service,
available to clients from
across the corporate
intranet or across the Internet.
These services are
then accessible over HTTP,
using URIs to identify
the pieces of information or
“resources” made available
through a service.
Using ADO.NET Data Services
ADO.NET Data Services consumes a concep-
tual model representing the entities that your
application wishes to expose, along with the as-
sociations between entities, and exposes them as
HTTP resources, each identified by a URI. The
conceptual model exposed by ADO.NET Data
Services is defined using the Entity Data Model
(EDM), which is supported by the ADO.NET
Entity Framework. By virtue of the ADO.NET
Entity Framework, ADO.NET Data Services can
expose high-level constructs which are mapped
by the framework to the underlying store. For
more information on the Entity Framework
and EDM, please see Michael Pizzo & Shyam
Pather’s articles, also in this issue of CoDe Fo-
cus.
Setting up a new ADO.NET Data Service is
straightforward and, at the time of this writing,
involves creating an EDM model and a data ser-
vice using wizards in Visual Studio 2008 and then
writing a couple lines of code to hook up the data
service to the model. The site http://msdn.microsoft.
com/data includes links to step-by-step instruc-
tions showing how to create a data service using
the latest release of ADO.NET Data Services.
The following examples use the read-only North-
wind sample database and the experimental
ADO.NET Data Services Online Service to de-
tail by example how to interact with an ADO.
www.code-magazine.com
NET Data Service. As we progress towards a fi-
nal version of the product, we will likely update
the Online Service to reflect our current think-
ing; however, since the service is experimental the
URLs noted below may change or be removed all
together over time, so I recommend visiting the
ADO.NET Data Services section on http://msdn.
microsoft.com/data after reading this article to see
if any updates to the URI syntax have been made.
For example:
http://astoria.sandbox.live.com/
northwind/northwind.rse/Customers[ALFKI]
returns a single Customer resource. When a URI
points a specific resource, you can perform CRUD
operations on the resource: An HTTP GET re-
quest to the URI enables retrival of the resource,
the POST verb is used to add a new instance, the
PUT verb to update and, finally, the DELETE
verb is used to remove a resource instance.
In addition, it is possible to traverse associations
between resources using a URI such as:
http://astoria.sandbox.live.com/northwind
/northwind.rse/Customers[city eq
'London']/Orders?$orderby=OrderDate
This URI represents all the Orders for the Cus-
tomers living in London, where the results are
returned sorted by the OrderDate.
The documents linked to from the developer cen-
ter noted above details all the URI construction
rules, available query operators as well as the
data formats supported in HTTP requests and
responses to/from an ADO.NET Data Service.
At the time of this writing, the latest CTP release
supports simple data formats such as XML and
JSON. For an in-depth overview of the formats
used by ADO.NET Data Services, please see
http://msdn.microsoft.com/data.
Rich Client Access
Since all ADO.NET Data Service resources are ex-
posed by simple HTTP entry points, any applica-
tion with the ability to make an HTTP request can
access a data service. In addition to direct HTTP
access using an HTTP API, client libraries are
available for the .NET Framework and Silverlight
1.1 applications. As well as abstracting the applica-
tion from the base semantics of HTTP, the libraries
surface results as .NET objects and provide rich
services to the application such as graph manage-
ment, change tracking, and update processing.
For AJAX applications, most AJAX libraries in-
clude some type of easy-to-use wrapper object
over the XMLHTTP object. To facilitate this us-
age model, ADO.NET Data Services supports
requests and responses in JSON format. This en-
ables a payload returned from a data service to
be used as the parameter to the JavaScript Eval( )
Elisa Flasko
Elisa.Flasko@microsoft.com
Elisa is a Program Manager with
the SQL Data Programmability team
at Microsoft focused on a number
of Data Platform Development
technologies including ADO.
NET, XML, and SQL Native Client
products. Prior to joining Data
Programmability, Elisa worked as
a Technical Presenter Business
Development Manager for
Microsoft, travelling across North
America. Elisa has previously
worked in both large and small
companies, in positions ranging
from sales to quality assurance
and from software development to
program management.
What Is REST?
In his dissertation, Roy Thomas
Fielding describes Representational
State Transfer (REST) as an
architectural style making use of
existing technologies such as HTTP
and XML over the Web. The REST
architectural style outlines how to
address and define resources.
The list below highlights some
key design principles that make
up the REST interface:
• Application state and
functionality are divided into
resources
• Every resource is uniquely
addressable using a
universal syntax for use in
hypermedia links
• All resources share a
uniform interface for the
transfer of state between
client and resource
• A protocol that is:
§ Client/Server
§ Stateless
§ Cacheable
ADO.NET Data Services 39
 Data Programmability Team
Pablo Castro
“Simplicity is what keeps the
Internet working. One of the
beautiful things about the
HTTP pattern and the concept
of REST is that they both focus
on just that—simplicity.”
Pablo works in the SQL Server
product group at Microsoft,
designing the future versions
of the programming models for
working with data. He has been
involved in several releases of
large scale projects at Microsoft
such as SQL Server 2005 and the
.NET Framework. Before joining
Microsoft, Pablo spent some time
running a start-up focused on
advanced application frameworks
development, and before that he
split his time between building
collaborative applications and
creating infrastructure for
inference system creation and
distributed execution used in risk
analysis applications.
40 ADO.NET Data Services
function, which enables the AJAX application to ADO.NET Data Services is currently in active de-
directly materialize JavaScript constructs from velopment. Unfortunately, this article is unable to
the response from a data service. cover all aspects of this new technology. We’ll strive
to be very transparent with our blog at http://blogs.
microsoft.com/astoriateam. There you’ll learn more
Going Beyond “Pure Data” about how ADO.NET Data Services is progress-
ing and find out more on our current thinking in
So far I have described how ADO.NET Data Ser- various aspects of the product design, plus you can
vices map a URI to each entity within an EDM- provide us with your feedback.
based conceptual model, which a developer de-
fines and provides to a data service to expose Mike Flasko
as HTTP resources. This approach works well Elisa Flasko
for classes of data (ex. blog articles) that can be
served from the storage medium all the way to
the application where it is likely to be directly
presented to the end user. While such data ex-
ists, it is clear that not all data fits this mold.
Some types of data will, for a variety of reasons
(validation, etc), need to be accompanied by busi-
ness logic that governs how the data is served.
ADO.NET Data Services enables a developer to
go beyond the pure data model where required
by defining service operations and/or intercep-
tors. Service operations allow the developer of
a data service to define a method on the serv-
er, which just like all other ADO.NET Data
Service resources, is identified by a URI. For ex-
ample, the URI /MyFavoriteBooks?category=spor
ts&$OrderBy=Title represents a call to the service
operation named MyFavoriteBooks that takes a
single category parameter. One of the interesting
features of service operations is that if the opera-
tion on the service returns a query representing
the resource to return as opposed to the resource
itself, then as shown in the example above,
the output of the service operation can be acted
on using standard ADO.NET Data Service op-
erations. This is shown in the query string of the
example above where the OrderBy operator is ap-
plied to the results of the operation.
Another mechanism known as interceptors is
provided to data services, which enables a devel-
oper to plug in custom validation logic into the
request/response processing pipeline of a data
service. Interceptors allow a developer to register
a method to be called when a particular CRUD
action occurs on a given type of resource being
exposed by ADO.NET Data Services. Such a
method then may alter the data or even terminate
the operation.
Looking Forward
Applications are moving to a model which divides
behavior and presentation from data access. This
trend has been driven by the ability of such archi-
tectures to deliver a rich, interactive client experi-
ence. ADO.NET Data Services provide a simple,
uniform way for applications to access their data.
Uniformity in this regard simplifies data access
and enables opportunities for generic UI controls
and libraries to be constructed based on a well-
known pattern.
www.code-magazine.com
ONLINE QUICK ID 0712092
Steve Lasker
SteveLas@Microsoft.com
Steve is a Program Manager at
Microsoft working to empower
developers to enable their
occasionally connected users.
For Visual Studio 2005, Steve
was on the Visual Basic team
working on many of the data
design-time features for building
client applications including the
Data Source Window, DataSet
Designer, Object Binding, and
enabling SQL Server Compact
for usage across all Windows
operating systems. For Visual
Studio and SQL Server 2008,
Steve focused on SQL Server
Compact 3.5 and, working with
an extremely talented group of
people across the company, led
the design for Sync Services 1.0.
Prior to joining Microsoft, Steve
was a technical architect with
an early Internet eCommerce
company before moving to a
consulting firm building Web,
client, and device applications
for corporate customers. With an
engineering background in the
remote broadcasting industry,
Steve lived the life of the mobile
workforce where he gained his
passion for the occasionally
connected user.
Steve speaks regularly at
various events, maintains a
blog with screencasts, Q&A and
architectures for occasionally
connected systems.
You can reach Steve via
e-mail or his blog:
blogs.msdn.com/SteveLasker
42 Caching with SQL Server Compact and the Microsoft Sync Framework for ADO.NET
Caching with SQL Server
Compact and the Microsoft
Sync Framework for ADO.NET
With Sync Services for ADO.NET, developers can easily optimize
their online experience by caching data locally within the easy-
to-deploy SQL Server Compact embedded database engine. In this
article, I’ll cover how Sync Services for ADO.NET was designed to fit the growing
developer needs for caching data locally in online-optimized, offline-enabled
applications.
T
here’s nothing new about wanting to cache Saving Leftover Pizza, or Shopping for
data, or enabling users to work offline. The Fresh Pasta and Sauce
problem has been that enabling this func-
tionally has been just too hard as the technologies
required have been complex to deploy, manage, When developers start thinking about caching, it’s
and haven’t really been focused on developers helpful to consider there are multiple models to
or productivity. Most applications are designed caching. I like to break them down into two dis-
to solve some business prob- tinct categories I call Passive
lem requiring the developer Fast Facts Caching and Active Caching.
to spend most of their time With Passive Caching, the cli-
focusing on the business com- Sync Services and SQL ent portion of the application
ponents of the application. All Server Compact aim to bring simply caches results to que-
the infrastructure bells and new levels of productivity, ries as they’re returned from
whistles are cool, but it’s typi- enabling developers to easily the server. There’s not a lot of
cally hard to justify the time add caching and offline complexity here and it can be
spent on something the end somewhat minimal impact to
features to their applications.
user doesn’t see. add to your existing applica-
With a focus on friction free, tion architecture, but as we’ll
Microsoft designed Sync Ser- non-admin deployment, see, the benefits are minimal
vices for ADO.NET to sim- developers no longer need as well. With Active Caching,
plify the integration of local to trade powerful, productive the application proactively
caching and synchronization features with an end-user seeks out to pre-fetch data
into your application. It focus- focused deployment model. that the client may require. In-
es on a goal of “Simple things stead of pre-fetching answers
should be simple while com- to the individual queries, the
plex things should be enabled.” Sync Services for application will retrieve the raw data required to
ADO.NET ships in two major upcoming products. answer the questions.
Visual Studio 2008 marks the first release of the
technology and ushers in unparalleled ease of use An alternative way to think about passive cach-
with an integrated Sync Designer leveraging SQL ing is the same way that you may shop for food.
Server Compact 3.5 as the local cache. The second You could order out for pizza, Chinese food, or
release of Sync Services for ADO.NET ships as a in some cities, you can have just about anything
part of SQL Server 2008 enabling further server delivered directly to your home. Now imagine
configuration simplicity with greater performance that instead of just saving the leftovers, you could
for sync-enabled applications. Sync Services 2.0 actually “copy” the delivery and save it in your
incorporates the Microsoft Sync Framework and refrigerator. The refrigerator represents your lo-
enables additional support for non-relational cal cache. The next time you’re hungry for Chi-
data stores in collaborative, peer-to-peer topolo- nese food or pizza, you’re all set; you simply open
gies. This article discusses the project goals and the cache. But what if you want spaghetti and
functionality enabled with Sync Services for meatballs? You could place another order for de-
ADO.NET. For more info on the Microsoft Sync livery or you could dissect the pizza w/meatballs
Framework, please read Moe Khosravy's article, and the Chinese noodles to create the spaghetti.
Introducing the Microsoft Sync Framework—Next This can get quite complex, not to mention
Generation Synchronization Framework. strange.
www.code-magazine.com
 End User
UI Designer Client Dev
Figure 1: Typical central management of all queries increasing the workload of the central servers, making the application rigid and
difficult to maintain.
Turning back to the data example, what if you
cached queries for Products.ProductName LIKE
‘a%’ or Products.Category = ‘Parts’. If you ask
those same questions again, you can leverage
the cache. But what if you ask for the top five
products? If you just look at the cache, you only
get the top five products that start with ‘A’ in the
‘Parts’ category. This starts to represent the limits
of Passive Caching.
With Active Caching, your application will have
brought down the raw materials to answer the
questions. Just as you may shop for the raw mate-
rial such meat, pasta, and spices, you can cache
the appropriate amount of raw data. Sometimes
raw data may be too detailed. Mom may not ap-
prove, but you might buy pre-prepped foods such
as canned tomato sauce, which you then spice up
to satisfy your unique taste. De-normalizing the
data as it’s brought down to the client may be your
tomato sauce. However, buying one of everything
at your food store isn’t practical either. Determin-
ing the right subset can help you cook up the re-
sults you need locally without having to replicate
the entire enterprise database. Microsoft Outlook
uses this same model with Exchange Server. The
appropriate filter of data, your inbox, calendar,
tasks etc., are brought down to your cache and
you then search, filter, and group the data locally,
offloading the details from Exchange Server.
Simplifying Data Access by Caching
Data Locally
In addition to the benefits of Active Caching,
another reason to cache data is it can actually
simplify your overall development. While many
End User
UI Designer Client Dev
Figure 2: Offloading detailed questions to the client, simplifying the server-side configuration and reducing the central workload.
www.code-magazine.com
Service Dev DBA
applications may start fairly simple, allowing the
client to directly query the database, many appli-
cations quickly move into the enterprise or need
to communicate over the public Internet. To limit
the server exposure, clients communicate via ser-
vices which are responsible for opening the con-
nections to the database behind the firewall. This
architecture quickly adds a lot of complexity to
your development. As seen in Figure 1, for each
question your users ask the server, the develop-
ment team would need some subset of the follow-
ing:
• Stored procedures and/or views for each
query on the database
• Configuration of permissions on each of the
objects
• Load testing and tuning of the queries as the
server has a multiplier of users asking inter-
secting questions
• Indexes supporting the WHERE clauses of
the sprocs & views
• Service contract wrapping each sproc/view
with potentially a custom object definition
for the result shape
• Client proxies for each service API
• And of course the supporting UI to ask and
answer each question
Making even the simplest change to the queries
or results can be painful to implement across the
entire stack requiring many roles in your devel-
opment team to get involved. Back to my pizza
example, what if you wanted extra cheese, or you
wanted everything but onions on the “everything”
pizza? Since you’re getting pre-cooked meals, you
must go back to the food store or possibly to the
supplier. Of course this isn’t very practical. To
solve the problem, many development teams au-
Service Dev DBA
Caching with SQL Server Compact and the Microsoft Sync Framework for ADO.NET 43
 tomate the creation of these various tiers, however
inevitably there are changes that require database
tuning for each interleaving query. Once the appli-
cation goes into production, it’s difficult to make
changes as the changes typically trickle across all
tiers, requiring a new version of the entire stack.
By leveraging this pre-fetch, Active Caching
model, you can now simplify your application
development. The DBA and service developer
now maintain a set of services that serve out raw
data, or essentially your version of your local food
store. Just as the food store has frozen pre-cooked
dinners, or even hot, ready-to-eat meals, the ser-
vices will likely offer a blend of sync services and
live services that may not be practical to “cook
up” locally. As you can see in Figure 2, with the
Sync Services and raw data local, the UI designer, client developer,
SQL Server Compact and end user can be quite productive, asking all
sorts of interesting questions without having to
Work on .NET worry about the impact to the central resources.
Framework 2.0 This can not only be productive for development,
To minimize the complexity for but extremely powerful to end users as they can
developers wanting to enable now ask complicated questions that could have
local caching, Sync Services caused havoc to the multi-user server. The local
for ADO.NET is built on .NET query may not always be the most efficient query,
FX 2.0. This means clients that but only the individual user is affected as there’s
have 2.0, 3.0 or 3.5 versions of no multiplier of users on a local cache. This re-
the framework can utilize Sync duces the workload to the production server as it
Services for ADO.NET. only needs to return changes to the raw data.
Micro Management or Delegation
Browsers offload In addition to the change management issues
above, how much duplication of effort should you
entry and minimal implement within your application? Though I am
validation to the writing this article for a special CoDe Focus issue
of CoDe Magazine, typically Rod Paddock (Edi-
client. Is bringing tor-in-Chief of CoDe) would create a theme for
reference each magazine and then find appropriate authors
for each article. They may enlist a writer like me
data to the client for a topic; they’d provide me a target page count,
just a continuum of a Word template, and CoDe Magazine publish-
ing guidelines. I then go offline and write the ar-
the PC evolution? ticle. When done, I submit it for editing. A few
days later I’ll get the article back with the edits.
(The number of edits typically relates to how well
I followed the guidelines.) I review the changes,
making sure the underlying meaning isn’t lost and
submit it again for “validation”. If all is accepted,
the article goes into layout and publishing. If not,
we repeat the same validation and editing loop.
This model scales as Rod employs the expertise
of great editors such as Erik Ruthruff for overall
content quality and delegates the technology ex-
pertise to the individual authors.
Now imagine this process in the “online” elec-
tronic model. As I write, I’d submit each para-
graph, heading, and figure to CoDe for validation.
Since I’m not as structured as some writers, I
tend to write out my thoughts, chunk them up,
and continually reorganize them. This can make
my style challenging to edit, so imagine if I sub-
mitted my content as I write it. This wouldn’t be
44 Caching with SQL Server Compact and the Microsoft Sync Framework for ADO.NET
very practical as Erik has other articles he needs
to edit. If Erik has to micro-manage my writing,
I’d create too much work for him, which begs
the question, why enlist me to write the article?
We’ve all been in this situation before thinking
it’s too much trouble to explain and you should
just do it yourself. But if you did that for every-
thing, could you get everything done you needed
to do? This delegation of responsibilities is how
most things scale.
Just Because You Can, Doesn’t Mean
You Should
Consider an order management system. For each
order, and each line item, how much of the system
needs to be involved? Does the server really need to
know the details for how each line item is added?
Or, similar to the article editing process, should the
user get the reference data they need to create the
order including the product catalog, categories, pric-
ing, and customer info. As they create orders, users
could save them locally until they are ready to be
submitted. The server then receives completed or-
ders and revalidates them for accuracy. It’s just soft-
ware, so why not have all the logic on the server?
As corporate data centers evolve, the amount of
central processing is causing them to grow at an
exponential pace. While data centers grow, client
computers are leaving lots of processing power
“on the table”. This is starting to look a lot like
the mainframe to PC revolution all over again.
The difference this time may be balancing both
the use of the client and the central servers. In the
mainframe days, the client sent each keystroke to
the server. Browsers offload entry and minimal
validation to the client. Is bringing reference data to
the client just a continuum of the PC evolution?
By offloading some of the validation to the client,
applications can better leverage each tier in the
architecture and empower users to make impor-
tant and quick decisions, while focusing the serv-
er on validating completed orders. Just because
companies can centrally run all the application
logic doesn’t mean it’s the best overall solution
for those data centers or the end users who get
frustrated at the latency and the requirement that
they be tethered to the network. Using this lay-
ered, empowered approach, you’ll likely find that
you can make many enhancements in one portion
of the application without affecting other tiers. If
you already have the reference data on the client,
you can likely cook up other enhancements by
simply versioning the client app and leaving the
central services alone.
How to Keep Your Cache Up to Date
Now that you may be convinced having a local
cache can help your clients make better decisions,
how do you keep the cache up to date? In my
www.code-magazine.com
food shopping comparison, you use up resources
and need to replenish them when inventories get
low. With reference data, you don’t “use up” data,
but rather need to get updates to the data. With
Active Caching, you bring down the raw materi-
als. But similar to food shopping, you don’t re-
purchase everything; you just get “the updates.”
There are a few models to tracking changes. The
server could keep track of all the clients or the
clients can track their own reference points.
As each client asks, “What changed for me?” the
server can send its change list. The problem with
this model is scalability. While this may work for
a small number of clients, as you get to the thou-
sands, this model tends to fall over as the server
must track changes for each client. Rather than
have the server track each client, what if the serv-
er simply tracked when changes were made? As
each client synchronizes with the server, the cli-
ent simply stores the reference point of the server,
so later on, it can ask for what’s changed since the
last reference point.
So imagine this scenario: On Monday Adam, your
user, installs the Order Entry application. Adam
downloads his product line from the overall cata-
log and the customers for his region. After a two-
day road trip, he reconnects to the corporate serv-
ers. Rather than asking the servers for what data it
has for Adam, the application asks what’s changed
since Monday for his product lines and his custom-
ers. In Sync Services for ADO.NET, the storing of
the server’s reference point on the client is known
as an anchor-based tracking system. This works re-
ally well for centrally managed data systems that
need to support a large number of clients.
Painting the Big Picture
Before jumping into the details for Sync Services
for ADO.NET, it may help to understand a num-
ber of key design goals and priorities that our
team had in mind.
• Simple things should be simple while com-
plex things should be enabled.
• Work within friction free, non-admin, Web
deployment models.
• Be developer-oriented and leverage their
domain-specific skills for each type of data
source.
• Deliver an end-to-end experience built on a
layered component model.
• Enable independent server and client con-
figuration empowering the client application
to determine the data it needs, while freeing
the server to version independently offering
additional services.
• Enable rich eventing on all tiers for conflict
detection and business rule re-validation for
domain-specific data sources.
• Enable easy movement/refactoring between
two-tier, n-tier, and service-oriented archi-
tectures.
www.code-magazine.com
• Be protocol and transport agnostic leverag-
ing the various transports, protocols, and se-
curity models developed by other smart folks
at Microsoft (Web services, WCF, …).
• Enable endless scalability of the number of
clients synchronizing with the server.
• Provide a common sync model for all types
of data, not just relational data, but files and
custom data sources.
• Ensure data can be exchanged in a variety of
connectivity patterns including centralized
and collaborative, peer-to-peer topologies.
I’ll expand on these points a bit below, providing
a bit more context. As we started to scope the ef-
fort of accomplishing these goals, we considered
the common data sources that developers re-
quired. The most obvious were files and relational
data. XML data sort of fell into a combination of
files and unstructured relational storage. In many
cases developers were choosing XML storage
as it was too hard to deploy anything else. For
Sync Services for ADO.NET version 1.0 (hereaf-
ter, V1) we decided to focus on relational data
in a centralized topology, but carefully designed
the system to support other data sources in the
future. Be sure to read Moe Khosravy’s article,
Introducing the Microsoft Sync Framework—Next
Generation Synchronization Framework, covering
the broader goals for the sync platform including
peer-to-peer topologies.
As we scoped V1 to relational data, we elaborated
on the domain-specific data sources:
• Leverage a developer’s ADO.NET experi-
ence while being DBA-friendly for server
configuration.
• Deliver a good, better, best programming
model enabling any ADO.NET server da-
tabase, provide a designer for SQL Server,
and use SQL Server 2008 to deliver the best
overall performance and simplest to config-
ure.
The Power of SQL Server in a
Compact Footprint
While many IT-supported applications were suc- One Size Doesn’t
cessful deploying SQL Server Express, the ma- Fit All
jority of applications didn’t have direct IT sup-
Sync Services is just one
port. Many of these applications were persisting
component of a larger
their data as XML using serialized DataSets as
architecture. On my blog I
they couldn’t use SQL Server Express due to its
describe how “Logical Queuing”
deployment requirements. With our V1 scop-
and “Notifications to Pull” are
ing to relational data and our design goals pri-
other important architectures
oritizing non-admin, Web deployment models, we
for using the right tool for the
knew we had to enable a local store that could
right job.
deliver the power of SQL Server in a compact
footprint.
Since 2001, SQL Server has had an embedded
database engine available to the Windows Mobile
platform known as SQL Server CE. Targeted at
Caching with SQL Server Compact and the Microsoft Sync Framework for ADO.NET 45
 Figure 3: High-level architecture for Sync Services for ADO.NET.
devices, Microsoft designed it for constrained en-
vironments, but it still had the power to handle
relatively large amounts of data in a compact
footprint. With a footprint under 2 MB, it became
a popular choice for applications that need the
power of SQL Server in a compact footprint. Sev-
eral applications in Windows Vista, Media Center
PC, and MSN Client all embedded this engine
within their applications. You likely didn’t even
know you had SQL Server Compact running, and
that’s one of its key advantages. It’s simply em-
bedded cleanly within each application.
In 2005, SQL Server CE became SQL Server Mo-
bile to recognize its expanded usage on the Mo-
bile platform of the Tablet PC. With Visual Studio
2005 focusing on client development, we knew
we couldn’t wait for Visual Studio 2008 to enable
developers looking to cache data for optimized
online or those building offline applications. We
knew we’d take a while to complete Sync Services
and SQL Server Mobile simply needed a licensing
change. With Visual Studio 2005 Service Pack 1
we released SQL Server Compact 3.1 as the suc-
cessor to SQL Server Mobile and SQL Server CE
to address all our current Windows desktop op-
erating systems.
Some of the key benefits of SQL Server Compact
include the following:
• ~1MB for the storage engine and query pro-
cessor.
• Consistent engine and data format for all
Windows operating systems from the phone
to the desktop.
• Runs in-process (i.e., embedded).
• Deployed as DLLs eliminating the need for a
Windows service.
• Supports central installation with traditional
MSI (requires administrative rights).
• Supports private deployment of the DLLs
within the application directory (enabling
non-admin deployment).
• Full side-by-side support for multiple ver-
sions enabling different applications to em-
bed SQL Server Compact without fear that
other applications using newer versions may
destabilize their application.
46 Caching with SQL Server Compact and the Microsoft Sync Framework for ADO.NET
• Rich subset of query programming of SQL
Server.
• Rich subset of data types of SQL Server.
• Robust data programmability through ADO.
NET with native programmability through
OLE DB.
• ISAM-like APIs for bypassing the query pro-
cessor, working directly against the data.
• Updatable, scrollable cursors eliminating the
need for duplicate, in-memory copies.
• Support for up to 4GB of storage, with in-
creasing storage in future versions.
• Rich sync technologies including Merge Rep-
lication, RDA, and Sync Services for ADO.
NET.
• Full support for SQL reporting controls.
• Single-file code-free file format enabling
a safe document-centric approach to data
storage.
• Support for custom extensions enabling dou-
ble-click association with your application.
• Support for network storage, such as net-
work redirected documents and settings.
• Concurrent application support from mul-
tiple processes or threads enabling sharing
of data on a single user’s machine.
• Best of all, it’s free to deploy, license, distrib-
ute, and embed within your applications.
With SQL Server Compact 3.1 shipping with Vi-
sual Studio 2005 SP1, developers can fill the need
between XML storage and SQL Server Express.
With Visual Studio 2008, SQL Server Compact
3.5 ships as the default local database with in-
tegrated designer support including the support
of Sync Services for ADO.NET and LINQ to
SQL. With the release of the ADO.NET Entity
Framework, SQL Server Compact will be used to
persist ADO.NET entities locally within your ap-
plication.
Future versions of SQL Server Compact will con-
tinue to take advantage of the unique embedded
nature to provide an integrated experience within
your application programming model. SQL Serv-
er Express will certainly continue for develop-
ers requiring the same programming model with
our data service SKUs, including Workgroup,
Standard and Enterprise. SQL Server Express
www.code-magazine.com
Figure 4: SyncAdapters follow a familiar pattern to ADO.NET DataAdapters.
and SQL Server Compact are not meant to com-
pete, but rather provide overlapping technolo-
gies enabling the unique needs of custom ap-
plications. They are best when used together
with SQL Server Express as the free entry point
to Microsoft’s data service platform and SQL
Server Compact as the local cache utilizing Sync
Services to synchronize the shared data and local
cache.
Simple to Complex
As developers are continually challenged with
new emerging technologies, it becomes diffi-
cult to invest a lot of time learning a technology
just to get the simple things done. To enable the
simple scenarios with the complexity as needed,
Sync Services took an incremental approach to
its design. This includes simple snapshot caching,
enabling developers to add incremental changes,
or ultimately the ability to send changes from the
local store back up to the server.
Additionally, many applications may start off as
two-tier applications and over time they expand
requiring n-tier architectures. Rather than assume
you must re-architect the application to make
this move, Sync Services was designed to easily
move from two-tier to n-tier architectures, includ-
ing service-oriented architectures (SOA). Figure
3 shows how the components are broken into
groupings. You can see server components, client
components, and a SyncAgent which orchestrates
the overall synchronization process.
Building On Your ADO.NET Skills
One of the significant improvements between
classic ADO and ADO.NET was the ability to
drill into and configure the commands which
were executed against the server. With ADO.
NET, components such as the DataAdapter re-
trieve data from your database populating a dis-
connected DataSet sending the data across the
wire. As inserts, updates and deletes are made,
the DataAdapter can shred the changes based
on the RowVersion in the DataSet, executing in-
www.code-magazine.com
sert, update and delete commands that may have
been configured for you, or customized to suit the
needs of your specific application using custom
T-SQL, sprocs, views or functions. However the
DataAdapter is designed for a one-time retrieval
of data. It doesn’t have the infrastructure to re-
trieve incremental changes. As seen in Figure 4,
Sync Services uses a SyncAdapter, which is based
on the DataAdapter programming model. The Se-
lectCommand is replaced with three new com-
mands for retrieving incremental changes.
Configuring each command and its associated
parameters can be quite tedious, especially when
you just need the basics. Similar to the SqlCom-
mandBuilder, Sync Services adds a SqlSync-
AdapterBuilder which allows you to specify a
few configuration options to create a fully-con-
figured SyncAdapter. While Sync Services can
work with any ADO.NET provider, the SqlSync-
AdapterBuilder is specific for Microsoft SQL
Server enabling the “Better on SQL Server” de-
sign goal.
Server Configuration
Figure 3 shows the green portions representing
the server-side configuration. As most database
servers are shared servers for several applications,
each application may only require a subset of the
data, or the data may need to be reshaped or de-
normalized before it travels down to the client.
For this reason, Sync Services exposes granular
configuration with ADO.NET DbCommands en-
abling any ADO.NET data provider. For each log-
ical table you wish to synchronize, a SyncAdapter
is configured.
One of the challenges with DataAdapters is the
ability to update parent-child relationships. With
common referential integrity patterns, child de-
letes must first be executed before the parents can
be deleted. Likewise, parents must first be created
before children can be inserted. The DbServer-
SyncProvider contains a collection of Sync-
Adapters that enables the hierarchal execution of
commands based on the order of SyncAdapters in
the collection.
Caching with SQL Server Compact and the Microsoft Sync Framework for ADO.NET 47
 Client Configuration
The client has two main components. The client
provider, in this case the relational store of SQL
Server Compact, and the SyncAgent which orches-
trates the synchronization process. You could sort
of think of the SyncAgent as the thing that does the
food shopping for you once you’ve given it the shop-
ping list, and location of the food store. While server
databases tend to be shared resources, client data-
bases are typically specific to the application. With
this design simplicity there’s no need for configura-
tion of the individual commands for each local ta-
ble. However as client applications get installed on
potentially thousands of clients, there are a number
of other deployment options. Other configuration
options such as what subset of server data each cli-
ent actually requires and the ability to group several
Resources for SQL tables within a transactional sync operation can be
Server Compact and achieved with a SyncGroup configured through the
Sync Services for Configuration object of the SyncAgent. Once the
ADO.NET SyncAgent is configured with a RemoteProvider,
LocalProvider and the SyncTables, you can simply
• www.SyncGuru.com. Rafik call the Synchronize method.
Robeal, our developer guru for
Sync Services, provides many In two-tier architectures the RemoteProvider is the
examples and insight into DbServerSyncProvider. Later in this article, in the
Sync services for ADO.NET. section called “Going N-tier with WCF,” I’ll dis-
• blogs.MSDN.com/ cuss how you move from a two-tier architecture to
SteveLasker. Includes something with multiple tiers. To enable different
many screencasts, demos, data sources ranging from relational databases to
answers to common files, as well as custom objects, Sync Services fol-
questions and blogicles lows a standard provider model. Sync Services 1.0
for occasionally connected includes a relational provider for SQL Server Com-
architectures. pact, or the SqlCeClientSyncProvider.
• www.Microsoft.com/SQL/
Compact. Product info and Configuring the SqlCeClientSyncProvider only re-
downloads for SQL Server quires a connection string for the local database
Compact. with the default option to automatically create the
local database and schema. Similar to the auto cre-
• blogs.MSDN.com/
ation of the database, the SyncTable configuration
SQLServerCompact. The
enables options for creating the tables and whether
SQL Server Compact team
local changes to the table should be synched back
blog.
to the server. Several events are available within the
• www.Microsoft.com/ client and server providers enabling conflict detec-
SQL/Editions/Compact/ tion and business rule validation.
SSCEComparison.mspx.
Choosing between SQL
Server Express and SQL Designer Productivity
Server Compact.
• MSDN.Microsoft.com/Sync. To meet our developer productivity design goal,
Info for the overall sync Visual Studio 2008 adds a Sync Designer to sim-
platform. plify configuration of Sync Services for ADO.NET.
Listing 1: Extending the Sync Designer, enabling bidirectional synchronization and N’ Tier sync with
WCF
Partial Public Class NorthwindCacheSyncAgent
Private Sub OnInitialized()
' Enable Customers to sync with the server
Me.Customers.SyncDirection = _
SyncDirection.Bidirectional
' Glueing the SyncAgent.Remote provider to
' a WCF service proxy
Me.RemoteProvider=New ServerSyncProviderProxy(
New SyncServRef.NWCacheSyncContractClient())
48 Caching with SQL Server Compact and the Microsoft Sync Framework for ADO.NET
Similar to the design goals for Sync Services, the
Sync Designer starts simple and lets you incre-
mentally add complex requirements as needed.
The Sync Designer focuses on the following:
• Configure SQL Server databases for change
tracking, including tracking columns, track-
ing tables (known as Tombstone tables) and
triggers to maintain the tracking columns.
• SQL scripts generation for later reuse and
editing when moving from development to
production.
• Generation of typed classes for instancing
the sync components, similar to the typed
Dataset and TableAdapter experiences.
• Auto creation of the local database and
schema, including primary keys.
• Separation of client and server components
to enable n-tier scenarios.
• Creation of WCF service contracts for n-tier
enabled applications.
The Visual Studio 2008 Sync Designer is focused
around read-only, reference data scenarios, so
you won’t see any configuration for uploading
changes. However, the designer does generate
the upload commands—they’re just not enabled.
Similar to typed DataSets, you can extend the
designer-generated classes through partial types
enabling the upload commands with a single line
of code per table. By selecting “View Code” from
the Solution Explorer context menu on the .sync
file you can reconfigure the Customers table to
enable bidirectional synchronization as seen in
Listing 1.
Other features of Sync Services that aren’t en-
abled through the designer but can be extended
through partial classes are:
• Filtering of columns or rows
• Foreign keys, constraints, and defaults
• Batching of changes
To get a feel for the designer simply add a Lo-
cal Database Cache item to any non-Web project
type. The Local Database Cache item aggregates
the configuration of Sync Services for ADO.NET
and SQL Server Compact 3.5 for the local cache.
The designer will enable connections to SQL
Server databases and will automatically create
a SQL Server Compact 3.5 database. With the
server database selected, you can add tables to be
cached based on the default schema of the user id
in the SQL Server connection string. By checking
one of the tables the designer will configure the
server tracking changes using “smart defaults”.
If you can’t make server-side changes to your data-
base, and the data is small enough to just retrieve
the entire result each time you sync, you can change
the “Data to download” combo box to “Entire table
each time,” also known as snapshot. When per-
forming snapshot sync, the tracking combo boxes
become disabled as the SQL Server configura-
tion options aren’t required. Assuming you want
www.code-magazine.com

incremental changes, you can then choose which
columns will be used to track the changes. The
designer will default to adding an additional Date-
Time column for LastEditDate and CreationDate. If
your table already has TimeStamp columns for last
updates, you can select the existing column. Sync
Services stores a common anchor value for last edit
and creation columns, so you’ll need to either use
DateTime or TimeStamp on both the last edit and
creation tracking columns. Since you can only have
one TimeStamp column per table, you’ll notice that
if you choose TimeStamp for the LastEdit compari-
son, the creation column will default to a BigInt.
Timestamps are really just big integers serialized in
binary form.
Tracking Deletes
Tracking deletes is an interesting problem. If you
synchronize with the server on Monday, and come
back on Wednesday asking what’s changed, how
does the server know what’s deleted unless it keeps
a historical record? There are a couple of different
approaches to tracking deletes. Prior to the Sar-
banes-Oxley (SOX) compliance days, it was typical
to just delete aged data. However, between SOX
and increased disk storage, applications are keep-
ing the deleted data around. While it’s important to
keep the deleted data on the server, it’s typically not
necessary to clog up your clients. When each client
synchronizes, Sync Services simply requests the list
of primary keys that should be deleted locally.
Tombstones
A standard model for tracking deletes is to create
a separate table that contains the primary keys for
deleted rows. Just as cemeteries use tombstones to
leave an indicator of what once was, tombstone
tables have become the standard in many sync-en-
abled systems. Another interesting analogy is how
tombstones, if not managed, could eventually cover
the earth leaving no room for the “active partici-
pants.” Luckily data is a lot less emotionally sen-
sitive so a standard model is to purge tombstone
records after a period of mourning. How long you
keep your tombstones is related to a balance of disk
space, database sizes, and how long you expect your
users to work offline. The pre-SQL Server 2008
www.code-magazine.com
Figure 5: Sync Services in a layered service-oriented architecture.
model could utilize a scheduled task to run daily to
purge tombstones that are older than a configured
number of days. Furthering our goal to make Sync
Services run “Best on SQL Server,” SQL Server
2008 adds a feature known as SQL Server Change
Tracking that dramatically simplifies this server
configuration and reduces the overhead by track-
ing changes deep within the SQL Server engine.
Change tracking incorporates a change retention
policy that automatically purges historical changes
based on the per-table configured value.
Since Visual Studio 2008 ships prior to SQL
Server 2008, and developers may need to target
pre-SQL Server 2008 servers, the Sync Designer
will default to creating a Tombstone table and the
associated triggers to maintain the tracking infor-
mation. As with the rest of the SyncAdapter con-
figuration, if the designer-generated commands
don’t fit your needs, you can easily customize these
commands using sprocs, views, and functions.
Sync Services simply needs a DbCommand with
results for the specific command, in this case the
list of primary keys to delete. As you move to SQL
Server 2008, your configuration becomes easier,
and your sync-enabled database will simply per-
form better.
Setting the Hierarchical Order
Once you’ve added the cached tables, the Sync
Designer will configure your server and save the
creation and undo scripts for inspection and re-ex-
ecution when moving from development to produc-
tion. Back in the Configure Data Synchronization
dialog box, you’ll see your list of tables. It may not
be that important for reference data, but if you in-
tend to send your changes back up to the server,
and you want to make sure parent records, (such
as the OrderHeader table) are inserted before Or-
derItems and OrderItems are deleted before Order-
Header rows are deleted, you can use the arrows to
shuffle the list of tables sorting the parents above
their children. The designer sets the order of Sync-
Adapters within the DbServerSyncProvider. Click-
ing OK to finish the Sync Designer configuration
will configure the Sync Services runtime, generate
the classes for use within your application, and ex-
ecute the Synchronization method to automatically
generate the SQL Server Compact database with
Caching with SQL Server Compact and the Microsoft Sync Framework for ADO.NET 49
CODE COMPILERS
the schema and data you’ve selected. As the Sync
Designer completes it adds the newly created SQL
DATA DEVELOPMENT Server Compact database to your project which
triggers Visual Studio to automatically prompt to
October 2007 Volume 4 Issue 3 create a new typed DataSet for your newly added
SQL Server Compact data source.
Group Publisher
Markus Egger
Associate Publisher
Rick Strahl
Sync Services, SQL Server Compact
Managing Editor and LINQ
Ellen Whitney
Content Editors
H. Kevin Fansler Erik Ruthruff Similar to building houses, it’s difficult to build
the second floor when the basement is still being
Writers In This Issue
Brian Beckman José Blakeley completed. Because of the parallel development
Anthony Carrabino Andrew Conrad of LINQ, Sync Services and the Microsoft Sync
Gert Drapers Elisa Flasko Framework, Sync Services for ADO.NET 1.0 uses
Mike Flasko Moe Khosravy
Stan Kitsis Christian Kleinerman DataSets and DbCommands for its programming
Steve Lasker Chris Lee APIs. Future releases of Sync Services will be ex-
Erik Meijer Shyam Pather tended to work with the new LINQ programming
Michael Pizzo David Schach
Vaughn Washington Jonathan Wells models.
Technical Reviewers
Elisa Flasko The Sync Services, Active Caching approach syn-
Ellen Whitney chronizes data between two or more data sources.
Art & Layout The orange box in Figure 3 shows how interacting
King Laurin GmbH with the data in SQL Server Compact is complete-
info@raffeiner.bz.it
ly up to the application developer. SQL Server
Production Compact continues to support the typed DataSets
Franz Wimmer
King Laurin GmbH and TableAdapter programming model from Visu-
39057 St. Michael/ Eppan, Italy al Studio 2005 as well as the updateable, scrollable
Printing cursor using the SqlCeResultSet. With the new
Fry Communications, Inc. LINQ programming model, developers may use
800 West Church Rd. LINQ to SQL or LINQ to Entities. While Visual
Mechanicsburg, PA 17055
Studio may auto-prompt to create typed DataSets,
Advertising Sales
Vice President, Sales and Marketing
this is simply based on providing a consistent expe-
Tom Buckley rience with what developers may already be used
832-717-4445 ext 34 to with Visual Studio 2005. Developers wishing to
tbuckley@code-magazine.com use LINQ to SQL or LINQ to Entities can simply
cancel this dialog and use the appropriate tools
Sales Managers to generate classes over tables within their SQL
Erna Egger Server Compact database. For more information
+43 (664) 151 0861
erna@code-magazine.com on LINQ to SQL and LINQ to Entities, see the
Tammy Ferguson article, LINQ to Relational Data: Who’s Who? by
832-717-4445 ext 26 Elisa Flasko in this issue of CoDe.
tammy@code-magazine.com

Circulation & Distribution

General Circulation: EPS Software Corp. Going N-tier with WCF
Newsstand: Ingram Periodicals, Inc.
Media Solutions One of the challenges developers faced in Visual
Worldwide Media
Studio 2005 was how to easily refactor Typed Da-
Subscriptions
Subscriptions Manager
taSets from two-tier to n-tier architectures. ADO.
Cleo Gaither NET factored DataSets and DataAdapters as
832-717-4445 ext 10 separate components allowing the DataAdapters
subscriptions@code-magazine.com and their associated commands and connections
US subscriptions are US $29.99 for one year. Subscriptions outside the US to be hosted on the server. DataSets could be
are US $44.99. Payments should be made in US dollars drawn on a US bank. shared on the client and server reusing common
American Express, MasterCard, Visa, and Discover credit cards accepted. business rule validation and schema. However, in
Bill me option is available only for US subscriptions. Back issues are available.
For subscription information, email subscriptions@code-magazine.comor contact Visual Studio 2005, the Typed DataSet Designer
customer service at 832-717-4445 ext 10. didn’t easily enable this scenario. Late in the Vi-
sual Studio 2005 cycle we prototyped a number
Subscribe online at of different solutions which eventually lead to the
www.code-magazine.com following design.
CoDe Component Developer Magazine
EPS Software Corporation / Publishing Division In Visual Studio 2008, both typed DataSets and
6605 Cypresswood Drive, Ste 300, the Sync Designer now support full separation of
Spring, Texas 77379
Phone: 832-717-4445Fax: 832-717-4460 the client- and server-generated code. Within the

50 Caching with SQL Server Compact and the Microsoft Sync Framework for ADO.NET www.code-magazine.com
Sync Designer, clicking the advanced chevron will
expose the n-tier configuration options. Here you
can target the client- and server-generated classes
to different existing projects within your solution.
If you close the dialog and add a new “WCF Ser-
vice Library” project to your solution you can use
the designer to generate your server classes into a
WCF service project.
The Sync Designer will not only generate the
server provider classes into the WCF project, it
will also generate a WCF service contract wrap-
ping the configured server sync provider and its
associated interface. At the top of the gener-
ated file, a few commented snippets are includ-
ed for configuring your WCF service. With the
classes separated and the WCF service fully config-
ured, you simply need to glue them back together
by generating a service and its associated proxy.
Multiple Inheritance and Organic
Onions
One of our design goals for Sync Services was to
minimize dependencies on external components
and leverage other teams building transports, ser-
vices, and security models. Developers may need
to work over Web services, use the more powerful
features of WCF, work over REST, use their own
custom transports, or serialize data in a custom-
ized format. As you can see in Figure 5, we de-
signed Sync Services to work over stateless service
models. All that’s needed is a matching service and
proxy. With Sync Services 1.0 using DataSets, de-
velopers can leverage the out of the box serializa-
tion of DataSets, or they can use LINQ to trans-
form DataSets to their own custom payload. Keep-
ing with the food metaphor, they may convert the
DataSets to serialized jellybeans. On the client, the
developer simply converts the jellybeans back to a
DataSet, handing it to the SyncAgent, and they are
good to go.
One of the design challenges we faced with en-
abling transport-agnostic programming was the
base class requirements of many components. Web
services and WCF proxies require their own base
classes. The RemoteProvider on the SyncAgent re-
quires its own base class of a SyncProvider. To get
around the multiple inheritance issues we added a
wrapper class to Sync Services enabling delegation
to the appropriate base class. Since the wrapper
class was generic, but wrapped other proxy classes,
we initially called it the OrganicOnion class. As
we moved into production this was changed to
the less creative, but more professionally named
ServerSyncProviderProxy class. To glue the Syn-
cAgent back together with the configured Server-
SyncProvider in the WCF project, developers can
add the code similar to Listing 1 in the partial class
of the Sync Designer.
With this configuration, your application can eas-
ily work over two-tier architectures, n-tier or ser-
www.code-magazine.com
vice-oriented architectures with minimal impact
to your code.
A Quick Recap
In this article I’ve covered why caching data isn’t
just about enabling offline functionality and is a
common practice in other scenarios such as the
food supply chain. Caching can ease your develop-
ment cycle, enabling decoupled tiers so you can
independently update components of your appli-
cations, and of course, you’re that much closer to
enabling full offline scenarios. By comparing the
caching scenarios to other real world issues, and
walking through the configuration of Sync Ser-
vices for ADO.NET using Visual Studio 2008, I’ve
hopefully demonstrated the need and productivity
gains for caching reference data, tracking deletes,
sending updates and simply moving from two tiers
to n-tier over WCF. As your applications scale to
more complex requirements, Sync Services can be
reconfigured to your unique needs. As you need to
enable different data sources or support collabora-
tive topologies, your investments in Sync Services
can be expanded with the Microsoft Sync Frame-
work. As your companies roll out SQL Server
2008, features like SQL Server Change Tracking
and Sync Services for ADO.NET will deliver the
best developer experience with the greatest perfor-
mance for sync-enabled applications.
It’s Not a Matter of If, but When.
Are You Ready?
You can always tell the difference between expe-
rienced and novice motorcycle riders by the pro-
tective gear they wear. As the experienced will tell
you, it’s not a matter of if you fall; it’s a matter of
when. By building systems that can leverage a local
cache, your users, and your business will be pro-
tected from the inevitable. With SQL Server Com-
pact, Sync Services for ADO.NET, and the Micro-
soft Sync Framework you have more productive
and powerful tools enabling applications to easily
integrate redundancy into your applications. Done
right you can reduce the workload to your cen-
tral services empowering your users to work any-
time, anywhere. When they’re connected, things
may work better, but when they’re not, they can
continue to be productive producing revenue for
your company. Although networks and wireless
continues to be available in more locations with
faster bandwidth, they’re not going to be available
everywhere, and if you assume your users won’t
have a problem, well, I guess you haven’t fully ex-
perienced motorcycle riding.
Steve Lasker
Caching with SQL Server Compact and the Microsoft Sync Framework for ADO.NET 51
ONLINE QUICK ID 0712102

Introducing the Microsoft

Moe Khosravy
Moe is the Lead Program
Manager on the Microsoft Sync
Framework and has been with
the project since incubation. You
can find him roaming the halls of
campus preaching content flow
and applying quotes from
The Simpsons to everyday life.
He was previously an architect in
the Advanced Technologies group
Sync Framework:
Next Generation
Synchronization Framework
The Microsoft® Sync Framework is the new framework and
runtime for adding synchronization, roaming, and offline
capabilities to applications. It supports peer-to-peer scenarios,
works with devices and services, and is agnostic of data types,
stores, and protocols. In this article, I’ll cover the high-level vision for the
platform as well as the enabled scenarios made possible by the framework for
developers, ISVs, and OEMs.

S
for Vital Images Inc, working on everal years ago, Microsoft assembled archi- audio, video, settings, and files/folders across any
advanced medical visualization tects and researchers from across the com- number of PCs, services, and devices—all directly
and analysis software solutions. pany to understand the limitations prevent- via peer-to-peer sync or through any number of
ing the seamless flow of data intermediaries such as PCs or
across applications, devices, Fast Facts services.
and services to allow consum-
ers access to their data wher- For years developers At the same time, more and
ever they need it. Throughout have been looking more applications were look-
the investigations, several ob- for an easy way to ensure ing to enable an Outlook-like
servations quickly emerged: cached mode of operation
that their applications may
in which an application oper-
• Lack of generic synchroni- be taken offline ates offline and periodically
zation technology lead to and synchronized with other synchronizes with the server.
the creation of numerous devices such as cellphones, Clearly, the common frame-
single-purpose solutions. PDAs, and other PCs. work would need to excel at
• Solutions tended to re- scenarios such as:
invent the wheel; often
falling into the same pitfalls explored by oth- • Caching and line of business (LOB) appli-
ers. cations that can take data from relational
• Solutions could not be bridged as they made databases and disparate back-end services
contradictory assumptions about the syn- offline.
chronization methodology (i.e., relied too • Taking Web services and Web applications of-
heavily on specific topologies, data types, fline, providing richer experiences or simply
stores or business logic). network resiliency.

It became clear that the only way to make prog-

ress towards enabling seamless “content flow”
was to build a common framework with the fol-
lowing attributes:
• Powerful. Able to solve the numerous hard
problems related to interoperability.
• Flexible. Able to be used in all endpoints in
the sync-enabled ecosystem.
• Simple. Easy to use in any architecture to
compose new systems.
Only then could the framework support roaming
and sharing scenarios for content such as PIM,
Project Goals
The Microsoft Sync Framework was created with
the following goals in mind:
1. Provide a common runtime for developers
building synchronization solutions, allowing
them to reuse a reliable and optimized code
base capable of addressing the numerous subtle
issues that developers run into when building
caching, offline, sharing or roaming scenarios.
2. Facilitate content flow across solutions, even
if they utilize different protocols and stores

52 Introducing the Microsoft Sync Framework: Next Generation Synchronization Framework www.code-magazine.com
 by standardizing on the synchronization
metadata.
3. Simplify the actual development of sync
solutions by providing domain-specific and
end-to-end components for common sce-
narios such as syncing relational databases,
file systems, lists, devices, PIM, music, video,
etc…
Microsoft designed
the Microsoft Sync Framework
as a componentized and
layered architecture to
allow developers to pick
and choose only what they
require to enable their
scenarios and ensure
that assets built on the
framework could participate
in the content flow ecosystem
we’re working to grow.
The Microsoft Sync Framework
Approach
A key tenet of the Microsoft Sync Framework is
to bring existing applications and devices into the
sync ecosystem by leveraging as much as possible
from their existing implementations. This meant
supporting synchronization between data stores
ranging from large distributed Web stores backed
by relational databases down to the file systems
found on removable USB drives.
The framework would also have to support nu-
merous data types and schemas (such as relation-
al data or files), as well as various sync topologies
(such as peer-to-peer meshes or hub-and-spoke).
These requirements emerged prominently when-
ever multiple sync solutions were bridged in a
heterogeneous mesh.
For these reasons the heart of the Microsoft Sync
Framework consists of the common synchroni-
zation metadata model. By agreeing on only the
metadata without forcing agreement on protocols
or storage, the Microsoft Sync Framework lays
the foundation for content-flow as well as a ge-
neric solution for synchronization, roaming, and
sharing.
The metadata at the core of Microsoft Sync
Framework is highly efficient and compact, yet
it provides full support for correct multi-master
www.code-magazine.com
synchronization (that is, allowing concurrent up- Data Programmability Team
dates on multiple endpoints). The Microsoft Sync
Framework runtime implements the algorithms
required to work with the metadata, allowing
endpoints to easily participate in the ecosystem.
To make it easier and more productive for de-
velopers building on Microsoft technologies to
leverage these capabilities, Microsoft has start-
ed integrating the enabling technology into our
next-generation flagships and platforms. For
more information on Sync Services for ADO.
NET and for an example of a Microsoft Sync
Framework-enabled solution for rich synchro-
nization and caching of relational databases,
please see Steve Lasker’s article, Caching with
Sam Druker
SQL Server Compact and the Microsoft Sync
“I think we’re moving to
Framework for ADO.NET in this issue of CoDe
Focus. the point that the cloud is
becoming the next great
platform for people to build
Components of the Microsoft Sync applications on.”
Framework Sam currently leads the Data
Programmability team at
Microsoft, in the SQL Server
The synchronization framework team designed
division; Sam is responsible for
the Microsoft Sync Framework as a componen-
tized and layered architecture to allow develop- building out the data platform
development strategy and
ers to pick and choose only what they require to
engineering, shipping XML and
enable their scenarios and ensure that assets built
on the framework could participate in the con- Data platform technology in the
tent flow ecosystem we’re working to grow. .NET Framework and throughout
the SQL Server product line. Sam
The Microsoft Sync Framework’s designers di- is also responsible for core XML
vided these components into the following logical and Data access components in
layers, which are available to both managed and the Windows platform, Internet
native code developers. (Note: Device builds for Explorer, and Office.
ARM, SH, x86, MIPS as well as Apple Mac sup-
port are in development and will be provided in a Sam has been in the software
future CTP on MSDN). industry for more than 18 years.
Prior to joining Microsoft, Sam
• Core Sync Runtime. Infrastructure that in- was Vice President Engineering
cludes the algorithms to utilize the metadata as for LinkExchange, an early
well as components to drive roaming, sharing, provider of Internet marketing
offline, and sync on behalf of applications. services acquired by Microsoft
• Sync Provider Framework. Components in 1998 and Vice President
designed to make it easier to expose data to Engineering for Cygnus Solutions,
the platform. This is effectively the plug-in a provider of compiler and
model to the Microsoft Sync Framework by development tools to the open
which developers can either configure exist- source community, since
ing Sync Providers or write their own. acquired by Red Hat Software.
• Domain-specific components. Infrastruc- In the early nineties, he was a
ture to facilitate the rapid development of Principal Software Engineer at
end-to-end solutions involving specific stores
Cognex Corporation, a leading
such as SQL Server, SQL Server Compact,
manufacturer of machine vision
FAT, NTFS, etc.
systems for factory automation
and robotics. Sam started his
Core Sync Runtime software engineering career as
Vice President Engineering for
The core runtime contains the metadata services Zortech, which shipped the first
used by all of the Microsoft Sync Framework cli- native code compiler for C++.
ents. Some of the features include: Sam is a graduate of MIT's VI-3
EECS program, a former offensive
• Multi-master metadata representation and lineman for the Engineers, and
management, including conflict detection, now lives in Seattle, WA.
Introducing the Microsoft Sync Framework: Next Generation Synchronization Framework 53
 custom and preset conflict resolution han-
dling, change enumeration assistance, etc.
• Conflict preservation management and rep-
resentation of conflict resolution.
• Support for change units (tracking changes
at a property level) and consistency units
(dealing with logical groupings of objects).
• Handling of filtering changes and filter sync
(both for items and for parts of items).
• Recovery from a multitude of failure sce-
narios, such as tombstone cleanup, interrup-
tions, network failures, etc.
• Synchronization session management, such
as cancellation, progress reporting, etc.
The Microsoft Sync Framework provides an
out-of-the-box Synchronization Agent that can
synchronize endpoints when requested. The
endpoints are abstracted using the notion of a
Sync Provider, which is responsible for expos-
ing the capabilities of a given endpoint, store or
protocol. Applications use Sync Providers for
the stores they need to synchronize and develop-
ers use the Microsoft Sync Framework to create
custom Sync Providers for virtually any type of
data.
Core to the framework is support for Simple
Sharing Extensions (SSE). The Microsoft Sync
Framework natively supports endpoints that wish
to interoperate using SSE extensions for RSS
and ATOM feeds. Furthermore, the framework
provides services for feed generation and con-
sumption, including the requisite conflict detec-
tion and preservation. While the Microsoft Sync
Framework offers RSS/ATOM support out-of-
the-box, developers can extend this support to
other formats as well. High level producer/con-
sumer APIs help easily turn any Microsoft Sync
Framework provider into an SSE-enabled end-
point!
Sync Provider Framework
The Microsoft Sync Framework supports several
ways of writing Sync Providers. Each is intended
to make the experience as easy and efficient as pos-
sible. In addition to full-featured (that is, “full par-
ticipant”) providers that are capable of true peer-
to-peer sync, the framework provides support for
allowing legacy and existing devices and applica-
tions to participate in the content flow ecosystem.
Specifically, the Microsoft Sync Framework Sync
Providers support the following scenarios:
• Partial Participants. Providers that sim-
ply store but do not understand most of the
sync metadata (e.g. a USB keychain, legacy
phone, media device). Despite being very
easy to develop even on endpoints that do
not host the Microsoft Sync Framework en-
gine, these providers can participate in all
multi-master content flow scenarios.
• Simple Participants. Endpoints that lack
the ability to detect changes and that lack
54 Introducing the Microsoft Sync Framework: Next Generation Synchronization Framework
the ability to store metadata. The Microsoft
Sync Framework has services to allow data
from these participants to flow within the
ecosystem on behalf of a fully featured par-
ticipant/provider.
• Anchor-Based Providers. Support for pro-
viders that rely on a simple tick-count based
enumeration mechanism for their change
detection mechanism (e.g. timestamps, tick
counts, etc.).
Storage Specific
Components
In addition to the Microsoft Sync Framework-en-
abled Sync Providers in development across Mi-
crosoft, the Microsoft Sync Framework includes
several components to simplify the development
of offline, sync, sharing, and roaming scenarios
using specific stores and protocols. These compo-
nents include:
• Relational Data Providers. Synchronization
Services for ADO.NET 2.0, the next version
of Sync Services discussed in Steve Lasker’s
article, (Caching with SQL Server Compact
and the Microsoft Sync Framework for ADO.
NET in this issue of CoDe Focus) will ship
with SQL Server 2008. Sync Services 2.0
is built on the Microsoft Sync Framework
aligning these two evolving sync platforms. It
continues to dramatically simplify the cach-
ing of a remote or local database for scenar-
ios such as line of business and branch of-
fice/point-of-sale scenarios, while providing
peer-to-peer and other advanced sync capa-
bilities. These providers take full advantage
of the new change-tracking features in SQL
Server 2008 and SQL Server Compact.
• SQL Server Compact Metadata Store. A
ready-to-use component for storing sync
metadata such as versions, anchors, and
change detection information. This compo-
nent greatly simplifies the development of
custom providers that do not have a natural
place to store metadata.
• File and Folder Sync Provider. A ready-to-
configure provider capable of representing
any Win32-compatible file system (e.g. FAT,
NTFS, removable device). This provider han-
dles challenges such as change detection on
FAT volumes (including move and rename
heuristics), name-name collision resolution,
update-delete conflicts (including hierarchi-
cal update-delete), and the ability to preview
a synchronization action.
• ADO.NET Data Services Offline Provider.
ADO.NET Data Services providers are cur-
rently being explored through prototypes
that allow synchronizing data using REST-
style interfaces for taking data services of-
fline.
• Plus many more providers currently in de-
velopment across Microsoft!
www.code-magazine.com
Sync-Enabled Content
Flow Scenarios

By removing the requirement that endpoints

need intimate knowledge of one another in
order to share or roam data, the Microsoft Sync
Framework creates a sync-enabled occasionally
connected ecosystem capable of providing:

• Offline support for rich Internet applica-

tions. Ability to take a Web service or an
Internet application offline with all changes
synced back to any other endpoint without
conflicts.
• Calendar, Contact, Task List sync. Support
for field-level sync such as “first name” “last
name” in an item-enabled rich PIM sync sce-
nario.
• Rich media sync. Ability to efficiently sync
a full media library, or smaller subset, to the
cloud or to devices either directly from an-
other endpoint or through intermediaries.
Ability to sync properties such as ratings and
play count without transferring the entire
media file.
• Easy caching. The ability to easily cache a
subset of the information from a remote or
local store to support offline scenarios.
• Sharing. Ability to have endpoints, such as
devices, share data between any number of
devices, clients, and services with edits pos-
sible at any node. This makes tethered or
over-the-air sharing or roaming of media,
PIM, files, and settings a breeze.

Conclusion
While this is an introductory, high-level overview
of the Microsoft Sync Framework CTP1, you’ll find
plenty of material including “How-to” documents,
full-featured samples, articles, and white papers as
we unveil the framework on MSDN and at confer-
ences in North America and Europe.

Please visit http://msdn.microsoft.com/sync to get a

copy of the Microsoft Sync Framework CTP1 SDK!
I’m personally looking forward to reading your com-
ments and feedback as we extend the framework
and deliver on the content flow vision and strategy.
I encourage developers to evaluate the SDK and re-
quest the features and improvements that will help
you become more productive and the platform eas-
ier to use. We can be reached at syncsdk@microsoft.
com and look forward to your feedback!

Moe Khosravy

www.code-magazine.com Introducing the Microsoft Sync Framework: Next Generation Synchronization Framework 55
ONLINE QUICK ID 0712112
Anthony Carrabino
anthony.carrabino@microsoft.
com
Anthony is a Sr. Product Manager
for SQL Server at Microsoft.
He has been developing and
marketing commercial software
products for developers since
1994. Before joining Microsoft,
Anthony was President/CEO of
Vista Software for eight years.
During that time he and his team
created VistaDB™ as the world’s
first fully managed SQL database
engine for Microsoft .NET.
Before VistaDB, he established
APOLLO™ as one of the world's
most popular data engines for
managing legacy CA-Clipper
and FoxPro files. Other career
highlights include selling a suite
of data access components
called Firefly for Flash™ to
Macromedia in 2002 and working
with the team that licensed a
graphics framework to Computer
Associates in 1995 that became
the GUI for CA-Clipper 5.3. Chess,
soccer, reading and working out
fill his spare time.
56 What’s New in SQL Server 2008?
What’s New in SQL Server
2008?
SQL Server 2008 is scheduled for release in 2008 and promises
to deliver an array of new and exciting benefits to both
developers and IT Pros alike.
T
he list below highlights some of the new and quired to keep backups online and enables
improved capabilities planned for SQL Serv- backups to run significantly faster.
er 2008. Enjoy! • Occasionally connected systems (OCS).
New unified synchronization support across
applications, data stores, and data types
Enterprise Data Platform with SQL Server 2008 and SQL Server
Compact 3.5.
• Transparent data en- • Partitioned table parallel-
Fast Facts
cryption. Enables en- ism. Improved performance
cryption of an entire da- Did you know that on large partitioned tables.
tabase, data files, and log SQL Server is the fastest • Star join query optimiza-
files, without the need growing DBMS and BI vendor tions. Improved query per-
for application changes. in the world? formance for common data
• Hot Add CPU. Scale And did you know that SQL warehouse scenarios by
your databases by dy- recognizing data warehouse
Server ships more units than
namically adding CPU join patterns.
resources to supported
ORACLE and
hardware platforms IBM combined?
without requiring appli- Beyond Relational
cation downtime.
• Policy-based management. New policy- • New spatial data types. New vector-based
based management architecture. spatial data types that conform to industry
• Extensible key management. New encryp- spatial standards allowing location-aware
tion and key management with support for applications to be developed.
third-party key management and hardware • New date and time data types. DATE
security module (HSM) products. (date-only type), TIME (time-only type),
• Performance data collection. New central- DATETIMEOFFSET (time zone aware DA-
ized data repository stores performance TETIME type), and DATETIME2 (new DA-
data and new reporting and monitoring TETIME type with larger fractional seconds
tools provide performance insights to ad- and year range than DATETIME).
ministrators. • New HIERARCHYID system type. HI-
• Data compression. Improved data com- ERARCHYID is a CLR UDT that provides
pression stores data more effectively and methods for creating and operating on val-
provides significant performance improve- ues that represent hierarchy nodes.
ments for heavy I/O workloads. • New FILESTREAM data type. Allows bina-
• Streamlined installation. New installation, ry data to be stored in an NTFS file system
setup, and configuration architecture sepa- while maintaining transactional consistency
rates the installation of physical bits on the with the database.
hardware from the configuration of the SQL • Integrated Full-Text Search. Perform high-
Server enabling custom installation configu- speed text searches on large text columns.
rations to be used. • Large user-defined types (UDTs). 8 KB
• Resource Governor. Define resource lim- limit for UDTs has been removed.
its and priorities for different workloads
enabling concurrent workloads to provide
consistent performance. Dynamic Development
• Predictable query performance. Greater
query performance, stability, and predict- • Sparse columns. Provides very efficient
ability with new functionality to lock down management of empty data in a database by
query plans. enabling NULL data to consume no physi-
• Backup compression. Reduces storage re- cal space.
www.code-magazine.com
 • CLR integration and ADO.NET object ser-
vices. Developers can program against a da-
tabase using CLR objects that are managed
by ADO.NET.
• Change data capture. Captures and main-
tains changes to data and schema across
tables.
• MERGE SQL statement. Enables develop-
ers to handle common tasks such as check-
ing if a row exists before executing an insert
or update.
• GROUPING SETS. Enables multiple
groupings to be defined in the same query,
producing a single result set that is equiva-
lent to a UNION ALL of differently grouped
rows.
Conclusion Disclaimer
Developers and organizations can count on SQL This article is based on a pre-
Server 2008 to deliver a powerful set of capabili- release build of SQL Server 2008
ties to solve the growing needs of managing data and makes no guarantees that
in the enterprise, on desktops, and on mobile features and/or capabilities listed
devices. As an integrated part of the Microsoft here will be available in the final
Data Platform vision, SQL Server 2008 is the release of SQL Server 2008.
most comprehensive release of Microsoft SQL
Server to date. For more information, please visit
http://www.microsoft.com/sql/prodinfo/futureversion/
default.mspx
Anthony Carrabino

Pervasive Insight

• New design of SQL Server Integration

Services (SSIS) Pipeline. Improved scal-
ability of runtime into multiple processors
allows Data Integration packages to scale
more effectively.
• Integration Services Persistent Lookups.
Improved performance of large table look-
ups.
• Block computations. Provides a significant
improvement in processing performance,
enabling users to increase the hierarchy
depth and computation complexity.

ADVERTISING INDEX
• New MOLAP-enabled writeback capa- Advertisers Index Data Development with Microsoft ® technologies

ET
.N
bilities. Removes the need to query ROLAP
partitions and provides users with enhanced Code Magazine 41, 63
writeback scenarios from within analytical www.code-magazine.com Volume 4 / Issue 3

applications. Microsoft Talks Data!

• Scalable reporting. Improved reporting en- Link up with LINQ
EPS Software Corp. 69 ADO.NET Data
gine and tools for creating, processing, for- Services
www.eps-cs.com/TabletPC Microsoft Sync
matting, and viewing reports with extensible Framework
architecture that enable easy integration of SQL Server 2008

reporting services. Microsoft 92 ODBC Rocks!

• Internet report deployment. Effortlessly http://www.microsoft.com/sql/bigdata

deploy reports over the Internet. www.code-magazine.com
US $ 5.95 Can $ 8.95

• Manage reporting infrastructure. Control SXSW Interactive Conference 87

server behavior with memory management, www.sxsw.com
infrastructure consolidation, and easier Advertising Sales:
configuration through a centralized store Vice President,
Tech Conferences Inc. 2-3 Sales and Marketing
and API. www.devconnections.com Tom Buckley
• Improved scale-out configuration. New 832-717-4445 ext. 34
support for managing multiple report serv- tbuckley@code-magazine.com
VFP Conversion Tools 79
ers. Sales Managers
www.vfpconversion.com/tools
• Built-in forms authentication. Built-in Erna Egger
forms authentication enables users to easily +43 (664) 151 0861
West Wind Technologies 9 erna@code-magazine.com
switch between Windows and Forms.
• Report Server application embedding. Re- www.west-wind.com Tammy Ferguson
port Server application embedding enables 832-717-4445 ext 26
tammy@code-magazine.com
the URLs in reports and subscriptions to Xiine 91
point back to front-end applications. www.xiine.com
• Microsoft Office integration. New Micro-
soft Office rendering enables users to con-
sume reports directly from within Microsoft
Word, and the existing Microsoft Excel® This listing is provided as a courtesy to our readers and
renderer has been greatly enhanced to sup- advertisers.
port nested data regions and sub-reports as The publisher assumes no responsibility for errors
well as merged cell improvements. or omissions.

www.code-magazine.com What’s New in SQL Server 2008? 57


ONLINE QUICK ID 0712122
Vaughn Washington
vaughn.washington@
microsoft.com
Vaughn Washington currently
leads the development team for
native data access in the SQL
Server division at Microsoft;
Vaughn is responsible for building
and maintaining Microsoft’s
world class data providers for
SQL Server: ODBC and OLE
DB. His responsibilities include
delivering scalable, performant,
and secure client components
that are consumed by Microsoft
internal and external customers
through APIs that adhere
to industry standards and
expectations while showcasing
SQL Server’s unique value
proposition.
58 Programming SQL Server 2008
Programming SQL Server
2008
SQL Server Katmai, now officially announced as SQL Server
2008, introduces a significant amount of new and improved
functionality, including enhanced data types and greater
programming flexibility for database application developers.
I
n this article I’ll look at the range of data access sions. One important takeaway is that you don’t have
technologies available for leveraging the power to rewrite your application from the ground up to take
of SQL Server 2008. After I show you how to advantage of SQL Server 2008. Instead, you can gain
select the right technology for significant value from incremen-
your needs, I’ll dive into the im- Fast Facts tal changes to your existing data
provements, first examining a access layer. With that said, let’s
new construct for programming Did you know SQL dive in.
data. Then I’ll turn to expanding Server 2008 has already
the set of building blocks—data released more than
types—available to developers. 16 improvements in the July New Programming
Taking a brief pause, I’ll discuss CTP? You can register to Constructs: Table-
how SQL Server 2008 removes
limitations for some scenarios.
participate in this and future Valued Parameters
Then building on what the ar-
technology previews at:
ticle has covered, I’ll examine http://connect.microsoft.com/sql One of the most requested new
new out-of-the-box solutions for features by developers is the abil-
common application scenarios. ity to cleanly encapsulate tabular
The article will hopefully leave you with a new un- data in a client application, ship it to the server in a
derstanding of the possibilities for taking advantage single command, and then continue to operate on the
of Katmai: SQL Server 2008. data as a table in T-SQL. The simplest such use case is
the long-desired array data type. Traditionally, applica-
tions solve this need by doing one of the following:
Choosing a Data Access Technology
• Defining stored procedures with large numbers
SQL Server offers a wide range of data access tech- of parameters and pivoting the scalar parameter
nologies to developers. The best place to start the dis- data into rows.
cussion on taking advantage of new programmability • Using an out-of-band bulk insert mechanism like
features is how to choose the right one. For new ap- SqlBulkCopy in ADO.NET to create a temporary
plication development, using the .NET Framework table.
and specifically the SQL Server provider in ADO.NET, • Using parameter arrays in the application and
System.Data.SqlClient, will be the best choice in most repeatedly executing logic that operates on a
cases. If you’re working within an existing application scalar “row” of data.
with business logic and associated data access code in
native C/C++, you can choose from the SQL Server None of these solutions is ideal. The pivoted param-
Native Client ODBC driver or the OLE DB provider. eter solution, in addition to being ungraceful, creates
Both options allow you to take advantage of the full set code that is difficult to maintain that’s also tough to
of features of SQL Server 2008 and the choice will usu- move forward when the time comes to add a new “col-
ally be based on your application requirements. umn” to the conceptual “table.” The bulk insert solu-
tion is clumsy when the application needs to do any
Additional data access options include Windows Data additional filtering or apply more complex logic. The
Access Components (WDAC)—new in Windows Vista, parameter array solution, while it may perform well
previously named Microsoft Data Access Components for small data volumes, becomes untenable for larger
(MDAC)—and the Microsoft SQL Server 2005 JDBC batches both on the client where memory consump-
Driver for Java environments. For purposes of this dis- tion may become a problem and on the server where
cussion, I’ll focus on SqlClient and SQL Server Native per-row invocations of the procedure provide non-op-
Client; for more information on these and other data timal performance.
access technologies, visit http://msdn.com/data.
Table-valued parameters (TVPs), believe it or not, ad-
To get the most out of the new functionality, you’ll need dress all of these problems. TVPs provide an improved
to use .NET Framework 3.5 or SQL Server Native Cli- programming model and significant performance ben-
ent 10.0 which works side-by-side with previous ver- efits in certain scenarios.
www.code-magazine.com
 Existing Type Accuracy Date/Time Range
Smalldatetime 1 minute 1900-1-1 through 2079-6-6
Datetime 3.33 milliseconds 1753-1-1 through 9999-12-31
Table 1: Existing SQL Server 2005 date and time data types along with their range and maximum precision.

Imagine a simple order processing scenario. Using eter class to take a DataTable as a value. DataTable’s Connecting the SQL
TVP starts with defining a table type in T-SQL on ubiquity in data application programming makes it an Server Developer
the server: ideal choice for this simple model. SQL Server Native
Client OLE DB accomplishes the same model by lever-
Community with the
-- TSQL to CREATE a TABLE TYPE tt_OrderItems aging the COM IRowset interface and by introducing Product Team
an implementation that allows buffering. SQL Server You may have heard about
CREATE TYPE tt_OrderItems AS TABLE ( Native Client ODBC uses a similar method modeled MSDN forums, http://forums.
[ItemId] int NOT NULL, after parameter arrays where applications allocate and microsoft.com, as a tool for
[Quantity] int NOT NULL) bind arrays of buffers. For the streaming model, ADO. seeking and providing answers
NET supports specifying DbDataReader and IEnum for technical problems, but did
Next, create a stored procedure that uses the table erable<SqlDataRecord> as a parameter value, which you know that in addition to
type you just created and additionally takes the cus- provides a solution for both external and user-defined MVPs, Microsoft development
tomer who placed the order as a parameter: data sources. In much the same vein, SNAC OLE DB product teams participate in
accepts a custom IRowset implementation and ODBC the forums? For SQL Server
-- TSQL to CREATE a PROCEDURE sp_AcceptOrder builds on the data-at-execution (DAE) paradigm by programming questions,
-- that performs set-based operation on TVP accepting DAE table-valued parameters. All providers check out these forums:
also expose a rich set of services for discovering and .NET Development > .NET
CREATE PROCEDURE sp_AcceptOrder ( describing table-valued parameter metadata. Framework Data Access and
@CustomerId int, Storage, and SQL Server > SQL
@OrderItems tt_OrderItems READONLY) When you can update an application to use table-val- Server Data Access. SQL Server
AS ued parameters, it should gain the benefits of having 2008 has versions of these
INSERT dbo.AcceptedOrders cleaner, more maintainable code in both the client and forums under SQL Server Katmai
SELECT O.ItemId, O.Quantity server application tiers; faster performance, particu- for any questions you have using
FROM @OrderItems AS O larly when you use set-based operations on the server new functionality.
INNER JOIN dbo.Items AS I side by leveraging the power of the SQL query proces-
ON O.ItemId = I.ItemId sor; and better scalability for large data volumes. In
WHERE I.BackOrdered = 0 addition to enhancing the programming model, SQL
Server 2008 introduces new intrinsic data types to bet-
This example is fairly simple, but it illustrates a big ter align with the precise needs of applications.
win for developers, which is that if you can update
an application to implement its business logic to use
set-based operations on a batch of data, it should see
significant performance gains. When you can update an
application to use table-valued
Here are the application changes needed to use table-
valued parameters. When using table-valued param- parameters, it should gain
eters, client applications generally have two possible the benefits of having cleaner,
programming models:
more maintainable code in both
• Bind in-memory table data as a parameter. This the client and server application
is usually the simplest and fastest to code at the
expense of not being as scalable in the applica-
tiers; faster performance,
tion for large batches of data due to increased particularly when you
memory consumption.
• Stream row data to the provider from a disk or
use set-based operations on
network-backed data source. This model takes the server side by leveraging
a bit more code to set up with the advantage of
having a fixed memory usage profile in the ap-
the power of the SQL query
plication. processor; and better scalability
For small batches of data, the performance difference
for large data volumes.
on the client will usually be negligible, so choosing the
simpler programming model may be your best choice.
On the server side there will be no performance differ- New Building Blocks: New and
ence between the models. Enhanced Date & Time Types
Diving into the details for a moment, ADO.NET ac- In previous releases of SQL Server, the database ex-
complishes the first model by extending the SqlParam- posed date and time data primarily through two data

www.code-magazine.com Programming SQL Server 2008 59

 types: DATETIME and SMALLDATETIME (Table 1).
In SQL Server 2005, these types began to show their
age and developers began to bump into their limita-
tions.
You can break down problems with the existing date/
time types into five categories:
• Applications that work in terms of only date or
only time data must implement a layer of abstrac-
tion on the server data often writing their own
validation routines. While feasible to accomplish,
this increases the burden on developers.
• Table column storage requirements for either
date or time only are considerably less than com-
bined date and time. This means that as the size
of a database increases, storage costs increase at
a rate faster than necessary.
• Existing ranges are often not large enough to rep-
resent the data that applications need to handle
(like process control and manufacturing).
• You cannot commonly represent time-zone data.
Some applications choose to work around this
by defining an additional column for the offset
and storing date/time in UTC. Along with that
solution comes the baggage of performing all the
necessary calculations to treat this as a single
type and the inability to straightforwardly take
advantage of built-in date/time functions in SQL
Server.
• Many other database vendors (and the ANSI
SQL standard) support unique date, unique time,
and time-zone aware date/time types, such that,
migrating an application using this functionality
to SQL Server was sometimes a cumbersome
process that might even require changing appli-
cation requirements.
In order to address these problems, SQL Server 2008
introduces support for four new types: DATE, TIME,
DATETIME2, and DATETIMEOFFSET; along with a
rich set of built-in T-SQL function support for the new
types:
• DATE. Provides a broader range of day value
than DATETIME by starting at 1/1/1 rather than
1/1/1753, and provides better storage scalability.
• TIME. Also provides storage scalability over ex-
isting types and introduces user-defined, fraction-
al-second precision from 0 to 7. In other words,
based on your needs you can define a table col-
umn of type TIME to be accurate to the second
or 100 nanoseconds and you’ll pay only for the
accuracy you need in storage costs.
• DATETIME2. A composition of DATE and TIME
in that it supports both a wider range and a user-
defined, fractional-second precision to 100 ns.
• DATETIMEOFFSET. Includes all of DATE-
TIME2 and additionally introduces time-zone
offset support, which should significantly reduce
the amount of custom code you need for time-
zone-aware applications.
As you can see, all new types support a broader range,
and where appropriate, user-defined, fractional-sec-
ond precision (Table 2). This allows developers to tune
60 Programming SQL Server 2008
the storage size of columns to exactly fit application
needs, which for large databases, translates to signifi-
cant savings in storage costs. Now let me discuss how
data access stacks expose these types.
The new SQL Server types DATE and DATETIME2
correlate to existing types in all data access technolo-
gies. For ADO.NET this is DateTime for both, for
ODBC it’s SQL_DATE and SQL_TIMESTAMP re-
spectively, and for OLE DB it’s DBTYPE_DBDATE
and DBTYPE_DBTIMESTAMP. The new SQL types
TIME and DATETIMEOFFSET were more difficult
to express in some cases because the conceptual type
didn’t already exist. While you could map TIME to
TimeSpan in ADO.NET, Microsoft needed to invent
new provider types in ODBC and OLE DB (SQL_
SS_TIME2 and DBTYPE_DBTIME2) because exist-
ing types don’t support fractional-second precision.
As you’d guess, these match their pre-existing types
in every way with the addition of a fraction compo-
nent matching the component that already exists in
SQL_TIMESTAMP/DBTYPE_DBTIMESTAMP. The
server type DATETIMEOFFSET is unique in that, in
addition to containing all date/time parts of other
types, it also includes a time-zone offset. To accom-
modate this type, Microsoft introduced new types
across the board. The .NET Framework 3.5 release
includes a new system type conveniently named Da-
teTimeOffset while OBDC and OLE DB introduce
SQL_TIMESTAMPOFFSET and DBTYPE_DBTIME-
STAMPOFFSET; all of which should look familiar to
people used to working with their non-time-zone-
aware equivalents. These new types are first-class
citizens in every way—with the goal of being better
data-type alternatives for new application develop-
ment and replacement options for application en-
hancements. In addition to introducing new data
types and programming constructs, SQL Server 2008
also removes limitations on existing types to open the
door to new application scenarios.
Breaking Barriers: Removing Size
Limitations
Here is a close look at new support for large common
language runtime (CLR) user-defined types (UDT) and
All new types support a
broader range, and where
appropriate, user-defined,
fractional-second precision.
This translates to developers being
able to tune the storage size
of columns to exactly fit
application needs, which for large
databases, translates to significant
savings in storage costs.
www.code-magazine.com
 New Type
date
time
datetime2
datetimeoffset
Table 2: New SQL Server 2008 date and time data types along with their range and maximum precision.
the introduction of storing large object (LOB) column
data transactionally in the file system.
Support for .NET Framework CLR UDTs first appeared
in SQL Server 2005 with the goal of providing database
application developers a more flexible programming
model. UDTs gave developers a way to create complex
types and also to express and encapsulate computa-
tionally expensive business logic alongside that data.
Since the release of SQL Server 2005, customers have
been adopting the technology and using it in interesting
ways; however, the current 8,000-byte maximum size
has limited the set of scenarios UDTs can address.
SQL Server 2008 removes this restriction allowing a
CLR UDT to be “unlimited” length. In practice the
storage size is actually limited at the SQL large object
(LOB) limit of 2^31 -1 bytes, or about 2 GB, in much
the same fashion as the varbinary(max), varchar(max),
and nvarchar(max) types that were introduced in SQL
Server 2005. Theoretically, you can expose any .NET
Framework CLR class or structure as a SQL CLR
UDT as long as it meets several requirements involving
constructors, annotations, and implementing a set of
interfaces. Large UDTs in particular must implement
a user-defined, serialization-to-bytes method that SQL
Server relies on both when consuming parameters
of that type from clients and when sending column
result sets. A value of Format.UserDefined in the
SqlUserDefinedTypeAttribute annotation indicates
that the UDT has user-defined serialization and also
introduces the requirement for the developer to imple-
ment the IBinarySerialize interface.
The analogy for other “max” types remains useful
when describing how to take advantage of large ver-
sions of UDT in client applications. Integrating into
applications currently using UDTs should be seamless,
with only minor changes in metadata returned for col-
umns and parameters of these types to allow discrimi-
nating applications to distinguish large versions from
their less-than-8,000-byte counterparts. To see this dif-
ference, ADO.NET applications for result sets will in-
voke the SqlDataReader GetSchemaTable() method
that returns a DataTable and examine the ColumnSize
column where a value of “-1” indicates a large UDT.
For parameters, they’ll examine or set the SqlParam-
eter Size property where “-1” has the same meaning.
In ODBC, you specify and distinguish large UDTs
by the use of the SQL_SS_LENGTH_UNLIMITED
macro originally introduced for “max” types. In OLE
DB, you represent the large UDT column or parameter
length as “~0”.
www.code-magazine.com
Maximum Accuracy Date/Time Range
1 day 1-1-1 through 9999-12-31
100 nanoseconds 00:00:00 through 23:59:59.9999999
100 nanoseconds 1-1-1 through 9999-12-31
23:59:59.9999999
100 nanoseconds 1-1-1 through 9999-12-31
23:59:59.9999999
In addition to being able to
fully utilize the functionality of
the CLR UDT on both server and
the application tier when working
in ADO.NET, using user-defined
serialization can allow native
applications to access a UDT as a
stream of bytes with a well-defined
format. This, in turn, enables
scenarios where applications
interpret these bytes by parsing or
by overlaying structure and provide
similar business logic
in middle tiers or client
applications to what exists as CLR
methods in the assembly.
In addition to being able to fully utilize the func-
tionality of the CLR UDT on both server and the
application tier when working in ADO.NET, using
user-defined serialization can allow native applica-
tions to access a UDT as a stream of bytes with a
well-defined format. This, in turn, enables scenarios
where applications interpret these bytes by parsing or
by overlaying structure and provide similar business
logic in middle tiers or client applications to what
exists as CLR methods in the assembly. While these
techniques can solve a range of scenarios involving
structured data, a different limitation removal assists
the growing number of document management-style
applications.
On the unstructured data side of the fence, SQL Serv-
er 2008 also enables applications to store unstructured
data directly on the file system—outside of the database
file—leveraging the rich streaming APIs and perfor-
mance of the Windows file system. Though accessible
through the file system, the data also remains compat-
ible with the T-SQL programming model. Using this
dual programming model—T-SQL and Win32—appli-
cations can maintain transactional consistency be-
tween unstructured data stored in the file system and
structured data stored in the relational database.
Programming SQL Server 2008 61
 You use this functionality—SQL Server Filestream—
by adding a new storage attribute, FILESTREAM,
on varbinary(max) columns, a type introduced in
SQL Server 2005. The beauty of the feature is that
other than removing the 2 GB LOB storage limita-
tion, filestream columns work seamlessly in other
operations, including data markup language (DML)
operations like SELECT, INSERT, DELETE, and
MERGE. In case you haven’t already heard, MERGE
is yet another new SQL Server 2008 programmabil-
ity feature that allows expressing multiple other DML
operations in one statement, but that’s a detail for
another day.
Getting back to Filestream, take a look at how this is
exposed in different data access stacks. Because the
only real change from the database application per-
spective is the difference in maximum size, existing
programming patterns continue to work unchanged.
Having said that, once you’ve made the transition to
storing large data sizes, streaming data in your appli-
cation both into and out of the server becomes more
important for scalability. Even though coding patterns
for streaming haven’t changed in SQL Server 2008,
it’s worth doing a quick refresher. For ODBC, the
application binds parameters using SQLBindParam-
eter with ColumnSize set to SQL_SS_LENGTH_UN-
LIMITED and sets the contents of StrLen_or_IndPtr
to SQL_DATA_AT_EXEC before it calls SQLExecDi-
rect or SQLExecute. You must unbind and retrieve
result columns via SQLGetData. For OLE DB, the
application uses DBTYPE_IUNKNOWN in parame-
ter and result bindings. For parameters, the consumer
must implement an ISequentialStream; for results,
the provider returns an ISequentialStream interface.
For optimal streaming performance, applications can
use a Win32 file access API to read or write data us-
ing a UNC path returned by new server built-in func-
tion PathName() available from filestream columns.
The advantage of streaming using Win32 API over
T-SQL grows as the data grows in size and the benefit
can be seen as early as data 1 MB in size. SQL Server
Native Client also includes a new method, OpenSql-
Filestream, that combines and simplifies the opera-
tions of opening the file and associating it with an ex-
isting SQL Server transaction context using another
new server built-in function, Get_filestream_trans-
action_context().
Building on New Foundations: System
Types
Beyond the plumbing and infrastructure improve-
ments discussed so far, SQL Server 2008 introduces
two new CLR system types as out-of-the-box solutions
for common application scenarios:
• HierarchyId. Allows database applications to
model tree structures like organizations or file
systems in a more efficient way than currently
possible. The type includes a rich set of func-
tions for answering questions about relationships
between nodes in the tree and reorganizing the
tree.
62 Programming SQL Server 2008
• Geometry. Implements the Open Geospatial
Consortium (OGC) geometry type and encapsu-
lates the entire OGC type hierarchy.
These CLR system types are available in every database
and the CLR assemblies are available as a standalone
redistributable package for applications to install with
classes named SqlHierarchyId and SqlGeometry re-
spectively. This allows .NET Framework applications
in the middle tier or client to interact with the type like
any other class. Non-.NET Framework applications
will generally rely on server-side conversion or seri-
alization to a documented well-known format. In the
case of the Geometry type, applications will be able
to retrieve the serialized bytes in a well-known format
by using a new built-in function when issuing queries,
choosing between STAsText() or STAsBinary(). These
provide, respectively, the Open Geospatial Consor-
tium (OGC) well-known text and well-known binary
format. Alternatively, for all system types, developers
can create user-defined functions on the server that
operate on the data providing access to the full func-
tionality of the type.
Putting the Pieces Together
In summary, SQL Server 2008 introduces a new pro-
gramming construct, table-valued parameters, to more
rationally model scenarios where applications are
operating as tables of data. Table-valued parameters
provide better performance and scalability than exist-
ing solutions, particularly when you can implement
server-side logic with set-based operations. Along
with this new programming construct, new intrinsic
data types more efficiently handle date and time data
and will help developers reduce custom code in ap-
plications by choosing a type that closely matches
their needs. This comes with the benefit of aligning
database storage costs to those same needs. For exist-
ing types, SQL Server 2008 removes two limitations:
lifting the 8,000-byte size limit on CLR user-defined
types and allowing you to store and access binary
large object columns through the file system. Building
on enhanced CLR type support, SQL Server 2008 also
introduces two system types to provide out-of-the-box
solutions for planar spatial and hierarchal data. This
is far from an exhaustive list of new programming fea-
tures, but it should start to get you thinking about how
you can get the most out of SQL Server 2008.
Vaughn Washington
www.code-magazine.com
Get 1 year of CoDe plus this
book for ONLY $37.95*

Pay the special combination price of just $37.95

and you will receive a 1 year subscription to
CoDe Magazine AND a copy of Pro VS 2005 Reporting!
Two great developer references, one low price! With this special offer
you get cutting edge, in-depth Visual Studio .NET articles delivered
to your door six times a year with Code Magazine plus a copy of
Pro VS 2005 Reporting by Kevin Goff and Rod Paddock (the editor of
CoDe Magazine).
This combination offer is only available online using this special URL:
www.code-magazine.com/subscribe/ap37

Visit www.apress.com for more great .NET titles!

* While supplies last. Please allow 4-6 weeks for delivery. US addresses only.
ONLINE QUICK ID 0712132
José Blakeley
joseb@microsoft.com
José Blakeley is a software
architect in the SQL Server
division at Microsoft Corporation
working with the SQL Server
Engine team. Previously he
was lead architect in the Data
Programmability team and helped
to build the ADO.NET Entity.
José was a lead designer of the
SQLCLR integration in SQL Server
2005 and has contributed to
numerous programmability and
extensibility features in various
SQL Server releases. Before
joining Microsoft in 1994, José
was a Member of the Technical
Staff at the Computer Science
Laboratory at Texas Instruments
where he was a principal
investigator in the development
of DARPA Open-OODB, an
object-oriented database system.
He has over 20 granted or
pending patents. José received
a computer systems engineering
degree from ITESM, Monterrey,
Mexico, and M.Math and Ph.D.
degrees in computer science
from the University of Waterloo,
Canada.
64 Use SQL CLR 2.0—Advancing CLR Integration in SQL Server 2008
Use SQL CLR 2.0—Advancing
CLR Integration in SQL
Server 2008
The integration of the .NET Common Language Runtime (CLR)
inside SQL Server 2005 (SQL CLR 1.0) enabled database
programmers to write business logic in the form of functions,
stored procedures, triggers, data types, and aggregates using
modern .NET programming languages.
This article presents the advances to the CLR integration introduced in SQL Server
2008, which significantly enhances the kinds of applications supported by SQL
Server.
I
n particular, this article describes the support of a CLR class or struct instance to a binary stream.
for large (greater than 8000 bytes) user-defined The deserialization routine performs the inverse oper-
types and aggregates, multiple-input user defined ation. SQL Server supports two forms of serialization:
aggregates, and order-aware native (Format.Native) and user-
table-vauled functions. The Fast Facts defined (Format.UserDefined).
CLR integration in SQL Server All forms of UDTs in SQL Server
2008 will also leverage the da- Geographical information 2005 have a size limit of up to
tabase scalar type extensibility is rapidly becoming 8000 bytes. SQL Server 2008 re-
introduced in SQL CLR 1.0 to mainstream to many business moves the size restriction of For-
provide a hierarchical identifier applications. mat.UserDefined UDTs allowing
data type to enable encoding SQL Server 2008 provides UDTs up to 2 GB.
of keys describing hierarchies
new spatial data types for
(folders, inheritance, etc) as Listing 1 shows a code fragment
well as a built-in framework for
developers to build describing the definition of a
spatial applications. This frame- location-aware applications. Polygon type as a large UDT. To
work includes: a class library define a UDT as large, one only
for geometry types based on the needs to specify a value of -1 for
Open Geospatial Consortium for both flat earth and the MaxByteSize property of the SqlUserDefinedType
round earth solutions, as well as a spatial index. custom attribute.
Large UDTs Handling User-defined Aggregates
SQL Server 2005 introduced user-defined types A very welcome extensibility feature in SQL Server
(UDTs) as a mechanism to extend the scalar type sys- 2005 is the ability to implement user-defined aggre-
tem of SQL. UDTs are not a general object-relational gate functions. As with any such feature, the usage
(O-R) mapping mechanism. That is, you should not scenarios expand very quickly beyond those initially
use UDTs to model complex business objects such as envisioned and require the platform to support ad-
employee, contact, customer, or product. For more ditional behaviors and capabilities. SQL Server 2008
general O-R mapping mechanisms, consider the expands user-defined aggregates to support multiple
LINQ to SQL or LINQ to Entities frameworks avail- input arguments as well as the ability to have results
able in Visual Studio 2008. Microsoft designed SQL for the aggregate be larger than 8000 bytes.
Server UDTs, implemented as CLR classes or structs,
to model atomic scalar types such as monetary cur- Customers have frequently asked the SQL Server
rency, geometry and spatial types (see section on spa- team to add support for a string concatenation aggre-
tial types), specialized date or time, etc. gate function. As an explanation for why they want
the request, a customer would typically explain how
To persistently store UDT instances, it must be possible they’d like to stitch together a set of e-mail addresses
to transform the state of the instance to and from its with a semicolon as a delimiter. SQL Server 2008’s
internal binary format. You can achieve this through user-defined aggregate function addresses this re-
a pair of routines for serialization and deserialization. quest. The accumulate method of the aggregate class
The serialization routine transforms the run-time state may look something like:
www.code-magazine.com
public void Accumulate (SqlString Value)
{ erful aggregates such as weighted averages or other
if (!Value.IsNull)
result+= Value.Value + ";";
}
If you tried this approach in SQL Server 2005 you
quickly realized that the resulting concatenated string
is constrained to an 8000-byte size limit. SQL Server
2008 doesn’t have this restriction and allows you to
have an aggregate as large as 2GB. In addition, you do
not need to change the managed code (C# in the snip-
pet above)—you only need to change the T-SQL reg-
istration. The example below specifies nvarchar(max)
as the return type of the aggregate function:
CREATE AGGREGATE dbo.concat
(@Value nvarchar(4000))
RETURNS nvarchar(max)
EXTERNAL NAME [concatProject].[concat];
You can use this aggregate in a query as you would
use any other aggregate function:
SELECT dbo.concat(email) FROM dbo.attendees
WHERE …
Imagine now a scenario where the format of each
concatenated value depends on some other e-mail-
specific flags included in a separate column of the
attendees table. SQL Server 2008 will let you create
aggregates that take multiple columns as input as in
the following query:
SELECT dbo.concat (email, format_flags)
FROM dbo.attendees WHERE …
This capability makes it simple to write more pow-
statistical functions such as median.
Order-Aware TVFs
One of the most powerful extensibility features en-
abled by the CLR integration in SQL Server 2005 is
the ability to write table-valued functions in any .NET
language. These are functions that have an initializer
method to set up any necessary context and provide
an enumerator and a method that is invoked once per
row. You can see a canonical example in Listing 2 that
shows return entries in the Event Log as a rowset in
SQL Server. Christian Kleinerman
Chirstian.Kleinerman@
Imagine now a query that wants to retrieve the first microsoft.com
few (e.g., 10) records from the log in chronological
order: Christian Kleinerman has
over 12 years of experience
SELECT TOP 10 * FROM dbo.Initmethod('System') working with SQL Server—the
ORDER BY timeWritten last seven as part of the SQL
Server development team in
If you examine the execution plan for this query, you Redmond, WA. He is currently
will notice that the optimizer introduces a sort opera- the Group Program Manager of
tor before returning the results so it can guarantee the the Relational Engine team where
top events given the particular requested order (Fig-
he has worked on a variety
ure 1).
of features and technologies.
You can see from the execution plan that a large por- Before Microsoft he worked
tion of the estimated execution cost goes into the on eCommerce Web site
sort operation. However, what happens if the table- development and previously was
valued function is already returning the data in the co-founder of a development
right order? The sort operation introduced by the company specializing in
optimizer is unnecessary. SQL Server 2008 intro- scheduling software.

Listing 1: Definition of a Polygon type as a large UDT

[Serializable] public void Read(BinaryReader r) {…}
[SqlUserDefinedType(Format.UserDefined, IsByteOrdered = false,
MaxByteSize = -1, I public void Write(BinaryWriter w) {…}
sFixedLength = false)]
public class Polygon : INullable, IBinarySerialize [SqlMethod(InvokeIfReceiverIsNull = true)]
{ [return: SqlFacet(MaxSize = -1)]
public Polygon(SqlString value) {…} public string ToString() {…}

public bool IsNull {…} public static Polygon Parse(SqlString s) {…}

public static Polygon Null {…} }

Listing 2: UDT
public class TabularEventLog public static void FillRow(Object obj, out SqlDateTime
{ timeWritten, out SqlChars
[SqlFunction(FillRowMethodName = "FillRow", message, out SqlChars category,
TableDefinition = "timeWritten datetime, out long instanceId)
message nvarchar(2000), {
category nvarchar(500), EventLogEntry eventLogEntry = (EventLogEntry)obj;
instanceId bigint")] timeWritten = new
public static IEnumerable InitMethod(String logname) SqlDateTime(eventLogEntry.TimeWritten);
{ message = new SqlChars(eventLogEntry.Message);
return new EventLog(logname, category = new SqlChars(eventLogEntry.Category);
Environment.MachineName).Entries; instanceId = eventLogEntry.InstanceId;
} }
}

www.code-magazine.com Use SQL CLR 2.0—Advancing CLR Integration in SQL Server 2008 65

Figure 1: The optimizer introduces a sort operator before returning the results so it can guarantee the top
events given the particular requested order.
References
Balaji Rathakrishnan, et al.
Using CLR Integration in
SQL Server 2005.
Alazel Acheson, et al. Hosting the
.NET Runtime in Microsoft SQL
Server, ACM SIGMOD Conf. 2004.
Figure 2: With ordered TVFs the optimizer does not need to introduce a sort operator to the execution plan.
66 Use SQL CLR 2.0—Advancing CLR Integration in SQL Server 2008
duces an additional ORDER clause in the CREATE
FUNCTION statement that tells the optimizer the
expected ordering of the results from the table-val-
ued function:
CREATE FUNCTION [dbo].[InitMethod](@logname
[nvarchar](4000))
RETURNS TABLE (
[timeWritten] [datetime] NULL,
[message] [nvarchar](max) NULL,
[category] [nvarchar](4000) NULL,
[instanceId] [bigint] NULL
) WITH EXECUTE AS CALLER
ORDER ([timeWritten])
AS
EXTERNAL NAME
[SqlServerProject1].[TabularEventLog].[InitMethod]
Note how with this small enhancement, the execu-
tion plan for the same query shows no sort operation
and the bulk of the cost happens in the CLR function
itself (Figure 2).
Spatial Framework
Even though location is a concept prevalent in every-
thing that people do, in recent years location aware-
ness has become an integral part of everyday soft-
ware applications. With the proliferation of mapping
frameworks, GPS, and other location-aware devices,
it is increasingly common to encounter spatial data,
which often needs to be stored, queried, and reasoned
upon. SQL Server 2008 leverages the CLR integration
to provide native spatial data support.
Many business applications have some form of loca-
tion data: sales regions, delivery routes, factory or
point of sale locations, or employee addresses. Not
only do business managers often require that applica-
tions store this information in a database, but business
managers also commonly desire the ability to run que-
ries that make use of the spatial semantics: What is
the average distance that customers have to the near-
est point of sale location? How do sales for a region
compare to those from adjacent regions?
SQL Server 2008 supports storing and querying of
geospatial data, that is, location data referenced to the
earth. Without going into too much detail, two com-
mon models of this data are the planar and geodetic
coordinate systems. The main distinction between
these two systems is that the latter takes into account
the curvature of the earth. SQL Server 2008 intro-
duces two new data types: geometry and geography,
which correspond to the planar and geodetic models.
SQL Server implements these data types as CLR types
leveraging the User Defined Type infrastructure avail-
able in SQL Server 2005. Microsoft registers and in-
cludes these types in every installation of SQL Server,
making it easy to create columns of either of these
types on a table:
CREATE TABLE points_of_sale
(id int, name nvarchar(50), location geometry);
The Open Geospatial Consortium (OGC) defines ca-
nonical textual representation for a geometry known
as the Well-Known Text (WKT). You can use the
WKT to insert data into the points_of_sale table by
using a static method that converts from WKT to an
actual instance:
INSERT INTO points_of_sale (id, name, location)
VALUES (1000, N'Main Store',
geometry::STPointFromText('POINT (50 50)', 0));
You can later write queries against the table created
above to answer business questions. For example,
assume that you have a geometry variable that holds
a polygon representing a given urban area. You can
then write a query to retrieve from the database all
points of sale that fall within the given urban area:
SELECT id, name
FROM points_of_sale
WHERE location. STIntersects(@urban_area) = 1;
At the same time, SQL Server 2008 provides new
spatial indexing capabilities to speed up processing of
queries involving spatial operations. As an example,
the following statement would create an index on the
table above:
CREATE SPATIAL INDEX SIndx_points_of_sale
ON points_of_sale(location)
WITH ( BOUNDING_BOX = ( 0, 0, 500, 500 ) );
The geometry data type works well for operations
performed on instances spanning small areas. When
operating on larger distances, you must take into ac-
count the curvature of the earth and this is when the
www.code-magazine.com
geography data type comes into play. This is imple-
mented as a separate CLR class. Even though the
usage is very similar to that of the planar data type,
methods invoked on the geodetic type operate on an
ellipsoidal model of the earth.

Leveraging the integration of the CLR into the data-

base engine, SQL Server 2008 introduces spatial data
support, which will make development of location-
aware applications become mainstream. Figure 3: An organizational hierarchy.

DECLARE @mnode hierarchyid, @lc hierarchyid Data Platform Team

Hierarchical Identifiers SELECT @mnode = node FROM organization WHERE
empid = @mgrid
SQL Server 2008 introduces the HierarchyId type, de- BEGIN TRANSACTION
signed to make it easier to store and query hierarchi- SELECT @lc = max(node) FROM organization
cal data. Hierarchical data is defined as a set of data WHERE @mnode = node.GetAncestor(1)
items related to one another by hierarchical relation- INSERT organization (node, empid, name)
ships—relationships where one item of data is the par- VALUES (@mnode.GetDescendant(@lc, NULL), @empid,
ent of another item. Common examples include: an @name)
organizational structure, a hierarchical file system, a COMMIT
set of tasks in a project, a taxonomy of language terms, END
a single-inheritance type hierarchy, part-subpart rela-
tionships, and a graph of links among Web pages. Microsoft implemented the HierarchyId built-in SQL
scalar type as a CLR UDT. Table 1 lists some of the
Consider the example organizational hierarchy in Fig- key methods exposed by this type.
ure 3.
You can expect that this new data type will enable the
The number sequence indicated at the upper-right efficient implementation of applications that access Peter Spiro
corner of every box represents a hierarchical identifier hierarchically organized data. “I didn’t get my first real
for the corresponding department. One could model corporate job till age 29…and I
this organization using the following table and index: think that’s a good thing to do,
Summary because when you do that, you
CREATE TABLE organization ( don’t get burned out at 40.”
node hierarchyid primary key clustered, SQL Server 2008 builds on the CLR integration and From a forestry degree to
level as node.GetLevel() persisted, scalar data type extensibility foundation laid out in making charcoal in the Peace
empid int unique, SQL Server 2005. Microsoft has extended the UDT Corps, Peter is now one of 14
name nvarchar(100) capabilities by allowing UDTs of size up to 2 GB. Technical Fellows at Microsoft.
… The SQL Server team have also illustrated two new Recruited to Microsoft from DEC
) capabilities—the spatial data type framework and the in 1994, Peter was brought in to
CREATE UNIQUE INDEX org_idx ON organization hierarchical id—built on the scalar UDT extensibility help re-engineer the database
(level, node) mechanism introduced in SQL Server 2005. These management system and move it
new capabilities will significantly enhance the kinds to the next level.
Inserting the CEO record: of applications supported by SQL Server. Other ca-
“So just building this world-class
pabilities not described in this paper include often-
database was the first thing
INSERT organization (node, empid, name) VALUES requested enhancements such as the support for
we did. Now we are moving
(hierarchyid::GetRoot(), 123, 'Frank Smith') INullable<T> for describing parameters to functions
from a database to a suite of
and procedures and table-valued parameters.
products, building a better
Adding an employee reporting to a manager:
Josè Blakeley database for the cloud, building
Christian Kleinerman better management capabilities,
CREATE PROC AddEmp (@mgrid int, @empid int, @name
building synchronization
nvarchar(100) ) AS BEGIN
technologies, building in-
memory caching and mapping
technologies.
Method Description
It’s not a core database anymore;
Static SqlHierarchyId GetRoot() Returns the root of the hierarchy type it’s a collection of services on
SqlHierarchyId GetDescendant ( Returns a child Id x such that child1 < x and top of the database that create a
SqlHierarchyId child1, SqlHierarchyId child2) child2 > x. database platform.”
SqlBoolean IsDescendant (SqlHierarchyId child ) Returns true if child is >= this.
SqlInt16 GetLevel() Returns an integer representing the depth of the
node this in the tree. Returns 0 if this is the root.
SqlHierarchyId GetAncestor (int n ) Returns a hierarchical id representing the nth
ancestor of this.
Table 1: A list of some of the key methods exposed by CLR UDT.

www.code-magazine.com Use SQL CLR 2.0—Advancing CLR Integration in SQL Server 2008 67
ONLINE QUICK ID 0712142
Jonathan Wells
blog.onoj.net
Jonathan is currently a product
manager in Microsoft’s Developer
Division and focuses on Visual
Studio 2008. During his seven
years at Microsoft, Jonathan has
served as a Software Design
Engineer / Test (C# team), .NET
Compact Framework product
manager, and as an Architect
Evangelist.
Visual Studio 2008
Beta 2
This beta includes a Go-Live
license so that you can start
taking advantage of Visual
Studio 2008’s new features
today. With mulit-targeting you
can use Visual Studio 2008’s
new designer and code editing
features with projects that target
.NET Framework version 2.0 and
3.0; today!
68 Visual Studio 2008: RAD Gets RADer
Visual Studio 2008:
RAD Gets RADer
Visual Studio 2008 is all about making it easier for developers and
development teams to create software for the latest and greatest
platforms with technologies such as .NET Language Integrated Query
(LINQ), ASP.NET AJAX, and the Windows Presentation Foundation
(WPF) designer, to name just a few. In this brief article I will highlight just a
few of the code editing and designer improvements that are new to Visual Studio 2008.
LINQ and IntelliSense
Fast Facts Studio 2008 so that you can take
advantage of the new design and
Integrating query syntax into Vi- Many of the new designer code editing capabilities without
sual Basic and Visual C# enables and code editing capabilities have to target a new framework
functionality such as statement introduced in version.
completion and IntelliSense Visual Studio 2008 are also in
when working with data queries. the free and lightweight Visual Web Designer
The approach for querying data
Studio Express Editions.
with LINQ is the same whether Visual Studio 2008 uses the same
you are accessing an in-memory Web designer that ships with the
list, XML data, a SQL database or any combination Microsoft Expression products. The result is a snap-
of these together. In addition to the design-time assis- pier Web development experience and a swag of new
tance provided by statement completion, smart com- designer features. The same multi-targeting capability
pile auto correction (that squiggle under syntax errors) applies to the new Web designer capabilities.
and IntelliSense, LINQ enables compile-time valida-
tion of queries so that you catch potential bugs before Visual Web Developer 2008 Express Edition now sup-
your code is executed. ports split-view editing allowing simultaneous display
of HTML source and the design view. Changes made
LINQ’s expressive style enables a cleaner code base in one view are immediately displayed in the other.
which is ultimately easier to understand and main-
tain. Working with Cascading Style Sheets (CSS) is now a
whole lot easier with the addition of CSS Style Man-
Working with JavaScript ager, CSS properties windows, and CSS Source View
IntelliSense. These features reduce the effort required
Visual Studio 2008 includes IntelliSense support for to understand and manage your project’s CSS format-
JavaScript, which will improve the ASP.NET AJAX ting rules.
development experience. IntelliSense greatly improves
the discoverability of variables, objects, and their meth- Other Improvements Worth Checking Out
ods without having to context switch by ALT-TAB’ing
to some form of documentation. Speaking of context Client developers should check out the new WPF
switch, a simple but neat new feature that pertains to designer and project support. Both Windows Forms
any IntelliSense dropdown is IntelliSense Transpar- and WPF application types can now take advantage of
ency Mode. In the past, developers had to escape out ASP.NET Application Services (Membership, Roles,
of IntelliSense to see the code under the dropdown. Profile) for roaming smart client user data. Developing
Visual Studio 2008 lets you make the dropdown semi- solutions for Microsoft Office is much richer and now
transparent by holding the Ctrl key. includes the ability to integrate with the Ribbon. Prac-
titioners of agile software development methodologies
JavaScript debugging is sure to be a popular feature and distributed teams will be pleased to discover that
with Web developers and is available in Visual Web Continuous Integration is now built into Visual Studio
Developer 2008 Express Edition, which is free. The Team System 2008 Team Foundation Server.
JavaScript debugger is fully loaded with the debugging
features that developers have come to expect of Visual You can find more information about the many im-
Studio. You need not upgrade your existing applica- provements and features in Visual Studio 2008 by
tions to Visual Studio 2008 and the .NET Framework visiting the MSDN Visual Studio Developer Center:
3.5 to take advantage of the debugger, or indeed any http://msdn.microsoft.com/vstudio.
of the new Web designer and features. Multi-target-
ing allows you to open existing projects that target Jonathan Wells
version 2.0 or 3.0 of the .NET Framework in Visual
www.code-magazine.com
The slate.
The pen.
The mightier you.
When you’re ready to become mighty,
EPS Software can help you build
state-of-the-art Tablet PC and Mobile PC
applications. EPS Software employs
experts like Markus Egger (according
to Microsoft, one of the ten most important
Tablet PC influentials in the world),
Claudio Lassala (Microsoft MVP) and other
well-known names. EPS Software (we are
also the publishers of CoDe Magazine) has
what any organization needs to implement
a successful Tablet PC project.

www.eps-cs.com/TabletPC
ONLINE QUICK ID 0712152
Gert Drapers
gert.drapers@microsoft.com
Gert Drapers is the Software
Architect and Development
Manager of the “Data Dude”
project. Gert has been with
Microsoft since 1991 where he
has worked in various capacities
on SQL Server from version 7.0
to SQL Server 2005. He also
created System.Transactions.
Figure 1: Adding a New Project to SCC.
70 The Data Dude Meets Team Build
The Data Dude Meets Team
Build
“Integrate the data tier developer into the core development life
cycle and process.” That is one of the main objectives of Visual Studio Team
Edition for Database Professionals, also known under its project name “Data Dude.”
Bringing the data tier developer into Visual Studio is the first step in enabling closer
integration between the application and data tier developer. Having both environments
leverage the same Team Foundation Build (Team Build) system enables daily and
automatic integration of changes into the build process, enforcing closer integration and
shorter feedback cycles between the two originally disjoint disciplines.
V
isual Studio Team Edition for Database Pro- presents the physical layout of all the directories
fessionals (VSDBPro) enables you to manage and files used to store the objects inside the proj-
your SQL Server database schema (definition) ect (Figure 2). The Schema View provides a logi-
the same way as you handle your application projects cal representation of the complete schema orga-
and source code. It starts by representing your da- nized by schema and/or object type (Figure 3).
tabase schema inside the newly
added Database Project (.dbproj) Fast Facts
(see Figure 1). It does so through Build & Deploy
a collection of T-SQL DDL frag- Visual Studio Team Edition
ments. Each fragment is stored in for Database Professionals Now that you have a collection of
a single .SQL file and represents adds new capabilities to .SQL fragments, how do you go
a single schema artifact, for ex- help with the configuration about deploying your schema?
ample a table, queue, constraint management, tracking,
or procedure definition. Because collaboration, and testing of This is where “build” comes
fragments only represent a single in. When you build a Database
your SQL Server databases.
schema object, tracking changes Project, the build engine takes
and versioning these schema ob- all the fragments inside the proj-
jects is a lot simpler and precise. ect and compares them with the schema inside the
target database.
After you have established the Database Project
you have two views: The Solution Explorer view Build is based on a difference-based build engine (Fig-
ure 4). The build engine takes two inputs: the project
state; which is “what you want” and the target data-
base; which is “what you have.” The comparison re-
sults in a set of schema objects that are different, which
are sorted to reflect the correct dependency order. The
build engine determined this dependency information
during the load of the project by parsing and interpret-
ing all DDL fragments for the lifetime of the project.
Finally, the build engine generates a deployment script
in the form of a .SQL script which contains a set of
T-SQL DDL statements to incrementally update the
schema on the target server/database. This alleviates
the need to manually create and/or maintain incre-
mental update scripts to keep your database schema
up-to-date.
Using the incremental deployment script you can up-
date the target database, using the Deploy command
inside Visual Studio.
The Truth Is in the Project
The big change is that the Database Project is now
the authoritative source of the schema definition,
www.code-magazine.com
not the database instance(s) the schema is deployed
to. Since the Database Project has become the cen-
ter of truth for our database schema, placing the
project under source code control enables the next
step in our crusade of integrating the data tier into
the development life cycle: versioning of the sche-
ma objects (Figure 5).
This enables placing the project under source code
control (SCC), the versioning of the individual
schema objects, and the tracking of the changes of
these objects. It also enables alignment in version-
ing between the application and the data tier. Both
can use the same labels to indicate versions at a giv-
en point in time and use branching when projects
are derived or merged.
So when a developer syncs his or her sources to a
certain label inside source code control, this con-
tains the definition of the state of the database
schema as it was used at that point in time when
the label was created.
Now that you understand how the individual proj-
ects work, let me switch gears and focus on how
you can integrate this into the team environment.
MSBuild Is Your Friend
Like most Visual Studio-based project systems,
the core tasks inside the projects like compilation
and linkage are implemented as MSBuild tasks.
The same is true for VSDBPro. The two core ac-
tivities for Database Projects (.dbproj), “Build” and
“Deploy” are implemented by two MSBuild tasks
named “SqlBuildTask” and “SqlDeployTask.”
All VSDBPro-related MSBuild tasks are defined in
the .targets file located in:
%ProgramFiles%\MSBuild\Microsoft\VisualStudio\
v8.0\TeamData\ Microsoft.VisualStudio.TeamSystem.
Data.Tasks.targets
The properties exposed by the MSBuild tasks are doc-
umented via an accompanying XSD file located in:
%ProgramFiles%\Microsoft Visual Studio
8\Xml\Schemas\1033\MSBuild\ Microsoft.
VisualStudio.TeamSystem.Data.Tasks.xsd
Any project file that wants to include the functional-
ity offered by the build task includes the .targets file;
the .XSD file is used to provide IntelliSense support
when you edit the project file inside the XML Edi-
tor in Visual Studio.
Building from the Command Line
The biggest benefit of all this? You can build and
deploy your Database projects from the command
line. When needed you can use the command line
to override settings and use MSBuild to automate
almost all possible tasks in a normal build process,
www.code-magazine.com
which now can include building and deploying your
database schemas.
To build from the command line you simply start a
Visual Studio Command Prompt (this makes sure
MSBuild.exe is in the PATH), navigate to the direc-
tory where your project file is stored, and execute
the following statement:
CD C:\src\AdventureWorksDB <enter>
MSBuild AdventureWorksDB.dbproj /t:build <enter>
This will use all the property settings as defined inside
the (.dbproj) project file and use them to build the proj-
ect. If you need to override some properties you simply
pass them to MSBuild.exe via the command line as
Figure 2: Solution Explorer—
name-value pairs using the property parameter.
AdventureWorksDB.
CD C:\src\AdventureWorksDB <enter>
MSBuild AdventureWorksDB.dbproj /t:build
/p:TargetDatabase="MyAdventureWorks"
/p:TargetConnectionString="Data
Source=(local)\sql90;User=sa;
Password=MySecret99;Pooling=False;" <enter>
This statement will change the target database name
to “MyAdventureWorks” and use your local SQL
Server instance named SQL90 and connect using
standard SQL Server authentication.
The result of the build is a .SQL file, which by de-
fault is named the same as your project and lives
in the sql directory under the location where your
project file is located.
Figure 3: Schema View—
To deploy the resulting build script you change the AdventureWorksDB.
target of the project to /t:Deploy:
CD C:\src\AdventureWorksDB <enter>
MSBuild AdventureWorksDB.dbproj /t:deploy <enter>
Difference Based Build
Figure 4: Difference Based Build.
The Data Dude Meets Team Build 71
 Figure 5: Initial Check-in.
You can combine the build and deploy into a single
invocation using:
CD C:\src\AdventureWorksDB <enter>
MSBuild AdventureWorksDB.dbproj /t:build;deploy
<enter>
MSBuild extracts all information needed to execute
the tasks from the project files and allows you to
Listing 1: Partial AdventureWorksDB.dbproj.user
<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<PropertyGroup Condition=" '$(Configuration)' == 'Default' ">
<TargetDatabase>AdventureWorksDB</TargetDatabase>
<GenerateDropsIfNotInProject>False</GenerateDropsIfNotInProject>
<DefaultDataPath>D:\DATA\</DefaultDataPath>
<TargetConnectionString>Data Source=TESTHOST\SQL90QA;
Integrated Security=True;Pooling=False</TargetConnectionString>
<AlwaysCreateNewDatabase>False</AlwaysCreateNewDatabase>
<BlockIncrementalDeploymentIfDataLoss>True
</BlockIncrementalDeploymentIfDataLoss>
<PerformDatabaseBackup>False</PerformDatabaseBackup>
</PropertyGroup>
</Project>
72 The Data Dude Meets Team Build
override property values via the command line. The
Database project file is split in two pieces; projects
settings which are stored in the .dbproj file and user
settings which are stored in the .dbproj.user file. Both
these files are aggregated into a single file image that
represent the project and are passed on to the tasks at
execution time. You can see the partial content of the
AdventureWorksDB.dbproj.user file in Listing 1. It
contains properties with their respective values that
are specific to the user. In general it contains settings
that you want the user to be able to change for him or
herself, but not affect other users of the project.
An example of a user setting is the “AlwaysCreat-
eNewDatabase” property which indicates that you
always want to drop and recreate your database
when it is being deployed.
NOTE: The .dbproj.user file should not be checked
in!
In the next section, Team Build, you’ll learn that you
need to be aware of which properties are defined in-
side the .dbproj.user file when you start integrating
Database Projects into a Team Build environment.
www.code-magazine.com
Figure 6: New Team Build Type Creation Wizard page 1.
Team Build
Now you’ll learn to integrate a Database Project into
Team Build. The first thing we need to do is add a
new Team Build Type, which we will do using the
“New Team Build Type Creation Wizard” (Figure 6).
This wizard will guide you through the process of cre-
ating a Team Build type in six steps. The wizard will
ask you to provide the name of the type; which solu-
tion you want to build; the configuration section to
use; some location settings, and some other options.
Figure 7 shows where you will select the solution.
Figure 8 shows the most important page in the wiz-
ard—the Configurations page. Here you would nor-
mally choose between Debug and Release, but in this
case the Database Project does not implement either
of these two configurations; it uses the Default con-
figuration, which is what you need to use when defin-
ing the build type.
In the fourth page (Figure 9) you specify the relative
locations that Team Build will use. For all other op-
tions in this article I will use the default values pro-
vided by the wizard.
Figure 8: New Team Build Type Creation Wizard page 3.
www.code-magazine.com
Figure 7: New Team Build Type Creation Wizard page 2.
With a completed Team Build Type you can right-
click on the build type inside Team Explorer to
build the project. Figure 10 shows the Build dia-
log box. Visual Studio
Team Edition
When you click “Build” you will see the Team
Build page which displays the progress of the
for Database
build process. The first time you build a database Professionals
project following the steps laid out in this article,
the build will fail! (Figure 11)
(VSDBPro)
enables you to
The best way to troubleshoot this is to look at the
build output placed in the drop location, which you
manage your SQL
specified for the Team Build Type when you cre- Server database
ated it. Since build output is shared between users,
it helps to make the drop location available through
schema (definition)
a share name, which in this example is \\mi-svr\ the same way
drops.
as you handle
dir \\mi-svr\drops\AdventureWorksDB_20070810.1 /s /b your application
\\mi-svr\drops\AdventureWorksDB_20070810.1\BuildLog.txt
projects and
\\mi-svr\drops\AdventureWorksDB_20070810.1\Default.txt source code.
\\mi-svr\drops\AdventureWorksDB_20070810.1\
ErrorsWarningsLog.txt
Figure 9: New Team Build Type Creation Wizard page 4.
The Data Dude Meets Team Build 73
 The BuildLog.txt file contains a complete recording of
all the steps performed during the build. The Default.
txt file contains the warning and error information for
the Default build configuration and the ErrorsWarn-
ingsLog.txt contains the warning and error informa-
tion for all configurations.

In the build that failed, look at the files named De-

fault.txt and ErrorsWarningsLog.txt, which will reveal
the same information.

Solution: AdventureWorksDB.sln, Project:

AdventureWorksDB.dbproj, Configuration: Default, Any CPU
(0,0): error TSD257: The value for $(DefaultDataPath) is
not set, please set it through the build property page.

The above results clearly point out that the Default-

DataPath is missing. This happens to be a user setting
and since .dbproj.user files are not checked in and
Team Build does not automatically create one when
missing like Visual Studio does, some properties do
not have values defined inside the .dbproj file.
In order to resolve this, you need to manually edit
the project and provide the values for the missing
properties. You will find that the .dbproj file con-
tains the tags for the properties, but either no value
is defined or they contain the value “Undefined.”
<PropertyGroup>
<Configuration Condition="
'$(Configuration)' == ''
">Default</Configuration>
Figure 10: Build Team project.
<DefaultDataPath>Undefined</DefaultDataPath>
<TargetConnectionString></TargetConnectionString>
<TargetDatabase></TargetDatabase>
</PropertyGroup>
You need to change the .dbproj file to reflect the
correct property values that the Team Build service
should use when building the project. Keep in mind
that the security privileges of the Team Build ser-
vice can be different than one of an interactive user
building the project inside the Visual Studio IDE.
Using the MSBuild-based command line build in
combination with the Windows RunAs command

Figure 11: Build failure.

74 The Data Dude Meets Team Build www.code-magazine.com

Figure 12: Build success.

provides a good test environment for interactive Wrap Up Overview of Team

troubleshooting.
Now you know how to integrate Database Projects
Foundation Build
<PropertyGroup> into a Team Build environment. You could achieve You can find out more about
<Configuration Condition=" '$(Configuration)' == '' more things by extending the approach described the various pieces of Team
">Default</Configuration> above. For example, you could automatically de- Foundation Build at:
<DefaultDataPath>D:\DATA</DefaultDataPath> ploy the database schema as part of the build pro- http://msdn2.microsoft.
<TargetConnectionString>Data cess by adding the SqlDeployTask to the post-build com/en-us/library/
Source=TESTHOST\SQL90QA;Integrated events, or you could integrate database unit testing ms181710(VS.90).aspx
Security=True;Pooling=False</TargetConnectionString> and data generation into the test execution part of
<TargetDatabase>AdventureWorksDB</TargetDatabase> Team Build.
</PropertyGroup> Gert Drapers

Make sure to check in the changes to the .dbproj

file so that when the Team Build service performs
a build and syncs the sources, it will automatically
pick up the new project file with the changed val-
ues. The build should now complete successfully
and without errors as shown in the resulting build
report in Figure 12.

As the last step, check the output of the build

by inspecting the drop location using the com-
mand:

dir \\mi-svr\drops\AdventureWorksDB_20070810.4 /s /b

The drop location now contains the build script

(.sql) and the metadata file (.dbmeta) which you
can use to deploy the database schema to the TES-
THOST\SQL90QA database instance.

www.code-magazine.com The Data Dude Meets Team Build 75

ONLINE QUICK ID 0712162
Stan Kitsis
stan.kitsis@microsoft.com
Stan Kitsis is a Program Manager
in the Data Programmability team
at Microsoft. Stan has over 10
years of software development
experience. His primary focus is
on the XML tools, which include
XML Editor, XSLT Debugger, and
XML Schema Designer. Prior to
working on the XML tools, Stan
was responsible for parts of the
System.xml and MSXML APIs.
XML Editor comes
with a full range
of features you
would expect
from a Visual
Studio editor,
which includes
IntelliSense, color-
coding, brace
matching, and
formatting.
76 XML Tools in Visual Studio 2008
XML Tools in Visual Studio
2008
XML is everywhere from XML Web Services to databases to config
files to Office documents. This article will show you tooling support offered
in Visual Studio 2008 that will make working with XML easier. It will cover editing
XML files, working with XML schemas, debugging XSLT style sheets and extending
Visual Studio by writing your own custom XML Designers.
W
hen you open a file with an XML extension So what do you do when there are no snippets and
in Visual Studio 2008 (for example, .xml, .xsd, you need to create an XML file based on an existing
.xslt, svg, or .config), you will invoke its XMLschema? XML Editor offers a wide range of features
Editor. XML Editor comes with a full range of features when you associate your XML documents with XML
you would expect from a Visual Studio editor, which schemas. Schema-based IntelliSense, real-time valida-
includes IntelliSense, color-coding, brace matching, tion, and error reporting are just a few of them. In
outlining, and formatting. It provides full XML 1.0 addition, XML Editor can dynamically generate snip-
syntax checking, end-tag completion, as well as DTD pets based on an existing schema. Once you provide a
and XML schema support with real-time validation. name of the element you want to add, the XML Editor
can parse the schema for required
Fast Facts information, generate a snippet,
Editing XML Files and insert it for you. To invoke
“Visual Studio seems to dynamic snippet functionality, all
Face it, manual editing of XML be the best editor for XML you need to do is type the name
files can be very tedious and time (compared to Eclipse plugins of the element as in the following
consuming. To help with this, the and NetBeans) and I use it example:
Visual Studio 2008 XML Editor even when my Web project is
comes with a number of pro- <element_name
Java based.”
ductivity enhancement features.
One such feature is an extensible and press TAB. The XML Editor
library of XML code snippets— - Customer statement from will create a snippet, which looks
XML files that contain a configu- the MSDN Forums very much like the one in Figure
rable code segment, which acts 1 except this time you didn’t have
as a template to use while editing to do anything up front. This is
documents. Visual Studio installs a number of XML a very powerful feature, especially when you need to
code snippets that help developers to write XML sche- create documents with large content models.
mas and XSLT style sheets. To invoke a snippet while
editing an XML file, select “Insert Snippet” from the By default the XML Editor generates only the required
“Edit > IntelliSense” menu. Once you have inserted a content, but this can be customized by annotating
snippet, you can TAB between highlighted modifiable your XML schemas. More information is available on
fields to enter data. Figure 1 shows an example of in- MSDN under, “How to: Generate an XML Snippet
serting a snippet. From an XML Schema”.
You can also write your own snippets. If you want to
create a simple snippet, follow these easy steps: Schema Cache and Schema Catalogs
1. Create a new XML file and type in: For advanced users, XML Editor offers features such
as schema cache and schema catalog files. Schema
cache is a folder that contains a number of well-known
<snippet
W3C schemas, as well as a number of Microsoft-spe-
2. Press ESCAPE to close the IntelliSense win- cific schemas. It serves as a repository of widely used
dow. schemas that are unlikely to change. You’ll find the
3. Press TAB. default location for the schema cache at %vsin-
4. Fill in the blanks. stalldir%\xml\schemas, where “%vsinstalldir%”
5. Press ENTER to finish. is a variable representing the location in which Vi-
sual Studio itself was installed. When you declare one
For more detailed information about creating snip- of the namespaces defined by these schemas in your
pets, I recommend VSEditor’s blog post “Code
Snippet – Schema Description” (http://blogs.msdn.
XML files, XML Editor will automatically associate
appropriate schemas from the cache location and in-
com/vseditor/archive/2004/07/14/183189.aspx). stantly provide you with IntelliSense and validation.
www.code-magazine.com
Schema catalogs are XML files located in the Schema 1. Debugging of standalone
Cache directory (catalog.xml file is the default). They transformation, which is
give advanced users more granular control over vari- useful when your primary
ous namespaces they might want to use. For example, interests are input docu-
you can associate specific namespaces with external ment, transformation itself,
locations: and output document.
2. CLR-integrated debugging,
<Schema which is useful for debug-
href="mylocation/myschema.xsd" ging transformations in the
targetNamespace="http://myschema"/> context of your CLR appli-
cation.
You can also use catalog files to create associations
between schema files and file extensions, which you
can find particularly useful when your schema has Debugging Standalone
no targetNamespace: Transformations
<Association
extension="config" Figure 2 showcases the debug-
schema="xml/schemas/dotNetConfig.xsd"/> ging environment when you
Figure 1: Invoking an XML snippet in Visual Studio 2008.
work with XSLT transforma-
New in Visual Studio 2008, you can also add condi- tions. In the default configura-
tions: tion, you can see the XSLT file and a tabbed view
of the input and output documents. You can also
<Association configure Visual Studio to simultaneously show
extension="config" all three documents. When you step through the
schema="xml/schemas/dotNetConfig30.xsd" transformation code, you can see the input data,
condition="%TargetFrameworkVersion% = 3.0" /> the transformation that handles the input, and the
generation of the output file.
This condition means that the dotNetConfig30.xsd
schema should only be associated when the current If you are used to either the C# or Visual Basic de-
project is targeting .NET Framework version 3.0. bugging environment, you will notice that debugging
XSLT looks very similar to debugging other CLR lan-
Finally, you can create a chain by pointing one cata- guage programs. All the controls and keystroke com-
log file at another: binations are the same. In Figure 2 you can also see
the Locals window, which can show implicit XSLT
<Catalog href="http://mycompany/catalog.xml"/> variables (self, position, and last), as well as all local
and global variables declared in your template. You
can also see the call stack window, which you can use
Working with Large Files to navigate to various XSLT templates up and down
the call stack. You can also use the XSLT debugger to
Another important editing feature I would like to
highlight is the XML Editor’s support for editing
large files. While you could work with large files in Feature Visual Studio 2008 Support
previous versions, Visual Studio 2008 supports in-
Auto-completion X
cremental parsing of the XML documents. Now if
you work with a 10 MB file, you don’t have to wait Syntax coloring X
for Visual Studio to parse the entire file every time Outlining X
you make an edit. The XML Editor will isolate the IntelliSense X
edits and reparse only what’s needed, offering better
Extendable snippet library X
performance and responsiveness.
Dynamic snippets X
I have covered a few interesting features of XML Go to definition (from XML to XSD) X
Editor, but obviously couldn’t go over all of them. Back/Forward navigation X
Table 1 shows some of the other features available
Project and user profile support X
to Visual Studio users.
Real-time validation X
Full XSD Support X
Debugging XSLT Style Sheets Namespace support X
DTD support X
XSLT is a W3C standard transformation language,
which is very popular among a large group of devel- XML-to-XSD generation X
opers. Visual Studio Professional edition provides DTD-to-XSD conversion X
support for both editing and debugging XSLT style Unicode support X
sheets. Editing XSLT files is very similar to editing
Large file support (10 MB+) X
other XML files. When it comes to XSLT debug-
ging, Visual Studio supports two main scenarios: Table 1: Overview of XML Editor features available in Visual Studio 2008.

www.code-magazine.com XML Tools in Visual Studio 2008 77

 Figure 2: Debugging standalone XSLT transformations. Figure 3: Debugging an XSLT transformation from a C# application.

set breakpoints in both your XSLT code and in your Table 2 shows a summary of various features available
input XML documents. when debugging XSLT transformations.

Integrated CLR Debugging XSLT Debugger is tightly

While debugging standalone transformations is use- integrated with other CLR
ful, sometimes you need to debug XSLT as a part of debuggers, which lets you
your C# or Visual Basic application. XSLT Debugger,
which is tightly integrated with other CLR debuggers, seamlessly step from C# to XSLT to
lets you seamlessly step from C# to XSLT to Visual Visual Basic while debugging
Basic while debugging your application. The follow-
ing example shows a C# code snippet that uses the your application.
XslCompiledTransform class to initiate an XSLT
Mutual transformation:
registration Extending XML Tools
XslCompiledTransform xsltcmd =
new XslCompiledTransform(true); XML Editor provides a good foundation for devel-
Events xsltcmd.Load(XSLTFile); opers to build custom designers on top of it. Figure
XmlUrlResolver resolver = new XmlUrlResolver(); 4 shows how it was done in the previous version of
XmlWriter writer = XmlWriter.Create(OutputFile); Visual Studio (Visual Studio 2005). Developers could
build on top of XML Editor by sharing the buffer.
xsltcmd.Transform(XMLFile, writer); They would create their custom designer by sharing
Figure 4: Old way of building IVsTextLines buffer and parsing XML using System.
custom XML designers on top of If you set a breakpoint on the call to Transform Xml.XmlReader or XmlDocument. Don Demsak used
XML Editor. method and step into it, the debugger will take you this approach when he created XPathmania, an XPath
to the XSLT style sheet (Figure add-on to Visual Studio (for more information about
Mutual 3). All of the features described XPathmania, see http://www.donxml.com). However, be-
registration in the standalone debugging sec- cause the integration happened on the buffer level, this
tion are also available here. The method created a lot of large buffer edits. In addition,
only exception is the ability to set it was somewhat limited because System.Xml parser
Events breakpoints in the data files. This consumes only valid XML while designers, by defini-
feature is not available when step- tion, should work with invalid files.
ping into XSLT from another CLR
program. Everything else behaves XML Editor, on the other hand, has its own parser
Figure 5: Visual Studio 2008 way of extending XML Editor. exactly the same. with error recovery. It also builds its own parse tree
representing the contents of the buffer. In Visual Stu-
dio 2008, Microsoft exposed this LINQ to XML tree
Feature Visual Studio 2008 Support to third-party developers. You can see the new archi-
Browser view X tecture in Figure 5. The new API allows developers
Locals, watch, call stack X to create custom views over XML Editor parse tree,
provides them with XML Editor’s error recovery logic,
Viewing input XML, output, X and the LINQ to XML parse tree with a transacted up-
and XLST during debugging date that goes all the way back to the buffer, integrated
Breakpoints in XML X with Visual Studio UndoManager. These changes will
Breakpoints in XSLT X make it easier for developers to build great XML tools
on top of Visual Studio 2008.
CLR language debugger integration X Stan Kitsis
Table 2: Overview of XSLT Debugger features available in Visual Studio 2008.

78 XML Tools in Visual Studio 2008 www.code-magazine.com

Migration Headache?

For immediate relief,

visit
www.VFPConversion.com
today!

VFP Conversion is a migration services brand of EPS

Software Corp., providing expert upgrading of VFP
applications to the latest technologies available from
Microsoft. To learn more about how VFP Conversion can
assist your enterprise, call toll free: 1 (866) 529-3682.
info@VFPConversion.com www.VFPConversion.com
ONLINE QUICK ID 0712172
Chris Lee
Chris.T.Lee@microsoft.com
Chris Lee is a program manager
in the Data Programmability
Runtime Connectivity team
with responsibility for Microsoft
SQL Server Native Client. Prior
to joining Microsoft he worked
for Micro Focus, Merant, Blyth
Software, Inmos and Perkin-
Elmer Data Systems. His career
experience includes work
on operating systems, data
access methods, compilers,
development tools, and database
access.
80 ODBC Rocks!
ODBC Rocks!
Fifteen years after its launch, ODBC is a firmly entrenched
cornerstone of the software industry. This article explains why and will
explore the relationship between Microsoft SQL Server and ODBC and discuss where
ODBC may go in the future.
C
onceived as a broadly based, multi-platform, developers, third-party tools and application packages
multi-database data access technology, ODBC are ADO.NET, JDBC, ODBC and OLE DB. Devel-
has been an outstanding success. Probably the oping drivers for all of these requires significant time,
best known implementation of ISO/IEC 9075-3:2003 effort, and expertise. What is the best strategy if re-
SQL Call Level Interface (part 3 of the complete SQL sources are limited or time to delivery is important?
standard), ODBC is included The answer is simple. Implement
in Windows, MacOS, all major ODBC first and evaluate require-
Linux distributions, and is readily Fast Facts ments for the others afterwards.
available for many Unix versions ODBC drivers are available Why? First, there are bridges
including AIX, HP-UX, Solaris, from all the other APIs to ODBC,
for every major database
and FreeBSD. Even PDAs and so as soon as ODBC is enabled so
including Microsoft SQL
smartphones have ODBC! are all the others. Second, even
Server, Microsoft Access,
ignoring the API bridges, ODBC
Though often thought of as an Oracle, DB2, MySQL, will provide users of the data
API for C and C++ applications, Informix, Ingres, Sybase, source with the widest choice of
ODBC is frequently used with PostgreSQL, Interbase, third-party software since ODBC
other languages. For example, FoxPro, Paradox, Progress, has been around longest and ac-
many COBOL applications use ADABAS, Supra, dBase, cumulated the widest selection of
ODBC for database access as do SQLite, Teradata, Rdb, etc. tools and applications.
dynamic languages such as PHP,
Perl, Python and Ruby along ODBC drivers are available ODBC can, and often is, imple-
with Microsoft Access and other for many other data sources mented on top of a proprietary
RADs. including Excel, Btrieve, API. In the early days of ODBC,
many perceived this to be a weak-
C-ISAM, RMS, text files,
Despite being conceptually lim- ness in first generation ODBC
ited to relational databases, no VSAM, IMS, IDMS, drivers. Research soon showed
significant data source lacks an LINC, 4D, AutoCAD, Pick, that layering ODBC on top of
ODBC driver: text files, Excel FileMaker, QuickBooks, proprietary native APIs usually
spreadsheets, ISAMs such as COBOL EXTFH, SAS, etc. has minimal performance impact.
dBase, Paradox, C-ISAM, Btrieve In some cases ODBC even out-
and VSAM—you name it and an performed proprietary APIs when
ODBC driver is most likely available. For any data the driver adopted strategies to overcome weak default
source that lacks an ODBC driver there are numerous behavior in the underlying native API. For some data-
driver development kits and driver development shops bases, including Microsoft SQL Server, ODBC actually
waiting to help you plug that gap! It goes without say- fulfills the role of proprietary native API.
ing that no respectable relational database lacks an
ODBC driver. Corporate IT and enterprise developers live in a very
heterogeneous environment and have to support an
ODBC is very popular for custom enterprise applica- accumulated legacy spanning multiple languages, op-
tion development and is widely supported by market erating systems, and data sources. There may also be
leading ISVs. ERP, CRM, and SCM packages all use pressures to consolidate and standardize application
ODBC as do query, analysis, reporting and ETL pack- platforms alongside pressure to modernize existing
ages as well as productivity applications such as Mi- applications, re-engineer business processes, increase
crosoft Office. information integration, and introduce business intel-
ligence. Phew! How to juggle these pressures and stay
So what makes ODBC popular with different industry sane?
factions and why will ODBC remain popular for the
foreseeable future? In data integration and consolidation scenarios from
ETL to BI, ODBC’s ubiquity is invaluable in bringing
For the data source owner ODBC is a must. The data from both packaged and custom applications to-
industry standard APIs required to enable access by gether.
www.code-magazine.com
The .NET platform is compelling where new develop-
ment is required, but in many cases timescale, busi-
ness risk, and overall cost factors favor code re-use
and incremental modernization. ODBC has a lot to
offer here. Often application developers can recom-
pile existing code for use in .NET applications with the
Microsoft Visual Studio 2005 C++ compiler using the
/clr compiler switch. This greatly simplifies mixing ex-
isting C++ and ODBC business logic with a modern-
ized user interface, which can be developed most ef-
ficiently using the .NET Framework. ODBC provides
a common API for a diverse spectrum of data sources
and operating systems and can access multiple data
sources simultaneously. These are essential capabili-
ties for information consolidation, aggregation, and
integration whether used in native applications or via
the .NET Framework Data Provider for ODBC.
In addition to directly reusing existing business logic,
ODBC is also a prime candidate to accelerate con-
version from other proprietary APIs. Consider license
consolidation via database migration, where it may be
more cost effective to convert applications that use
proprietary APIs such as DB-Library, CT-Library and
OCI to ODBC rather than to rewrite completely for
.NET since ODBC is conceptually quite similar to
other native code APIs.
ISVs often choose to support multiple databases to
increase addressable market and customer appeal.
ODBC offers several approaches to this requirement.
Above all, ODBC offers a common API that can ad-
dress multiple data sources and, as seen above, a wide
range of available drivers. However, there are differ-
ences among data sources and this is reflected in a
range of behaviors across different ODBC drivers.
The simplest approach to solving this problem uses a
combination of the most restrictive behavior patterns
and SQL dialect subset. ODBC offers escape sequenc-
es that iron out minor differences among SQL dialects
to assist in this approach. Next, if this is insufficient,
applications can query ODBC to determine the char-
acteristics of the particular driver and database in use
and adapt dynamically to them. An application can
also determine the actual driver and database in use
and trade off generality to exploit unique characteris-
tics in meeting the most stringent functional and per-
formance demands. ODBC places no restrictions on
the statements sent from applications to data sources,
so does not suppress the richness of the SQL dialect
on a connection.
Some ISVs use an internal data abstraction layer to
enable use of different APIs for different databases.
ODBC has much to offer, even in this scenario. For
some databases, notably Microsoft SQL Server,
ODBC is the best performing API available.
System integrators and value-added resellers (VARs)
benefit from ODBC in two ways. They can guarantee
that whatever the operating platform and other in-
frastructure, ODBC will be available for all the data
sources they will encounter, which simplifies devel-
opment of a specialist proprietary toolset. Secondly,
staff familiar with ODBC can be deployed to satisfy
www.code-magazine.com
a wider range of customer situations than staff whose
skill sets span a narrower spectrum.
ODBC and Microsoft SQL Server
ODBC has broad ongoing appeal across the software
industry and remains a key, though often unsung, ele-
ment of the Microsoft Data Platform. Let me now dis-
cuss the relationship between ODBC and Microsoft
SQL Server in more detail.
When first launched, DB-Library was SQL Server’s
only API for client applications. Microsoft later sup-
plemented the SQL Server API with ODBC then with
OLE DB and most recently with ADO.NET. DB-Li-
brary has been deprecated due to technical limitations
and is today only supported to provide backwards
compatibility to legacy applications that have not yet
converted to one of the other APIs.
At the time of its release, the SQL Server team at Mi-
crosoft believed OLE DB would supersede ODBC.
This is no longer the case and ODBC’s future is com- At the time
pletely secure. ODBC is more widely used than OLE
DB and it is better suited to some key scenarios that I of its release,
will discuss later in this article. experts at
Both OLE DB and ODBC are true native APIs for Microsoft believed
SQL Server in that they map API calls directly into OLE DB would
SQL Server’s network protocol, Tabular Data Stream
(TDS). When Microsoft recommended best practic- supersede ODBC.
es are followed, ODBC is a very thin wrapper over This is no longer
TDS with no intermediate buffering between network
packet buffers and the application. It therefore has ex- the case …
cellent performance and scalability characteristics.
ODBC and OLE DB support for Microsoft SQL Serv-
er is available in WDAC (Windows Data Access Com-
ponents), originally known as MDAC (Microsoft Data
Access Components), and in Microsoft SQL Server
Native Client, a component of Microsoft SQL Server
2005 and later versions. Support in WDAC targets
legacy and generic applications that do not exploit
the unique features of Microsoft SQL Server 2005
and later versions. Applications that do wish to fully
exploit the unique features of Microsoft SQL Server
2005 or later should instead use Microsoft SQL Serv-
er Native Client, which enables use of features such as
snapshot isolation, database mirroring, query notifi-
cation and data types such as xml and varchar(max).
Microsoft will continue to add support for new SQL
Server features to all of the APIs it supports: ADO.
NET (via SqlClient), ODBC and OLE DB (via SQL
Server Native Client) and JDBC.
Ongoing Roles for ODBC
I’ll now examine some key scenarios where ODBC will
play an important role in SQL Server’s future, starting
with data integration and business intelligence. Some
of the SQL Server components involved here rely on
OLE DB internally and reflect this in their external
interfaces. In recognition of the popularity of ODBC,
Microsoft has decided to continue support for the Mi-
ODBC Rocks! 81
 ODBC Overview
ODBC (which stands for Open
DataBase Connectivity) was
designed to be a common Call
Level Interface (CLI) for relational
databases. ODBC 1.0 was based
on work by the SQL Access Group
and released in September 1992,
following work by Microsoft and
Simba Technologies. ODBC 2.0
was an impoved and extended
version of ODBC 1.0. Microsoft
aligned ODBC 3.0 with the X/
Open and ISO CLI specification
that was adopted as part three
of the overall SQL standard.
Subsequently Microsoft added
Unicode and 64-bit support to
ODBC and the current version is
3.52.
At one time Microsoft considered
extending ODBC to allow it to
be used with other data sources
but the consensus among users
was that ODBC should remain
focussed on relational databases
and so Microsoft designed OLE
DB to fulfil the universal data
access role.
Easysoft and OpenLink Software
developed ODBC driver managers
for non-Windows platforms based
on Microsoft’s specifications
for ODBC on Windows, and
these became the open source
unixODBC and iODBC projects.
A wide range of ODBC drivers are
available for use with these driver
managers.
Data Direct Technologies also
supplies a wide range of ODBC
drivers for both Windows and
non-Windows platforms.
Although conceptually limited
to relational databases, drivers
have been developed for a wide
range of non-relational data
sources. Today ODBC remains
one of the most widely used data
access APIs with broadly based
support from database vendors,
ISVs, and third-party connectivity
specialists.
82 ODBC Rocks!
crosoft OLE DB Provider for ODBC in both 32- and
64-bit versions for Windows Server 2003, Windows
Vista, and Windows Server 2008 to ensure that ODBC
can be used when it is the best choice available.
Dynamic languages such as PHP, Perl, Python and
Ruby account for a significant amount of developer
activity. Some of these are supported on the .NET
platform but are also popular on other platforms. All
of these languages can access Microsoft SQL Server
via ODBC, though the libraries available to achieve
this vary in quality and performance. Microsoft will
continuously review the need for libraries optimized
for Microsoft SQL Server and is developing a PHP ex-
tension library based on SQL Server Native Client’s
ODBC driver.
Business interest in application migration and server
consolidation as a means of reducing operational and
license costs is increasing. SQL Server’s excellent to-
tal cost of ownership profile makes it very attractive
for this role. ODBC has significant roles to play here
in both data consolidation and application migration,
as described earlier. Microsoft SQL Server Migration
Assistant helps customers migrate schema and the da-
tabase itself to Microsoft SQL Server and Microsoft is
working on documentation and tools to help migrate
application source code to use ODBC and Transact
SQL in the future.
Non-Windows clients and mid-tier servers add an
extra dimension to the scenarios described above.
Currently third-party ODBC drivers meet customer
requirements, but Microsoft continuously evaluates
providing its own drivers to ensure customers are get-
ting the best experience.
Last but not least, Microsoft wants existing customers
to be even more successful in the future. This involves
maximizing their return on existing code assets by ap-
plication modernization and code reuse. If existing
business logic continues to meet business needs then it
makes more sense to reuse it than to rewrite it. For ex-
ample, approvals and forecasting in financial applica-
tions use the same algorithms regardless of application
architecture. The ability to mine routines from batch
and call center applications and re-deploy in SOAs,
where they then become available for B2B and self-ser-
vice consumer scenarios, makes sound business sense.
Naturally, .NET is a strong contender for developing
the new communication and UI elements of the mod-
ernized application, but core C++ and ODBC-based
business logic and data access routines can often be re-
used with very little modification and at very low cost.
Visual Studio and C++ CLI provide excellent re-use
and interoperability capabilities and there is an oppor-
tunity to extend this further in the future by providing
increasingly more sophisticated re-factoring tools.
What Do the Next 15 Years Hold in
Store for ODBC?
There are two key and inter-related questions to an-
swer here. First, how will ODBC continue to adapt
to the changing capabilities of SQL Server in its role
of principal native API? Second, how will ODBC
evolve in its role as an industry standard cross-plat-
form, cross-database, API?
ODBC defines areas where drivers may add their
own driver-specific extensions above and beyond the
core ODBC specification, and Microsoft SQL Server
Native Client uses this capability to add support for
new features added to Microsoft SQL Server. Other
drivers have also added their own extensions for
this purpose. The following two examples will dem-
onstrate how Microsoft SQL Server Native Client
adapts to SQL Server enhancements.
SQL Server 2008 adds additional date/time data
types to supplement the existing datetime and
smalldatetime types: date and datetime2 corre-
spond to ODBC’s existing SQL_TYPE_DATE and
SQL_TYPE_TIMESTAMP types; time (time with 0
to 7 digits fractional seconds scale) and timestam-
poffset (effectively datetime2 plus a timezone offset)
do not match any existing ODBC data types and so
new types must be added to support them. These are
SQL_SS_TIME2 and SQL_SS_TIMESTAMPOFF-
SET.
In SQL Server 2008, a table-valued parameter (TVP)
is a parameter to a T-SQL statement or stored proce-
dure that can consist of multiple rows and columns.
For example, consider the following statement:
Insert into OrderItems (OrdID, ProdCode, Qty)
Select ?, ProdCode, Qty from ?
The first parameter is the primary key of a newly in-
serted Order and the second is a TVP that has a row
for each order item with product code and quantity
values. The TVP allows multiple OrderItem rows to
be inserted in a single statement from a single param-
eter. TVPs add performance by reducing round trips
between client and server and by enabling optimi-
zations in TDS and the relational engine. They also
enhance encapsulation by enabling a single stored
procedure to perform a complete business trans-
action, with exceptional code clarity, by relaxing
normal “rectangular” restrictions on parameter sets
to stored procedures. Note this example of the lat-
ter:
create type OrdItemType as table(
ProdCode integer, Qty integer)
create procedure OrderEntry
(
@CustCode varchar(5),
@Items OrdItemType READONLY,
@OrdNo integer output,
@OrdDate datetime output)
as
set @OrdDate = GETDATE();
insert into Orders (OrdDate, CustCode)
values (@OrdDate, @CustCode);
select @OrdNo = SCOPE_IDENTITY();
insert into TVPItem (OrdNo, ProdCode, Qty)
select @OrdNo, ProdCode, Qty from @Items
www.code-magazine.com
Without a TVP you would have to split this into two
stored procedures, one for each table.
Since you can mix table-valued and traditional single-
valued parameters in a statement, SQL Server Native
Client must provide a way to handle what amounts to
a nested table when binding ODBC parameters. This
is achieved by first binding a TVP as type SQL_SS_
TABLE, a new type representing a nested table. Then
a statement attribute SQL_SOPT_SS_PARAM_FO-
CUS is set to the parameter ordinal of the TVP. This
directs subsequent calls to SQLBindParameter to
columns of the TVP (rather than the “top-level” pa-
rameter set) to define the nested row structure. What
seems at first thought to be a complex and difficult
extension to ODBC turns out to be quite simple in
practice and the resulting application code is very
compact and readable.
The complete TVP implementation in SQL Server
Native Client builds naturally on existing ODBC con-
cepts of array binding and “data at execution” param-
eter values to allow rows of a TVP to be supplied at
runtime either in an in-memory array or as batches
of one or more rows that are streamed to the server
when the statement is executed.
So far, so good. SQL Server Native Client shows that
an ODBC driver can add significant functionality
without requiring any changes to the ODBC specifica-
tion. Although, what happens if and when a new SQL
Server feature goes beyond what ODBC extensibility
allows? Are there new SQL Server application sce-
narios where ODBC’s architecture is less than ideal?
These are tricky questions, to which possible solutions
might be: (1) update the core ODBC specification, or
(2) turn SQL Server Native Client into a native API
that can work without an ODBC Driver Manager and
so is no longer constrained by the ODBC specifica-
tion. Each of these paths have implications.
Changing the ODBC specification is not entirely risk
free. One of ODBC’s greatest virtues is its stability.
Change the specification and existing drivers and ap-
plications may start behaving in new and unexpected
ways. Branching away from the current specification
may satisfy some scenarios but abandon others. Is
there a utopian “third option” for SQL Server Na-
tive Client and ODBC? IBM’s call level interface for
DB2 can operate in dual modes as ODBC driver and
extended call level interface. This could be an evo-
lutionary path for SQL Server Native Client and/or
ODBC. Another alternative may be to enable current
and future versions of ODBC to co-exist side-by-side
in some way. The current ODBC Driver Manager al-
ready achieves this by supporting the slightly different
behaviors of ODBC 2 and ODBC 3 side-by-side.
Are other organizations interested in extending ODBC
beyond the current core specification? This is an area
where Microsoft would like to encourage discussion
among interested parties and assess the potential for a
broadly based consensus. Some form of community-
based evolution might be able to react more quickly to
a wider range of input than a full-blown ISO or ANSI
process. You can rest assured of one certainty, what-
www.code-magazine.com
ever happens; nothing will be allowed to undermine
ODBC’s rich legacy and enduring value throughout
the industry.
Microsoft regards the Entity Data Model (EDM) as
a major generational step change across the whole
software industry, applicable well beyond the cur-
rent ADO.NET Entity Framework. Thinking how
ODBC and the EDM might come together provides
an additional factor in ODBC’s future. ODBC could
be extended to handle queries in Entity SQL return-
ing entity-shaped results with polymorphic rows and
nested entity values. The experience with non-rectan-
gular data for table-valued parameters suggests that
handling this via driver-specific extensions in SQL
Server Native Client would be feasible. However, Mi-
crosoft is enabling the EDM for a broad range of data
sources, not just SQL Server. EDM support will be
available for other relational database products via
the ADO.NET Entity Framework, so why not do the
same by extending ODBC’s core specification to sup-
port the EDM? In part the answer depends on the
level of adoption EDM would see in ODBC applica-
tions, and the nature of the demand: should EDM and
relational access use separate drivers, a common driv-
er but separate connections or co-exist on the same
connection? Would existing code adopt EDM access
incrementally or is the EDM attractive only in new
code? This is another area where Microsoft would be
interested in hearing the views of interested parties.
ODBC has come a long way and hopefully this article
has given you new insights into the breadth and depth
of what it has and will continue to offer for many
years to come; how it might evolve in the future; and
of the ongoing long term commitment Microsoft has
made to it.
To comment on the points raised in this article, please
send e-mail to odbcfut@microsoft.com.
You can find further information about ODBC in the
following references:
1. Mike Pizzo’s very entertaining history of Mi-
crosoft’s data access APIs at http://blogs.msdn.
com/data/archive/2006/12/05/data-access-api-of-
the-day-part-i.aspx
2. The Wikipedia ODBC page at http://en.wikipedia.
org/wiki/Open_Database_Connectivity
3. Ken North’s article on SQL and ODBC in in-
tegration frameworks for Business Integration
Journal at http://www.sqlsummit.com/PDF/BIJ_
North_Nov2004.pdf
4. Ken North’s ODBC portal at http://www.sqlsum-
mit.com/ODBCPORT.HTM
5. Online ODBC documentation at http://msdn2.
microsoft.com/en-us/library/ms710252.aspx
6. Online information about Microsoft SQL Server
Native Client at http://msdn2.microsoft.com/en-
us/data/aa937705.aspx
7. The SQL Native Client Blog at http://blogs.msdn.
com/sqlnativeclient/
Chris Lee
ODBC Rocks! 83
ONLINE QUICK ID 0712182
Brian Beckman
bbeckman@microsoft.com
Brian Beckman has worked
on many projects at Microsoft
over the last fifteen years from
Research to Crypto to BizTalk
to games to LINQ. Before
Microsoft, Brian worked at the
Jet Propulsion Laboratory on
research operating systems and
radio astronomy. He holds a PhD
in Astrophysics from Princeton
University.
84 TESLA: Democratizing the Cloud
TESLA: Democratizing the
Cloud
In our service-oriented world, users need the same experience on
any device, whether mobile phone, office PC, or Internet café.
Moreover, they want the same experience any time they access applications, offline
or online. For developers, this means tackling multi-tier, distributed, and concurrent
programming. LINQ 1.0 radically simplified multi-tier programming with unified
query and deep XML support. TESLA is a broad engineering program by the authors
to extend the success of LINQ with external relationships, reshaping combinators,
assisted tier-splitting, and join patterns.
LINQ 1.0: Standard Query Operators and composable—while the number of basic operators
Deep XML Support is small, the number of combinations is unlimited.
That way, one small set of primitives covers all sce-
narios. Furthermore, SQOs can just as easily trans-
It seems that developers spend the majority of form data from one data model to another, say from
development hours and com- XML to objects, or entities to
puter cycles just plumbing: Fast Facts JSON, or relations to XML and
reshaping, reformatting, wrap- back, etc., in a regularized way.
ping, connecting, serializing, TESLA will bring much
and marshaling from one data greater expressive power and This is classic engineering by
model to another just to inter- robustness to distributed data separation of concerns. Separate
face with supplied technologies. applications by integrating the data models from the query
Most developers would rather monadic comprehensions, languages, look for common se-
spend their time and energy mantics amongst the query lan-
composable mapping,
on presentation, business logic, guages, and capture them in a
data analysis, and data mining:
assisted tier-splitting, and higher-level definition that each
the parts of applications that join patterns into mainstream data model can implement in its
generate value. .NET programming own natural way.
languages.
To mitigate this situation, some In current practice, every data
academics and industry experts model comes with its own id-
recommend picking one of the data models—XML, iosyncratic, tightly coupled query language. XML
JSON, objects, entities, or relations—as the univer- comes with XQuery; objects require hand-written
sal one and mapping all the other data models into query logic; and the data tier has SQL. Consider
it. Other experts prefer to invent grandiose new the following three ways to write the same example
data models that purport to encompass all the other query. In XQuery, on the client, one writes:
data models.
from $c in customers/customer
where $c/city == "Seattle"
One Query Language, Many Data return <result>{$c/Name}{$c/Age}</result>
Models
On the middle tier, one would write in C#:
We (the authors of this paper) do not believe in ei- class Result { string Name; int Age; }
ther of these “universalist” approaches. The allure ...
of a unified data model is naïve in practice. The pro- var r = new List<Result>();
posals, especially with their mapping solutions, are foreach(var c in Customers)
at least as complicated as the jumbles they contend if(c.City == "Seattle")
to replace and their learning curves just add more r.Add(new Result{c.Name, c.Age});
developer cost.
And, on the data tier, one would write SQL:
It is preferable to unify at the level of operations,
that is, to define a small, universal set of Standard Select Name, Age
Query Operators (SQOs) that work the same way From Customers
on all data models. These operators are limitlessly Where City = "Seattle"
www.code-magazine.com
In each case, the meaning of the query is the same:
“Give me the collection of name, age pairs from
customer records such that the customer’s city is Se-
attle.”
It is intolerable to impose the burden of managing such
superficial and gratuitous differences on the developer.
It doesn’t matter whether, under the covers, Customers
are objects, rows, XML elements, or any other kind of
collection, and developers should insist on writing the
same query the same way. This is exactly what LINQ 1.0
query comprehension syntax does. All three queries look
the same in C# 3.0:
from c in Customers
where c.City == "Seattle"
select new { c.Name, c.Age }
Visual Basic 9 has a very similar syntax. The com-
piler simply translates comprehension syntax into
nested calls of the SQOs, and the SQOs, in turn,
call methods in the interface IEnumerable. Only
the implementations of the various IEnumerables
for the various types of Customers know whether
to scan elements, iterate over objects, or join over
tables.
Mathematicians recognized this “design pattern”
decades ago. Each data model defines a monad
and the query is a monad comprehension. Don’t
let the terminology intimidate; monads are very
simple structures and monad comprehensions are
just like the “set” comprehensions you learned in
school, for example, “the set of all integers x such
that x is divisible by 5.” Just as sets work the same
whether they contain integers, apples, or even oth-
er sets, so monads work the same whether they
contain objects, elements, or rows. IEnumerable
captures monadic “collection-ness” and hides the
differences. Any class that implements IEnumer-
able automatically supports the SQOs.
Operators and Syntax
The following six SQOs are the core of LINQ 1.0.
These are precisely the monadic primitives for fil-
tering, transforming, joining, grouping, and aggre-
gating over arbitrary collections of arbitrary types
(abbreviating IEnumerable as IEn):
static class Sequence
{ static IEn<T> Where<T>
(this IEn<T>, Func<T, bool>) { }
static IEn<S> Select<T, S>
(this IEn<T>, Func<T, S>) { }
static IEn<S> SelectMany<T, S>
(this IEn<T>, Func<T, IEn<S>>) { }
static IEn<V> Join<T, U, K, V>
(this IEn<T>, IEn <U>,
Func<T, K>, Func<U, K>, Func<T, U, V>) { }
static IOrderedSequence<T> OrderBy<T, K>
(this IEn<T>, Func<T, K>) { }
static IEn<IGrouping<K, T>> GroupBy<T, K>
(this IEn<T>, Func<T, K>) { } }
www.code-magazine.com
Consider the first one, Where<T>, with T repre-
senting a type such as Int or String or Customer.
Where takes an IEnumerable<T> and a Func<T,
bool>. This Func is a predicate function that takes
a T and returns a Boolean. Where returns a new
IEnumerable<T> that contains only the elements of
its input IEnumerable<T> that satisfy the predicate
function.
Read the rest of the operator definitions in a similar
fashion and look up details and examples in any
number of places on the Web such as http://www.
codeplex.com/LINQSQO and http://www.hookedonlinq.
com. There are a few more convenience operators
in the SQO specification, but these six are the ker-
nel set.
Erik Meijer
All the SQOs are static methods that take any erik.meijer@microsoft.com
IEnumerable<T> as their first argument and re- Before joining Microsoft, Erik was
turn an IEnumerable of another potentially differ- associate professor at Utrecht
ent type S. Because they are static methods with University where he worked
no implicit this parameter, you can write SQOs for on languages such as Haskell,
a class without changing or even recompiling the XMLambda, and Mondrian
class. This critical design choice ensures modularity and directed the Microsoft lab
(see sidebar, “Modularity Saves Money”). Further- (now defunct). He is currently
more, since every SQO just consumes and produces
working on .NET language and
IEnumerables, you may compose and nest SQOs in
type-system support for bridging
an unlimited variety of ways. This is another criti-
object-oriented (CLR), relational
cal design choice (see sidebar, Composability Saves
(SQL), and hierarchical (XML)
Money). TESLA carries forward these design prin-
data with first-class functions.
ciples through external relationships and reshaping
combinators. I’ll discuss more about them below.
He is a regular contributor to
Lambda-The-Ultimate.org.
It is preferable to unify
at the level of operations,
that is, to define a small,
universal set of Standard Query
Operators (SQOs)
that work the same way on
all data models.
History of Monad
Deep XML in Visual Basic 9 Comprehensions
The functional language Haskell
Sometimes you are working with a single data was the first to introduce monad
model and would like detail-control leverage from comprehensions for universal
the programming language. Therefore, in LINQ 1.0, query (http://www.haskell.org).
Microsoft also built deep XML integration into Vi- Haskell requires all mutable data
sual Basic 9. If you are working directly with XML to be in monads. Instead of, say,
documents, you can write the query as follows: tacking characters onto the end
of a string by side effect, as you
From C in Customers.<customer> do in ordinary languages, string-
Where C.@City = "Seattle" building in Haskell is a monad
Select <result> that yields new strings with the
<%= C.<Name> %> needed characters every time
<%= C.<Phone> %> you run it. The monadic approach
</result> gives Haskell uniquely high
clarity and hygiene, making it the
The compiler just translates such queries into ex- “programming tool of choice for
pressions over the SQOs, which, in turn, are over discriminating hackers”
the LINQ-to-XML provider. (http://icfpc.plt-scheme.org/).
TESLA: Democratizing the Cloud 85

LINQ 1.0 solved the impedance
mismatch problem by reframing
it from the intractable one of data
models to the tractable one of
query languages.
LINQ 1.0 Summary
LINQ 1.0 solved the impedance mismatch prob-
lem by reframing it from the intractable one of data
models to the tractable one of query languages.
LINQ 1.0 also dramatically simplified the work of
XML users.
TESLA: Reaching Across the Cloud in
Space and Time
Much work remains to achieve the larger goal of
democratizing the Cloud: making distributed, data-
intensive programming easy enough for everyone.
The next big issues are object-relational mapping,
distributed programming, and concurrent program-
ming. This article’s solutions for these issues are
external relationships, reshaping combinators, as-
sisted tier-splitting, and join patterns.
Co-Dependency and Non-
Composability
The “last mile” of data programming, mapping one
data model to another, remains painful. In par-
ticular, the mapping between relational data and
objects is very thorny. Typically, one finds code
generators that read relational schemata and write
class definitions (Figure 1) or one finds external
mapping-specification files, perhaps in XML. At
first glance, these would seem to satisfy goodness
criteria like eliminating hand-written code or sepa-
rating the concerns of mapping and data represen-
tation. However, the cost with both approaches
is complexity. If you can configure the code gen-
erators at all, they entail a configuration language
more complex than either the relational model or
the object model. Similarly, the external mapping
languages can be much more complex than either
side. Some will argue that their complexity is un-
avoidable, because mapping is dealing with the dif-
ficult problem of impedance mismatch. To be rich
enough for real-world scenarios, they must support
the Cartesian product of features from both sides.
This article argues, however, that since their com-
plexity admittedly arises from the beginning with
the union of the data models, everything from both
sides, you can eliminate it by beginning with the
intersection, the parts the two sides have in com-
mon. To build up richness, you can resort to the
proven technique of composability: define a small
number of things that fit together in an unlimited
variety of ways.
External Relationships Cure
Co-Dependency
Because you want ordinary object-programming
dot notation to traverse primary-key and foreign-
key relationships between classes in both direc-
tions, typical mapping solutions interpolate ex-
plicit properties into class definitions. See Listing
1 for a typical example. These properties introduce
tight, bi-directional, internal coupling between the
Customer and Order classes. Changing one may re-
quire changing the other, particularly if the primary
or foreign key specifications change. Even worse,
introducing a new relationship, say between Cus-
tomer and PostalAddress, requires changing the
Customer class, with changes possibly cascading
to other classes.
In the database, coupling is usually only one way.
Relational databases encode one-to-many relation-

Listing 1: LINQ to SQL and monolithic mapping

[Table(Name = "Customers")] public class Order
public class Customer {
{ [Column(IsPrimaryKey = true)]
[Column(IsPrimaryKey = true)] public int OrderID;
public string CustomerID;
... [Column(IsForeignKey = true)]
private EntitySet<Order> _Orders; public string CustomerID;
[Association(Storage = "_Orders", OtherKey = "CustomerID")] ...
public EntitySet<Order> Orders private EntityRef<Customer> _Customer;
{ [Association(Storage = "_Customer", ThisKey = "CustomerID")]
get { return this._Orders; } public Customer Customer
set { this._Orders.Assign(value); } {
} get { return this._Customer.Entity; }
} set { this._Customer.Entity = value; }
}
[Table(Name = "Orders")] }

86 TESLA: Democratizing the Cloud www.code-magazine.com

ships upstream from the target table (Orders) to the
source table (Customers) by foreign key. However,
once you map to objects, you get bi-directional
coupling. Even if you only want one navigation
direction, the most natural one is the other way,
downstream from the source table (Customers) to
the target table (Orders), ignoring foreign keys. In
other words, the natural object solution defines a
property in Customer referencing the collection of
Orders, but this property has no overt reflection in
the store.
Recall that the SQOs, as static methods, avoid gra-
tuitous tight coupling. Perhaps you can do some-
thing similar here and—just as in LINQ 1.0—intro-
duce convenient syntactic sugar. This is exactly
what the authors have done, working with our
colleague, Kasper Osterbye of IT University at Co-
penhagen.
Developers should have notation for declaring re-
lationships as properties externally to the class def-
initions, hence to have ordinary dot notation for
accessing properties bidirectionally, but without
modifying or even recompiling the original class
definitions. Customers and Orders do not need
to know exactly how they are related or even if
they are related. All that matters is that application
code can get from a customer to its order and from
an order to its customer.
Dot notation naturally chains, so the properties
remain composable. Internally, the relationship
www.code-magazine.com
machinery must honor the primary-key and for-
eign-key constraints from the relational model, but
will yield much more stable and decoupled class
definitions at the object level. While it is too ear-
ly to propose the precise syntax, it will resemble
Listing 2.
Reshaping Combinators Cure
Non-Composability
Since existing mapping solutions go all the way
between tables and classes, and never tables-to-
tables or classes-to-classes, it is impossible to build
up mappings incrementally by composing smaller,
more understandable mappings in sequence. You
must make all mapping decisions at one time, dra-
matically increasing the cost (see sidebars). Con-
trast that with the composability of traditional
database views, which you can build up block-by-
block because combining one view with another
yields a new view.
To cure mapping non-composability, we define a
small set of reshaping combinators. Like the SQOs,
these are static methods that take a reshaping and
return a reshaping, and which can combine in near-
ly arbitrary ways for an unlimited variety of solu-
tions. The authors worked with our colleague, Mark
Shields, to determine a small set that suffices for all
practical scenarios. While it is much too early to pro-
pose specifics, a rough explanation means that you
TESLA: Democratizing the Cloud 87
 Listing 2: External relationships sketch
public class Customer }
{
... // Notice no explicit reference to Orders public extension Order
} {
private EntityRef<Customer> _Customer;
public class Order
{ [Association(Storage="_Customer", ThisKey="CustomerID")]
... // No need to link to Customer data here, either public Customer Customer
} {
get { return this._Customer.Entity; }
public extension Customer set { this._Customer.Entity = value; }
{ }
private EntitySet<Order> _Orders; }

[Association(Storage="_Orders", OtherKey="CustomerID")] ... // This shows how to use the induced properties
public EntitySet<Order> Orders foreach (Order o in c.Orders)
{ {
get { return this._Orders; } var a = o.Customer.Address;
set { this._Orders.Assign(value); } // can be chained ... they’re COMPOSABLE
} }

Listing 3: Sample of join patterns

Class OnePlaceBuffer(Of T) Function __Take_And_Contains(ByVal t As T) _
As T
Public Synchronous Put(t As T) When Take, __Contains
Public Synchronous Take() As T Me.__Empty()
Asynchronous __Empty() Return t
Asynchronous __Contains(t As T) End Function

Sub __Put_And_Empty(ByVal t As T) Public Sub New()

When Put, _Empty Me.__Empty()
Me.__Contains(t) End Sub
End Sub
End Class

work in a simplified monad of tuples and pose an Mapping Solved

effective collection of operations like the following:
• Adding, dropping, and renaming fields
• Nesting and un-nesting tuples
• Generalizing and specializing field types
• Scalar transforms
• Arithmetic on field data
With external relationships and reshaping com-
binators, you only need a very thin layer of non-
programmable default mapping at the edges be-
tween the relational world and the object world.
Developers will express everything else in their
favorite LINQ-enabled programming languag-
es.

Just as LINQ 1.0 solved impedance mismatch by

Just as LINQ 1.0 solved turning it into a solved problem (monad com-
impedance mismatch by turning prehensions), TESLA will solve the mapping
problem by turning it into something solvable:
it into a solved problem external relationships and reshaping combina-
(monad comprehensions), tors.
TESLA will solve the mapping
problem by turning it into The Cloud Reaches Across Space
something solvable: and Time
external relationships and
reshaping combinators. The other open problems we will attack in TESLA
are distribution and concurrency, the new applica-
tion design patterns for the Cloud.

88 TESLA: Democratizing the Cloud www.code-magazine.com

In an increasingly distributed and service-oriented
world, users demand the same experience every-
where they may be, using any device, and at any
time, connected or not. For developers, this means
tackling distributed programming (multiple places)
and concurrent programming (multiple times). How
do you get there from the familiar world of one-ma-
chine, step-by-step applications?
Assisted Tier-Splitting
For the space dimension, how can you get, with
minimal agony, from a program that runs on one
machine to a program that runs on two machines,
and from there, to any number of machines? You
want to avoid maintaining different application
builds for each client platform. You would even
like to delay assigning functionality to client and
server until the very last responsible moment,
when you know the capabilities of the client de-
vice.
var o = new ListBox();
//... set up UI ...
i.KeyUp += delegate {
o.Items = Format(d.Suggest(i.Text, 10));
You want to test and tweak this application purely
on the client tier until satisfied with its functional
behavior. To deploy the Dictionary class as a ser-
vice on the middle-tier, just add a custom attribute
[RunAt(Server="...")] to the class declaration:
[RunAt(Server="http://foo.baz.qux/...")]
class Dictionary {...}
Guided by this metadata, the compiler rewrites
the MSIL of the client-side Dictionary class with a
proxy that calls into a server-side stub to the origi-
nal implementation. Client and server communi-
cate via a stateful RPC-style protocol using JSON,
XML, or S-expressions as the serialization format.
The run-time infrastructure picks the format based
on additional metadata in the custom attribute.

We want to start with an ordinary, single-tier ap-

plication, and then successively refactor the code
into a distributed application. To be clear, auto-
matic repartitioning is bad. However, there is plen-
ty of opportunity just for good tooling for program-
mer-specified repartitioning.
Tier-splitting refactoring honors the converse
statement of the expansion theorem from process
algebra. In its forward direction, the theorem de-
scribes the parallel composition of two programs
as an arbitrary interleaving of a single sequential
program. We go the other way, describing a se-
quential program as a parallel composition of two
programs, ensuring that the parallel execution
of the new programs maintains invariants of the
original sequential program. The transformation
inserts synchronization, marshaling, and security
logic using join patterns, described below.
To see how this works in practice, imagine a tiny
“dictionary-suggest” application that proposes
completions when a user types part of a word. The
Suggest method in class Dictionary returns the
topmost n descriptions that match a given prefix
p. As a single-tier, sequential program, this is a
straightforward LINQ query over a collection of
dictionary entries:
IEnumerable<Entry> Suggest(string p, int n)
{ var o = new ListBox();
return (from e in this.entries
where p.Length > 0 && p.Matches(e)
select e).Take(n);
} entries => {o.Items = Format(entries); });
By the converse of the
expansion theorem, parallel
execution of the new programs
maintains invariants
of the original sequential program.
As with all distributed applications, you must worry
about network latency and availability. All remote
calls should be asynchronous. Let the compiler
write a new overload for the Suggest method with
an [Async] custom attribute:
[Async]
extern void Suggest(string p, int n,
Action<IEnumerable<Entry>> cont);
This new overload takes an additional parameter, a
continuation delegate of type Action<...>. This con-
tinuation takes as input the IEnumerable<Entry>
that was the output of the old, synchronous Suggest.
This is a completely automatable rewrite into con-
tinuation-passing style:
...
i.KeyUp += delegate {
d.Suggest(i.Text, 10,
Modularity Saves
Money
“Modularity” dictates that
you reduce co-dependencies
amongst parts of a system. This
reduces the number of changes
you must make at one time when
you update a system. That saves
money because cost goes up
very quickly with the number
of changes made at one time.
The reason is just entropy: the
number of ways of making a
mistake grows exponentially with
the number of changes, while the
number of ways of getting it right
does not grow. Modularity is a
design dogma.

The client program, at each keystroke, takes the

value of the textbox i, looks up the completions of
the word in the dictionary, and populates listbox o
with the first 10 results, properly formatted:
var d = new Dictionary();
var i = new TextBox();
Don’t Stop at Just Two Tiers
With our colleague Dragos Manolescu, we are pur-
suing much more broadly distributed real-world
applications. This means continuing tier-splitting
recursively beyond client and server until all com-

www.code-magazine.com TESLA: Democratizing the Cloud 89

 putation happens close to its data and resources.
Let’s move the dictionary query into the data tier
and, perhaps, split the client tier to run UI in Sil-
verlight, but leave the Dictionary proxy class in the
HTML DOM.
On all tiers, we do not assume there is a native
.NET runtime. Instead, we target any available run-
time, adopting the point of view of a pluggable com-
piler back end. In the near term, the minimal client
will be a browser with JavaScript. Therefore, we
translate IL into JavaScript with very high fidelity,
by deep embedding. Similarly, in the near term, the
typical server will be a cluster of commodity-hard-
ware blades. Therefore, we have a version of LINQ
where GroupBy aggressively exploits the parallel-
ism inherent in horizontally partitioned data.
message arrive, and __Take_And_Contains will fire
when at least one Take message and at least one
__Contains message arrive.
Conclusion: Repeating the Success
Formula
LINQ 1.0 broke ground for radically simplifying
distributed data-intensive applications in .NET.
TESLA will stretch the programming model to cov-
er the Cloud.
Brian Beckman
Erik Meijer

Join Patterns

Finally, it has to be easier for developers to specify

Composability Saves
Money
“Composability” dictates that you
build up complex operations from
smaller, generic operations, each
as combinable or composable
with as many of the others as
possible. Both the SQOs and the
reshaping combinators follow
this principle. Composability
implies modularity, and therefore
saves money for exactly the
same reason: by reducing co-
dependencies and therefore
the entropy costs of changing
a system. Composability is a
design dogma.
synchronization and replication of program states.
This is essential for occasionally connected and
collaborative scenarios. In the past, Microsoft of-
fered little more than thread pools, message queues,
monitors, semaphores, and so on; that is, low-level
communication primitives. Overt process notations
like the Pi-calculus also seem to be off-the-mark for
mainstream developers working with object-orient-
ed programming.
We deem C-omega style join patterns to be the most
attractive alternative for TESLA and we are pursu-
ing them for .NET languages with our colleague
Claudio Russo.
Join patterns are a generalization of declarative
events that developers already know well. With de-
clarative events, the developer declares a disjunc-
tion (OR-chain) of events to which a set of event-
handler methods must respond. In the following
example, the Visual-Basic Handles clause below
specifies that the method __Click_Or_Focus must
run whenever either the Button.Click or the Text-
box.Focus event is received:
Sub __Click_Or_Focus _
(ByVal S As Object, ByVal E As EventArgs)
Handles Button.Click, TextBox.Focus
...
End Sub

Declarative join patterns substitute a conjunction

(AND-chain) of messages as opposed to a disjunc-
tion of events. The new When clause is the paral-
lel construction to the existing Handles clause. The
When clause marks a method as one that must fire
when all the specified messages have arrived. In
caller code, messages just look like method calls,
with asynchronous ones returning immediately and
synchronous ones blocking.

For just a taste of this style of programming, con-

sider Listing 3. __Put_And_Empty will fire when
at least one Put message and at least one __Empty

90 TESLA: Democratizing the Cloud www.code-magazine.com

Have You Xiine It?

All You Can Read.

Digitally.
Experience the next generation
in digital reading FREE.
Go to www.xiine.com to read
CoDe Magazine and other
great publications.