Sei sulla pagina 1di 107

The "Software 2.

0" in the
Context of Design Automation
Dr. T V09Gopal
October 2010]
Professor &
Chairman, CSI Division II [Software] &
Honorary Chief Editor - CSIC
Department of Computer Science and Engineering
College of Engineering
Anna University, Chennai – 600 025

e-mail :
Home Page
Cobb’s Paradox:
"We know why projects fail, we know how to prevent their failure – so why
do they still fail?"
“The tar pit of software engineering will continue to be sticky for a long time to
come. One can expect the human race to continue attempting systems just
within or just beyond our reach; and software systems are perhaps the most
intricate and complex of man’s handiworks”
– F.P. Brooks
The Crux
"On a large software project one is lucky if
one person in 50 has anything resembling
an overall understanding of the conceptual
structure of the project, and divinely
blessed if that person has the ability to
explain it in lay terms"
- G. Robinson
"The outcome of any software project is necessarily uncertain…There
is no problem ‘producing’ software – the problem is knowing what to
produce" - A. Hall
Winning with Software
“While technology can change quickly, getting your people to change takes a great deal longer. That is why the
people intensive job of developing software has had essentially the same problems for over 40 years. It is also
why, unless you do something, the situation won’t improve by itself. In fact, current trends suggest that your
future products will use more software and be more complex than those of today. This means that more of your
people will work on software and that their work will be harder to track and more difficult to manage. Unless you
make some changes in the way your software work is done, your current problems will likely get much worse.”
- Watts Humphrey, Winning with Software: An Executive Strategy, 2001
Rapidly Changing Global
Risk Landscape
 Unimagined new sources of risk
 Unimagined risk correlations
 Unimagined emergent vulnerabilities
Traditional Models
Aimed at business

Conceptual Aimed at business

process owners

Aimed at architects
and designers
Aimed at designers
and developers
Old Model – Microsoft Solutions
Framework [MSF 3.0 + Views]
Business strategies &
Contextual processes
Applications View

Technology View
Information View
Applications to facilitate
Business View

Conceptual business process

Information needed to
Logical manage business

Technology to support
Physical business & application
The Need of Architecture
The Winchester “Mystery” House

 38 years of construction – 147 builders 0 architects

 160 rooms – 40 bedrooms, 6 kitchens, 2 basements, 950 doors
 65 doors to blank walls, 13 staircases abandoned, 24 skylights in
 No architectural blueprint exists
Limitations of Traditional Models
 Standard linear statistical models inadequate to new world of
 Increasing emergence of fat tails in data patterns
 Extreme events cut across products, coverage,
policyholders & financial statement categories
 Traditional linear Financial Risk Management
models lack needed flexibility
 Stochastic methods cannot accommodate
interactions between multiple marketplace forces
New Model – Domain Specific
Languages [DSL] (set of views and artifacts)
Capabilities Manual Architecture
Reconciliation Procedures
Business Processes Constraints
and Entities

Services, Messages, Logical

Applications, Endpoints Constraints Data Center
Abstraction/ Refinement
Refinement Physical servers
XML, Projects,
& segments
DBs, Classes, Code
packaged into Units deployed on
MSF “can be mapped to” DSL but..
Business Applications Information Technology
Contextual Manual Architecture
Reconciliation Procedures
Business Processes Constraints
Conceptualand Entities
Logical Services, Messages,
Applications, Endpoints Constraints Data Center
Abstraction/ Refinement
XML, Projects,
Physical servers
& segments
DBs, Classes, Code
packaged into Units deployed on
Enterprise Models
 Characterizes ERM risk as threats
to organization
 Reveals interconnectedness of operational,
event, asset, liability, information & strategic
 Cannot accurately value the effects of multiple
& non-linear correlations, cascading risks or
positive feedback loops
Rational Unified Process [RUP]
US Department of Defense
Architecture Framework
[DODAF (3 Main Views)]
Zachman Framework

Data Function Network People Time Motivation

(What) (How) (Where) (Who) (When) (Why)

Scope (Ballpark) view

Owners View (Enterprise Model)

Designers View (System Model)

Builder’s View (Technology Model)

Out of Context View (Detailed Model)

Operational View (Functioning)

Vital Reality Checks
 Software projects are still late, over budget, and
 26% Project Success Rate
 On average, each Fortune 1000 company spends $3
million on software development tools each year and
sees less than a two percentage point improvement
in project success rates.

40+ Years and….

• an IT project is more likely to be unsuccessful than successful
• about 1 out of 5 IT projects is likely to bring full satisfaction
• the larger the project the more likely the failure
Complex Systems
 Need: model simultaneous changes
to multiple variables in complex
 Today all risks are potentially cascading risks
 Traditional tools never intended to work with this
expanded risk
 Statistical modeling cannot meet the challenge of
today’s non-linear dynamics
Knowledge Capture / Strategic
 The complexity advantage:
 Citibank uncovered $200M in hidden
credit risk
 Proctor and Gamble saved 22% in distribution

Dupont saved $500M annually in
manufacturing expense

Predicting the Dynamics
Key Challenges
1. The levels of professionalism observed in software engineering are
generally lower than those in other branches of engineering, although
there are exceptions.
2. Education in many universities is not producing IT practitioners with the
IT application and project skills they need.
3. The importance of project management is not well understood and
usually under-rated and senior managers are often ill qualified to
handle issues relating to complex IT projects.
4. Risk management is critical to success in complex projects but is
seldom applied effectively in the case of IT and software.
5. The vital role of the systems architect in major IT projects is frequently
not appreciated and there is a shortage of appropriately skilled
6. There is an urgent need to promote the adoption of best practice
amongst IT practitioners and their customers.
7. Basic research into complexity and associated issues is required to
enable the effective development of complex, globally distributed
1. Complexity Science
Software 2.0 – Emerging 2. Systems Thinking
Competencies 3. Cybernetics
4. Web Intelligence
5. Software Aesthetics
Understanding Complexity
Examples of complex adaptive systems include:
 The patterns of birds in flight
 The interactions of varied life forms in an ecosystem
 The behavior of consumers in a retail environment
 The rise and fall of species in evolution
 The interactions of people and groups in a community
 And other complex social, biological and ecological phenomena.

The findings of Complexity science can be used for more natural, more
productive, more enjoyable and far more innovative results managing people,
organizations and communities.
Old Science and Reality Checks
 Things happen because something causes them to
happen (cause and effect).
 We can understand what happened by reducing things to
their components or parts and examining those parts
 The universe is orderly, follows natural laws, and works
like an incredibly complicated machine.
 The best way to manage people is to organize them into
a clear structure and control them with clear directions.
 The best results occur when work is streamlined to be as
efficient as possible, with a minimum of wasted effort,
producing the most output in the least amount of time
(the "lean machine").
Chaos – Making a New Science
 The new science of Complexity evolved
out of general systems theory and a field
of study known as chaos theory.
 The New York Times reporter popularized
such insights as the Butterfly Effect,
whereby a butterfly flapping its wings in
India causes a series of air movements
that eventually result in a thunderstorm
over Chicago.
Complexity Science
 Then in the 1990s a group of brilliant scientists (including several
Nobel winners) affiliated with the Santa Fe Institute in New Mexico
said, in effect, "There's not much point in studying chaos. It's too
chaotic. Let's study Complexity, where with the help of computers
we can actually figure something out."

 They have discovered many things about "the real world" with
practical applications for business, management, community and
economic development.

 They have discovered some profound properties of life forms, order

and structure using advanced computer modeling, which suggest
powerful new ways by which organizations can emerge, evolve
and thrive in the increasingly complex technological-economic
Complex Nonlinear Systems
 "Small inputs can lead to dramatically large
 "Very slight differences in initial conditions produce
very different outcomes"
 “Global properties flow from aggregate behavior of
 "Complex adaptive systems are pattern seekers. They
interact with the environment, 'learn' from the experience,
and adapt as a result."
- Murray Gell-Mann, Nobel prizewinner in physics
 "Most complex systems exhibit what mathematicians
call attractors, states to which the system eventually
settles, depending on the properties of the system"
Software – Some Home Truths
 "Managers are finding that many of their long-
established business models are inadequate
to help them understand what is going on, or
how to deal with it"
 “Nonlinear, complex adaptive systems are
the best way to understand systems
involving people”
 "Emergence (of order) is certain, but there is
no certainty as to what it will be.”
 "Greater diversity of agents in a system leads to
richer emergent patterns”. Seek Diversity….
The Magic of Complexity
 Interconnected and interdependent elements and
 Feedback processes promote and inhibit change within
 System characteristics and behaviors emerge from simple
rules of interaction;
 Nonlinearity;
 Sensitivity to initial conditions;
 Phase space – the ‘space of the possible’;
 Attractors, chaos and the ‘edge of chaos’;
 Adaptive agents;
 Self-organization;
 Co-evolution.
Guest Editor:
Prof. K S Narendra
Yale University, USA
Coping with Complexity
– Prof. K S Narendra, Guest Editorial, CSI
Communications, October 2007
 “One of the fundamental ideas of engineering is the
abstract concept of feedback. Any dynamical system,
whether or not it can be represented by a mathematical
equation, can improve its performance using feedback.
However, feedback can also lead to instability. Hence,
when designing complex systems, adequate precaution
must be taken to use fedback information without leading
to instability. This in turn permits the designer to improve
performance and achieve robustness”
 This issue is about systems that are at present
not readily amenable to mathematical modeling.
Feedback (close-loop) Control

Controlled System
control manipulated
control Actuator
input variable

sample controlled
+ Monitor
- variable

Closed-Loop Controller
 The output of the system y(t) is fed back
through a sensor measurement F to the
reference value r(t).
 The controller C then takes the error e
(difference) between the reference and the
output to change the inputs u to the
system under control P.
Advantages of Closed-Loop

 Disturbance rejection
 Guaranteed performance even with model
uncertainties (when the model structure does not
match perfectly the real process and the model
parameters are not exact)
 Unstable processes can be stabilized
 Reduced sensitivity to parameter variations
 Improved reference tracking performance
Open-loop control

 Compute control input without continuous variable measurement

 Simple
 Need to know EVERYTHING ACCURATELY to work right
• Cruise-control car: friction(t), ramp_angle(t)
• E-commerce server: Workload (request arrival rate? resource
consumption?); system (service time? failures?)
 It does not use feedback to determine if its input has achieved the
desired goal
 It is useful for well-defined systems where the relationship between
input and the resultant state can be modeled by a mathematical
 Open-loop control fails when
 We don’t know everything
 We make errors in estimation/modeling
 Things change
Complexity and Systems
 Systems characterized by interconnected and
interdependent elements and dimensions are a
key starting point for understanding complexity
 Feedback processes crucially shape how
change happens within a complex system.
 Emergence describes how the behavior of
systems emerges – often unpredictably – from
the interaction of the parts, such that the whole
is different to the sum of the parts.
Complexity and Change
 Within complex systems, relationships between dimensions
are frequently nonlinear, i.e., when change happens, it is
frequently disproportionate and unpredictable.
 Sensitivity to initial conditions highlights how small
differences in the initial state of a system can lead to massive
differences later; butterfly effects and bifurcations are two
ways in which complex systems can change drastically over
 Phase space helps to build a picture of the dimensions of a
system, and how they change over time. This enables
understanding of how systems move and evolve over time.
 Chaos and edge of chaos describe the order underlying the
seemingly random behaviors exhibited by certain complex
Complexity and Agency
 Adaptive agents react to the system and to each
other, leading to a number of phenomena.
 Self-organization characterizes a particular form
of emergent property that can occur in systems
of adaptive agents.
 Co-evolution describes how, within a system of
adaptive agents, co-evolution occurs, such that
the overall system and the agents within it
evolve together, or co-evolve, over time.
Cybernetics - Norbert Wiener
 Theoretical-sciences of complexity--including AI, neural
networks, dynamical systems, chaos, and complex
adaptive systems.
 Practical-Many of the concepts used by system scientists
come from the closely related approach of cybernetics:
information, control, feedback, communication.
 There was also a return from the machine to the living
organism, which accelerated progress in neurology,
perception and the mechanisms of vision.
 In the sixties MIT saw the extension of cybernetics and
system theory to industry, society, and ecology.
Engineering Cybernetics Biological Cybernetics Social Cybernetics

The view of A realist view of epistemology: A biological view of epistemology: A pragmatic view of
epistemology knowledge is a “picture” of how the brain functions epistemology: knowledge
reality is constructed to achieve
human purposes

A key distinction Reality vs. scientific theories Realism vs. Constructivism The biology of cognition vs. the
observer as a social

The puzzle to be Construct theories which Include the observer within the Explain the relationship
solved explain observed domain of science between the natural and
phenomena the social sciences

What must be How the world works How an individual constructs a How people create, maintain,
explained “reality” and change social
systems through
language and ideas

A key assumption Natural processes can be Ideas about knowledge should be Ideas are accepted if they
explained by scientific rooted in neurophysiology. serve the observer’s
theories purposes as a social

An important Scientific knowledge can be If people accept constructivism, By transforming conceptual

consequence used to modify natural they will be more tolerant systems (through
processes to benefit persuasion, not coercion),
people we can change society

Three Versions of Cybernetics

Engineering Cybernetics - 1
 A realist view of epistemology: knowledge
is a picture of reality
 A key distinction: reality vs. scientific
 The puzzle to be solved: construct
theories which explain observed
Engineering Cybernetics - 2
 What must be explained: how the world
 A key assumption: natural processes can
be explained by scientific theories
 An important consequence: scientific
knowledge can be used to modify natural
processes to benefit people
Biological Cybernetics - 1
 A biological view of epistemology: how
the brain functions
 A key distinction: realism vs.
 The puzzle to be solved: include the
observer within the domain of science
Biological Cybernetics - 2
 What must be explained: how an
individual constructs a “reality”
 A key assumption: ideas about
knowledge should be rooted in
 An important consequence: if people
accept constructivism, they will be more
Social Cybernetics - 1
 A pragmatic view of epistemology:
knowledge is constructed to achieve
human purposes
 A key distinction: the biology of cognition
vs. the observer as a social participant
 The puzzle to be solved: explain the
relationship between the natural and the
social sciences
Social Cybernetics - 2
 What must be explained: how people create,
maintain, and change social systems through
language and ideas
 A key assumption: ideas are accepted if they
serve the observer’s purposes as a social
 An important consequence: by transforming
conceptual systems (through persuasion, not
coercion), we can change society
The Contributions of Cybernetics
 Develop a theory of circular or regulatory phenomena
including goal seeking and goal formulation
 Create a theory of perception, learning, cognition,
adaptation, meaning, understanding
 Include the observer within the domain of science
 Create a theory of the use of knowledge in society,
 “Scientific discovery consists in the interpretation for our
own convenience of a system of existence which has
been made with no eye to our convenience at all.”
- Norbert Wiener
Software Cybernetics
 Software cybernetics explores the interplay between
software/software behavior and control.
 This emerging and inter-disciplinary area addresses issues
and questions on
1. the formalization and quantification of feedback and self-
adaptive control mechanisms in software,
2. the adaptation of control theory principles to software
processes and systems,
3. the application of software engineering principles and
theories to control systems, and
4. the integration of the theories of software engineering and
control engineering.
Sample Problems
Control of the software test process:
How much and what additional effort is to be applied to
achieve the desired quality objective under time/cost

Optimal selection of tests:

What is an optimal set of tests for achieving the
desired quality objective given time constraints?
Sample Problems (continued)
Software performance control
How best to adjust software parameters so that an
optimal level of performance is maintained?

Control of the software development process:

What is an optimal set of process variables required to
achieve delivery objectives within cost/time constraints?

1. Use instinct and experience.

2. Use (1) supported by quantitative
(a) Use simulation: “forward” approach.

(b) Use simulation: “forward” approach.

(c) Use (a) plus feedback control: “inverse” approach.

Software cybernetic approach..

Sample Problem Scenario

r0 observed
remaining errors

Approximation of how r is likely to change

r - number of

schedule set by
the manager

cp1 cp2 cp3 cp4 cp5 cp6 cp7 cp8 cp9 t- time
t0 cpi = check point i
Physical and Software Systems: An

External Dashpot
To err is
Human Block Rigid surface

Xequilibrium Spring Force

Software Effective Test Effort

X: Position Mass of Viscosity

the Quality of the
Number of remaining
Software test process
Physical Systems: Laws of Motion [1]

First Law:

Every object in a state of uniform motion

tends to remain in that state of motion unless
an external force is applied to it.

Does not (seem to) apply to testing because

the number of errors does not change when
no external effort is applied to the
Physical Systems: Laws of Motion [2]

Newton’s Second Law:

The relationship between an object's mass m, its

acceleration a, and the applied force F is F = ma.

CDM First Postulate:

The relationship between the complexity Sc of an

application, its rate of reduction in the number
.. of
remaining errors, and the applied effort E is E=S c
Physical Systems: Laws of Motion [3]

Third Law:

For every action force, there is an equal and

opposite reaction force.

When an effort is applied to test software, it leads to

(mental) fatigue on the tester.

Unable to quantify this relationship.

Challenges (Sources of difficulty)

Physical Logical (software)

Laws of physics Yes No

Object controlled Physical Logical (and Humans)

Time dependence Stationary and non- Non-stationary

Rate of change Very slow to very fast Slow

Parameter variability Relatively low High

Taking Stock
 The interplay between software and control is not new.
 The existing interplay between software and control is,
however, far from comprehensive.
 Control theories seldom play a major role in the quality
assurance of software analysis and design.
 Software theories such as temporal logic, CSP and CCS,
have not been fully applied in control systems analysis or
synthesis either.
 The core of software engineering and that of control
theory / engineering have been developed independently
of each other.
Web Intelligence

Guest Editor:
Dr. Raghu
Chief Scientist for
Audience and Cloud
Web Intelligence
 "The information superhighway is a dirt road that won't be
paved over until 2025.“
- Sumner Redstone, CEO of Viacom/Blockbuster
- Web Intelligence includes domain expertise in Social
Networks and Social Intelligence, Web Mining and
Farming, Semantics and Ontology Engineering, Web
Information Filtering and Retrieval, Web 3.0, Web Agents
and Agent-based Systems and Knowledge Grid and Grid
- How all these strategies relate to the way Human Brain
process information and operates on it is a more
challenging question.
As We May Think
 Article by Dr. Vannevar Bush in The
Atlantic, July 1945
 the idea of hypertext or "instant cross-
referencing," decades before the term
itself was conceived by Ted Nelson and
before the concept was widely
implemented on the World Wide Web.
 Bush called it "memex" device
WWW – At a Glance
Current Trends
 Digital Libraries
 Hypermedia [Entertainment]
 Agent Technologies
 E-Commerce
 E-Learning
 E-Governance
 Multi-Tiered Architectures
 Database Connectivity Methodologies
Web Db Functions
Semi-Structured Data
 RDBMS – Fully structured data ; each
element has a well defined type
 Multimedia data is unstructured
[Impossible to represent]
 To better represent such data, one
introduces some (partial) structure.
 Such data are called semi-structured data
Web Services
 Web Services address the need for interoperability
between web applications by specifying a standard
XML based protocol.
 Please recall the hassles with CORBA, DCOM and
the likes.
 One of the better-known early implementations of
XML-based communication protocols is XML–RPC
(XML Remote Procedure Call).
 Most vendors did not fully embrace it, but rather
chose to embrace its successor SOAP (Simple
Object Access Protocol)
Web Services Layers

WSDL : Web
Web Services
 Can be secured using standard strategies
adapted to XML
 Can be adapted to GRID
 Can be adapted to ‘semantic web’
 Appear to be the best bet for EAI
 Can be dynamically composed [There are
composing tools]
Technical Perspectives of
‘The Web’
 Computer science perspective - infrastructure
and intelligent systems
 Information science and knowledge
management perspectives - data, information,
knowledge, wisdom hierarchy
 Social intelligence perspectives - connectivity,
social network intelligence
 Application perspectives -E-commerce, E-
Governance, E-Learning and so on.
Perspectives of ‘Science’
 Physical/biological science perspectives -analytic
disciplines that aim to find laws/processes that generate
or explain observed phenomena
 Social science perspective– scholarly or scientific
disciplines that deal with the study of human society
and of individual relationships in and to society
 Computer science perspective - a basically synthetic
discipline that creates mechanisms (e.g., formalisms,
algorithms, etc.) in order to support particular desired
Web Science
 The Web is a new technical and social
phenomenon and a growing organism
 The Web needs to be studied insitu and
understood and it needs to be engineered
 Web Science is a new field of science
that involves a multi-disciplinary study and
inquiry for the understanding of the Web
and its relationships to us
Intersection of Disciplines
Roots of Web Science
 Web Ecology (Bernardo Huberman) – “The Web becomes a
gigantic informational ecosystem that can be used to
quantitatively measure and test theories of human behavior and
social interaction.” (The Laws of the Web, 2001)
 Web Engineering – “…covers the realization of solutions within
the Web, its applications and its advancement, in particular its
approaches, methods, models, principles and tools, which are
based on the information and communication technologies of the
Internet” (ISWE)
 Web Science suggests that

We can use non-technical concepts to understand
the complexity of our Web applications so that we
can engineer them to have new and predictable

We can better understand the impact of Web
technology in all areas of communication and
social interaction
 We may be able to reliably predict the future
evolution of the Web
Software Agents
 An intelligent agent is a piece of software that
performs a given task using information from its
environment and acts in such a way that it can
complete the given task successfully.
 Some desirable properties for such agents are:
autonomy, adaptability, mobility, reactive,
pro-active, social and communication ability.
 To deal with complex real world problems, it is
desirable to have different type of agents.
Key Challenges
1. Semantic Heterogeneity: Structured
and Unstructured data
2. Support for Dynamic Evolution of
Information Sources
3. Scalability and Performance
4. Generalization across application
Information Architecture
 Certifies the authenticity, cost and degree of
completeness of the content.
 Establishes and documents the context (who,
what, why, where and how aspects that are
extrinsic) of a content. The special context of
preserved and archived collection if any is also
 Identifies and exploits structural relationships that
exist between and within a document.
 Provides a range of intellectual access points for
an increasingly diverse range of users.
Information Architecture
 Provides some of the information a professional
might have provided in a physical reference or
research setting.
 Facilitates multi-versioning of the document (various
forms of the same document).
 Tracks acquisition, stocking and accessioning
information over the entire network.
Note : Document = Information Object
Semantic Web
 "The first step is putting data on the Web in a
form that machines can naturally understand,
or converting it to that form. This creates what
I call a Semantic Web—a web of data that can
be processed directly or indirectly by
 --Tim Berners-Lee, Weaving the Web, Harper
San Francisco, 1999
Key Reasons
1. Information Overload
2. Stovepipe Systems : No sharing of data
3. Poor Content Aggregation
Smart data Continuum
The Stages of Smart Data
 Text and databases : The Applications are
 XML documents for a single domain
 Taxonomies and documents with mixed
vocabularies [Classification]
 Ontologies and rules [Inference of new data ;
facilitate knnowledge sharing and reuse]
 Semantic Web: a machine-processable web of
smart data.
Web Services
Semantic Web Vision
RDF : resource Description Format
Software Aesthetics
 “Beautiful Code: Leading Programmers Explain How
They Think”, Andy Oram and Greg Wilson [Editors],
O'Reilly and Associates, Sebastopol, CA, USA, 2007.
 “33 essays by eminent programmers on the art and craft of

“For both essays and computer code, it’s always important to look at how each
one is written. Even if the idea itself is good, it will be difficult to transmit to the
desired audience if it is difficult to understand. The style in which they are
written is just as important as their purpose.”
- Yukijiro Matsumoto, Chief Designer of the Ruby Programming Language,
Beautiful Code, O’Reilly, 2007
Some Case Studies
NCSL : Non-Comment Source Lines
Growth in functionality provided by software to pilots of military aircraft
Systems that must manage complex interactions and high coupling are more
prone to accidents. Space missions are among these high-risk systems.
US DoD - Software Engineering and System
Deputy Directorate Laments [2007]
 Software requirements not well defined, traceable,
 Immature architectures, COTS integration,
 Software development processes not institutionalized,
planning documents missing or incomplete, reuse
strategies inconsistent.
 Software test/evaluation lacking rigor and breadth.
 Unrealistic schedules (compressed, overlapping).
 Lessons learned not incorporated into successive builds.
 Software risks/metrics not well defined, managed.
Standish Group Top 10 Factors Most Responsible for Project
Success or Failure
Complexity and Success / Failure
1. Complexity Science
Software 2.0 – Emerging 2. Systems Thinking
Competencies 3. Cybernetics
4. Web Intelligence
5. Software Aesthetics