Sei sulla pagina 1di 43

Fixing an Insecure

Software Life Cycle


Practical Techniques for Building Security
into Existing Software Development Programs

April C. Wright
Fixing an Insecure
Software Life Cycle
Practical Techniques for Building Security into
Existing Software Development Programs

April C. Wright

Beijing Boston Farnham Sebastopol Tokyo


Fixing an Insecure Software Life Cycle
by April C. Wright
Copyright © 2018 O’Reilly Media, Inc. All rights reserved.
Printed in the United States of America.
Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472.
O’Reilly books may be purchased for educational, business, or sales promotional use. Online edi‐
tions are also available for most titles (http://oreilly.com/safari). For more information, contact our
corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com.

Editor: Courtney Allen Interior Designer: David Futato


Production Editor: Nan Barber Cover Designer: Karen Montgomery
Copyeditor: Octal Publishing, Inc. Illustrator: Rebecca Demarest
Proofreader: Sonia Saruba

May 2018: First Edition

Revision History for the First Edition


2018-04-27: First Release

The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Fixing an Insecure Software Lifecy‐
cle, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc.
While the publisher and the author have used good faith efforts to ensure that the information and
instructions contained in this work are accurate, the publisher and the author disclaim all responsi‐
bility for errors or omissions, including without limitation responsibility for damages resulting from
the use of or reliance on this work. Use of the information and instructions contained in this work is
at your own risk. If any code samples or other technology this work contains or describes is subject
to open source licenses or the intellectual property rights of others, it is your responsibility to ensure
that your use thereof complies with such licenses and/or rights.

978-1-492-02821-5
[LSI]
Table of Contents

Fixing an Insecure Software Life Cycle. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1


The Status Quo of Software Development Life Cycles 1
Understanding Stakeholders and Existing Process Mechanics 2
Preparing to Rebuild the Program 12
Working as a Unified Team 19
Setting Expectations for Stakeholders 23
Sample Checklists and Planning Documents 30
Conclusion 37

iii
Fixing an Insecure Software Life Cycle

The Status Quo of Software Development Life Cycles


The geography we have created is all about speed, convenience, and scale; security is an
afterthought.
—General Michael Hayden, retired head of CIA, NSA

Reid Hoffman, the founder of LinkedIn, famously said, “If you are not embar‐
rassed by the first version of your product, you’ve launched too late.” Although
this has been the status quo in software innovation to achieve a proverbial leg up
on the competition, it can be a dangerous status quo when it comes to the secu‐
rity of a software product.
Embarrassment arising from software might indeed be an acceptable risk when it
applies to functionality, to features, or to the user interface (UI) or user experi‐
ence (UX). However, when embarrassment is the potential result of a compro‐
mise of your users’ security or privacy, it should be considered a completely
unacceptable risk.
The challenge within the Software Development Life Cycle (SDLC) is to include
both offensive testing and defensive building in organizational security strategy.
Security is not one or the other—because offensive testing and incident response
are not replacements for good code, strong architecture, and threat-based design.
This report discusses how security practitioners can work with other stakehold‐
ers in your organization to improve the security of an existing SDLC. Software is
often created and released before formal security oversight is implemented, so it
is important for security practitioners to understand how to observe and change
existing software development processes, analyze the human elements of soft‐
ware life cycles, build relationships, and generate measurable changes over time.
The basis for the security “blue” team—which handles incident response, foren‐
sics, log analysis, and monitoring—is the idea that software will be under attack.
If you build software securely from the ground up, its operational state will be

1
stronger, you will lessen the frequency of incidents, catch incidents earlier, and it
will be easier to respond to an incident. Therefore, builders—the people who
design, develop, test, and deploy software—are critical to software security.
Builders are on the front lines of defense, and when they are empowered to build
securely and knowledgeably about security, it can make an enormous difference
in software assurance. Yet a gap tends to separate software builders and security
blue teams, often to the detriment of the software’s security and users’ privacy.
It is your duty and privilege as a security practitioner to share knowledge with
others and continuously work to improve the risk posture of your organization.
When everyone involved in the SDLC is trained in software security principles
and practices, it benefits both your users and your organization.
This report teaches you best practices for understanding the goals of different
stakeholders, gaining support across teams, assessing existing SDLC processes,
and taking incremental steps to create measurable changes over time.
Keep in mind that you must take care when implementing new processes because
there is always a risk of alienating other teams or inadvertently making them
work against the process instead of accepting it and championing it. Accordingly,
this report will also teach you how to analyze and understand why software is
being built insecurely before starting any improvements within the software eco‐
system. Only then can you and your organization’s security team elicit change.

Understanding Stakeholders and Existing Process


Mechanics
Before you make a change, it is critical to understand what needs to change. This
understanding is based on examination of historical data and, ideally, supple‐
mented with observation and inquiry.
You can begin making changes based on process and practices, but focusing on
the mechanics of an SDLC ignores its most important component: people. Resist‐
ance to change is a natural human response, which, when met with empathy and
positivity, you can minimize.
An empathetic approach supports stakeholder motivation and adoption of
change. As people are often both the cause and solution of any problem, elicita‐
tion of change will start with a study and analysis of the SDLC’s stakeholders as
well as their goals.

Stakeholders Have Differing Points of View


In the article “How to Get a Stakeholder to Buy in on a Project”, Scott Shpak pro‐
vides a clear definition of project stakeholders:

2 | Fixing an Insecure Software Life Cycle


A stakeholder is any person or group that affects or is affected by a particular
project. Along the path to completing your project, stakeholders can be partners,
resources, or roadblocks—and potentially all three rolled into one. Stakeholder
buy-in, the cooperation or positive participation of a stakeholder, is the preferred
condition for any successful project.
Organizational stakeholders include people who design, build, test, review, and
implement software, as well as people who organize the activities that make up
the SDLC.
Regardless of the formal development methodology in use (Agile, Waterfall, etc.),
security teams want software to be built with the following traits:

• Created with confidentiality, availability, and integrity in mind


• Monitoring ready
• Capable of being maintained properly and safely maintainable
• Capable of being deemed compliant,1 as needed
• Resilient when attacked
• Decoupled from other components such that it can be decommissioned
when no longer capable of being secure

These seem like relatively simple goals, but the realities of achieving this end state
are far more complex.
Understanding the motivations of stakeholders is critical to building relation‐
ships that will lead to a stronger and more responsive organizational approach to
security. This section helps you to better understand stakeholders’ differing goals
and views.

View: builders
The builders of software include its developers, engineers, architects, and design‐
ers. Builders create.2 They solve problems. They are the source of software.
Their primary responsibility is to deliver software that is as follows:

• On time
• Correct
• Optimized
• Minimally defective

1 Compliance is certainly not equivalent to a secure end state, but it is for better or worse a common
organizational goal.
2 Code is art.

Understanding Stakeholders and Existing Process Mechanics | 3


Because of tight time constraints, builders are working to make code that does
what it is supposed to do3 in as little time as possible.
Builders will build in whatever way the organization requires and demands, so to
build more secure software, you should make allowances during planning and
build stages for necessary resources, including leeway from management in
terms of any additional time required to address security.
Security measures do not directly create profit. Secure practices are therefore a
bit like insurance against future damage. After software is released, builders ide‐
ally should be able to focus on working to add to or improve features as well as
creating greater stability, because these are the parts of software that make money
and have a clearer path to Return on Investment (ROI).
The frequency of security testing can never keep pace with the speed of the build‐
ers’ work, so it is important that this first line of defense be security
knowledgeable and permitted to do what needs to be done.

View: project managers


Project managers (PMs) are like the SDLC’s supervising adults. Project manage‐
ment is a very social job that involves driving human contributors toward meet‐
ing deadlines, which can be challenging. PMs make sure everyone does what is
needed to produce software, pushing teams to complete tasks and managing the
master plan for the software effort.
The primary limitation of a PM’s effectiveness originates from the information
(or rather, lack of information) given to them by external sources. In other
words, a PM can work only with what has been provided to them. If given
impossible timelines or incomplete requirements, the project will suffer. Thus,
with project management, it is a “garbage in, garbage out” situation, and a
higher-quality product will be created when a good set of requirements, tasks,
and estimated times are provided to a PM.
The PM will develop projected completion dates and project milestones. The
requirements gathering stage will occur at the very beginning of a project, and
security requirements need to be included before this stage is completed. Other‐
wise, there is a risk that security requirements could be the reason for a timeline
not being met, causing the software to be either delayed or released insecurely. If
a timeline is in danger of slipping, security-related tasks might not be completed.4
PMs want to help security teams achieve relevant goals, but a PM must know
what those goals are to provide effective help. You will need to develop and pro‐

3 If you have ever written code, you truly know that making code work can be the most difficult part!
4 Unfortunately, security functions are too often one of the first things to fall off the table.

4 | Fixing an Insecure Software Life Cycle


vide detailed instructions and timelines for what needs to be done to reach every
security goal. Be ready to answer questions such as: “After this step, what are the
next step(s)?” “What should be the outcome of this?” and “What needs to be
ready to start this?”
To create change, the PM’s master plan is one of the most important places to
ensure that security tasks are included. This will guarantee that security-related
tasks are completed, and that security does not negatively affect timelines by
causing surprise delays later.

View: quality assurance


Quality assurance (QA) teams will validate that all predefined testable require‐
ments have been met. This is usually performed after software has been created
and before it goes into production. The QA team’s responsibility is basically to
mimic the actions and viewpoint of the customer before the customer has access
to the software. The QA team can carry out some QA activities in an automated
way, but most are performed manually.
If the customer’s expectation is that data will be secured by the software, QA
should be utilized to ensure that the software satisfies security requirements.
Note that in this way, QA’s limitation is a bit like the PM’s limitation; QA teams
can validate only those requirements that have been adequately provided and
only if the requirements are testable.
Although the primary objective of QA teams is to test and assess the quality of a
product, you can empower this team to seek out security defects if the team is
knowledgeable about how and where to look. The security requirements laid out
during the early phases of the SDLC, as well as use cases and misuse cases,5 can
be measured for “quality” if these cases are written in a way that makes them
capable of being tested.
In other words, if you have not included security teams as stakeholders during
the requirements phase of the SDLC and were not able to create a set of testable
security requirements, QA teams will not be able to validate that any of those
requirements have been met. The same is true of use and misuse cases that are
not testable, so keep that in mind while developing security cases.
If you include QA activities in an SDLC, those activities are normally planned far
in advance, which is why it is so important that you include security tests in any
QA planning as early as possible during the life cycle.

5 A misuse case is a use case with hostile intent. It is the opposite of a use case. For example, a misuse case
might be: “As an attacker, I want to be able to escalate privileges within the cloud so that I can gain
access to other customers’ data.”

Understanding Stakeholders and Existing Process Mechanics | 5


View: the organization itself
A commonly used business metaphor involves the concepts of “fast,” “good,” and
“cheap,” explaining that it is possible to have any two of those concepts, but not
all three.
This metaphor holds true with software, where “good” includes quality and secu‐
rity, and it is often “fast” and “cheap” that are chosen. By not selecting the option
of “good” and combining it with either of the other options, the software concept
that is created might translate (at least in the security world) to “very expensive
later.”
If software is not built with “good” in mind, customers might be happy initially,
but builders are taking risks in the long term that might not ultimately leave the
organization or its customers happy.
Additionally, if the organization starts with “good” software, the operational state
can be “good” as well, which means less resources spent on maintenance, incident
management, and other production state concerns.
One of the primary goals of implementing software is to support making revenue
and keeping customers and stakeholders satisfied. A security bug that has been
exploited by a malicious entity can have significant negative financial impacts on
an organization. Therefore, if profit is an organizational goal, security should be,
as well.

View: DevOps
The DevOps approach represents a collaboration of efforts by builder, opera‐
tional, security, and QA teams. DevOps processes integrate multiple functions
into a single output using automated, semiautomated, and/or manual tools to
ensure consistency and efficiency during software releases.
The DevOps team has a goal of delivering software continuously, as well as
improving software continuously, which represents an obstacle to rigorous secu‐
rity testing. Security testing teams face short turnaround times for code rollouts
and releases, which makes thorough testing difficult.
DevOps teams can assist in implementing security measures during the SDLC in
the following ways:

• DevOps teams are responsible for configuration management (CM). Assur‐


ance of security configurations can be validated regularly as part of CM pro‐
cesses. Security monitoring configuration can be standardized as part of the
DevOps process to make incident response easier and more effective.
• Bridging the gap between development and operational states, DevOps
teams should be responsible for performing automated code reviews for

6 | Fixing an Insecure Software Life Cycle


security and vulnerability scanning of code, infrastructure, and components
being implemented.
• Security teams can use the automated nature of DevOps testing to decrease
security risk prior to delivery into production state. DevOps represents a
particularly good opportunity for including automated security testing as
part of the existing process.
• DevOps teams should be trained in security awareness and armed with
knowledge of security best practices. Throughout their activities, DevOps
should be sure to follow security principles such as least privilege and separa‐
tion of duties; for example, no DevOps practitioner should have complete
end-to-end control of a release. Checkpoints should exist within DevOps
processes to ensure that multiple people review work that is being performed
to reduce the chance for a vulnerability being introduced to a production
environment.

DevOps teams can unite with security teams by embracing the shared goal of cre‐
ating better (more secure) software using as much automation as possible.

View: legal
Legal teams aim to protect all contractual, regulatory, privacy, and compliance
interests. They are concerned with limiting liability and meeting formal expecta‐
tions to protect the organization.
Security-relevant legal interests include the following:

• Maintaining the confidentiality of sensitive internal and proprietary data


• Maintaining the confidentiality of other parties’ data to which the organiza‐
tion is somehow obligated
• Being able to attest to the integrity of legally relevant documents and artifacts
• Meeting formal contractual obligations and expectations for customers
• Ensuring the organization’s third-party supply chain meets expectations and
obligations related to confidentiality, integrity, and availability.

To protect these interests, a security practitioner must at any time be able to


explain to legal teams the risks of a real or potential organizational failure. From
due diligence, to contractual promises, to concerns about advertising of “secure‐
ness” of software, legal teams share many of the same viewpoints and goals as
those of the security team, and is therefore a true ally, providing minimal resist‐
ance to changes related to decreasing risk.

Understanding Stakeholders and Existing Process Mechanics | 7


View: customers
From the first interaction with a customer to the procurement of software,
expectations are being set. Whether it is punctuality of sales teams, knowledge of
technical staff, or merely timelines and protocols for internal interactions
between teams, organizations promise and deliver many things to customers.
Although some promises are more formal than others, such as the contractual
obligation of an organization to maintain an industry compliance or governmen‐
tal regulation, adherence to all laws and commitments is a reasonable expecta‐
tion.
Customers begin by granting an initial set of their trust to the organization.
When the organization continuously meets expectations, trust is built and solidi‐
fied.
Even though a single minor misstep might be capable of being overlooked in a
long-term partnership, more serious violations of trust can include topics that
legal teams would also be very concerned with, such as violation of compliance
or regulatory objectives, exposure of proprietary data, or breach of contract.
Additionally, when the customer consistently experiences poor behavior such as
lack of due diligence, failure to be transparent, and/or incapability of fixing prob‐
lems, the organization is likely to lose customer trust and, in some cases, never
regain it.
Security is one of the most important aspects of the customer relationship to
manage, because a failure to meet your customers’ expectations of security can
hurt the organization’s trustworthiness and irreparably damage the brand.

View: suppliers and third parties


Third parties generally see themselves as suppliers to customers, not as critical
offshoots of the organizations they support via a supply chain. Making things
more complicated, third parties have their own supply chain to manage, each
with its own policies and practices. Ultimately, every organization will have third
parties each with their own supply chain with policies and practices, ad infini‐
tum. The links in a supply chain adhere to their own individual governance,
compliances, and regulations and might operate independently unless they are
validated by another link.
The supply chain generally does not care as much about an organization’s secu‐
rity as the organization does. It is therefore extremely important for an organiza‐
tion which cares about security to audit its own supply chain regularly. You must
assess any new third parties brought into the supply chain prior to their supply‐
ing products or services. The goal is to provide assurance that third parties
adhere to security policies and practices that are equal to or better than the pri‐
mary organization’s policies and practices. Audits of third parties should deter‐
mine a level of risk associated with the supply chain entity. Security and legal

8 | Fixing an Insecure Software Life Cycle


teams can then evaluate and score the risk(s), and request or demand changes to
meet requirements and expectations. A third party should not be introducing
unacceptable or unknown risk to the primary organization.
The introduction of a third-party relationship into an organization’s supply chain
might happen long before the SDLC officially starts, so if necessary, you might
need to conduct an audit and analysis of supply chain terms later than would be
otherwise ideal. You can amend contracts as needed or use them to request that
changes be made by a third party that does not meet requirements or have secure
practices.

Analyzing Existing Processes


Interviews, observations, process and procedure analysis, and information gath‐
ering activities will make it possible to define the organization’s current security
issues and gaps. By pinpointing issues and gaps, it becomes possible to determine
and propose solutions. Then, you can create a plan for reaching the goal of clos‐
ing the gaps, addressing the issues, and improving security practices in the SDLC.
With management buy-in, you can implement changes in reasonable ways over
time to create a level of security maturity for the software program. This section
shows you what steps to take to find and analyze potential program improve‐
ments.

Risk management
A working risk management program is essential to a mature SDLC. If you don’t
know what the financial and security risks to the business are, you cannot fully
address the risks. Lack of risk management is a business gap.
You must understand and track risks. The “how” and “why” of building a func‐
tioning risk management program is beyond the scope of this text; however,
numerous quality resources on this topic available. Even a basic risk rating, rank‐
ing, and tracking process can help with SDLC security.

Observing and evaluating the existing SDLC


Well before approaching anyone about making changes, it is important that you
identify what is broken and what can be improved or fixed.
Start by gathering as much data as possible about the SDLC as it is currently per‐
formed. Determine the key players and their motivators: are SDLC participants
pushing to meet deadlines, or is thoroughness more important? Is there one par‐
ticular stakeholder, group, or mindset that seems to “drive” the SDLC?
Review any existing documentation related to the SDLC process, and then simply
observe a release cycle from start to finish. Ask questions and seek explanations,
but try not to impede. Document your observations (especially those related to

Understanding Stakeholders and Existing Process Mechanics | 9


security) and be sure to note what the major phases and milestones are, including
any unofficial things that happen. Keep track of both the good and bad.
This exercise is not intended to get anyone in trouble; this is an objective set of
notes that will eventually turn into a more formally documented assessment and
recommendations.

Performing a gap analysis on current processes


Security bugs in software are often remediated reactively, not prevented proac‐
tively. Reliance on only offensive testing to create assurance is ineffective. The
preferred alternative is a security program that includes security in every connec‐
tion, every design element, every supply chain link, and every line of code as it is
written.
Being able to build a brand-new security program for a brand-new software
product is ideal and inherently easier; however, there is probably already an
established SDLC process that you must modify. Unknown gaps cannot be
closed, so you must identify, document, and rate gaps in terms of urgency and
complexity; socialize proposed solutions to the stakeholders; and remediate.
The goal of a gap analysis is to perform an objective review of the existing pro‐
gram with the intent of making the business more successful by reducing risk.
Here are some gaps to look for while observing the SDLC:

• Are there avoidable delays in procedures?


• Would it be better if security were engaged earlier in the process?
• What improvements could be made that would lead to security being consid‐
ered in each phase of the SDLC?
• What milestones should require approval from security teams?

After you have identified areas in need of improvement, you should take the fol‐
lowing steps to create a gap analysis document:

1. Summarize the existing life cycle for anyone who is unfamiliar with the pro‐
gram.
2. Describe the current state of the SDLC from a security perspective.
3. Point out any good choices that have already been included in the SDLC.
This can help provide recognition for people having done the right thing.
This also helps establish that the gap assessment was conducted in a thor‐
ough, unbiased way.
4. List specific gaps in the program that could lead to an insecure product.
Clearly explain the risk(s) that the organization is exposed to as a result of
each process or procedural gap, providing examples whenever possible.

10 | Fixing an Insecure Software Life Cycle


5. List gaps chronologically based on the SDLC phases and timeline.

We discuss the path to developing and documenting possible solutions and rec‐
ommendations to close the gaps later in this report.
When considering language, tone, and content for a gap analysis, keep in mind
that this report will likely be passed around various levels of management and
coworkers, reaching exposure in potentially every part of the organization. The
analysis is therefore not the time or place for blaming any person or team, and an
appropriate business format is preferred.

How Does Security Affect the Stakeholder?


Security teams are responsible for ensuring that the organization demands a
high-quality product by making each stakeholder understand why security is
important for them. Security teams can sometimes be viewed as an impediment
to the execution of primary responsibilities and viewpoints. Why should other
stakeholders take more time and effort than what they are doing now? They want
to know how it will benefit them.
Until security practitioners are viewed as critical stakeholders in the existing
SDLC, the “us versus them” mentality will perpetuate. When someone under‐
stands how important security is, how can they possibly neglect it? Thus, security
teams must approach and engage stakeholders in an empathetic way to ensure
their cooperation.
You need to acknowledge that change is going to take some adjustment, but also
explain how any change will affect the stakeholder and in what positive ways it
will make their responsibilities easier or better.
For example, to builders, time spent fixing security bugs is not conducive to their
goals. Following is a list of factors that you can address in motivating builder
teams to want to work with security teams:

• When software is built securely from the start, builders will have more time
to work on new features and tackle new opportunities.
• Builders will receive training, allowing them to learn new skills, potentially
increase compensation, and be more competitive in the job marketplace.
• Gamify security:
— Offer perks or prizes for the person or team with least security bugs at the
end of the month, year, or quarter.
— Participation in or completion of security training can award entries into
a raffle for a gift card or other prize.

Understanding Stakeholders and Existing Process Mechanics | 11


— Decreases in security bug count numbers month-to-month can also
become raffle entries.

When security teams can acknowledge and empathize with other teams and
demonstrate shared needs and goals with stakeholders, it helps build rapport.
Stakeholders need to believe that security teams are not just creating unnecessary
work, but that the goal is to behave as one team and benefit the organization.

How Does Security Affect the Process?


If your organization’s SDLC processes are not already documented, that is an
important business gap to address. Process flows provide a clear path to stake‐
holders and set shared expectations across all stakeholders. Being able to review
the steps taken during a process allows stakeholders to modify steps and add
steps, as well as remove any unnecessary steps.
Within a documented process, security can be added as either an information
input source (such as providing requirements) or as a checkpoint (such as
approving the SDLC move to the next stage). Walk through the process and,
when other teams provide input, ensure that security also provides input. Inject
security reviews and collaboration into the process wherever necessary. As the
process matures, security’s role within the process will become clearer.

Preparing to Rebuild the Program


Without a plan, it would be challenging or impossible for you to shift the culture,
to get buy-in, and to elicit change.
Change is not easy for most people. Cognitive bias makes humans want to stick
with the status quo, no matter how ineffective it is. Anticipate pushback on
change and be ready to address concerns; not everyone will want change.
It is ultimately in everyone’s best interest to secure the organization’s software—
the employees, the customers, investors, shareholders, etc. Security should be a
“need,” not a “want” for the organization. The ultimate achievement is to make
stakeholders feel that a secure end state is an imperative for the organization—
the right thing to do for themselves and the greater good.

Key Program Metrics


Lack of adequate SDLC metrics is a business gap that you need to address as soon
as possible.
To measure improvements to software security, start collecting and analyzing
metrics (if they are not already being tracked). If metrics are already being

12 | Fixing an Insecure Software Life Cycle


tracked, determine whether they are adequate to aide in analysis and comparison
over time as the SDLC improves, and then calibrate as needed.
Here are some examples of key metrics to track:

• Number of security bugs trending over time


• Types of security bugs found
• Comparisons of number of security bugs versus other bugs
• Severity of security bugs trending over time
• Regressions seen between releases
• Recurrence of similar types of security bugs (e.g., buffer overflow conditions
found repeatedly)

Metrics do not answer the fundamental question of why security vulnerabilities


are occurring. In “Using Metrics to Manage Your Application Security Program”,
SANS recommends the following, after you have collected metrics:
This live data needs to be acted on, shared, and used. DO NOT keep it in PDFs or
a spreadsheet. Make sure to get metrics back to the people who need to use them
in the program and be sure it is in the format they need. Get metrics out of your
tools and reports and get it to your developers’ development backlog...The faster
you can find real problems and get them back to developers, the faster—and more
likely—these problems will get fixed.
Whereas the gap analysis you performed pertained to high-level gaps in process
or procedure, metrics can help identify individuals and teams responsible for
certain gaps.
Have a policy in place that details how the organization will handle recurrent cre‐
ation of insecure code. If a person continues to produce insecurity (insecure
code, insecure workflows, insecure architecture, etc.), you need to begin by edu‐
cating this individual on how to do things properly. Your response to continued
bad coding should initially be training, and policy might want to provide the pos‐
sibility of termination of employment if improvement is not seen from additional
training. Hiring a replacement who has security experience will take more time;
statistics show that bringing someone new into the organization costs more than
retaining an existing employee. Any new hire would need to be trained on
organization-specifics, anyway. Ultimately, termination for insecure practices is a
last resort and would hopefully consider a number of factors.
You should track individuals who create the least security bugs or who identify the
most security bugs before release. People who do the right things and make good
choices should be held up as models of good action. Positive reinforcement is
always better than negative reinforcement when trying to change behavior.

Preparing to Rebuild the Program | 13


Presenting this data gathered by tracking metrics to management is not a one-
time effort but a regular (i.e., weekly, monthly, quarterly, yearly) set of reporting
that will support the time and other resources spent on securing the SDLC. Pro‐
cess improvement should also show ROI of time that is not spent, any decreases
in costs, and downward trending in terms of number and severity of security
bugs.
Metrics can help rate and evaluate identified gaps and, more important, can be a
key aspect of goal setting for improvement of the SDLC program.

Improvement over time


Few things are as complicated as cost-ROI for organizations.
Not only can security practices be a general cost center, generating substantial
revenue, but lack of security can become a remarkable burden on the organiza‐
tion in terms of money and time. According to recent data, the cost of a single
data breach can be upward of $3.62 million. Every day, there are roughly 85 new
zero-day exploits,6 and software vulnerabilities on commercial off-the-shelf
(COTS) products are key entry points for hackers.
As previously mentioned, the priorities of builders are on time, working, and cor‐
rect. Some significant security flaws will be effectively unfixable (or at least,
unable to be fixed fast, cheap, or in a quality way).
Bugs are easy to fix compared to flaws. A bug is a code problem. Flaws manifest
in design and architecture, not during code development. Flaws are significantly
more complex and difficult to fix as time goes on, which is why they are increas‐
ingly more expensive to fix as software moves through the SDLC phases. For this
reason, identifying and fixing “flaws” before software enters the implementation
stage, development stage, and certainly before the production stage is so impor‐
tant.
According to “Using Metrics to Manage Your Application Security Program” by
SANS, key impacts that will need to be continuously reported on include the fol‐
lowing:

• How has the application security program improved the organization’s secu‐
rity risk posture overall?
• How is the program helping to improve the availability and operational risk
of critical systems and services?
• How is the program helping to reduce time-to-delivery for important pro‐
grams or cost of operations for existing services?

6 A “zero-day” is a vulnerability that exists but does not have an associated patch to prevent exploitation.

14 | Fixing an Insecure Software Life Cycle


• What serious risks remain to critical operations and key products or pro‐
grams?
• How do these risks translate to threats to key customers and organizational
reputation?
• What is being done to systematically reduce these risks?

Phased Goals
You cannot perform a complete renovation of an SDLC program in a day. After
performing the gap assessment, prioritize the gaps and consider recommenda‐
tions for improvement. If you cannot carry out any remediation quickly or easily,
you might need to implement gap fixes in phases.
Phased goals show change progress to management and other stakeholders
sooner.
Include each of the gaps in one of three initial phases:

1. Phase 1 gaps are to be addressed first, including a few “low-hanging fruit” or


easy-win items that can show progress to management. You should address
the largest, most dangerous, or gaps that will make other gaps easier to close
in Phase 2. Some gaps to include in this phase are failure to engage security
early enough in the process, failure to include security in certain SDLC activ‐
ities, or failure to require security to sign off before a critical process moves
forward. After you create gating conditions to address these types of gaps,
you can close other gaps methodically. The primary goal for Phase 1 activi‐
ties is not to create all the changes necessary, but to make it clear that secu‐
rity is becoming part of the process and that management believes security is
an important part of the SDLC, so stakeholders will be amenable and expect‐
ant of further, possibly more difficult change.
2. Phase 2 addresses gaps that are important but cannot or should not be
addressed until you address the Phase 1 gaps. Utilizing existing resources
and staff is the key priority for Phase 2. Gap-closing activities should inte‐
grate other stakeholders, which will help garner support through active par‐
ticipation and input into decisions related to solutions.
3. Phase 3 is not the absolute final phase, of course, and Phase 3 activities are
not unchangeable. This phase provides a long-term view of where the pro‐
gram is going, lays out what SDLC stakeholders can expect and look forward
to, and allows time for SDLC to prepare for ongoing change or change that
requires additional time or planning.

You will present your phased goal plan to management as part of the business
case, which we discuss later in this report. Management and stakeholders will use

Preparing to Rebuild the Program | 15


the business case to review the proposed plan and justify its implementation. It is
therefore imperative that the plan provide as much detail as possible about what
needs to be done, who needs to be involved, any costs or resource shifts neces‐
sary, and why the plan is important.
The gap assessment and phased goal plan are the basis of a long-term plan for
change. This is effectively a roadmap to where the program is going, including
small and large steps that will make the program integrate and require security at
critical stages. Perform an internal security team review of the business case
before it is socialized to other stakeholders.7
You should hold a plan review internal to the security team before socializing the
results to other parts of the organization. Security management should confirm
the existence and prioritization of each gap that needs to be addressed. Gaps
become less of an insurmountable SDLC issue when handled individually.

Gaining Management Support


The most critical step in an SDLC transformation is obtaining buy-in from man‐
agement. Management can help set expectations with other stakeholders and
provide support when there is reluctance to cooperate. There might be some
pushback initially from different teams for various reasons, so being backed by
management will help make change easier and more palatable for many stake‐
holders.
To gain the support of another team, the best path to success is an “up-across-
and-down” approach. This means sending ideas up to the security team’s man‐
agement, across to the other team’s management, and then back down to the
intended team members. Peer-to-peer management relationships can be
employed to work out details and make compromises, and ideas can be spread en
masse by management. This approach is more efficient and effective than
approaching individuals on the other team with the intent of changing minds or
processes.
Although concerns about money, risk, and time are salient, personal and organi‐
zational success associated with receiving less-urgent escalations can be a signifi‐
cant motivator for management. There are increased costs associated with
waiting to fix a bug or flaw, as well as intangible costs such as harm to the brand
in the event of a breach, representing unexpected risk to the budget and level of
effort. You should provide management with documentation and analysis of any
identified gaps and risks.

7 You might also want to have a personal plan that is based on observations that you should possibly keep
discreet (especially any observations about stakeholders, hierarchy, and power you might find useful,
but would not be appropriate in circulation).

16 | Fixing an Insecure Software Life Cycle


Most organizations have limited resources, so change will likely require a shift of
resources from one place to another. To push for change and succeed, you must
make a strong argument for a feasible strategy to the organization’s leadership.
You do this using a business case.

The business case


Understanding is the beginning of approving.
—André Gide

Making changes to an existing and functional (yet insecure) SDLC might require
some convincing of stakeholders before they approve or abide. A simple discus‐
sion with a stakeholder regarding the benefits of more secure software might be
enough to start making changes. However, at larger organizations and for extra
credibility with high-level stakeholders, it might be preferred or required that
you present a formal business case. A business case will contain the gap analysis,
metrics and statistics, the phased goal plan, and possible solutions and recom‐
mendations for desired outcomes. This business plan provides a full set of infor‐
mation as to exactly what change will mean for the organization, pro and con.
A business case is a way to inform a stakeholder that an idea is sound. It is a
document that helps justify an idea by explaining how it will align with the
organization’s strategic goals. It might sound daunting to create, but you just need
to present the data obtained through previous activities in a way that someone
else can understand, and you need to add possible solutions.
The most basic business case document has four main sections and follows an
outline like this:

1. Statement of the problem or opportunity


a. Include data to back up this statement
2. Description of possible solutions
a. For each possible solution, explain the pros and cons, as well as the
following:
a. Benefits
b. Risks
c. Costs (if known)
d. Technical requirements
e. Timeline

3. Recommendation
a. Choose a preferred solution

Preparing to Rebuild the Program | 17


b. Explain why this is preferred
4. Describe how the preferred solution would be implemented
a. What is the approach?
b. Who or what teams would be involved?
c. Milestones for success of the implementation
d. Overall time frame

Socialize the business case to management and to stakeholders who can help
contribute to its success. If you give stakeholders the opportunity to contribute to
creation of the business case, they will be more engaged in its success. The plan
will be more likely to be acceptable if it is feasible for the stakeholders who will
participate in its implementation. After stakeholders have reviewed and either
approved or made changes to a plan, it is less likely that they will back out later.

Active Stakeholder Participation


Numerous studies have proven that stakeholder involvement in planning and
decision making increases ownership and commitment. Participative decision
making (PDM) fosters an environment in which people have a choice whether to
be motivated and contribute.8
To build and support strategic engagement in the implementation of process
change, it is important to remember a few things:

• First, open and transparent communication between teams is necessary to


build trust and maintain engagement. Every stakeholder will bring their own
agenda to the table, so being able to effectively communicate a topic helps
conversations stay on-point and produce results. Management needs to com‐
municate to stakeholders that their work goals need to align with the direc‐
tion of the Plan.
• Next, include stakeholders in all decisions related to the Plan and actively
involve them. Ensure everyone understands all choices and decisions, as
greater understanding leads to ownership of the ideas by the stakeholders.
Similar to the “up-across-and-down” management approach described ear‐
lier, it is more effective to hold smaller, targeted meetings to get buy-in from
individual groups and then cumulatively add support from one group to the
whole. Then, when teams meet as a larger group, everyone is already in

8 Tzu-Shian Han, Hsu-Hsin Chiang, and Aihwa Chang, “Employee participation in decision making, psy‐
chological ownership and knowledge sharing: mediating role of organizational commitment in Taiwa‐
nese high-tech organizations,” The International Journal of Human Resource Management (2010),
2218–2233, http://bit.ly/2J9CE67.

18 | Fixing an Insecure Software Life Cycle


agreement and can discuss the minute differences in agendas and require‐
ments in a cross-team way.
• Lastly, stakeholders need to understand why they are involved, what is
expected of them, and how their contributions are valuable. Their role may
be simply a representative of a team who has been asked to listen and report
back, or it may be that their role is a result of heavy impact to their team.
Each stakeholder should be aware and reminded of how they fit in to achiev‐
ing the common goal of decreasing risk to the business and creating more
secure software.

When stakeholders know how they contribute to the achievement of the Plan,
they will take ownership of that Plan, which will pave a pathway to success.

Working as a Unified Team


A “purple” team (the “red” or offensive security team acting in conjunction with
the “blue” or defensive/detective security team) is a great example of two organi‐
zational teams working together toward a common goal of better security. In
purple team exercises, these teams join forces and create a “same-team mindset”
where each team can think more like the other.
Purple team activities have limitations, such as the following:

• Not addressing the upstream source of vulnerabilities—where are security


bugs coming from, and why are there so many? Root-cause analysis might
not be not performed.
• There is an inherent delay between inception of a vulnerability when it is
created by a builder and identification of a vulnerability by an offensive red
security team.
• Offensive red teams are essentially looking for known vulnerabilities and
exploiting weaknesses in unpatched systems or bad code to achieve infiltra‐
tion of the software or infrastructure.

A security program that relies purely on offensive testing to secure its software is
always going to fall behind in proactivity. Offensive testing of software created
without defensive building is a bit like trying to remove a needle from a haystack
rather than preventing a needle from getting into that haystack anyway.

The Importance of Collaborating as One Team


Security teams can improve software by partnering with builders, not policing
them. Everyone contributes to the security of an organization. All project con‐
tributors are technically on the same team, though each also has different chal‐
lenges.

Working as a Unified Team | 19


There is a goal shared among all stakeholders, and it is also a shared challenge:
each stakeholder is trying to make the organization—not just herself or himself—
successful.
Security teams cannot protect what they do not know about, but software build‐
ers have many of the answers—builders know what libraries are in use and how
everything fits together.
Builders want to do things in a quality way and have no reason not to want to do
things securely, but they most likely do not have the resources to do so—whether
those resources are time, knowledge, or permission. Security teams need to create
more opportunities for the builders to learn security concepts and practices and
get builders more involved in the process of creating secure software.
Most challenges are lessened when all teams work together and communicate,
because the challenge of one stakeholder is quite often solved by the inherent
knowledge of another.
Imagine a security analyst is reviewing an incoming advisory of vulnerability and
notification of release of a patch to fix an issue. The analyst might not know and
might not be able to look up what components or libraries or extensions were
used to create each piece of software in the infrastructure, and thus they might
not be sure whether the advisory applies to the organization. By simply reaching
out to a builder and asking the question “Does this apply anywhere?” the builder
might be able to easily say that no, the vulnerable library was not something that
was used, and the analyst can move on to another task. If the vulnerability were
to apply to the software, the analyst could kick off a remediation process to
ensure that the organization addresses the issue properly. Without that commu‐
nication, it would be much more difficult or even impossible to determine applic‐
ability or impact.
Builders need to be considered part of the same “team” as security specialists.
Builders are a prime resource and on the front lines of defense. If software is
audited for security on a yearly or quarterly basis, the builders are responsible for
security the remainder of the time. Builders should be able to reach out to secu‐
rity teams to discuss or model possible solutions or choices. An open-door policy
between teams allows collaboration for working through and addressing issues as
they arise. Any delays in being able to collaborate will likely cause the builder to
make a choice on their own and move on. The availability of security team
resources to assist with builder questions and concerns is critical to the success of
a one-team mindset.
The entire organization needs to approach its challenges as opportunities—
together.

20 | Fixing an Insecure Software Life Cycle


Discussions, Not Just Bug Submissions
It is always better to prevent a bug than to find it and fix it later, but you should
treat every discovered security bug or flaw as a learning experience.
When output from testing is produced (i.e., a penetration testing report, code
scanning report, etc.), you should view it as an opportunity to teach builders
about adversarial critical thinking. When you include security best practices in a
builder’s personal frame of “correctness” or “accuracy,” and when security is a
core business objective, building securely becomes an important part of a build‐
er’s goals.
For each identified security issue, you should provide a detailed written descrip‐
tion to the builders followed by a related verbal discussion to help reinforce the
concepts. Reports and discussions should not focus on only identifying the issue
itself but should consider what causes the issue, how it can be exploited, the risk
and impact it can have to organizational security, and how similar issues can be
prevented or addressed.
You should hold detailed sessions that extend beyond executive summaries and
separately from broader-scope briefings; for instance, management-level presen‐
tations of findings.
By having deeper conversations, builders will learn the nature of vulnerabilities,
not just where to find and fix them. Builders, in some cases, might realize they
have created the same bug elsewhere, out of scope, or have been repeating the
same mistake time and time again, perhaps by use of a function in a library or
just out of habit. This can cause a slew of improvements throughout other areas,
leading to wide-ranging improvements.

Positive Interactions
Keep in mind that the coding process and creation of software is rather personal
for some people, and builders can sometimes see criticism as a personal attack.
When discussing bugs and giving feedback, make sure any criticisms are con‐
structive. Here are some helpful tips on making interactions more positive:

• Focus on the issue, not the person who caused it.


• Be sincere when giving praise, and acknowledge the effort, not the ability.
• Use the “compliment sandwich.” Start with a positive comment, then provide
constructive criticism, then close with another positive comment.

In whatever way feedback is delivered to a builder, the goal should be to collabo‐


rate and agree on a solution. Be specific about what is expected, what needs to be
done, and ensure that the builder understands why it needs to be done. Remem‐

Working as a Unified Team | 21


ber, this is not an attack on the person behind the bug—feedback is meant to help
create more effective team members and a more successful organization.

Rotating Work Assignments


Job rotation occasionally moves employees around from team to team, partici‐
pating in different tasks to promote variety and gain experience. Job rotation
gives an employee the ability to experience other parts of the business and learn
skillsets different from their own. A rotational assignment is usually for a finite
time with a specific goal in mind.
Rotating software builders into security assignments can be particularly power‐
ful, but the goal is not complete cross-training. Instead, the goal should be to
expose builders to real-world scenarios and emphasize the skill of “thinking like
an attacker” or “thinking like a responder.” Adversarial thinking will help build‐
ers create more effective and automated defenses and responses for things attack‐
ers would do.
For example, a builder can sit with a security analyst in a Security Operations
Center (SOC) and watch incoming attack activities as well as organizational
response in real time. For the builder, this experience can shed light on the con‐
stant, hostile attacks occurring against software. SOC team members can directly
explain to the builder what is occurring in terms of attacks or logs or indicators
of compromise (IoC). When a security incident occurs, the builder can learn
what caused it and consider how code and design could have prevented, sand‐
boxed, or decreased the impact of attack.
Through job rotation, the temporarily rotating builder would hopefully learn to
consider various defenses, such as addition of traps to contain attackers or the
generation of additional logging for critical functions. By working with an SOC
response team, a developer could better understand what information is needed
to help detect an attack and conduct a successful investigation, as well as how
software can help contain and withstand attack.
A developer could also sit with the offensive red team to learn more about the
anatomy of an attack, what attackers are looking for, and how defensive design
can prevent attack. By working with the adversarial testing team, a builder can
learn concepts like how to make software a less-attractive target, why it is impor‐
tant to separate administrative interfaces from user interfaces, and why it is so
important to handle untrusted user input with care.
Knowledge gained through job rotation should be brought back to the builder’s
normal job duties and shared with other team members. Builders who have par‐
ticipated in rotating work assignments can discuss what they have learned, learn
from one another, and teach others.

22 | Fixing an Insecure Software Life Cycle


The downside of rotating builders into security roles is that of temporary lost
productivity from both sides. Security must spend more time explaining what
they are doing, and builders are not actively developing. However, rotation can
lead to better performance of builder duties and even potentially better time-to-
detection or time-to-resolution of incidents. This is an effort striving for quality,
not speed or cheapness. There is always a trade-off.
Conversely, you can rotate security personnel into builder teams, acting as an
“on-call” security liaison or sitting with the team to see how software is created.
This can lead to improvements in style guides and promote security discussions
pertaining to decisions made by the builders as they plan and create the software.
Job rotation can have a net positive impact on security, but you need to do it in a
way that is sustainable to the organization. Limited engagements with specific
frequency are ideal, and an organizational policy should dictate that rotations
occur in order to make the activity mandatory. If rotation is optional, teams
might never be able to “spare” an employee for rotation.

Embedded Security Liaisons


For organizations capable of doing so, a dedicated, embedded security liaison
might be a quick way to jumpstart conversations and build rapport between
builder and security teams. When the security liaison is seen as being (or perhaps
actually is) on the same team as the builders, it is more likely that the builders
will be exposed to security topics, issues, and concerns, and the security liaison
can provide immediate feedback for design choices and other decisions made
during both formal and informal meetings. Many critical choices are made by
shouting over cubicle walls or at a team lunch, so it is important that the liaison
works closely with the builder team(s).
A high level of dedicated interaction via a liaison has a caveat: headcount for
security will be either diminished by one or a new employee will need to be hired
to fill the liaison role. There is also potential for decreased productivity for a
dedicated security specialist when their help is not needed to handle builder
questions and decisions, though any downtime can be spent creating training,
improving the style guide, or helping the primary security team. This represents
another compromise among speed, quality, and cost.

Setting Expectations for Stakeholders


Security professionals have an idea what a secure end state should look like, but
the builder and PM teams need to understand exactly how to get there.
Plan time to work with the other teams to set expectations, which will probably
involve lots of explanatory conversations with builders, PMs, and product man‐
agers.

Setting Expectations for Stakeholders | 23


Employees and stakeholders will need to be fully aware of any steps that need to
be taken, and organizational policy defines the things your organization must do.

Using Organizational Policy to Create a Need


Policy is an important instrument for implementing any change or requiring a
security control. Policy should be a tool in every security practitioner’s arsenal
because it is a powerful way to elicit change.
It is often mandatory requirements originating from policy that can be significant
drivers of change. The same is true for contractual obligations and promises
made to customers or third parties. Getting a security requirement into a policy
can be a critical step to ensure organizational compliance with the policy state‐
ment. (You can find a sample policy statement later in this report.)
Extra time and effort will be necessary to a secure the SDLC, so it is imperative to
have leadership support for allowance of any additional resources needed. The
policy should mandate that security teams are a common stakeholder in all
projects and are involved as early as possible in the SDLC.
Following are some actionable recommendations for using policy to generate the
need for security in the SDLC:

• Before implementing new policy or changing existing policy, engage stake‐


holders to agree on the terms and ensure future compliance. Everyone
should clearly understand and support the reason changes are necessary and
how the change is pertinent to success of the organization.
• The security policy should be extremely clear and easy to understand. Suc‐
cess will be limited if stakeholders do not know what they need to do to com‐
ply.
• Ensure that the organizational policy lays out all relevant stakeholders and
what their responsibilities are. In addition to socialization, training on new
policy is also going to help solidify understanding and compliance.
• Policy language is very important, so use statements phrased as “only” or
“always” or “must”; the use of words like “should” or “may” or “generally”
make policy less effective and capable of being circumvented.
• Lay out a process for identifying, handling, and tracking any exceptions that
need to be made. Do not lose sight of exceptions that have been granted, and
periodically check to make sure they are still required. A path to compliance
with policy should be created and carried out in conjunction with any excep‐
tion that is filed, whenever possible.
• Change is going to take some time to occur, so follow the exception process
and be sure to manage and track any risk related to noncompliance. Under‐
stand that new projects will be capable of following the policy more closely,

24 | Fixing an Insecure Software Life Cycle


whereas legacy software might either take time to comply—or never com‐
pletely comply—with new policy.
• A policy that is not enforced is ineffective, so stand firm on mandates and
communicate any issues or concerns that arise to management. If the policy
is unenforceable, you might need to make tweaks to the new policy, because
a policy that is not enforced or cannot be enforced will be resisted by stake‐
holders.

Almost everything an organization does starts with organizational policy, so


include requirements for audits and be sure to create processes and procedures to
support the policy.
Most important, make sure that every element of SDLC policy, process, and pro‐
cedure documents produce artifacts that you can present as evidence to auditors.

Compliance
Compliance itself is not an end state; rather, it is just a validation of the organiza‐
tion doing what it says it does. The need for compliance, much like policy or con‐
tractual obligations, can be a significant driver of change in an organization.
Compliance can also open doors to additional types of customers and revenue
streams.
Compliance will not dictate how you should implement security or whether you
need to perform things like automated code reviews. Instead, compliance will
provide a question such as, “What controls are in place to ensure the integrity of
infrastructure?”
The organization will answer these questions by explaining something along the
lines of: “Software must be created following organizational software develop‐
ment life cycle policies, including policy A, B, and C,” and then the organization
will provide the “A, B, and C” policies to the auditor as relevant artifacts proving
that the policies exist. An auditor will then ask for examples to support the claim
that a policy is being followed and expect that you provide documentation to
prove that you are adhering to the processes and procedures that support the
policy.
This is where policy becomes so important and useful: an auditor will drive
change to meet each security requirement laid out in policy.
If policy states that you must perform automated code reviews before every
release, an auditor would need to ask for proof that such a policy statement is
being followed. Therefore, the organization will need to produce copies of
reports showing that code reviews were conducted (which means that code
reviews will occur).

Setting Expectations for Stakeholders | 25


Although the compliance requirements themselves do not normally state that
code reviews need to be performed or how, the organization did state that it does
such reviews via its policy, so the organization must prove that these reviews are
being performed.
So, using policy to mandate security makes it critical for the organization to per‐
form security tasks to meet compliance. This is one way to implement policy to
the advantage of security. Policies are powerful!

Compliance planning
At the beginning of the SDLC, stakeholders must determine what is being devel‐
oped and to what compliance or regulatory requirements it must adhere. You
should base compliance planning on the software end user and target market.
The target market and intended users of the software should help identify com‐
pliance goals; for example, if a target market is related to healthcare, compliance
with the United States Health Insurance Portability and Accountability Act
(HIPAA) might be a requirement. If the organization plans to do business with
the European Union, the General Data Protection Regulation (GDPR) regulates
that specific security controls around user data will need to be satisfied. Review
the business plan for the project to fully understand the user base and overall
objectives.
If any commercial or regulatory compliance is a requirement, work with the PM
to create a compliance timeline.
Think backward from a compliance goal, answering important questions such as
the following:

• Who is likely to use the software?


• What are the laws or regulations in our users’ countries?
• Are there unanticipated users that might benefit?
• What data will be stored and processed by the software?
• To what other resources will the software connect?
• What funding will need to be procured to hire third-party auditors?
• How long will it take to perform the audit?
• When does compliance need to be achieved?

Compliance might not be possible with the initial prototype, so you can create a
roadmap to compliance in lieu of a fully compliant and tested solution on launch
day. For existing software, a roadmap to compliance might be similar to the
“phased goal plan” discussed earlier. A “soft launch” can then occur, via silent
rollout to a few willing customers, which will make certain stakeholders happy.

26 | Fixing an Insecure Software Life Cycle


Knowledgeable Humans
To bring about change, the organization’s cultural ideal of “production-ready,”
“quality,” and “correct” software needs to shift to definitions that include security.
The entire organization needs to hold security and privacy as core values shared
by all teams, and it is the security team’s job to direct and impart these values—to
stakeholders, peers, management, the board, and direct reports. Everyone who
can help make culture change happen needs to embrace the change in culture.
Consider what kind of training can be offered to the stakeholders, builders, and
other teams as well as how to make it available. Here are some questions to ask
when considering software security awareness training:

• Will new processes require training?


• Can the security team take time to offer in-house training?
• Is the security team equipped to train others?
• Is there training that you can purchase and make available throughout the
enterprise?
• Should a third-party develop custom training?

In terms of in-person versus online training, the ideal scenario is dedicated time
spent in class with a teacher who can answer questions and explain concepts to
full comprehension. Logical and monetary limitations might not make such
dedicated training feasible. Some teams might prefer on-demand virtual training
due to geographic limitations or other needs. Find out what type of training and
what level of interactivity works for stakeholders by experimenting or just asking
the teams which they believe will be the most effective for them. You should not
perform training to mark off a checkbox; rather, you should view it as an impor‐
tant part of fulfillment of employee goals.

The Development Style Guide and Standard Libraries


Much like organizational policy, an effective way to make security unavoidable is
to build it in to the development style guide9 and include security in any standard
libraries that builders use, especially code developers.
Creating and using a style guide for software design and development can aide in
teaching best practices and settling disagreements between builders or DevOps
teams.

9 A style guide (sometimes known as a pattern library) was originally used in print design for achieving
consistency of logos and imaging, but guides have become useful for coding as a method for ensuring
development consistency and quality across projects and teams.

Setting Expectations for Stakeholders | 27


Examples of items to include in a style guide include the following:

• How and when to use comments


• Tabs or spaces for indentation (and how many spaces)
• Appropriate use of white space
• Proper naming of variables and functions
• Code grouping an organization
• Patterns to be used
• Patterns to be avoided

You can use a style guide to the advantage of security teams by consistently pro‐
viding guidelines for standards in coding, standardizing the design and build
process, and detailing common interactions between software functions.
Generally, a style guide is the result of agreement of opinion on how a coding
topic should be done, so not every builder will agree with every detail. However,
builders must follow the coding conventions at the risk of creating problems for
the rest of the team.
You can include security in a style guide by, for example, adding statements
regarding sanitization of input from users or methods for avoiding buffer over‐
flows, which will give builders a way to implement buffer overflow prevention
without much consideration or additional discussion. This can limit the time and
resources needed to develop securely and serves as an educational tool for newly
hired builders. Use of a style guide could likely also curtail other nonsecurity
bugs or design flaws.
Because any guide is a living document, style guides can and should change as
new versions of programming languages are released, new programming lan‐
guages are put into use, or new vulnerabilities are possible. Security teams should
regularly review and update a style guide to address emerging attack or defense
techniques.

You can find a sample, language-agnostic set of secure coding practices


in the Open Web Application Security Project (OWASP) Secure Coding
Practices.

Automated code scanning


Sometimes referred to as static code analysis, you can use automated code scan‐
ning tools to detect known, easy-to-find bugs like buffer overflows or unchecked
user input.

28 | Fixing an Insecure Software Life Cycle


Any automated tools have both benefits and limitations. The benefit of code
analysis tools is that you can run them repeatedly at various stages of the SDLC
and can help pinpoint areas of code that you might need to review more thor‐
oughly. You should supplement automation with knowledgeable humans who
have adequate time to review code.
False positives and false negatives from automation are to be expected, and this is
one risk associated with the use of such tools. Static code analysis is similar to a
“spelling and grammar check,” a comparison which should help explain the
inherent constraints and valid use cases.
Overreliance on any automation without some human validation is not a good
idea, but inclusion of automated code analysis tools as part of a development
workflow can be a quick way to start recognizing security bugs without dedicat‐
ing additional person-hours to code review that technology can perform ade‐
quately.
Despite its limitations, static code analysis can be a “quick win” improvement for
any SDLC program.

Checklists
The best way to set expectations and track results is via the use of checklists.
Inclusion of security teams during the early SDLC phases means assurance can
be created via checklists and formalized procedures. Checklists help everyone
understand what to expect and what is needed.
One way to design a checklist is to start with a goal and work backward to the
starting point. Imagine the ideal end state for software:

• What does it look like, and how does it behave?


• Who has vetted its functionality?
• What does it do and, more important, what does it not do?

Think about what the organization and security teams need in order to feel com‐
fortable with software risk before approving progression to the next SDLC phase.
You can find sample checklists at the end of this report. For additional ideas
about where to start with a checklist, search for sample SDLC phase checklists
online and tailor them to your organization’s needs.
With any checklist, certain steps might not be applicable to every project, but it is
important to not remove steps from checklists so that a step remains there for the
next project to which it is applicable.
New software requires additional and differing steps versus updates to existing
software, so multiple sets of checklists might be needed.

Setting Expectations for Stakeholders | 29


Be sure to note who the stakeholders are for each checklist item and what steps
are required to complete the task. What kind of artifacts should you create by a
process or procedure? When should you consult security teams and at what
points should they be required to approve before moving to the next stage?
The next section of this report provides example checklists as a starting point for
tailoring steps and requirements to your organization.

Sample Checklists and Planning Documents


When PMs, management, builders, and security teams understand what exactly
needs to be done and when, it makes the software life cycle scalable and manage‐
able. By laying out required steps early via well-documented checklists, other
teams will not feel as if security teams are causing “scope creep,” and inclusion of
security stakeholders or performance of security activities will not hurt deadlines
or launch dates.
Each SDLC phase activity can utilize a checklist, from planning to design, and
architecture to development, and implementation to testing, and eventually
operational state.
This section shows you how to create checklists for different SDLC phases and
provides sample checklists for you to build upon. These samples are not exhaus‐
tive; they are meant to show you what kinds of items would appear in each
checklist. The best checklist is one that is tailored specifically for your organiza‐
tion based on your observations and knowledge about a specific environment.

Handling Checklists
Security teams need to provide every checklist as soon as possible at the begin‐
ning of a project to the PM, so the master project plan will include the security
checklist tasks. As many checklists as possible should be prepared and provided
before a project begins.
You might want to consider running through mock scenarios and socializing the
checklists internally before handing them over to the PM. It is a good idea to
meet with the project management organization (PMO) and explain the require‐
ments, format, and any other new processes that will need to be followed.
Create a simple bullet list of what tasks need to be completed and then work with
a PM later to turn it into a spreadsheet or standalone project plan document that
a PM can easily import into each master plan. You can tailor checklists by remov‐
ing or adding requirements based on need or risk. Be sure to document and
retain any changes or exceptions to a checklist for later review.
Initially, task duration might not be defined or clear, but a best guess is good
enough at first for project planning purposes. For example, if the task is adding

30 | Fixing an Insecure Software Life Cycle


an asset to a vulnerability scanning tool, the turnaround might be a few days, but
less than a week. Timeline estimates will help the PM lay out the project plan,
and it is acceptable if the duration is not exact. Talk to process owners and pro‐
cess implementers to help narrow down time frames for as many tasks as possi‐
ble. Estimated task timelines become tuned with process maturity.
If possible, you might want to include the specific team or role that will complete
each checklist task, but it is not necessary. If you do not know who is responsible
for a task, stakeholders and roles will be defined as the SDLC process matures.
Keep in mind that task responsibility might change between software efforts.
You should add buffer time to estimated time frames to account for any delays or
problems that might occur. PMs sometimes add buffer time to the project plan
anyway, so find out whether the PM will buffer the estimates to avoid adding too
much.
Tell the PM exactly how to initiate each action that must be completed. The per‐
son who initiates or performs an action should follow a standard operating pro‐
cedure (SOP) for doing so. For example, to add a new asset to a vulnerability
scanning tool, the process initiator will need to know what information to gather,
how and where to submit a ticket containing the IP addresses and physical/logi‐
cal location, and how and where to route the ticket to the scanning administra‐
tion team. Then, the SOP should have a step to provide a confirmation of
completion artifact (e.g., a “procedure is complete” email). The PM needs to
know precisely where the SOP is located and who needs to complete the SOP, in
addition to the estimated time to completion of that task.
Each checklist might have items that do not apply to every project, and that is
fine. When a checklist item does not apply to a certain project, the item should
not be removed, because it might be required for future projects. Document any
checklist items that do not apply to the current project and why, and centrally
store that document as an artifact for later reference.
Each checklist itself is an artifact that you can reference later for compliance or
risk review. Following are some tips for handling checklists:

• Tailor the checklist(s) to each project:


— Not every control will apply
— Some new controls might need to be added
• Provide all checklists up front to the PM
• Be prepared to explain how to initiate each process that is to be followed
• Be prepared to explain what indicates that a process is complete
• Keep track of who is responsible for each task
• Completed checklists are compliance artifacts

Sample Checklists and Planning Documents | 31


• Store artifacts in a backed-up, central repository
• Store artifacts for an adequate amount of time
• Protect artifacts based on privacy requirements of the organization

Sample Planning Phase Checklist


The planning phase of the SDLC should have a checklist to capture security-
related planning steps, in order, such as the following:

1. Engage security to participate in the project


2. Define the project scope
3. Identify compliance goals and determine a compliance timeline
4. Define security-specific use cases
5. Security review of all nonsecurity use cases
6. PM generates concept, timeline, and budget; everyone agrees or makes
changes
7. Obtain the security team’s approval before moving to the next phase

If any step has a prerequisite step that must be completed, be sure to make note
of it. The last item in the above list is the most important because software should
not proceed through the phases of the SDLC without the security team’s
approval.

Sample Privacy Questionnaire


A privacy questionnaire is a great way to learn about the data being protected.
The legal team should be included as a stakeholder in all things related to data
privacy because legal will have the most concern in this area.
The privacy questionnaire is a “living document.” Keep it up-to-date throughout
the SDLC. The questionnaire should help define details such as the following:

• Who are the data privacy stakeholders?


— Who knows about the data that the application will store and process?
— Legal is always a stakeholder in privacy, but engineers and architects are,
too.
• What sensitive data is kept?
— If there is not already an organizational data classification policy that
describes types of data with a rating or criticality and prescribes controls
that must be present for each type of data, now is a great time to imple‐
ment one.

32 | Fixing an Insecure Software Life Cycle


• Where will sensitive data be stored and transmitted?
— Where sensitive data will be stored and transmitted determines the foot‐
print that should be protected with the most resources.
— Do not spend more time and effort protecting inconsequential areas than
is necessary.
• What mitigations will be used to protect that data?
— Will the software or its architecture encrypt database fields, implement
Internet Protocol Security (IPsec) tunnels, or make use of a demilitarized
zone (DMZ)?
• What is the risk if data is exposed?
— Who would be affected?
— What is the opportunity for attackers?
• What needs to happen if there is exposure?
— Do users need to be notified when a breach occurs?
— Are there additional compliance requirements for international deploy‐
ments?

Sample “Design/Architecture” Phase Checklist


Design and architecture diagrams should be required artifacts, and you should
also create data flow diagrams when relevant.
Security teams must review and, if necessary, adjust all network and data flow
diagrams based on risk and/or practicality. Security reviews should consider
design and architecture topics such as: “What internal connections will be
required?” or “Will a web application firewall be required?”
You should include all interconnections on diagrams. Consider what other
resources (such as operational monitoring systems, email alerts, etc.) will inter‐
face with the infrastructure:

• How will users access the environment?


• How will administrators access the environment?
• Will there be a bastion host and, if so, will there be a bastion host in every
physical location?
• What assets need to be within the DMZ?
• Is the environment accessible from the corporate virtual private network
(VPN)?

Sample Checklists and Planning Documents | 33


• Does an asset need to communicate with every node to which it is connec‐
ted? Are all connections bi-directional? What protocols and ports are
required?

A software prototype must include all security requirements for design and
architecture. The checklist for this phase must minimally include the following
steps:

• Create network diagrams


• Create data flow diagrams
• Review diagrams with security
• Incorporate the security team’s feedback, as needed
• Ensure capacity is allocated for security-related functions (e.g., log server,
span ports available on a switch for network intrusion detection, separate
virtual LANs for storage and admin traffic)
• Security has reviewed vendors prior to selection

As with all phases of the SDLC, the design and architecture of the solution must
be approved by the security teams before moving on to the next SDLC phase.

Sample “Development/Implementation” Phase Checklist


The development and implementation checklist is a mix of things that the build‐
ers need to do, as well as verification testing that the QA and security teams need
to perform.
This is the phase in which all design, architecture, and requirements come
together and are actually “built” into software. DevOps activities, if used, will be
incorporated during this phase.
Critical operational controls like monitoring and patching need to be in place
when the project is eventually deployed to a production environment. The orga‐
nization should not be able to push off security features to later; security assur‐
ance must be in place and working before a production deployment.
During this phase, certain elements from design and architecture might need to
change if they simply do not work in practice. Suppose that encryption of all
storage traffic ends up being too costly in terms of performance. Perhaps you can
move the storage network to a separate port on the switch or even to a dedicated
switch with further restricted access and the organization can accept the risk of
sending unencrypted storage traffic.
You need to document built-in redundancy and mitigations, and you need to
review any design or architecture changes after the design/architecture phase by
all stakeholders, especially the security team.

34 | Fixing an Insecure Software Life Cycle


After verification has occurred and the security team is comfortable approving
deployment, a push to production can occur. If there are any outstanding con‐
cerns, you can perform a “soft launch” for a limited number of users.
Following is an example of minimal requirements that you must satisfy prior to
implementation or launch:

• Prototype design must address both security and functional requirements


• Users have been created within the application for authenticated penetration
testing
• Logs are being sent to a centralized server for processing and alerting
• A secure baseline is in use for all operating systems and applications
• Procedural and process documentation has been created and approved
• A patch management process and procedure have been created
• Monitoring and related Access Control Lists (ACLs) are put in place
— Intrusion detection and prevention mechanisms are configured and
tested
— Network monitoring traps are set up
• Vulnerability scans and internal penetration testing have been performed
• Remediation of security flaws, bugs, and vulnerabilities has occurred, based
on testing
• Obtain the security team’s approval before moving to the next phase

Sample Policy Statement


As discussed earlier, organizational policy can play a critical role in creating a
more secure SDLC. Following is a sample policy statement that includes a secu‐
rity requirement that you can adapt to meet your organization’s specific needs.

Title
Software Security Requirements Policy

Date of Implementation
January 1, 2017

Change Log

Date Editor What changed?


January 1, 2017 Jane Smith Initial document

Sample Checklists and Planning Documents | 35


Date Editor What changed?
January 1, 2018 Drew Wilson Added new definitions

Next Review Date


January 1, 2019

Policy Owner
Susan Stamp, ISSO

Approved By
Sam Brown, IT
Owen Taylor, legal
Gwen Richards, development engineering

Purpose
This policy defines mandatory activities to be implemented during the planning
phase of the Software Development Life Cycle (SDLC) to achieve a high level of
security assurance.

Scope
This policy applies to all software developed by the organization, including new
and existing software projects, as well as custom code used to integrate third-
party software implemented by the organization.

Policy Statement
The security team must be engaged at the start of any new project during the
beginning of the Planning phase of the SDLC.
All software security requirements provided by the security team must be incor‐
porated into the design, architecture, development, and deployment of the soft‐
ware.10 Each requirement must be testable and verifiable as being implemented.
Any potential exceptions to requirements must be discussed, documented, and
approved by the security team. Exceptions must be reviewed on at least a yearly
basis to determine if they are still required. Exceptions should be remediated
whenever possible.
The security team’s approval of all requirements is required prior to software pro‐
gressing to the design phase of the SDLC. If full approval is not given, steps must

10 This allows the security team to alter the requirements, as needed.

36 | Fixing an Insecure Software Life Cycle


be taken to ensure approval prior to proceeding, or any contingencies on appro‐
val must be noted and handled as soon as possible.
All artifacts related to security activities, including completed checklists and a list
of requirements, must be stored centrally in the main compliance archive.

Definitions
SDLC—Software Development Life Cycle
VLAN—Virtual Local Area Network
IT—Information Technology
Approval contingency—Any specific requirement that is being permitted to be
completed after overall approval is given. Assumed to be capable of being com‐
pleted within a finite amount of time.

Related Documents/Standards/Procedures
Software Style Guide: http://xyz.corp/softwarestyleguide/
Secure Baseline Standard: http://xyz.corp/securebaseline/
Security Team Formal Approval Procedure: http://xyz.corp/procedures/
SDLC_phase_approval.html

Conclusion
Although it is ideal to develop software securely from its inception, it is possible
to evaluate existing processes and fix an insecure software life cycle. Getting
security teams involved as early in the SDLC process as possible cuts down cost
and time required to manage and minimize risk, especially when security teams
understand how to work successfully across teams and gain buy-in from your
organization.
Through understanding the unique goals and challenges of organizational stake‐
holders, analyzing deficiencies in existing processes, planning for change, and
formalizing security within the SDLC program, you and your team can help to
assure that security is built into every piece of your organization’s software.

Conclusion | 37
About the Author
April C. Wright is a hacker, author, teacher, and community leader with over 25
years of breaking, making, fixing, and defending global critical communications
and connections. She is an international speaker and trainer, educating others
about personal privacy and information security with the goal of safeguarding
the digital components we rely on every day. A security specialist for a Fortune
15 company, April has held roles on offensive, defensive, operational, and devel‐
opment teams throughout her career, and been a speaker and contributor at
numerous security conferences including BlackHat, DEF CON, DerbyCon, Hack
in Paris, DefCamp, ITWeb, as well as for the US government and industry organ‐
izations such as OWASP and ISSA. She has started multiple small businesses
including a nonprofit, is a member of the DEF CON Groups Core Team, and in
2017 she cofounded the Boston DEF CON Group DC617.