Disruption Opportunity Special Notice

DARPA-SN-18-02
Understanding Group Biases (UGB)

I. Opportunity Description
The Defense Advanced Research Projects Agency (DARPA) Defense Sciences Office (DSO) is
issuing a Disruption Opportunity (DO) Special Notice (SN) inviting submissions of innovative
basic or applied research concepts in the technical domain of understanding group biases. This
DO SN is issued under the Program Announcement for Disruptioneering, DARPA-PA-18-01.
All proposals in response to the technical area(s) described herein will be submitted to DARPA-
PA-18-01 and if selected, will result in an award of an Other Transaction (OT) for prototype
project not to exceed $1,000,000.
A. Introduction
Cultural models i are models that are shared among individuals within groups. These models
shape social identities by creating interpretive frameworks that in part help to define those
groups. Cultural models impose structure on the world by classifying objective phenomena that
might otherwise be incoherent, inconsistent, or inexplicable into subjective cultural categories —
things, features, values, actions, events— where they are then interpretable within a larger
cultural logic. As particular categories are often specific to unique groups, cultural models often
act as a “glue” that binds a group together by providing its members with a collective tendency
to interpret the world in culturally-specific but shared ways. Examples of categories that vary
among cultural models include what foods are edible ii, who is a preferred marriage partner iii,
what is a sacred value iv, or even what might constitute physical or mental illnesses v. In this
regard, cultural models can also be thought of as a collection of shared group biases, where bias
is defined as an inclination or a tendency to act or think in a certain way vi. These biases help
explain how the same world can often be interpreted very differently by different people and
groups, i.e., because the world is lived in and interpreted through these models that
anthropologist Clifford Geertz called “webs of significance.” vii

Social scientists, including anthropologists, have traditionally tried to capture these cultural
models using largely qualitative methods, of which ethnographyviii is perhaps the best-known
example. Ethnography is an approach that often relies on interviews, longitudinal engagement,
and participant observation ix to try to capture “the native’s point of view.” However,
ethnography suffers from significant limitations. This largely qualitative, often in situ study of
humans—which results in “thick data” x —is often resource-intensive, requiring significant
amounts of time and often relying on a single researcher’s perspective. As such, ethnography is
frequently limited in scope, scale, and speed. Additionally, interviews and participant
observation may be subject to significant selective reporting from those being studied, as groups
often are unaware of—or unwilling to acknowledge—their own biases and cultural models. This
may be, in part, because many models are implicit and, in part, because group members, aware
that others may have different models, often seek to manage their impressions and reputations.
Finally, ethnographic research often relies on limited data collection capabilities, necessarily
calling into question how exhaustive or robust the conclusions may be.

DARPA-SN-18-02 UGB Disruption Opportunity 1

In the context of these limitations, ethnography’s “thick data” stand in contrast to current “big
data” approaches, which leverage the explosion of digital text and other media and are
increasingly common in computational social science and network science. However, just as
there are limitations to ethnographic methods, there are concerns that these computational
approaches often fail to capture important social contexts, subjective interpretations, and cultural
nuances for providing a deeper understanding of human beliefs, perspectives, and identities–
without which our ability to explain (and potentially predict) many social phenomena will
remain unnecessarily hampered.
B. Objective/Scope
DARPA hypothesizes that there may be new opportunities for overcoming this current trade-off
between qualitative, “thick” data and quantitative, “big” data. These opportunities could lead to
a revolution in understanding group biases, i.e., developing capabilities to capture different
cultural models at new speeds and scales, in part by leveraging big data via media sources and
cultural texts that may contain important latent group biases, which can help explain how
different groups interpret the same world differently.

As some recent work has identified, machine-learning algorithms often come to learn and reflect
the biases that are implicit in the data on which those algorithms are trained xi. Many of these
biases are hidden in the datasets, often being revealed by machine learning outputs that reflect—
often unintentionally—certain social categories, judgments, or associations assigned to events or
topics as evidenced by the algorithm’s performance in classifying new data. Further, such biases
also exist in the outputs of algorithms relying on fixed ontologies, such as semantic role
labeling xii and event coding xiii. As these ontologies are created by humans, they necessarily
reflect the biases of their designers, which may be at odds with the biases of the datasets they are
used to analyze xiv. While social biases are clearly inappropriate and potentially dangerous in
cases where machine-learning algorithms are used to make important medical, personnel, or
legal decisions xv, this same tendency to reflect biases that are implicit in a dataset or algorithm
might be used to more effectively identify different cultural models. Hence, in different
situations, machine learning could reveal biases of different cultural models within large
datasets, and it could also reveal cultural models of those who designed the ontologies that are
used to code those datasets. From this perspective, the otherwise unwelcome bug of machine
learning to pick up non-obvious biases from large datasets may—if used and validated
appropriately—be leveraged as a feature to give us deeper insight into human variability and the
diversity of “webs of significance” that define our worlds through our different cultural models.
Hence, the objective of the UGB program is to test whether there may be new capabilities for
radically enhancing the scale, speed, and scope of automated ethnographic-like methods for
capturing group biases and cultural models from increasingly available large digital datasets.
C. Structure
Proposals submitted to DARPA-PA-18-01 in response to this DO SN must be UNCLASSIFIED
and must address two independent and sequential project phases (a Phase 1 Feasibility Study
(base) and a Phase 2 Proof of Concept (option)). The periods of performance for these phases
are 6 months for the Phase 1 base effort and 12 months for the Phase 2 option effort. Combined
Phase 1 base and Phase 2 option efforts for this DO should not exceed 18 months. The Phase 1
(base) award value should not exceed $150,000. The Phase 2 (option) award value should not
exceed $850,000. The total award value for the combined Phase 1 and Phase 2 is limited to

DARPA-SN-18-02 UGB Disruption Opportunity 2

$1,000,000.

Phase 1 objectives are to secure access to relevant, publicly available datasets; evaluate the
appropriateness of machine learning approaches; determine and justify the necessary level(s) of
analyses for datasets and cultural model detection; provide early evidence of the technical
credibility for use case scenario(s) (described below) for their algorithms and prototypes, and;
outline specific validation approaches.

Phase 2 objectives are to develop and test algorithms on identified datasets to derive cultural
models and group biases from those datasets; instantiate algorithms and prototypes in one or
more use case scenarios; validate algorithms; and transition prototypes and/or identify further
promising research avenues for future UGB capabilities.

All proposals must address each of the following Research Project Objectives. Proposals that
fail to address all of the objectives may not be considered for funding.

Research Project Objectives

- Data: Proposers shall identify current or potential datasets that they can use for their
approach. DARPA anticipates that all data used will be publicly available data, which have
been or will be ethically collected in ways that allow the performer to train their algorithms
for purposes of the program, but also enable a computationally reproducible workflow for
independent evaluation. Data should both avoid personally identifiable information (PII)
concerns as well as be available for another researcher wishing to computationally
reproduce xvi a performer’s results. If proposers wish to use a sensitive or proprietary dataset in
Phase 2, the proposal should clearly identify and justify the rationale for this approach and
propose mitigation strategies for ensuring computational reproducibility while avoiding
unnecessary collection or use of PII.
- Machine Learning: Proposers shall identify one or more machine learning methods that will
be used to investigate the hypotheses described above. Proposals relying on large volumes of
annotated data must be clear about how such data will be acquired. Explicit model
assumptions (e.g., variable independence, normally distributed data) should be detailed, as
well as their anticipated effects on approaches and their interplay with biases implicit in the
data. Anticipated feature representation(s) of the data should be similarly detailed and
justified.
- Levels of Analysis: Proposers shall identify the anticipated levels of analysis at which their
algorithms will capture various biases. For example, proposers should identify whether they
anticipate using biases in datasets in order to capture cultural models at a purely semantic or
syntactic level, an ontological or typological level, a conceptual or nomological level, or some
combination thereof.
- Use Case Scenarios: Proposers shall identify one or more potential use case scenarios to
which they will apply their approaches and which they may use for validating their methods
and prototype algorithms. These scenarios should credibly address one or more of the
following capabilities:
• Incorporate Perspective: The prototype algorithm helps quantify the diversity of
interpretations of an event or message in the world, helping to add nuance to event coding
datasets or social media analyses

DARPA-SN-18-02 UGB Disruption Opportunity 3

 • Improve Accuracy: The prototype improves the accuracy of event coding algorithms by
leveraging the degree to which different cultural models reach similar or divergent
conclusions about the occurrence and interpretation of an event or message
• Quantify Change: The prototype algorithm documents how group biases evolve over
time and identifies quantitative indicators that may signal the emergence, fusion, fission,
and/or blending of identities and cultural models
• Enhance Modeling: The prototype algorithm improves agent-based modeling by
enabling simulations using agents with empirically-derived biases that can better simulate
different cultural models and social identities xvii

Proposers shall provide one or more approaches to validating their work and deliverables for the
program. Validation approaches should reflect the primary goals of the program as well as the
proposers’ approach to the Research Project Objectives. Validation should include at least one
quantitative measure for comparing the performance of the proposer’s deliverable(s) against
current capabilities and baselines.

Human Subject Research (HSR): DARPA expects that proposed Phase 1 work will not include
HSR. Proposals with HSR work in Phase 1 may be excluded from consideration for funding.
DARPA further expects that Phase 2 work will also not include HSR. However, proposed Phase
2 work that may include HSR and, therefore, will require Human Subjects Research Protection
and IRB review (to include HSR that is determined to be “Exempt”) should be clearly identified
and justified in the proposal, and credible mitigations for impacts on the proposed
timeline/research schedule should be offered and justified.
D. Schedule/Milestones
Proposers must address the Research Project Objectives, metrics, and deliverables, along with
fixed payable milestones requested in this Special Notice. Proposers must provide a technical
and programmatic strategy that conforms to the entire 18-month program schedule and presents
an aggressive plan to fully address all program goals, metrics, milestones and deliverables. The
task structure must be consistent across the proposed schedule, Task Description Document
(TDD), and Vol. 2 – Price Volume. If selected for award negotiation, the fixed payable
milestones provided in Section 9.c of the Vol. 2 – Price Volume will be directly incorporated
into Attachment 2 of the OT agreement (“Schedule of Milestones and Payments”). Please see
the sample OT for Prototype provided as an attachment to DARPA-PA-18-01.

For planning and budgetary purposes, proposers should assume a program start date of June 1,
2018. Schedules will be synchronized across performers, as required, and monitored/revised as
necessary throughout the program. Anticipated milestones for Phase 1 and Phase 2 are the
following:

Phase 1 Milestones:

M1 Acquire and demonstrate accessibility of UGB datasets. If the Phase 2 approach is

dependent on HSR data, synthetic data may be used. (Month 3)
M2 Receive successful independent pre-review of Phase 2 research plans, via a credible,
independent technical evaluation that can also serve as a pre-registration of Phase 2 plans
(e.g., In-Principle Acceptance of a Registered Report xviii with a journal in good standing,

DARPA-SN-18-02 UGB Disruption Opportunity 4

 Letter of Support from a potential Phase 2 transition partner, written positive evaluation
of Phase 2 plan from a professional committee or society). (Month 5)
M3 Demonstrate the feasibility of the proposed machine learning approach on a
representative sample of the dataset. (Month 5)
M4 Deliver Final Report (only required if Phase 2 is not awarded) (Month 6)

Phase 2 Milestones:

M5 Receive IRB and HSR approval of Phase 2 research approach if identified as being
required during Phase 1. Acquire and demonstrate accessibility of any data requiring
HSR. (Month 9) (only required if research includes HSR)
M6 Demonstrate machine learning approaches and appropriate level(s) of analyses to
identify group biases/cultural models on complete datasets for at least one scenario.
Demonstrate computational reproducibility of results. (Month 15)
M7 Validate UGB prototype algorithms using methods identified in Phase 1 on at least one
scenario. (Month 15)
M8 Deliver UGB datasets, algorithms, and software in format to be agreed upon with
DARPA as appropriate (Month 18)
M9 Deliver UGB Final Report including (but not limited to) technical overview of Phase
1/Phase 2 work, deliverables associated with the work, lessons learned, and
recommendations for potential future work in UGB (Month 18)

All proposals must include the following meetings and travel in the proposed schedule and costs:
• To foster collaboration between teams, and disseminate program developments, a two-
day Principal Investigator (PI) meeting will be held approximately every six months, with
locations split between the East and West Coasts of the United States. For budgeting
purposes, plan for three two-day meetings over the course of 18 months: two meetings in
the Washington, D.C. area and one meetings in the San Francisco, CA area.
• Regular teleconference meetings will be scheduled with the Government team for
progress reporting as well as problems identification and mitigation. Proposers should
also anticipate at least one site visit per phase by the DARPA Program Manager during
which they will have the opportunity to demonstrate progress towards agreed-upon
milestones.

E. Deliverables
Performers will be expected to provide at a minimum negotiated deliverables specific to the
objectives of the individual efforts. These may include registered reports, experimental
protocols, publications, intermediate and final versions of software libraries, code, and APIs,
including documentation and user manuals, and/or a comprehensive assemblage of design
documents, models, modeling data and results, and model validation data. Specific deliverables
will be established in any resultant OT award.

II. Award Information

Selected proposals that are successfully negotiated will result in award of an OT for prototype
project. See Section 3 of DARPA-PA-18-01 for information on awards that may result from

DARPA-SN-18-02 UGB Disruption Opportunity 5

proposals submitted in response to this notice.
III. Eligibility
See Section 4 of DARPA-PA-18-01 for information on who may be eligible to respond to this
notice.
IV. Opportunity Responses
Responses to this DO SN must be submitted as full proposals to DARPA-PA-18-01 as described
therein. All proposals must be unclassified.
A. Proposal Content and Format
All proposals submitted in response to this notice must comply with the content and format
instructions in Section 5 of DARPA-PA-18-01. All proposals must use the templates provided
as Attachments to the PA and follow the instructions therein.

Information not explicitly requested in DARPA-PA-18-01, its Attachments, or this notice may
not be evaluated.
B. Proposal Submission Instructions
See Section 5 of DARPA-PA-18-01 for proposal submission instructions.
C. Proposal Due Date and Time
Proposals in response to this notice are due no later than 4:00 PM on April 11, 2018. Full
proposal packages as described in Section 5 of DARPA-PA-18-01 must be submitted per the
instructions outlined therein and received by DARPA no later than the above time and date.
Proposals received after this time and date may not be reviewed.

Proposers are warned that the proposal deadline outlined herein is in Eastern Time and will be
strictly enforced. When planning a response to this notice, proposers should take into account
that some parts of the submission process may take from one business day to one month to
complete.
V. Proposal Evaluation and Selection
Proposals will be evaluated and selected in accordance with Section 6 of DARPA-PA-18-01.
Proposers will be notified of the results of this process as described in Section 7.1 of DARPA-
PA-18-01.
VI. Administrative and National Policy Requirements
Section 7.2 of DARPA-PA-18-01 provides information on Administrative and National Policy
Requirements that may be applicable for proposal submission as well as performance under an
award.

VII. Point of Contact Information

Adam Russell, Program Manager, DARPA/DSO, UGB@darpa.mil

VIII. Frequently Asked Questions (FAQs)

DARPA-SN-18-02 UGB Disruption Opportunity 6

All technical, contractual, and administrative questions regarding this notice must be emailed to
UGB@darpa.mil. Emails sent directly to the Program Manager or any other address may result
in delayed or no response.

All questions must be in English and must include name, email address, and the telephone
number of a point of contact. DARPA will attempt to answer questions publically in a timely
manner; however, questions submitted within 7 days of the proposal due date listed herein may
not be answered.

DARPA will post an FAQ list under the Special Notice on the DARPA/DSO Opportunities page
at http://www.darpa.mil/work-with-us/opportunities. The list will be updated on an ongoing
basis until one week prior to the proposal due date. In addition to the FAQ specific to this
notice, proposers should also review the Disruptioneering General FAQ list on the DARPA/DSO
Opportunities page under the Disruptioneering Program Announcement (DARPA-PA-18-01).

IX. Additional Information

For additional information, please review the attached slides.

i
E.g., http://www.arizona.edu/sites/default/files/cultural_models.pdf
ii
E.g., Mary Douglas Purity and Danger: An Analysis of Concepts of Pollution and Taboo. 2002 (1966) Routledge.
iii
E.g., https://en.wikipedia.org/wiki/Types_of_marriages
iv
http://sites.lsa.umich.edu/satran/wp-content/uploads/sites/330/2015/10/michelejgelfand_ch06_copy_2.pdf
v
E.g., https://en.wikipedia.org/wiki/Culture-bound_syndrome
vi
Note that while bias often is used in a pejorative sense, this Special Notice uses the term in its most neutral sense
of probabilistic tendencies to act, think, or attend to things in certain ways. Importantly, biases are not always bad,
since they can help in directing limited attentional resources to what matters most in certain contexts. Most experts
have biases, which in fact often define them as experts: that is, they are biased – or inclined - to attend to the most
important features of often complex situations, which makes them more effective (faster, accurate) than novices.
However, biases may also lead people to make snap judgments that are wrong.
vii
Clifford Geertz. The interpretation of cultures: selected essays. 1973
viii
Brian A Hoey. "A Simple Introduction to the Practice of Ethnography and Guide to Ethnographic Fieldnotes"
Marshall University Digital Scholar (2014)
ix
Geertz, Clifford. “From the native's point of view": On the nature of anthropological understanding." Bulletin of
the American Academy of Arts and Sciences (1974): 26-45.
x
Geertz, Clifford. "Thick description: Toward an interpretive theory of culture." Readings in the philosophy of
social science (1994): 213-231.
xi
http://www.sciencemag.org/news/2017/04/even-artificial-intelligence-can-acquire-biases-against-race-and-gender
xii
https://framenet.icsi.berkeley.edu/
xiii
data.gdeltproject.org/documentation/CAMEO.Manual.1.1b3.pdf
xiv
E.g., http://science.sciencemag.org/content/353/6307/1502
xv
https://singularityhub.com/2017/01/31/the-struggle-to-make-ai-less-biased-than-its-creators/
xvi
http://science.sciencemag.org/content/354/6317/1240
xvii
E.g., http://www.aaronbfrank.com/wp-content/uploads/2011/06/Identity-Theory-and-Agent-Based-Modeling-v4-
DRAFT1.pdf
xviii
https://www.nature.com/articles/s41562-016-0034

DARPA-SN-18-02 UGB Disruption Opportunity 7

 Understanding Group Biases using Big Data and
Computational Social Science (UGB)
Adam Russell

NOTE:
These slides are meant to be supplementary materials to the UGB Special Notice for the
purposes of providing additional clarifying information on the UGB background,
approach, and goals.

HOWEVER, the information in the Special Notice supersedes any information in these
slides. In case of perceived contradiction or confusion between information in these
slides and information in the Special Notice, please reference the Special Notice.

Distribution A – Approved for public release 1

 What are we trying to do?
Goal: Exploit biases in data and machine learning models to simulate cultural
models in specific use case scenarios
E.g.,
Objective Feature learning from Classification of group Predicted cultural model
Event in biased datasets biases
the World
Reported
E.g., Media
in Media Outlet A
Sources

E.g., Media
Outlet B

E.g., Media
Outlet C

Distribution A – Approved for public release 2

 How is it done today?

• “Ethnography - or field research - • Strengths

is a sociological method that • Produces “thick” data, providing
explores how people live and deep(er) context for identities,
makes sense of their lives with beliefs, and behaviors
one another in particular places.” • Can explain social phenomena that
appear “irrational” to outsiders
• Limitations
• Highly subjective, depends on
individual ethnographer
• Lengthy, resource intensive process
(months-years, travel)
• Often non-reproducible due to data
collection approach

http://sociology.columbia.edu/ethnography
https://www.loc.gov/exhibits/mead/field-sepik.html
https://en.wikipedia.org/wiki/Bronislaw_Malinowski
https://global.si.edu/projects/recovering-voices
http://www.newsweek.com/2013/08/16/human-terrain-system-sought-transform-army-within-237818.htm
https://en.wikipedia.org/wiki/The_Chrysanthemum_and_the_Sword
Distribution A – Approved for public release 3
 Data are “social mirrors” that reflect group biases

• A group’s biases are reflected in

data that:
• groups choose to consume
• groups produce
• researchers solicit
• researchers infer

• Cultural models can be thought of

as a collection of shared group
biases

• Hypothesis of program: Focused How

do we
“big” data can also become “thick” get
data for better understanding here?

cultural models
Dunbar, Robin IM, et al. "The structure of online social networks mirrors those in the offline world." Social Networks 43
(2015): 39-47.
http://chainsawsuit.com/comic/archive/2014/09/16/on-research/
Lada Adamic, personal webpage: http://basementrejects.com/review/west-side-story-1961/
https://medium.com/ethnography-matters/why-big-data-needs-thick-data-b4b3e75e3d7

Distribution A – Approved for public release 4

 Machine learning algorithms capture biases in data

“Even artificial intelligence can acquire

biases against race and gender” –
Science, April 2017

Machine learning models can exhibit data bias and model bias:
• Biases exist in the data labels due to class imbalances, “data as a social mirror,”
cognitive biases of human annotators in hybrid models
• Models themselves, independent of the data, may make assumptions that are not
consistent with reality, e.g. linear relationships between variables

Understanding Group Biases (UGB) will purposefully incorporate appropriate data bias and
model bias to capture group biases and learn cultural models
• Incorporation of vast data volumes limits subjectivity
• Modern machine learning and computing methods greatly reduce time to analyze
data
• Reproducible collection approaches facilitate reproducible research

http://www.sciencemag.org/news/2017/04/even-artificial-intelligence-can-acquire-biases-against-race-and-gender

Distribution A – Approved for public release 5

 Can UGB help us…

..quantify the diversity of interpretations? …improve event coding?

Feature learning, Learned models/group Predicted group Feature learning, Learned models/group Predicted group
classification biases interpretation classification biases interpretation

More likely to
be correct coding

…identify change in biases over time? …simulate variability?

Feature learning, Learned models/group Predicted group Feature learning, Learned models/group Agents that simulate
classification biases behavior/interpretation classification biases group biases

Distribution A – Approved for public release 6

 UGB Program Structure
Goal: Identify and extract biases in data using machine learning models to
simulate cultural models for specific use case scenarios

Tasks: See the UGB Special Notice for Phase 1 (6 months): No HSR!
more detail! Acquire datasets reflecting group biases
Receive successful independent pre-review of
Task: Identify publicly available datasets Phase 2 research plans
Task: Identify machine learning approaches Demonstrate approaches to encode biases
into models
Task: Identify the anticipated levels of
analysis
Phase 2 (12 months): HSR possible
Task: Use UGB algorithms to build prototypes
Demonstrate approach on chosen use case(s)
for one or more Use Case Scenarios
• Quantify Diversity of Interpretations Validate on chosen use case(s)
• Improve Event Coding Evaluate performance
• Identify Cultural Change over Time
• Simulate Variability
Task: Validation approaches

Success: Performance and validation of UGB algorithms and prototypes against

proposed use case scenarios and metrics
Distribution A – Approved for public release 7