Organisation for Economic Co-operation and Development

GOV/RPC/RD(2019)13

For Official Use English - Or. English

4 November 2019
PUBLIC GOVERNANCE DIRECTORATE
REGULATORY POLICY COMMITTEE

Algorithms and (big) data for evidence-based regulations

Concept note

21st meeting of the Regulatory Policy Committee

6-7 November 2019

OECD Conference Centre
Paris, France

This note highlights possible avenues for future work on the use of new technologies for assessing and
evaluating the impacts of laws and regulations. It builds on GOV/RPC(2019)6.

Christiane Arndt-Bascle, christiane.arndt@oecd.org and

Miguel Amaral, Miguel.amaral@oecd.org .

JT03453999
OFDE

This document, as well as any data and map included herein, are without prejudice to the status of or sovereignty over any territory,
to the delimitation of international frontiers and boundaries and to the name of any territory, city or area.
2  GOV/RPC/RD(2019)13

1 Concept note
1.1. Background

1. The increasing availability of open data, ‘big data’ and new analytical techniques have
considerable potential both to enhance evaluations of laws and regulations and enable innovations in how
these are conducted. Patterns and responses may be discernible that would not have been possible using
traditional statistical methods. This is a relatively new area and one that holds out considerable scope for
learning across jurisdictions (OECD, 2019)
2. The Programme of Work and Budget for 2019-20 [GOV/RPC(2018)2/REV2] therefore includes a
project on (big) data for evaluating laws and regulations. The planning of activities to deliver on the
2019-2020 Programme of Work of the RPC on regulation and emerging technologies(GOV/RPC(2019)6
further specifies that this work could focus on how to collect and make use of (big) data and artificial
intelligence for better design and evaluation of regulations. This could include the following outputs:
1. Conference on Measuring Regulatory Performance (Norway) June 2019;
2. Identification of examples through the OECD survey on experiences with regulatory impact
assessments related to emerging technologies and the use of (big) data and new analytical tools
to the design and the evaluation of regulations (GOV/RPC/RD(2019)12)
3. Preliminary findings on using (big) data and artificial intelligence for designing and evaluating
regulations;
4. Key lessons on collecting and using (big) data and artificial intelligence for better design and
evaluation of regulations (draft version).
3. The OECD conference in Norway revealed many challenges related both to the use of new
technologies but also to traditional ways of data collection, sharing and analysis for assessing and
evaluating impacts of legislation. Concrete examples for the use of new technologies for evaluating
regulations outside the financial sector seem to be hard to find. This note builds on the findings of Norway
as well as relevant work of other OECD committees to identify possible avenues for future work.
4. Relevant work from other parts of the OECD include the “Using digital technologies to improve
the design and enforcement of public policies” (OECD, 2019), “Enhancing Access to and Sharing of Data
(EASD): Reconciling Risks and Benefits for Data Re-use across Societies” (OECD, 2019, forthcoming), “A
data-driven public sector: Enabling the strategic use of data for productive, inclusive and trustworthy
governance” (Ooijen, Ubaldi, & Welby, 2019), “Hello World: Artificial Intelligence and Its Use in The Public
Sector” (Berryhill, Kok Heang, & Clogher, Forthcoming), “State of the art in the use of emerging
technologies in the public sector (Ubaldi, et al., 2019) “The Path to Being a Data Driven Public Sector”
(OECD, 2019), the 2018 OECD Open Government Data Report (Open Government Data
Report: Enhancing Policy Maturity for Sustainable Impact, 2018) and “Which Strategies for NSOs in the
Digital Era? Towards ‘Smart Data’ Strategies” (OECD, 2018). A number of OECD Recommendations are
also relevant to this work including the 2019 OECD Recommendation of the Council on Artificial
Intelligence, (OECD, 2019), the OECD Recommendation of the Council on Digital Government Strategies

ALGORITHMS AND (BIG) DATA FOR EVIDENCE-BASED REGULATIONS

For Official Use
 GOV/RPC/RD(2019)13 3

(OECD, 2014); the OECD Recommendation of the Council concerning Access to Research Data from
Public Funding (OECD, 2006) and the OECD Recommendation of the Council for Enhanced Access and
More Effective Use of Public Sector Information (OECD, 2008) which address data access and sharing,
and provide guidance and best practices.
5. Current work at the OECD however does not provide examples and practical guidance on using
new technologies for the design and evaluation of regulations. The suggested work therefore would fill this
gap.
6. The work is subject to voluntary contributions as outlined in the PWB. Findings will furthermore
depend on the availability of country examples in the use of (big) data and artificial intelligence for designing
and evaluating regulations.

1.2. Proposed work

7. Following discussions with the Steering Group on Measuring Regulatory Performance the
Secretariat suggests undertaking case studies in a selected number of countries. These could focus
narrowly on identifying examples of the use of new digital technologies such as artificial intelligence, big
data, smart data or crowdsourcing of data for better assessment and evaluation of regulations. The work
could also embed these examples into a wider analysis of the enabling conditions for better data and
analysis for designing and evaluating laws and regulations.
8. This would follow the findings from the 2019 OECD conference in Norway that often government
do not lack data but that the issues lie elsewhere. For example, ministries are sometimes not aware of the
“ordinary” data that are already at their disposal and would fit their purpose. Some data might be of bad
quality and the key issue is to improve its quality – otherwise artificial intelligence or other analytical tools
might provide misleading results. The work could also look into ways of institutional support to identify the
best source of data and provide officials in charge of the analysis with assistance in the use of analytical
tools including the use of artificial intelligence.

1.2.1. Types, sources and quality of data

9. Many different types of data including more traditional sources such as census data are relevant
for policy evaluation. Digital technologies potentially give access to a much richer set of relevant
information, at a level of granularity and periodicity that was not previously possible. Challenges might
differ according to the nature of this data. There is a for example a difference to be made between
‘structured’ (e.g. numbers) and ‘unstructured’ information (e.g. pictures). Quality of data is also often
insufficient, making it impossible to use or, if users are not aware or decide to ignore the quality issues,
results of analysis will be misleading. Furthermore, the extent to which digitally-generated data used for
policy evaluation is based on a representative sample of the regulated population is probably a concern.
10. The work could:
a. Provide a typology of different types of data relevant for evaluation of laws and regulations;
b. Identify examples of use of new sources of data such as big data, smart data or crowdsourcing for
assessing, monitoring or evaluating impacts of regulations;
c. Identify challenges related to the quality and representativeness of data, and examples of how
countries have developed strategies to enhance data quality.

ALGORITHMS AND (BIG) DATA FOR EVIDENCE-BASED REGULATIONS

For Official Use
4  GOV/RPC/RD(2019)13

1.2.2. Accessing data and privacy protection

11. A key finding of the conference in Oslo was that data is often available but either officials are not
aware that they exist or there are barriers to accessing the data. It is not always necessary to collect data
from scratch - relevant data for evaluating legislation could come from other ministries and agencies, from
statistical offices, from the private sector or even from other countries who have collected more complete
datasets. Yet, this implies to know about the existence of these data and to be able to access and use it.
Digitalisation precisely offers opportunities to co-operate on data sharing. Building on work done in other
parts of the OECD on data governance and privacy (e.g. (OECD, 2019, forthcoming), (OECD, 2019)), this
work could:
a. Identify strategies for simplifying information and access to existing data relevant for assessing and
evaluating impacts of regulations, for example through central portals;
b. Identify examples for successful data sharing across government agencies, the private sector, civil
society organisations or even across countries;
c. Look into strategies to protect individuals’ privacy while enhancing governments’ capacity to assess
the impact of their laws and regulations.

1.2.3. Analysis of data

12. New technologies including the rise of artificial intelligence offer many opportunities for collecting
and analysing data that are currently mostly exploited in the private sector. They may also help to increase
the focus on inductive reasoning — that is, let the data ‘tell its own story’ — rather than on a deductive
basis where we typically have a hypothesis in mind that we wish to confirm or reject and use the data for
that purpose.
13. This work could look into:
a Typology of techniques available for data analysis with a particular focus on new technologies;
b Examples of the use of new technologies for assessing or evaluating the impacts of laws or
regulations;
c Identify where the use of new technologies adds value and what enabling factors need to be in
place to use it.

1.2.4. Continuous monitoring1

14. Regulations are experiments and new technologies can offer the possibility to monitor its impacts
continuously instead of a one-off evaluation. This can reduce the duration of uncertainty on the impacts of
regulations or even allow for more flexible forms of regulation such as performance-based regulation which
require monitoring of outcomes. At the same time governments need to ensure burdens on the regulated
subjects to provide continuous data are justified and they need to create a stable regulatory environment
to allow for business and citizens to take investment decisions.
15. This work could:
a Identify the different types of digital technologies that enable governments to continuously
monitor impacts;
b Identify concrete examples for continuous monitoring of the impacts of laws or regulations

1
The concept of “continuous monitoring” is quite close to the notion of “real-time monitoring” that has been examined,
in particular, during the joint GOV/STI workshop on “Governance Innovation” held on 15 may 2019.

ALGORITHMS AND (BIG) DATA FOR EVIDENCE-BASED REGULATIONS

For Official Use
 GOV/RPC/RD(2019)13 5

c identify who collects/provides the data (e.g. regulated subjects, governments themselves,
private sector);and the challenges raised in situations where monitoring is delegated to a third
party
d Look into the benefits and costs to real time monitoring of impacts of regulations

1.2.5. Incorporating data needs into regulatory proposals

16. Data requirements are best considered at the time a regulation is being made, as part of wider
consideration of the type of ex post review that would be most appropriate (OECD, 2019), (OECD, 2019).
RIA systems should also have an in-built monitoring, evaluation and refinement mechanism in place. This
includes early plans for data collection or access to data. Before it is decided that new data need to be
collected, regulators need to be aware of already existing data to avoid double collection of data. Where
there is a need to collect new data, those who will be in charge of collecting the data and the corresponding
resources need to be identified.
17. The OECD Secretariat could:
a Identify countries that systematically tackle data needs at the ex-ante stage;
b Identify concrete examples of RIAs that specify what data will be used to evaluate the regulation
and where to access it or how to collect it;
c Identify examples where data specified at the RIA stage was actually used to evaluate the
regulation ex-post;
d Attempt to find examples where data needs identified at the ex-ante stage involved new
technologies.

1.2.6. Skills and institutional support

18. The OECD conference in Norway revealed the lack of skills in government as a key challenge in
adopting new technologies. Analysts in charge of preparing or evaluating regulations are not necessarily
data experts and often not aware of data that is available to them. In some countries statistical offices, or
evaluation units might have more capacities in these areas and provide support or do take over part of the
analysis. Computer scientists and experts on artificial intelligence are mostly found in the private sector
and it is challenging for the public sector to attract talent from a highly competitive market.
19. The work could look into:
a Strategies of countries, where they exist, to attract computer scientists and other data experts
including on AI to improve monitoring and evaluation of regulations;
b How countries train officials in charge of developing or evaluating laws in data collection and
analysis;
c What role oversight bodies can play to ensure consideration of a broad range of data and evidence
by officials preparing and reviewing regulations
d What role other institutions including statistical offices can play to support officials to identify, collect
and analyse data, with a particular focus on the use of new technologies.

1.2.7. Learning from other policy areas or the private sector

20. Participants to the last MRP conference did not provide a lot of practical examples on the use of
big data/digital tools for evaluation. In case examples from the public sector in the use of new technologies
for evaluating legislation are not sufficient, the Secretariat could:
a Look into relevant examples from policy evaluation more broadly;

ALGORITHMS AND (BIG) DATA FOR EVIDENCE-BASED REGULATIONS

For Official Use
6  GOV/RPC/RD(2019)13

b Examples from the private sector.

1.3. Questions for discussion

 Which of the themes mentioned above would you like to see covered by the work? Which themes
and research questions are a priority to you?
 Would you like us to take a broad approach looking into the elements for better data for better laws
or exclusively identify examples of the use of new technologies for data and analysis such as big
data or AI?
 Would case studies be the best way to approach this topic at this stage?
 Would you like us to focus on design, monitoring and evaluation or only one or two of these phases
in the policy-cycle?
 Would you like to participate in this work?

ALGORITHMS AND (BIG) DATA FOR EVIDENCE-BASED REGULATIONS

For Official Use
 GOV/RPC/RD(2019)13 7

References
Berryhill, J., Kok Heang, K., & Clogher, R. (Forthcoming). Hello World: Artificial Intelligence and its Use in
the Public Sector. OECD Working Papers on Public Governance.
OECD. (2006). Recommendation of the Council concerning Access to Research Data from Public
Funding, [OECD/LEGAL/0347]. Retrieved from
https://legalinstruments.oecd.org/en/instruments/OECD-LEGAL-0347
OECD. (2008). Recommendation of the Council for Enhanced Access and More Effective Use of Public
Sector Information, [OECD/LEGAL/0362]. Retrieved 07 24, 2019, from
https://legalinstruments.oecd.org/en/instruments/OECD-LEGAL-0362
OECD. (2014). Recommendation of the Council on Digital Government Strategies, [OECD/LEGAL/0406].
OECD, Paris. Retrieved from https://legalinstruments.oecd.org/en/instruments/OECD-LEGAL-
0406
OECD. (2018). Open Government Data Report: Enhancing Policy Maturity for Sustainable Impact. In
OECD Digital Government Studies. OECD Publishing, Paris.
doi:https://dx.doi.org/10.1787/9789264305847-en
OECD. (2018). Which Strategies for NSOs in the Digital Era? Towards ‘Smart Data’ Strategies. OECD
Publishing.
OECD. (2019). Best Practice Principles for Regulatory Policy: Reviewing the Stock of Regulation.
OECD. (2019). OECD Best Practice Principles for Regulatory Policy: Regulatory Impact Assessment
(RIA). Final Draft.
OECD. (2019). Recommendation of the Council on Artificial Intelligence. OECD Publishing.
OECD. (2019). The Path to Being a Data Driven Public Sector. OECD Publishing.
OECD. (2019). Using digital technologies to improve the design and enforcement of public policies.
OECD Publishing. doi:https://doi.org/10.1787/20716826
OECD. (2019, forthcoming). Enhancing Access to and Sharing of Data (EASD): Reconciling Risks and
Benefits for Data Re-use across Societies. OECD Publishing.
Ooijen, C., Ubaldi, B., & Welby, B. (2019). A data-driven public sector: Enabling the strategic use of data
for productive, inclusive and trustworthy governance. OECD Publishing.
doi:https://doi.org/10.1787/19934351
Ubaldi, B., Le Fevre, E. M., Petrucci, E., Marchionni, P., Biancalana, C., Hiltunen, N., . . . Yang, C.
(2019). State of the art in the use of emerging technologies in the public sector.
doi:https://doi.org/10.1787/19934351

ALGORITHMS AND (BIG) DATA FOR EVIDENCE-BASED REGULATIONS

For Official Use