Total Words: 3,580 Dexter Pante - 613439 Capstone Final Assessment Lessons from International Practices on Evaluation Standard

Development and its Implication to Philippine Education Context: A Mixed Method Case Study using Boolean Algebra Section 1: Introduction While evaluation culture and practice in Philippine basic education remains weak, this condition is slowly being addressed since 2000, through the combined efforts of the national government and international partners. Among the initiatives were directives that aimed at reforming the system in the areas of planning, resource programming, and reporting. Given this backdrop of reforms, the Philippine National Government is pouring huge investment into the K to 12 Program for 2014 to the amount of Php 338 Billion (AUD 8.2 B). As a developing country, the government is persistently faced with resource constraints for its programs and services to the public. In this light, there is a need for evaluation to ensure that resources are effectively, efficiently and economically utilised. In the Philippine context where evaluation is not a well-entrenched practice, there is a perception that evaluation services are not responsive to the needs of the clients, especially with regard tothe feasibility and utility of policy recommendations. In countries where there is a relatively mature evaluation practice, evaluators are guided by sets of evaluation standards, or code of ethics, which are designed to ensure that the interest of the public and the clients are best served (Kakande, 2013; Marklund, 1984; Patel, 2002; Virtanen & Laitinen, 2004). In the Philippines, no such standard exists. The aim of this study is to stimulate discussion regarding the development of such standards, by looking at the contextual factors that have enabled countries to develop their own evaluation standards. The hope is that by exploring these macro causal factors, some patterns and implications can be drawn that are relevant to the effort of the countrys largest government bureaucracy, the Department of Education (DepEd), to develop evaluation standards for basic education. This literature review is an attempt to explore international practices regarding the development of evaluation standards as a benchmark for the Philippines effort to develop its own internationally-comparable, yet culture-responsive, evaluation standards. The final product of this literature review is a program theory that will guide the DepEd in crafting its development strategy (Greene, 2007). Towards this end, the research questions are the following: General Question: What are the different causal pathways or configurations resulting from Boolean algebraic analysis of the contextual conditions that have enabled countries to develop their own evaluation standards?

Sub-questions: Within a country, what are the conditions that have led to the development of evaluation standards? Across different countries, what are the themes, if any, that cut across contexts? Section 2: Implication for evaluation theory and practice This section discusses the contribution of this literature review to the areas of evaluation theory and practice. This review has four modest contributions. The first two contributions add knowledge to evaluation theory, by identifying the cross-cutting causal factors that influenced the development of evaluation standards in different countries, and by identifying the social science theory that could explain the reason countries develop their standards of evaluation practice.1 The other two contributions relate to the evaluation practice: first, is the articulation of Qualitative Comparative Analysis (QCA) using the models of current mixed method literature and second, the demonstration of the procedures of QCA as a viable approach to improve the reliability of data transformation in mixed method research, as described by Creswell and Plano Clark (2007). In this literature review, no less than a thousand publications were searched using the key terms country evaluation standard national association. Of these, around a hundred articles were short-listed for a quick appraisal of the contents. Annex 1 shows the list of countries and available literature for each, while Annex 2 shows the journals searched, the results of the search, and the number of articles reviewed. The majority of the reviewed articles discussed national evaluation capacity, monitoring systems, and results of commissioned evaluation with regard to a specific evaluand. A minority discusseda specific historical evolution of evaluation within national contexts. I was only able to find ten publications that attempted to consolidate country case studies on national evaluation system. Even these ten (See Annex 3) do not provide a comprehensive picture about the development of evaluation standards; hence, the mismatch between those articles cited in Annex 1 and the ten articles in Annex 3. Only the articles,Hovlet and Dewachter (2013) and Furubo, Rist, and Sandahl (2002) made a systematic, cross-country comparison. Given this lack of study on the topic, this review intends to stimulate further discussion on crosscountry causal factors affecting the development of evaluation standards. As part of the QCA analysis, I also identified a suitable theory for understanding the burgeoning demand for evaluation standards. Of the nine publications that made a comparative analysis, only two studies, Furubo et al. (2002) and Hovlet and Dewachter (2013), presented theories for understanding the development of evaluation standards. However, I found their theories to be inadequate because they only explained the diffusion of evaluation in relation to the development of evaluation and national evaluation societies. These theories did not explain the development of evaluation standards. Hence, my exploration on the alternative theory will enhance our theoretical knowledge about the rationale for the increasing need to develop evaluation standards.

I use standards, ethics, guidelines and codes interchangeably.

The other two contributions relate to evaluation practice. The added value of this study concerns the application of existing mixed method models to QCA design. I surveyed all the articles from the websites of the Comparative Methods for Systematic and Cross-Case Analysis (www.compass.org), which is a network of researchers worldwide that aims to advance the use of systematic comparative case to research, and from the Journal of Mixed Method Research (http://mmr.sagepub.com/). With regards to the former, there are 73 published working papers on QCA from 2003-20013 and for the latter, there are 162 mixed method articles published from 2007-2013. While it has been recognised by Greene (2007) that QCA is one of the pioneers in the mixed method research, I have not found any example of published article from the COMPASS and the Journal of Mixed Method Research that articulate QCA using the frameworks of mainstream mixed method literature. The last contribution of this study to evaluation practice is the practical strategy of transforming qualitative data to quantitative variables in mixed method research. According to Greene (2007), data transformation is one unique feature of mixed method research that separates it from pure quantitative and qualitative methods. As noted by Sandelowski, Voils, and Knafl (2009), there are a number of qualitative assumptions and judgments involved in quantitizing. In this study, I demonstrated that identifying the presence or absence of themes in a case may be facilitated by using inductive approach. Through the inductive approach, I was able to identify preliminary patterns and used these patterns to define and describe the themes, which I used for coding the chunks of textual data in the Nvivo (Bernard & Ryan, 2010). This strategy ensures that the study is reliable, and it also adds to the increasing body of literature on the procedure of transforming one type of data to another form, so as to enable higher order analysis. Section 3: Engagement and critical analysis of relevant literature A. Presentation of analytic process and findings In this section, I present the cross-cutting causal factors, develop a program theory using my findings and engage this finding in a dialectical conversation with existing social theories. Before identifying the cross-cutting factors, a brief description of the data collection and transformation processes is necessary to understand the results of the analysis. After the formulation of the themes and identification their presence or absence in a case, the data was organised in Table 1 as shown below. The process of determining the set membership scores or calibration of the cases in each theme was very challenging, because the articles were not written in a standardised form. In the absence of objective criteriawhich allows transparent calibration of the data, the best recourse was the use of indepth theoretical knowledge of the cases.

Table 1: Data Set Matrix

Countries Australia Switzerland Japan Canada Sri Lanka New Zealand Uganda Korea China Brazil
A : Research and B : Internal C : Influence of D : Donor Support E : Critical Mass of evidenced-based Demand International (DONOR) Practitioners tradition in policy making (INDEMAND) Practices (PRAC) (RES) (INFINTL) 1 1 1 0 1 1 0 1 0 1 1 1 1 1 1 1 1 1 0 1 0 0 0 1 0 0 0 1 0 1 F : Diversity of Evaluation Approaches (DIVERSE) 1 1 1 1 1 1 G: Concern on the H: Evaluation Quality of Evaluation Standard Practice and Outcomes (OUTCOME) (QUAL) 1 1 0 1 1 0 1 1 1 1 1 1 1 0 0 0

0 1 1 0

1 1 0 1

0 0 0 0

1 0 1 1

0 0 0 0

1 1 0 0

1 1 0 0

In Table 1, the header contains the themes. Due to lack of space, the descriptions of the themes are found in Annex 4. Interpreting the causal configurations requires an understanding of the definition of the binary variables (1 or 0), which means either true or false, present or absent, high or low, respectively. For example, the causal configuration of Australia can be interpreted as, high research and evidenced-based tradition (RES), high internal demand (INDEMAND), high influence of international practices (INFINTL) coupled with presence of critical mass of practitioners (PRAC), high diversity of evaluation approaches (DIVERSE), and high concern for the quality of evaluation (QUAL) which, even in the absence of donor support (DONOR), lead to the development of evaluation standards (OUTCOME). The next step is to summarise the data set matrix to identify the cases with common causal configurations (Scheider & Wagemann, 2012). The purpose of this truth table is to show the observed causal configurations that are unique. Given that QCA emphasizes, multiple conjunctural causation, which means diverse causal pathways lead to the same outcome, it is possible that there are other causal configurations beyond those listed in Table 2. Using the expression 2K, whereby K represents the number of themes (7) and 2 represents the two possible result of the outcome, the total possible combinations is 128 (Bernard & Ryan, 2010; Ragin, 2007; Scheider & Wagemann, 2012). This total is spread between 8 observed and 120 unobserved configurations. In this truth table, I replaced the binary variables, 1 and 0, with upper case and lower case letters to facilitate understanding and analysis of the configurations later on.

Table 2: Truth Table

Configurations RES INDEMAND INFINTL DONOR PRAC DIVERSE QUAL OUTCOME Number of Cases Cases

1 2 3 4 5 6 7 8

RES RES res res res RES RES res

INDEMAND INFINTL donor INDEMAND INFINTL donor INDEMAND INFINTL donor INDEMAND infintl INDEMAND infintl INDEMAND infintl indemand infintl INDEMAND infintl

PRAC prac prac

DIVERSE QUAL DIVERSE qual DIVERSE QUAL DIVERSE QUAL DIVERSE qual DIVERSE QUAL diverse qual diverse qual

OUTCOME 3 OUTCOME 1 OUTCOME 1 OUTCOME 1 OUTCOME 1 outcome outcome outcome 1 1 1

Australia, Canada, NZ Switzerland Japan Uganda Sri Lanka Korea China Brazil

DONOR prac DONOR prac donor prac

DONOR prac DONOR prac

In the analysis of multiple causes, it is important to identify the necessary and sufficient conditions of the outcome of interest. The first type of analysis is the test of necessary conditions. A theme is a necessary condition if, whenever the outcome is present, the condition is also present (Losee, 2011; Benot Rihoux & Ragin, 2009; Scheider & Wagemann, 2012). Stated otherwise, the outcome cannot be achieved without the necessary conditions. For the test of necessity, only those configurations (1 to 5) where the outcome is present have to be analysed (Scheider & Wagemann, 2012). Using the Mills Method of Agreement, the necessary conditions for the outcome of interest are high internal demand (INDEMAND) and the presence of diversity in the use of evaluation approaches (DIVERSE). The other themes are not necessary because the outcome will still occur, regardless of their values. Unlike the necessary condition, a theme is a sufficient condition whenever the condition is present, the outcome is also present, but it is possible that the outcome is a result of other conditions as well (Benot Rihoux & Ragin, 2009; Scheider & Wagemann, 2012). For the test of sufficiency, only those themes whose value is true should be analysed (Scheider & Wagemann, 2012). Since the test of sufficiency involves more elaborate and complicated computation, I utilised Kirq which is a software designed for QCA analysis (Reichert & Rubinson, 2013). Based on the computer analysis as shown in Figure 1, the resulting causal configuration is as follows:

Figure 1: Sufficient Conditions using Kirq2

Further simplification using distributive and associative rules in algebra gives a more parsimonious and readable formula. Simply put, this final configuration contains two variables:
1st Variable {INDEMAND*DIVERSE} AND A. B. C. D. 2nd Variable {(RES* INFINTL *PRAC* QUAL*donor) OR (DONOR*res*infintl* prac) OR (RES*INFINTL*donor*prac*qual) OR (INFINTL*QUAL *res*donor*prac)} Outcome Development of Evaluation Standards

CAUSES

As can be seen from the final configuration, the first variable (INDEMAND*DIVERSE) constitutes the necessary conditions identified in the previous paragraphs; this finding validates the analysis of the necessary conditions. On the other hand, the second variable has four alternative configurations (A, B, C and D). These four alternatives correspond to the observed causal configurations with positive outcome, which are found in Table 2. This point confirms Scheider and Wagemann (2012)s observation that all combinations with positive outcome are sufficient conditions. The final configuration means that the presence of necessary conditions, combined with either the sufficient conditions A or B or C or D, lead to positive results in terms of the outcome of interest. A careful review of the second variable shows the presence of seemingly contradictory themes. Examples of contradictory themes are found in configurations A and B. The combination of these themes in a configuration points to the observation that the presence of necessary conditions (1st Variable), combined with either (A) the presence of RES, INFINTL, PRAC, QUAL and absence of DONOR or (B) the absence of RES, INFINTL, PRAC and presence of DONOR, lead to the development of evaluation standards. The explanation for this contradiction is that QCA is an analytic method based on set-theory. In set-theory, set membership and its negation are two qualitatively different phenomena (Scheider & Wagemann, 2012). Unlike the concept of necessary conditions, where superfluous themes
2

The asterisk (*) here refers to the coordinating conjunction AND, while the plus sign refers to the disjunction OR.

are not considered, in the test of sufficiency it is perfectly plausible that either the presence or absence of a theme is a sufficient condition for the outcome. This idea lends support again to the concept of multiple conjunctural causation in QCA. B. Discussion and Analysis of Relevant Theories The next step after the Boolean simplification is to use the necessary conditions as the program theory, which shall form the basis in the development of evaluation standards in the Philippines. However, the QCA analysis does not end with the theory development. The theory should be tested against the existing body of social theories and knowledge, in order for it to attain analytic generalisability (Berg-Schlosser, De Meur, Rihoux, & Ragin, 2009; Rihoux & Lobe, 2009; Scheider & Wagemann, 2012; Yin, 2009). According to Yin (2009), in analytic generalisation, previously developed theory is used as a basis with which to compare the results of other case studies; if that theory is confirmed by existing knowledge, then it is a good theory. As mentioned in the previous section, I have found two theories during the literature review, which attempt to explain the development of evaluation in other countries and the growth of national evaluation societies. These two theories are the diffusion of innovation and the resource mobilisation theory. The first theory claims that diffusion of innovation is the process by which innovation is communicated through certain channels over time among the social systems (Rogers, 2003, p. 5). The second theory is not discussed here because this theory primarily discusses the need for stakeholder support and other resources in building capacity of national evaluation societies. The diffusion of innovation theory explains the necessary conditions that arose within observed cases in my literature review. However, even the diffusion theory is not sufficient enough to account for the development of evaluation standard; it only explains the diffusion of evaluation practice. To explain the outcome, I looked for other social theories, for example, the status theory on professionalism. The causal configuration that I have identified is better understood in reconciling the theories of diffusion of innovation and the status theory on professionalism. In the succeeding paragraphs, I first explain how the diffusion theory fits with the observed cases and then use the status theory to argue that evaluation, once implanted in a system grows, evolves and matures. The clamour for standards of practice is a manifestation of maturation of evaluation into a profession. The theory of diffusion explains why evaluation is more widespread and institutionalised in some countries and not in others. This theory has four elements. The first element, nature and attributes of the innovation, explains that evaluation is adopted only when people see its advantages (i.e. evaluation is perceived as a useful tool in policymaking and the government demands it) and its compatibility with the existing values, traditions (RES) and needs (INDEMAND) of potential adopters (Rogers, 2003). This is certainly the case for Australia, Canada, New Zealand and Switzerland. The second element is the communication channels. Rogers (2003, p. 18) defines this term as the means by which messages get from one individual to another. The communication channel explains the importance of influencers overseas such as the United

States, with its mature evaluation culture and the support of international donors in stimulating internal demand for evaluation. This fact is proved by the 7 country cases where the outcome is positive (Australia, Canada, New Zealand, Switzerland, Japan, Uganda and Sri Lanka). In some case studies, there were discussions that the cross-cultural exchanges (Switzerland and Canada with the United States and Australia with New Zealand) facilitated the diffusion of evaluation practice. Another manifestation of communication channel that exists in the observed cases is the presence of diversity (DIVERSE) of evaluation approaches. This theme refers to the discussions and debates around the use of different methods for evaluation. From the view of diffusion theory, these debates and discussions are examples of heterophilous conversations,3 which are important factors in mainstreaming evaluation within the national context (Rogers, 2003). The third element of diffusion theory is time. Time is the rate of adoption of innovation, that is usually measured by the number of adopters in a given period (Rogers, 2003). Although the causal order or sequence of the themes could be analysed using a QCA variant called Temporal QCA (TQCA), this was not included in my analysis.4 Nevertheless, I found in the literature that the diffusion of evaluation came in waves. Furubo et al. (2002) discussed two waves of early adopters, namely the first wave composed of the United States, Canada, Germany, Sweden, United Kingdom and the second wave, which included Denmark, Netherlands, Norway, France and Switzerland. This point shows that time is also an important element in the diffusion of evaluation (Ragin & Strand, 2008). The fourth element is the social system, which is a group of interrelated units engaged in a common activity to accomplish common goal (Rogers, 2003). Diffusion happens in a space, which usually has its own pre-established social structures. In all of the cases reviewed where the outcome was present, diffusion of evaluation occurred in the public sector because of the desire of the government to determine the effects of the program or to optimise resource allocation (Furubo et al., 2002). Within the public sector is the political organisational structure, which also influenced the diffusion evaluation. If there is a strong internal demand from the central government for the use of evaluation, there is a strong likelihood that evaluation will be mainstreamed. As mentioned previously, the theory of diffusion only accounts for the spread of evaluation but not the development of professional evaluation standards. However, the theory of diffusion serves as a good springboard in discussing the status theory that can account for the outcome under study. Basically, I am reviewing the applicability of the status theory to determine whether the final causal configuration provides a good analytic model for predicting the outcome. The criterion for success in this analysis is the commonality in the characteristics of the status theory with the conditions in the identified final causal configuration. According toScriven (2013), as evaluation is diffused and attains maturity, it becomes self-conscious about itself and its distinction with other professions or disciplines. He emphasizes that evaluation has an important role in society and it is for this reason that
3

A heterophilous conversation is defined as the degree to which two or more individuals who interact are different in certain attributes (Rogers, 2003, p. 18) 4 Readers who want to learn more about TQCA can refer to Caren and Panofsky (2005) and Ragin and Strand (2008).

he values objectivity and reflexivity as standards of good evaluations (Scriven, 2004, 2011). This internal need to elevate the standing of evaluation practice and the external demand for a normative value of evaluation capture what Abbott (1983) calls the intra and extraprofessional status of professional ethics. Otherwise known as the status theory, this theory offers pragmatic solution between the functionalist and monopolist theories in interpreting the development of professional ethics. The functionalist theory values standards and ethics because of the fiduciary nature of certain services, while the monopolist theory views standards as form of anti-competition measures (Evetts, 2003, 2011). Unlike both theories, the status theory emphasises the importance of a profession in a society, but this theory stops short of making absolute normative value claims; it also portrays the purpose of the relationship between an individual and the profession as one of collegiality and mutual assistance, and not solely for profit-making, control and hegemony (Abbott, 1983; Evetts, 2003). This pragmatic orientation of the status theory shares family resemblances with some themes or conditions present in the final causal configuration. For example, the emphasis of the status theory upon the importance of evaluation resonates with the themes internal demand (INDEMAND), donor support (DONOR), and the concern for quality of evaluation (QUAL). There is also evidence in the literature that the growing professionalisation of evaluation is more oriented towards mutual support rather than a measure to stifle competition, although this evidence is weak.This point leads to the conclusion that although the final causal configuration does not explain the development of professional standards, I could still advance a warranted assertion that the final configuration provides a framework for advancing evaluation profession. In summary, the necessary conditions are important causal factors in spreading evaluation practices. This claim is supported by the theory of the diffusion of innovation. However, once evaluation takes root in the specific national context, this phenomenon will give rise to the professionalisation of practice through the adoption of standards, guidelines and code of conduct. The analysis of the similarities of the causal conditions and the characteristics of the status theory reinforces this proposition. In developing evaluation standards for the Philippine basic education, these necessary conditions should be taken into consideration.

Annex 1 List of Countries and Title of Journals Reviewed

Countries 1. Australia 2. Brazil Source 1 Mackay (2002) Silva et al. (2006) Source 2 Sharp (2003) Dannemann, Firme, and Letichevsky (2005) Rutman and Mayne (1985) Tanaka (N.D.) Shim (2011) Turner and Washington (2002) Tudawe and Samranayake (2008) Bussmann (1995) CLEAR (2012) Houqi and Rist (2002) Source 3 Straton (2001) X Evaluation Standard Present Absent

3. Canada 4. Japan 5. Korea 6. New Zealand

7. Sri Lanka

Segsworth (2002) Yamaya (2002) Lee (2002) Lunt and Trotman (2005) Sivagnanasothy (2007) Spinatsch (2002) Hauge (2001) Lu and Xie (2005)

Lahey and Nielsen (2013) X Jang (2000) X

Present Present Absent Present

Present

8. Switzerland 9. Uganda 10. China

Balthasar (2009) X X

Present Present Absent

Annex 2 Library Research Results Source Evaluation and Program Planning Advances in Program Evaluation Journal of Multi-Disciplinary Evaluation Evaluation Review: A Journal of Applied Social Research American Journal of Evaluation Evaluation: The International Journal of Theory, Research and Practice Evaluation Journal of Australasia Total Search Result 232 40 239 154 183 131 128 Number of Articles Reviewed 8 0 37 2 12 20 13

Annex 3 Ten Publications with Cross-Country Case Studies 1. 2. 3. 4. 5. 6. 7. 8. Building Evaluation Community: Past, and Future by Love and Russon (2000) Program Evaluation Standards in International Settings by Russon (2000) International Atlas of Evaluation by Furubo et al. (2002) Creating and Developing Evaluation Associations by Segone and Ocampo (2006) Evaluation in South Asia by Williams and Sandar (2008) National Evaluation Capacities by UNDP (2009) African Monitoring and Evaluation System byCLEAR (2012) Performance Measurement and Evaluation Systems: Institutionalizing Accountability for Governmental Results in Latin America by Cunill-Grau and Ospina (2012) 9. Development Evaluation in Times of Turbulence by Rist, Boily, and Martin (2013) 10. Building national M&E systems in the context of changing aid modalities: The underexplored potential of National Evaluation Societies by Hovlet and Dewachter (2013)

Annex 4 Themes and Descriptions Themes Research and evidencedbased tradition in policy making (RES) Internal demand (INDEMAND) Influence of international practices (INFINTL) Donor support (DONOR) Critical mass of practitioners (PRAC) Diversity of discussion on evaluation approaches (DIVERSE) Concern of the quality of evaluation practice and outcomes (QUAL) Description Includes texts that convey the idea of well-entrenched research tradition within policy formulation Includes texts on social problems, program solutions, government need for program efficiency and effectiveness, and individuals who advocate evaluation. Includes texts on overseas models on evaluation, for example, discusses Extension Services Research, New Public Management and social experimentation Includes texts on the assistance of international funding institutions such as ADB and WB in building national evaluation capacity Includes texts on the existence of think tanks, research institutes, and research specialist in the country Includes texts on presence of various evaluation approaches, such as performance management, audit, formative and summative evaluation, quality management, research, etc. Includes texts on the importance of improving evaluation procedures and practices, issues on unethical practice, and ensuring accurate measurement of program outcomes

References
Abbott, A. (1983). Professional Ethics. American Journal of Sociology, 88(5), 855-885. Balthasar, A. (2009). Institutional Design and Utilization of Evaluation A Contribution to a Theory of Evaluation Influence Based on Swiss Experience. Evaluation Review, 33(3), 226-256. Berg-Schlosser, D., De Meur, G., Rihoux, B., & Ragin, C. (2009). Qualitative Comparative Analysis (QCA) as an Approach. In B. Rihoux & R. C. (Eds.), Configurational Comparative Methods: Qualitative Comparative Analysis (QCA) and Related Techniques. London: Sage Publications. Bernard, H. R., & Ryan, G. (2010). Analyzing Qualitative Data: Systematic Approaches. Thousand Oaks: Sage Publications Inc. Bussmann, W. (1995). Evaluations and grassroots politics: The case of Switzerland. Knowledge and Policy, 8(3), 85-98. Caren, N., & Panofsky, A. (2005). TQCA A Technique for Adding Temporality to Qualitative Comparative Analysis. Sociological Methods & Research, 34(2), 147-172. CLEAR (Ed.). (2012). African Monitoring and Evaluation Systems: Exploratory Case Studies. Johannesburg: University of Witwatersrand. Creswell, J. W., & Plano Clark, V. L. (2007). Designing and conducting mixed method research. Thousand Oaks, CA: Sage Publications. Cunill-Grau, N., & Ospina, S. (2012). Performance Measurement and Evaluation Systems: Institutionalizing Accountability for Governmental Results in Latin America. New Directions for Evaluation, 134, 77-91. Dannemann, A., Firme, T., & Letichevsky, A. (2005). Setting Up the Brazilian Evaluation Network: a Challenging Work With no Boundaries. Paper presented at the 2005 Joint Conference of the Canadian Evaluation Society and American Evaluation Association, Toronto. Evetts, J. (2003). The Sociological Analysis of Professionalism: Occupational Change in the Modern World. International Business Research, 18(395). Evetts, J. (2011). Sociological Analysis of Professionalism: Past, Present and Future. Comparative Sociology, 10(2011), 1-37. Furubo, J., Rist, R., & Sandahl, R. (Eds.). (2002). International Atlas of Evaluation. New Brunswick: Transaction Publishers. Greene, J. (2007). Mixed Methods in Social Inquiry. California: Jossey-Bass. Hauge, A. (2001). Strengthening Capacity for M&E in Uganda: A results-based management perspective. ECD Working Paper Series 8. Houqi, H., & Rist, R. (2002). Evaluation Capacity Building in the People's Republic of China: Trends and Prospects. In J. Furubo, R. Rist & R. Sandahl (Eds.), International Atlas of Evaluation. New Bruswick: Transaction Publishers. Hovlet, N., & Dewachter, S. (2013). Building national M&E systems in the context of changing aid modalities: The underexplored potential of National Evaluation Societies. Evaluation and Program Planning (41), 47-57. Jang, S. (2000). The Appropriateness of Joint Committee Standards in Non-Western Settings: A Case Study of South Korea. In C. Russon (Ed.), The Program Evaluation Standards in International Settings: The Evaluation Center. Kakande, M. (2013). Professionalising Evaluation in Uganda: The Role of Uganda Evaluation Association, from http://gef.opm.go.ug/wp-content/uploads/2013/04/UEA-presentationEVALUATION-WEEK1.pdf Lahey, R., & Nielsen, S. B. (2013). Rethinking the Relationship Among Monitoring, Evaluation, and ResultsBased Management: Observations From Canada. New Directions for Evaluation, 2013(137), 45-56. Lee, Y.-S. (2002). Public Policy Evaluation in Korea: In Search for New Direction. In J. Furubo, R. Rist & R. Sandahl (Eds.), International Atlas of Evaluation. New Bruswick: Transaction Publishers.

Losee, J. (2011). Theories of Causality: From Antiquity to the Present. New Brunswick, New Jersey: Transaction Publishers. Love, A., & Russon, C. (2000). Building Evaluation Community: Past, and Future. Evaluation and Program Planning, 23(2000), 449-459. Lu, X., & Xie, D. (2005). Evaluation in the People's Republic of China. Journal of MultiDisciplinary Evaluation, 2(3). Lunt, N., & Trotman, I. (2005). A stagecraft of New Zealand evaluation. Evaluation Journal of Australasia (new series), 5(1), 3-10. Mackay, K. (2002). The Australian Government: Success with a Central, Directive Approach. In J. Furubo, R. Rist & R. Sandahl (Eds.), International Atlas of Evaluation. New Bruswick: Transaction Publishers. Marklund, S. (1984). Applicability of standards for evaluations of educational programs, projects and materials in an international setting. Evaluation and Program Planning, 7(4), 355-362. Patel, M. (2002). A meta-evaluation, or quality assessment, of the evaluations in this issue, based on the African Evaluation Guidelines: 2002. Evaluation and Program Planning, 25(2002), 329332. Ragin, C. (2007). Redesigning Social Inquiry: Fuzzy Sets and Beyond. Chicago: University of Chicago Press. Ragin, C. C., & Strand, S. I. (2008). Using Qualitative Comparative Analysis to Study Causal Order Comment on Caren and Panofsky (2005). Sociological Methods & Research, 36(4), 431-441. Reichert, C., & Rubinson, C. (2013). Kirq [Computer Programme], Version 2.1.9. Houston: University of Houston. Rihoux, B., & Lobe, B. (2009). The Case for Qualitative Comparative Analysis (QCA): Adding Leverage for Thick Cross-Case Comparison. In D. Byrne & C. Ragin (Eds.), The SAGE Handbook of CaseBased Methods. London: SAGE Publications. Rihoux, B., & Ragin, C. (2009). Configurational Comparative Methods: Qualitative Comparative Analysis (QCA) and Related Techniques. (Vol. 1-51). Thousand Oaks, CA: SAGE Publications, Inc. Rist, R., Boily, M., & Martin, F. (Eds.). (2013). Development Evaluation in Times of Turbulence. Washington: World Bank. Rogers, E. (2003). Diffusion of Innovations (5th ed.). New York: Free Press. Russon, C. (Ed.). (2000). Program Evaluation Standards in International Settings: The Evaluation Center. Rutman, L., & Mayne, J. (1985). Institutionalization of Program Evaluation in Canada. In M. Q. Patton (Ed.), New Directions for Program Evaluation (Vol. 25). San Francisco: Jossey-Bass. Sandelowski, M., Voils, C. I., & Knafl, G. (2009). On Quantitizing. Journal of Mixed Methods Research, 3(3), 208-222. Scheider, C., & Wagemann, C. (2012). Set-Theoric Methods for Social Sciences. Cambridge: Cambridge University Press. Scriven, M. (2004). Reflections. In M. C. Alkin (Ed.), Evaluation Roots: Tracing Theorists' Views and Influeneces (pp. 183-195). Thousand Oaks: Sage Publications, Inc. Scriven, M. (2011). Evaluation Bias and Its Control. Journal of MultiDisciplinary Evaluation, 7(15), 7998. Scriven, M. (2013). In Facebook Retrieved 16 November 2013, from https://www.facebook.com/MichaelScrivenEvaluator Segone, M., & Ocampo, A. (Eds.). (2006). Creating and Developing Evaluation Organisations. Peru: Asociacion Grafica Educativa. Segsworth. (2002). Evaluation in the 21st Century: Two Perspectives on the Canadian Experience. In J. Furubo, R. Rist & R. Sandahl (Eds.), International Atlas of Evaluation. New Bruswick: Transaction Publishers.

Sharp, C. (2003). Development of program evaluation in Australia and the Australasian Evaluation Societythe early decades. Evaluation Journal of Australasia, 3(2), 7. Shim, O. (2011). Performance Evaluation and Management in Korea: An Integrated System for Planning, Management and Budgeting Paper presented at the Improving the Quality of Public Services: A Multinational Conference on Public Management, Moscow. http://umdcipe.org/conferences/Moscow/papers/O-Taeg%20Shim_present%20paper.doc Silva, R., Brandao, D., Chianca, T., Firme, T., Marino, E., Lustosa, P., & Meirelles, C. (2006). Brazil: The Brazilian Evaluation Network. In M. Segone & A. Ocampo (Eds.), Creating and Developing Evaluation Organisations Peru: Asociacion Grafica Educativa. Sivagnanasothy, V. (2007). Institutionalization of Monitoring and Evaluation System in Sri Lanka Lessons learnt, Best practices, Issues, Challenges and the Way Forward. Spinatsch, M. (2002). Evaluation in Switzerland. In J. Furubo, R. Rist & R. Sandahl (Eds.), International Atlas of Evaluation. New Bruswick: Transaction Publishers. Straton, R. (2001). Evaluation research in Australasia: Moving forward. Evaluation Journal of Australasia, 1(2). Tanaka, H. (N.D.). Evaluation in Local Governments in Japan, from http://www3.grips.ac.jp/~coslog/activity/01/04/file/Bunyabetsu-14_en.pdf Tudawe, I., & Samranayake, M. (2008). Civil society partnership in promoting an evaluation culture in the development process experience of the Sri Lanka Evaluation Association. In B. Williams & M. Sankar (Eds.), Evaluation in South Asia. Kathmandu: UNICEF. Turner, D., & Washington, S. (2002). Evaluation in the New Zealand Public Management System. In J. Furubo, R. Rist & R. Sandahl (Eds.), International Atlas of Evaluation. New Bruswick: Transaction Publishers. UNDP. (2009). National Evaluation Capacities, Casablanca, Morocco. Virtanen, P., & Laitinen, I. (2004). Beyond Evaluation Standards. European Journal of Spatial Development, 13(1). Williams, B., & Sandar, M. (Eds.). (2008). Evaluation South Asia. Kathmandu: UNICEF Regional Office for South Asia. Yamaya, K. (2002). The Art of Policy Evaluation in Japan. In J. Furubo, R. Rist & R. Sandahl (Eds.), International Atlas of Evaluation. New Bruswick: Transaction Publishers. Yin, R. (2009). Case Study Research (Vol. 5). Los Angeles: Sage Publications