HERMAN KUI PERS Royal Netherlands Military Academy, Breda and Technical University, Eindhoven RUDY RI CHARDS ON Royal Netherlands Military Academy, Breda This article discusses active qualitative evaluation and modern sociotechnical design as parts of an approach for the evaluation of organizational change. Intervention in interactions is the most important similarity between these parts when considering the evaluation of organizational change.The two parts are discussed in terms of their theoretical core elements, practical consequences for research and dilemmas. Because, in practice, intervening in a changing organization is always accompanied by some important paradoxes, the different parts of the approach are presented as supplementary in practical evaluation of organizational change.The leitmotif on the basis of the case of the Royal Netherlands Military Academy here is that a paradoxical situation such as intervening in a changing organization needs a paradoxical intervention strategy. Introduction In this article we would like to highlight some starting-points and procedures used during evaluation research at the Royal Netherlands Military Academy (RNLMA). In particular, we would like to focus on a central aspect of our evalu- ation, the combination of different procedures combining various pre-existing methodological and theoretical orientations. This led to an evaluation approach and corresponding procedures: an active form of qualitative evaluation, guided by sociotechnical design-theory. We discuss the approach and its procedures as follows: rst, we elaborate on our approach by describing some core elements of the three parts, viz. qualitative evaluation, action research and sociotechnical design. Then, we highlight some differences and similarities between the three parts of the approach. Our next step is to elucidate the procedures involved as used during a study we are conducting at the RNLMA. Finally, we discuss some dilemmas and paradoxes that derive from our approach and its procedures in an active research setting. Evaluation Copyright 1999 SAGE Publications (London, Thousand Oaks and New Delhi) [13563890 (199901)5:1; 6179; 008196] Vol 5(1): 6179 61 05kuipers(ds) 11/2/99 3:45 pm Page 61 Since the changes in the international political and societal situation, the Dutch Armed Forces have reorganized and changed rapidly (Soeters and Richardson, 1996; Richardson et al., 1996). Organizational units in the Armed Forces, Army as well as Air Force and Navy, are being transformed into so- called Result Responsible Units (RRU): smaller and more exible units, managed in a more economic, business-like and performance-based manner. These units are in the process of formulating Mission Statements, introducing contract management to arrange the vertical and horizontal relations within the organization, decentralizing the personnel function and delegating responsi- bilities. This process of change was launched at the strategic level of the Dutch Armed Forces by the introduction of the policy of Improvement of Conduct of Business (Richardson et al., 1996). The most important features of this policy are similar to the aspects of New Public Management as described by Pollitt (1995). In practice, these changes are not without problems (Soeters and Richardson, 1996). One of the major problems is that strategic concepts are not taken for granted at the operational level. At this level, strategic concepts undergo a process of transformation and modication. The most difcult ques- tions at the operational level are: how to shape strategic concepts in an organization, given the organizational context? what are the consequences of this shaping process for organizational change? and, how can we organize support for organizational change? It is to answer these three questions that systematic and thorough evaluation of organizational change in the Dutch Armed Forces is needed. Our goal is to learn from several change initiatives in order to decide how to deal with change in future. To this end, we launched an active evaluation approach, that was combined with ideas about organizational design. One of our cases is the RNLMA. We will focus on this case in this article and at the same time discuss our overall approach. Evaluation of Organizational Change Before starting we formulated a research design for the evaluation (see Richard- son et al., 1996). This design was based on ideas of constructivist evaluation, on which we will elaborate later in this article. During the evaluation, the members of the evaluation team discovered synergy between different approaches in this evaluation project: a synergy between an active form of qualitative evaluation and design. By qualitative evaluation we refer to evaluation based on open qualitative strategies and methods in order to obtain integral knowledge of the organization as a unique system. By active we mean the constant dialogue with participants in the process of change in order to transform and modify ideas about organizational change, based on the results of interim evaluations. By design we refer to modern sociotechnical system theory as a tool to analyse and implement ideas on organization change. Therefore, our emerging knowledge of and our interventions in the organization are not only based on the interaction of several Evaluation 5(1) 62 05kuipers(ds) 11/2/99 3:45 pm Page 62 participants in the organization, but are also inspired by the principles of sociotechnical design as a normative framework. This does not, however, imply that we wanted to take over the position of the designers in this project, but we did wish to evaluate the design effort from the normative and analytical per- spective of sociotechnics. We consider qualitative evaluation and action research to be parts of the qualitative framework and we consider design as a specication of the modern sociotechnical framework. We discuss the parts of the framework as core elements of our approach, starting with qualitative evaluation and action research as specications of the qualitative framework. There are those who consider change and intervention to be a total contrast to evaluation with its more hands-off quality. We see this as part of a broader set of distinctions. In our opinion evaluation research can be classied along three inter- related dimensions: the degree to which evaluators distance themselves or become involved; the degree to which evaluators use a neutral or a normative analytical frame- work of interpretation; the degree to which evaluators passively observe instead of using the (interim) results to actively inuence the proceedings, as well as to nd out more about the system. The more the evaluator is involved, uses an explicitly normative analytical frame- work for the evaluation, the more actively evaluators participate in the discus- sions concerning what is to be done with (interim) results, the more possible it becomes, in our opinion, to speak of active qualitative evaluation (AQE). Involvement is demonstrated through the participative and interactive manner of evaluation. The evaluation is then regarded as the mutual product of the evaluator and those being evaluated with the goal being understand- ing the complex world of lived experience from the point of those who live it. The normative and analytical framework becomes clear when the results of the qualitative evaluation are positioned in a more general framework. Select- ing that framework is determined by the goal that has to be attained: in this case the evaluation of structural organizational change. The added value of the evaluator is then determined by the link that can be established between the world of lived experience and more general design and development prin- ciples (in this case what light can the evaluator shed on the RNLMAs unique project of change using a sociotechnical framework and what conclusions can be drawn concerning possible changes in the process of change?). Activeness is expressed by the deliberate use of intervention techniques by the evaluator in role. The evaluator intervenes deliberately in order to maxi- mize the impact of evaluation results. In practice this has meant that, on a number of occasions, use was made of conference techniques with the (interim) evaluation results providing the input. The conferences were set up according to familiar principles from the literature on organizational design (e.g. Weisborn, 1991). What remains to be resolved is whether such Kuipers and Richardson: Active Qualitative Evaluation 63 05kuipers(ds) 11/2/99 3:45 pm Page 63 interventions, which are designed to enable a discussion of evaluation results and to enhance their impact, still fall within the domain of the evaluation itself. We believe that they do and that an active, professional transfer of results is part of the evaluation process. In addition, such conferences yield evaluation data. In the RNLMA case the conferences uncovered a multi- tude of opinions on what responsibility would mean for the Academy and the stance of some of the stakeholders. In this article we particularly discuss this active side. The reader, however, must keep in mind that this is done from the perspective of the evaluator and not from that of the designer or change agent though now and again we could not resist the temptation to look at matters from their perspectives. Our choice of AQE framework is largely determined by the chosen object of evaluation, i.e. the evaluation of organizational change. We believe that a distant, neutral and passive description would contribute little. A factual representation of the proceedings in the change project is also of little value if the evaluator is unable to provide a link with more general insights in the eld of organizational change and design. For this we have used the sociotechnical framework. Although each process of change is unique and can only be understood from the world of experience of the participants, the added value of evaluation, we believe, is determined by its capacity to interpret this unique process within a more general analytical and normative framework. In the case of organizational change and design, sociotechnics provides such a framework. The RNLMA was given the assignment to take the concept of Result Responsibility and translate it into a exible organization with responsibilities placed at the lowest levels possible. Sociotechnics is a theory which is ideally suited to enable just that. It can be employed in two ways: as an evaluative and diagnostic framework for the analysis and explanation of structural dysfunctions in existing organizational processes; as a design methodology to nd structural solutions for such dysfunctions which will suit the specic organization. In our role as evaluation researchers we employed sociotechnics mainly as an evaluative framework to start a dialogue with the organization on the basis of those evaluations in order to nd possible solutions. This was consistent with our appointment as evaluation researchers and not as designers or change agents. Our role was to critically evaluate the design team that was to implement the Result Responsible Unit. Characteristic of our approach to evaluation research is that we devote atten- tion to the process as well as to the content. Both are crucial: an evaluation of only content would disregard the fact that the process of evaluation is essential for the acceptance of change and for mobilizing the knowledge and experience necessary to nd the right solutions. Evaluation 5(1) 64 05kuipers(ds) 11/2/99 3:45 pm Page 64 Qualitative Evaluation and Action Research as Parts of the Quali- tative Framework Qualitative methodology, research and methods have made tremendous progress during the last three decades. We think it is justied to mention Glaser and Strauss (1967), Denzin (1989), Glaser (1978), Lincoln and Guba (1985), Ham- mersley and Atkinson (1983), Miles and Huberman (1984), Strauss and Corbin (1990), Tesch (1990), Denzin and Lincoln (1994) and Stake (1995). Qualitative research methods and methodology have grown into a complex spectrum of more or less elaborated approaches, strategies, methods and techniques. However, this spectrum can be traced back to no more than three major theoretical perspec- tives, viz. constructivism, interpretism and human inquiry (Schwandt, 1994). The purpose of these perspectives is a common one: understanding the complex world of lived experience from the point of view of those who live it (Schwandt, 1994: 118). The theoretical basis of this general framework is that the qualitative researcher has to grasp the actors denition of the situation in order to under- stand the complex world of lived experience. This denition of the situation is made by actors in order to give meaning to events, phenomena and physical objects in their environment. Such meanings are derived from the process of social interaction with others and it is the structured meanings actors compose and use in everyday life that constitute the objective world. Society, from this perspective, takes on the form of a meaning structure. It is based on the de- nitions which meaningful others have in interaction with the actor. Social order in this context is the result of the agreement between actors on the denition of the situation. In order to understand meaning, the qualitative researcher must elucidate the process of meaning construction and clarify what and how meaning are embodied in the language and actions of social actors (Schwandt, 1994: 118). This elucidation itself is an interpretative process in which the researcher makes second order constructs of the construction made by actors. Of course, there are differences between the various qualitative perspectives within the overall paradigm, as between relativist and realist perspectives, more action oriented and more distant perspectives, pure qualitative strategies and mixed quantitative and qualitative strategies, etc. (see Schwandt, 1994). We will not, however, discuss these differences. Rather, we will focus on the combination of qualitative evaluation and action research (Schwandt, 1994; Lincoln and Guba, 1985). In this perspective, evaluation is part of a discourse and action with par- ticipants in a specic context, striving for democratic and educative goals. Eval- uating changing social networks, whether organizations or society as a whole, is not a distant, intellectual business of one or more so-called scientic experts. Evaluating these changing networks is an (inter)active interchange between (the knowledge of) the researched and the researcher, in order to bring forward fundamental change and to increase the democratic and educative possibilities of actors in their social network. In this view, there is no strict division between changing by acting and having scientic knowledge, rather, there is mutual learning by participants as well as the researcher (Boog et al., 1996). The pro- cesses and outcomes of evaluation can be seen as interactive constructions of the Kuipers and Richardson: Active Qualitative Evaluation 65 05kuipers(ds) 11/2/99 3:45 pm Page 65 meaning both researcher and researched give to intentions, rules, norms, values and actions and of the results of these constructions (see Lincoln and Guba, 1985; Patton, 1987, 1990; Chen, 1990; Scriven, 1991; Schwandt, 1994). The most impor- tant consequences for evaluation practised in the way described are: (a) that the aim evaluation is change, i.e. we want to communicate and gener- ate knowledge in and about a changing setting in which actors, goals and processes are continuously changing, as are the aims and procedures of the evaluation itself; (b) that the aim is to communicate and generate integral knowledge, i.e. knowledge in the sense of broad, complete knowledge of the setting or the settings processes in their context (holistic approach); (c) that evaluation itself is messy, i.e. interwoven with political processes; 1 (d) that as a result of (c), evaluation is a process of negotiation and inter- action; (e) that the object to be studied is a unique object, i.e. the knowledge we generate in the evaluation is time- and place-limited; (f) that the evaluation is formative, i.e. the purpose is to improve (the development of) a program and to make participants aware of its goals, developments and bottlenecks; (g) that as a result of (f) relevance, acceptability and understandability of (evaluation) processes and their outcomes become important; (h) that starting points of evaluation are sensitizing concepts, i.e. prior implicit or explicit theoretical notions on goals, (side) effects, processes, contextual inuences, etc.; and, (i) that a combination of research strategies, methods and techniques, quali- tative as well as quantitative, can be used in the evaluation. In our view, there are three dilemmas to be faced in this active evaluation approach. The rst dilemma is between participation (openness) and direction (closeness). Most researchers want to bring about change but they do not want to (and cannot) dictate this change. They try to solve this dilemma by acting as non-directive facilitators, i.e. the change agent as non-directive orchestrator of the democratic dialogue. However, here the researcher is confronted with the paradox: how to change a fundamentally undemocratic system by democratic means. The second dilemma of this approach is the double role of the researcher: as change agent as well as (qualitative) researcher. Both roles can interfere with each other and the borderline between both roles may be fuzzy. Apart from that, the role of change agent is often very demanding. One possibility is to divide the roles between different persons, but is that sufcient? The last dilemma we want to raise is how to deal with scientic theory and theory of practice. Is it important or even possible to use or develop theoretical insights in this kind of research? If we use theory, how can we prot most from it? If we develop theory, what is the (scientic) status of this theory? As we will discuss below we see an important contribution for theoretical insights into active evaluation to cope with all these dilemmas. Evaluation 5(1) 66 05kuipers(ds) 11/2/99 3:45 pm Page 66 The Core of the Modern Sociotechnical Framework Having discussed qualitative evaluation and action research we want to focus on the design part of our approach. More specically, by design we refer to Modern Sociotechnical Design (MSD), which was selected because of its use as a normative and analytical framework. We would like to emphasize once more that we did not use this framework as designers but as evaluators of the designs and the design process. We have diagnosed and interpreted actual developments against the norms and procedures of this theory. The Dutch version of sociotech- nical design is an applied theory, concerning the integral (re-)design of produc- tion processes, including the wider organization. This theory of practice is founded in a more fundamental theory of social systems (Luhman, 1984; Weick, 1969; De Sitter, 1973; 1993, 1994). Theory and practice are discussed in the work of Kuipers and Van Amelsoort (1990) and, in greater detail and sophistication, in the work of De Sitter (1994), the founder of the Dutch approach of sociotechni- cal design. MSD has its roots in the rst generation sociotechnical approach of the Tavistock Institute in the UK (see for a recent overview Passmore, 1995). In the Dutch approach, however, some fundamental theoretical problems are addressed on that basis, a rened design theory has been developed (De Sitter, 1994). In this article we only discuss a few of the basic characteristics of this approach. A rst characteristic is a fundamental view that people labour in a framework of social exchange and interact with the other players in the organization as part of a social network. Every workplace is a junction of exchange-relations within such a network of interactions. Norms, motives, aspirations and needs are devel- oped in this social interaction. People are conceived as social referring, sense- making systems. There is no inborn norm that guides human behaviour. Norms are always produced and reproduced in social networks operating in changing environments, not inside people. A second characteristic of practical design theory is the contrast made between classical and exible organizations. Classically designed organizations are based on simple tasks, multiple rules and complex formalized networks. The junctions (the workstations) in such mechanized networks offer very little space for the development of norms, sense-making or the solving of unexpected problems. In addition, the load of coordination and the loss of regulation capacity in such complex networks increase exponentially with the degree of the division of labour. The incapacity of the nodes in the classic networks to regulate and the heavy weight of coordination explain a lot of serious dysfunctions within classic organizations especially in circumstances of continuous change. The inability to handle change is an inherent characteristic of what we call classic design. These dysfunctions are precisely analysed in terms of the characteristics of different network structures in Dutch sociotechnical theory. It is a truism to say that these days organizations have to cope with a turbulent environment. Far more is demanded of most organizations in terms of exibility, innovation, quality, deliv- ery times, reduction of stocks, quality of work, etc. A central point of departure of sociotechnical design is that radical redesign of classic network structures is Kuipers and Richardson: Active Qualitative Evaluation 67 05kuipers(ds) 11/2/99 3:45 pm Page 67 needed to handle these new demands connected with change and turbulence. New network structures are based on complex tasks, few rules and simple net- works. The junctions in such new networks offer space, i.e. regulation capacity for norm-making, sense-making, problem-solving and commitment in interaction with other network-nodes, so that the burden of coordination can be drastically reduced. A third characteristic of this approach is the idea of redesign. A key idea of sociotechnical redesign is reduction of structural complexity, so that many forms of new networks become possible, depending on local conditions. Mixed self- directed teams are chosen most of the time at the micro-level and self-supporting strategic units at the meso-level. Drastic reduction of staff departments often results from redesign and these departments may be regrouped in mixed teams for integral support. Dutch sociotechnical theory has a well-developed methodology at its disposal, helping organizations to nd suitable solutions for regrouping and recoupling network structures. Much experience has been gained with the trans- formation process itself. The sociotechnical designer is confronted with a paradox of change, which is the counterpart of the earlier mentioned paradox: how to arrive at a participative or democratic organization through essentially directive or undemocratic means. On the one hand, top management need a clear and con- sistent change policy, on the other hand everyone in the organization must par- ticipate in making new sense of the organization. This should lead to a process in which a meaningful structure is the product of sense-making and interaction within the social network: redesign is self-design. The actors in the social network have to produce their own redesign based on knowledge of design principles. The central question MSD tries to answer is how to design and develop network structures which improve the quality of work and organization, especi- ally in uncertain and dynamic conditions. A central indicator of both the quality of work and organization is the self-regulating and self-learning capacity of people, units and the organization as a whole. In certain networks (bureaucratic, centralized, functionally specialized) the opportunities for selective and meaning- ful interaction are few and in other networks (organic, decentralized, function- ally unspecialized) they are many. MSD as a design-theory is focused on the precise architecture of organizations as networks depending on the characteristics of order-ows and the uncertainty of circumstances. MSD is strongly inuenced by interactionism. Of course, there are differences between active evaluation and MSD, as we will discuss below. Nevertheless, there is also a clear congeniality of spirit. Both approach change from an active, integral and holistic orientation. Also the uniqueness, internal validity and meaningfulness of relations and inter- actions are stressed in all the approaches referred to. They have much in common in their scientic philosophy and they all deviate clearly from mainstream posi- tivist scientic approaches. Active Evaluation and Sociotechnical Design Dutch modern sociotechnics offers a general design and implementation theory, without denying the uniqueness of each organization. The core laws of Evaluation 5(1) 68 05kuipers(ds) 11/2/99 3:45 pm Page 68 sociotechnical design, which are analytical in character and therefore true by de- nition, should be used as a general guide to nd suitable local solutions. The sociotechnical designer will teach organization members the essence of design theory so that they can participate in the design and development of their own organization. In this sense, the sociotechnical designer is a teacher and an expert who teaches the members of the organization how to change and redesign their own network structures, i.e. their own organization. Designers are directive in the sense that they teach fundamental design principles to all members of the organization, and non-directive in the sense that the members have to transform their own network structures. Organization members have to select and interpret sociotechnical design principles in the light of local conditions and constraints. The sociotechnical designer may support the organization in this transformation process in different ways. Initial power differences in the existing network are accepted as a fact of life. Paradoxically, however, the power of the top level has to be used to transform the hierarchical and rigid network structures and to strive for power equalization. Dutch experience has taught us that absol- ute commitment of one or more top managers to the transformation project is the most critical pre-condition for success. An important condition for cooper- ation between managers and designers is that top managers must subscribe to the values of MSD and they must be able to face the consequences of their actions. Mostly, this will, however, only be clear during implementation. In this way, the designer also has a conicting role. The modern sociotechnical framework con- trasts with the active evaluation on directivity. The active evaluator is not in a position to design in a directive way, but he or she is in a position to operate as a sounding-board inspired by the sociotechnical framework. Furthermore, notions on the direction of change are mostly absent in active evaluation. Most qualitative evaluation researchers argue that the inquirer should not take the lead in the implementation of the results of evaluation to improve the organization. This view is most obvious in the new realist evaluation approach (Pawson and Tilley, 1997). Another difference lies in the notion of power which is best described within a qualitative framework in constructivist and stakeholder evaluation (Lincoln and Guba, 1985; De Bourgoyne, 1994; Greene, 1994). Here, power is seen as a given and neutral even though major aspect or research topic in the evaluation. Power differences are to be ascertained and analysed in networks and are not, as in the modern sociotechnical framework, to be accepted and strategically used by the researcher. At best, even the activist inquirer who aims at revitalization, renewal and increasing the effectiveness of an organization is no more than an organizer of the democratic dialogue 2 . The statement evaluation is politics (Greene, 1994) therefore has no validity for the researchers commitment to lower or higher level members in the network structure. In the modern sociotechnical framework, as we have shown, it means a strategic choice for commitment of higher level members as a vital factor to break through the existing situation. There is also a difference in theoretical orientation between active qualitative evaluation and sociotechnical design. As we have shown above, MSD has its theoretical roots in system thinking. In both active qualitative evaluation and Kuipers and Richardson: Active Qualitative Evaluation 69 05kuipers(ds) 11/2/99 3:45 pm Page 69 modern sociotechnics, actors are seen as social-referring and sense-making. In both paradigms social norms, order, structure and meaning are produced in the interactive process. But active qualitative evaluation and the Dutch version of system thinking differ on some major points. In modern sociotechnics, for example, structure is not only a product of interaction, but also a determinant. Thus, people run the risk of becoming the prisoners of rigid, unproductive or chaotic networks created and formulated by themselves or by elites without their consent. Organizations can behave for long periods of time as if they were closed systems (Weick, 1969). In Weicks terms, we can speak of a real structure of the organization. Usually qualitative perspectives are inclined to stress that struc- tures are the ever-changing products of spontaneous and creative interaction processes. In the most active qualitative evaluation, objectivity in terms of the world of facts and what we take as objective knowledge and truth is mostly absent. Reality is pluralistic: people construct their understandings from experi- ence and from being told what the world is, not by discovering it whirling there untouched by experience (Stake, 1995: 100). Even in symbolic interactionism (social) structure and social culture are abstract systems that only exist for and have inuence on people by their sense-making. Structure can only be dened and seen through the eyes of actors that construct and preserve that structure (Wester, 1992). Modern sociotechnics, by contrast, even speaks of core laws of structure design, which are analytical in character and therefore true by de- nition. For example: if an individual or a group is given responsibility for a com- plete product instead of only part of it, a structural basis is created, by denition, for intrinsic involvement. These differences in theoretical orientation did not prevent us combining frameworks, especially in research practice. On the contrary, we felt that both frameworks are supplementary as we will describe in our approach below. It has to be mentioned that in this context sociotechnical design is not used as a design methodology, but as a normative perspective to analyse, judge and inuence ini- tiatives of organizational change. Active Qualitative Evaluation and Design in the Case of the Royal Netherlands Military Academy One of the units in the Dutch Armed Forces that has to change into a Result Responsible Unit is the Royal Netherlands Military Academy (RNLMA). We use the RNLMA to develop, test and rene our approach. The RNLMA was selected for evaluation on the basis of accessibility and the progress of change at the time (Richardson et al., 1996). The RNLMA trains and educates ofcers for the Dutch Army as well as the Air Force. The Academy has existed for 165 years, and focuses on military training as well as scientic education and research. Scientic education and research deal with several aspects of Management Science (materiel, logistics, economics, organization and personnel) as taught at regular universities. In 1992 the Academy was reduced in size and its educational system was reorganized. Scientic education in the academy was broken up into two phases: RNLMA I and RNLMA II, with two years of military service outside the Evaluation 5(1) 70 05kuipers(ds) 11/2/99 3:45 pm Page 70 Academy in between. RNLMA I is divided into twelve modules of nine weeks each: three modules of military training and nine modules of scientic education. RNLMA II, the period after a two-year military service period, 3 contains ve modules of scientic education. Having completed the RNLMA I course, cadets become Lieutenants in the Army or Air Force, and are ready to function at middle level (Captain, Major) and higher level (Lieutenant-Colonel, Colonel, Brigadier, Major-General, Lieutenant-General and General) management func- tions in the Dutch Armed Forces. Since January 1996 the RNLMA has had the status of Result Responsible Unit. This means that the Commandant of the RNLMA has to reorganize it along the lines of a small, exible, economical and performance-based unit. Specications of control of resources within the unit, between the unit and other units and between the unit and the parent organization must be formulated in a manage- ment contract with the Commander-in-Chief of the Dutch Armed Forces (Richardson et al., 1996). This contract came into force in November 1996. This process of organizational change in the RNLMA was prepared by a special internal project group, set up by the Commandant of the RNLMA in January 1995: the Project Group Performance Accountability. This group consisted of ve persons: a professor in information science (the chairman), the chief of the department of planning of the RNLMA, the controller of the RNLMA (chief department of nances) and two advisors (nancial and organizational) from the Department of Defence in The Hague. This group had the task to investigate the consequences of performance accountability for the internal structure and func- tioning of the RNLMA and to prepare its internal implementation. The project group had six work groups, covering the most important elements of preparing a new organization (budgeting, processes/product description, management system, information system, implementation and performance indicators). During this process of preparation, a small evaluation team was installed by the Commandant of the RNLMA, alongside the project group, consisting of three members of the RNLMA Military Management Faculty. It is this team that we are part of. The installation of the evaluation team rst became an item when the faculty discussed its new research program for 1995/1996. One of the projects, called Evaluation of Initiatives in Organizational Change in the Dutch Armed Forces, was formulated in terms of qualitative evaluation and in-depth case studies (see: Richardson et al., 1996). The board of project-leaders of the facultys research program decided to investigate the process of change at the RNLMA. The evaluation team started this research in 1995. As described here, we speak of an internal evaluation (Richardson et al., 1996). Procedures in Active Qualitative Evaluation and Design It is important to stress that we had no inuence in designing the process of organizational change as described above and initiated by the Commandant of the RNLMA. Also it is important to stress that this design was not explicitly based on sociotechnical principles. When we were installed as an evaluation team, we decided to use these principles as a normative mirror to evaluate the process of Kuipers and Richardson: Active Qualitative Evaluation 71 05kuipers(ds) 11/2/99 3:45 pm Page 71 change so far and to inuence this process. The Commandant and the project team were not familiar with our evaluation approach. In the following we will describe which procedures we used, concentrating on the active side of the evalu- ation process. However, the reader should bear in mind that this was always based on (the interpretations of) evaluative data about the change project and the design. Preliminary Moves The rst step (January 1995) was to make clear the outlines of our approach and research questions to both Commandant and project group. After discussion, the following research questions were formulated: (a) to what extent and in which way will the ideas of the strategic level be transformed, (re)shaped and modied at the RNLMA? (b) which interpretations and mechanisms play a central role in this trans- formation process? (c) to what extent can the process and possible outcomes of this transform- ation process at the RNLMA be improved? (d) how can the outcomes be implemented in the existing organization? (e) to which kind of organization will this process of implementation lead? After agreeing on these research questions, it was decided that we attend the (two-weekly) meetings of the project group. In these meetings, the interim results of the several work groups were discussed and future steps for the preparation and implementation of the new organization were formulated. During these meetings our role was to register the discussions and other interesting events such as the standpoints of certain participants, the way these were legitimized and the way coalitions were formed. Apart from the meetings, we gathered documents dealing with the ideas of the strategic level of the Armed Forces on (aspects of) conduct of business (see also Richardson et al., 1996). These documents can be seen as the strategic sensitizing concepts, which the project group has to trans- late, rene and elaborate on the operational level of the RNLMA. We stored all these data in a database for qualitative analysis, called KWALITAN (Peters et al., 1989) and analysed them using qualitative coding techniques (Wester, 1992; Miles and Huberman, 1984). We discussed the results of the analyses weekly in our evaluation team. The Miniconference During the analyses and discussions in the evaluation team, we discovered some major and regular points of discussion within the project group. Approaching the summer of 1995, we decided to organize a discussion of these and other topics with the members of the project group and the chairs of the work groups. The goal of the evaluation team when organizing this discussion was to generate some views and concepts of project group and work group members on these topics as well as to give feedback from a sociotechnical point of view. The discussions took place in July 1995 in the form of a half-day miniconference. Here we see clearly the active participation of the evaluation researcher in the discussion of Evaluation 5(1) 72 05kuipers(ds) 11/2/99 3:45 pm Page 72 organizational change at the RNLMA. The following four main topics were brought forward at the conference and will be discussed briey below. i) The composition of the project group and its work groups. The work groups consisted solely of members from the four departments of the academy. Each work group had to deal with one aspect of the reorganization: one with per- formance indicators, another with budgeting, yet another with the denition of processes and products, again another with the management system. The func- tion-based work groups were to produce their reports independent of each other after which these reports were integrated. The discussion centred around whether this method was adequate to nally arrive at an integrated picture of the new organization. In the view of some participants it was not. The method would lead to a differentiated, fragmented picture of the organization, leading to much more confusion rather than to an integrated picture and more clarity. The members of the work groups were hardly committed to the whole task. The project organiz- ation was based on a functional division of labour and the dysfunctions men- tioned could easily be predicted from a sociotechnical perspective. ii) The nature of the documents of the strategic level of the Armed Forces. As pointed out earlier, these documents are meant to be dealt with as sensitizing concepts for reorganization. Two different opinions on the status of these docu- ments emerged. One camp was convinced that reorganization should take place exactly as dictated by the strategic level in terms of contents and path, while another camp wanted to leave as much room as possible for their own organiz- ation to transform and modify these concepts. From our sociotechnical point of view, transparency and agreement concerning strategy are absolute prerequisites for internal (re)design. iii) The contents of these documents. The importance of embedding responsibility and performance accountability as deep as possible in the organization is stressed constantly in these documents. However, the development of structural con- ditions for the organization to be able to implement these ambitions had not yet been thought of. A central design-rule of sociotechnical design is that certain pre- conditions in the production structure should be fullled before responsibilities can be decentralized. iv) The (lack of) methodology of change. From the start, there was much con- fusion in the project group on the time path of their work. Should their nal docu- ment be denitive in January 1996, when the RNLMA was ofcially registered as a Result Responsible Unit? Or was that the date for the presentation of a rough starting framework, which would be rened during 1996 and become denitive in January 1997? Another bottleneck was the lack of agreement on the composi- tion of the new organization according to the principles of result responsibility. What is the core of result responsibility? What does the organization want to achieve? What is needed to achieve a minimal level of result responsibility? How are these ideas to be implemented in the future? Kuipers and Richardson: Active Qualitative Evaluation 73 05kuipers(ds) 11/2/99 3:45 pm Page 73 Our conclusion at the end of the conference was that the conference was very helpful in understanding the (dys)functioning of the changing organization. Informing the Management Team The next step for the evaluation team was to analyse outputs and discussions of the miniconference, in terms of future actions. For the evaluation team, these outputs were the immediate input for the presentation of the progress of our research to our principals, the Commandant of the RNLMA and his manage- ment team. We decided to try and t the results of the miniconference into a single perspective. This was intended to improve the quality of the discussions on (aspects of) organizational change and to initiate a strategy of implemen- tation. The perspective was mainly inspired by the Modern Sociotechnical Design described earlier. We were convinced that this perspective would give an integrated view of the elaboration and implementation of organizational change at the RNLMA to date. The perspective would in our view also offer the struc- tural pre-conditions within which several initiatives for the organization of responsibilities and performance accountability could optimally be developed. It is clear that the design part of our approach is put on stage here. The presen- tation of our perspective in the management team took place on three occasions in OctoberNovember 1995. The three main parts of this presentation were as follows. Firstly, we presented our evaluation of organizational change and result responsibility at the RNLMA. Some global but important starting points were stressed. For example, we stressed that the introduction of result responsibility in an organization is much more than merely introducing (technical) systems, bureaucratic rules and norms, and a punishment/reward system to obtain such responsibility. We stressed that structural conditions must rst be developed in order to create room for the denition of responsibilities and tasks at each level of the organization (Kuipers and Van Amelsfoort, 1990; De Sitter, 1994). At the same time, a cultural change can be initiated and made visible through the development of new forms of management and leadership, new forms of com- munication and through new initiatives. We stressed that such a strategy leads to more conciseness, involvement and initiative in the organization. Secondly, a possible strategy for elaboration and implementation of the changing process at the RNLMA was presented. This strategy was intended to solve the problems in the existing strategy as they had surfaced in the evaluation, and was inspired by sociotechnical notions. The strategy started with the organization of a broad strategic management conference of RNLMA staff and line managers and their immediate subordinates. A rst goal of this strategic conference was to dene and stress the importance of the global guidelines and starting points of the strategy for change. This was to be mirrored in the environ- ment of the RNLMA following on from a Strengths, Weaknesses, Opportunities and Threats analysis (SWOT). Another goal was to translate global guidelines into specic principles of result responsibility and to dene future actions for each part of the organization. The conference took place over two days in Feb- ruary 1996. The evaluation team participated in this event: one member as Evaluation 5(1) 74 05kuipers(ds) 11/2/99 3:45 pm Page 74 facilitator and one member as active stakeholder. The results were described in a substantial report which contains a SWOT-analysis, an outline business model and an outline management contract. As a result of the conference, the pro- ject group was disbanded and the responsibilities for further elaboration and implementation of the change process were brought back into the line organization. Thirdly, our presentation contained a possible strategy for implementing the process of change. This proposal was based on the observation that the top-down implementation strategy was rather impotent. The members of the project team and the top management complained that they got no response. The main prin- ciple of our proposal was the organization of bottom-up development of ideas on further aspects and principles of result responsibility: what does it mean for our line organization? This is further described below. Bottom-up Discussion and Integration of the Result Responsibility Principles Having presented a strategy for the elaboration and implementation of the global guidelines and starting points of change, this strategy was approved during the conference. The idea is to organize bottom-up discussions to give concrete form to the guidelines of result responsibility in each part of the organization. This started around November 1996. Within the global guidelines and starting points, the different departments are formulating their view on result responsibility and organizational change. One integration team per organization line is responsible for the integration of their discussion. The goal is to complete the discussion in each division with a memorandum of standpoints of that specic organizational division. The role of the evaluation team was at rst facilitative by formulating the rules of the game. 4 Some of these rules are: (a) the different departments per division create their own view on result responsibility and organizational change as input for discussion to the integration team; (b) the integration team per division must include at least one member of each department who must be acceptable to their department; (c) the members of the integration team must have some expertise on result responsibility and organizational change; (d) the members of the teams are responsible for the feedback to their department and should constantly put forward the standpoints of that department; (e) the means (time, budget, expertise etc.) for those participating in the teams must be available. Apart from these rules some cultural norms for the support of the process in the teams were also articulated. These included open discussion, tolerance for mis- takes, expertise is more important than hierarchy, teamwork, constant awareness of the situational context and active involvement. Furthermore, one member of the evaluation team was to participate in the Kuipers and Richardson: Active Qualitative Evaluation 75 05kuipers(ds) 11/2/99 3:45 pm Page 75 discussions of each integration-team. While participating in the discussion the theoretical insights on organizational change of members of the evaluation team were put forward. This is the point in time that the process of organizational change and result responsibility at the RNLMA and the accompanying work of the evaluation team have reached. Lessons Learned The main purpose of this project was to learn how to deal with fundamental organizational change as an active evaluator. The active evaluator aims at maxi- mizing the effect of his/her evaluative efforts in support of a sound process of change, with regard to contents as well as form. The criteria for what is sound are derived from insights from organizational design and development. In particular, we have made use of insights from sociotechnics. From the above description it is apparent that whilst the active form of evaluation, based on a framework of sociotechnical concepts, has been very useful, it has not been a guarantee of success. Intervention techniques for organizational design proved to be useful to actively disseminate the results of the evaluations. The conferences themselves were also a means to gather information about the change project. The sociotechnical frame- work proved to be a potent means to interpret the dysfunctions regarding the change process at the RNLMA in a normative-analytical way. It also allowed structural remedies to be identied in such a way as to give the Result Respons- ible Unit (RRU) concept more weight within the RNLMA. However, this did not imply that these structural remedies were introduced without resistance. The gap between the professed RRU ambitions and actual practice clearly manifested itself. In practice the RRU concept was seen mainly as an administrative, bureau- cratic intervention, shifting hitherto central responsibilities to the Commandant of the RNLMA without many consequences for internal management. Ofcially, the RRU philosophy meant that the awareness of result responsibility was to be mobilized down to the level of people carrying out the work and that the con- ditions for this were to be created in the organization. The normative evaluation also helped isolate empty rhetoric within the RRU philosophy. By empty rhetoric we mean that people say one thing and do the other. On the one hand they commit themselves to structural innovation within the RRU framework an innovation aimed at dening the right conditions for the mobilization of self-responsibility in an atmosphere of mutual trust. On the other hand the actual routine of the organization is strongly dominated by deeply ingrained bureaucratic mechanisms, such as micro-management, the drive for promotion, risk avoidance, and a need for regulation and control. Such mechan- isms can be perfectly explained from a sociotechnical point of view as belonging to the pre-existing structure. Fundamental innovation means sticking out your neck and learning from mistakes. But avoiding real risks and preventing mistakes is still the best guarantee for promotion in the old structure; and promotion is sacred. Although people have mastered the rhetoric of modern management, it still goes against deeply rooted bureaucratic principles. Consequently, in spite of Evaluation 5(1) 76 05kuipers(ds) 11/2/99 3:45 pm Page 76 the professed support, the RRU project has ultimately yielded little in terms of innovation, and mainly adopted the guise of administrative, bureaucratic regu- lation. All in all, we have concluded that active qualitative evaluation (AQE) proved to be a useful means to put the RRU project in a broader and more principled framework and to make the implications of a consistent translation down to the basic operations of the organization clearly visible. However, this has not led to a broad implementation. Only in one part of the organization the faculty was a more or less serious attempt made to translate the RRU philosophy down to the level of people carrying out the work. This has resulted in an increased responsibility of the departments, which now manage their own budgets. The sub- culture in the faculty proved to be a relatively favourable breeding ground for RRU ideas. Insight into the ultimate consequences of RRU may have frightened off certain key gures in other parts of the RNLMA. Of course, AQE in itself is not and cannot be a guarantee for organizational innovation. But it can support it. In our experience successful innovation projects in organizations have one thing in common: there is always one central gure who leads the project, and who inspires it to break through the vicious circle of bureau- cracy. AQE cannot supersede such gures, but in organizations where they are present AQE can full an important role. It can act as a sounding board, allow- ing them to broaden their vision and stimulating dialogue, whilst helping form the process of participation necessary to mobilize members of the organization and exploit present know-how to the benet of the design process. Is this a reason only to embark on a project with organizations in which the potential for change is present in the gure of an inspiring change manager who guarantees a serious attempt at innovation? We think not. At the start of this kind of project, it is very seldom clear whether such persons are there in the organiz- ation. AQE can make a contribution towards mobilizing and developing poten- tial talent. It may make potential change managers aware of the structural causes of problems in their organization and the possible benets of fundamental inno- vation; though this can never be fully guaranteed. Those engaged in AQE have to take the risk, to have been a Don Quixote, in hindsight. Without trying in this active way you can be certain of making no impact as an evaluator of organiz- ational change. Once again, it is necessary to emphasize here that our role in this project is essentially different from that of a sociotechnical designer. As (active) evaluators we did not, on principle, become designers ourselves, but we acted as a sounding-board for an organization that was supposed to produce its own re- design. That, on rare occasions, we could not resist the temptation to climb into the chair of the designer does not detract from this principle and is partly under- standable in terms of the paradoxes that accompany fundamental organizational change. Notes The authors would like to thank Sjaak Rovers and Harry Kirkels for their help in trans- lating this article. Kuipers and Richardson: Active Qualitative Evaluation 77 05kuipers(ds) 11/2/99 3:45 pm Page 77 1. Antonio (1995: 21) denes evaluation as a common process of searching and negoti- ating for shared meaning and values that connect each other between different stake- holders. 2. For instance, this was the case in the Swedish LOM project. 3. This does not apply to cadets who become pilots. They will have four or ve years of instruction and training in the Netherlands as well as in the United States. Cadet- engineers also receive longer training. 4. In this case the role of (active) evaluator changed into that of process facilitator. With hindsight it may not have been a wise choice because explicit responsibility was now taken for the process of change itself, mixing evaluation and process responsibility. References Antonio, S. (1994) Theory-driven Evaluations. Verslag van een Workshop van Prof. H. T. Chen [Theory-driven Evaluations: Report of a Workshop with Prof. H. T. Chen], Belei- dsanalyse 94(4): 1622. Boog, B., L. Keune, H. Coene and R. Lammerts (1996) Theory and Practice of Action Research. Tilburg: Tilburg University Press. Chen, H. T. (1990) Theory-driven Evaluation. Newbury Park, CA: Sage. De Bourgoyne, J. G. (1994) Stakeholder Analysis, in C. Cassell and G. Symon (eds) Qualitative Methods in Organizational Research, pp. 187207. London: Sage. De Sitter, L. U. (1973) A System-theoretical Paradigm of Social Interaction; Towards a New Approach to Qualitative System Dynamics, Annals of System Research (3): 10940. De Sitter, L. U. (1993) A Sociotechnical Perspective, in F. van Eynatten The Paradigm That Changed The Workplace. Stockholm/Assen: The Swedish Centre of Working Life, Van Gorcum. De Sitter, L. U. (1994) Synergetisch Produceren [Synergetic Producing]. Assen: Van Gorcum. Denzin, N. K. (1989) The Research Act, 3rd edn. Englewood Cliffs, NJ: Prentice-Hall. Denzin, N. K. and Y. S. Lincoln (eds) (1994) Handbook of Qualitative Research. Thousand Oaks, CA: Sage. Glaser, B. (1978) Theoretical Sensitivity: Advances in the Methodology of Grounded Theory. Mill Valley, CA: Sociology Press. Glaser, B. and A. L. Strauss (1967) The Discovery of Grounded Theory: Strategies for Qualitative Research. Chicago, IL: Aldine. Greene, J. C. (1994) Qualitative Program Evaluation, in N. K. Denzin and Y. S. Lincoln (eds) Handbook of Qualitative Research, pp. 53044. Thousand Oaks, CA: Sage. Hammersley, M. and P. Atkinson (1983) Ethnography: Principles in Practice. London: Tavistock Publications. Kuipers, H. and P. van Amelsfoort (1990) Slagvaardig Organiseren [Alert Organizing]. Deventer: Kluwer. Lincoln, Y. and E. Guba (1985) Naturalistic Inquiry. Beverly Hills, CA: Sage Publications. Luhman, N. (1984) Soziale Systeme: Grundriss einer Allgemeinen Theorie [Social Systems: Basics for a General Theory]. Frankfurt am Main: Suhrkamp. Miles, M. B. and A. M. Huberman (1984) Qualitative Data Analysis: A Method to the Madness. London: Sage. Passmore, W. (1995) Social Science Transformed: The Social Technical Perspective, Human Relations 48(1): 121. Evaluation 5(1) 78 05kuipers(ds) 11/2/99 3:45 pm Page 78 Patton, M. Q. (1987) How to Use Qualitative Methods in Evaluation. Newbury Park, CA: Sage. Patton, M. Q. (1990) Qualitative Evaluation and Research Methods. Newbury Park, CA: Sage. Pawson, R. and N. Tilley (1997) Realistic Evaluation. London: Sage. Peters, V., F. Wester and R. Richardson (1989) Kwalitatieve Analyse in de Praktijk en Han- dleiding bij KWALITAN versie 2.1 [Qualitative Analysis in Practice and Manual of KWALITAN version 2.1]. Nijmegen: SWI-reeks. Pollitt, C. (1995) Justication by Works or by Faith? Evaluating the New Public Manage- ment, Evaluation 1(2): 13354. Richardson, R., J. Soeters and H. Kuipers (1996) Evaluation of Organizational Change in the Dutch Armed Forces, Evaluation (2)1: 722. Schwandt, T. (1994) Constructivist, Interpretivist Approaches to Human Inquiry, in N. K. Denzin and Y. S. Lincoln (eds) Handbook of Qualitative Research, pp. 11837. Thou- sand Oaks, CA: Sage. Scriven, M. (1991) Evaluation Thesaurus, 4th edn. Newbury Park, CA: Sage. Soeters, J. and R. Richardson (1996) Van Macro- Naar Micro-Evolutionaire Veranderin- gen in de Krijgsmacht [From Macro- to Micro-Evolutionary Changes in the Armed Forces], Bedrijfskunde 68(4): 5160. Stake, R. (1994) Case Studies, in N. K. Denzin and Y. S. Lincoln (eds) Handbook of Qualitative Research, pp. 23647. Thousand Oaks, CA: Sage. Stake, R. (1995) The Art of Case Study Research. Thousand Oaks, CA: Sage. Strauss, A. L. and J. Corbin (1990) Basics of Qualitative Research: Grounded Theory Pro- cedures and Techniques. Newbury Park, CA: Sage. Tesch, R. (1990) Qualitative Research: Analysis Types and Software Tools. New York: Falmer Press. Weick, K. E. (1969) The Social Psychology of Organizing. Reading, MA: Addison Wesley. Weisborn, M. R. (1991) Productive Workplaces. San Francisco, CA: Jossey-Bass. Wester, F. (1992) Strategien Voor Kwalitatief Onderzoek [Strategies for Qualitative Research]. Muiderberg: Coutinho. HERMAN KUI PERS is Professor of Psychology at the Department of Social Sciences and Philosophy at the Royal Netherlands Military Academy, Breda,The Netherlands. He is also Professor of Sociotechnical Design at the Technical University, Eindhoven. His current research interests include (evaluation) research into organizational and strategic changes in civic and military organizations. RUDY RI CHARDSON is Assistant Professor of Methodology at the Department of Social Sciences and Philosophy at the Royal Netherlands Military Academy, Breda,The Netherlands. His current research interests include (evaluation) research into organizational and strategic changes in civic and military organizations. Kuipers and Richardson: Active Qualitative Evaluation 79 05kuipers(ds) 11/2/99 3:45 pm Page 79