Documenti di Didattica
Documenti di Professioni
Documenti di Cultura
Abstract
This case is about monitoring and evaluation (M&E) challenges in the Punjab Department of Health
(DOH). Despite his substantial experience of working in senior managerial positions, Arif Nadeem, the
department secretary, finds himself somewhat lost in the numerous department related reports and
data sheets that keep on coming from various quarters. He feels under-informed and over-informed
at the same time.
DOH regularly collects data on a range of indicators and there are multiple systems in place to
collect the data from various health facilities. The following four systems are important. First, the
District Health Information System (DHIS) collects data on around eighty indicators covering treatment
and spread of communicable and non-communicable diseases, human resources, facility utilization, etc.
Second, the M&E assistants (MEAs) inspect primary health care facilities and report on fourteen indica-
tors covering the number of patients attended, staff presence, medicine availability, public opinion, etc.
Third, the DOH field operatives visit health facilities and report on various aspects of functioning in
monthly meetings of officials at the district level. Since 2011, they have been using simple smartphone
based applications to enter data on site and transmit it to the points of analyses instantaneously. Fourth,
progress on development projects is reported every month on prescribed pro forma covering physical
progress as well as the amount spent. In addition, tertiary care hospitals report on various aspects of
their functioning on need basis and receipt/expense statements are regularly prepared by the budget
and accounts section in the department.
Often there is too much information to absorb. There is no effective system of filtering and
processing information according to the needs of various managerial tiers. Arif realizes that he needs
a dashboard that can provide just enough detail to various users. Arif and his team deliberate on the
choice of indicators for the dashboard. There is substantial disagreement on what to include and what
to leave. The disagreement partly emanates from a lack of clarity on the mandate of the department
and its senior management. Farasat, a key team member, proposes eight key areas for the dashboard.
He suggests displaying these eight areas on the main screen and creating links to detailed district,
tehsil and facility-wise data on selected indicators.
Keywords
Public sector governance, performance dashboard, monitoring and evaluation, collecting and managing
performance data
1
Suleman Dawood School of Business, Lahore University of Management Sciences, Pakistan
Corresponding author:
M. Ahsan Rana, Suleman Dawood School of Business, Lahore University of Management Sciences, Pakistan.
E-mail: ahsan.rana@lums.edu.pk
April 2013: Arif Nadeem felt overwhelmed. He had been working as the Secretary of the Punjab
Department of Health (DOH) for almost a year and was still finding it difficult to keep track of what
was happening in the department and the provinces health sector. This was hardly due to inexperience.
He had previously been the secretary in two other departments and was well versed in the technique
and practice of management. However, he did not feel on top of things. Although DOH regularly
collected and passed on to him substantial volumes of information on a range of indicators, there
was hardly a system that could filter information to suit his decision needs. No wonder, he felt under-
informed and over-informed at the same time. He realized that he needed a dashboard that presented
updated information on key indicators to suit his everyday information needs.
coordination officer (DCO)the chief bureaucrat in the district. He was assisted by a number of
officials, such as district officers (DOs), deputy district officers (DDOs), assistant district officers
(ADOs) and medical superintendents of various THQ and DHQ hospitals (Exhibit III).
DHIS
DHIS was by far the most comprehensive data collection system in the department. It was launched in
2006, initially in a few districts but was extended to the entire province by 2009. It was based on data
reported by the staff of PHC and SHC facilities on various aspects of functioning of their facilities.
This data covered seventy-nine and eighty-three clinical and nonclinical indicators for PHC and SHC
facilities, respectively. Clinical indicators covered forty-three communicable and noncommunicable
diseases, whereas nonclinical indicators covered vacancies, presence of staff, utilization of facilities,
patients treated, availability of medicine, budget, etc. (Exhibit IV for DHIS indicators).
PHC and SHC facilities initiated monthly reports on separate pro forma. The information was derived
from twenty-four registers maintained by concerned officials in the facility. The whole system was
paper-based. Although it was the responsibility of the facility incharge to ensure that reporting was accu-
rate, in practice he/she neither had the time nor incentive to carefully check what was being reported.
To minimize the possibility of misreporting, DOH introduced lot quality assurance sampling (LQAS)
as a quality assurance mechanism. LQAS involved drawing samples in each district and cross checking
data through field visits. In practice, however, LQAS testing did not become a regular feature of data
collection at DOH.
DHIS data was consolidated in the district DHIS cell each month. The district cell converted the data
into soft form and generated district reports for the benefit of EDO, DCO and other managers at the
district level. These reports were also circulated to incharges of PHC and SHC facilities in the district.
Shortly thereafter, the EDO held his monthly meeting with his district-based and field teams to review
current progress on various indicators.
The district DHIS coordinators sent a copy to the Directorate General of Health Services (DGHS),
where the director (MIS) consolidated district reports into a provincial DHIS report. DGHS published
quarterly and annual reports, which were widely shared with a range of stakeholders including the
secretary, other senior DOH managers, EDOs, heads of teaching institutions, various project managers
and international aid agencies.
Potentially, DHIS reports contained useful information that could inform management decisions.
For example, the annual reports for 2011 showed that per capita attendance at the outpatient departments
in PHC and SHC facilities was only 0.90 for Punjab, that is, on average, staff comprising 100 persons
was serving around ninety patients per unit of time. These reports also showed wide variation among
districts0.31 for Lahore and 1.41 for Chakwal. These were useful information bits for the secretary
and other managers when they had to make decisions and policies regarding allocation of human and
financial resources. Whether or not this actually happened was a different question altogether.
format to record this information and a report was made usually when an aberration was found, but not
otherwise. Mostly verbal feedback was provided in monthly meetings of EDOs with facility incharges.
In case of a serious observation, a written report was made to the DCO and the DOH, which could initiate
appropriate action thereupon. These reports were rarely sent for the secretarys perusal.
To facilitate reporting from field inspections, DOH introduced smartphone-based data reporting in
February 2011. The new regime differed from DHIS and MEA data collection streams to the extent that
data from the field was directly recorded in soft form and transmitted to several locations instanta-
neously. Simple android-based applications were specifically developed for this purpose and installed
on smartphones provided to field officials. The new system was introduced in eighteen districts in the
province and 392 smartphones were provided.
Each time a supervisory official visited a health facility, he recorded observations in the smartphone.
Date, time and location were electronically stamped on the reports, which were instantaneously fed into
a central backend system at PHSRP. Pictures could also be uploaded to ward off the possibility of fake
reporting. So a visiting official could simply have himself photographed with the facility staff and send
the group photograph to the data repository as proof of his visit as well as staff attendance.
A set of twelve indicators was used for data collection (Exhibit VII). These related to staff attendance,
medicine stock out, number of outpatient visits and deliveries, functionality of equipment and general
upkeep of the facility. Data received from the field was not aggregated at the provincial level. Similarly,
time-series analyses were not carried out to see long-term trends.
Occasionally, DOH sent its senior managers to tour health facilities in various districts. Sometimes,
but not always, observations from these visits were recorded as tour notes. Being infrequent and unstruc-
tured, these visits did not produce data that could be regularly and reliably used in decision-making.
EDOs (health) in the districts maintained category-wise data on staff employed in various wings/
facilities. These officials were responsible for recruitment, posting, transfer and other service matters
of DOH employees according to a hierarchical system of exercise of authority. Service matters of
employees up to basic pay scale 16 were mostly dealt with by the respective EDOs, and service matters
of employees from basic pay scale 17 and above were dealt with by DGHS. There was no comprehen-
sive database on employees listing their professional qualifications and experience, service history,
special skills, salary, performance against targets, complaints, etc. In the absence of such a database, it
was difficult for the secretary to plan an effective deployment of this large workforce. Information on
the human resource was presented to the secretary as and when desired by him.
The DOH budget was consumed at two levels: 5060 per cent was consumed at the provincial level
and the remaining was spent at the district level. In 20112012, salaries comprised about 25 per cent of
the total current expenditure and utilities consumed 44 per cent of the total budget. Record of receipts
and expenditure was kept at the district and provincial levels. Monthly reports were sent to the deputy
secretary (budget), who consolidated these into a single statement for the perusal of the secretary out-
lining total budget allocation under various heads and expenditures so far. Since most of the budget was
committed upfront for salaries, utilities, maintenance, development, etc., there was little discretion to be
exercised by the secretary. Nevertheless, it was possible for the secretary to make minor adjustments
here and there during a fiscal year to accommodate exigencies or political priorities. A slightly larger
opportunity to do so presented itself at the time of budget making when resource allocations for the next
fiscal year were being finalized.
progress on regular basis, but Arif wanted to focus on a few of these items, at least initially. However,
he was far from clear regarding which items these could be. He realized that selection must be done
carefully, as this would effectively set priorities for health professionals in the public sector. Once
they knew which indicators progress would be reviewed regularly, health practitioners would have
limited incentive to excel on other indicators.
Second, Arif often found himself and his team caught between the priority of the week as deter-
mined by the CM and regular work of the department. The CM frequently shifted from one priority to
another in response to media reports, judicial activism and his own caprice or political expediency.
It was, therefore, not uncommon for the department to pursue as the top priority, say, dengue control one
week and disposal of hospital waste in the next one. Arif had learnt during the previous year that he had
to be very up to date on the priority of the week if he wanted to keep the job. At the same time, he
also appreciated that his real challenge was to improve routine functioning of his department. It was
the everyday interface of the common citizen with an official in the EDOs office or in the Secretariat,
or with a health professional at a typical facility, that needed to be improved, simply because this consti-
tuted the point of interaction between the department and its clients. Balancing these competing informa-
tion needs was a challenge.
Third, there was some tension between the frontend and the backend work that the DOH performed.
The former comprised treating patients, conducting procedures, implementing development projects,
etc. These were more visible and usually concrete. The latter included mundane and relatively lacklustre
tasks, such as maintaining databases and keeping inventories. But could the former be accomplished
without attending to the latter? Arif tended to answer in the negative, which meant he had to devote some
space in the dashboard to measuring progress on the backend work.
Last, the dashboard had to satisfy information needs of a set of stakeholders. If it were just him
who was the primary user, the task would have been easy. He could select a few indicators of his choice
that directly contributed to achievement of priority policy objectives and monitor progress on these
indicators regularly. But he was only one user of the dashboard. Others included senior managers in
the department, such as the DGHS, programme director PHSRP, additional secretaries and EDOs, et al.
in the field. These users had different information needs. For example, while DGHS was interested in
child immunization and disease outbreaks, the additional secretary (establishment) was keen to know
how many posts were vacant and how many disciplinary proceedings were pending at various levels.
In order to be useful for a variety of users, the dashboard had to present information on several groups
of indicators and some opportunity for customization. Arif also realized that managers and field officials
needed different levels of detail. Therefore, the dashboard had to be capable of presenting summary
information for one group of users (viz., managers) and detailed information for another group of users
(viz., field officials).
Babar said that a useful starting point could be to identify the decisions that Arif made on daily, weekly
or monthly basis. The next step could be to identify the information that could inform these decisions.
From this list could be filtered the most vital statistics to include in the dashboard. Perhaps a month
would be a reasonable time span to start, said Babar.
As the chief manager, Arifs principal responsibility was to efficiently deploy the DOHs human, physi-
cal and fiscal resources in pursuance of Punjab Governments health policy objectives. Thus, he was
routinely making decisions in these two broad areas. As for the human resource (HR), his decisions related
to staff recruitment, promotion, transfer, seniority, leave, professional development, disciplinary proceed-
ings and complaints. Although most HR issues were dealt with by EDOs or DGHS, eventually it was
the secretarys responsibility to ensure that the DOH personnel were efficiently utilized and that rules
and regulations were followed. Furthermore, he was the supervising officer for a small number of senior
managers. Summary statistics on category-wise number of posts, vacancies, staff presence, qualification,
length of experience, place of posting, training, special skills, etc. were relevant for decision-making in
this area.
Similarly, Arif made decisions regarding allocation of financial resources to various facilities and
projects. This was mainly at the time of budget making, but progress had to be monitored on regular
basis. Additionally, there was some space for reallocation of budget from one project to another and
from one head to another. Often, service provision at a facility would suffer because it could not incur
an unanticipated expense on, say, the repair of a particular equipment or purchase of medicine. DHIS
provided detailed data on these aspects regularly, but Arif did not receive summary statistics on, for
example, the X-ray machines not working or the essential medicines being out of stock. Similarly, the
PC III reported monthly progress on each project, but Arif wanted a summary instead. The following
information was relevant: budget allocation and utilization in various heads and for various facilities,
physical and financial progress of development projects, equipment functionality and number of tests
carried out, medicine stock out, physical condition of the DOH buildings, etc.
Beena suggested expanding this list to include those items that Arif was required to do but was unable
to attend to for any reason whatsoever:
There are several items in the Rules of Business, which Arif does not seem to be very bothered about in his
routine functioning. Should we continue to ignore them? If we do not include them in the dashboard, they
are likely to remain ignored. If items such as juvenile smoking, nutrition, collection of data on the state of
health in the province, etc. are still important for the DOH, these should be put on the dashboard. Otherwise, they
should be removed from the Rules of Business.
She also asked Arif, Do you have a Terms of Reference (TOR) or a Charter of Duties that we can use to
specify what to put on the dashboard?
No, replied Arif, we dont have anything specific to my working in this department. A Secretarys
duties and functions are given in the Rules of Business (Exhibit VIII), but they are generic.
Anwar suggested, Perhaps we can have several sections and each section can focus on one aspect of
the working of the department as specified in the Rules of Business. This way, we will cover both what
we are currently doing and what we should be doing but are not.
Asim, who had previously worked with the school education department to develop a district report
card, shared a sample of the report card (Exhibit IX). The report card contained summary statistics on
staff strength and vacancies, teachers attendance, student enrolment and attendance, students exam
performance, missing facilities, budget allocation and utilization, and inspection visits by supervisory
authorities. He highlighted four aspects of the report card. First, it showed trends by comparing data
across several months. Second, it presented data for the entire district as well as for constituent tehsils.
Third, data was gender segregated for some indicators. Lastly, several data items were ratios, which
provided a relational analysis of variables. Can we develop something on these lines? he asked.
Zahid suggested that the dashboard should have layers. It should provide summary statistics for the
entire province, but should also support deeper exploration, if one so wished. For example, it should
be possible to view district (or tehsil or facility) level statistics for the current year as well as for previous
years. This would make it useful for several tiers of functionaries. The same dashboard could be used by
the minister and senior managers as well as by EDOs.
Arif liked the idea. We can give different access rights to various users. Can we make at least a
portion of the summary statistics available for viewing by the general public as well? he asked.
Beena wondered why that should be a problem. She was of the view that other than information on
individuals, the entire information on working of various facilities, resources at their disposal, budget
allocation and utilization and targets achieved should be available to the public at large. The dashboard
should have a permanent tab on the DOH website. This will help generate public pressure on DOH func-
tionaries to improve their performance, she opined.
Farasat shared a few dashboard samples. He highlighted that these sample dashboards presented data
on output and outcome level indicators. He said:
We receive a lot of data every month on dozens of indicators, but except for a handful, all of these indicators
either relate to inputs or to activities. So we end up measuring how many officials we have posted, how much
money we have allocated and which equipment or building we have provided. At best, we measure whether or
not the doctors et al. are present in the health facility. But we hardly make any effort to measure what we are
achieving. For this we shall have to focus on outputsor better stilloutcome level indicators.
He proposed that the dashboard should have data on the following indicators:
Arif wondered if it was possible to calculate cost per patient or per procedure. He asked, How will
you apportion cost of the District Offices and of the support provided by DGHS?
Farasat responded in these words: Yes, this is tricky. But there must be a way to do this. I am sure
there are accounting conventions to handle such apportionment. The private health facilities do it on
regular basis. Let us ask them to help us with this.
Farasat further emphasized that by focusing on outputs and outcome level indicators, the dashboard
could become an important instrument of assigning targets to health facilities and districts. It would
also enable a performance comparison across districts. Beena proposed that at least some indicators
on preventive health care should also be included in the dashboard. She complained that the DOH was
very indifferent to this aspect of health care, and consequently ended up spending substantial sums on
ailments that could have been prevented at a fraction of the cost.
Arif asked how health awareness would be measured. After all, raising awareness was also an impor-
tant function of the department and a key component of preventive health care. Farasat said that surveys
could be designed to measure awareness among the public at large.
Asim cautioned against putting too much into one dashboard:
In your efforts to make it comprehensive, you will make the dashboard unwieldy and complex. Then you wouldnt
look at it. The whole point is to present key information in a visually-friendly manner. Also, this dashboard will
not be the only thing the secretary will look at. It is just one of the several data sources at his disposal.
Arif then asked if having a layered structure would solve this problem. He stated, We do not have
to display everything on one screen. Instead, we can have links to lead the interested viewer to more
detailed information.
Thats true, replied Farasat, but we have to be careful in choosing what goes in the first screen
thats the one most looked at.
Arif asked if anyone had an idea of how much resource it would take to build a dashboard. Asim
suggested that it should not cost much, given that most of the data was already being collected by
DGHS and/or PHSRP. He said, It is just a question of deciding which data you want and in what form.
Once that has been done, your IT people should be able to do it for you.
How frequently will we need to update it? asked Arif. Farasat replied that it would vary for different
indicators:
Some values change daily e.g. outpatient or medicine stock out. But some values change over a long time
e.g. population per hospital bed or proportion of children immunised. Then there are variables whose values
change over a very long time e.g. infant mortality rate. We collect data on these indicators accordingly. We will
update the dashboard as and when new data is available for an indicator. So some indicators will be updated
daily, while others will be updated monthly and some annually.
Beena enquired about those indicators for which data was not readily available, such as juvenile smoking
or nutrition. Farasat replied that PHSRP could conduct surveys and update the dashboard as soon as
results from a survey were available.
Moving Forward
The meeting was inconclusive. Even after two hours of discussion, Arif and his team did not agree
regarding the choice of indicators. Farasat proposed the following eight sections for the dashboard:
1. HR management
2. Physical assets and financial resources
3. Key activities, outputs and outcomes
4. Development schemes
5. Medical education
6. Implementation of laws
7. Findings from various periodic surveys
8. Emergent issues.
He wanted the aggregate statistics in these eight areas to fit on one screen, which Arif could quickly
glance through and see how things were. He proposed that most of these statistics should be shown in a
graphic, rather than a tabular form. He also wanted each section to lead to detailed district-wise or
facility-wise information. Such detailed information would be available on a click to different users, who
would have variable access levels.
Arif spent the next few days pondering about the appropriateness of the above eight sections. He did
not want to put too much on the dashboard, but neither did he want to miss important indicators that
ought to be included. He was also thinking about how he could use the dashboard to improve his man-
agement in particular and the effectiveness and efficiency of health care provision in general.
Exhibit I. Health Facilities and Practitioners in PunjabNumber of Public Sector Health Facilities
in Various Categories
Exhibit II. Functions of the Health Department, 2nd SchedulePunjab Government Rules of
Business, 2011, Health Department
iv. Punjab Juvenile Smoking Ordinance, 1959 (W.P. Ordinance XII of 1959)
v. Punjab Prohibition of Smoking in Cinema Houses Ordinance, 1960
(W.P. Ordinance IV of 1960)
vi. Eye Surgery (Restriction) Ordinance, 1960
vii. Punjab Pure Food Ordinance, 1960 (W.P. Ordinance VII of 1960)
viii. Allopathic System (prevention of misuse) Ordinance, 1962
ix. Pakistan Medical and Dental Council Ordinance, 1962
x. Unani, Ayurvedic and Homoeopathic Practitioners Act, 1965
xi. Pharmacy Act, 1967
xii. Drugs Act, 1976
xiii. Medical & Dental Degree Ordinance, 1982
xiv. Punjab Health Foundation Act, 1992
xv. Punjab Transfusion of Safe Blood Ordinance, 1999
xvi. Mental Health Ordinance for Pakistan, 2001
xvii. Protection of Breast Feeding and Child Nutrition Ordinance, 2002
xviii. Prohibition of Smoking and Protection of Non-smokers Health
Ordinance, 2002
xix. Punjab Medical and Health Institutions Act, 2003
xx. Injured Persons (Medical Aid Act), 2004
xxi. King Edward Medical University, Lahore Act, 2005
xxii. Human Organ Transplant Ordinance, 2007
xxiii. Pakistan College of Physicians and Surgeons Ordinance, 1962
xxiv. The University of Health Sciences Lahore Ordinance, 2002
15. Matters incidental and ancillary to the above subjects.
Source: Punjab Government Rules of Business, 2011.
Exhibit IV. DHIS Indicators for PHC and SHC Facilities (p1 of 3)
(Exhibit IV continued)
Sr. No. Indicator PHC SHC
Community Meetings
52 Number of community meetings : :
53 Number of participants : :
Diagnostic Services
54 Total lab investigations : :
55 Total X-rays : :
56 Total ultra sonographics : :
57 Total ECGs
Stock out : :
58 Stock out of drugs/vaccines : :
Indoor Services
59 Allocated beds : :
60 Admissions : :
61 Discharged not on same day of admission : :
62 Discharged on same day of admission : :
63 LAMA : :
64 Referred : :
65 Deaths : :
66 Total of daily patient count : :
67 Bed occupancy : :
68 Average length of stay : :
Procedures
69 Operations under general anaesthesia : :
70 Operations under spinal anaesthesia : :
71 Operations under local anaesthesia : :
72 Operations under other type of anaesthesia : :
Human Resource Data
73 Sanctioned : :
74 Vacant : :
75 Contract : :
76 On general duty in facility : :
77 On general duty out of facility : :
Revenue Generated and Financial Report
78 Total receipts : :
79 Deposits : :
80 Total allocation for the fiscal year : :
81 Total budget released to-date : :
82 Total expenditure to-date : :
83 Balance to-date : :
Weightage
Indicator BHU RHC
Cleanliness/general outlook 7 8
Displays 5 8
Availability of utilities 8 4
Functioning of equipment 6 10
Availability of medicines 21 21
Public opinion 10 10
Doctors presence 23 21
Preventive staff presence 5 5
Administrative staff presence 5 5
Paramedics presence 10 8
Total 100 100
Source: PHSRP Records, 2014.
Exhibit VIII. Duties and Functions of Secretary, Punjab Rules of Business, 2011
References
Government of Punjab (2008). Stocktake of private practitioners in Punjab. Punjab: Government of Punjab.
. (2011). Punjab Government Rules of Business. Punjab: Government of Punjab.
Department of Health. (2011). DHIS quarterly report 4th Quarter 2011. Lahore: Directorate General of Health
Services. Punjab: Government of Punjab.
. (2014). Lahore: Directorate of General Health Services (DGHS). Punjab: Government of Punjab.
Ministry of Finance (2011). Pakistan Economic Survey 201112. Islamabad: Ministry of Finance, Government of
Pakistan.
Pakistan Bureau of Statistics (2011). Pakistan Labour Force Survey 201011. Islamabad: Pakistan Bureau of
Statistics, Government of Pakistan.
. (2012). Punjab development statistics. Lahore: Punjab Bureau of Statistics, Government of Punjab.
Punjab Health Sector Reforms Programmes (2014). Lahore: Punjab Health Sector Reforms Programmes. Punjab:
Government of Punjab.
Punjab School Education Department (2014). Lahore: Punjab School Education Department. Punjab: Government
of Punjab.
World Bank (2013). Punjab performance expenditure review. Islamabad: World Bank.