The International Conference on Information and Digital Technologies 2017

Evaluation of Healthcare Enterprise Information

Systems: A Structural Equation Model

Ahed Abugabah
College of Technological Innovation,
Zayed University
Dubai, United Arab Emirates
ahed.abugabah@zu.ac.ae

Abstract—The aim of this study is to evaluate Healthcare
Enterprise Information Systems from both system and user
perspectives. The study uses a structural equation modelling to
analyse the impacts of healthcare information systems on
employees performance and health care service provided by
hospitals. This was measured using several dimensional factors
including, system quality, service quality and information quality.
The study used a questionnaire survey to gather data from
healthcare organizations and their staff in different departments
and at different levels. The study findings indicated that Hospital
Enterprise Information Systems (HEIS) helped improve service
quality by providing complete and accurate information related to
patients and patient service processes
Keywords— Healthcare Enterprise Information Systems,
Information Systems Models, Healthcare service
I. INTRODUCTION
Information systems have significant potential to improve
patient service, health care efficiency, patient satisfaction and
health care process. For example, physician systems can help
reduce the serious medication error. Over the past years,
substantial initiatives were proposed in health care industry to
prompt the adoption and implementation of healthcare
information systems and their applications [1]. As a
consequence, health care organizations made huge investments
and spent enormous amounts of money and resources in shifting
to or adopting Health care Enterprise Information Systems with
the hope of improving organizational performance and patient
service, and operational efficient [2].
General speaking, it is believed that the use of health care
Information Systems provides great opportunities for improving
the healthcare process, service quality and the efficiency health
care operations. Recent research in health care industry reported
that if these organizations do not adopt relevant and advanced
applications of healthcare information systems, they will
become ineffective and face many operational and functional
deficiencies [1,2]. From practical perspective, health care
practitioners and managers continue their quest for more
efficient techniques and methods of gathering patient
information to meeting patient requirements and work needs.
These requirements place increased demands on healthcare
information systems and their applications [4, 5, 6]. This also
makes it interesting to evaluate the actual benefits of the Health
care Enterprise Information Systems (HEIS) and their role in
health care organizations [3].
Consequently, researchers as well as stakeholders of health
care organizations seem to be interested in evaluating the
outcomes of HEIS projects implementations and analysing the
level to which HEIS applications are able to meet the varied
needs of groups and individual in their organizations [3].
Fortunately, there has been a number of Information Systems
(IS) models and frameworks available for researchers in this
endeavour [2, 6].
This study therefore attempts to evaluate the (HEIS) and
assess their impacts on health care service quality and staff
performance from both system and user perspectives. The study
uses a structural equation modelling to analyse the impacts of
healthcare information systems on employees and
organizational performance and healthcare services provided by
hospitals. The study is a part of a large research project
investigating information systems in health care organizations at
different levels and from different user perspectives.
II. LITERATURE REVIEW
As an attempt to build a rigor foundation for this research,
prior IS and heath care informatics research were reviewed with
a focus on IS models and IS evaluation frameworks research in
health care context. This provided a theoretical basis for the
main factors investigated in this study and their dimensions and
measurements. This also helped avoid the limitations of
previous evaluation frameworks and make improvements to the
current study framework [2, 5].
Most existing evaluations of HEIS focused on either technical
issues or behaviour issue leading to improper evaluation and or
poor evaluation results. Ammenwerth et al. [3] stated that,
despite the growing body of research literature on healthcare
information systems evaluation, there have been many
concerned issues in conducting a system evaluation. These
issues include the complexity of the systems themselves, the
evaluation goals and the lack of clarity [12, 13, 14]. Whilst many
frameworks proposed for HIS may have been complementary to
one another because they focus on different aspects of
evaluation, however none of them provided explicit evaluation
categories for the evaluators, including technical, behavioral and
systematic categories [11]. In addition, one of the major

978-1-5090-5689-7/17/$31.00 ©2017 IEEE 1

Evaluation of Healthcare Enterprise Information Systems: A Structural Equation Model
shortcomings in previous HEIS frameworks used in previous
studies is the lack of a proper reporting format for HEIS
evaluation. The lack of consideration in addressing the what,
why, who, how and when of the HEIS evaluation is evident
[5,6,7, 19]. Whilst previous frameworks complement each other,
it appears that caution should be exercised in applying the
frameworks based on their shortcomings [10].
A. Research Model
In order to build a rigor research framework, the research
model was built based on previous IS models. Namely, the IS
success model, the Task-Technology Fit (TTF) model and
Technology Acceptance Model (TAM) [8,9,10]. These models
have been used widely in information technology/systems
fields to evaluate numerous types of IS applications in various
business environments, and have been proven to provide valid
findings [13, 14, 15, 18].
B.
C. Hypotheses
This research examined the following hypotheses, which are
derived from the integrated models and based on an extensive
review of the IS in health care
 H1a: TTF is associated with user performance and
health care service quality in health care organizations.
 H1b: TTF is associated with health care service in health
care organizations.
 H2a: system quality of HEIS is associated with user
performance in health care organizations
 H2b: system quality of HEIS is associated with health
care service in health care organizations.
 H3a: Information quality of HEIS is associated with
user performance and health care service quality in
health care organizations.
Figure 1. The Study Model
2 978-1-5090-5689-7/17/$31.00 ©2017 IEEE
 H3b: Information quality of HEIS is associated with
health care service in health care organizations.
Notwithstanding the expansion of frequent frameworks for
HEIS evaluation, there appears to be a lack of consensus about
what aspects to evaluate, when to conduct the evaluation, and
how to carry out such evaluation [7, 8, 9, 16]. In addition to who
are the main stakeholders of the evaluation as well as why
carrying that evaluation, meaning the main the purpose of the
evaluation must be clearly identified.
III. RESEARCH MODEL AND HYPOTHESES
That is, jointing these models together will provide a more
comprehensive and solid research framework that can examine
many factors including the HEIS and user related factors, as
illustrated in Figure 1. Besides, the integrated model will help
overcome the limitations of each model standing alone [6,7,8].
The study model along with the hypotheses examined in this
study are illustrated in figure 1.
IV. METHODOLOGY
The results reported in this study represent a part of larger
research project investigating HEIS. The study utilized a survey
questionnaire to collect data from different types of users
including decision makers and IT managers in healthcare
organizations. The questionnaire was constructed based on an
extensive review of the literature in the areas of information
technology and health care information systems [3,7,8,9, 17].
The questionnaire first was pre-tested with 5 users and
academicians in order to improve the face validity. The study
was conducted in five hospitals that had implemented the HEIS
more than two years ago in UAE. A total of 197 questionnaires
were analysed after data filtration and cleaning as summarized
in Table I.
 The International Conference on Information and Digital Technologies 2017

TABLE I. SAMPLE CHARACTERISTICS (N=197). than the correlation coefficient of all constructs demonstrating
Factor Classification No % thus a high level of discriminant validity.
Male 104 52.8% V. ANALYSIS AND FINDINGS
Gender
Female 93 47.2%
A Structural Equation Modelling (SEM) was used to test the
Vocational 11 5.6%
research structural model. The structural model was examined
Qualification Bachelor 166 84.3% by using the goodness-of-fit (GFI) of the hypothesized research
Postgraduate 20 10.1% model [4], the 2/df, Confirmatory Factor Analysis (CFI), and
Physician 97 49.2% the Root Mean Square Error of Approximation (RMSEA) were
Position Nursing 52 26.4% used to test the goodness-of-fit of the structural model. , the
Administrative 21 10.7% goodness-of-fit indices for the structural model were
IT staff 27 13.7% x2/df=1.54, GFI=0.93, CFI=0.93, Normed Fit Index (NFI)
=0.92, and RMSEA=0.81, indicating that the model provides a
good fit with the data.
A. Reliability and Validity
The validity of the study model was examined using item TABLE IV. RESULTS OF HYPOTHESES TESTS
reliability, discriminant validity and convergent validity. The
reliability of the measurement items was confirmed by Hypothesis Path t-
Support
calculating factor loadings. A factor loading greater than 0.5 is Coefficient  Value
considered as an acceptable threshold for item reliability [11,4]. H1aTTF performance 0.39 6.8 Yes
As presented in Table II , the factor loadings for all items H1b TTF Health care
exceeded 0.5, indicating acceptable level of reliability. In 0.27 3.5 Yes
service
relation to construct validity, the Average Variance Extracted H2a System Quality 
(AVE) was used to measure construct validity [4]. An AVE must 0.49 6.3 Yes
performance
be higher than 0.5 to ensure that variance captured by the H2b System Quality 
construct is greater than the measurement error. As shown in 0.41 3.0 Yes
Health care service
Table II all values of the AVEs were above 0.60, indicating that
H3a Information Quality
construct validity was satisfied. 0.25 4.5 Yes
 performance
TABLE II. CORRELATIONS AND AVE VALUES. H3b Information Quality
0.23 3.4 Yes
 Health care service
Factors AVE TTF SQ IQ UP HS Alphaα
TTF 0.89 (0.94) 0.84
A Structural Equation Modelling (SEM) was used to test the
IQ 0.83 0.78 (0.91) 0.87 research structural model. The path coefficients and t-test values
SQ 0.87 0.63 0.59 (0.93) 0.81 were used to examine the relationships between all study
UP 0.89 0.52 0.39 0.47 (0.94) 0.85 variables.
HSQ 0.72 0.48 0.29 0.59 0.41 (0.85) 0.82 The findings indicated a significant support for all
hypotheses. The path coefficient between all study factors
TABLE III. GOODNESS-OF-FIT indicated positive relationships significantly. As shown in
Table IV P<0.05. Task technology fit significantly affects both
Recommended Measurement Structural
Criteria/Indices health care service and user performance, where (=0.27, t=3.5,
Value Model Model and (=0.39, t=6.85) respectively. This indicates that whenever
Chi-square (2) --- 307 301 HEIS provides the appropriate level of fitness with staff needs
Degree of and work requirements, health care staff tend to realize more
--- 192 195 system benefits and significant impacts on both performance and
Freedom
healthcare service quality. That is, having a HEIS with the right
2/df >2 1.60 1.54 outputs on time will lead to enhanced patient’s services as it will
GFI >0.90 0.94 0.93 save staff time and expedite service process with more accuracy
NFI >0.90 0.91 0.92 and less errors. Thus, both H1a and H1b are supported as listed
NNFI >0.90 0.93 0.93 in Table IV.
CFI >0.90 0.92 0.92 The findings also indicated that the SQ of HEIS was the most
RMSEA >0.08 0.81 0.81 powerful variable that affects performance and service quality
(=0.41, t=3.0, and =0.49, t=6.39, p<0.05) respectively.
Hence, both H2a and H2b were significantly supported.
Furthermore, discriminant validity was tested for each Similarly, the findings indicated a positive effect of IQ of HEIS
construct using the square root of its AVE, which must be on both service quality and User Performance (UP) =0.23,
greater than the correlation coefficient with other constructs in t=3.4 and, (=0.25, t=4.5 p<0.05) respectively and therefore,
the model. As shown in Table II all square root values are higher H3a and H3b are supported. Unexpectedly, the study revealed
no significant impact of IQ on Perceived Ease of Use (PEOU).

978-1-5090-5689-7/17/$31.00 ©2017 IEEE 3

Evaluation of Healthcare Enterprise Information Systems: A Structural Equation Model
However, other factors such as system quality showed relatively
significant but small impact on both PU and PEOU, as shown in
figure 1. To sum up, the findings indicated that HEIS plays a
critical role in helping health care staff performing their job and
tasks more effectively, leading to improved health care services.
VI. CONCLUSIONS AND IMPLICATIONS
This study aimed to evaluate HEIS from user perspective in
order to identify main benefits and impacts of the HEIS on users
and service quality. The main conclusions of the study bring into
our attention some important highlights on critical factors
relevant to the evaluation of HEIS at both system and user levels.
Evaluation of HEIS is in itself a complex and
multidimensional task. A proper framework then must address
the what, why, who, how and when of the HEIS evaluation is
evident and provide appropriate solutions from health care
context. Some observations have been clearly made in this
study. These include identifying a need for an evaluation
framework that clearly answers the aforementioned questions
raised by many researchers. An evaluation framework must
consider also contextual factors when evaluating health care
information systems. In addition to the need for a better
approach for identifying the right system stakeholders, instead
of using simple classifications.
The finding of the study indicated that system
incompatibility leads probably to disrupted system interaction to
system users including medical staff and administrative staff.
Consequently, information and data needed in some certain
circumstances are inaccessible and/or unreadable, causing
serious difficulty for doctors to and managers to make decision
related to patient diagnosis or treatment. Poor system
functionality is also attributed to different factors, such as
administration disputes and missing user and task requirements.
Therefore, technology fit between the system functionality and
user needs and task requirements is a priority that should be
considered by both system designers and health care
management.
Furthermore, system quality of HEIS is associated with ease
of use and staff performance leading to better services quality.
The study revealed that when the HEIS is designed to be easy to
use and user friendly, better system usage will occur leading to
more positive impacts on both user tasks and service quality
provided to patients. This can happen because of time saving in
answering questions and obtaining relevant and complete
information when needed. Relevant training on the system itself
also helps users to become competent and utilize the system
functionality more effectively.
Information quality is measured from systems outputs and
user interface displays in various forms such as patient report,
record, and prescription. Information quality in many cases is
subjected to user perspective on information characteristics such
as accuracy, consistency and completeness. Poor quality of
information retrieved from the HEIS will lead to medical errors
and task mistakes, leading consequently to poor performance
and poor services quality. Overall, the potential of HEIS system
impact is influenced by system ease to use and system quality,
4 978-1-5090-5689-7/17/$31.00 ©2017 IEEE
which in return affect the level of system use, leading to major
effects on patient services and user performance.
REFERENCES
[1] A. Andargoli, , D. Rajendran and A Sohal, Health information systems
evaluation frameworks: A systematic review. International Journal of
Medical Informatics, (97), January 2017; 195-209
[2] A.Azizi, R. Abdolkhani, The case study of effect of hospital information
system in improvement of Razi hospital performance, J. Jundishapour 2
(2011) 185–190.
[3] Abugabah, A and Alfarraj, O, 2015. Issues to Consider in Designing
Health Care Information Systems: A User-centred Design Approach,
electronic Journal of Health Informatics, 9(1), pp1-15.
[4] Abugabah, A Sansogni, L and Alfarraj, O., 2015. Evaluating the impact
of ERP systems, International Journal of Information and Learning
Technology, 32 (1), pp. 45-64
[5] Ahed Abugabah, Louis Sanzogni (2014). Exploring factors affecting end-
user performance of information systems: the International Journal for
Infonomics, 7 (3/4), 956-973.
[6] Alfarraj, O and Abugabah, A (2017). Extending Information System
Models to the Health Care Context: An Empirical Study and Experience
from Developing Countries, The International Arab Journal of
Information Technology. 14 (2), p159-167.
[7] E. Ammenwerth, N. de Keizer, An inventory of evaluation studies of
information technology in health care: trends in evaluation research 1982–
2002, Methods Inf. Med. 44 (2005) 44–56.
[8] J. Hair, B. Babin, A. Money and P. Samouel, P, Essential of business
research methods. John Wiley & Sons: United States of America, 2003.
[9] J. Sligo, R. Gauld, V. Roberts and L. Villa, International Journal of
Medical Informatics, (97), January 2017; 86-97. A literature review for
large-scale health information system project planning, implementation
and evaluation
[10] L. Ahmadian, S. Nejad and R. Khajouei, Evaluation methods used on
health information systems (HISs) in Iran and the effects of HISs on
Iranian healthcare: A systematic review, International Journal of Medical
Informatics, 84, (6), 2015; 444-453
[11] L. Lapointe, M. Mignerat and I. Vedel , The IT productivity paradox in
health: A stakeholder’s perspective. International Journal of Medical
Informatics. 2011. 80 (2):102-115.
[12] L. Politi, S. Codish, I. Sagy and L. Fink, Use patterns of health
information exchange systems and admission decisions: Reductionistic
and configurational approaches, International Journal of Medical
Informatics, 84, (12), 2015; 1029-1038
[13] M. Beuscart-Zéphir, S. Pelayoa and S. Bernonvillea, Example of a human
factors engineering approach to a medication administration work system:
potential impact on patient safety. International Journal of Medical
Informatics. 2010. 79(4): E43-E57.
[14] M. Lluch, Healthcare professionals’ organizational barriers to health
information technologies: A literature review. International Journal of
Medical Informatics. 2011. 80(12): 849–862
[15] M.M. Yusof, J. Kuljis, A. Papazafeiropoulou, L.K. Stergioulas, An
evaluation framework for health information systems: human,
organization and technology-fit factors (HOT-fit), Int. J. Med. Inf. 77
(2008) 386–398.
[16] P. Sockolow, P. Crawford, H. Lehmann, Health services research
evaluation principles, Methods Inf. Med. 51 (2012) 122–130.
[17] R. Khajouei, A.A. Azizi, A. Atashi, Usability evaluation of an emergency
information system: a heuristic evaluation, J. Health Admin. 16 (2013)
61–72.
[18] S. Martikainen, M. Korpela and T. Tiihonen. User participation in
healthcare IT development: A developers’ viewpoint in Finland:
International Journal of Medical Informatics, 2014. 83 (3); p 189–200
[19] T. Teixeira, C. Ferreira, B. Santos, User-centered requirements
engineering in health information systems: Computer Methods and
Programs in Biomedicine. 2012. 106 (3):60