http://informahealthcare.

com/jic
ISSN: 1356-1820 (print), 1469-9567 (electronic)

J Interprof Care, 2015; 29(6): 579–586

! 2015 Taylor & Francis Group, LLC. DOI: 10.3109/13561820.2015.1049340

THEMED ARTICLE

Refining a self-assessment of informatics competency scale using

Mokken scaling analysis
Sunmoo Yoon1, Jonathan A. Shaffer2, and Suzanne Bakken1
1
School of Nursing and 2Center for Behavior Cardiovascular Health, Columbia University, New York, NY, USA

Abstract Keywords
Healthcare environments are increasingly implementing health information technology (HIT) Interprofessional care, interprofessional
and those from various professions must be competent to use HIT in meaningful ways. In collaboration, interprofessional education,
addition, HIT has been shown to enable interprofessional approaches to health care. The interprofessional evaluation, methods,
purpose of this article is to describe the refinement of the Self-Assessment of Nursing quantitative method, statistics
Informatics Competencies Scale (SANICS) using analytic techniques based upon item response
theory (IRT) and discuss its relevance to interprofessional education and practice. In a sample of History
604 nursing students, the 93-item version of SANICS was examined using non-parametric IRT.
The iterative modeling procedure included 31 steps comprising: (1) assessing scalability, (2) Received 20 September 2014
assessing monotonicity, (3) assessing invariant item ordering, and (4) expert input. SANICS was Revised 30 March 2015
reduced to an 18-item hierarchical scale with excellent reliability. Fundamental skills for team Accepted 5 May 2015
functioning and shared decision making among team members (e.g. ‘‘using monitoring Published online 3 December 2015
systems appropriately,’’ ‘‘describing general systems to support clinical care’’) had the highest
level of difficulty, and ‘‘demonstrating basic technology skills’’ had the lowest difficulty level.
Most items reflect informatics competencies relevant to all health professionals. Further, the
approaches can be applied to construct a new hierarchical scale or refine an existing scale
related to informatics attitudes or competencies for various health professions.

Introduction team possess informatics competencies (Laramee, Bosek, Shaner-

McRae, & Powers-Phaneuf, 2012). A number of efforts have
Patient-centered care is supported by interprofessional collabor-
focused on defining informatics competencies (American Nurses’
ation (Lindberg & Vingard, 2012; Manojlovich & DeCicco,
Association, 2008; Centers for Disease Control and Prevention,
2007). Healthcare environments are increasingly implementing
2009; Curran, 2003; Kulikowski et al., 2012; QSEN Institute,
health information technology (HIT) and those from various
2012) and designing educational programs focused on achieve-
professions must be competent to use HIT in meaningful ways. In
ment of the competencies (Desjardins, Cook, Jenkins, & Bakken,
addition, HIT has been shown to enable interprofessional
2005; Feldman & Hersh, 2007). An important component of
approaches to healthcare (Kubias, 2011). For example, in primary
strategies to ensure such competencies is having reliable and valid
and acute care settings, care notes entered into an electronic
competency measurements. In addition to efforts such as specialty
health record (EHR) by one profession are subsequently available
certification examinations for informatics (Lehmann, Shorte, &
to other members of the patient care team and data from infusion
Gundlapalli, 2013), a number of instruments have been developed
pumps and medication barcode scanners are shared and monitored
for self-reported competencies during the last several decades
by the interprofessional care team.
(Hwang & Park, 2011; McNeil et al., 2005; Ozbolt & Graves,
In other instances, applications have been explicitly designed
1993; Staggers, Gassert, & Curran, 2002; Westra & Delaney,
to facilitate collaboration among care team members from
2008; Yoon, Yen, & Bakken, 2009). The purpose of this article is
different professions. For example, Vawdrey, Stein, Fred,
to describe the refinement of the Self-Assessment of Nursing
Bostwick, and Stetson (2013) designed a computer-based handoff
Informatics Competencies Scale (SANICS) and discuss its
tool integrated with an EHR that is used by different professions
relevance to interprofessional education (IPE) and IPP.
including physicians, nurses, social workers and pharmacists for
communication and coordination of care. These technologies can
Background
be meaningfully used for reducing medical errors and enhancing
team collaboration and accurate communication among team Self-assessment of nursing informatics competencies
members (Collins et al., 2012). scale
Given the importance of HIT in interprofessional practice
The SANICS is a 93-item self-report instrument for measuring
(IPP), it is essential that members of the interprofessional care
informatics competency among nurses at different stages of their
educations and careers (Yoon et al., 2009) and has been used for
Correspondence: Dr. Sunmoo Yoon, PhD, School of Nursing, Columbia evaluation of pre-licensure students and graduate students
University, 617 W. 168th Street, New York, NY 10032, USA. E-mail: (Desjardins et al., 2005). SANICS incorporates items based
sy2102@columbia.edu upon published informatics competencies (Staggers & Thompson,
580 S. Yoon et al.
2002) as well as interprofessional informatics competencies
such as using standardized terminologies (e.g. International
Classification of Disease), evidence-based practice and wireless
communication (Desjardins et al., 2005).
As part of a continuous effort to improve SANICS, we applied
exploratory principal component analysis (PCA) with oblique
promax rotation to the 93-item SANICS that resulted in a five-
factor, 30-item version (Yoon et al., 2009). The 30-item SANICS
has been used for measuring nursing informatics competency in
national (Choi & De Martinis, 2013) and international academic
and non-academic settings over the past 5 years. Given the
increased demand for SANICS and our interest in more
sophisticated analytic techniques (Carroll, 1945; Choi & De
Martinis, 2013) to refine our existing scale, we decided to apply
approaches based upon item response theory (IRT).
Item response theory
Item response theory (IRT) relates response on a test item to an
underlying ability or trait. Unlike scale development using factor
analytic techniques, which are informed by classical test theory,
scales constructed using IRT have higher reliability and homo-
geneity (van Schuur, 2003). Originally used in education, IRT has
increased in popularity given its more recent use in the Patient
Reported Outcomes Measurement Information System
(PROMIS), a large National Institutes of Health initiative aimed
at improving the reliability and validity of patient-reported
outcomes for use in research and clinical practice (Fries, Bruce,
& Cella, 2005; Fries, Krishnan, Rose, Lingala, & Bruce, 2011).
There exist both parametric and non-parametric approaches
to IRT.
Parametric IRT has the following theoretical assumptions: a
unidimensional trait (y, theta), local independence of items and
latent monotonicity. Unidimensionality means the scale measures
only one ability. Local independence indicates that the response is
only influenced by the abilities not by other factors such as help
from others, and this assumption is established once unidimen-
sionality is true (Lord, 1980). Latent monotonicity means that the
item step-response functions are non-decreasing functions of the
trait being measured.
Although the assumptions of unidimensionality, local inde-
pendence and latent monotonicity are common across parametric
and non-parametric approaches, there are some differences
between the two. Parametric IRT is based upon stronger statistical
assumptions about the shape of the item response curves between
the score on a latent trait and the probability of obtaining that
score (Meijer, Sijtsma, & Smid, 1990). Whereas parametric IRT
assumes strictly perfect item characteristic curves, non-parametric
IRT approaches do not. Rather non-parametric IRT requires that
the item response curves be monotonically increasing and non-
intersecting. This has been recognized by several psychometri-
cians since the 1940s (Mokken, 1971). Although there are a few
research studies reporting similar results using parametric and
non-parametric approaches (Lei, Dunbar, & Kolen, 2004), the
consistency of results from the two methods depends on the
datasets under certain conditions. Thus, researchers need to be
cautious about defaulting to parametric IRT without checking that
assumptions are met so that the results obtained are valid (Sass &
Schmitt, 2011).
Mokken scaling is a non-parametric IRT measurement
modeling procedure used to assess the hierarchies of items
measuring individual abilities or attitudes from the perspective of
item difficulty and has been used more than 30 years across
different professional fields (Mokken, 1971). The goal of Mokken
scaling is to establish a hierarchical scale among a set of items
based on item difficulty. To do this, Mokken scaling includes
J Interprof Care, 2015; 29(6): 579–586
iterative assessment of scalability, monotonicity and invariant
item ordering (IIO) in conjunction with expert review as items are
removed from the scale. The specific methods associated with
these techniques are described in data analysis.
Despite the acknowledged benefits of IRT in general (Ligtvoet,
van der Ark, te Marvelde, & Sijtsma, 2010) and Mokken scaling
in particular, these approaches have not typically been applied to
scales that measure nursing informatics competencies. Therefore,
the aim of the current study was to use Mokken scaling to develop
a hierarchical scale derived from the 93-item version of SANICS.
Methods
Dataset
The 93-item SANICS was administered to pre-licensure students
from 2006 to 2008 (n ¼ 604) at Columbia University School of
Nursing. Respondents rated their competencies on a 5-point
Likert scale (1 ¼ not competent, 2 ¼ somewhat competent,
3 ¼ competent, 4 ¼ proficient and 5 ¼ expert). Higher scores on
the SANICS are indicative of higher perceived competence. After
checking for low missing rates (0.53%) without patterns, an
individual with missing data was excluded from the analysis
(Schafer, 1999) resulting in a sample size of 603.
Mokken scaling procedures
Mokken scaling was performed using R package Mokken version
2.7.5 (Sijtsma, Meijer, & van der Ark, 2011) applied in our
analysis along with the resulting decrease in number of scale
items: (1) scalability assessment; (2) monotonicity assessment; (3)
IIO assessment; and (4) expert review.
Scalability assessment
This assessment includes homogeneity (Loevinger, 1948) in
assessment of unidimensionality assumption. Homogeneity is
measured through Loevinger’s H statistic. It can be calculated for
an item (Hi), each pair of items i and j (Hij), and averaged across
all items for the scale (H). Hij is an item scalability coefficient to
indicate the degree of discrimination. High Hij value indicates
high discriminating item response function (item step response
function for a polytomous scale). Hi is an item-level coefficient to
evaluate the fit of item i with respect to the other items (n  1) in
a set of item I (Molenaar, 1997). H is a scale-level coefficient; the
higher H, the smaller error in the rank ordering of items. A zero
value of H means the discrimination between subjects is only
based on random error.
Loevinger’s scalability coefficients Hi and H (and also Hij) of
our sample were calculated using coefH of the Mokken module in
R. The item and scale level scalability coefficients (Hi, H) were
evaluated according to the established criteria: 0.3  H50.4
(weak scale), 0.40  H50.50 (moderate scale); H  0.5 (strong
scale) (Ligtvoet et al., 2010).
Unidimensionality (homogeneity) means that the scale meas-
ures only one latent ability, and local independence is used to
determine if the response of the scales depends on a person’s
latent ability, rather than other factors such as learning behavior
from the other items during the test. When unidimensionality is
true, local independence is satisfied (Lord, 1980; Lord & Novick,
1968).
Unidimensionality and local independence assumptions were
assessed using automated item selection procedure (AISP), which
returns a vector with as many elements as there are items and
indicates to which scale an item belongs (Molenaar & Sijtsma,
2000). This algorithm was run repeatedly to identify unidimen-
sional clusters of items with different customizable parameter c
values; the higher c, the more confidence in the ordering of
DOI: 10.3109/13561820.2015.1049340
persons with their total scale score (Mokken, 1971). The
customizable lower bound c parameters were incrementally
specified from 0 to c ¼ 5.0 (default 0.3) in order to reveal the
patterns (Molenaar & Sijtsma, 2000). The patterns of the AISP
results were examined to determine if the patterns followed the
typical sequence: (1) most items belong to one scale, (2) one
smaller scale found, (3) a few small scales found and (4) several
items excluded.
Monotonicity
This allows for the alignment of respondents on a latent
continuum for that item. In this step, we assessed if the item
response step functions for the 93 SANICS items monotonically
increased as latent trait levels increased. During the visual check
of the graphical displays for each iteration, the plots of patterns of
scores for items were checked for strictly increasing monotonicity.
For each violation, a significance test at level a ¼ 0.05 without
Bonferroni correction was computed (Molenaar & Sijtsma, 2000).
Invariant item ordering
Assessing if a set of items holds invariant item ordering (IIO),
which means that items are in the same order according to
difficulty across respondents, is essential in the construction of a
hierarchical scale (Sijtsma et al., 2011). For example, if someone
can calculate ‘‘374 7 173’’ (i.e. the most difficult item) at the top
of a hierarchical ordering, the assumption is that the person can
calculate ‘‘3 7 3’’ (i.e. the least difficult item) at the bottom of
the hierarchical order. Technically, this means that item response
step functions do not intersect (Ligtvoet et al., 2010) which is a
requirement of Mokken scaling. Despite its importance in
measurement practice, assessment of IIO assumptions has been
often ignored for construction and evaluation of scales (Ligtvoet,
van der Ark, Bergsma, & Sijtsma, 2011).
Command check.iio was used to apply method MIIO (manifest
invariant item ordering), which uses a t-test to assess for the
presence of violations, i.e. intersecting item response step
functions. The process was repeated 26 times until there were
no significant violations, and then the coefficient HT, a measure
of distance between item response step functions, for the selected
items was computed; the greater the distance, the greater the
accuracy of the IIO. The computed HT was evaluated according to
the following criteria: HT50.3 inaccuracy, 0.3  HT50.4 low
accuracy, 0.4  HT50.5 medium accuracy and HT  0.5 high
accuracy (Ligtvoet et al., 2010).
Expert review of items
The goal of expert review is to select the item from the set at each
hierarchical level to represent all items at that level. After the first
round of Mokken modeling of scalability, monotonicity and IIO,
two informatics competency experts (S. Y. and S. B.) selected the
most representative item among the items with the same hierarchy
Table I. Results of the Mokken Scale analyses using AISP to check assumptions of unidimensionality and local
independency, c is the lower bound scalability coefficient H specified by the user (default 0.3).
SANICS-93 c¼0 c ¼ 0.10 c ¼ 0.20
No. scales 1 1 1 1
No. items 1st scale 93 93 93
1st scale (%) 100 100 100
No. excluded items 0 0 0 0
The pattern of having one dominant scale with the increment of c value confirms unidimensionality and local
independency of the instrument.
Informatics competency Mokken 581
from the 65 items on the 22 different hierarchical levels. The 22
selected items were further assessed to confirm scalability,
monotonicity and IIO to generate the final set of scale items.
Results
Scalability
No Hi coefficients were below 0.3 for the 93 SANIC items, and all
items met criteria for retention in the first round of the scalability
assessment. The Hi ranged from 0.53 to 0.93. The highest item
(0.93) included ‘‘Extract data from clinical data sets’’, and the
lowest item (0.53) included ‘‘Use telecommunication devices to
communicate with other systems’’. The scale-level coefficient (H)
on 93 items was 0.53 in the first round.
From the repeated AISP to check unidimensionality (theta
being a unidimensional latent variable) and local independence
(item response only depending on theta) with incremental lower
bound c parameter from 0 to 0.5, the patterns of the AISP results
followed the typical sequence of unidimensionality, in which most
items belong to one dominant scale and one or several small
scales with the increment of c. Table I shows the results of the
AISP of the 93-item SANICS with the increment of parameter c-
as the minimum value of coefficients Hi. At c ¼ 0, only one scale
was found which included all 93 items. At c ¼ 0.5, only one
dominant scale (68% of items loaded to the one dominant scale)
was found with several small scales. This pattern supports the
unidimensionality and local independence of the instrument.
After the first round investigation of other assumptions and
expert selection of representative items, we re-assessed the
scalability of the 22 items. The item-level and scale-level
scalability coefficients are summarized in Table II. None of the
Hi coefficients were below 0.3, and all of items were at a level
worthy of retaining items at the second round of the scalability
assessment. The most difficult item was noted as ‘‘Describe
general applications/systems to support clinical care (Hi ¼ 0.61)’’
followed by ‘‘Applies monitoring system appropriately according
to the data needed (Hi ¼ 0.61)’’. And, the easiest item was
depicted as ‘‘Demonstrate basic technology skills (e.g. turn
computer off and on, load paper, change toner, remove paper jams
and print documents; Hi ¼ 0.34)’’ followed by ‘‘Use e-mail’’
(Hi ¼ 0.38). The scale level scalability coefficient (H) was 0.51,
which indicates a strong scale.
Monotonicity
Based on the assessment of monotonicity in the first round
analyses, three items (e.g. ‘‘Explain use of networks for electronic
communication’’) were removed from the 93 item SANICS. After
three iterations of the second round, the four items (e.g. ‘‘Use the
Internet to locate, download items of interest’’) were also
removed iteratively. Table II and Figure 1 show that the final
18-item scale included no violations of monotonicity and that all
curves are monotonically increasing.
c ¼ 0.25 c ¼ 0.30 c ¼ 0.35 c ¼ 0.40 c ¼ 0.50
1 1 2 5
93 93 93 80 63
100 100 100 86 68
0 0 0 0
582 S. Yoon et al. J Interprof Care, 2015; 29(6): 579–586

Table II. Psychometric properties of the 18-item scale using Mokken analysis.

Item Mean Item Hi (SE) #ac #vi #tsig

1 Demonstrate basic technology skills (e.g. turn computer 4.40 0.344 (0.032) 120 0 0
off and on, load paper, change toner, remove paper
jams, print documents)
2 Use e-mail 4.64 0.377 (0.034) 119 0 0
3 Conduct on-line literature searches 4.02 0.378 (0.029) 119 0 0
4 Use applications to manage aggregated data (e.g. excel, 3.32 0.409 (0.026) 119 1 0
database, statistical software)
5 Recognize that the computer is only a tool to provide 3.62 0.410 (0.026) 119 0 0
better nursing care and that there are human functions
that cannot be performed by computer
6 Recognize the value of clinician involvement in the 3.51 0.430 (0.026) 121 1 0
design, selection, implementation and evaluation of
applications, systems in health care
7 Extract data from clinical data sets (e.g. Clinical Log 2.13 0.492 (0.025) 119 2 0
Database, Minimum Data Set)
8 Incorporate structured languages into practice (e.g. 1.69 0.505 (0.030) 120 0 0
ICD9 codes, CPT codes, nursing diagnoses codes)
9 Describe ways to protect data 2.49 0.528 (0.022) 119 0 0
10 Assess accuracy of health information on the Internet 2.99 0.528 (0.023) 118 0 0
11 Identify, evaluate and apply the most relevant 2.86 0.549 (0.020) 118 1 0
information
12 Use application to document patient care 2.45 0.553 (0.020) 119 0 0
13 Identify, evaluate and use electronic patient education 2.38 0.573 (0.020) 119 1 0
materials appropriate to language and literacy level at
the point of care
14 Use decision support systems, expert systems and aids 2.00 0.578 (0.022) 119 0 0
for differential diagnosis
15 Act as an advocate of system users including patients 2.16 0.586 (0.020) 119 0 0
and colleagues
16 Participate as a content expert to evaluate information 1.85 0.593 (0.021) 119 1 0
and assist others in developing information structures
and systems to promote their area of nursing practice
17 Applies monitoring system appropriately according to 1.88 0.606 (0.021) 119 0 0
the data needed
18 Describe general applications/systems to support clin- 2.15 0.612 (0.020) 119 1 0
ical care

Coefficient of scalability: scale H ¼ 0.51 (standard error [SE] ¼ 0.02); Coefficient of invariant item ordering: HT ¼ 0.72; Reliability: MS ¼ 0.94,
alpha ¼ 0.93, lambda2 ¼ 0.94; Skewness ¼ 0.45 (SE ¼ 0.10); Kurtosis ¼  0.32 (SE ¼ 0.20); Sum score: 21–90 (reference 18–90); Sum score
median: 49.
#ac: active comparisons; #vi: violations; #tsig: significant violations.

Figure 1. Process of Mokken scale modeling.

Invariant item ordering
A total of 25 items (e.g. ‘‘Recognize that one does not have to be
a computer programmer to make effective use of the computer in
nursing’’) were removed throughout 25 iterations in the first
round. This process resulted in 65 of 90 items with IIO. After 65
items were further reduced by the experts and the second round
scalability and monotonicity check procedures (Figure 2), the
second round IIO assessment on the 18 remaining items (expert
selection; 65 ! 22 items, monotonicity check; 22 ! 18 items)
revealed an IIO coefficient of HT ¼ 0.72 and no violations.
Expert selection
After the first round assessing scalability (93 ! 93 items),
monotonicity (93 ! 90 items) and IIO (90 ! 65 items),
DOI: 10.3109/13561820.2015.1049340 Informatics competency Mokken 583

Figure 2. Monotonicity plots of the 18-item scale.

584 S. Yoon et al.
two experts (S. B. and S. Y.) selected the representative items
among the items in each hierarchy (Hi) resulting in 22 items from
65 items. For example, among three items on the same
hierarchical level (Hi ¼ 0.37), ‘‘Demonstrate basic technology
skills’’ (e.g. turn computer off and on, load paper, change toner,
remove paper jams, print documents) was chosen over the others:
‘‘Use word processing’’, and ‘‘Use telecommunication devices
(e.g. modems or other devices) to communicate with other
systems (e.g. access data, upload and download)’’. The 22
selected items by the experts were further reduced to the final 18
items during the second round scalability, monotonicity and IIO
assessments. The reliability of the final 18 item scale was
Cronbach’s alpha ¼ 0.932.
Discussion
The hierarchical scale was developed using non-parametric IRT
Mokken scaling and resulted in an 18-item scale with excellent
reliability as measured by traditional and IRT standards (van
Schuur, 2003). The 18-item scale provides hierarchical informa-
tion on the difficulty of each item and this contextual information
may be useful for interpreting competency assessments and
designing curricula accordingly (Appendix). This study has
substantive and methodological implications for competency
measurement in IPE and IPP.
Most items in the 18-item scale reflect informatics compe-
tencies relevant to all health professionals. Sixteen items reflect
interprofessional informatics skills that support interpersonal
communication, patient-centered care and team collaboration –
key concepts of IPE and IPP (D’Amour, Ferrada-Videla, San
Martin Rodriguez, & Beaulieu, 2005; Zwarenstein, Goldman, &
Reeves, 2009). These include general competencies such as using
email, protecting data and appraising systems for patient care. The
ability to conduct on-line literature searches to assess health
information on the internet, and to use electronic patient
education materials appropriate to language and literacy level,
are also essential skills in IPP and IPE. Moreover, the ability to
use monitoring systems appropriately is a fundamental skill for
team functioning and shared decision making among interprofes-
sional team members (Meyer & Lavin, 2005; Nightingale, 1860/
1969). The remaining two items can be specified according to
each discipline: ‘‘Participate as a content expert to evaluate
information and assist others in developing information structures
and systems to promote their area of [discipline name here]
practice’’ and ‘‘Recognize that the computer is only a tool to
provide better [discipline name here] care and that there are
human functions that cannot be performed by computer’’.
In terms of methodological implications, this study found that
Mokken scaling offered benefits for competency measure refine-
ment. For example, it (1) provides a hierarchy, (2) is based on a
single latent trait, (3) has accurate reliability, (4) is shorter in
length and (5) does not make assumptions that are characteristic
of other psychometric methods (Mokken, 1971). The hierarchical
scale is particularly advantageous for constructing scales designed
to be administered to those with varying levels of expertise, for
differentiating between novice and expert practice and for
measuring change in competencies over time.
This study has several limitations. The dataset reflected a
single site and single sample. In addition, SANICS is based on
self-reported informatics competencies rather than observed
competencies and thus responses may be influenced by social
desirability bias.
In conclusion, this study applied Mokken scaling procedures to
develop an 18-item scale to measure informatics competency. The
strong results related to IIO suggest its potential relevance for
application in IPE and IPP. In addition, the decrease in length with
J Interprof Care, 2015; 29(6): 579–586
excellent reliability increases the utility of the use of the 18-item
scale in both education and practice settings. Moreover, the
approaches can be applied to construct a new hierarchical scale or
refine existing scales related to informatics attitudes or compe-
tencies for multiple health professions.
Declaration of interest
The authors report no conflicts of interest. The authors alone are
responsible for the writing and content of this article.
Funding
This study was supported by Wireless Informatics for Safe and
Evidence-based APN Care (D11 HP07346).
References
American Nurses’ Association. (2008). Nursing Informatics: Scope and
Standards of Practice. Washington DC: American Nurse Publishing.
Carroll, J.B. (1945). The effect of difficulty and chance success on
correlations between items or between tests. Psychometrika, 10, 1–19.
Centers for Disease Control and Prevention. (2009). Competencies for
Public Heatlh Informaticians. Retrieved from http://www.cdc.gov/
informaticscompetencies/downloads/PHI_Competencies.pdf.
Choi, J., & De Martinis, J. E. (2013). Nursing informatics competencies:
Assessment of undergraduate and graduate nursing students. Journal of
Clinical Nursing, 22, 1970–1976.
Collins, S.A., Mamykina, L., Jordan, D., Stein, D.M., Shine, A., Reyfman,
P., & Kaufman, D. (2012). In search of common ground in handoff
documentation in an Intensive Care Unit. Journal of Biomedical
Informatics, 45, 307–315.
Curran, C.R. (2003). Informatics competencies for nurse practitioners.
AACN Clinical Issues, 14, 320–330.
D’Amour, D., Ferrada-Videla, M., San Martin Rodriguez, L., & Beaulieu,
M. D. (2005). The conceptual basis for interprofessional collaboration:
Core concepts and theoretical frameworks. Journal of Interprofessional
Care, 19 Suppl 1, 116–131.
Desjardins, K.S., Cook, S.S., Jenkins, M., & Bakken, S. (2005). Effect of
an informatics for evidence-based practice curriculum on nursing
informatics competencies. International Journal of Medical
Informatics, 74, 1012–1020.
Feldman, S.S., & Hersh, W. (2007). Evaluating the AMIA-OHSU 10x10
program to train health care professionals. AMIA Annual Symposium
Proceedings, 2007, 953.
Fries, J.F., Bruce, B., & Cella, D. (2005). The promise of PROMIS: Using
item response theory to improve assessment of patient-reported
outcomes. Clinical and Experimental Rheumatology, 23, S53–S57.
Fries, J.F., Krishnan, E., Rose, M., Lingala, B., & Bruce, B. (2011).
Improved responsiveness and reduced sample size requirements of
PROMIS physical function scales with item response theory. Arthritis
Research and Therapy, 13, R147.
Hwang, J.I., & Park, H.A. (2011). Factors associated with nurses’
informatics competency. Computers, Informatics, Nursing, 29,
256–262.
Kubias, D. (2011). Health and clinical management – From the pen and
paper to the digital era. Adapting the healthcare environment to take
full advantage of information and communication technology.
Yearbook of Medical Information, 6, 48–50.
Kulikowski, C.A., Shortliffe, E.H., Currie, L.M., Elkin, P.L., Hunter, L.E.,
Johnson, T.R., . . . Williamson, J.J. (2012). AMIA Board white paper:
Definition of biomedical informatics and specification of core
competencies for graduate education in the discipline. Journal of the
American Medical Informatics Association, 19, 931–938.
Laramee, A.S., Bosek, M., Shaner-McRae, H., & Powers-Phaneuf, T.
(2012). A comparison of nurse attitudes before implementation and 6
and 18 months after implementation of an electronic health record.
Computers, Informatics, Nursing, 30, 521–530.
Lehmann, C.U., Shorte, V., & Gundlapalli, A.V. (2013). Clinical
informatics sub-specialty board certification. Pediatrics in Review,
34, 525–530.
Lei, P.W., Dunbar, S.B., & Kolen, M.J. (2004). A comparison of
parametric and nonparametric approaches to item analysis for multiple-
choice tests. Educational and Psychological Measurement, 64,
565–587.
DOI: 10.3109/13561820.2015.1049340
Ligtvoet, R., van der Ark, L.A., Bergsma, W.P., & Sijtsma, K. (2011).
Polytomous latent scales for the investigation of the ordering of items.
Psychometrika, 76, 200–216.
Ligtvoet, R., van der Ark, L.A., te Marvelde, J.M., & Sijtsma, K. (2010).
Investigating an invariant item ordering for polytomously scored items.
Educational and Psychological Measurement, 70, 578–595.
Lindberg, P., & Vingard, E. (2012). Indicators of healthy work
environments – A systematic review. Work, 41 Suppl 1, 3032–3038.
Loevinger, J. (1948). The technic of homogeneous tests compared with
some aspects of scale analysis and factor analysis. Psychological
Bulletin, 45, 507–529.
Lord, F.M. (1980). Applications of item response theory to practical
testing problems. Hillsdale, NJ: Lawrence Erlbaum.
Lord, F.M., & Novick, M.R. (1968). Statistical theories of mental test
scores. Reading, MA: Addison-Wesley.
Manojlovich, M., & DeCicco, B. (2007). Healthy work environments,
nurse-physician communication, and patients’ outcomes. American
Journal of Critical Care, 16, 536–543.
McNeil, B. J., Elfrink, V.L., Pierce, S. T., Beyea, S. C., Bickford, C. J., &
Averill, C. (2005). Nursing informatics knowledge and competencies:
A national survey of nursing education programs in the United States.
International Journal of Medical Informatics, 74, 1021–1030.
Meijer, R.R., Sijtsma, K., & Smid, N.G. (1990). Theoretical and
empirical-comparison of the mokken and the rasch approach to irt.
Applied Psychological Measurement, 14, 283–298.
Meyer, G., & Lavin, M.A. (2005). Vigilance: The essence of nursing.
Online Journal of Issues in Nursing, 10, 8.
Mokken, R.J. (1971). A theory and procedure of scale analysis with
applications in political research. The Hague: Mouton.
Molenaar, I.W. (1997). Nonparametric models for polytomous responses.
New York: Springer.
Molenaar, I.W., Sijtsma, K. (2000). User’s Manual MSP5 for Windows.
Groningen, The Netherlands: IEC ProGAMMA.
Nightingale, F. (1860/1969). Notes on nursing: What it is, and what it is
not. New York: Dover.
Appendix
Self-assessment of informatics competency scale for health professionals
For each statement, indicate your current level of competency on the scale of 0–4, where: 0 ¼ Not competent, 1 ¼ Somewhat competent,
2 ¼ Competent, 3 ¼ Proficient and 4 ¼ Expert.
Informatics competency Mokken 585
Ozbolt, J.G., & Graves, J.R. (1993). Clinical nursing informatics.
Developing tools for knowledge workers. The Nursing Clinics of
North America, 28, 407–425.
QSEN Institute. (2012). Graduate-Level QSEN Competencies,
Knowledge, Skills and Attitudes Retrieved from http://www.aacn.nch-
e.edu/faculty/qsen/competencies.pdf.
Sass, D.A., & Schmitt, T. A. (2011). Introduction to the special issue:
Moving beyond traditional psychometric approaches. Journal of
Psychoeducational Assessment, 29, 299–303.
Schafer, J.L. (1999). Multiple imputation: A primer. Statistical Methods
in Medical Research, 8, 3–15.
Sijtsma, K., Meijer, R.R., & van der Ark, L.A. (2011). Mokken scale
analysis as time goes by: An update for scaling practitioners.
Personality and Individual Differences, 50, 31–37.
Staggers, N., Gassert, C.A., & Curran, C. (2002). A Delphi study to
determine informatics competencies for nurses at four levels of
practice. Nursing Research, 51, 383–390.
Staggers, N., & Thompson, C.B. (2002). The evolution of definitions for
nursing informatics: A critical analysis and revised definition. Journal
of the American Medical Informatics Association, 9, 255–261.
van Schuur, W.H. (2003). Mokken scale analysis: A nonparametric
version of Guttman scaling for survey research. Political Analysis, 11,
139–163.
Vawdrey, D. K., Stein, D.M., Fred, M.R., Bostwick, S.B., & Stetson, P.D.
(2013). Implementation of a computerized patient handoff application.
AMIA Annual Symposium Proceedings, 2013, 1395–1400.
Westra, B.L., & Delaney, C.W. (2008). Informatics competencies for
nursing and healthcare leaders. AMIA Annual Symposium Proceeding,
2008, 804–808.
Yoon, S., Yen, P.Y., & Bakken, S. (2009). Psychometric properties of the
self-assessment of nursing informatics competencies scale. Studies in
Health Technology Informatics, 146, 546–550.
Zwarenstein, M., Goldman, J., & Reeves, S. (2009). Interprofessional
collaboration: Effects of practice-based interventions on professional
practice and healthcare outcomes. The Cochrane Database of
Systematic Review, 3, CD000072.

Basic computer skills

1. Demonstrate basic technology skills (e.g. turn computer off and on, load paper, 0 1 2 3 4
change toner, remove paper jams, print documents).
2. Use e-mail 0 1 2 3 4
3. Conduct on-line literature searches (e.g. PubMed). 0 1 2 3 4
4. Use applications to manage aggregated data (e.g. excel, database, statistical software). 0 1 2 3 4
Role
5. Recognize that the computer is only a tool to provide better [discipline name here] 0 1 2 3 4
care and that there are human functions that cannot be performed by computer.
6. Recognize the value of clinician involvement in the design, selection, implementation 0 1 2 3 4
and evaluation of applications, systems in health care.
Applied computer skills : clinical informatics
7. Extract data from clinical data sets (e.g. Clinical data warehouse, Minimum Data Set). 0 1 2 3 4
8. Incorporate structured languages into practice (e.g. ICD9 or 10 codes, CPT codes, 0 1 2 3 4
diagnoses codes).
9. Describe ways to protect data. 0 1 2 3 4
10. Assess accuracy of health information on the Internet. 0 1 2 3 4
11. Identify, evaluate, and apply the most relevant information. 0 1 2 3 4
12. Use application to document patient care. 0 1 2 3 4
13. Identify, evaluate and use electronic patient education materials appropriate to 0 1 2 3 4
language and literacy level at the point of care.
14. Use decision support systems, expert systems and aids for differential diagnosis. 0 1 2 3 4
15. Act as an advocate of system users including patients and colleagues. 0 1 2 3 4
16. Participate as a content expert to evaluate information and assist others in 0 1 2 3 4
developing information structures and systems to promote their area of [discipline
name here] practice.
17. Applies monitoring system appropriately according to the data needed. 0 1 2 3 4
18. Describe general applications/systems to support clinical care. 0 1 2 3 4
586 S. Yoon et al. J Interprof Care, 2015; 29(6): 579–586

Use and scoring

Please enter your discipline name in item 5 and 16 prior to use. Total score ranges from 0 to 44.064 using the following weights. Each subscale score
can be calculated separately. As the scale measures the level of self-confidence, absolute score varies by target population.

Item Score weight

1 0.344
2 0.377
3 0.378
4 0.409
5 0.410
6 0.430
7 0.492
8 0.505
9 0.528
10 0.528
11 0.549
12 0.553
13 0.573
14 0.578
15 0.586
16 0.593
17 0.606
18 0.612
Copyright of Journal of Interprofessional Care is the property of Taylor & Francis Ltd and its
content may not be copied or emailed to multiple sites or posted to a listserv without the
copyright holder's express written permission. However, users may print, download, or email
articles for individual use.