TAO DAYS Berlin, 10/09/2012

Applying Open Source to

Adaptive Testing
Michal Kosinski
mk583@cam.ac.uk

Michal Kosinski
mk583@cam.ac.uk

www.psychometrics.cam.ac.uk

PLAN
PART I: Open-source

PART II: Item Response Theory and Computerized

Adaptive Testing
PART III: What do you need to build your CAT (or any
other test or e-learning platform)
and where to get a right open-source software

Part I:
Open-source

Michal Kosinski
mk583@cam.ac.uk

Using open-source
Who is not convinced to open-source software?
Problems? Concerns?

Why open-source instead of

commercial one?
High reliability / free updates / constant debugging

Open source gives you independence:

Own your software copy and access all the updates

Enhance your software as (and for how much) you wish
See what is under the hood
Low switching costs = innovation and progress

Usually relatively low costs

Open-source elements as a part of

your own software dev. project
all the aforementioned advantages, plus:
Avoids rediscovering the wheel
Minimize dependence on your suppliers / employees
Decrease maintenance costs (e.g. debugging /
updating)

Publish your stuff as open-source

It doesnt mean that you have to stop offering commercial
s0lutions (e.g. mySQL model)
Low-cost marketing
Build trust
Get free input from the community
Ideas / Improvements / Enhancements / Debugging
Increased opportunities for funding
Improve the world! (and get competitive edge!)

Problems and myths

How do I sell products based on open-source?
In the same way as commercial ones!
Most open-source can be used for commercial purposes
Open-source based products based can be sold / licensed / etc.

Who do I blame/sue when something goes wrong?

Read your agreement with a commercial software developer
can you really blame them?
Choose your software wisely
Pay someone to give you a warranty

Part II:
Item Response Theory
and Computerized
Adaptive Testing
Michal Kosinski
mk583@cam.ac.uk

www.psychometrics.cam.ac.uk

IRT

Modelling on the item level (and not on the test level)

Robust item parameters
Can be used for test development and scoring
Score is item independent

Advantages:
Increased accuracy of the tests
Cheaper test development
On-going test development
Test security / IP protection
Enables Adaptive Testing (CAT)

CAT
Select items fit for a person and purpose

Advantages:
Much shorter tests
Less frustration / more attention
Detect react to inattention, cheating, etc.
Tests fit for purpose (e.g. screening)
Further increased test security and protection

IRT and CAT beyond testing!

Adaptive learning
Innovative question formats

Incorporate non-question information

Part III:
What do you need to build
your CAT (or any other test
or learning process)
Michal Kosinski
mk583@cam.ac.uk

www.psychometrics.cam.ac.uk

What you need:

1. Build a test = develop an Item bank
Item writing / translations

2. Establish the psychometric properties of the test

CAT / IRT analysis

3. Choose the most appropriate CAT approach

CAT simulation

4. Deliver your test on the testing platform

5. Analyse the results

Build an item bank

Consider open source items:
Avoid rediscovering the wheel!
IPIP.orgi.org
myPersonality.org/wiki

Crowd-source your items / translations

e.g. Amazon Mechanical Turk

Wiki-source your items?

Generate your items automatically:
Prof. Heinz Hollings
Cambridge IQ items generator (coming soon!)

Establish the psychometric

properties of the test
R language for statistical analysis
Classical and IRT analyses (+ anything else you may
want)
ltm, irttoys, psych, and loads of other psychometric
libraries
Powerful graphing capabilities
Flexible programming interface
Great wealth of analyses available within one
environment

Choose the most appropriate CAT

approach
Issues:

Where to start?
How to select next item?
When to stop?
What is the expected accuracy and test length?

CAT simulation :
catR package
catIrt
MAT (Multidimensional Adaptive Testing, M3PL model)

Testing platform choice

Use web-based technology:
Compatibility
Can be used locally

Plan ahead:
Choose the platform on the piloting stage
Use CAT/IRT enabled platform even if do you need it yet

Concerto:
an open source adaptive testing
platform
It is open source
It relies on open-source software (e.g. R, MySQL)
Ease of use develop advanced tests without a software
engineer
Flexible:
Robust R engine
Works with any CAT / IRT algorithm

Nicely integrates with other software e.g. Moodle, TAO, etc.

Design of Concerto
MySQL

R server

Database

Statistical
Engine

CKEditor

QTI
Editor

Item Editor

Concerto Platform

Open-source elements
Flexible R engine
Standard language in
psychometrics
Reliable / up-to-date methods
Powerful MySQL database
Modular design
Add / Replace modules if
necessary

Administration channels
Internet

Secured LAN e.g. testing centre

Locally e.g. Tablet with local copy of the platform

Concerto: brief tour

Analyse your results:

R
or
GNU Octave

TAKE-HOME MESSAGES
Avoid rediscovering the wheel
Do your homework BEFORE you decide to buy/develop
software

Stay independent:
Insist on using open-source software EVEN if you decide
to develop your own platform

Use IRT and CAT it is cheaper, more accurate, and

gives you a competitive edge!
Insist on using R in statistical analyses
Help us to fully integrate TAO with Concerto
Anyone has spare programmer or few euros?

More about Concerto:

www.psychometrics.cam.ac.uk
Michal Kosinski, mk583@cam.ac.uk
Prof. John Rust, jnr24@cam.ac.uk

Michal Kosinski
mk583@cam.ac.uk

www.psychometrics.cam.ac.uk