Dr Mike Sewell

Director of Admissions for the Cambridge Colleges

Cambridge Admissions Office

31 May 2013 Kevin Brennan MP House of Commons London SW1A 0AA Dear Mr Brennan, As promised, we should like to offer some further reflection on the analysis put forward to you by David Laws MP in his letter of 14 May. An initial analysis of the DfE study suggests that this could be a case of policy-based evidence. No attempt appears to have been made to recognise the effect of the variables at any level of detail (as far as can be determined given the lack of information about the base data employed), and that in turn suggests that the original proposition being explored was either poorly considered or loaded towards a particular result or, more charitably, that the raw data the researchers had to work with was simply not up to the job. In our judgment the study has three fundamental design flaws:The authors choose two oversimplified measures of GCSE and AS attainment, and compare them to a crude measure of degree success. Selective HEIs do not admit on a chance of getting a II.1. They seek to admit the students who are likeliest to perform really well which means a good II.1 (65%+) or First. Extrapolating from logistic regression in the manner in which this study does is highly problematic, since the binary outcome treats the lowest II.1 as equivalent to the highest First, and simply asks whether AS and KS4 performance can correctly predict which side of the II.2/II.1 divide a student will end up on. A linear regression model incorporating the full range of degree results (including failures) allows for finer distinctions between student performances, and so would have been much more appropriate to the question of whether AS Levels provide university admissions officers with useful additional information about student potential, over and above GCSE. The model fails to adjust for the fact that, at any given institution, students will tend to have roughly similar levels of prior attainment. This is because students in large part select HEIs based on their own level of academic attainment. Since, as the study admits, a II.1 is a relative measure of performance within an institutions cohort rather than a nationally calibrated standard, lumping all universities together, as this analysis does, is destined significantly to weaken correlations. This is because, while students obtaining II.1s at a highly competitive-for-entry university might average grades AAAA at AS Level, students obtaining II.1s at a less competitive-for-entry university might only average grades BBBB. A serious and rigorous analysis would have run multiple separate regression models to analyse the relationships within each institution.
Cambridge Admissions Office Fitzwilliam House 32 Trumpington Street Cambridge CB2 1QY Tel: +44 (0) 1223 333308 Fax: +44 (0) 1223 746868 Email: admissions@cam.ac.uk

The reporting of the studys findings is incomplete in a number of respects. In particular the report provides very little information about the factors considered when securing base data and conducting the analysis. A complete report should, for example, provide information on how the analysts counted GCSE results and AS results: which grades did they include, A*-C, or A*-F? Did they exclude subjects like General Studies and Critical Thinking which are not accepted by most selective universities? Did they include short courses, and did they cap the total number they counted? If they used a points system for each grade, what points did they allocate to each grade? Similarly for AS level, the key information should include whether capping was applied and the points allocated per grade. Finally, does this analysis include degree failure or drop-out? A good sense of what the data being analysed actually is and how it is being used is essential to a sound statistical report. We therefore requested the missing information, and the data-set, from the DfE in order to carry out our own modelling. The DfEs reply confirmed that the analysis is based on uncapped average scores in both cases, that KS4 qualifications are being used rather than GCSE in isolation, and that no subjects have been excluded. The University of Cambridge uses an average score calculated across the best three, or three most relevant, AS UMS scores (depending on whether the application is for arts or sciences), and excludes General Studies and Critical Thinking. While we await a decision on our request for the full data-set, we have run a similar logistic regression model to that to which Mr Laws refers using our own data, and using the yardstick of a good II.1/First, which around 50% of our students achieve. In this case we see a clear effect when adding UMS to GCSE: the predictive validity of the model is improved and the GCSE coefficient is rendered insignificant. That is, if we have UMS, GCSE does not add value, but if we already have GCSE, UMS adds value over GCSE while also eradicating any value GCSE had. So far as Mr Laws argument that GCSE and AS tell us the same thing goes, we can show that UMS tells us everything that GCSE does, and much more besides. Our position remains that AS UMS is key to accurate and fair university admissions. If other universities also adopted UMS as a key selection-tool, we believe that they too would find that it was much more valuable than GCSE. We hope that this is helpful. As ever, if you would like to discuss the issues in more detail, please dont hesitate to get in touch. Yours sincerely,

Jon Beard Director of Undergraduate Recruitment cc:

Dr Mike Sewell Director of Admissions for the Cambridge Colleges

David Willetts MP, Minister of State for Universities and Science, Department for Business,

Page 2 of 3

Innovation & Skills, 1 Victoria Street, London SW1H 0ET Michael Gove MP, Secretary of State for Education, Department for Education, Sanctuary Buildings, Great Smith Street, London SW1P 3BT David Laws MP, Minister of State for Schools, Department for Education, Sanctuary Buildings, Great Smith Street, London SW1P 3BT Dr Julian Huppert MP, 16 Signet Court, Swann Road, Cambridge CB5 8LA Dr Wendy Piatt, Director General, The Russell Group, 1 Northumberland Avenue, Trafalgar Square, London WC2N 5BW Nicola Dandridge, Chief Executive, Universities UK, Woburn House, 20 Tavistock Square, London WC1H 9HQ Mary Curnock Cook, Chief Executive, UCAS, Rosehill, New Barn Lane, Cheltenham GL52 3LZ Janet Graham, Director, SPA, Rosehill, New Barn Lane, Cheltenham GL52 3LZ

Page 3 of 3