At the Manchester meeting someone -- I think it was Tim Carey, but I'm not sure, asked me about a paper I'd come across about the difficulty of predicting percentile rankings. I've tracked the paper down:
David Rogosa
"Accuracy of Individual Scores Expressed in Percentile Ranks: Classical Test Theory Calculations"
Stanford, July 1999
http://www-stat.stanford.edu/~rag/ed351/TECH509.PDF
It studies the question: given a test (e.g. an educational assessment test) of less than perfect reliability, how accurate is someone's percentile ranking on the test likely to be?
Test reliability is defined as 1 - Ve/Vo, where Vo is the population variance of observed scores and Ve is the noise variance, which can be estimated by testing people more than once (and making various assumptions I won't go into). This is the same as the correlation between test and retest scores. Values of around 0.9 are generally considered acceptable for assessment tests for practical use.
He gives exhuastive calculations, of which I'll just quote one example: for a reliability of 0.9, if a student's score is at the 50th percentile, the probability is 0.309 that their true score is within percentiles 45 to 55.
ยทยทยท
--
Richard Kennaway, jrk@cmp.uea.ac.uk, http://www.cmp.uea.ac.uk/~jrk/
School of Computing Sciences,
University of East Anglia, Norwich NR4 7TJ, U.K.