May 2005: Comments on 2005 3rd Grade Reading CSAP Test Results - See Below
Between November 2001 and December 2002 I performed statistical and probabilistic analyses of CSAP test results. I had been using state and district data available on the Colorado Department of Education (CDE) website and the Lewis-Palmer School District #38 raw test scores for 2001 and 2002 which were kindly provided by my local school district.
I have had a life long interest in random phenomena and the probabilistic description of such. Indeed, when I was an adjunct professor at the University of California at Davis and at the University of Denver, it was mostly statistical physics that I taught. Calculating the probabilities or liklihoods of events occuring has been a hobby for decades and is an area in which I consult.
My interest in the CSAPs came about when I was chairing the school district accountability advisory committee, which was during the period when the Colorado Student Assessment Program was being put into place. Upon seeing the details of the program my immediate question was how can anyone make sense of the testing results when the sample sizes are so small? Random fluctuations would dominate the testing results and overwhelm the trends that one would hope to find in the test scores. It is what electrical engineers would call a signal-to-noise issue.
After a few years worth of CSAP test results were available, I began the analytical work described in the documents below. I pursued this out of scientific curiosity, it's the kind of thing scientists do.
I had several goals in this work:
1. Characterize and quantify the random fluctuations that are universally observed in the fractions of students who achieve in the Proficient or Advanced groups on the CSAP tests at the individual school level and at the state level.
2. Use probability theory to provide guidelines to help determine when year-to-year changes in test outcomes are statistically significant.
3. Find methods by which one can detect improvement hidden in the statistical noise that is the natural consequence of the small sample sizes associated with any given test.
My results and conclusions are summarized in the Handout that was prepared when I discussed this work at the annual conference of the Colorado Association of School Boards in December 2002. The summary points are:
1. Random fluctuations in CSAP performance are unavoidable.
2. Since the magnitude of the fluctuations is proportional to 1/Sqrt(N), where N is the class size, the CSAP outcomes fluctuate greatly enough to mask any real quantitative educational improvement.
3. Because N is always small the CSAP results are extremely sensitive to the "cutscore" boundaries.
4. These fluctuations lead to great uncertainty in the total overall score used to rank entire schools.
I have not performed further analyses since 2002. It is clear from news reports, however, that statistical considerations have been significant issues to the extent that the details of the CSAP outcomes are no longer widely reported. Newspaper reports confine themselves to stating that schools are holding steady or show slight improvement and so on.
That this web page receives many hundreds of hits and several hundred downloads per month indicates that people are interested in this topic.
The details of my work and numerous examples are presented in the large document, which can de downloaded from below.
My December 2002 presentation entitled "The CSAP See-Saw: Understanding the Fluctuations in CSAP Test Scores" at the Colorado Association of School Boards (CASB) Annual Conference and the Handout can be downloaded from below.
May 2005: 3rd Grade Reading CSAP Results
The results of CSAP tests of 3rd grade reading proficiency are released by the State of Colorado in the spring, whereas all the other CSAP results are released in the late fall of the testing year. The 2005 3rd grade reading results were released on Monday, May 2, 2005 and appeared in the newspapers the following day.
Here, I will comment on the CSAP results for the Lewis-Palmer School District #38.
The D-38 results of the 3rd grade reading CSAP tests for the eight years, 1998-2005, that they have been given are shown in the following graph:
The graph shows the scores for the six D-38 elementary schools as well as the D-38 average.
This is a busy graph but that it is so busy is a demonstration of the points that I have stated above. Random fluctuations are the norm, not the exception. The 3rd grade class sizes in each of these schools are typically 60 to 70 students. Using the rough rule-of-thumb that I stated above, we would expect +/- 12% -15% random fluctuations in the proficiency level or CSAP score from year to year. This is pretty much just what we see here.
The inference to be drawn is that no meaning can be attributed to year to year changes within that uncertainty range. This behavior is immutable; the random fluctuations will continue in this manner for as long as the tests are given. The fluctuations impose a fundamental limitation on what can be learned from tests such as the CSAPs about the educational improvement or decline in schools from year to year, as is obvious from the graph, over intervals of eight years or longer. Assuming that the CSAP tests themselves are invariant in time, which is a dubious assumption, at this level of fluctuations (also known as statistical noise) decades would be required to observe a trend.
About the Author:
Final Documents & Presentations:
Power Point presentation to the Colorado Association of School Boards Annual Convention on December 6, 2002.
Handout for CSAB Convention (December 6, 2002, 230 kB PDF)
A comprehensive guide to understanding the origins of fluctuating proficiency levels and to determining when changes in scores are significant.
Previous Documents (work in progress):
Statistical Analysis of 2001 CSAP Test Data - by W. L. Morgan, Ph.D. (February 2002, 1.6 MB PDF)
Summary of February 2002 Document
Statistical Analysis of 2002 3rd Grade Reading CSAP Test - by W. L. Morgan, Ph.D. (July 2002, 1.5 MB PDF)
Power Point Presentation to Pike's Peak Area Superintendents (April 2002)