How to Assess the Educational Health of San Francisco Schools

From Dr. Kathy Emery's great website - Education and Democracy

How to Assess the Educational Health of a School District?
by Kathy Emery
The last two superintendents of the San Francisco Unified School District have based their reputations on raising standardized test scores. Both claimed to have done so, but the FACTS belie the claims.

High-stakes testing advocates such as Mayor Gavin Newsom and Superintendent Arlene Ackerman claim that they are "not ideological, but results driven." But test scores as "results" are based on ASSUMPTIONS about what signifies intelligence and which knowledge is important. Basing assessment on test scores alone, or even primarily is highly ideological.

• "It is wrong to evaluate schools based on students' standardized achievement test scores. Educators and non-educators alike need to understand why." James Popham, The Truth About Testing: an Educator's Guide to Action

1. Standardized tests are created so that the results closely correlate with socio-economic status. This is done ON PURPOSE. This ensures their "reliability," i.e., students get the same score every time they take the test. For an example of how tests are strongly correlated to SES view a map comparing selected census track's median income with the API of the neighborhood's elementary school in San Francisco in 2003. More data: look at a map of median income and compare it to API scores of all elementary schools in San Francisco, all middle schools and all high schools.
You can look at the test score data from one elementary school in SF during the last 5 years. And then look at Popham's analysis of a standardized test to determine how many items are correlated to SES and how many to "inherited aptitude." Not much is left over for other kinds of interventions. [more by Popham and Berlak]

2. There are many and various "threats" to the "validity" of a test score. Here are just a few:

1. Test scores are supposed to be reported as a range, not a single number. Test makers do not claim that their tests are accurate enough to be reported precisely. There are too many variables that interfere with a test taker when they are taking the test (too hot in the room, not feeling well, distracted by a sudden but brief commotion outside, low blood sugar, not enough sleep the night before................ ). The conditions under which every child in every school in every district of the state are not standardized. The validity of the test scores depends on such standardization of conditions.
2. Most comparisons of a school from one year to the next are based on the combined test scores of the students (such as the Academic Performance Index in California). These scores are NOT "matched scores," that is the student population tested from one year to the next is not the same.
3. The degree to which the student population taking the test differs in its demographics from the student population used to norm the test is the degree to which the test is invalid. For example, if the norming sample has only 2 percent English language learners but the school has 25 percent English language learners, the test is not very valid at all.

See Harold Berlak's Handbook for more information about why using test scores to evaluate student achievement is not a good idea.

What is the Alternative to Using Test Scores to Measure Achievement?

One can start thinking about how to develop alternatives by looking at Debbie Meier's Six Alternative Assumptions to the high-stakes testing agenda. Meier expands upon these assumptions in her book, In Schools We Trust.

The real question today, a question provoked by the Business Roundtable's successful implementation of high-stakes testing, is how can parents, teachers and other local community members work together so they begin to trust each other enough to have honest, deep and ongoing conversations about educational goals, assessment, pedagogy, curriculum, standards, governance, and so forth?

If the members of a school's community cannot develop alternative vision to high-stakes testing, then our public school system will continue on its current path of punishing "low-performing" schools and allowing "high-performing" schools to produce task-completers rather than problem-solvers. This distinction is addressed in Don Arnstine's Democracy and Arts of Schooling in which he distinguishes between education and socialization. He argues that it is crucial for any possibility of democracy that we do both.

SEE alternative reform models in SF and the Bay Area.
For more go to Education and Democracy.