ducation researcher Walter Haney likes to play a little trick on educators and board members. During presentations, he hands out data on several schools and asks people in the audience to rate the schools based on those statistics.
The educators and board members don't know the imaginary numbers are designed in such a way that it would be virtually impossible for these schools to exist in the real world. For example, Haney might present data on a school that has a high dropout rate, a low graduation rate, and a small percentage of students enrolled in core academic courses. Yet this same school, according to the hypothetical data, scores in the top 10 percent on state achievement exams--highly unlikely, given the rest of the statistical profile. Over the years, Haney says, only two people have ever realized something was strange about the conflicting numbers. Most people have expressed confidence in their ratings before Haney reveals the secret. Why are we so gullible when it comes to statistics? "People tend to be very uncritical about data and where they come from," says Haney, who is director of the Center for the Study of Testing at Boston College. "When they're faced with a complexity of data, they end up focusing in on one or two or, at most, three variables. And most people almost always rely primarily on test scores." Of course, there is much more to student achievement than test scores. But what other indicators should you use to measure school and student success? How should you interpret the additional data? And when you do use other indicators to evaluate schools and districts, do new problems arise? Take the graduation rate, or the percentage of high school students who earn standard high school diplomas, not the General Equivalency Diploma (GED). Districts, states, and the nation often set goals to raise graduation rates--an indicator most researchers agree is a good one. Yet this gauge of success is not without its own problems. For instance, when schools are mandated to raise graduation rates, some researchers say, educators might feel pressured to lower standards, especially for students who are doing poorly.
Richard M. Haynes, an education researcher at Western Carolina University in North Carolina, says grade inflation is a real possibility in districts and states where there is an overemphasis on high school graduation rates. But he says it would be hard for U.S. schools to inflate grades now "because there's so little room to inflate them" any further. The high school completion rate--that is, the percentage of 18 to 24-year-olds who have completed a high school education and have either a diploma or a GED--is another important indicator, according to Richard Rothstein, a research associate for the Economic Policy Institute in Washington, D.C. Both high school graduation rates and high school completion rates have an impact on future as well as present generations, says Rothstein, who is the author of What Do We Know About Declining (or Rising) Student Achievement, a 1997 report by the Educational Research Service. "One of the best predictors of a student's achievement is the educational attainment of the parents," he says. "Parents who have finished high school are more confident to encourage their kids to finish high school. So [high school graduation and completion rates] have a significant impact on the next generation."
Rothstein concedes that efforts to raise graduation rates are "somewhat in conflict" with a simultaneous goal of raising academic standards. But he says he has not seen any trends showing that higher high school graduation rates resulted in lower academic standards or that higher academic standards led to lower graduation rates. Gordon Newton, division director for school improvement for Kentucky, is more skeptical. "Graduation rates, by themselves, do not indicate a higher level of skills on the part of kids," he says. "An A in one school is not an A in another. And the message we're getting from business is that high school graduates are not prepared to function well on the job because their [academic] skills are not as high as they used to be. If we make high school courses more rigorous, that diploma will have more meaning." PROMOTION OR RETENTION? Inevitably, though, many kids are passed along from grade to grade without ever meeting tough academic standards, a process called social promotion. Lately, districts across the country have begun clamping down on social promotion by holding kids back from progressing to the next grade if they don't meet certain standards. In Chicago, for instance, school officials have developed a policy that says age should have nothing to do with whether a student is retained or promoted. Under the policy, all eighth-graders must read at about a seventh-grade level on the Iowa Test of Basic Skills or risk being forced to repeat eighth grade. The value of tracking retention rates is questioned by some researchers, however. "Retention rates are useless," says Rothstein. Unlike the graduation rate, which is an easily measurable standard, he says, retention rates are difficult to track and interpret. Is an increasing retention rate good, because it means the school is holding kids to higher standards? Or is it bad, because it means teachers are doing a poor job of educating students? Those questions are impossible to answer on the basis of retention rates alone. Newton of Kentucky agrees: "The whole notion of retention is highly questionable. You do not raise the rigor of the courses without providing the supports necessary for kids." Also, Newton says, most research shows that the more often a student is retained, the more likely that student is to drop out of school. Of course, researchers could not have reached that conclusion had they not tracked retention rates in the first place, so this indicator does have some value. Also, some researchers say, school districts can use retention rates to pinpoint the grade levels at which students are most likely to be retained. Even so, Peter McCabe, an administrator in the Office of Policy and Evaluation at the California State Department of Education, says he believes retention figures are just too murky to interpret. "It's kind of like rates of school suspensions," says McCabe. "It's not really clear if a number is good or bad. So how do you rank it?"
TRACKING RESULTS A rise in school suspensions, McCabe says, could be interpreted as a sign that a school is losing control of its students. Or, he adds, it could be read as a positive trend, showing that school officials are getting tougher on kids who violate school policies. Despite the possible ambiguity, many school districts find such figures useful when they are reviewing discipline policies. The best indicators, researchers say, are closely linked to student outcomes, such as the percentage of kids who go on to two or four-year colleges. In Missouri, state officials track not only the percentage of students who go on to college, but also the percentage who enter the military and the percentage who go to work in a field they were trained for in a high school vocational education program. Kent King, coordinator of supervision for the Missouri School Improvement Program, a seven-year-old reform effort, says that this postsecondary success rate is one of a dozen indicators used to evaluate school districts in the Show Me state.
King believes it's important to look at more than just the percentage of students who go to college, because a school district could be turning out a large number of graduates who go into the military or enter well-paid technical fields straight out of high school but not a high percentage of college students. Such a district could be criticized for a low college-attendance rate when in fact it is doing a stellar job of preparing its students for the real world. Haney cautions against overemphasis on the college attendance rate, too, because the numbers could be easily manipulated. One district might exclude its vocationaleducation students from the percentage count, giving it an unfair advantage over a district that includes vocational students in the count. Even so, he says such data can be valuable. Equally important, several researchers say, is tracking the percentage of students who take core academic courses and those who take Advanced Placement courses (as well as the percentage passing AP exams). These figures give school officials solid data on what percentage of students are willing to stretch themselves beyond the normal academic expectations.
Other student data worth watching include student absenteeism rates, teen pregnancy rates, the percentage of students experimenting with drugs or alcohol, the percentage of students whose parents are involved in schools, and the percentage of students performing community service. With so many indicators to follow, researchers suggest that school district officials focus on the indicators that best match district priorities.
STANDARDS FOR ALL Should poor urban schools be compared to other poor urban schools? Or should all schools--regardless of socioeconomic level--be expected to reach the same standards? "Frankly," says Rothstein, "we should have different standards for different kids." But not everyone agrees. This year, for example, California will stop including demographic data in the state's High School Performance report card. Setting the bar lower for less advantaged students and districts is "sending the wrong message," says California's McCabe. "Everybody has to get up to the same standards." Expecting less of poor inner-city and rural schools, McCabe says, means dooming kids in those communities to lives of low expectations and poor performance. In a sense, he says, it's a backhanded form of discrimination against poor people and minorities. In Kentucky, school officials say they have tried to sidestep the debate by focusing more on improvement within individual schools than on comparisons between schools. Still, expectations for improvement are not watered down for schools serving lower socioeconomic communities. "We don't want schools racing against each other--we want them racing against themselves and the clock," says Newton. "In the past, we were always focused on inputs and opportunity. The major shift here is toward measurable results." Kentucky officials track such indicators as attendance, dropout rates, and percentage of kids entering the military or going on to college. But the bulk of the evaluations are made on the basis of a battery of tests, including multiple-choice assessments, writing portfolios, performance events, and essay and short-answer exams. All the indicators are boiled down to a single composite number. The state has set a standard composite number that all schools are expected to meet by 2012. Schools might be at different levels now, but they must be progressing toward the 21st-century standard. And to reach that standard, low-achieving schools, having further to go than high-achieving schools, must improve more rapidly. If a school falls 5 percent below its baseline--which is its 1992 composite score--it is placed in "crisis." A school can also be placed in crisis if it fails to meet its annual improvement goals two years in a row. When a school is in crisis, a state education expert, in effect, takes over the role of the principal and the school council. The expert reviews the staff every six months and sends a recommendation to the district superintendent regarding which staff members should be fired or kept on board. In addition, parents who have a child in a crisis school have the option of transferring their child to a more successful public school at the district's expense. On the flip side, there are 51 categories of monetary rewards for schools that exceed their improvement goals. School councils--which are made up of the principal, two teachers, and two parents--decide, along with the school staff, what to do with the reward money. For a program like Kentucky's to be successful, Newton says, it must have a clear definition of success (in Kentucky, it is performance on the student assessments). Moreover, curriculum must be closely aligned with the assessments (misalignment, he says, is often the root cause of problems in poorly performing schools); and each school must have a thoughtful and detailed plan for how it is going to transform itself. Rothstein, however, is skeptical of reform programs that are heavily linked to test scores. And he says comparing kids from disparate time periods--in the Kentucky case, the timeline ranges from 1992 to 2012--is ludicrous. Much of the criticism of today's public schools comes from comparing them with schools from 20 or more years ago, he says, when the world was a very different place. "No institution can design appropriate reforms based on faulty diagnoses or utopian standards," Rothstein writes in What Do We Know About Declining (or Rising) Student Achievement? "If there was a golden age of education from which we have now fallen, we have no way of knowing about it because we have no way of making meaningful comparisons between achievement then and now." LOOKING FOR ACCOUNTABILITY Yet the nation's push for greater accountability brings with it the demand for more data to evaluate. The public, which once relied solely on district officials for statistical reports about the state of schools, is now aggressively seeking out ways to analyze the numbers independently. Columbus, Ohio, is a case in point. In 1996, The Columbus Dispatch hired a company called SchoolMatch to conduct an accountability audit of the city schools. The company analyzed school district policies, test data, attendance figures, dropout statistics, enrollment in Advanced Placement and honors courses, grading patterns, and the district's curriculum.
First, researchers collected reams of so-called input data--that is, data on the human and financial resources available to the district, as well as a demographic profile of the community. Among the factors analyzed were percentage of students living in poverty, minority population of the district, per-pupil expenditure, percentage of the teaching staff with a master's degree or higher, percentage of kids performing at grade level in first grade, percentage of community members with a high school education or higher, and "mean cognitive ability index," a measure of the intellectual ability of young children at the beginning of their school careers. On the basis of these data, the auditors established a set of standards for the district, including a target student attendance rate of 94 percent and expected average scores for state and national achievement tests such as the Ohio Proficiency Test and the American College Testing (ACT) assessment. The recommendations also called for the district to keep the dropout rate under 18 percent, and for at least 10 percent of high school juniors and seniors to be enrolled in Advanced Placement courses. SchoolMatch then compared those expectations with reality. Columbus found the city's schools were doing well in some areas--such as attendance and the percentage of kids taking Advanced Placement courses. But the dropout rate was a disappointingly high 30 percent, and performance on Advanced Placement exams was not strong. The audit also compared Columbus to urban districts with similar demographic data. "From my perspective, the audit was something we should welcome, embrace, cooperate with, and learn from," says Mark Hatch, president of the Columbus school board. "We'll make a stronger, better school system by opening ourselves up to close scrutiny. We know we need to be doing a better job in some areas." Like Walter Haney, Hatch says he can't overemphasize the danger of focusing in on only a few pieces of data. The Columbus audit involved hundreds of statistics and analyses. Says Hatch: "You have to look at the whole picture." Kevin Bushweller, project editor for Education Vital Signs, is a senior editor of The American School Board Journal. Sources: Charts 1, 2, 3, 5, and 6--U.S. Department of Commerce; chart 4--American College Testing Program; charts 7 and 8--U.S. Department of Education Reproduced with permission from the December 1997 issue of Education Vital Signs, a supplement to The American School Board Journal. Copyright ©1997, National School Boards Association. The American School Board Journal is an editorially independent publication of the National School Boards Association. Opinions expressed by this magazine or any of its authors do not necessarily reflect positions of the National School Boards Association. This web page may be saved to disk, printed out for individual use, or reproduced in quantities of less than 100 copies for academic use only, provided this copyright notice remains intact on each copy. This web page may not be otherwise transmitted or reproduced without the consent of the Publisher. For more information, contact Magazines Coordinator Jo Surette, (703) 838-6739.