Student Data Collection: Purpose, Costs, Risks?

Secretary of Education Arne Duncan says this is a once in a lifetime opportunity to do something different. The current policies are different, but are they better, more effective, and more efficient?

According to the President's Blueprint, the state longitudinal data base initiative will provide the targeted accountability to raise achievement by ensuring students are making progress and by linking student achievement to teacher performance. The recent proposed FERPA regulations requires additional safeguards for the collection, release, and safeguarding of student data. The fiscal impact on State and local-levels is not mentioned.


Four reasons the public should have deep concerns about the development and proposed regulations and the collection of data from birth to college on every student in the nation:

April 20, 2011, SONY CEO apologizes for major security breach.
Sony Corp. Chief Executive Howard Stringer apologized for "inconvenience and concern" caused by the security breach that compromised personal data from more than 100 million online gaming accounts.

May 17, 2011, Data Breech Infects Massachusetts Unemployment Office.
An estimated 225,000 Massachusetts residents could become fraud victims as a result of a computer data breach in the state unemployment system.

May 27, 2011, Data Breech at Security Firm Linked to Attack on Lockheed.
Lockheed Martin, the nation’s largest military contractor, has battled disruptions in its computer networks this week that might be tied to a hacking attack on a vendor that supplies coded security tokens to millions of users, security officials said on Friday.

June 2, 2011, Google Mail Attack Blamed on China.
Suspected Chinese hackers tried to steal the passwords of hundreds of Google email account holders, including those of senior U.S. government officials, Chinese activists and journalists, the Internet company said.

June 3, 2011, Hackers Attack another Sony network.
Hackers broke into Sony Corp’s computer networks and accessed the information of more than 1 million customers to show the vulnerability of the electronic giant’s systems, the latest of several security breaches undermining confidence in the company.

NUT Report: "We have to do something."

When asked reasonable questions and explanations regarding education reform initiatives, the replies are predictable. When it comes to merit pay based on value-added measurement (VAM), the rationale is described as:
1) We need to know the truth.
2) We have to do something.

In addition to the two recent studies reported here and here, another group of assessment experts from the Economic Policy Institute sent a letter to the NY Board of Regents showing evidence that VAM is unstable and prone to error. Among the concerns expressed, this team of experts noted the following:

Teachers’ ratings are affected by differences in the students who are assigned to them. Students are not randomly assigned to teachers – and statistical models cannot fully adjust for the fact that some teachers will have a disproportionate number of students who may be exceptionally difficult to teach (students with poor attendance, who are homeless, who have severe problems at home, etc.) and whose scores on traditional tests have unacceptably low validity (e.g. those who have special education needs or who are English language learners). All of these factors can create both misestimates of teachers’ effectiveness and disincentives for teachers to want to teach the neediest students, creating incentives for teachers to seek to teach those students those expected to make the most rapid gains and to avoid schools and classrooms serving struggling students.

These experts also cited a RAND Corporation study that concludes:

....the research base is currently insufficient to support the use of VAM for high-stakes decisions about individual teachers or schools. It is important that policymakers, practitioners, and VAM researchers work together, so that research is informed by the practical needs and constraints facing users of VAM and that implementation of the models is informed by an understanding of what inferences and decisions the research currently supports.

Nevertheless, the NY Board of Regents ignored the evidence and joined the wave of national experimentation with no reasonable rationale nor justification.

How many expert studies are required before education reformers respond to the legitimate concerns over this massive implementation effort? Unstable, unreliable data will not reveal the "truth" and "We have to do something" are not adequate explanations. Decades of doing "something" that does not work is irresponsible policy making.

The following experts signed the letter:
Eva Baker, Distinguished Professor, UCLA Graduate School of Education
Director, National Center for Research on Evaluation, Standards and Student Testing (CRESST)
President, World Educational Research Association, 2010-2012
Past President, American Educational Research Association

Linda Darling-Hammond, Charles E. Ducommun Professor of Education, Stanford University
Past President, American Educational Research Association
Executive Board Member, National Academy of Education

Edward Haertel, Vida Jacks Professor of Education, Stanford University
Chair, Board on Testing and Assessment, National Research Council
Vice-President, National Academy of Education
Past President, National Council on Measurement in Education

Helen F. Ladd, Edgar Thompson Professor of Public Policy and Professor of Economics, Sanford School of Public Policy, Duke University
President, Association of Public Policy and Management

Henry M. Levin, William Heard Kilpatrick Professor of Economics and Education, Teachers College, Columbia University
Past President, Evaluation Research Society
Past President, Comparative and International Education Society

Robert E. Linn, Professor Emeritus, University of Colorado at Boulder
Past President, American Educational Research Association
Past President, National Council on Measurement in Education

Aaron Pallas, Professor of Sociology and Education, Teachers College, Columbia University
Fellow, American Educational Research Association

Richard Shavelson, Dean Emeritus and Margaret Jacks Professor Emeritus, Stanford University
Past President, American Educational Research Association

Lorrie A. Shepard, Dean & Distinguished Professor, University of Colorado at Boulder
Past President, American Educational Research Association
Past President National Academy of Education
Past President National Council on Measurement in Education

Lee S. Shulman, Charles E. Ducommun Professor Emeritus, Stanford University
President Emeritus, Carnegie Foundation for the Advancement of Teaching
Past President, American Educational Research Association

SB736/HB7019: The trouble with value-added measurement

SB736/HB7019 intends to apply value-added measurement (VAM) to connect student achievement on a test as a performance indicator of the teacher, or a poor student score = poor teaching. VAM uses statistical tools to calculate the contributions a teacher makes to student achievement gains. The formula for the calculation includes factors the developer chooses to include. What do experts have to say about the value of value-added measurement?

1) Jim Angermeyr, Director of Research, Evaluation & Testing for Bloomington Public Schools and "one of the designers of a widely respected value-added test lots of Minnesota schoolchildren take two or three times a year" was interviewed by Beth Hawkins. In the article, "Do 'value-added' teacher data really add value?", Angermeyr is described as "something of a standardized testing skeptic. He believes that economists tend to believe in using value-added data in evaluation. Educators and psychometricians, not so much."

“It’s not necessarily that the methodologies are wrong,” he said. “It’s that the inferences we’re drawing can be wrong.”

"The kids are the greatest of the variables, of course. The tests may tell you a student is reading better or sliding in math, but they don’t tell you whether she spent the summer with a tutor or he is so young the test isn’t as accurate as it would be in an older child.

Nor is the same test used from year to year. A particular student or teacher may fare better on a test closely normed with curriculum vs. one aligned with a set of knowledge-based standards."

“You leave out a lot of the potential variables,” Angermeyr said.
“They’re just not at the point where we should use them to make decisions about jobs.”

2) The National Research Council and the National Academy of Education gathered experts in the this field and held a workshop titled "Getting Value Out of Value Added." There was general agreement by participants that there is no single statistical model and still a work in progress for applicability when measuring teacher quality.

Discussion resulted in these additional conclusions:

• Results generated by existing models have a high degree of instability.
• Good results require good tests and good test results.
• Experts find it useful for low stakes use to identify areas that need improvement.
• Experts recommend against using it for high stakes such as, teacher pay.

According to Henry Braun, the group determined the following:

"To nobody’s surprise, there is not one dominant VAM. Each major class of models has shortcomings, there is no consensus on the best approaches, and little work has been done on synthesizing the best aspects of each approach. There are questions about the accuracy and stability of value-added estimates of schools, teachers, or program effects. More needs to be learned about how these properties differ, using different value-added techniques and under different conditions. Most of the workshop participants argued that steps need to be taken to improve accuracy if the estimates are to be used as a primary indicator for high-stakes decisions; rather, value-added estimates should best be used in combination with other indicators. But most thought that the degree of precision and stability does seem sufficient to justify low-stakes uses of value-added results for research, evaluation, or improvement when there are no serious consequences for individual teachers, administrators, or students." (p.54)

While Florida plunges into creating dozens of new tests, North Carolina's legislature in bipartisan agreement is sending a bill to the Governor to end some end-of-course tests, maintaining those to meet federal requirements and to measure student achievement. Two years ago, they voted to end a few others. This legislature wishes to stop paying for so many tests that are both expensive and failing to yield the returns once thought beneficial.

Concerns over the costs for implementing SB736/HB7019 by Florida Senators and by school boards are being reported. In the end, this is an unfunded mandate using an experimental statistical model, and an expensive test development and implementation scheme that extends far beyond the reach of RT3 dollars. Legislators remain silent on the issue of costs and cost benefits. The public has a right to know.

To read the "Getting Value Out Of Value Added" report for free, go to the widget on the right side of this page, select the icon that looks like an open book with the word Read underneath, and then Open Book in the small screen area. The document will open so you can read it easily.

The experts in the National Research Council and the National Academy of Education in this conference included:

Rita Ahrens, Education Policy Studies
Joan Auchter, National Board for Professional Teaching Standards
Terri Baker, Center for Education, The National Academies
Dale Ballou, Vanderbilt University
Henry Braun, Boston College
Derek Briggs, University of Colorado at Boulder
Tom Broitman, National Board for Professional Teaching Standards
Alice Cain, House Committee on Education and Labor
Duncan Chaplin, Mathematica Policy Research
Naomi Chudowsky, Center for Education, The National Academies
Pat DeVito, AE Concepts
Beverly Donohue, New Visions for Public Schools
Karen Douglas, International Reading Association
Kelly Duncan, Center for Education, The National Academies
John Q. Easton, Consortium on Chicago School Research
Stuart W. Elliott, Center for Education, The National Academies
Maria Ferrão, Universidade da Beira Interior, Portugal
Rebecca Fitch, Office of Civil Rights
Shannon Fox, National Board for Professional Teaching Standards
Jianbin Fu, Educational Testing Service
Adam Gamoran, University of Wisconsin–Madison
Karen Golembeski, National Association for Learning Disabilities
Robert Gordon, Center for American Progress
Jeffrey Grigg, University of Wisconsin
Victoria Hammer, Department of Education
Jane Hannaway, Education Policy Center
Patricia Harvey, Center for Education, The National Academies
Lloyd Horwich, House Committee on Education and Labor
Lindsey Hunsicker, Committee on Health, Education, Labor, and Pensions
Ben Jensen, Organisation for Economic Co-operation and Development
Ashish Jha, Harvard School of Public Health
Moshe Justman, Ben Gurion University, Israel
Laura Kaloi, National Association for Learning Disabilities
Michael Kane, National Conference of Bar Examiners
Judith Koenig, Center for Education, The National Academies
Michael J. Kolen, University of Iowa
Adam Korobow, LMI Research Institute
Helen F. Ladd, Duke University
Kevin Lang, Boston University
Sharon Lewis, House Committee on Education and Labor
Valerie Link, Educational Testing Service
Dane Linn, National Governors Association
Robert L. Linn, University of Colorado at Boulder
J.R. Lockwood, RAND Corporation
Angela Mannici, American Federation of Teachers
Scott Marion, National Center for the Improvement of Educational Assessment
Daniel F. McCaffrey, RAND Corporation
Alexis Miller, LMI Research Institute
Raegen Miller, Center for American Progress
John Papay, Harvard University
Liz Potamites, Mathematica Policy Research
Ali Protik, Mathematica Policy Research
Sean Reardon, Stanford University
Mark D. Reckase, Michigan State University
Andre Rupp, University of Maryland
Sheila Schultz, HumRRO
Lorrie Shepard, University of Colorado at Boulder
Judith Singer, Harvard University
Andrea Solarz, Director of Research Initiatives, National Academy of Education
Gerald Sroufe, American Educational Research Association
Brian Stecher, RAND Corporation
Justin Stone, American Federation of Teachers
David Wakelyn, National Governors Association
Greg White, Executive Director, National Academy of Education
J. Douglas Willms, University of New Brunswick
Mark Wilson, University of California, Berkeley
Laurie Wise, HumRRO