NUT Report: Decades of test-based incentive programs do not yield expected results

Grumpy Educators reported on a study conducted by the National Research Council here. The widget with the full text remains available on the right side of the screen. Last week, another study was published. The National Research Council released the results of a study examining the decades of implementing high-stakes standardized testing and concludes the practice has not achieved the expected impact on student achievement.

Isn't it time to ask why the persistence in continuing to pour time, money, and effort into test-centric policies?

This study was sponsored by Carnegie Corporation of New York and the William and Flora Hewlett Foundation. The National Academy of Sciences, National Academy of Engineering, Institute of Medicine, and National Research Council make up the National Academies. They are private, nonprofit institutions that provide science, technology, and health policy advice under a congressional charter. The Research Council is the principal operating agency of the National Academy of Sciences and the National Academy of Engineering.

To read the entire report on this study, select the widget on the right at the top.

The following press release describes the study and its findings:

WASHINGTON — Despite being used for several decades, test-based incentives have not consistently generated positive effects on student achievement, says a new report from the National Research Council. The report examines evidence on incentive programs, which impose sanctions or offer rewards for students, teachers, or schools on the basis of students' test performance. Federal and state governments have increasingly relied on incentives in recent decades as a way to raise accountability in public education and in the hope of driving improvements in achievement.

School-level incentives -- like those of No Child Left Behind -- produce some of the larger effects among the programs studied, but the gains are concentrated in elementary grade mathematics and are small in comparison with the improvements the nation hopes to achieve, the report says. Evidence also suggests that high school exit exam programs, as currently implemented in many states, decrease the rate of high school graduation without increasing student achievement.

Policymakers should support the development and evaluation of promising new models that use test-based incentives in more sophisticated ways as one aspect of a richer accountability and improvement process, said the committee that wrote the report.

Incentives' Effects on Student Achievement

Attaching incentives to test scores can encourage teachers to focus narrowly on the material tested -- in other words, to "teach to the test" -- the report says. As a result, students' knowledge of the part of the subject matter that appears on the test may increase while their understanding of the untested portion may stay the same or even decrease, and the test scores may give an inflated picture of what students actually know with respect to the full range of content standards.

To control for any score inflation caused by teaching to the test, it is important to evaluate the effects of incentive programs not by looking at changes in the test scores tied to the incentives, but by looking at students' scores on "low stakes" tests -- such as the National Assessment of Educational Progress -- that are not linked to incentives and are therefore less likely to be inflated, the report says.

When evaluated using low-stakes tests, the overall effects on achievement tend to be small and are effectively zero for a number of incentives programs, the committee concluded. Even when evaluated using the tests attached to the incentives, a number of programs show only small effects.

Some incentives hold teachers or students accountable, while others affect whole schools. School-level incentives like those used in No Child Left Behind produce some of the larger achievement gains, the report says, but even these have an effect size of only around .08 standard deviations – the equivalent of moving a student currently performing at the 50th percentile to the 53rd percentile. For comparison, raising student performance in the U.S. to the level of the highest-performing nations would require a gain equivalent to a student climbing from the 50th to the 84th percentile. The committee noted, however, that although a .08 effect size is small, few other education interventions have shown greater gains.

Effects of High School Exit Exams

The study also examined evidence on the effects of high school exit exams, which are currently used by 25 states and typically involve tests in multiple subjects, all of which students must pass in order to graduate. This research suggests that such exams decrease the rate of high school graduation without improvements in student achievement as measured by low-stakes tests.

Broader Measures of Performance Needed

It is unreasonable to implement incentives tied to tests on a narrow range of content and then criticize teachers for narrowing their instruction to match the tests, said the committee. When incentives are used, the performance measures need to be broad enough to align with desired student outcomes. This means not only expanding the range of content covered by tests but also considering other student outcomes beyond a single test.

Policymakers and researchers should design and evaluate alternate approaches using test-based incentives, the committee said. Among the approaches proposed during current policy debates are those that would deny tenure to teachers whose students fail to meet a minimal level of test performance. Another proposal is to use the narrow information from tests to trigger a more intensive school evaluation that would consider a broader range of information and then provide support to help schools improve. The modest and variable benefits shown by incentive programs so far, however, means that all use of incentives should be rigorously evaluated to determine what works and what does not, said the committee.

In addition, it is important that research on and development of new incentive-based approaches does not displace investment in the development of other aspects of the education system – such as improvements in curricula and instructional methods -- that are important complements to the incentives themselves, the report cautions.

###

The study was sponsored by Carnegie Corporation of New York and the William and Flora Hewlett Foundation. The National Academy of Sciences, National Academy of Engineering, Institute of Medicine, and National Research Council make up the National Academies. They are private, nonprofit institutions that provide science, technology, and health policy advice under a congressional charter. The Research Council is the principal operating agency of the National Academy of Sciences and the National Academy of Engineering. For more information, visit http://national-academies.org. A committee roster follows.

Contacts:
Sara Frueh, Media Relations Officer
Shaquanna Shields, Media Relations Assistant
Office of News and Public Information
202-334-2138; e-mail news@nas.edu

Pre-publication copies of Incentives and Test-Based Accountability in Education are available from the National Academies Press; tel. 202-334-3313 or 1-800-624-6242 or on the Internet at http://www.nap.edu. Reporters may obtain a copy from the Office of News and Public Information (contacts listed above).

NATIONAL RESEARCH COUNCIL
Division of Behavioral and Social Sciences and Education
Committee on Incentives and Test-Based Accountability

Michael Hout (chair)*
Professor and Natalie Cohen Sociology Chair
Department of Sociology
University of California
Berkeley

Dan Ariely
James B. Duke Professor of Psychology and Behavioral Economics
Fuqua School of Business
Duke University
Durham, N.C.

George P. Baker III
Herman C. Krannert Professor of Business Administration
Harvard Business School
Boston

Henry Braun
Boisi Professor of Education and Public Policy
Boston College
Chestnut Hill, Mass.

Anthony S. Bryk (until 2008)
President
Carnegie Foundation for the Advancement of Teaching
Stanford, Calif.

Edward L. Deci
Professor of Psychology;
Gowan Professor of Social Sciences; and
Director
Human Motivation Program
University of Rochester
Rochester, N.Y.

Christopher F. Edley Jr.
Professor and Dean
School of Law
University of California
Berkeley

Geno Flores
Deputy Superintendent
San Diego City Schools
San Diego

Carolyn J. Heinrich
Professor
LaFollette School of Public Affairs
College of Letters and Science
University of Wisconsin
Madison

Paul Hill
Director
Center on Reinventing Public Education, and
Professor
Daniel J. Evans School of Public Affairs
University of Washington
Seattle

Thomas J. Kane**
Professor of Education and Economics
Graduate School of Education
Harvard University , and
Deputy Director for Research and Data
Education Program
Bill and Melinda Gates Foundation
Seattle

Daniel M. Koretz
Professor
Graduate School of Education
Harvard University
Cambridge, Mass.

Kevin Lang
Professor
Department of Economics
Boston University
Boston

Susanna Loeb
Professor of Education and Business
Graduate School of Education
Stanford University
Stanford, Calif.

Michael Lovaglia
Professor of Sociology, and
Director
Center for the Study of Group Processes
Department of Sociology
University of Iowa
Iowa City

Lorrie A. Shepard
Dean and Distinguished Professor
School of Education
University of Colorado
Boulder

Brian Stecher
Senior Social Scientist and Associate Director
RAND Education
RAND Corp.
Santa Monica, Calif.

STAFF

Stuart W. Elliot
Study Director

* Member, National Academy of Sciences
** Was not able to participate in the final committee deliberations

NUT Report: Parents Head to D.C. in July

Save Our Schools (SOS) was covered in the Washington Post by Valerie Strauss:

Rita Solnet is a Florida businesswoman, parent and education activist. She is a co-founder of the nonprofit Parents Across America.

Why, as a parent, are you involved with the SOS March?

As a business owner and as a 15 year volunteer in the school system, I’ve had a front row seat and witnessed our public education system in decline. I attribute it to NCLB’s fixation on high stakes tests and the obsession with privatizing public education. NCLB has robbed all children of a solid curriculum inclusive of arts, music, literature, language, PE, history along with reading, math and sciences in favor of drills and prep for mindless bubble tests. I’m angry.

Education cannot be treated as a corporate business. It is that simple. Children’s minds are not commodities to be bought, sold, and exchanged. I frankly don’t see policy makers or many legislators grasping the totality of this destruction. With those I’ve met, most have little knowledge of what is truly happening inside classrooms.

As a result of making decisions in a vacuum, they are not making the best possible decisions. In fact, they are illogical decisions.

The past two years I’ve witnessed multi billion dollar testing & technology companies being the sole beneficiaries of policy makers’ decisions — not our children. And I see children as collateral damage in a control struggle, it seems, with unions and political parties.

They must take this conflict into a conference room to devise workable contracts to resolve that situation. They must stop meddling in the classroom with desperate, ill-fated attempts to fix those issues.

As a parent, I’ve concluded that if we don’t step up and take action to ensure we are heard, who else will do so?

I am also a co-founder of Parents Across America. Our motto is: Our children. Our schools. Our voices must be heard. I believe this with all my heart.

What do you think this SOS March will accomplish?

This rally and March will accomplish a few goals.

1) It will open up a candid and genuine dialogue once and for all. No longer can a news network air a week of one-sided perspective.

2) It will send a message to Congress that NCLB must end as it stands today. It failed our children miserably for 9 years. It devastated public education unintentionally but undeniably.

3) It will heighten the awareness that this goes beyond “disgruntled teachers.” Participation of parents and business men and women will demonstrate this goes beyond the worn out rhetoric that this is merely a union issue or a band of whiny teachers. These are real issues facing millions of voters. We are fed up.

4) It will help to raise a red flag on reckless spending of federal education dollars on ill-fated reforms which all research shows have proven to fail.

SANDRA IN BREVARD SUPPORTS THE EVENT AND WISHES SHE COULD ATTEND.

I AM A NUT.

Education Reform: Basic Questions Go Unanswered

Billionaires donating their wealth to causes that are intended to improve the quality of life for humankind is not bad, evil, or conspiratorial. However, it is anti-democratic when those billionaires, in a highly organized effort, push out parents, the public, all critical views, and the taxpayer from any role whatsoever in shaping or reshaping the educational landscape. In an extraordinarily dramatic federal education reform policy, fueled by a multitude of non-profits funded by unending stream of dollars from the Bill & Melinda Gates Foundation and others, parent and local control has been lost.

News outlets have failed to provide basic descriptions, commentary, analysis, and critique. A few hundred articles have been published on the subject By and large, critical voices have been shut out completely. Questions that demand answers regarding elevated testing requirements, complex longitudinal data systems, and serious privacy concerns go unanswered, ignored, or trivialized. I do not recall electing non-profits to legislative and regulatory roles. Is the May 22 New York Times article a sign that current policies will finally get the media attention the public rightly deserves? Looks like the Times article got the attention of the Tampa Gradebook, so at least one Florida newspaper covered the story. Will the topic make it to Meet the Nation, Fox News, or talk radio in ways that ordinary people can understand? In the end, we are stuck with the goods.

In the the blogosphere, I found smart fact-based Conservative and liberal, Republican and Democrat, parents, community members, and education experts voicing legitimate and reasonable concerns. On the heels of the NY Times article, long-time blogger Susan Ohanian fills in significant details regarding the players and extent of the Gates Foundation influence. There are some surprises. Read the list of players and their funding sources here.

Ohanian also describes the apparent news blackout:

Wanting to see which "independent experts" reporters called upon to explain Race to the Top and the Common Core standards, I examined over 700 articles published between mid-May 2009 and mid-July 2010. I eliminated cites from state ed officials, union officials and politicos. This left me with 152 outside experts quoted in 414 articles. Of the 23 experts quoted five times or more, 15 have connections with institutions receiving Gates funding and 13 with strong charter advocacy institutions. Who doesn't gets cited, raised very troubling questions.

Frederick M. Hess, American Enterprise Institute Resident Scholar and Director of Education Policy Studies writes a blog for Education Week, is quoted in the NY Times article:

Mr. Hess, a frequent blogger on education whose institute received $500,000 from the Gates foundation in 2009 “to influence the national education debates,” acknowledged that he and others sometimes felt constrained. “As researchers, we have a reasonable self-preservation instinct,” he said. “There can be an exquisite carefulness about how we’re going to say anything that could reflect badly on a foundation.”

“Everybody’s implicated,” he added.

Something must change in this regard. The Bill and Melinda Gates Foundation announced that it was starting another foundation, called Teaching First and covered by Washington Post blogger Valerie Strauss.

The plan includes campaigns to reach out to parents, teachers, students, business and civic and religious leaders, and to build “strong ties to local journalists, opinion elites, and local/state policymakers and their staffs.” The plan explains how the organization will ensure “frequent placement ... in local media coverage of issues related to teacher effectiveness and equitable distribution of effective teachers” in accordance with the Gates approach.

The proposal calls for supporting local groups that promote the value-added evaluation systems, and who even get involved in unions so they can demand this approach in collective bargaining for teachers contracts.

The motives of billionaire venture capitalists are really not the issue in my opinion. The issue is a lack of critical analysis and inclusion of the public on the massive education reform initiative underway right now. Hess wrote an interesting blog titled "What We've Got Here Is...Failure to Communicate" stepping over the Gates controversy into the policy itself.

Even mainstream conservatives are being radicalized. Last weekend, standards guru Sandra Stotsky, a longtime champion of standards-based reform and generally regarded as an NCLB supporter, blasted the very notion of federal involvement in schooling. In an e-mail exchange regarding the Common Core, she wrote, "I've tried to think of sound federal policies in education (with positive effects on student achievement), and the closest I've come are the Land Grant Acts of the 19th century...In my lifetime, I can't think of ONE federal policy that has improved student achievement."

It's kind of amusing, really. The self-proclaimed reformers just can't imagine that, confronted with data showing that many children are poorly served, any sensible adult could look askance at their favored policies. When confronted with skepticism that the measures will work as intended, the would-be reformers ask with wide-eyed shock, "Are you willing to just let those children fail?" If a conservative House staffer suggests that maybe the feds lack the ability or purview to solve the problem, would-be reformers seem to think they've stepped through the looking glass.

The problem is deeper than a failure to communicate.

President Obama's Weekly Address: Replacing NCLB

Education Reform and Privacy Concerns Collide

In February, Grumpy Educators covered U.S. Office of Education initiatives to build and develop longitudinal data systems for education here and here. The requirement for data systems that track student data from preschool through high school and beyond is part of current education reform policy on data-driven decision-making. The data, to be accessed by researchers, auditors and other agencies, may reveal what reforms, methods or textbooks work or do not work so well.

A spokesperson from the Data Quality Campaign, a non-profit founded by the Bill and Melinda Gates Foundation, praises the proposed changes:

"We can't afford not to use this information if we want to meet our big policy goal of graduating students ready for college and career."

However, accessing the data requires changes in current privacy protection laws or the Family Educational Rights and Privacy Protection Act of 1974 (FERPA).

The expansion of state student-record systems is central to President Obama's accountability agenda, which seeks to improve education through the better use of data. In a statement issued on Thursday, the U.S. secretary of education, Arne Duncan, said the proposed new rules would "strengthen privacy protections and allow for meaningful uses of data."

According to the Chronicle, the proposed changes would allow sharing of student-level data with researchers, auditors, and other agencies without violating FERPA. The article further notes that Congress prevented the Education Department from creating a "national 'unit record' data system in 2008, but has also funded states to develop these systems.

There are many serious concerns and unanswered question about these proposed changes that affect that rights of parent consent and the collection and use of vast amounts of data on the nation's children. How long will this data remain available? How will it be stored? When will it be erased? Will it be erased? Will parents and the children themselves when adults know how the information is used?

According to a Missouri Education Watchdog alert, public input and objections to the proposed rules changes is available until Monday, May 23. Background information, examples of specific objections to the rules, and the website location for registering objections is offered.

If privacy concerns matter and the lack of interest by Congressional oversight on this matter matter to you, review the Missouri Education Watchdog blog here and consider writing an objection.


http://chronicle.com/article/New-Rules-Would-Allow-for/127047/

http://chronicle.com/article/Why-Privacy-Matters-Even-if/127461/

NUT Report: No parent involvement wanted

Gretchen Herrera, a parent of a child with Asperger's Syndrome and Type 1 diabetes asked to opt out of testing on medical grounds. Her requests were received with threats, resistance, and ultimately denied. The professional opinions from his medical team were ignored. Her son liked the virtual charter school and took the first round of tests, not wanting to "disappoint" anyone.

Mom checked his blood sugar level after the test and it was sky high, just as the lead of his medical team indicated the testing would affect blood sugar levels. In a posting on Facebook, Herrera says that she will not infict the rest of the tests on her son.

Why must I be forced to do what my doctors, my son, and I know will harm him? I want my son to have access to the joyful, useful, relevant, real, and interesting learning experience that our tax dollars pay for? I’d happily take my tax dollars elsewhere, but unfortunately, our compulsory system of compliance doesn’t afford parents such options. I am forced to subject my son to that which will make him physically and emotionally ill if he is to get the education he he deserves. The system has failed and my child is being left behind.

Read the full story here.

NUT = No Unnecessary Tests

Ritalin: The jury remains out

In a study funded by Irish pharmaceutical Shire, a manufacturer of ADHD drugs, University of Pennsylvania researchers determined that kids who take medication for ADHD are “at no greater risk for heart problems than kids not taking ADHD medications.” The lead researcher says the findings of the study should "reassure parents that drugs such as Ritalin and Adderall aren’t associated with an increased risk of cardiovascular events." However, the researchers noted that "it was not possible to definitely rule out an association between ADHD drugs and increased rates of heart problems."

On the other hand, Dr. Steven Lipshultz, professor and chair of pediatrics at the University of Miami Miller School of Medicine is concerned these published results may be misinterpreted. HealthDay Reporter Steven Reinberg interviewed Lipshultz and presented his evaluation of the study this way:

"This study wasn't able to look carefully at where the major concerns are," he added.

The study did not specifically separate those children with heart disease from others, Lipshultz said, adding that the FDA's greatest concern was with the use of these medications in patients with underlying heart disease. In fact, the drugs may be contraindicated for such children, he said.

These drugs are a great benefit to children with ADHD, Lipshultz said. "On the other hand, there are still considerable safety concerns about the use of stimulant therapy in children, especially children with underlying health conditions," he said.

In 2007, the FDA ordered manufacturers of ADHD drugs to produce a "medication guide to alert doctors and parents" of heart-related risks associated with these medications. The results of FDA directed research on these risks are forth-coming. Manufacturer Shire supports the "careful medical evaluation of cardiovascular risk by physicians for patients diagnosed with ADHD as they seek appropriate treatment."

Fifteen years ago, Dr. Lawrence H. Diller, wrote a book “Running on Ritalin” describing what he viewed as an over-diagnosis of ADHD and over prescription of Ritalin and similar drugs to many children “who would respond well to family therapy and tailed programs and routines at home and at school.”
“Diller warned that as harried parents, teachers and physicians attached the ADHD label to more and more children who were dreamy, unmotivated, forgetful, restless, impulsive or distractible, the nation's tolerance for children's natural temperamental variance would narrow. Instead of helping children work around weaknesses and choose strategies and paths that played to their strengths, society's growing inclination to medicate them, Diller cautioned, could turn many into lifelong patients.”

Diller has just released a new book, “Remembering Ritalin”, in which he describes the lives of ten of his former patients now adults.

Watch one of those former patients from www.rememberingritalin.com website:



In a L.A. Times interview, Diller maintains his view on over diagnosis and over medication while noting value in medicines having prescribed them for 32 years:

Pills represent efficiency, and effective nondrug interventions like special education or behavior-modification value engagement with the child. The medical and educational systems value efficiency. Parents, when offered a choice initially between efficiency and engagement, almost always choose engagement. However, when offered the choice of only a pill or nothing, they'll take the pill. And that's often the only choice they're given.

So I remain a relatively lonely professional voice pointing out this moral dilemma. But it is greatly edifying that when people hear the full message, they invariably say, "You know, he's right."

For more information on Dr. Diller and ADHD, visit his website here.

NUT Report: Parents Taking Their Case to D.C.

NUT Report: No parent involvement wanted

Gretchen Herrera, a parent of a child with Asperger's Syndrome and Type 1 diabetes asked to opt out of testing on medical grounds. Her requests were received with threats, resistance, and ultimately denied. The professional opinions from his medical team were ignored. Her son liked the virtual charter school and took the first round of tests, not wanting to "disappoint" anyone.

Mom checked his blood sugar level after the test and it was sky high, just as the lead of his medical team indicated the testing would affect blood sugar levels. In a posting on Facebook, Herrera says that she will not infict the rest of the tests on her son.

Why must I be forced to do what my doctors, my son, and I know will harm him? I want my son to have access to the joyful, useful, relevant, real, and interesting learning experience that our tax dollars pay for? I’d happily take my tax dollars elsewhere, but unfortunately, our compulsory system of compliance doesn’t afford parents such options. I am forced to subject my son to that which will make him physically and emotionally ill if he is to get the education he he deserves. The system has failed and my child is being left behind.

Read the full story here.

UPDATE: Read the medical recommendation here.

NUT = No Unnecessary Tests

SB1620/HB7197: Digital Learning Bill Passes

Late Monday, I noticed that the House ditched its version of so-called digital learning and voted on a new bill that matched the Senate version. Last week I read the 50 plus page Senate bill and called my legislator to voice my concerns. The entire Brevard delegation supported the bill. As written, the bill opens the door for outsourcing virtual charters. The requirements for teachers include a type of certification for individuals who may live outside the United States, but no Florida teacher certification requirements are needed. These virtual charters do not require a physical presence in Florida; however, all who offer services must be approved by the Florida Department of Education. This bill is also an unfunded mandate requiring school districts to provide access for those who do not have the proper equipment and internet access. The bill was introduced by Florida Senator Anitere Flores, who is listed as a member of the digital learning committee on the education foundation established by former Governor Jeb Bush.

On another note, UCF won a Bill and Melinda Gates Foundation grant to develop national standards for blended learning that seems will apply to higher learning. Blended learning means students attend class AND complete some coursework online. This approach has years of supportive research as the most successful application of online learning and so noted on the UCF website. More students successfully finish courses in blended learning than other formats. SB1620/HB7197 does not acknowledge the research and takes a step backward in successful application of virtual capabilities.

This blog marks #50 and coincides with the near end of the Florida legislative session. So-called education reform sped through legislatures nationwide with barely any public discussion and even less media coverage. Non-public entities with no accountability are establishing national policy on national standards, national assessment, and national curriculum. Local control has been vastly diminished. Soon the U.S. Congress will be taking up the reauthorization of No Child Left Behind. Many of the issues connected with education reform will likely re-emerge. Grumpy Educators serves as a resource for Brevard County residents and others who are interested in education reform and may be helpful for those who are interested in the reauthorization when it comes up. Hopefully, the national media will take greater interest in covering these issues when they are debated in Congress.

Education reform is a complex topic and moves faster than a speeding bullet. I have attempted to break it down into small enough chunks so the entire picture could be understood. I hope it has been helpful. Now, as last year, Sandra In Brevard will take a blogging break. However, there are two separate groups who have organized to push back on the efforts to nationalize education that I am following closely.

Best regards to all readers of Grumpy Educators.


SandraInBrevard

Where Real Reform Must Start