Community Members: Log In / Join Us

Is the Scientific Method Becoming Less . . . Scientific?

In my ongoing search to better understand how we reconcile the creative tension between subjective and objective measures of the world — including our ongoing (and thus far) elusive search for a better way of tracking how people learn — I took note of a recent New Yorker article that cast light on some emerging problems with the ostensible foundation of all objective research — the scientific method.

In the article, author Jonah Lehrer highlights a score of multiyear studies — ranging from the pharmaceutical to the psychological — in which core data changed dramatically over time. Drugs that were once hailed as breakthroughs demonstrated a dramatic decrease in effectiveness. Groundbreaking insights about memory and language ended up not being so replicable after all. And the emergence of a new truth in modern science — the “decline effect” — cast doubt on the purely objective foundation of modern science itself.

Without recounting the article in entire, there are several insights that have great relevance to those of us seeking to find a better way of helping children learn:

  • In the scientific community, publication bias has been revealed as a very real danger (in one study, 97% of psychology studies were proving their hypotheses, meaning either they were extraordinarily lucky or only publishing outcomes of successful experiments). The lesson seems clear: if we’re not careful, our well-intentioned search for the answers we seek may lead us to overvalue the data that tell us what we want to hear. In the education community, how does this insight impact our own efforts, which place great emphasis on greater accountability and measurement, and yet do so by glossing over a core issue — the individual learning process — that is notoriously mercurial, nonlinear, and discrete?
  • In the scientific community, a growing chorus of voices is worried about the current obsession with “replicability”, which, as one scientist put it, “distracts from the real problem, which is faulty design.” In the education community, are we doing something similar — is our obsession with replicability leading us to embrace “miracle cures” long before we have even fully diagnosed the problem we are trying to address?
  • In the scientific community, Lehrer writes, the “decline effect” is so gnawing “because it reminds us how difficult it is to prove anything.” If these sorts of challenges are confronting the scientific community, how will we in the education community respond? To what extent are we willing to acknowledge that weights and measures are both important — and insufficient? And to what extent are we willing to admit that when the reports are finished and the PowerPoint presentations conclude, we still have to choose what we believe?

Why We Measure Things

To conclude my recent bender on the “data craziness” that is plaguing our national education reform efforts, and once again in an effort to highlight a more thoughtful approach that resists either extreme — i.e., “all data all the time or no data none of the time” — I want to share, courtesy of my friend Lisa Kensler, this wonderful 1999 (read: pre-NCLB!) article by Meg Wheatley.

See what you think, and please share your thoughts and reactions.


The X Factor of School Reform

In case you missed it, there was a great piece in yesterday’s New York Times, the core message of which has a lot of relevance for those of us who, barely a week removed from not one but two major reports of misleading test data being used to evaluate schools and school districts, continue to search for the simplest way of evaluating what may be the most complex undertaking in the professional world — creating a challenging, engaging, relevant, supportive and experiential learning environment in which all children can learn.

The Times article had nothing to say about school reform — it was about the Fed’s inability to decide whether to stimulate the economy now or later. And it was about how even in a social science flush with quantitative data, the “social” aspect of the science — i.e., human behavior — is sufficiently complex and nonlinear to make certainty a chimera. “One point I always make to my graduate students,” said Robert Solow, a Noel Prize winner and MIT professor, “is never sound more certain than you are.”

Would that such caution were commonplace in our current conversations about education reform!

Of course, the message is not that economics is a boundless free-for-all discipline that uses numbers to hide its own guesswork — charges that are sometimes made to rebut the growing push in education circles to embrace a greater use of student information to guide adult decision-making — but one message seems clear: beware the worship of “data” in your search for certainty, as long as human beings are part of the equation. “The entire question of how emotion will change people’s behavior is pretty much outside the standard model of economics,” said Dan Ariely, a professor at Duke. “Pride is not in the model. Fear is not in the model. Revenge is not in the model. Even simple things like disenchantment of people who are fired from their jobs — the model doesn’t account for how devastating that experience can be.”

Reform leaders, are you listening?


Data-Driven Decision Making . . . and Soccer?

Great timing.

A week after I wrote about what the World Cup can teach us about school reform, the New York Times published an article about the growing push for more detailed data in the relatively data-free world of professional soccer.

I am not, for what it’s worth, against the use of more sophisticated data in making decisions about how to improve the learning conditions for kids (or, for that matter, how to make better decisions on the soccer pitch). Who would be? In fact, I’ve written in the past about how a balanced scorecard in schools would help educators do their jobs more effectively.

That being said, I am very much against the glorification of data as a way to make extremely subjective, non-linear things — like learning how to use one’s mind well, or watching a collective burst of creativity and synchronicity that leads to a beautiful soccer goooooooaaaaaaal — into extremely objective, linear things for which we can appropriately plan and script out a desired, predictable response.

I don’t think it’s coincidental that this new push for soccer data is reported the same week as an announcement in my home city that Chancellor Michelle Rhee intends to significantly expand the use of standardized tests so that “every D.C. student from kindergarten through high school is regularly assessed to measure academic progress and the effectiveness of teachers.” What’s afoot in both instances is, on one hand, the (appropriate) desire to take human ingenuity and apply it to situations that in the past have lacked specificity, and, on the other, the (inappropriate) effort to make everything quantifiable, resulting in an overreliance on that which can be measured — at the expense of everything else.

Notably, the push for soccer data seems far more measured than what I see in education. According to Mark Brunkhart, the president of a company that provides soccer data for a fee to clubs and news organizations, he and his staff do not blindly evangelize statistics. Every month or two, he says, he gets a call from a professor or graduate student who is a rabid soccer fan and just finished Moneyball, the book that brought sabermetrics into the mainstream in 2003. (I wrote about Moneyball and its potentially positive implications for school reform in a 2009 column titled “What Would Theo Do?”)

“Every single one comes with the idea that they’re going to solve soccer with the ‘Moneyball’ approach,” Brunkhart said, “and I try to talk them all down.” Similarly, the president of the Society for American Baseball Research pointed to Miroslav Klose’s second goal in Germany’s 4-0 victory against Argentina in the World Cup quarterfinals as an example of how statistics seem to overlook the nuance and elegance of soccer. “A series of three or four absolutely beautiful passes — how do you capture that?” he said. “It’s just the nature of the game.”

Would that I were seeing similar restraint among our education leaders. As longtime educator Ted Sizer once said, “Inspiration, hunger: these are the qualities that drive good schools. The best we educational planners can do is to create the most likely conditions for them to flourish, and then get out of their way.”

© 2022 Faces of Learning
Website by AndiSites.   |  Go back to top ↑