Tuesday, 1 July 2014

The conventions of Randomised Control Trials are too limiting

I just read a research paper slamming "synthetic phonics" and extolling "whole language".
Why do academics have to come down on one side or another like they are supporting a football team?

This to me epit​omises a fundamental problem that wrecks scientific research - we can't cope with cognitive dissonance.
Our culture requires us to assess stuff on one scale from good to bad, and then come up with a binary conclusion - it's good, or it's bad. 
What's wrong with a curate's egg - good in parts, bad in other parts?

Synthetic phonics (one of my pet subjects) has lots of good points.  "Whole language" (the other team) has lots of good points.  They are not mutually exclusive. 
The problem with most research is it's looking for a holy grail of "your one single hypothesis is probably true".   Everyone bow to the Probability Deity.

It's present in our justice system - a person is either guilty or not guilty.  What if he's partially guilty?  I know you can't be partially pregnant.  But you can be partially guilty.  You can be partially right.  You can be an almost complete shit and still be nice to your mother.

This problem besets research into vital development procedures which effect the lives of millions.
Some very smart people from MIT, I think it was, did a study which found that the benefits of microfinance initiatives are not proven.
That is so not helpful.
What we need is to know all the aspects of microfinance that are beneficial, and to what degree.
And what aspects of microfinance as it is now practiced are not beneficial, and what can we do about it.

We ourselves (ReadingWise) just did an RCT where we came up with the probability that our literacy program worked.  (It did, and remarkably well.)
That's all very well.
But what are its good bits, and how far do they work?  What aspects of it don't work so well?  That's what we need to know.  Shades of grey, please.  But we are trapped in an academic culture that disapproves of that.  As it so happens, I'm writing a supplementary paper where I can go into these shades of grey.  But to get a research trial published, you have to follow a very restrictive set of conventions which just don't help illuminate the issue you are trying to explore.

So please, academics, it's time to drop this convention of proving just one hypothesis with one parameter.  It's just so arbitary.  So 20th century.  

Test yourself - is Luis Su├írez (the footballer with a habit of biting people) a good or bad person?  You probably thought bad, because he bites people.  But maybe he looks after his Mum and donates lots of money to orphanages.  See - nothing is completely black and white.
  

No comments:

Post a Comment