Wednesday 13 July 2011

How to survive in psychological research

A Handbook of Skills and Methods in Behavioural Research is not the place you’d expect to find something to make you smile, but many years ago one of my graduate students pointed me to a wickedly funny piece by Ray Hodgson and Stephen Rollnick. Since then I’ve found myself loaning an increasingly dog-eared photocopy of the article to new generations of students and postdocs. Sadly, the article is not available electronically, though copies of the book can be tracked down. So here’s a summary of Hodgson and Rollnick’s laws, all of which are as pertinent to the older, seasoned researcher as to the intended readership of the ‘young, lively, questioning researcher who has great expectations but a lack of practical experience’:

Law #1. Getting started will take at least as long as the data collection
The barriers are various: perhaps the most salient for the newcomer is dithering induced by fear of commiting to a non-optimal design. Another barrier is having too many people involved; this just multiplies the dithering, as each person tries to include additional measures or graft on subsidiary projects. It’s vital to have someone who will take control for decision-making - a point emphasised in my previous post on the NationalChildren’s Study.
Hodgson and Rollnick also mention the need to get ethics approval, another topic that has featured on my blog. It’s got a lot worse in the years since they wrote their article: there’s even a kind of ‘meta-research’ in which the goal is to quantify the baleful influence of ethical scrutiny on research efficiency (e.g. Elwyn et al, 2005).
Law #2. The number of available subjects will be one-tenth of your first estimate
Note to young readers: ‘subjects’ are what we used to call ‘participants’ until someone decided that the term implied an unheathily controlling attitude to those taking part in experiment.
Quite simply, "as soon as somebody starts to research a particular condition, people with that condition leave the district". It’s totally true and totally mysterious.
Law #3. Completion of a research project will take twice as long as your last estimate and three times as long as your first estimate
This may be moderated by whether you are a pessimist or optimist, but no true pessimist would ever embark on a research project.
Law #4. A research project will change twice in the middle
Hodgson and Rollnick cite their experience with a one-year project to test the effectiveness of a Drinkwatchers program for problem drinking. “All we needed were thirty subjects from amongst the estimated 10,000 problem drinkers in South Glamorgan. One hundred and sixty problem drinkers answered the advertisement…Of these only eight volunteered to join a Drinkwatchers group, three turned up to the first meeting and one of these came to the second.” Since the study had been funded, the researchers decided the best they could do was an alternative study to discover what kinds of help problem drinkers really want. Needless to say, these days, such a change of plan would necessitate fresh ethics approval which would consume all the remaining time on the grant.
Law #5. The help provided by other people has a half-life of two weeks
Yes, yes, yes. Never do a study that depends on the kindness of strangers.
Law #6. The tedium of research is directly proportional to its objectivity
You really do need to know this when you start out in research. If you detest mundane, repetitive activities, try another career.
Law #7. The effort of writing up is an exponential function of the time since the data were collected
If the person who collected the data has left by the time you come to write it up, then it can be hard to remember exactly what was done, so you’d better be sure to have had a real obsessive in charge, who will document thoroughly every step of the research collection and data coding. Hodgson and Rollnick reckon that data that sit in a filing cabinet for 4 years will never escape.
On a more serious note, failure to get stuff written up is incredibly wasteful, especially if the funding for the study came from public funds. Sometimes the failure just comes from writer’s block, and sometimes because the researcher discovers a flaw that makes the study unpublishable. More commonly, though, the failure to write up is because the results are deemed uninteresting. This has the unfortunate effect of distorting the research literature, as null results are left in the file drawer. I'd like to see journal editors adopting a policy of determining ‘publishability’ of a paper on the basis of Introduction and Methods alone: if an interesting problem has been identified and the study is well-designed and adequately powered to answer it, then it should be published, regardless of the results.
Yet another reason for failure to publish is researchers who bite off more than they can chew. As I’ve suggested in a previous post, we need to move away from a system whereby the rewards for researchers are proportional to the amount of grant income they receive, to one that rewards thrift. And if research funders find themselves overwhelmed with far more proposals than they can fund, they should consider vetoing those who already have substantial funding, even if they are ace researchers. There is a limit to how much research someone can do and do well.
Law #8. Evidence is never enough
So you are lucky enough to get an interesting result, and are confident that this will change the field and make your reputation. And what happens? Nobody takes any notice. Hodgson and Rollnick note that research that conflicts with the prevailing view is likely to be ignored, but that’s not the only problem. You do also have to sell your science. But that does not need to mean cutting corners or distorting findings. But learn to write accessibly, get out there and give talks, start a blog (!) and, most important of all, focus on problems that are important.
Get hold of the original Hodgson and Rollnick chapter if you want positive tips on how to be a successful researcher. And for further advice, it’s hard to better Peter Medawar’s 1979 book Advice to a Young Scientist.

References
Elwyn, G. (2005). Ethics and research governance in a multicentre study: add 150 days to your study protocol BMJ, 330 (7495), 847-847 DOI: 10.1136/bmj.330.7495.847
Hodgson, R., & Rollnick, S. (1989). More fun, less stress: How to survive in research. In G. Parry & F.-N. Watts (Eds.), Behavioural and mental health research: A handbook of skills and methods (pp. 3-13). Hove, England: Lawrence Erlbaum Associates.
 

17 comments:

  1. Great post!

    It reminded me of a similar article by Steve Hayes, published in The Behavior Therapist in 1998. Fewer laughs, but the take-home-message is the same (i.e., say 'yes' to everything, and have fun!).

    Here's the reference:
    Hayes, S. C. (1998). Thirteen Rules of Success: A Message for Students. The Behavior Therapist, 21, 47-49.

    http://www.cs.dal.ca/~eem/gradResources/13Rules.pdf

    ReplyDelete
  2. This is fabulous stuff - and all absolutely true!

    ReplyDelete
  3. Enjoyed the post. The elements are applicable across the sciences not just in psychology. Law#2 about participant number (I prefer participants - subjects are bacteria or rats) can also be extended to include a larger than expected attrition rate for your clinical trial. Law#8 about evidence takes me back to a famous quotation "it is not enough to prove something, one has also to seduce or elevate people to it" (Nietzsche).

    ReplyDelete
  4. Charlie Wilson (@crewilson)14 July 2011 at 12:05

    I really like the idea of assessing "publishability" in advance based on idea and methods, and then publishing no matter what the results. Is this in serious consideration anywhere, to your knowledge?

    ReplyDelete
  5. thanks - will tell all my PhD students to read it! :-)

    ReplyDelete
  6. Thanks to all for comments - it is amazing how the laws have stood test of time and still seem to hit the spot all these years later.
    Charlie: I don't know of a journal that operates with these criteria, but they aren't my original idea.I first heard of them years ago at an editorial meeting of Journal of Child Psychology and Psychiatry. Your question makes me curious to track down the source, so I'll ask around to see if I can locate it.

    ReplyDelete
  7. I'd also add:

    Before you start your project, run yourself as the first subject and then analyze your data, in full, before you go any further.

    This is the only way to make sure that your protocol isn't fatally flawed, but it's so easy to skip doing this, leave all the analysis to the end, and only then realize that something is terribly wrong.

    ReplyDelete
  8. I'd also like to add a couple of laws.

    The first I am fairly sure I learned from Dorothy herself:

    Law #9: Your last subject will never ever ever ever turn up.

    This leads to ...

    Law #10: Make sure you add one extra subject to your proposed subject size.

    ReplyDelete
  9. Law 2 finally explains the mysterious absence of children with SLI in Manchester! Fortunately for my thesis, law 5 turns out to be false in Edinburgh where, thanks to the kindness of strangers, I was able to track down a few participants :)

    ReplyDelete
  10. hank one to make the tips! I wouldn't now have become this approach without help out! Can it be o . k in order to reference aspects of it in this little website basically if i include a back-link to this fact web page?

    ReplyDelete
  11. Those of us with a long memory can compare Mr. Busch's prose of today with that of years ago -- and be delighted at how much more interesting and incisive it has become.

    ReplyDelete
  12. Hi,
    Great information! Indeed very useful for me, Thanks for sharing
    Assignment Help
    Assignment Writing

    ReplyDelete
  13. I have to agree with most of your points, however not fully with #8. Starting a blog alone won't make you money. Each scientist knows how hard it is to make profit with his research papers.

    ReplyDelete
  14. Geraldine Pearson17 October 2013 at 05:56

    I totally agree with you on law #5 that help will be provided but never for the whole journey. So, it would be a good idea to maximize what you can get for the thesis topic instead of just taking them for granted which may greatly affect not only the project but also the relationship toward each other.

    ReplyDelete
  15. Well thanks for posting such an outstanding idea. I like this blog & I like the topic and thinking of making it right.
    Dissertation Writing Help
    Dissertation Help

    ReplyDelete