Sunday, 23 April 2017

Red flags: Assessing sources

This was originally posted on 'The Scarlet Onion' webpage but it has since ceased to exist :( 

I once had an online 'discussion' with a chap claiming that the twin towers were, in fact, brought down by the US government and that 9.11 was all an inside job. He sent me a link to, in his words, a ‘peer reviewed academic journal article’ to back this theory up. 

The link led to the ‘Journal of 9.11 studies. One of the editors was Kevin Ryan who coincidentally was also the article’s author. Editors do occasionally write articles for their own journals but still...I was dubious. So I googled Kevin Ryan and found that he’d written a book on why 9.11 was actually 'an inside job' and was fired from his previous university job his views. The article was about Nano-thermites (which he said the government used to blow up the twin towers), a type of explosive, and yet Ryan was working on water testing. Things didn’t add up. 

O Red Flags  
Imagine going to a restaurant and seeing no customers, the paint peeling and a smoke coming from the kitchen. None of this necessarily means the food there is bad, -hey it might be great, but these are all the kind of things which Dorothy Bishop refers to as ‘red flags’. These red flags can acts as a kind of early warning system. 

This brings me a conversation about the relative pros and cons of learning styles/MI and the Montessori Method I had on twitter earlier in the year. I was sent a link to a series of educational articles by a teacher who thought they supported her case. When I read them i noticed a number of red flags. I thought it would be useful to blog about these and hopefully this will be useful for other teachers assessing the quality of sources. So here are a few things to look out for.

O Mode of publication
Is the article in a book or on a website or in a journal? 

This matters because anyone can say anything in a book or on a website. Books often seem impressive as if somehow putting something in a book makes it more weighty and serious. Usually, academic journals, which have been through peer-review, are more likely to have credible information than books. Websites are almost always a no-no as there is usually zero quality control. Disclaimer (of course it always depends on exactly what you are looking for and what kind of website/book it is).

What about the papers that I was directed to look at? They seem to be on a website, which is an initial red flag. However, when we get to the articles in the download section they actually seem to be from real journals, namely, the MASAUM Journal of Reviews and Surveys (volume 1), MASAUM Journal of Open Problems in Science and Engineering (volume 1) and the International Journal of Engineering Research and Applications (volume 2); so far so good. But O (first red flag) why publish education articles in journals of engineering? 

O Dubious journals  
So these articles are in journals but what kind of journals?

Ideally the journals would be Peer-reviewed, and well established. The first two of these journals fail the ‘well-established’ test as they are all only seemingly on their second or first volume and are not available anywhere on the web. The third one actually does exist and looks (roughly) like a journal should. The site looks a bit cheap and unprofessional (Gmail address for submissions etc) but it exists, which is a start.

O Pay for play
A little bit of digging around the FAQ section of the third journal and we find this:

Q: How much do I have to pay for publication fee
A: $150

Paying for publication isn’t a great sign. As it’s an open-access journal (anyone can view the articles for free) this doesn’t necessarily mean it’s a bad thing, as long as the quality control isn’t affected. If we go to the ‘stats’ page though, we can see that they accept roughly 40% of all submissions. This is quite high considering journals like ‘Applied Linguistics’ accept only about 10% of submissions. It’s also notable that they record stats monthly and from those we can see they accept about 250 papers each month which means each volume has well over 300 articles. For context that’s about 7 years’ worth of ELTJ articles in one volume. 

Peer-reviewed
Peer-review is sometimes mistakenly thought of as the ‘end’ of the process. But actually all peer review means is that a couple of other people in the profession have read it and think it’s good enough to be published. This doesn’t mean it’s perfect, or that no one can question it, just that it has reached a certain level of acceptability. After peer-review, the academic community at large get their teeth into and then we often see criticisms, repeat studies and sometimes retractions. The International Journal of Engineering Research and Applications is actually peer-reviewed so that seems pretty reassuring. Except, considering they boast that peer-review only takes 4-6 days (not the couple of months journals usually take) and because they read about 800 papers a month and only have about 27 reviewers, they are getting through those papers at an astonishing rate! You have to worry about quality control. 

O The author 
If you’re at all in doubt you can always check out the author’s credentials. According to his website Dr. Qais Faryadi (or ‘Dr. Prince’ as his website calls him) has a PhD in computer science and a master’s in Sharia Law as well as being an expert in curriculum design, criminal law, software engineering and Islam. He lectures on all these subjects and has even published a number of books (or rather ebooks) on such diverse topics as teaching, Islam and “Magnesium The Health Restorer: The Missing Link To Recovery”. 

O The nose-test
When you actually start reading the paper, does it sound plausible or like an academic paper should? 

Dr. Prince’s starts his learning styles paper with the statement that:

This evaluation examines teaching and learning from the lenses of mind blowing scholars such as David Kolb, (1984), Honey, (1982), Dick and Carey model (1990), Anthony, Sudbury Model, VAK Model and Madeline. (Faryadi, 2012:222)

Language like this should set alarm bells off instantly. This paper isn’t going to be an unbiased review of learning styles, not when the writer believes their creators to be ‘mind blowing’. Not only this, the article is littered with errors. In this small section alone we have such oddities as:

  •        ‘Honey’ should be ‘Honey and Mumford’ (as in the reference section),
  •        He starts by talking about ‘scholars’ but then switches to ‘models’
  •        Anthony? Who he?
  •        Madeline? Who she?


Wouldn’t peer-review usually sort errors like these out? Then again, with reviewers getting through 50 or so papers a month in 4-6 days each, it’s perhaps not surprising that the quality suffers.

The article itself is a relatively unremarkable laundry list of things the author believes ‘good teachers’ should do. There is no attempt to critically engage with the various learning styles models presented or to talk about why and how they differ from each other.

So in conclusion we have sloppy education articles written by an expert in computer science and law who claims expertise in a number of other fields. They were published in science and engineering journals that either no longer exist or have very little reputation. The one journal that does exist, charges for publication and has numerous credibility issues.


The food may be great here, it really might…but I’m going to look elsewhere.