The Arrowsmith Program claims to help students with a wide range of learning disabilities. It is offered at schools in North America, Australia, New Zealand, and Asia. The method relies heavily on “cognitive exercises” that it claims have been drawn from neuroscientfic research by its founder, Barbara Arrowsmith Young.

In this post, I describe how Arrowsmith Young’s program can serve as an excellent case study illustrating several remarkably effective techniques for boosting the credibility of a product regardless of its merit.

In the enthusiastic response to my last post about Norman Doidge’s book “The Brain That Heals Itself”, I was asked to join in on the ongoing conversation around Barbara Arrowsmith Young, who has spent the past 30 years promoting a controversial program for treating learning disabilities. Dawn Brown is a Ph.D. psychologist in my network who is very familiar with the world of psycho-educational assessments. She pointed me to the controversy over the Arrowsmith Program’s use of intensive “cognitive exercises” with children and adolescents. Dawn was no doubt alluding to similarities between Arrowsmith Young’s emphasis on neuroplasticity in her program and Doidge’s enthusiasm for methods that “rewire” people’s brains. Like my investigations into the scientific underpinnings of Doidge’s book, my research into the Arrowsmith Program took me in a direction I didn’t expect. In the end, it taught me some important lessons about how to sell a product even if it may have no scientific basis, and why we all need to think very carefully about the implications of methods we use to sell our products and services, no matter how good our intentions.

Lesson 1: Boost your credentials

If you are an expert in your field you can skip this lesson. However, if you are like most people and are trying to hype something that is out of your league, the first thing you will have to fret about is how to massage your personal credentials. First, assuming you have a college diploma or university degree, be sure to describe it. Then write some inspiring stuff about some amazing books written by well-known brainiacs that inspired your great idea or product. The pairing of your basic educational qualifications and these genius level writers will encourage most people to assume you are an expert too.

Barbara Arrowsmith Young’s degrees are in child studies and school psychology.  In her bio, she also refers to being head teacher at the University of Guelph “Lab Preschool”. Most important of all, she makes a big deal about reading a textbook by A.R. Luria, an iconic neuropsychologist, and being inspired by research by “Rosenzweig”. Everything is true. None of it actually claims she’s a bona fide neuro psychologist or researcher. Yet somehow the teacher at a preschool comes off sounding like she’s a world authority on learning disabilities.

Lesson 2: Most of your staff don’t need strong credentials either

Most businesses are trying to save money. If you are providing a service that usually requires professionals who are credentialed, registered with a college, or regulated in some other way, this can get expensive. Sometimes you can simply argue that the professionals are not required to deliver the service because you are providing something different. Uber does this to justify not hiring taxi drivers. Many e-therapy sites do this by referring to their counselors as “coaches”. If you are trying to compete with services that are usually provided by expensive professionals, like teachers, there is yet another way to save money: just include a few in your staff. Then you can boast about the quality of your service, like the Arrowsmith School in Toronto does, by suggesting that “several of our staff are accredited, teachers…”. By the way, the Meriam-Webster dictionary definition of several is “more than two, but not very many”.

Lesson 3: Distract everyone with a good story

In the world of startups, this is known as the founder’s story. It can be part of a good pitch, and in the world of startups, the pitch is everything since early stage companies don’t usually have strong traction, good revenue, or other things that investors want. Most good stories include something about why you personally are closely tied to the product (i.e., the school of hard knocks). In Barbara’s case, she has a riveting tale of her own lifelong struggle to overcome multiple learning disabilities. In fact, she’s published a book titled “The Woman Who Changed Her Brain”. Better yet, her book has an introduction by none other than Norman Doidge, and Barbara’s story is the focus of the second chapter in his own bestseller “The Brain That Changes Itself”. If my child has ADHD the sad fact is that I will value these stories more than a dozen placebo controlled double blind research studies published in leading journals, since they are a lot more readable. Besides, isn’t a book as credible as a journal article?

Lesson 4: Make it sound like rocket science

Another version of this rule is “the best lie is the one that is closest to the truth”. You cannot read anything about the Arrowsmith Program without being inundated by difficult neuropsychological terms. Does this conceal lies? Probably not. Is it the truth and just the truth? It doesn’t really matter since you just have to mention the word neuropsychology and people’s eyes start to glaze over. It immediately conjures up images of lab coats, heavy framed glasses and pocket protectors laden with pens and pencils. Throw in some words like “neuroplasticity”, “connectivity”, and “cognitive exercises” and you’ll ensure that you are talking over everyone’s heads, even if the simple translation is that the brain is changeable, has an awful lot of connections, and can learn new stuff when we practice or rehearse things. Add some graphical images of neurons and they’ll conclude your method is revolutionary. Most important of all, when you use these terms a lot in your sentences, no one but a neuroscientist will be aware that what you are saying is nonsense.

Lesson 5: Pretend that research is in any document that contains charts and numbers.

You haven’t got a stitch of sound research to back up your claims? Don’t despair. Hire some academics to write policy papers, summarize your methods, or tabulate any basic statistics you have on user behavior. Better yet, get someone to present at an academic-sounding event. They probably won’t have the content required to get chosen as a speaker, but conferences will always offer them a poster spot so long as they pay the registration fee. Posters are just that – you get to put a poster on a partition wall in a vast room teeming with other poster presenters that summarize your information, and you spend a couple of hours answering the questions of anyone who feels compassionate enough to stop to have a look.

If you have some extra time in the evenings, you can also write up a case study. This is scientific terminology for a testimonial. Lots of journals publish case studies. Yup, researchers can’t resist the opportunity to plug themselves either, and case studies are rife in the research literature despite the fact that everyone knows they prove absolutely nothing.

How does the Arrowsmith Program perform on this measure? The “Arrowsmith Program Research Study Document” available on their website provides an impressive list of “studies” under headings such as “Independent Research in Progress”, “Completed Peer Reviewed Research”, and “Other Completed Studies”. Take a closer look, and you’ll find 3 poster presentations (there are 4 listed but two are the same poster presented at different events), 1 case study that made up a graduate student’s dissertation, and a handful of reports summarizing data collected on the same students that were presented to various school boards. There is not a single published, peer reviewed study in the entire collection. If I was a teacher, I’d give this a failing grade.

Lesson 6: Ignore negative results

You won’t do well trying to argue a point that you know you will lose. It is far better to avoid the argument entirely. After all, most people want to believe you. So when it comes to evidence that your product doesn’t work, simply pretend it doesn’t exist. Good research studies are expensive to fund, so anyone trying to prove you wrong is going to take it in the pocket. It’s safe to bet they have more important things to do. The only real threat will come from well-intentioned admirers who decide to try and prove your point but fail. The Arrowsmith Program has this problem. The only result you’ll get if you key in “scholarly research Arrowsmith” is this dissertation study by Rhonda Hawkins. She followed a class of students in a private prep school using the Arrowsmith method. Her results? Teachers and students found the process terribly stressful. Most performance indicators did not change significantly over the year. English scores actually declined. The only thing that improved was students’ self-esteem, which one would expect from a course that requires hours of 1:1 support every day 1. While the study easily matches anything cited by Arrowsmith schools in their lists of research as far as rigor, there is no sign of it in any of their literature. What?

Lesson 7: Blame the system

People need an outlet for their frustration. Parents of children with learning disabilities or ADHD are often frustrated and upset. They have to watch their children struggle in the current system, and fail. Year after year they are presented with IEPs that seem to fall flat. It’s a fight all the way and feelings get hurt, despite the fact that everyone’s doing their best. It is tempting to blame the system for a problem that is far deeper in its scope. Feeding mistrust and dislike of the system exploits what psychologists call cognitive dissonance. When we are frustrated and upset, and failing to meet our goals, we have two choices. Blame our selves, or blame the system. Dissonance theory suggests that most people will blame the system in order to reduce their inner discomfort. Barbara plays into this when she describes her master’s thesis work, which demonstrated how children educated using traditional methods seemed to show no real change or improvement after years of hard work in the system. According to Barbara, traditional methods offer only “workarounds” or ways to “compensate” for our weaknesses. The theme that plays constantly in all the information provided by Arrowsmith is that the traditional educational system consistently fails children with learning disabilities. In the startup world, this is called “defining the problem”. Every good pitch begins by giving the audience a heavily skewed account of how an existing product or process is failing everyone to an epic degree. People like it when you arouse their ire over how life can shortchange them, and as every psychologist or psychotherapist knows, anger can be very motivating.

Lesson 8: Be as inclusive as possible

Making slips of the tongue? Messy handwriting? Spelling mistakes? Distractible? Forgetful? Didn’t think through the consequences of your actions? Lose your train of thought? Shy or withdrawn when you meet new people? Trouble reading people’s emotions or body language? Get tongue tied? Struggle with math? Struggle with english?

Whew! Most people who read this list will relate to at least one or two of the items. When the full list includes 19 different deficits that are based on neuropsychological testing, as is the case with the Arrowsmith Method, you are covering most of the major cognitive skills that people possess. This is brilliant! By including something for everyone in your sales pitch, you will encourage people to mistakenly conclude that you are speaking directly to their issue when you are just being inclusive. This is also how many psychics succeed, and what makes a really good horoscope. The funny thing about life is that all the problems you’ve experienced that you’re likely to remember and count as important are pretty common and predictable, even if you don’t know it.

Lesson 9: Make your customers into advocates

As anyone who is a fan of Game of Thrones knows, little sparrows can be very, very powerful and disruptive agents of change. By rewarding your admirers and treating them like disciples, you can inspire them to fight your cause. Need to boost a shout out? Want an obvious sign of traction when your company is negotiating a new contract or sale? Or if you are with Arrowsmith, want someone to heckle the local school board as you try to sell them your curriculum? Advocates will lobby for you, protest for you and write reviews for you. Best of all, they’ll do this and more for free. If you confer a more salubrious title on some of your sales staff, such as “Program Representative”, you can even get your advocates to feed prospects directly into your revenue funnel. Proponents of the Arrowsmith Program do this very effectively by providing a detailed advocacy guide. It gives parents lists of resources including websites, videos, books, and pamphlets. It coaches parents on who to approach and how to present Arrowsmith’s material. Everyone knows that the best advocate for a child is that child’s parents. Just imagine if you could harness that sort of influence for your sales team!

Lesson 10: Don’t control for group differences

I should have put this one back with the other lessons that relate to research methodology, but I wanted to save the best for last. In scientific studies, it is well established that all sorts of subject characteristics can confound even the most carefully engineered experiment. This is not just about controlling for people’s expectations to rule out the influence of positive expectations (the famous “placebo effect). People also differ in predictable ways based on things like cultural background, gender, or age – to name just a few examples. Researchers who overlook these effects do so at their peril.

So how does the Arrowsmith Program perform in this department? It’s easy to find out. Just have a closer look at that “Research Document” I mentioned in Lesson 5 above. A quick scan of the studies they’ve included suggests that the vast majority had no control groups. But it gets better. Of the couple of studies that used comparison students, the largest and most recent one carried out for the Toronto Catholic School Board by William Lancee tells all. To begin with, Lancee used unequal samples, with 30 Arrowsmith students compared to only 10 regular students. Also, the samples weren’t randomized, nor were they matched for age or gender.

To the uninitiated, these may seem like minor issues that are quickly overshadowed by the apparently startling results. Basically, the students in the Arrowsmith Program appeared to vastly outperform those in the regular program on a wide variety of standardized educational tests. Yup, they improved their scores by an average of 30-50%, whereas the comparison students showed no improvement at all. But here is where things get interesting: if you look at the scores of the Arrowsmith Program students in Table 1 (which I have taken from the pdf copy of the paper), you’ll notice that they are lower at the start of the study than those for the regular students (that’s the column under the heading “30 AP Students” in the “Pre” group. In fact, they are significantly lower than the “Pre” scores of the “10 Comparison Students” on every single test. This suggests that we are not just talking about a random lack of correspondence in our comparison groups. Instead, this indicates pretty clearly that the groups were predictably different from the start.

Based on these numbers, it looks like the children in the Arrowsmith Program were as much as 2 grades behind the children in the regular group at the start of the study. On average, Arrowsmith Program Students started at the grade 2-3 level, whereas the Comparison Students started at the 4-5 level. Given that the age range for tests like the WRAT is 5-11 years, this means that the Comparison Students started at the upper limit of the test. In other words, there wasn’t a lot of room for improvement – a common problem in research that is aptly named the ceiling effect. On the other hand, the Arrowhead Program students started closer to the bottom of the age range for the tests, where there was much more room for improvement.

Where’s Alex?

I noted earlier that Barbara Arrowsmith Young claims to have been inspired by the legendary Russian neuropsychologist Alexander Romanovich Luria when she first developed her program. What would Luria think of the result? I suspect he would be rolling in his grave. Luria was not some technician who enjoyed administering learning assessments so much that he wrote a book on the topic. He was a brilliant thinker and theorist who understood that the kinds of tests that neuropsychologists conduct provide very limited signs of what is actually going on in the brain. His real passion went far beyond learning how discrete brain functions operate. Luria wanted to understand how our collective histories, intricate social forces, and diverse cultures exert an influence on – and are influenced by – the “adaptive flexibility” of the brain 2. His theories anticipated the kind of developments in neural network research that are current in today’s deep learning and AI movements. His discoveries in the field of neuropsychology were never meant to be used to justify a reductionist, oversimplified, and just plain inaccurate rendering of the brain as a collection of discrete parts that require focused exercise and practice to “fix” problems.

So there you have it – 10 lessons on how to blur the distinction between facts and fictions, or alternately, 10 ways that any person trying to sell a product or service can lose sight of the truth. In the end, this should remind all of us that science is a method, not a product, and neuroscience is simply one application of this noble pursuit.

When used properly, it promises to help us to understand, improve, and perhaps even heal our amazing brains. When used poorly, it’s just a quick and easy way to sell ideas, regardless of their truth or merit.