Tag Archives: feedback

What’s in a Stat? Using Data to Impact Small Choices

1 Aug
salutations-chart, from blog.okcupid.com

salutations-chart, from blog.okcupid.com

In a blog post titled, ‘We Experiment on Human Beings,” OKCupid founder Christian Rudder shares how the site manipulated user profiles to gain data on what led to interactions and meaningful conversations on the site.

His post is in reaction to the ruckus over the Facebook emotional contagion study, but I thought it was much more interesting how OKCupid uses it’s data.  Yes, they mine their data for information that will help them make the site more successful (and by extension, profitable) but they also share their information on their blog.  Oktrends is a veritable gold mine of information about our habits, preferences, and our often misguided assumption about what will appeal to a potential mate (or ourselves.)

What’s awesome about their data is that they interpret it for us–so when they share their thousands of data points about what first messages gained the most traction in their post, Exactly What to Say in a First Messagethey get very specific with advice–open with “how’s it goin” or “What’s up,” but not the more formal “Hi” or “Hello.”  Express interest by using the phrases, “I was curious,” or “You mentioned…”  Contrast that with the general advice we often get, like “be casual,” or “show specific interest in the other person.”

Imagine this transferred to other fields.  Let’s take the classroom.  We give feedback to students, but it’s often a general, not-easily-applied kind of message.  For example, “Johnny is not very engaged in his reading.  He needs to focus on his books for longer.”  We have a very general piece of data here–not engaged in reading–that’s sort of equivalent to OKCupid telling users, “you’re not successful at getting dates.”  It’s accurate, but it’s describing a problem, rather than being helpful.  In fact, it’s pretty discouraging to hear.

We could get more specific with our data.  “Johnny reads for an average of 5 minutes before he finds an alternate activity, like going to the drinking fountain or sharpening a pencil.”  But we’ve really just described the problem in more detail, like saying, “people look at your profile on OKCupid an average of 8 seconds before they click away.”

We need some data for when Johnny is successful at engaging in reading to see the difference–or barring that, some data about when other students similar to Johnny are successful.  “Johnny reads for an average of 15 minutes when his book is a series with characters he knows well,” or “Johnny focuses for more than ten minutes at a time when he’s sitting in a favorite spot, facing away from other students so he’s not distracted.”  Suddenly we have some strategies for how to help Johnny, like OKCupid telling us that pictures that show activities spark more meaningful conversations on average than selfies that just focus on a smiling face.

Now imagine we give this information to Johnny, instead of just sharing with parents at conferences or keeping the knowledge tucked in our head.

Johnny, I’ve been marking when you’re reading and when you’re doing a different activity, and I noticed something interesting.  Usually when you’re reading, you read for about 5 minutes before you get distracted.  But sometimes when you read, you can focus for ten or fifteen minutes at a time!  Usually that’s when you’re reading your series books, like Animorphs or The Lightening Thief.  What do you think of that?”

Johnny can make the cognitive leap.  And now he can devise some plans for how to stay engaged more in reading.

Data is a powerful tool for noticing trends, and what works and what doesn’t, but it’s often held by a those in charge.  OKCupid has opened up some of their data to benefit their subscribers, and they’ve made that data specific, comprehensible, and useful.  Too often in education the data is vague (such as “below standard in math” or “5 on the API”) or not shared with the ultimate actors–the students.  If we are really specific about the issue (struggles in reading because doesn’t notice when a vocabulary word is unknown) then we can be very specific about solutions (repeated lessons with short texts working on identifying and attacking unknown words.)

Data can help us to identify problems, but it can also help us to identify solutions.  We can share data with our students, in the form of grades, percentages, or smiley-faces, but the more specific we are with our observations, the more our students can respond with a positive solution to the problem.


Reform Reality Check: Just Because Something Works, Doesn’t Mean We Should Do It.

17 Jun

Research in education is a funny business.  You can find support for just about anything you want–whole language or phonics, back to basics or project-based learning, charter schools or public.

John Hattie, the author of Visible Learning and the Director of the Melbourne Education Research Institute at the University of Melbourne Graduate School of Education, shows us why.  In his research into what works in education, Hattie did a synthesis of over 800 meta-analyses related to educational achievement.  He looked at over 100 interventions and found that almost all of them produced positive results.  Class size reduction from 30 to 15 students, for example, showed a positive effect size of .2, or about 9 months of learning, which is statistically significant and sounds impressive. What Hattie points out, however, is that we shouldn’t be comparing an intervention’s results to the results if we do no intervention.  We should be comparing it to the results of doing something different.  Class size reduction might sound good, but it’s actually in the bottom half of interventions that produce results.  An intervention must produce an effect size of at least .4, in Hattie’s research, to be at the midline of effectiveness.*

Very few interventions, it turns out, produce negative results.  One that does is retention– a result of -.16.  Seeing that almost no interventions negatively impact a child’s learning, the idea that politicians are actually advocating for a policy that does is unfathomable.  It’s another example of people promoting ideas in education that have very little support in research.

So what does impact achievement the most?  Hattie ranked the interventions in order of effectiveness.  A few interesting ones?  Teacher-student relationship produced an effect size of .72.  

That’s incredible.  And it makes sense.  As Rita Pierson proclaimed in her Ted Talk that went viralkids don’t learn from people they don’t like.

Other interested findings?  Feedback and formative evaluation produced an achievement effect of .73 and .9 respectively.  This also makes sense.  Both are tools used to inform future teaching and learning.  Too bad we seem mostly obsessed with summative evaluation right now, in the form of standardized end-of-year tests.  Placing all of your emphasis on summative evaluation is a little like closing the barn door after the horse is out.  If your teaching wasn’t working, it’s too late to do anything about it.

Overall, Hattie found that only five items show a negative impact on student achievement in research:  mobility, television, retention, being on welfare, and summer vacation.  Of the 95 interventions showing positive results, it’s some of the weakest that are getting the most attention right now, such as charter schools, with an effect size of only .20. (I would add the caveat that this suffers from the problem of lumping all charter schools into one category.  It’s fair to say that some probably produce a far larger effect size, while others may be much more ineffective.  But with such weak overall results, it still begs the question of why there’s so much focus on this one intervention.)

We can’t do them all–we don’t have the money, time, or expertise to invest in every positive intervention.  So the question for educators, parents, and policymakers becomes not, “What can we change that positively impacts student achievement” but “What positively impacts student achievement more than all of the other interventions we can put in place?”


*I do want to throw in a plug for class-sized reduction.  When it was implemented, class sized reduction created a sudden and immediate need for significantly more teachers, more classrooms, more curriculum (teacher’s manuals) etc, which schools often didn’t have.  The result was thousands of teachers hired on emergency credential with very little back ground in teaching or their subject matter, classes taught in closets or other inappropriate rooms, and students and teachers with no learning materials.  With all of these negative competing factors, it’s a wonder that class sized reduction showed any improvement in learning at all.

See “Visible Learning: Part 1” and “Visible Learning: Part 2” to hear Hattie explain his research more in depth.

Enjoy the Moment

6 Mar

Most of us get far more positive feedback and reinforcement than negative during the day, but somehow it’s the negative that can burrow into our brains and stay there, gleefully building a pyramid of worry and anxiety.  We get so caught up in the details (admiring the problem, I heard a colleague say yesterday) that we fail to step back and look at our situation objectively.

Students do this too, become so obsessed with one thing they want, or one thing that goes wrong, that they struggle to pull back and think, “How important is this?”  When I see children having an argument, the first thing we often discuss is, “What’s the size of this problem?” (i.e. Does it really matter if you write with the purple pen or the red pen?)  Grudgingly, they will often acceed that the problem is on the small scale rather than the master-disaster-tornado-blew-down-your-house-scale.

Wisdom comes from Spanish NBA player Ricky Rubio, consoling his teammate (who’s been having some trouble hitting his shots lately.)

I’m sure both of these guys have a lot riding on these games, but if they don’t enjoy it now, when will they?

It brings to mind the kid president video I posted earlier.  Even more amazingly, kid president (otherwise known as Robbie Novak) was born with osteogenisis imperfecta, or brittle bone disease.  But instead of worrying, he prefers to dream like he’s flying.  Good advice kid president.  Good advice.


A world of film, a house of stuff.

Literacy Changes Everything!

Teaching and Parenting as a Dedicated Reader and Writer

To Make a Prairie

A blog about reading, writing, teaching and the joys of a literate life

sunday cummins

Experience Nonfiction

Shanahan on Literacy

Literacy in Education


A meeting place for a world of reflective writers.

The Quick and the Ed

Literacy in Education

Shanker Blog


Free Technology for Teachers

Literacy in Education


Smarter Charts from Marjorie Martinelli & Kristine Mraz

%d bloggers like this: