Everyone wants to compare themselves to Netflix, whose data-driven, personally tailored movie suggestions improve customer satisfaction and retention. Among the latest domains to see this trend: “learning analytics” in higher education. The basic idea is to use institutional data to help students successfully navigate towards their college degrees. Doesn’t sound controversial yet – data-driven decision making is usually just plain common sense.
But the details can get a little tricky. Consider the following effort out of Austin Peay State University in Clarksville TN. What caught our attention was that in addition to sensible suggestions for ways to meet course requirements en route to graduation, the system also predicts the grades students are expected to receive in their upcoming classes. The author of the article is impressed with the accuracy of prediction, saying that end of semester GPA was predicted within 0.02 on the 4 point scale, and that individual class grades were predicted within .6. Furthermore, the probability of getting a C or better was predicted with 90% accuracy.
Seems like one of those numbers is very likely wrong, and the others are not that impressive given a little thought. There is no way that end of semester GPA is predicted to within 0.02 unless previous GPA is used as part of the prediction, in which case the weighted sum might not move much. To predict individual class grades, keep in mind that offering a certainty of +/- .6 means offering a range as wide as A to B– . A prediction interval that wide, centered on the course average from previous semesters, will cover most of the students in the class. Add in a tweak for the student’s own prior GPA and voila, good prediction. There’s not much magic required to achieve that level of accuracy and I’m sure any university could replicate it in a heartbeat.
But would that be wise? Injecting grade predictions into the student decision making process is likely to affect choices. Oh, of course the student will decide which courses to take based on interests, goals, sense of personal identity, and other thoughtful and reflective criteria. But like all rational agents they will also want to maximize their utility and it would be hard to ignore differences in expected reward (grade) “all else being equal”. That kind of feedback could easily operate as a market force changing the distribution of supply and demand. And instructors might begin to feel pressure to make their classes easier if they see students voting with their feet.
Isn’t it already good practice to notify a student during a course that their performance on assignments and tests has them at risk for failure? Sure, but such feedback is focused on those who are at risk for facing the consequences (financial and otherwise) of failure, and is based on evaluation of effort and achievement in the course. Whatever the predicted outcome for a student before a course begins, a grade must still be earned. Netflix predicts how someone will rate a future movie experience based on how they rated previous experiences. But a grade (or salary or other reward) must be earned, and the prediction accuracy rests entirely on the assumption that future effort will resemble past effort. A college student makes a sequence of choices - they choose which classes to take and then they choose whether to expend the necessary effort to succeed.
And here’s where it is unclear how much an individual’s course selection should be driven by their own prior success, and by comparison with the average successes of others. Should we tell people their actual likelihood of overcoming difficult challenges? The smart money will always bet against anyone trying to lose weight or quit smoking. Should predicted failure be emphasized up front? Maybe, as it’s certainly not irrelevant to have full knowledge. But what about the risk of discouraging effort, and of enabling complacency? For all the virtues of helping college students avoid getting themselves into classes that are too difficult, there is considerable risk for driving a market for classes that are too easy. It sounds great to make data-driven decisions, but sometimes introducing data into a system can create unintended feedback dynamics.
Here's some other suggestions for so-called “learning analytics” in higher ed: we could apply “learning analytics” to actual learning, and guide instruction and assessment within classes in a tailored and evidence based manner. Or we could build a course selection system with a much broader view. If it’s fair game to predict grades in upcoming courses, how about also giving hard data on the career success of students who have taken those courses? That would extend the decision making horizon and add more context to the process. And how about also expanding a course selection system to the broader higher education market, including online providers? Imagine an advising system that suggested a dozen alternatives for meeting a linear algebra or intro stats requirement, each much cheaper than the cost of the resident instruction class at the student’s current school. That would be a disruptive technological innovation.
