I’m taking a break from the BlendKit posts to reflect on this EDUCAUSE article: Penetrating the Fog: Analytics in Learning and Education. It caught my eye because I’ve been sitting on a hiring committee charged with finding a top-rate instructional designer to join the team working on hybrid and online courses at UC Davis. One of the questions we asked our candidates was about learning analytics–what they are and how one might use them in a hybrid or online class.
The premise of the article seems to be that the evolving world of higher education must include effective, strategic use of data and the tools we have for gathering more and better of it. The authors, Siemens and Long, do a credible job of defining learning analytics as different from simply data collection and analysis about students in higher education. As they see it, the focus of learning analytics is on understanding behavior and contexts of learners and then making changes to optimize the learning. It sounds simple stated thus, something that good instructors have always done. The difference, according to this article, is that we are now in a world of huge data sets, collected and recorded instantaneously as learners interact with digital content. Some examples they give are course-level discourse analysis of social network activity and predictive modeling based on patterns of success or failure. The authors suggest that faculty could use this data for early intervention to better support at-risk learners.
As with many conceptual and prescriptive articles, this one was interesting to think about but left me wanting more. What might this look like in a specific classroom? Better yet, what has it looked like in specific classrooms? The field of learning analytics is new enough that the empirical literature is still emerging. This leaves me reflecting only on my own experience until I can find more evidence.
In an online course I taught earlier this year, I used a fairly simple application of learning analytics. The learning management system (LMS, which at UC Davis is SmartSite) allowed me to see when specific students had logged in, which pages they had visited, how long they had spent, and where they had interacted with the site (posted on the discussion forums, etc.). Siemens and Long cite evidence that “time on task” indicators like these correlate positively with student mastery of material and with grades. In a small class like the one I taught, I primarily used these stats for quick and easy grading. In a large class, an instructor who could not personally interview every student could glance over these data charts and quickly identify students with the lowest statistics for personal intervention. In a class where students must master complex terminology, formulae, or processes, an instructor might compare such stats to scores on a midterm and see where students need more review, as indicated by many students spending a long time on certain pages of content but still getting low scores on the test items pertaining to that content.
But Siemens and Long advocate that we move beyond such basic LMS learning analytics, and my questions take me in that direction anyway. In a consultation I had last week, two faculty members described a course they might be redesigning for hybrid offering, and we discussed possible benefits and drawbacks. The class involves activities in which students reflect on ethical dilemmas in a specific field and decide what they might do and why. One of our ideas included having students grapple with case studies online and post their thoughts about the ethical implications to discussion boards. The instructors could then sample discussion threads to see whether and how students were developing in their ethical sophistication and care. This use of learning analytics is what Siemens and Long call discourse analysis.
Such discourse analysis intersects with learning analytics because the online discussions are recorded, not ephemeral like the in-class discussions that an instructor might have had on the same topic. However, it still leaves me wondering. Online tools can quickly display numerical data in charts, and they can even provide word searches as a beginning of qualitative data. But discourse analysis, ultimately, comes down to a person reading the record, deciding what it means, and deciding what to do about it. It seems to me that learning analytics are important to a course–but only if the course includes enough instructors (whether faculty and TAs) to handle the accumulation of data, who know how to use the tools and will use the data to make changes. It’s a classic challenge of research all over again.
{ 4 comments… read them below or add one }
Is it fair to say that the types of learning analytics they discussed are primarily geared towards online courses? It seems like the online course paradigm lends itself much more easily to this type of thing than do more traditional courses, but perhaps there are some learning analytics that are only available in a traditional course? You make at the end — if there aren’t enough people to go over all the data, it is likely not going to get made use of. That’s a tough problem.
Luke, you are right–the term “learning analytics” and the surrounding discussion seem to occur almost exclusively in the context of online or hybrid courses. I don’t know what kind of learning analytics are available only in a traditional course, except the art of reading student facial expressions, body language, tones of voice. That’s what many instructors say they would miss the most if they moved toward online interactions. But I don’t think I would classify that kind of data analysis in the same category as Siemens and Long’s learning analytics, because the data is ephemeral and the analysis is really the personal skill from practice. Instructors who undertake to study their own classrooms usually find that designing a data collection system for this kind of thing is difficult, and they end up focusing on written transcripts (qualitative) or records of specific events (quantitative), which is what gets collected in the online learning analytics tools after all. But it still comes down to what we do with the data, beyond just collecting it.
I have seen two great examples of what I would consider “learning analytics.” One was relatively old school and applied within the lecture hall and the other incorporated Smart site as well as “clicker” technology. The first old school style was one in which the lecture was based on powerpoint slides with no text. The lecturer successfully encouraged lively and interesting discussions throughout his course, which no doubt allowed him to collect data sets on the material and methods he used to impart information. The second method using Smart Site, gave a mixture of powerpoint and chalk talk lectures supplemented with several activities related to the lectures including visual animation of processes. The data collection from the questions prior to the activities as well as questions related to the activities were used as an indication of student participation and understanding of the lecture. An additional method to “instantly” evaluate the student understanding is the “clicker” technology, from which the lecturer can identify the effectiveness of imparting information on a particular subject. All of these techniques I find somewhat foreign in that they were not available during my undergraduate years. I am intrigued and interested in the best mix of technology within and outside the classroom.
Very interesting post! I had never thought of querying the “Learning Management System” (Smartsite) in order to assess student progress. My undergrad institution had a similar system available (albeit 5 years less technologically advanced), but it was rarely used for classes. The classes I’ve been in as a graduate student have used Smartsite in variable ways. Most often Smartsite is just used to post syllabi and resources for download, so such measures as described in the above post wouldn’t be very useful to examine. One class required weekly posting of questions before class, but I tended to write these questions offline and just paste them on Smartsite–so “time on task” would be an inaccurate statistic of my level of participation. In short, to become more useful, instructors would need to change the way they use learning management systems. (However, these questions were a more permanent/less ephemeral way of recording my statements, as noted above).
Similarly, the classes I’ve TA’d for at UCD have tended not to use Smartsite for anything other than posting lecture slides and allowing chat rooms for students. I sometimes find it useful to “troll” the chat rooms and pick out common questions/themes the students are having trouble with, and try to address it on smartsite (or at least get them going in the right direction). I guess this would be more like discourse analysis.
But you’re right: given time/resource constraints, even with an appropriate, valid, and reliable set of data on students, how do we properly analyze and interpret? For one, we should all (even non-mathy folks like me) get familiar with statistical methods and tools (go learn R!). There are entire courses and books on how to properly deal with massive amounts of data (e.g. STA 141), so maybe educators could stand to delve more deeply into such material?