Understanding Analytics

So I’ve been reading alot of posts lately trying to explain the importance of Tin Can or Experience API from folks doing their best to simplify the opportunity as well as the posts condemning Tin Can prior to ever having used it. In reading the posts I get the overwhelming sensation that ‘data’ is not one of those things that is easily understood by most instructional designers. I’ve spoken to people like Patti Shank and Julie Dirksen who know a whole lot more about research science than I do, discussing the need to educate L&D about data. The data you may want to gather about users and usage of instructional content is no different than data that researchers want to collect during scientific inquiry. Its no surprise therefore that I find a connection between instructional designers not thinking critically of research and missing the opportunity and challenges of Tin Can.

Here’s an example. I recently read this about Tin Can (This is not meant to criticize):

Ever since I first heard about the Tin Can API, I launched a full-fledged quest to learn and understand as much as I could about this development which was hailed by many in the elearning fraternity as a game-changer.

That’s primarily because it has the potential to change the paradigms of workplace learning by tracking and recording learner data outside the traditional elearning environment.

Although its true that the Tin Can spec can allow you to ‘track learner data outside the traditional elearning environment’ (whatever a traditional eLearning environment is) this is not its main value proposition or how the spec can potentially ‘change the paradigms of workplace learning’ as implied. Tin Can doesn’t track ‘learner data’ (It can store data related to a specific actor). Tin Can sets out specifications for tracking online interactions performed by a specific actor. The data captured is agnostic and is not ‘learner data’, ‘learning data’, etc. The relationship between any one data point, or their collection thereof that signals ‘learning’ is happening or has happened is not defined by Tin Can at all, but rather leaves it up to us (the L&D professional) to define. You see, the potential to change the workplace learning paradigm is not inherently built into Tin Can. Tin Can won’t change anything. What Tin Can does that previous specifications don’t do is place the onus of change into the hands of the designers (us). Thats how its a game changer. Tin Can is not an evolution of SCORM. The two accomplish two very different things, but what SCORM and previous specifications did was implement existing tracking paradigms into the ‘packaging’ of the specification. In other words, as a designer you were stuck delivering instructional content using a classroom tracking paradigm (even though it was ‘e’). All SCORM did was make the classroom interoperable. Tin Can on the other hand has no default data points. It has no default data streams and has nothing to say about what to track. It simply says if you want to capture an online interaction, do it like this and store it here.

Aside: Yes, you can track offline activities as long as their proxied through online interactions.

To reiterate and reaffirm – the potential to change workplace learning rests with the designer and not Tin Can. Tin Can gives us a defined toolset to collect data resulting from designs that do not follow protocol. Its greatest asset is that it doesn’t provide us with tracking (take a moment here and re-read this)). It allows us to create our own. That being said, we now need to understand or work with data science. Leveraging Tin Can to its potential means being able to set up experiments (experiences) that collect data. How that data is interpreted and used depends on the assumptions inherent in the experiment (experience), the quality of the experiment (experience) and the sort of conclusion we want to draw.

To complicate things, not all data is to be used for reporting sake and the experiences we setup need to consider using data for reasons other than reporting. Some data is used purely as a system feedback loop and some data is meant to remain invisible but feed into a sort of ‘analytic’. Examples: Recommendation engines use user data to provide recommendations to you, but do not expose the raw data from which recommendations are made. Thats data being used as a feedback loop within a system. An example of data feeding into an analytic, imagine I wanted to show you something called the ‘engagement index’ (a construct I created just now). This index is a construct based on how much time someone spends on a web page, choices for viewing videos versus reading text, choices to engage in a game or read text, and viewing all material. All the separate data points (how much time someone spends on a web page, choices for viewing videos versus reading text, choices to engage in a game or read text, and viewing all material) are never shown to anybody but rather become part of a calculation or analytic for displaying an ‘engagement index’. That calculation and algorithms need to be defined up front and content needs to be designed and structured to create that data.

So you see this isn’t about reporting learner data. There is a purposeful construct that needs to happen to make data valuable, reliable and significant. Setting up your construct begins with a series of assumptions. When you read a research report, understanding what the assumptions of the researcher are is extremely important. A classic example of assumptions made in ‘learning research’ is the relationship between retention and ‘learning’. Lots of data about how different interventions affect both short and long term retention and thats great especially if you think of ‘learning’ as retention. But what if you don’t? Thats why assumptions are critical. Next is setting up the experiment, or in the case of an instructional intervention, the experience,. An experiment (experience) must be designed to feed the data you are looking for. If done well, other than an explicit relationship between experience and data gathering tools, data gathering tools can be set up to collect data that you may not be looking for expressly but may influence conclusions reached. Again, there must be an explicit relationship between the experience thats set up and the data you wish to gather first and foremost. Lastly you have to figure out the types of conclusions you want to reach. To some degree your conclusions should include affirmation of your assumptions, but also the implications for those conclusions. In the world of experience design, this means setting up an experience that collects data that tells a story about user interactions. What that story is, is your story.

Its hard not writing more about this topic. I really believe that our educational systems and our corporate universities could all use a reset with a focus from the start about what experiments do we want to set up. What assumptions do we want to make? How do we set up our institutions so that we’re getting the right data feeds? How do we make these systems ‘intelligent’ so that they can self evolve? What conclusions do we want to be able to draw? If we ask those questions of ourselves and our institutions, I’m not sure we would end up with what we have. I guess for me thats the potential of Tin Can and Big Data in general.

This entry was posted in Uncategorized and tagged , , , . Bookmark the permalink.

One Response to "Understanding Analytics"

Leave a reply