An interactive model of the two-alternative forced choice test

If you are interested in machine learning or data science, you have probably encountered the idea of a confusion matrix and its representation in 2D, the receiver operating characteristic (ROC). There are many fantastic technical descriptions available, so my purpose here is not to recapitulate. Rather, several years ago, I shared an interactive model of the two-alternative forced choice test that I made for a Girls who Code event. The idea was to teach how these two metrics are derived from and give insight onto the data. I have updated it and, now that I have a place to put it, want to share it again (for only those who have Mathematica – see below).

Apologies, you must have access to Mathematica. Change the suffix to *.nb

I have struggled to deploy this into a version that is usable by anyone (*.cdf). To date, none of my attempts have worked. I have the Enterprise version of Mathematica, so if you know how to properly deploy a CDF, please contact me. There is so much more I would like to add!

In future days, I plan to expand this model to show how the model scales to n-alternatives, but I think there are some useful insights to be gleaned from this model. For instance, I have seen explanations suggesting that ROC curves below the 1:1 line are bad (for instance, inverse of an ideal observer is the worst possible outcome). While not technically wrong, the worst outcome is generally thought of as answers along the 1:1 line in the ROC. This is referred to as “chance.” Another way of thinking about it is, if performance is always wrong, the observer can simply reverse any answer and get the “right” answer. To see this, try transposing the two curves (move the slider bars for each curve to reverse their positions).

Leave a Comment

You must be logged in to post a comment.