Heed this warning as you wrestle with data and strategies to incorporate it into your organization: dashboards are dangerous. This sounds counter intuitive to many. Dashboards evolved to sort through data for us. In fact, people create dashboards to present concise and clean summaries of amassed information. However, multiple studies indicate that dashboards in fact create blindness to large and critical trends.
Dashboards encourage a false sense of security because of their passive nature. This leads to reliance on people to view and interpret them. Attempts to mitigate this problem involve more prescriptive data views, presenting an “expected result” line or an “alarm” line. Some organizations may send a weekly summary or report to subscribers so they don’t forget to check the data. However, a subscriber must still view the information, ingest it, draw the appropriate conclusion and take the best action to influence the outcome. Viewers do sometimes open the report on the relevant week, connect the dots and take appropriate action based on dashboards and reports. However, studies prove organizations cannot rely on this approach, even if the org is lucky enough to have the subscriber open the report on the relevant week.
In 1999, Daniel Simons and Christopher Chabris ran the famous “Invisible Gorilla Test.” This test showed that people asked to focus on a task suffer from “inattentional blindness.” In the original experiment, Simons and Chabris asked people to count the number of basketball passes made within a small group. During the exercise, a person in a gorilla costume walks through the scene. In the Simons and Charbris study, they found that half the people missed the gorilla.[1] Ask yourself if your dashboard can highlight an unexpected result more prominently than the gorilla in this video.
A subsequent study showed that people aware of the 1999 test and watching for the gorilla did better recognizing the gorilla. However, people watching for the gorilla did worse than test subjects unaware of the gorilla scenario at recognizing other unexpected events. In short, increased focus for a gorilla in the video made people less cognizant of other unexpected results. Put another way, if you look for one particular unexpected result, your blindness increases for other truly unexpected activity.[2] (video)
This phenomenon of “inattentional blindness” relates to the second flaw in dashboards and how people use them. In short, people looking and focused on particular patterns will miss glaring and potentially more significant issues. Dashboards clearly fit this pattern. They get designed to present certain trends and alert for certain issues, so people read them looking for those patterns. As a result, people create blind spots in their interpretation and miss valuable information.
You may believe you can mitigate this risk by having experts look at the data, but research indicates otherwise. Studies building on the invisible gorilla paradigm demonstrate that training and expertise do not mitigate the risk. Trafton Drew extended the work of Simons and Chabris and set up a study with radiologists and how they read x-rays. In this case, x-rays had an image of a gorilla placed into the image. 83% of the radiologists failed to spot the gorilla.[3]
Dashboards certainly provide invaluable context to diagnose and understand a problem. However, the key takeaway from this post is that you should not expect dashboards to consistently alert you to unexpected but critical and glaring issues. Instead, rely on monitors and alerts to set your context, bring you to a dashboard and/or data, and then address the issue. In addition, don’t use dashboards as monitors but instead as something more akin to a Rorschach test. In a subsequent post, I’ll prescribe monitor/alert methodologies, best practices for interpreting the data, and look at research showing the benefits of unfocused data interpretation.
Related Articles:
NPR, Why Even Radiologists can Miss a Gorilla Hinding in Plain Sight