Should I go to the store? This used to be an easy question. The decision could be made by asking a series of basic questions like –
Do I need an ingredient?
Am I craving something?
Am I procrastinating putting together a report for my boss?
If the answer to any of the basic questions was yes, then I chose to go to the store.
Today, however, the answer to that first, very simple question is not nearly as straightforward. Before heading to the store many of us are gathering more data than we did just a few short years ago. We may want to know –
Does Google say the store is busy?
What do the CDC guidelines say about how safe it might be?
What does the heat map of cases in the area reveal about which location could be riskiest?
We are now using more data to make a decision, and yet not finding the decision any easier. If anything, the choice may be more challenging. The conundrum that access to data does not equal easy decision-making is true whether we’re planning our weekend errands or strategizing a million-dollar business move. Why? One reason is that in order to use the data at hand, we have to be able to trust the raw data and trust the insights gleaned from them.
Now, more than ever, people have access to data and recognize the importance of good data. But how can we tell if the data we’re using are useful? How do we determine their veracity?
Unfortunately, there is no silver bullet to tell if the data we’re appraising are useful, accurate, and true. There are, however, a series of questions we can ask so that we can process data with more confidence, whether we are reading a news story or sitting in a meeting with our data analytics team in the office.
What is the motivation behind the research?
Data do not appear out of nowhere. Before approaching data, it is necessary to know the context in which the data exist, specifically “why” the data exist. It is key to understand:
- What are the political motivations behind the research, both in the traditional sense of politics and in regards to inter- and intra-company politics?
- What are the financial motivations behind the research? For instance, is the conclusion of the study being used for marketing purposes where a particular outcome would help sell a product or service?
- What are the personal motivations behind the research? Is someone’s job going to change based on the findings?
How is the research funded?
When it comes to motivations, one stands out above the rest: money. To see the information in its full context, it is necessary to know who or what organization might have a financial investment in the data gathering and insights.
- Who funded the research?
- What future funding opportunities might result from the data saying one thing over another?
Do these data pass the common sense “smell test”?
If something seems too good to be true, it probably is. Trust your intuition when it comes to appraising data.
- Does the study make a wild claim?
- Is it too good to be true? E.g. a study saying it had “real-time data from 1200 hospitals in a matter of months”
- How big is the sample? Does it claim to have an abnormally impressive amount of responses? Alternately, are the findings based on just a handful of responses?
Is there visibility into the research?
Even being able to find the answers to these questions is a consideration in and of itself. Typically, transparency about the research leads to greater trustworthiness.
- What is the description of how data were collected? Do the researchers reveal how they chose the sample, what code used in gathering and analysis, etc.?
- Are the errors and uncertainties listed? Perfect data doesn’t exist!
- Are the limitations of the data clearly stated?
- Do they outline the research process, such as how they found respondents and went about the research? With human respondents, were questions asked subjectively or quantifiably?
Do the data support the expounded data insights?
When it comes to being confident about your own conclusions, it is important to question the findings and “look past the headline.”
- Do the data support truly the conclusion? For instance, you have a sample of 100k respondents and findings say X is true of 60% of the top 3% of data points. But if only 0.02% of data points are in the top 3%, the group is such a small sample size that the implied finding becomes meaningless.
Is human bias influencing the insights?
People tend to see what they want to believe and reject what they don’t want to believe. When interpreting data, it is necessary to be aware of the ways human bias can influence those deductions.
- What do you want the data to say?
- Does the data finding challenge your worldview or the worldview of a large portion of the population?
- Are you using critical thinking or just seeing what you’ve been told to think reinforced? Company norms, “this is how it’s always been done” and various societal pressures are powerful forces that can morph interpretations.
Perhaps the most important guidance around data is to be aware that there are risks involved. Be aware that the data have limitations and act accordingly.
With the current prevalence of data in the world, it’s important to know what data you can trust and what data cannot be trusted. Having solid information provides valuable insights in order to determine which course of action is best. One way to approach data with confidence is to work with those who have the expertise, experience, and knowledge to gather, interpret, and act on data. If you are facing a data challenge, check out InfoWorks’ data solutions to see a better way to harness data.
