Caveat Funder: The Limits of Big Data

Um, hey guys, where's the needle?

Um, hey guys, where’s the needle?

Many in the international charitable sector are grappling with the potential and the perils of big data as it applies to their work and the work of their grantees.  Measurement and evaluation of grantees has entered interesting new territory as new technologies make it easier for grantees to produce data to help them tell their their story. Big data is also helping big NGO’s like the UN use “Global Pulse” real time data feeds from people on the ground to try to observe weapon and migration flows and predict where wars might emerge in fragile states. Cool stuff.

That said, we can’t forget that data is a construct from the moment one prioritizes the units of measurement and designs the method of observation.  And that’s even before your data visualization gurus like the Tufte’s of the world get in there to make it all look pretty (yet another level of manipulating the sense we make of data).

Big data is one tool for helping us assess certain dynamics in the world, but it is not all things to all projects.

Despite the seductive lure of lots and lots of data, it’s important to keep the limitations in mind as we build the haystacks of information and look for the needles of value and meaning within them.

In the February 18 NY Times there’s a nice recap from David Brooks on some important limitations of this powerful tool, which I have taken the liberty of recapping and applying to the philanthropic sector:

  1. Data struggles with the social. It can estimate the number of people in a network of communities in the Amazon, but it can’t easily tell you who the influencers are within those communities, or how overt and covert social power structures function.
  2. Data creates bigger haystacks. The more data, the more statistical correlations we’re able to make. But as the first rule of statistics warns us — correlation does not mean causation. Just because the murder rate and ice cream purchasing volume increase in an urban center in the summer, that doesn’t mean ice cream causes murders. Misplaced causal conclusions can route precious charitable resources in the wrong direction.
  3. Big data has trouble with big problems. As David Brooks notes, “Let’s say you are trying to stimulate an economy in a recession. You don’t have an alternate society to use as a control group.” The entire Council on Foundations Family Philanthropy conference this past January was dedicated to complex systems and the levels of awareness we need to bring to those systems and global challenges. The more sophisticated our information gathering technologies become, the more educated and sophisticated we need to become as producers and consumers of that data.
  4. Data favors memes over masterpieces. Brilliant new ideas and products often don’t get picked up by the masses at first — in fact they often antagonize the masses or get blank stares. Point is — valuable new services, projects and products don’t always show up as the hot new thing on Twitter or Amazon.
  5. Data obscures values. The decisions and values embedded in the collection and interpretation of data are often obscured to the reader, funder or other end consumer of that data. It is an art, not a science, to respond to data. I had a brief conversation with an African woman who talked about data reporting in her community for Western NGO’s. They had asked elders to report back on the number of children in her community in order to figure out what level of resources were needed. The woman I spoke to, frustrated, said “But your NGO’s only got half the story — in our culture girl children are traditionally not even counted!”  There is such a large margin of error especially when  engaging in intercultural data collection. Are data collectors aware of their bias and the bias of those from whom they collect information? Are you as a funder aware of these important inflection points along the data-collecting stream?

At the end of the day, your organization may value funding one population over another, or value engaging in one type of funding process over another, even if the numbers are hard to come by or seem to point in other directions. Data is one input in a complex system, especially for global funders.  They might not make for a sexy powerpoint presentation or neat and clean pie chart, but human relationships, ethics, curiosity and intercultural values are other essential inputs for any charitable decision making process.

* * *

Data visualization sages:

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s