Monday, February 26, 2007

Data: fundamental premises

About twenty years ago, I created a slide I called "Fundamental Premises" in reaction to what I saw at the time as an excessively eager approach to data collection in a particular manufacturing environment.

I rediscovered it recently. Here are its five points:

  1. One should only take data for a specific purpose; the quantity of data necessary for maintaining historical perspective and a report card is far less than we presently take;
  2. The value of the flow of information is epsilon less than the value of the flow of products, and the same attention should be paid to making both flows simple, easy to understand, and defect-free;
  3. Nothing beats talking to people for basic communications, but limited data helps to expand the capability of people to analyze a situation;
  4. Data collection is almost never free, although the costs are often well hidden;
  5. Manual data collection may be more valuable than computerized data collection (much as we have learned that manual, Kanban-oriented shop floor control may be preferred to computerized systems); for one thing, it is arguably easier to verify the accuracy of many kinds of data when manually collected and plotted.

While the original was an unnumbered list, I've added numbers to make commenting easier.

How might I modify those premises today?

Seemingly contrary to what I wrote in points 3 and 5, I do understand that automated data collection can be valuable, and I do understand that data helps us avoid subjective biases (even as talking with people helps us avoid missing important insights). I've described elsewhere a case in which people on a production line failed to report the most common problem they saw; when the problem was pointed out to them because it was evident in recorded data, they said, "Oh, that's not a problem; it happens all the time." Triangulation is important, as is paying serious attention to the data, not just letting a computer draw a few conclusions and accepting those conclusions without further thought.

I still stand by point 2 and the related point 4. Most of the organizational systems in which we work can be understood as feedback systems, and information feedback is a key determinant of system behavior in such systems. I would suggest that system dynamics can be a tool to help determine what data is important. That data feedback necessary to make the system dynamics model work well may be just the data needed to make the real system work well.

I'd largely stand by point 1, too. It's tempting to squirrel away all the data we can take and then have it just in case we need it. The problem comes in point 4; it costs time and money to ensure we're getting the data we think we're getting. If we don't need particular data, we're tempted to not worry about its accuracy as much. Then, later, if we do decide we need it, it may be hard to determine what it really means or how accurate it really is, and we may make bad and costly decisions by relying on data we only think we have.

What are your fundamental premises regarding data and its use in organizations?

Labels: , ,


Blogger tom said...


This is an interesting post and I tend to agree with your points. It makes me think of several things, not necessarily related:

1) Data and information are not the same thing. We have too much data and not enough information.

2) Knowing what is the current condition is lost to many people. As I've said in another post, advocacy without data is just dogma. It's so easy to make statements about the way things should be, let alone the way things are, without really knowing the data about what is.

3) Triangulation is one way we know what is. When we meet with another person and that person makes several inconsistent remarks, we would probably make a judgment that that person is not reliable. That assessment is based upon triangulations. So, I totally agree that "nothing beats talking to people". That's why salespeople travel so much: to get face to face. You get to send and receive a voluminous amount of information when you are physically with someone.

4) There's a great little movie called The Fog of War: Eleven Lessons from the Life of Robert S. McNamara. McNamara makes the point that we can rarely know things perfectly. We must make decisions with imperfect information.

5) Manual vs. automated data collection. I've seen people waste a lot of time trying to automate data collection when, if they had simply focused on manual data collection, they would have spent far less time. There is a benefit in manually transposing numbers because you are forced to really think about the numbers.

Again, thanks for your thoughtful post.



28 February, 2007 05:02  
Blogger Bill Harris said...

Tom, thanks for your thoughtful comment. I'm glad you took me up on my invitation to comment, for I'm wrestling with a few potential possible paradoxes in my thinking, and your comments are valuable as I work to understand them. I'm not convinced I have to eliminate paradoxes from my thinking, for I've found that holding onto paradoxes for a while can help me synthesize new ideas.

i) Your point 1 is likely one factor at the core of the reason I originally generated my "Fundamental Premises." We need a way to make sense of it all. I recall reading a book once with no footnotes but with 150-200 references to entire books and articles in a bibliography. I found that largely useless; without a guide, I wasn't inclined to search out and read all of those articles and books to try to discern the quality of the reasoning that went into the book.

ii) I agree with your point 2 that we often don't really know the current situation. Here's where I begin to wrestle with my "Fundamental Premises." One of the problems seems to be the scarcity of broadly based data in which we can ground our thinking; newspaper articles can only provide a fraction of the useful background. Your Data360 is a useful place to get grounded in what's going on. Gene Shackman has a page of links to data sources and insights. Statistics Finland seems to be another useful data source. I seem to find those useful in grounding my thinking, and yet I'm mindful of the risks in using generically collected data without knowing a lot about the definition of the data elements and how the data were collected. I guess part of the answer is that we should both look at generic data sources such as these and be skeptical of what we're seeing, not because it's wrong but because that's the way to increased understanding, even as it can frustrate those who want us to simply "get on with it."

iii) A second problem seems to be the need for good methodologies to make sense of that data easily. Sometimes the interpretations given by others may advocate for their positions instead of being disinterested analyses. Sometimes we as individuals never learned in school (or perhaps forgot) simple statistics or other tools useful in making sense of the data. I hope I've written about some of these approaches already, and I hope to continue to write about more approaches in the future.

iv) Not all data is quantitative. Chris Argyris reminds us that we often make decisions about what to do based on what we assume others will do in response. He probes to discover whether we've collected data and tested those assumptions. That doesn't mean we should take a survey; there are qualitative forms of data and ways to collect such data. That thought may align with your point 3.

v) Your point 5 relates to statistical advice I received years ago: start by plotting the data by hand. Even if you later use automated tools, having to touch each number and each datapoint does seem to help us think more deeply. With the availability of tools such as J, I don't always do this anymore; perhaps I should do it more than I do.

vi) Finally, your point 4 is a reminder of the tension we all experience and the need to balance our needs: wrong insights immediately are likely useless for solving the problems we face, as are perfect insights delivered a millennium from now. Blending the desire for correctness and immediacy is a key part of wisdom, I think, and it varies by the task.

I'm curious: did this spark any further ideas in you, Tom? Did any of the rest of you have any ideas you'd like to share?

28 February, 2007 10:00  

Post a Comment

<< Home