Thursday, November 16, 2006

System dynamics and program theory (evaluation)

I just returned from a meeting of the Program Evaluators Northwest, where seven of us related what we had observed at the recent Evaluation 2006 in Portland. I learned from lots of what I heard, but one idea really hit home.

Bill Leon of Geo Education & Research spoke about the King County Guidebook to Elements of Successful Programs. In a way, it is a meta-meta-evaluation, as it summarizes the results of meta-evaluations of effective, youth-serving programs so that local non-profits can compare their work to that of the best non-profits across the country. In practice, they found element 3, "Program Design Based on Theory and Research," to be an often-missed element, one they'll emphasize in new programs.

Program theory isn't nearly as intimidating as it sounds; it's simply your explanation of the actions you chose to achieve your goals and the rationale behind your decisions. That's awfully close to one use of system dynamics models: capturing, exploring, and exposing for review your dynamic hypotheses about why a system works as it does.

Would it help important or risky programs to include a system dynamics model as part or all of their program theory?

That's certainly not an appropriate approach for all programs. Some programs may not warrant the effort, although small models, which often provide some of the most insight, may not take that much effort to develop. Some may not operate over a long enough time horizon for many feedback loops to develop, and feedback is the raison d'ĂȘtre of system dynamics. Some may be better viewed through another systemic approach. Encouraging a singular approach to anything opens us up to missing important issues which that approach can't address.

But I can't help but wondering if system dynamics could make an important contribution in the development of solid program theories for many programs precisely because it forces one to be absolutely clear about one's theories and because it enables one to test those theories in ways few other approaches do. While the result of a system dynamics simulation isn't guaranteed to be correct, a well-done, useful model can help people see both the assumptions built into the program and the likely ramifications of those assumptions.

I'm still thinking about this one. If you're involved in program evaluation or program design, what do you think? Might this have utility? If not, why not? (I'd like to sharpen my thinking on this topic.) If so, what are the limits to its utility? What have I missed? Follow up with a comment here, or drop me an email.

By the way, there is a guidebook to the guidebook, Program Assessment and Improvement Plan: Using the Guidebook to Elements of Successful Programs, which you may find useful to make the afore-mentioned guidebook more approachable.

Labels: , ,


Post a Comment

<< Home