‘All models are wrong, some are useful’

There are good models and there are bad models.

For example, how many times have you pur­chased what amounted to be a garbage sack because it looked so much like a beau­tiful dress on the air­brushed model in the pic­ture online? If it’s half as many times as I have, then you know that models are not the real deal.

Whether it’s an archi­tec­tural model of the new building going up down­town, the model train you’ve got set up in your par­ents’ base­ment, or the man­nequins and air­brushed photos in our glossy mag­a­zines, models are some­times fairly inaccurate.

In con­trast, last week med­ical pro­fes­sionals trav­eled from five states (Mass., Conn., R.I., N.Y., Maine) to a health­care sim­u­la­tion work­shop at North­eastern to learn how to build models of health­care sys­tems that actu­ally are rep­re­sen­ta­tive and useful to help them improve crit­ical problems.

When it comes to a com­pu­ta­tional model, “just because it’s not real, doesn’t mean it can’t be used in real and useful ways,” said indus­trial and mechan­ical engi­neering pro­fessor James Ben­neyan, director of the Health­care Sys­tems Engi­neering Insti­tute, at a recent two-​​day work­shop for Boston area health­care sys­tems per­sonnel to teach them how to use com­puter sim­u­la­tion to improve crit­ical prob­lems. He quoted a pop­ular adage among his lot, uttered some years ago by sta­tis­ti­cian George E. P. Box: “All models are wrong, some are useful.”

The work­shop was part of Benneyan’s grant from the Cen­ters for Medicare and Med­icaid to create a national net­work of cen­ters that partner with local health­care sys­tems to help improve crit­ical prob­lems with sys­tems engi­neering methods. While I was there I met a few people from the Center for Clin­ical Excel­lence at Brigham and Women’s Hos­pital, an in-​​house con­sul­tancy group that aims to help the hos­pital estab­lish more effec­tive quality, safety, and oper­a­tions mea­sures. They told me that they often use sim­u­la­tion among their clin­ical staff to look at prac­tices and iden­tify more effi­cient approaches. But for them, sim­u­la­tion has always been of the “table-​​top” sort.

In health­care sys­tems engi­neering, prac­ti­tioners can draw from a number of tools in their arsenal. Most of the prob­lems they encounter, Ben­neyan likes to say, are pretty basic and can be solved with simple approaches. But when you get into the com­plex, weedy problems—like how to better manage of ICUs and emer­gency depart­ments, schedule pri­mary care appoint­ments, or redesign screening poli­cies for common diseases—basic approaches no longer cut the mustard.

I’d never heard of table-​​top sim­u­la­tion before the work­shop, but it’s exactly what it sounds like. You sim­u­late the real world sce­nario on the table: Think of Stannis Baratheon’s painted table in Game of Thrones or the XO’s war room map in Bat­tlestar Galac­tica. You can play out what did happen and then make changes to see how it might have hap­pened if you’d done some­thing differently.

This is all well and good until the number of vari­ables in your scenario—and their unreliability—starts rising. In the health­care world, these are often quite sig­nif­i­cant, as it deals with per­haps the least reli­able entity on the planet: humans.

“How do we design good processes given there’s so much out there we can’t con­trol?” Ben­neyan asked the work­shop par­tic­i­pants. Well, you stop playing around with the table-​​top-​​models and pull out the big guns: com­puter models.

Lou Keller, an expert in com­puter sim­u­la­tion who has been involved in sim­u­la­tion of health­care sys­tems his entire career span­ning mil­i­tary and civilian health­care sys­tems, walked the par­tic­i­pants through the FlexSim Soft­ware which he helped develop, as an example tool for these more com­plex sit­u­a­tions. The number-​​one rule, he said, is to always model what is sup­posed to happen first,” he said. “Once you know what’s sup­posed to happen then you can go looking for what it is that’s keeping it from happening.”

Health­care costs our nation $3 tril­lion each year. Ben­neyan says that one third of that could be elim­i­nated by dealing with the waste and inef­fi­cien­cies built into the system. The most common method for doing so is sim­u­la­tion. “These are really crit­ical prob­lems we are trying to help our health­care col­leagues with, so I’m thrilled with the interest and engage­ment,” said Ben­neyan. “For me, this also was a lot of fun, as it’s the first time I’ve taught with someone who was a bit of a mentor when I was starting my career.”

The mis­sion of the Health­care Sys­tems Engi­neering Insti­tute at North­eastern is to have broad national impact on health­care improve­ment and redesign through research, edu­ca­tion, and appli­ca­tions of sys­tems engi­neering methods.

 

Related Faculty: James Benneyan

Related Departments:Mechanical & Industrial Engineering