Question: Why are There so Many Data Points Outside of the Limits?

I got a message from a reader of my book Measures of Success: React Less, Lead Better, Improve More.

The questions were:

“I entered data from Press Ganey patient satisfaction responses (12-month running totals) and since the numbers were so close to each other, the PBC showed a bunch of data points outside the upper/lower process limits.

When I changed the limits to +- 3 sigma, this issue disappeared.  Have you run into this issue before?  Can you help me to understand the difference and its impact on analysis using the PBC?”

Read more and see my answer and analysis here:

[Updated] Reader Question: Why are There so Many Data Points Outside of the Limits?

Question: Why Not Use Other Types of Control Charts? Why Not Use Standard Deviation?

A reader from Hong Kong asks a question that has been asked by others:

“There are many types of control charts under the six sigma framework, depending on the data type (continuous/attributes). Do we need to consider it in crafting Process Behavior Charts?”

Yes, university statistics courses and Six Sigma programs teach a number of control charts:

  • When counting “defectives”
  • When counting “defects”

Process Behavior Charts are another name for the XmR Chart.

You can read chapter fourteen of Don Wheeler‘s book Making Sense of Data for more on this subject, as I cite that below.

Here is one table from that chapter that compares the chart types:

Wheeler teaches that the four charts in the bulleted list are “special cases of the X Chart, and since the XmR chart provides a universal way of placing count data on a process behavior chart,” we generally don’t need to use the other chart types.

Wheeler also writes:

“The XmR Chart gives limits that are empirical — they are based upon the variation that is present in the data (as measured by the moving ranges). The np-chart, the p-chart, the c-chart, and the u-chart all use a specific probability model to construct theoretical limits…

If you are not sure about when to use a particular probability model, then you may still use the empirical approach of the XmR chart. Remember, the objective is to take the right action, rather than to find the “right” number.”

The np-chart makes the assumption that the chance of an error, defect, etc. that’s being plotted is the SAME for each opportunity. That’s an assumption that’s unlikely to hold true in the real world. Does every patient have the exact same probability of getting a hospital-acquired infection? Probably not. So, the XmR chart might be a better choice. The c-chart makes the same assumption about constant probabilities of an event.

The p-chart makes the same assumptions about constant probabilities and requires additional work of essentially calculating different upper and lower limits for each data point (as illustrated here from this page). In my experience, this form of limits mainly serves to really confuse people. It’s easier (and arguably more valid) to use the XmR chart with their consistent limits that stay the same unless there’s a signal of a process shift.

Here is an example of a p-chart for the percentage of calls that go unanswered in different time periods:

Can we really assume that the probability of each individual call going unanswered is exactly the same? Queuing theory tells us NO.

Here is a PBC with that same data:

The PBC is easier to calculate and gives basically the same answer… without the bad assumption.

The c-chart and u-chart make also bad assumptions about the uniformity of what’s being measured and u-charts have the same varying-limits confusion as is caused by the p-charts.

Wheeler calls the XmR chart the “Swiss Army knife of control charts.” This website has a graphical flow chart for choosing the control type chart to use — it’s based on a diagram from Wheeler’s book.

When I have taught my workshop, I had one session where a Six Sigma Master Black Belt talk to me afterward and he said, basically, “My executives get lost and their eyes glaze over when I try explaining the various types of control charts. It’s great to have a single method that does the job well enough in all circumstances.”

What matters more is the thinking around our Process Behavior Charts. Can we stop from reacting to every up and down in a chart? Can we learn to filter out noise so we can find possible signals?

Wheeler also adds:

“The only instance of count data which cannot be reasonably placed on an XmR chart is that of very rare items or very rare events: data for which the average count per sample falls below 1.0.”

Wheeler addresses how to address this “chunky” data in his book.

For my readers, I address this in my book by offering a method of counting, for example, the days between rare events, like employee injuries or patient falls.

Below is the chart with an average <1:

And here is a chart of the days between infections:

That addresses the “rare events” situation for many, if not all, cases.

Question: When do we Recalculate the Limits in Process Behavior Charts?

A reader asked:

“I am wondering at what point do you recommend recalculating the average and upper and lower limits?”

Generally, we’d recalculate limits when there’s a signal that performance has shifted to fluctuate around a new average (evidence would be the “Rule 2” signal of eight consecutive data points above or below the old baseline average), like below. I shifted the limits downward starting in May 2017, the first below-average data point in a run of eight below the baseline average.

“I work in health care and have developed Process behaviour charts for many of our processes using the last 2 years of data. Do I leave the average calculation in my excel spreadsheet based on those initial 24 points? Or adjust the formula each month as I add data?”

No, it’s not a good practice to continually recalcuate each time there’s a new data point. We should only recalculate when there’s a signal / shift in performance. The average and the limits are established from a baseline timeframe of data points. If that baseline timeframe is a predictable system (meaning we have no signals in the chart), then we use the PBC to see if that predictable system is continuing or if it has changed.

Question: Why is there a Scaling Factor for the Upper and Lower Natural Process Limits?

A reader question:

“Mark, Thanks to your help, and some push from an IHI conference there is some pull in our organization for Process Behavior Charts.

As we are putting together a template and training, we are looking for the stats behind the 2.66 and 3.268 scale factors used in calculating limits.  I bought your book but it doesn’t go into detail.

Do you have a link you can share where I can find a basic explanation to share with MDs who are curious?”

My response:

Great question. The Natural Process Limits on the X-Chart are essentially plus or minus 3-sigma around the center line (average).

There is a scaling factor that makes the formula:

Natural Process Limits = Average +/- 3 * MR-bar /1.128

The 1.128 is a statistical scaling factor.

Or you can express the formula as:

Natural Process Limits = Average +/- 2.66 * MR-bar

That does not mean that the X-Chart has “plus and minus 2.66 sigma” limits. The scaling factor is used to create an estimate of the 3-sigma limits.

For the MR-Chart, the Upper Range Limit is:

Upper Range Limit = 3.268 * MR-bar

You can read Don Wheeler’s article: “Scaling Factors for Process Behavior Charts” for an in-depth discussion. You can also read more in his book Understanding Statistical Process Control.

I’ve read those and I’ve taken Wheeler’s four-day seminar where he goes into great depth and detail about these topics. For most practitioners of this method, I wouldn’t recommend worrying too much about these details. I’ll defer to Dr. Wheeler and his Ph.D. in statistics.

You can also read his articles:

 

 

Question: What Should the MR-bar be?

This question also came from a webinar viewer:

“How do you tell if a process is “controlled” or not?

Is there a sort of max MR-bar to have in terms of % of the average or something?

I mean, if the MR-bar is 100 for a process “A” with a data average of 200 and the process “B” as also MR-bar=100 but with an average of 10,000.

Process B is much more “controlled” than “A.”

So is there a rule of thumb for that also? Like MR-bar has to be less than 20% of the average or something?”

My response:

You tell if a process is “controlled” (or “predictable”) when there are no signals in either the X Chart or the MR Chart. You can have a predictable process with WIDE limits and you can have a predictable process with narrow limits.
I don’t understand what you mean by the “Max MR-bar” — that’s the voice of the process and the MR-bar is what it is. If you don’t like the width of the Natural Process Limits (as calculated from the MR-bar) then you can work to improve the system in a way that will reduce variation. Reducing the MR-bar will lead to tighter limits on the X Chart. d

Question: Should I be Re-Evaluating the Average and Limits Every Time?

This question came from a webinar viewer:

“You said that a minimum of 20 data points is a good place to start in order to have a realistic view of the process.

What I was wondering is: Do you re-evaluate the average every time a new data point enters the lot or do you set a window of time which will serve as a sort of benchmark for the rest?”

The short answer is “no.”

My longer response:

Once you’ve established the baseline, you should NOT continually recalculate the average and limits. You should really only do so when there’s been s shift in the system (like a “Rule 2” signal with eight or more consecutive data points above or below the baseline average).

An exception to this might be if I create a PBC with just six data points to start. I might continue revising the average and limits until I get to 15 data points or so, but then I’d lock that in as the baseline.