Suitability of Instruments: Risk Mitigation and Measurement Quality Assurance

“Each manufacturer shall ensure that all inspection, measuring, and test equipment, including mechanical, automated, or electronic inspection and test equipment, is suitable for its intended purposes and is capable of producing valid results.” (21 CFR 820.72).

While this quote is intended for Medical Device manufacturers, the concept behind it gets to the root of good manufacturing practices for any industry with an interest in minimizing rework, scrap, recall, and/or safety problems in order to maximize profits. And everyone likes more cash . . . well, except her (as Jimmy Fallon states in the Capital One commercial). The problem is some companies don’t make the connection that the instruments that are selected and used to quantify decisions about a process or about their product may be driving one or more of the root causes of these profit pilfering penalties.

We will cover different aspects of determining the suitability of instruments, including parameter, range, resolution, accuracy, process tolerances, Process Accuracy Ratio (PAR), Process Uncertainty Ratio (PUR), operator influence, storage/handling and other categories. You should expect to be able to formulate your own definition of Instrument Suitability so that you can compare it to your organization’s current definition or to help your organization develop a definition if it does not currently have one in place.

Introduction: Suitability Concept

In the medical device manufacturing world, as with many industries, there are regulations and guidance documents that are meant to communicate critical or important concepts that must or should be taken into consideration when manufacturing a product. Problem is, language gets in the way of transferring the ideal meaning across to the intended audience. As the talented Jimmy Page wrote and Robert Plant once sang, “…Because you know sometimes words have - two meanings…” (now you’re not going to be able to get that song out of your head, right?). Authors must take precautions in the way their writing might be interpreted, which is a good reason to have multiple editors review the work. Why? Because the other half of the communication problem is that each person in the intended audience has their own way of interpreting what the words mean. Even if there were no language errors in the original transmission, people will make different connections in their own context of what they think the message says, which results in a different message being ‘heard’ across the intended audience. Aaargh! I guess written and verbal communications simply have limitations and are not the most suitable means of communicating concepts. If we could just find a way to digitally transfer information between brains, we could eliminate the static noise level that gets in the way of clear conveyance of thought. Until such time, there is a delicate balance between an attempt to resolve this problem by over-communicating, being too verbose, and boring your audience to tears, and under-communicating, in which case you may keep the audience’s attention, but may get mixed results and possibly a large gap between where they stand with the concept vs. what you originally intended for them to perceive.

Look at me, over-communicating already. Aahhhh, but there’s a segue (‘segwā) into the topic of this paper buried in that first paragraph! The written and verbal forms of communicating are not as suitable as digital transfer between our brains if such a method were discovered. And so it goes with the topic of Measurement Quality Assurance. In some of my papers and articles, I had been using the more general term, “Measurement Assurance” until I attended the 2013 Measurement Science Conference and ran into a good friend of mine who pointed out the ‘multiple meaning’ problem that this term carries with different members of the intended audience. That friend is Scott Mimbs who recently retired from NASA and who has made it a point to insert ‘Quality’ in between those two words. This paints a slightly more verbose, but much clearer picture of the concept because Scott and other like-minded individuals want to be sure our intended audience knows we mean Measurement Assurance related to Quality . . . quality of processes which include manufacturing and testing among others and do not get this confused with the inter-laboratory comparison type of Measurement Assurance Program (MAP).

There are many components that make up Measurement Quality Assurance (MQA), whose universal goal is to ensure that the transfer of omnipotent measurement is not distorted or broken on its journey between the National Metrology Institutes (NMI) and the products being manufactured on this orbiting sphere known as the ‘third rock from the sun’, located just South of Mars. A bit verbose? Probably . . . you get the point . . . and now, hopefully, you also have a vivid picture of global Metrological traceability taking place on an inhabited planet within our solar system. So, why is MQA important? Because human beings are involved in making decisions (even if machines are involved, which can produce bias and variation errors) for the manufacturing and testing processes and along the way errors creep into those processes causing situations where we think we have good products/test results, but we really do not (concept of False Acceptance) or where we think we have bad products or test results, but we really do not (concept of False Rejection). Either of these situations is expensive and erodes profits. As long as these uncertainty components exist making it impossible to know we’re right or wrong until after the next calibration or measurement cross check or Gage R&R study that identifies errors that have been introduced (these are other components of MQA), it is in the best interest of the product and testing industries to understand all of the components of error that cause these false situations in order to keep them minimized.

Selecting the right tool for the job is one of the first and most important components of MQA. Another way of saying this is that it’s important to select an instrument that is suitable for its intended purpose or application; a close paraphrase from 21 CFR 820.72 which covers Inspection, Measuring and Test Equipment (IMTE) for the production and process controls related to medical device manufacturing. Before this was ever a Code of Federal Regulations document, it was a concept that likely existed in a number of minds which, through a series of hallway discussions, paper napkin notions, and various committee meetings, it evolved into international and federal documents (ISO 13485 also contains this concept). And, as it goes with regulations, in order to reduce liability on the author’s part (ISO committee, FDA, et.al.) as well as providing some flexibility for medical device manufacturers so that they can apply one concept to multiple processes, these documents are intentionally vague. The same can be said of ISO/IEC 17025 and ANSI/NCSL Z540.3, which are, again, intentionally vague to some degree so that it doesn’t become impossible to stay in business and still adhere to these standards. However, a commendable solution to this vagueness is one implemented for the Z540.3 document, which is accompanied by a handbook that serves as a guide to interpreting the Z540.3 document. It’s like the committee knew there would be difficulty in understanding it! How did they know? Because the members of that committee have been around the block a few times and know the troubles they have had in their past working lives in trying to interpret standards and guidance documents. Is it perfect? No – the digital brain transfer method (DBTM) is perfect. This is not. But it certainly helps to reduce the confusion.

Have you given much thought to a good process for selecting the right tool for the job? There’s no lesson like physics to teach you this concept. Ever use a wrench or screwdriver as a hammer? C’mon, you know you have! The nail (or screw, yeah you’ve used a screw like a nail before!) doesn’t go very far with each tap, does it? That’s because the hammer, a more suitable tool for the job of driving a nail, has some mass behind it and the momentum transfers the potential energy into kinetic energy as it makes contact with the fastening device (or whatever you artistically used as a ‘nail’), with a little loss due to thermal energy. Not much mass behind a screwdriver handle, is there? By the way, what were you thinking using a screwdriver as a hammer on a screw? Flip it around; you’ll get a much better result (the screwdriver, not the screw)! And that, ladies and gentlemen, is the concept of using a tool correctly once you’ve selected a suitable one; operator competence and training - yet two more components of MQA where the measurements can go awry. Hey, if it’s this easy to use simple tools and fasteners incorrectly, take a moment to imagine how easy it can be to arrive at false situations with more complex instruments and materials along with a variety of operators. That is exactly what industrial processes are up against. And with sometimes little to no operator training combined with a lack of defining Instrument Suitability within your Standard Operating Procedures (SOP), your company is leaving caution to the wind that can cause false situations to occur which are often untraceable or inconspicuous. It’s your profits – spend them how you like! Eh, who needs more bothersome cash anyway?!

Instrument Suitability

What is the first thing you must do to make any product? I know, get funding . . . I meant besides designing and funding it. You have to select the materials from which each component of your product will be manufactured. You have to select the right machines or tools to manufacture the components. For some components or subassemblies, you have to find a good supplier if you’re not going to manufacture those parts. And then you’ve got to measure all of these components to make sure they were produced correctly, whether the measurements needed are dimensional, mechanical, electrical, chemical, et.al. How will you know which instruments can make accurate measurements for these components? That, my friends, is the concept behind the term Instrument Suitability! Manufacturing requires someone to make a decision on the suitability of the materials used, the suitability of the manufacturing processes, and the suitability of IMTE that will help you determine whether or not the parts of your product (and the end product itself) met the design criteria for form, fit, and functionality. Another concept for you to think about is suitability related to a product’s life expectancy. There are a number of people with hip replacements (including me) who want assurance that the product is going to work and, hopefully, will last a lifetime because, and I think I can speak for my mechanically-assisted-ambulatory brothers and sisters, we don’t want a second or third surgical procedure due to parts wearing out. And for even more critical medical devices, like pacemakers, defibrillators, and artificial hearts, a worn out part can mean death! I don’t think I have to tell anyone in this audience that killing your customers is probably something you want to avoid.

OK, so that we’re clear on the direction that needs to be taken: Make sure ‘Suitability’ is an integral part of your selection processes, regardless of what piece of the manufacturing process your responsibilities include. Now let’s focus on the measurement quality assurance piece: IMTE Suitability. In this example, we’re selecting a multimeter that will be used to make some voltage and resistance measurements.

One of the many parameters monitored during a transfer process between the space vehicle (can’t say Space Shuttle any longer ☹) and the International Space Station (ISS) is temperature; heaters are used to prevent ice buildup on the modules and also in the astronauts’ spacesuits because they are also exposed to extreme temperatures while working outside of the space vehicle. The heaters that are built into the gloves and the helmet camera of a spacesuit are powered by a battery pack which provides 12.5 ±1.5 volts. The helmet camera heater requires 12 V ± 1 V while the glove heaters require a 9 V ±0.5 V source. The voltage from the battery pack is reduced and regulated by the In-Line Cable Voltage Regulator (ILCVR) to provide the correct voltage to each subsystem. The glove heaters have a resistance of 57.7 ±2.9 ohms.

Determine Initital List of Instruments for Consideration

Now we need to select an instrument that is suitable to make these measurements. Which measurements, you say? Voltage and resistance. Specifically, voltage at 12.5 V, 12 V, and 9 V, and resistance at 57.7 ohms. In determining suitability, a logical step-wise approach is recommended to make sure you haven’t missed anything in selecting the right tool for the job. While for some examples that you come across this may seem simple-minded and a waste of time, when you or someone else in your organization start taking shortcuts, the lines become blurred as to which situations are ‘simple’ and which ones require you to take a little more time thinking it through. In reality, if you have a solid process that you use consistently every time, it will be more difficult to make an ‘honest mistake’ that wastes time or costs your company money or, worst case, hurts someone, maybe even you! The FDA wants to see documented processes for this and other reasons, but I like to think it’s mainly for this common-sense reason.

Parameter Verification

The first step is to determine all the parameters that need to be measured. In this case, there is voltage and resistance. What if you need to measure these two parameters plus the intensity of a light source, the temperature of a heat source, the rotational speed (RPM) of a part, and the viscosity of the oil that lubricates the part? My point: Just how many measurements are you going to try to group together into one instrument before you recognize that you need to select multiple instruments? I’m half joking here because there are some instruments out there that seem to have everything but the kitchen sink built into them! And I have seen the OEM say they can calibrate the entire instrument for $99! A red flag should go up in your mind as to the quality of such a calibration (and possibly the quality of such an instrument). Seriously, don’t make it your ambition to find an all-in-one instrument. As for the example, obviously, a multimeter has the voltage and resistance functions to handle these two measurements. But which multimeter? Do you need an Agilent 3458A 8.5 Digit Multimeter or will a Centech 37772 Multimeter do the trick? The 3458A costs $9,100 while the 37772 is only $19.99; obviously, the price of the instrument will have an impact on your decision. But that must not influence the Suitability answer. Let’s throw a Fluke 87V and an Agilent 34401A multimeter into the mix as well for consideration in determining Suitability. Here’s the breakdown of these instrument’s respective performance specifications for the two parameters of interest:

Range Suitability Verification

The voltages we’re interested in measuring for this application are in the 9V to 12.5 V range. If we are considering the instrument for other applications, the measurements for those other applications would need to be compared to the specifications to make sure the instrument has a range that will cover all intended applications.

Regarding the voltage measurements, the 37772 has a 20 V range that covers all of the voltages of interest in this application. The Fluke 87 has a range of 60V that will cover all of these measurements. The Agilent 34401A has a 10 V range that would be used for the 9V measurement and a 100V range for the other voltages. Same goes for the 3458A model.

For resistance, the lowest resistance range on each of these models would cover the 57.7 ohm measurement; that is all but the 3458A in which case it’s second-lowest range would be used for this measurement.

So the good news is that, for the Range requirement, all four models can physically make the measurements that we need to take. Had one or more of these instruments failed to provide a suitable range for the measurement, those models would have been knocked off of the list.

Accuracy & Resolution Suitability Verification

Once we have selected a variety of models that will cover the range, the next step is to look at the accuracy and resolution of each instrument. Then we’ll exclude those that aren’t accurate enough.

Starting first with the voltage measurements, the accuracy for 37772, 20V range is ± (0.5% of rdg + 1 digit). The resolution column indicates that 1 digit = 10 mV. So, the tolerance for the 12.5 V measurement is: ± (0.5% x 12.5 V + 10 mV) = ±0.0725 V. See the table below for the determination of the tolerance limits for each model:

What is "Accurate Enough"?

Now it’s time to determine which of these instruments is accurate enough and which are not. We’ve gone through some moderate level math here; if that gave you a headache, be glad you’re not an engineer who is responsible for determining the Suitability of an instrument for a manufacturing process. If you are an engineer with a headache, then perhaps you would be interested in a class on Suitability (in the works for the near future) because, yes indeed – this is what it takes to determine suitability!

We need to make a comparison between the tolerance of the production process where the instrument will be used and the tolerance of the instrument for each reading. That’s why we went to the trouble of calculating tolerances for each of these measurements. That also means you have to know or be able to find the tolerances of the process. If you don’t have those, then you should stop at this point and go find them or, if they haven’t been determined, work on developing them. For this example, we have all of the process tolerances provided in the description of the measurements above. The way to make this comparison of accuracy between the Process and the IMTE being considered is using a ratio, specifically a Process Accuracy Ratio (PAR). While the term ‘accuracy’ is used, the ratio must be in like units of measure, so each accuracy statement must first be converted to a tolerance, as we have just done.

Here are the PAR calculations:

The formula for PAR is based on the formula for Test Accuracy Ratio (TAR) and Test Uncertainty Ratio (TUR) taken from the verbiage in NCSLI RP-9, section 8.2. The reason for using this formula is to set up the correct algorithm when applying this concept to unilateral tolerances (i.e., positive tolerance only or negative tolerance only) or asymmetrical tolerances (as found in the federal GGG spec for gage blocks, et.al.). Using a ratio like the PAR allows us to look across measurement values for different instruments to see how they compare. It is obvious that the Agilent models have extremely high Process Accuracy Ratios for these measurements. Given their purchase prices of roughly $1100 and $9100 respectively, these appear to be overkill for this measurement application. Their ongoing calibration costs are also high, relative to the handheld multimeters in the first two columns. So, it’s good that we can use a mechanism like PAR to put the decision making process on a level playing field in order to more easily make comparisons, but when is the PAR too low, indicating that the IMTE is not Suitable for the Intended Purpose? The rule of thumb has been a ratio of 4:1 (read as “4 to 1”) or better. Check your policy on the acceptable ratio, as this may change based on the parameter being measured or based on the particular manufacturing process.

In this example, all measurements meet or exceed a 4:1 PAR, except for the resistance measurement using the Centech 37772 multimeter. All of the other measurements appear to be suitable for their respective measurement applications. You may or may not decide to rule out the 37772 for this application. This is where a comparison of costs comes into the decision making process. Clearly, the Fluke 87V is more suitable for the resistance measurement than the Centech 37772. The Fluke costs about $400 while the Centech costs about $20. Again, you must consider whether this instrument will be solely dedicated to only this measurement application, for all time, forever and ever, infinity . . . or do you want to be able to use the selected instrument for other applications, either now or in the future? This could influence your decision in weighing measurement risk (less than 4:1 PAR for resistance, but better than 4:1 PAR for voltage) against cost. Take all of the variables into account before deciding. And, if in doubt, go with the lower measurement risk (i.e., the Fluke 87V in this example). The extra 380 bucks spent here may indeed save you thousands in rework, scrap, or recall later on as well as buying the flexibility of using one instrument across multiple processes!

Going a step further to include other errors that exist in making measurements in a manufacturing process gets into the concept of measurement uncertainty and changes the PAR to a more inclusive and realistic Process Uncertainty Ratio (PUR). Details on PUR can be found in another of this trilogy of papers entitled, “Process Accuracy vs. Process Uncertainty” by Jeremy Sims.

Conclusion

Many of the measurements made in the U.S. Space Program are similar to or the same as measurements made every day in manufacturing and testing processes across many industries. Measurement quality assurance is not just about the calibration of your instruments. It is equally about how you use these instruments to make decisions about the quality of a process or the quality of your products. Measurement quality assurance is about each individual who makes a decision regarding a step in the process of making your product(s) and how each person perceives the information that helps them quantify that decision. It’s really about making sure you are certain about your decision before moving on to the next step. Therefore, it is part of risk management. The more certain you are about a decision, the lower your risk in making the wrong decision (perhaps without even knowing it) and the better your odds of making a product right the first time. How does the old saying go . . . measure once, cut twice? No, that’s not right. Cutting twice means Rework or Scrapped Parts, and that is expensive. Inspecting the parts that your suppliers make is expensive. FDA or other regulatory audit findings are expensive. And, heaven forbid, recall of your products are not only expensive but can financially ruin your company. Let’s make sure, collectively, that you keep your company far from those dangerous areas. Measure twice, cut once. Better yet, measure right, cut once! To gain confidence in our measurements, we can start by understanding our role in Measurement Quality Assurance.

About the Author

Howard Zion is the Director of Service Application Engineering for Transcat, Inc. He holds a B.S. in Engineering Technology and a M.S. in Industrial Engineering & Management Systems from the University of Central Florida.

Howard has collected a wealth of knowledge in many disciplines during the span of 30 years in Metrology, and has been employed with:

• The United States Air Force (Strategic Air Command - PMEL)
• Lockheed Martin (Electronics and Missile Systems – Calibration Labs)
• NASA-Kennedy Space Center (Standards & Calibration Laboratories)
• Philips Electronics (Broadband Networks – Metrology/Test Engineering)
• Transcat, Inc. (B2B Resource for Industrial Manufacturing Support: Accredited Calibration, Instrument Repair, 3D Metrology/Inspection, Instrument Sales, Validation, Compliance, Remediation, and Consulting Services)