Weibull’s Wobble……………………..

I once wrote a maintenance recommendation for a DOE project meant for short term (less than 1 year) equipment operational requirements. The equipment types are a moot point for this article but if you must know; there were pumps and fans and filters and air handlers and hydraulic lifts and lift tables and electrical power transformers and electrical distribution panels and compressors and vacuum pumps and switches and relays and chillers and VFD’s and PLC’s and pressure switches and diesels and pipes and valves and a partridge in a pear tree…………. Yep everything you would expect to see in any typical production environment, temporary or not.

The scope of the project was to remove sludge at the bottom of expended nuclear fuel storage pools; it was a nasty job to say the least. Nuclear fuel had been in the pools for over 30 years, the material used to hold the fuel rods together had corroded and some of the rods had broken apart. The remaining sludge was a mixture of basalt dust, corrosion products and nuclear material from the busted fuel rods. To say the least, it was a very toxic and erosive material.

You should know that the project was SAFELY completed. However, schedule slip was measured in years and cost overruns were in the millions. Now it wasn’t that the project was poorly planned or that the engineering concepts and controls used to remove the material safely were compromised, the biggest problem, the one that the taxpayers ended up paying for was program management’s assumption that they could complete all operations with zero equipment redundancy and zero maintenance. It was a temporary installation after all and all work could be completed before the first oil change was required; WOW!!! What an assumption. And as a Monday morning quarterback; it turned out to be a really bad assumption…………and we had told them so.

Back to that maintenance recommendation I had written so long ago. So prior to getting this project underway, the contractor that DOE hired to do the job hosted a small business workshop that advertised they were looking for new and innovative contributions from the small business community that could support the goals of the project; Safe removal and interim storage of the sludge. I attended that workshop and since I am a maintenance ‘guy’ that ran a maintenance and reliability management company, I wanted to contribute my recommendations, and of course I wanted to include a plug my company’s expertise in these matters; it was a great sales opportunity. Not only that, maintenance engineers, planners and procedure developers that were employed by me (my company) had been instrumental in completing the nuclear fuel removal; my company had hands on experience managing similar and some instances, the same equipment. We had a developed list of maintenance lessons learned that could be leveraged into a very comprehensive maintenance strategy at minimal cost with maximum gain. WOW!!! A perfect storm; maybe. 

I wish I had a copy of that recommendation now, it was actually quite good. What I do remember introducing to that client was a highlight of Preventive Maintenance (PM) and Condition Based Maintenance (CBM, PdM) using simplified Reliability Centered Maintenance (RCM) tools, a FMEA plus our lessons learned, to establish the most economical methods in managing the short term operational schedule without equipment redundancy. Just a refresher here; a FMEA is a Failure Mode and Effects Analysis and based on the basic equipment types they had, it would be pretty easy complete, really just a cursory analysis and it would pay for itself. I based my case on the Random Nature of equipment failures and the Predicable Nature of consumable ‘end of life’ determination that are generically portrayed in the various Weibull distributions. So before the ink dried on my super-duper award winning expose on the magic of Managing Reliability, I set up a meeting with the client and charged off to impress upon them the importance of some form of routine maintenance, even on a project with short term operation goals.

The actual meeting went pretty smooth; the presentation was met with the usual chin stroking and polite nods of understanding as I hit each of the presentation points. In fact there were several comments from the project engineer, his deputy and the procurement representative that sounded quite positive; they were in, they understood. Then just as I wrapped up and had asked for any final questions, the project manager looks me square in the eye and said “the defined operational period (project scheduled) of the equipment is less than one year, based on our own internal risk assessment we have decided that no routine maintenance or inspection surveillances will be performed as a cost saving benefit to the project”. I was in awe, the project engineer buried his face on the table and as I left the meeting room, the procurement representative said thank you, great presentation. No Sale on the front end of this project. I was disappointed to say the least.

So let’s take a look at my recommendation(s) from a strictly probability of  failure (P_f) point of view and let you decide if some form of routine maintenance (PM, CBM, PdM) would be required.

And the story goes……………..

As part of the presentation, on a white board I threw up the 6 basic Weibull representations of equipment reliability. These were originally presented by Nolan in Heap in their study of commercial aircraft reliability in 1978. These basic reliability profiles were again verified as accurate by the US Navy in 1984(?). The only difference between the Navy and the commercial aircraft study was the percent occurrences. So what are the 6 curves and what do they represent?

We are generally most familiar with the ‘Bathtub Curve’, Pattern1. This is a generalized Weibull representation of a piece of equipment from commissioning through its operational phase to its end of life. The ‘Ski Slope’ on the front of the curve represents a higher P_f based on some run in period and then flattens out to a random P_f over the majority of the equipment’s life. And at or near the end of the equipment’s useful life the P_f creeps up again. What you need to understand that this curve is only representative of approximately 3 to 4% of failure profiles.   

Pattern 2, 4, 5 and 6 are similar variations of Pattern 1. What should be noted in these patterns, like Pattern 1, is that the P_f ‘flat lines’ over a majority of equipment life, e.g. Random Failure. In fact, Random Failures account for 80 to 85% of all failures. WOW!!! One other thing that should be noted about Pattern 6; this is the predominant failure pattern for equipment. It is representative of 29 to 60% of all failure profiles. Also note that this profile has a ‘Ski Slop’ on the front end just like Pattern 1; the P_f is higher early in equipment life, the ‘run in’ period. We will discuss this later because ‘run in’ isn’t the real issue here. It’s actually Introduction of Error. Say What????

Now Pattern 3 is a bit different, it is representative of consumable components. As time in service increases, P_f slopes up. Rolling element bearings, oil, drive belts, IGBT’s (power transistors) in VFD’s and SCR’s, filters, tires, gasket material in certain environments, UV exposed paints and coatings, etc… fall under this Weibull profile. If we look at an actual Weibull plot for bearings (to the right) it has a very predictable linier profile. I want to repeat myself here, Pattern 3’s focus is at the consumable component level and related to equipment only in the fact that if the component fails, it drags down the equipment with it.

Side Bar Comment: Remember the discussion in a previous article, Read the Fricking Instructions, the term Reductionism. And if you actually go back and review the definition I presented you will see I included the following statement; We Analyze for Failure and Evaluate Service Life at this level; the component level.

Since Pattern 6 is the predominant equipment failure pattern and Pattern 3 is representative of consumable components; it only makes sense that we manage our equipment based on these representations.

First and foremost; just based on the 6 variations of failure probability, failures are going to happen. If it rolls, slides, conducts electricity, is painted, uses oil, or goes bump in the night, if you are not thinking about failure prevention (mitigation) and failure detection, you will have setbacks. In this gamble, the odds are stacked in favor of the house. And if you have a process where there is no redundancy built into your design, it pretty much becomes mandatory that you develop a planned preventive (PM) and condition based (PdM) maintenance strategy. This is the only way you can ensure a reasonable level of reliability for your process.

Number 2; As I mentioned earlier, Pattern 1 and 6 have a ‘Ski Slope’ on the front end of the curve; an increased P_f. You generally hear that this increased P_f  is caused by ‘run in’, or maybe ‘burn in’ failures or maybe you’ve heard of it as ‘infant mortality’. The real reason; Introduction of Error. Whenever I manufacture, install and commission a piece of equipment, I increase the potential for the Introduction of Error. Whenever I disassemble a piece of equipment to perform a maintenance action, even something as simple as removing a guard or cover to gain access, I increase the potential for the Introduction of Error. During manufacture; I missed the occlusion in the casting, I missed the crack in the weld, I missed the flaw in the circuit board, I missed the material stratification in the transistor, getting the O-ring seated correctly…………… During installation; I mounted the components to an untested structure, grouting the base I left a hollow spot, the soil compaction was wrong for the installation, the space was not air conditioned, we did not re-align the coupling or the belts, the resilient mounts were installed incorrectly, power was wired to the common ground, we wired all of our single phase components to a single phase of the 3 phase distribution causing a voltage unbalance…………During commissioning; I ran the compressor for several minutes without oil, I ran the unit backwards for a while until I swapped leads, we operated the diesel for several minutes without cooling water…………..During Maintenance; I forgot to refill it with oil, I couldn’t get all of the bolts back in, the shaft was a little scared and the bearing fit was a bit loose, I couldn’t reach some of the fasteners to tighten them……….. Each and every one of these scenarios plays out as an Introduction of Error that increases the probability of failure during initial operation. And, initial operation could be measured in months and years, not just hours, adding to item 1 above, random failure.

What are your thoughts………….

So what do you think? Based solely on this discussion, should there have been a maintenance plan in place for the project, a maintenance strategy, some spare parts, maybe just a few simple PdM tasks? If you are or ever have been part of an operations or maintenance crew, I’m sure you would agree, the project manager’s assumption was flat wrong. Yes indeed, there should have been some defined routine maintenance process to support this project.

So did I profit from the several failures that plagued the project? Yes I did. After several pump failures my company had a full time engineer out there with one of my vibration meters providing more or less manual continuous monitoring. And based on what I remember, a good portion of those pump failures were due to equipment design and commissioning (or lack thereof) issues, e.g. Introduction of Error. To the U.S. taxpayer, I apologize that I could not make a stronger case.

My recommendation to you; include a maintenance plan, a strategy and budget for it. Failures are gonna’ happen.

Maintenance, what a Concept!!!!!

MMJennings

Read the Fricking Instructions…………….

One of my pet peeves is that people have a tendency NOT to read the Instructions. I suppose everyone is an expert; maybe. I think more of this has to do with the overload of information we receive; have you ever heard of the term ‘Aliteracy’?

Aliteracy

Aliteracy is the phenomenon of people being able to read, but choosing not to. In the workplace, aliteracy combined with normal job pressures and distractions results in people who simply won’t read text-heavy instructions. Yet, industry continues to produce a flood of conventional text-heavy technical instructions to manufacture, sell, install and maintain products and services of every description. Instead of reading these cumbersome instructions, many users will guess or start a trial and error solution. The result is a costly loss of efficiency, accuracy, quality and productivity. Not to mention the introduction of error and resulting asset failures when management, operation and maintenance instructions are not followed.

I’m not really sure where or when I ran into this definition. However, it was from the internet and in the days of dial up modem, so the concept has been around for a while. The emboldened text is something I added. When you don’t read the Fricking Instructions, you have a 99.99% chance of introducing error during operation and maintenance of the facility assets; whenever you touch or you ‘interface’ with that asset. WOW!!!! 99.99%!!!!

Even myself, as much as I pontificate on this subject in my training courses, I sometimes find reading instructions a pain in the butt prior to starting the task at hand. And I have to continually remind myself to Read the Fricking Instructions. And why do I place so much emphasis on the instructions? It is the most economical method in preventing the introduction of error. It is the most economical method to ensure consistency of performance. Preventing introduction of error and managing consistency of performance are the best tools we have for managing the probability of asset failure (P_f). The three human performance evils that we try to overcome by reading the instructions: Errors of Knowledge….Errors of Performance….Errors of Intent. And how do we generally manage these? A new procedure is issued, required reading is established, self-study becomes mandatory – Read the Fricking Instructions becomes the mantra for success.

Now my Human Performance Improvement (HPI) consulting counterparts may argue that management is at fault for not providing training and not glad handing or kissing your ass to have you do a job you are paid to do, and do it error free. And to a certain degree they are correct. However, if you have any pride, any ambition, any loyalty to the man that gives you dollars for the work you perform, you will do the best job you can and do it error free. And that means self-study, that means self-qualification, that means Read the Fricking Instructions. To become technically competent at any task you need to study and study hard.

For example, below is a list of required documentation for equipment procured for a new construction project. This is an old, old Fluor Global Services document that was borrowed many moons ago but it is still relevant.

VENDOR DATA REQUIREMENTS		REQUISITIONING ENGINEER		REV.	DATE
		ENGINEERS SIGNATURE
ITEM NO.:		RFQ/PO  NUMBER:
ITEM:		PROJECT NUMBER:
DOCUMENTS REQUIRED		REVIEW SUBMITTAL		CERTIFIED SUBMITTAL
		QTY.	DATE DUE	QTY.	DATE DUE
1.	PRELIMINARY GENERAL ARRANGEMENT DRAWING/SKETCH
2.	DIMENSIONED OUTLINE DRAWINGS
3.	CROSS SECTIONAL DRAWINGS
4.	COMPLETED EQUIPMENT DATA SHEETS	1P	WQ
5.	COMPLETED MOTOR DATA SHEETS	1P	WQ
6.	PERFORMANCE CURVES	1P	WQ
7.	CATALOG INFORMATION, CUTS, ETC.	1P	WQ
8.	FOUNDATION DIAGRAMS AND LOADING REQUIREMENTS
9.	SCHEMATIC PIPING DRAWINGS
10.	SCHEMATIC WIRING DRAWINGS
11.	ASSEMBLY AND/OR SHOP DETAIL DRAWINGS
12.	COMPONENT AND/OR SHOP DETAIL DRAWINGS
13.	DETAILED PARTS LIST (BILL OF MATERIALS)
14.	RECOMMENDED SPARE PARTS (WITH OEM NUMBERS) FOR (1) YR OPERATION W/ FIRM PRICING			1P	3 WK ARD
15.	RECOMMENDED LONG LEAD SPARE PARTS/ASSEMBLIES (WITH OEM NUMBERS)	1P	(PREL) 3 WK ARO	1P	3 WK ARD
16.	RECOMMENDED START UP SPARE PARTS W/ FIRM PRICING			1P	3 WK ARD
17.	INSTALLATION, OPERATION, MAINTENANCE & LUBRICATION MANUALS (Documents Shipped With Equipment Are Information Only, Not Acceptable as Submittal)			1P, 1E	6 WK PTS
18.	MANUFACTURER'S TEST, INSPECTION & DATA REPORTS
19.	MILL TEST CERTIFICATES WITH HEAT NUMBER
20.	MONTHLY PROGRESS REPORTS W/ENGINEERING, PROCURE, FABRICATION & INSPECTION SCHEDULES	1P	2 WK ARO	1P	MONTHLY
21.	WELDING PROCEDURES
22.	PAINT SPECIFICATION DETAILS
23.	QUALITY SYSTEM QUESTIONAIRE	1P	WQ
24.	DETAILED QUALITY PLAN	1P	3 WK ARO
25.	DETAILED SHIPPING LIST	1P	(PREL) 3 WK ARO	1P	3 WK PTS
26.	WEIGHT LIST OF FABRICATED PARTS FOR ERECTION, UNIT SHIPPING WEIGHT, ERECTED WEIGHT			1P	3 WK PTS
27.	SHIPPING SCHEDULE AND PACKING LIST			1P	1 WK PTS
28.	EQUIPMENT CALCULATIONS AS NOTED
29.	U-1 FORMS AND NAMEPLATE FACSIMILE
30.	EQUIPMENT LIST/INDEX			1P, 1E	3 WK ARD
31.	INSTRUMENT LIST/INDEX			1P, 1E	3 WK ARD
32.	DRAWING LIST/INDEX
33.	SUBSTITUTE CODES, STANDARDS AND SPECIFICATIONS
34.	SUBVENDORS AND SUBCONTRACTORS NAMES AND QUALIFICATIONS	1P	WQ
34.	MATERIAL SAFETY DATA SHEETS	1P	WQ	1P	3 WK PTS
DOCUMENTS AS QUANTIFIED ABOVE SHALL BE FORWARDED TO:                                     FLUOR GLOBAL SERVICES ATTN:  XXXXXXX 100 FLUOR DANIEL DRIVE GREENVILLE, SC  29607 VOICE: FAX: E-Mail:		NOTES: 1.      ALL DRAWINGS MUST BE MARKED WITH COMPANY NAME, DRAWING TITLE, DRAWING NUMBER, REVISION, PURCHASE ORDER NUMBER AND EQUIPMENT ITEM NUMBER. 2.      SUBMITTALS OTHER THAN DRAWINGS MUST BE MARKED WITH PURCHASE ORDER NUMBER AND EQUIPMENT ITEM NUMBER. 3.      ALL HARD COPY SUBMITTALS  MUST BE MICROFILM & SCANNING QUALITY.

Once this equipment is installed and operational, this data, these documents are relevant in the ongoing management of the asset. This is your knowledge base for this specific piece of equipment. These are the Fricking Instructions, DON’T LOSE THEM. And now I have to ask; “do you know where yours are?” When I write my SOP’s, when I write my maintenance plans, when I troubleshoot, when I plan my outages, when I work on the fricking equipment, when I ……………….., I need to reference this knowledge base, I have to be familiar with the instructions to manage this asset throughout its life cycle. If you don’t have them, you’re guessing and you can’t afford to play the lottery here boys, you need the exacts, you need the Fricking Instructions.

And that’s not all you have to read, it’s a start but it still may not be enough. When we think Read the Fricking Instructions, we also need to think ‘Reductionism’.

Reductionism

Reductionism is the process of breaking complex systems down to their component level. We do this in an attempt to see how things work; assuming that a complex system is nothing more than the sum of its parts. Modern science and engineering protocols are largely influenced by this process. We Analyze for Failure and Evaluate Service Life at this level; the component level.

Reduction…………..what? Not only do I have to understand how an assembled asset functions, I need understand how the motor, the coupling, the bearings, the seals, the belts, the sheaves, the shafts, the motor starter, the VFD, the fuses, the pressure switch, the PLC and all of the other parts and components that make up an asset function as well. I also need to Read the Fricking Instructions for all of these components.

The big difference between instructions for the individual component as compare to the assembly, you may have to do some research to find the instructions. Today’s best source is the internet. You can virtually Google up just about anything. The Fricking Instructions are at our fingertips yet many individuals that need to review the instructions still don’t.  It doesn’t seem all that many years ago I had a rolodex with the names of many, many vendors who I would call routinely and have them fax (another technology that is fading rapidly) me instructions and for larger manuals, they would send you me the instructions ‘snail mail’. Yet with all of the advances in communicating vendor information (the internet), Aliteracy is still prevalent and errors are still introduced.

It’s a shame that many, many errors can be avoided by a simple corrective action. Maybe my Human Performance Improvement (HPI) consulting counterparts should focus on a company’s management behavior so they will include the TIME needed by their engineers, technicians and operators to Read the Fricking Instructions. You know the old saying, there is never time to do it right the first time yet there is always time to do it over. If you Read the Fricking Instructions, you wouldn’t have this issue.

One of my all-time favorite ‘Read the Fricking Instructions’ stories is;

It was a bright and sunny day…………………….at the Hanford nuclear reservation in Richland Washington, I had one of my best planners (my employee) supporting two client millwrights and an electrician in the 200 areas. They were replacing the belts and a belt tensioning idler pulley on a confinement ventilation system exhaust fans. This was a rework job, the client had replaced the belts and idler two weeks earlier and the idler suffered a bearing failure. The original idler and belt lasted just over two years. And yes, you heard me correctly; this is about a belt tensioning idler pulley. This is something (the idler) I had included during a system restoration that included replacing that exhaust fan. My company had managed that exhaust system restoration several years earlier and the idea behind the tensioning idler was to allow a faster belt replacement in a radiologically contaminated area, minimizing exposure, etc… Just think, installing belts on a medium size fan with no alignment or tensioning issues; loosen the idler, slip off old belts, slip on new, retension the idler and start the fan. A two hour job reduced to twenty minutes or less. It would take longer to hang the LOTO then to replace the belt; maybe. And our maintenance strategy was to replace the idler pulley every two years. The reason; the idler incorporated sealed bearings without a means to collect any kind of vibration or condition data, you couldn’t verify condition so you replace routinely.

And the story goes…………………..Just to make sure there would not be a repeat failure two weeks later, I had my expert, the job planner, out there with the client’s crew. All were dressed out in anti-contamination clothing working in a radiation area. Just after 13:00 I get a frantic call from my planner. They had installed the new belts and idler and started the fan. There was squealing, smoke, lightning, plague and locusts; all because the belt was riding up on the edge of the idler. The client’s millwrights and electrician would like me to come out and investigate since I was the originator of this little change several years back. I was 70 miles in another direction at another client’s facility when this call for help came in.

The day was quickly becoming dark and gloomy……………..So I jump in my truck and drove an hour and a half plus spent thirty minutes dressing out in anti-contamination clothing. I met my planner and the client’s crew at the fan. They show me the issue and my planner presents me with his work package for the job. I thumbed through the pages but can’t seem to find what I’m searching for. I ask my planner for the box the new idler came in, the planner and crew fetch the box from the contaminated waste bin (there’s another story here also, we’ll save that for later). I look inside the box and pull out; you guessed it, the instructions for installing the idler. There are spacers and preloads that are required to ensure reliable service. The millwrights and electrician were in awe, my planner was astonished. I remember handing the sheet to my planner and saying “Read the Fricking Instructions”. I was not a happy boss at that moment. I left my planner and the client’s crew to finish the task at hand. They finally did a good job but at what price; the US taxpayer gets to pay.

Several days later my planner says to me; “that was the smartest thing I had ever seen, pulling those instructions from the box, without saying a word. And I will always remember what you said; Read the Fricking Instructions”.

Ever since that moment, my personal lesson learned, my goal, my windmill, my great white whale; pass on and train everyone you meet, the concept of Read the Fricking Instructions.

Maintenance, what a Concept!!!!!

MMJennings