W. Edwards Deming
Dr Deming was an American statistician and management consultant, remembered for his contribution to the success of post-war Japan. He was an avid systems thinker, which is evident from how many of his concepts are aimed at improving the performance of whole organisations. He continued to refine and teach these ideas until the age of 93. Perhaps the most important idea he taught was not a tool or a prescription but a different way to think about business, or as he wrote, “a map of theory by which to understand the organizations that we work in”[1993, p.92], which Deming called the System of Profound Knowledge. Like many of Deming’s published ideas, the aim is to help leaders to work on, not merely in, their organisations.
System of Profound KnowledgeJust as we gain a better understanding of a mechanical watch by looking at how the parts interact, we should look at how different parts of business thinking interact. Deming identified four knowledge areas or “lenses” that he believed were necessary for business leaders to understand their organisations better and to make positive changes. Notably, he saw them as an interdependent system, with the interactions being crucial. The system is in itself not a complete theory of organisations but a core that can be built upon, as demanded by situations. I believe it helps to delineate the boundary of business excellence and provides essential guidance for making sustainable changes.
1. Appreciation for a system
Deming considered many systemic aspects of organisations, including:
- The interests of multiple stakeholders: “stockholders, employees, suppliers, customers, community, the environment”[1993, p.51].
- The need for a common aim (usually customer-oriented) to align efforts within the system.
- Clarity of the aim is a prerequisite for “joy in work”.
- Many common practices conceal or distract from the aim.
- The interdependence of organisational functions and the need to optimise the whole rather than the parts.
- Many problems are best resolved by adjustments in a different area than the symptoms.
- A change beneficial to the performance of one area can cause unexpected problems elsewhere.
- Collaboration is a prerequisite for many improvements.
- Where competition supplants the aim of the system, results are suboptimal or worse.
- The feedback effects resulting from how performance is managed.
- Defining the purpose of a system and communicating its performance correctly is a prerequisite for continuous improvement.
- Incorrectly attributing system performance to people results in the gaming of the system and many other problems.
- The delays in how the effects of a change permeate an organisation.
- Many soft benefits (e.g. of training) are spread over a long time.
- The adverse consequences of a change can manifest much later.
2. Knowledge about variation
At first glance, statistics appears to be all about data, yet it really deals with more fundamental questions about real world uncertainty; “Use of data requires knowledge about the different sources of uncertainty”[1993, p.100]. The impact of this uncertainty on the prediction and interpretation of events is central to many of Deming’s ideas. He saw that the misunderstanding of variation was often the cause of bad business practices.
To paraphrase the concept with an example:
- The exact amount of time it takes you to get to work each day is uncertain and unpredictable. This is due to variation.
- For most people, the range of how long their daily trip to work takes is predictable. Waiting an extra minute at traffic lights is a common occurrence and has a limited impact on the total trip, however, introduces some uncertainty about when they will arrive, it is a common cause of variation.
- On some days commuters may encounter a special cause of variation, such as an accident on the road, a flat battery or other uncommon cause of delay. These are much less predictable and their impact is likely to push travel time outside of the usual range.
- While a special cause of variation in a daily commute may require a specific intervention on the day, a common cause is systemic — directing intervention efforts to any particular occurrence, rather than the whole system, is unlikely to be productive.
It is important to remember that despite being a quantitatively oriented statistician, Deming was strongly opposed to “Management by use only of visible figures”[1986, p.98] and took the position that “It is wrong to suppose that if you can’t measure it, you can’t manage it—a costly myth”[1993, p.35]. While he saw evidence as something important (see the next section), he also spent a lot of effort bringing attention to the myriad of ways that tantalising metrics, such as those based on thresholds, are grossly misleading to business leaders.
3. Theory of knowledge
How do we “know” things? It would be unusual to find epistemology on a longer list of leadership essentials, so why did Deming think it was something important enough to include in a list of just four? I believe it is fair to say that Deming was a pragmatist, concerned with theory that serves practical needs and can predict observed reality, however, even pragmatism requires something more than a “common sense” approach to separating truth from assumptions.
Deming’s discussion of epistemology can be paraphrased with questions like:
- How does one distinguish “fact” from “evidence” from “anecdote”?
- Which predictions are dependable and which are mere guesses?
- How ought one link “cause” with “effect”?
- How can we agree on subjective and contextual truths?
- How should one test hunches to build reliable knowledge?
Perhaps Deming’s reason for explicitly including theory of knowledge was because he rarely saw such questions being discussed in business. Intuitive “rules of thumb” provide poor answers and Deming saw mere “experience” as insufficient: “Without theory, experience has no meaning. Without theory, one has no questions to ask. Hence without theory, there is no learning… To copy an example of success, without understanding it with the aid of theory, may lead to disaster”[1993, p.103].
As intimidating as the word “theory” might be, Deming did not mean elaborate academic theory; even a simple theory can be helpful. For example, Bob wants to buy a new car that is cheap to run. He goes out to find a new car with only a simple theory: “Smaller cars are cheaper to run”. From the experience of seeing many cars, Bob takes notice of their sizes. Throughout the day he asks questions to test his theory about the ongoing costs for a car and learns that there are many other factors to consider. It turns out Bob’s theory was inadequate so he revises it. Note however, that until Bob learns that engine displacement affects fuel consumption, he is unlikely to take much notice of it — until that point, the experience of seeing engine sizes “has no meaning” to him. Similarly, many important things go unnoticed in organisations, however, unlike Bob’s theory, the implicit theories organisations work by are rarely questioned and revised.
While purely mechanistic theories of organisations are very appealing to some, a practical understanding of organisations requires an understanding of people. Deming emphasised two main areas of psychology in his system; motivation and biases.
Do employees come to work just to earn a living? The extrinsic motivating power of the paycheck can be mostly exhausted by the time an individual walks in through the front door; “I’m here, now pay me”. Fear can keep them going until they walk back out; “I’ve done everything they asked, safe for another day”. Although the pressure to comply can raise performance to a mediocre level, having extrinsic motivation designed into the organisation is a recipe for trouble.
How can business leaders foster intrinsic motivation in their employees? “The most important act that a manager can take is to understand what it is that is important to an individual”[1993, p.112]. Intrinsic desires of employees include doing meaningful work that they are proud of and opportunities for learning and development; these are often congruent with the long-term interests of an organisation, if one bothers to connect the dots.
The study of cognitive biases is a well-developed branch of psychology; many of the biases can result in harmful business decisions. As humans we have an innate need to make sense of the world, unfortunately this means seeing connections where there are none (illusory correlation) and dismissing the uncertainty of a situation after the fact (hindsight bias). We like to maintain our world-view, seeking out agreeable information (confirmation bias, selective perception) and stick to our first impressions (anchoring) or decisions (inertia). On the other hand, we tend not to notice important things when we are not looking for them (inattentional blindness) and view acts of omission more favourably than those of commission (omission bias). To top it off, we perceive ourselves as being more rational than we are (bias blindness) and as having greater influence on events than we do (illusion of control). Unfortunately this red pill of biases only gets more bitter…
Although mastering these four lenses seems overwhelming, Deming pointed out that “One need not be eminent in any part of profound knowledge in order to understand it and apply it”.
The System of Profound Knowledge in Businesses Today
Some of Deming’s examples might seem dated now but the concepts remain relevant, continuing to offer insights into how businesses function and explaining their outcomes. Rather than being a prescription, Deming’s system provides a good perspective to test the viability of proposed changes or methodologies. The four lenses are not independent and all have their place when dealing with business change:
- Are we answering the right question? (systems + variation + knowledge + psychology)
- What is the impact of people or on people? (systems + psychology)
- How much confidence should we have in the solution? (variation + knowledge)
- Are the full consequences being considered? (systems + psychology)
How well do modern organisations bring these areas of knowledge together? The (anecdotal) pattern seems to be:
|Area of Profound Knowledge||Location in a Traditional Business|
|Appreciation for a system||Various members practice informally|
|Knowledge about variation||Delegated to analysts|
|Theory of knowledge||Delegated to…|
|Psychology||Delegated to human resources|
It seems to me that most elements of Profound Knowledge, if present at all, are delegated to specialised functions in traditional organisations. That is not to say that these specialisations shouldn’t exist or have certain expertise but rather that they alone cannot perform the integrative role in the organisation — something only business leaders can do. Understanding businesses as systems is incomplete without understanding psychology, which is incomplete without understanding variation; keeping knowledge divided up in silos is only marginally better than having none. Thus the more general lesson from Deming is to bring together diverse knowledge to bear on business challenges.
Finally, why don’t we see Deming’s ideas in common use after all these years? Some possible explanations:
- The old (i.e. early 20th century) ways of thinking about business are alive and well, perpetuated by imitation — doing the same things as your boss is “safe” and when you become the boss you keep doing most of them.
- In many regards, Deming invites leaders to embrace a new paradigm of thinking. This takes more personal effort than just rolling the dice with another panacea.
- He wrote in a style that requires considerable patience to absorb. While this teaching approach can be effective, it is problematic for the time-poor business leaders whom he was trying to address.
- It seems to me that there was much interest in Deming’s ideas around the early nineties. However, in the late nineties there was a little distraction called the Internet. The number of Internet users grew by 526-1450% between 1995 and 1999; businesses needed to act! What followed were many fundamental changes to how companies do business, yet only minor changes to how managers think about business.
Consider a decision in your business that led to a change.
- What perspectives or “lenses” were applied in making the decision? Were these considered separately or together?
- Which lenses of Profound Knowledge might have been applicable yet were missing? What questions would they have prompted? How might the outcomes have been different?
(Self-assessments like this can be tricky, so let us know if you would like some help.)
1. ^ If you want to know more about Deming himself then I recommend this documentary, filmed when he was 91. Given that it is only 28 minutes, many of his ideas are introduced without adequate explanation but there is a good biography squeezed in.
As demonstrated in the video, there has been much fanfare associated with Deming. Reading various internet biographies, one might start thinking that Deming rebuilt Japan all by himself! We know that he taught in Japan but how much impact did he personally have on their economic growth? While a curiosity, it does not really matter I think. Instead we should ask about the usefulness of his ideas; all the fanfare suggests that there is something of value to be found in them.
2. ^ Some time ago I completed my degree in “business” with a “management” major. Most of the modules were taught well and I was happy with what I had learnt. Economic discussions on tv suddenly made sense, as did various business strategies and so on. Generally, I was a satisfied customer. In retrospect, I see now that the course had taught me a lot about working in a traditional business and precious little about working on a business, thus my learning continues.
Business managers might feel that deciding on and implementing a change, such as a new strategy, might be “transforming” their business, yet usually the basic assumptions in the organisation remain unchanged. A necessary skill for a leader, not taught in business school, is to surface and work on those basic assumptions (if they won’t, who will?). Deming, Ackoff and others saw this gap and dedicated much of their time to helping others see it.
3. ^ Many of the business concepts that Deming taught were related to variation. This led some to believe that his message was all about statistical process control, which spectacularly misses the point (a parallel would be thinking that excellent communication is all about accurate spelling). Making explicit the much broader System of Profound Knowledge was, I believe, in part a reaction to some of the misunderstanding. Looking at earlier material, such as this 1980 interview, we can see the focus on statistical control being more prominent in his message:
To provide a comprehensive explanation of problems stemming from the misinterpretation of variation I would need to rewrite a substantial part of Deming’s books, so I will just expand on one example. An important aspect of a stable system (one with only common cause variation) is that it performs consistently over the short to medium term; i.e. roughly half of the time performance will be above average and half of the time below. If performance is especially high or low one day, then by random variation alone, it is likely to return closer to the average performance the following day. This is called regression to the mean (for our purposes here “mean” is the same as “average”). The trap is that we might really, really want performance to keep improving the next day, however, unless the system to which the performance is attributable changes, ongoing improvement is just wishful thinking on our part.
The organisational psychology impact of regression to the mean has been recognised long ago. For example, Kahneman and Tversky noted the perverse effects of the phenomenon on the management of pilots:
Regression is inevitable in flight maneuvers because performance is not perfectly reliable and progress between successive maneuvers is slow. Hence, pilots who did exceptionally well on one trial are likely to deteriorate on the next, regardless of the instructors’ reaction to the initial success. The experienced flight instructors actually discovered the regression but attributed it to [what they thought was] the detrimental effect of positive reinforcement. This true story illustrates a saddening aspect of the human condition. We normally reinforce others when their behavior is good and punish them when their behavior is bad. By regression alone, therefore, they are most likely to improve after being punished and most likely to deteriorate after being rewarded. Consequently, we are exposed to a lifetime schedule in which we are most often rewarded for punishing others, and punished for rewarding. (p.251)
Kahneman, D. & Tversky, A. 1973, ‘On the psychology of prediction’, Psychological review, vol. 80, no. 4, pp. 237-51.
“Knowledge about variation” is thus not just about statistics and control charts but about asking much deeper questions of how we interpret and manage performance in organisations. As Bazerman and Moore put it, “Managers who fail to recognize the tendency of events to regress to the mean are likely to develop false assumptions about future results” (p. 45)
Bazerman, M. & Moore, D.A. 2013, Judgment in managerial decision making, Eighth edn, Wiley.
4. ^ Confusing the two types of variation is unproductive, wasteful or worse. When commuting to work for example, you might arrive earlier one morning, however, contemplating a unique reason for being early that day is unlikely to produce one; it would be a waste of time. When common cause variation is treated as a special cause, it can result in what Deming called “tampering” with the system, which actually amplifies the variation. For example, if you continually adjust your daily departure time based on the previous day’s arrival time, the range of arrival times will actually grow considerably wider; By pure chance (ignoring daily traffic patterns) you might have a quicker trip on days you leave early, or longer trip on days you leave late, resulting in a wider range of travel time. Improvement of a stable system with only common cause variation requires a fundamental change to the system; e.g. take a different route to work, move closer to work, etc.
While Deming promoted these concepts, they originated from Walter A. Shewhart. Shewhart’s contribution was in both recognising these two distinct types of variation as well as the creation of tools to differentiate between them (“Control Charts”), which are used to this day in guiding efforts to improve the performance of processes and systems. An intermediate method is to plot measurements over time (a “Run Chart”) — even this is much more informative than summarised data which can often give the impression that performance is improving while it is actually getting worse.
5. ^ I’m open to suggestions on alternate labels for Deming’s epistemological orientation, as philosophy of science is not my forte. Deming does cite the pragmatic philosopher Clarence Irving Lewis[1986, p.317 & 1993, p.101] and pragmatism generally seems to fit his position better than positivism. The PDSA (Plan-Do-Study-Act) cycle he advocated includes phases of both inductive and deductive thinking.
6. ^ In examining Deming’s idea, it seems that the “profound” aspect is the “system”, the integration of the various lenses, more than the “knowledge” within each lens taken separately; one could apply all four in isolation and make little progress. But I suppose “profound system of knowledge” just doesn’t have the same ring to it.
7. ^ The historical chain of managers mimicking what they see other (i.e. senior) managers do, extends far back in time similarly to any social ritual, hence the thinking prevalent in organisations today is essentially old thinking. There are however other explanations for why organisations are similar, provided by institutional theory and specifically the concept of isomorphism. While the espoused reasons for copying common practices might include performance improvement, the underlying driver is to make use of the popularity of established practices and thereby justify one’s own change as “legitimate”, sometimes at the cost of performance. The seminal work of Dimaggio and Powell on the topic serves as a good introduction:
DiMaggio, P.J. & Powell, W.W. 1983, ‘The iron cage revisited: Institutional isomorphism and collective rationality in organizational fields’, American sociological review, pp. 147-60.
To be truly “transformational” a leader needs to challenge not just what the business does (i.e. work in the business) but the basic assumptions of how the business does them (i.e. work on the business). Ricardo Semler of Semco comes to mind as a leader who has trashed traditional business assumptions by the bucket load:
The case of Semco provides good, albeit anecdotal, evidence of how powerful isomorphic forces really are. If such an unorthodox company can be so successful, companies which cling to orthodoxy appear to be doing so primarily due to herd mentality (i.e. isomorphism and legitimacy seeking) rather than any performance benefit.
8. ^ Some have argued that Deming was a poor teacher because his published works are difficult to learn from, however, his approach might have been spot on pedagogically, if we consider some context. Shouldn’t teachers make things easy for students? As David Kerridge wrote, Deming “never allowed students to ‘roll back the marbles’, as he put it: to simply repeat what they had been taught”.
Kerridge, D. 1994, ‘W. Edwards Deming – a personal memory’, Quality and Reliability Engineering International, vol. 10, no. 1, p. 3.
Rather than have people memorise and regurgitate concepts (as is all too common), Deming wanted them to absorb, interpret and adapt them; in other words, learn to think for themselves so that they can deal with unique situations in the unique contexts of their organisations. Perhaps his approach frustrates people who are used to step-by-step guides, however, it is precisely because they are not reduced to a step-by-step guide that Deming’s ideas have value. I always recommend that people should “study” rather than merely “read” Deming’s work, as his books, with their deceptively simple language, are not the kind that one can learn something useful from in a weekend speed-read.
Closely related is the need to study Deming’s work first hand. Why? Due to how Deming expressed his ideas in text, many who just breezed through his books have seriously missed the points he was making. Most misinterpretations of Deming we see floating around the internet today are nth-hand remixes of his ideas; a great, albeit unfortunate, illustration of “rule 4” in Deming’s funnel experiment[1993, p.190]. One cannot, other than by miraculous accident, extract the correct interpretation from a flawed one! Save yourself the trouble and go directly to the source.
9. ^ Between 1995 and 1999 the number of internet users worldwide grew from 16 to 248 million… or from 44.84 to 280.87 million; it depends on who you ask! Deming included “operational definitions” in his writing on the theory of knowledge and this discrepancy conveniently illustrates the need — different methods of collection will produce different data; to decide on the method we must first know its purpose. At least there is agreement that internet usage grew rapidly in the late nineties, which was fortunately the only purpose of including these sources:
10. ^ Most of the new thinking that the advent of the internet has inspired relates to the new opportunities that have emerged and the changes that have been enabled, however, these are largely compatible with classical management thinking. As dramatic as business changes have been, most of them are a reconfiguration of processes and interfaces; snail mail → electronic mail, sell through stores → sell through websites, offline word of mouth → customers writing online reviews, etc. Numerous high-impact changes, yes, whole industries have been shaken up by the internet and new ones created, however, someone who thought advertising was important before the internet would probably still think that now; they just have new options for how to do these things.
So, in the context of Profound Knowledge, what impact has the advent of the internet had on changing how we think about business? Not all that much, yet. There are however new pressures that are likely to result in changes of thinking. Some obvious ones:
- The relationship between organisations and their employees is changing. While the initial impact of the internet has been “more of the same” activity, such as outsourcing and downsizing, it is now leading to greater transparency, with services such as glassdoor.com allowing employees to share their experiences with the world. Even the most adamant Theory X manager is under pressure to change their thinking.
- Similarly the topic of “quality”, once championed by Deming, is back in the spotlight. Online reviews and social media ensure that if a customer has a disappointing experience then a great many people will hear about it (arguably there is also an increase in cases where this is being abused). In any case, if continuous quality improvement is back in fashion, then the thinking it depends on will see more interest too.
Latest posts by Marcin Kreglicki (see all)
- Thresholds: The “Wishful Thinking” Approach to Designing Metrics - January 27, 2018
- There Is No Recipe for Excellence, nor Is One Necessary - January 25, 2016
- Lean and Six Sigma - March 17, 2015