Process Metrics are Key
It is well known that ITIL® describes four types of process metrics: process efficiency, process effectiveness, process progress and process compliance [see, in particular, the discussion in Service Design, §3.7.5]. However, given the ambiguities of this terminology, it is not always evident to débutants how these types may be characterized and used. If efficiency and effectiveness are relatively easy to understand, progress and compliance are less easy to comprehend. Because a good understanding of metrics is a key underpinning of good management, it is worth looking at what these types really imply. In this article, I will discuss process compliance.
Process Compliance as a Metric Type
ITIL describes the process compliance metric type as
Compliance of the process to governance requirements, regulatory requirements and compliance of people to the use of the process.
Hiding behind this all too brief enumerative description is a very wide array of controls concerning different life cycle phases and widely varying types of governance. We may eliminate the problem of using a term to define itself and generalize the description of the process compliance metric as:
The definition of how to measure the respect by a process of the controls placed on it.
The controls might be external to the service provider organization, such as governmental regulations or industry standards. They might be internal to the service provider organization, but external to the process itself. And they might be controls that are internal to the design of the process itself. And, as we shall see, we may assess the definition of the process as well as the execution of the process.
Given this definition, we see that the list of “controls” provided by ITIL consists, first of all, of an active agent that truly control a process (the process owner); next, of the references the controller uses to perform the controls (policies, objectives, documentation); and finally the measurements that are compared to the reference values (feedback). Missing from ITIL’s list are the technical agents that may control a process autonomically (typically, software tools); the measurements of the process activities themselves (and not just the output); the measurement of the inputs (whose volume can be compared to the volume of outputs); and the assessment of the outcomes at the process’ customers that were fostered by using the process. These outcomes may indicate if the objectives of the process were appropriately defined.
Compliance and Conformity
Living in an age of semantic lassitude, we should clarify what we mean by “compliance”, in an effort to overcome the indolent indifference to the meanings of words that characterizes so many people. Too, many organizations have developed idiosyncratic uses of the term “compliance”, leading to divergent understandings of what it means. It is worth distinguishing compliance from conformity as a way of clarifying our understanding.
In the English language, “to comply” has a strong nuance of a situation where one party requests a certain service or behavior from another party. When the second party responds in a way that meets the intention of the first party, the second party may be said to comply with the first party’s request. There is a further nuance that the second party may not be entirely in agreement with the request, but nonetheless acts as requested. In other words, left to its own devices, the second party may well have acted differently. We comply with a governmental requirement to pay taxes, even though we would likely not pay them if not so required.
“To conform” implies a cultural or organizational norm or standard that shapes our behavior. We conform, not because anyone in particular asked us to (that would be compliance), but because we have an intrinsic understanding of social expectations and we behave accordingly (or not). Conformity may be valued or may be despised but, like self-censorship, it is a force with which to be reckoned. We conform (or not) to our perception of social imperatives.
Although we do distinguish these nuances in English, many languages do not do so. They may well use the same terms for both compliance and conformity, with the underlying understanding of that language’s term being both richer and more ambiguous. But, as our intention here is to disambiguate, we will insist on the English language sense of the term compliance.
Process Compliance with external regulations and norms
How may a process comply with external laws and regulations? In general, these laws do not regulate the activities of the process, but regulate instead the output of the process activities. For example, the law might require that a company’s accounts be a true and faithful representation of its financial situation. However, if that company cannot demonstrate that it follows a recognized accounting process to achieve that goal, it is likely to be subject to a long and unpleasant scrutiny.
In the realm of service management, compliance is more concerned with the data manipulated by a process, rather than the process itself. Laws might require that a given process include, or exclude, certain types of data from the records handled by the process. Countries with bank secrecy laws, for example, might require that records, such as service request or incident records, not contain information about bank customers if those records are visible to people not subject to those same laws (with the egregious exception of the Internal Revenue Service and other U.S. agencies best left unnamed). Or the European Union might require that personal data of EU citizens not be visible in other countries that cannot, or will not, harbor that data safely, as per EU standards. Such safe harboring would normally be ensured by the processes used to handle those data. So, using a standardized process in a consistent way is merely a strategy for achieving compliance of the output of that process. (The same strategy is often used as a means to reduce error, increase productivity and to make a process more effective.)
Industry standards often influence the scope of application of a process. An eminent example would be the application of a compliant change management process to the IT systems used in the design and manufacture of pharmaceutical products. If the company cannot demonstrate compliance with those standards for managing changes, then it might be refused permission to market its products in the more finicky jurisdictions. If a company objects that it is perfectly capable of attaining an appropriate quality for its medicines without the overhead of following such processes, its plaint is likely to fall on deaf ears.
A third type of external compliance would be with a national or international standard, such as ISO 20000-1. But once again, the standard is more concerned with the existence of processes that achieve certain outputs, and the definition and use of clear responsibilities, rather than regulating the detailed nature of the process activities, or who might play the process roles.
Process Compliance with Organization-wide Policies
Often, a process is required, or is encouraged, to comply with certain organizational policies. The best known of such policies concern information security. As we know, the risks to information confidentiality, integrity, availability and authenticity may be managed via organizational controls, technical controls, physical controls or procedural controls. Thus, the existence of a process to handle such risks may be an acceptable way of complying with an information security policy.
Another example of a policy that directly impacts process design would be a policy regarding the segregation of responsibilities. Such a policy would require that potential conflicts of interests be identified and separate roles created to mitigate those potentialities. Furthermore, the mapping of persons or organizational entities to those roles would have to respect the policy. Other examples would be a policy requiring that high risk data require four eyes to modify it, or the segregation of roles to prevent individuals from circumventing process checks.
We may expand this concept of compliance to include the alignment of processes to organizational strategies. I cannot overemphasize the importance of this concept, as it goes to heart of the way organizations are evolving today and why certain industry frameworks are lagging behind, rather than leading, that change.
An excellent example of this would be the adoption of agile methods for the execution of the organization’s activities. Once such a strategy adopted, processes may be assessed for their compliance with that strategy. In such a context, it is questionable if a process designed on pure waterfall principles were compliant with the organization’s strategy of agility. This is perhaps one of the reasons why lean-oriented professionals might prefer to speak of value streams rather than processes.
Process Compliance with Internal Process Policies
Compliance with internal process policies may take two forms. In the first case, the process owner has preferred to articulate a policy for the regulation of the process, rather than articulating the detailed procedures to follow. This is a way of saying to the people performing the process activities, “However you perform those activities, you need to comply with the following policy.” In other words, rather than telling people what they must do, you describe the constraints or limits on what they do. A certain margin is allowed to each process actor to perform the activities, within those limits, in his or her own way.
A good example of this would be the application of a policy to user support concerning the behavior of the support personnel. Everyone might be expected to be cheerful, respectful and polite. However, the policy recognizes that there are many different ways to be polite, depending on the contexts and the personalities involved. No attempt is made to regulate how to comply with the policy.
In the second case, the process execution is assessed in terms of the degree to which it is performed as agreed. Many people mean exactly this, and this only, when they speak of process compliance. To distinguish it from the broader scope of process compliance, we may call this “process adherence”. And when 32’491 people on LinkedIn tell you they have this skill, many of them are talking about how to get people to adhere to performing a process as agreed.
To continue with the above example, support personnel might be provided with scripts telling them precisely what they must say and when, with all the please’s and thankyou’s included. Compliance is measured in terms of having followed the script, rather than in terms of the spirit underpinning it. It might be argued that such forms of compliance are more appropriate to silicon-based, rather than carbon-based, life forms.
But internal process compliance also concerns whether the process itself is indeed triggered or performed when it is agreed that it should be used. Thus, we are often concerned with whether changes are performed under the control of a change management process, or whether new applications are deployed without first having performed the relevant tests, or whether a release process was followed at all.
One might ask what role service level agreements and other types of organizational agreements play with respect to process compliance metrics. Such agreements typically define a set of reference values to which a process should comply. These reference values are, of course indicators of process performance (whether they be key or not). When making an agreement with a customer of a process, that customer is more likely to be concerned with effectiveness and efficiency metrics than with compliance metrics, although some customers might also insist on meeting standards of ethical behavior or environmental standards, for example. However, agreements with teams performing the process activities might be asked to agree to certain levels of compliance, in addition to the typical effectiveness and efficiency metrics. It is important, however, to distinguish between the definition of compliance metrics, as we have described until now, and the compliance with targets for other types of metrics, such as effectiveness and efficiency service level objectives. The latter is a completely different notion of compliance that is out of the scope of a discussion of compliance metrics.
Measuring Process Compliance
Given the breadth of the types of compliance a process might have, we might wonder what is the best way to measure process compliance. After all, we are talking about metrics—definitions of how to measure something. Compared to measuring process efficiency, measuring process compliance is something of a conundrum. It is very difficult to roll up all the different types of compliance into a single measurement, other than in a meaningless statement such as “Compliant” or “Not compliant”. Let’s look at the measurement of each type of compliance in turn.
Measuring whether a process is used or not
The metric itself is fairly straightforward. For example, if 100 changes are performed and 98 of them are done following an agreed change control process, then we might measure the compliance as 98%. The difficulty is not in defining the metric, but in making the measurement. Non-compliant actions do not usually leave traces in the process tracking tool.
However, such measurements may be made either via an auditing procedure or, in certain cases, via technical controls. An auditing procedure depends on the existence of records of process execution and adequate sampling techniques. Let’s take release management as an example. It is possible to take a random sample of the applications currently in a production environment, identifying the versions in use. By checking for the existence of testing records or release records, it might be possible to estimate the percentage of applications that have used those processes as part of their transition into production. Of course, this method is limited by the compliance of those processes in creating the requisite records!
In other cases, identifying whether a process has been used is rather a hit-or-miss affair. Take the example of changes. Only the result of a change, but not the change itself, is tangible. We can only detect unauthorized changes as a result of a laborious cause analysis, or by using technical means to automate the detection of configuration changes of system components. In such cases, the measurement of non-use of the process might have a very low precision and questionable accuracy. The measurement method itself might introduce great bias into the measurements.
Measuring whether a process follows internal process policies
In theory, this sort of compliance could be measured by comparing the cases where a policy was successfully applied to the cases where is should be applied. But this sort of measurement is also extremely difficult to obtain. Measurement systems are likely to be burdensome and unreliable. For example, customers might be asked after receiving support if the agent were polite (as a way of measuring compliance to a politeness policy). For sure, customers are more apt to complain about a lack of politeness, thereby introducing a strong bias in the measurements. Furthermore, one man’s Mede is another man’s Persian. There is hardly any objective measurement of politeness.
Another approach to such measurement is via an auditing procedure wherein the actors in a process are shadowed and their compliance with policies is assessed by the auditor. This is essentially what happens when a supervisor listens in to a conversation between an agent and a customer. Useful measurements depend on an appropriate sampling technique, else the results are heavily influenced by the auditor’s biases. Lacking such audits, wee are reduced to measuring internal policy compliance by looking at the exceptions and complaints that might arise.
Measuring process compliance with general policies, external regulations and norms
Once again, auditing is the principal means of detecting failure to comply with organization-wide policies. It depends on the existence of records that might be audited and on the auditor’s ability to sample those records such that the results may be extrapolated with a minimum of bias in sampling and analysis.
The lack of records does not mean that the process is non-compliant. It merely makes life difficult for the auditor and expensive for the service provider. Once again, the process may be audited by shadowing the actors and drawing conclusions from first hand observations. But such methods may well influence the subjects being shadowed, resulting in performance levels that are different from what might otherwise be the case [as, for example, the Hawthorne effect].
One way of measuring such compliance is to count the audit points raised and somehow factor in the importance of the different points. The “somehow” is important, insofar as most weighting factors used for ordinal statistics, such as “minor, medium, major” are completely arbitrary forms of voodoo statistics. Ideally, the impact of non-compliance issues should be translated into economic terms, which would then allow for an objective aggregation of the results. Even better would be the measurement of the cumulative impact of those issues over time.
We should avoid defining metrics as simple counts, whether they be counts of audit points, counts of complaints, or any other form of count. Counts should always be expressed relative to the key drivers of those counts. This point is easiest to exemplify using an effectiveness metric. Many organizations are tempted, for example, to count the number of incidents (or changes, or problems, etc.) as a process metric. But such counts mean nothing on their own. A very large, technically complicated organization is certain to have far more incidents than a small, simple one. So, we prefer to measure the count of incidents as a ratio to something else, such as the number of users or the number or different technologies in use, or the number of system components, etc. Applying this same concept to a compliance metric based on auditing, we would prefer to state the ratio of audit points to the total number of issues audited, on the assumption that those issues are selected in a random way or in a way that does not unduly introduce measurement bias.
Summary
We have seen that a process is subject to a wide variety of controls. Compliance with those controls may be measured, with greater or lesser precision, in a wide variety of ways. Certain controls only concern the design of the process; other controls concern the execution of the process. Controls are usually concerned with the outputs of processes, but may also concern the performance of the process activities, as well as the design of the roles and responsibilities and the mapping of people to those roles.
It can be exceedingly difficult to measure compliance. It is virtually impossible to state categorically if a process is “compliant” or not. Such an assessment would not, in any case, be very useful. More useful would be an assessment of the impact of process compliance of the economic value of performing the process. This would have the further advantage of being directly comparable to measurements of process efficiency and effectiveness (assuming, too, that effectiveness be translated into economic terms). Alas, many organizations ignore the economic cost and benefit of the policies and other controls they put in place.
In a future article, I will make a similar analysis of progress as a process metric.
¹
The diagrams in this article by Robert S. Falkowitz are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
Source of the image of the centrifugal governor: “Shaft governors, centrifugal and inertia; simple methods for the adjustment of all classes of shaft governors (1908) (14757599606)” by Collins, Hubert Edwin, 1872- – (no longer available there) book page: https://archive.org/stream/shaftgovernorsce00coll/shaftgovernorsce00coll#page/n15/mode/1up. Licensed under No restrictions via Wikimedia Commons – https://commons.wikimedia.org/wiki/File:Shaft_governors,_centrifugal_and_inertia;_simple_methods_for_the_adjustment_of_all_classes_of_shaft_governors_(1908)_(14757599606).jpg#/media/File:Shaft_governors,_centrifugal_and_inertia;_simple_methods_for_the_adjustment_of_all_classes_of_shaft_governors_(1908)_(14757599606).jpg
Excellent article – really useful information on the topic.
Additionally, I’d be interested to know how, or what the best method would be, to translate into economic terms, the impact of non-compliance issues and what ways would be best to measure (and present) the cumulative impact of those issues over time.
Measuring non-compliance in economic terms would require two approaches, depending on the type of non-compliance. Suppose you fail to comply with regulations of an external regulator. This is presumably an occasional non-compliance, rather than a structural non-compliance. For example, you are a bank that fails to communicate its daily closing position to the central bank. That doesn’t happen all the time, but you need to estimate the probability of it happening. Say you calculate that it is likely to happen once every 3 years. The direct cost impact is easy, because it is the amount of the fine. The indirect cost impact is hard, because it covers things like lost reputation. So you make a hypothesis along the lines of “we have lost 1% of our business due to the tarnished image.” From that, you can calculate a cost impact. The difficulty is that people generally have no idea how to estimate that loss of business and the figures are produced thanks to vivid imagination, rather than facts. You can actually use approaches using Bayes theorem to come up with defensible hypotheses. See Douglas Hubbard’s book, “How to measure anything”, for more information about this.
The economic impact of internal rules non-compliance requires a different approach. In other words, what is the cost of not performing the process in the prescribed way. First of all, you have to accept that your agreed process might not be very efficient and by working differently, you actually save money. But let’s assume the contrary, for the sake of argument. If non-compliance results in poorer performance, then you will see it using statistical control charts for the process. In other words, you might see that your process is out of control (outliers), or you might see that the standard deviation for process lead time (or whatever metric you are using) has become greater, or the mean lead time has become greater. All that translates into two measurable impacts: 1) the cost of greater inefficiency, generally measured by the increase in FTEs required multiplied by the mean cost of the FTEs; and 2) the impact of the increased inefficiency of the customers of the process. This latter metric is itself composed of two factors: 1) once again, the impact on customer process efficiency (e.g., do I need more employees because IT is slow?); and 2) the opportunity costs of IT inefficiency (e.g., have I lost a contract because I could not make an offer in time?). I think you can see that this calculation can become iterative, depending on the overall complexity of the business and how the business processes support each other.
Is this the best method to translate into economic terms? I guess that depends a lot on the capability levels of your organization to make such calculations and on the purpose to which you will put them. Whether it is best or not, it is at least factual rather than subjective, and holistic rather than ad hoc.
Great article and comments, I would add to the above the fact that non compliance to processes can also increase the risk of fraud and error.
How about the compliance against the project’s defined process ?
That aims to measure the compliance level of actual process implementation against the planned one ?
If you are talking about the case where an organization has one more processes for managing a project and whether a given project is executed according to those processes, then this would be an example of compliance that could be measured using the metrics described above.
However, if you are talking about whether or not the tasks of a project are executed as per the plan, then this is an issue of performance, in my view, not an issue of compliance. If you have a task to perform as part of a project, and if that task is supposed to be performed according to a certain process, you can measure whether it was done in compliance with the process. But this is not a project management issue. If you are more concerned with the output of the task, if it was delivered when planned and to the required level of quality, then this is performance, not compliance.