IT organizations turn to outsourcing for any number of
reasons, and to fulfill a variety of needs. Whether the goal is to obtain
expertise or to reduce costs, to offload application maintenance or help
desk operations, outsourcing is here to stay. The typical outsourcing
engagement will last for a number of years, and be governed by a contract
setting the terms and conditions between the client and outsourcer for the
duration of their relationship. To measure whether that relationship is
working, and how well, Service Level Agreements are established.
A Service Level Agreement (SLA) is an essential part of
any outsourcing project. It defines the boundaries of the project in terms
of the functions and services that the service provider will give to its
client, the volume of work that will be accepted and delivered, and
acceptance criteria for responsiveness and the quality of deliverables. A
well-defined and crafted SLA correctly sets expectations for both sides of
the relationship and provides targets for accurately measuring performance
to those objectives.
At the heart of an effective SLA is its performance
metrics. During the course of the outsourcing engagement, these metrics
will be used to measure the service provider's performance and determine
whether the service provider is meeting its commitments. When properly
chosen and implemented, the SLA metrics:
-
measure the right performance characteristics to
ensure that the client is receiving its required level of service and
the service provider is achieving an acceptable level of profitability
-
can be easily collected with an appropriate level
of detail but without costly overhead, and
-
tie all commitments to reasonable, attainable
performance levels so that "good" service can be easily
differentiated from "bad" service, and giving the service
provider a fair opportunity to satisfy its client.
This article focuses on the issues surrounding the
selection and implementation of SLA metrics. Although application
outsourcing is used for many of the examples, the principles described
within are applicable for any type of outsourcing engagement. This article
does not attempt to define an exhaustive list of metrics that should be
included in a SLA; the topic is too large and project variations are too
great. Rather, it concentrates on the principles for selecting metrics,
the categories of metrics, and how those metrics should be represented in
a SLA. These topics are necessarily presented in an introductory manner.
Organizations without extensive metrics experience are urged to consider
professional assistance to guide them through the process of creating
their first few SLAs.
Five Principles for Selecting SLA Metrics
Selecting the appropriate metrics to gauge project
performance is a critical preparatory step for any outsourcing engagement.
A variety of metrics is required to manage the numerous aspects of an
outsourcing project. While some metrics will be unique to a given project,
many are common to all outsourcing projects. Often, a metric that works
well on one project may be ineffective, inaccurate or too costly to
collect on another project. A poor choice of metrics will result in SLAs
that are difficult to enforce and may motivate the wrong behavior or even
cause a dispute that ends up in court.
The selection process is complicated by the enormous number of
potential metrics and must be tempered by considerations such as
organizational experience with metrics, the type of behaviors to be
motivated and cost and effort of collection. Common sense must prevail
when selecting metrics. Remember that the goal is to ensure a successful
and positive working relationship between the service provider and the
client. To meet these goals, organizations should consider the following
five principles.
- Choose measurements that motivate the right behavior
The first goal of any metric is to motivate the
appropriate behavior on behalf of the client and the service provider.
Each side of the relationship will attempt to optimize their actions to
meet the performance objectives defined by the metrics. If the wrong
metrics are selected, the relationship can go astray quickly. For
example, paying programmers by the number of lines of code they produce
will certainly lead to an increase in production, but may play havoc
with quality and the true quantity of real work accomplished.
To motivate the right behavior, each side must
understand the other side, its expectations and its goals, and the
factors that are within its control. Realism must prevail. Clients have
to anticipate that service providers will want to make a profit; service
providers have to expect that clients will want to control costs.
When choosing metrics, first focus on the behavior
that you want to motivate. What factors are most important to your
organization? Reducing costs and/or defects? Increasing production or
speeding time-to-market? Which factors are you willing to trade for
improvements in another area? Pick an initial set of metrics that
measure performance to these behaviors.
Put yourself in the place of the other side and test the selected
metrics. How would you optimize your performance? Be creative. Does that
optimization lead to the desired results? Often, secondary metrics are
needed to provide checks and balances to avoid missteps. Also, consider
whether the metrics are truly objective or are subjective enough to
leave room for interpretation. Metrics that are based upon a subjective
evaluation are open to different interpretations, and will likely lead
to disagreement over whether a service provider has met its commitments.
For example, state that "all printed invoices will be delivered to
the post office within 4 hours after completion" rather than
"all printed invoices will be delivered to the post office in a
timely manner."
- Ensure metrics reflect factors within the service provider's control
Ensure that the metrics measure items within the
other party's control. Continuing the example from above, the service
provider has control over bringing the invoices to the post office, but
has no control over the speed by which the post office delivers the
mail. Thus, a requirement that "all printed invoices will be
delivered to our customers within 48 hours after production
completion" is unfair and likely to be de-motivating to the service
provider.
Service providers should ensure that the SLA is
two-sided. If the service provider's ability to meet objectives is
dependent on an action from the client, the client's performance must
also be measured. For example, a service provider may be held
accountable for the speed and quality of a system enhancement, but the
quality is affected by the accuracy of client-developed specifications,
and speed/delivery is held up by the client's approval cycle.
Conversely, refrain from choosing SLA metrics that attempt to dictate
how the service provider is to do its job. Presumably, an
outsourcing provider's core competence is in performing IT tasks, and
embodies years of collected best practices and experience. Attempting to
regulate these tasks will only introduce inefficiencies. Instead,
concentrate on ensuring that the delivered work products meet quality,
time and cost expectations.
- Choose measurements that are easily collected.
If the metrics in the SLA cannot be easily gathered, then they will
quickly lose favor, and eventually be ignored completely. No one is
going to spend an excessive amount of time to collect metrics manually.
Ideally, all metrics will be captured automatically, in the background,
with minimal overhead; however, few organizations will have the tools
and processes in place to do so. A metric should not require a heavy
investment of time and money; instead use metrics that are readily
available, compromising where possible. In some cases, it will be
necessary to devise alternative metrics if the required data is not
easily obtainable. For example, measuring whether a newly written
program meets published IT standards require an arduous manual review.
Conversely, a commercially available metric analysis tool can quickly
and automatically calculate the program's technical quality. While the
end result is not identical, the underlying goal -- motivating enhanced
quality -- is met at a fraction of the manual cost.
- Less is more.
Avoid choosing an excessive number of metrics, or metrics that
produce a voluminous amount of data. At the outset of drafting the SLA,
an organization may be tempted to include too many metrics, reasoning
that the more measurement points it has, the more control it will have
over service provider performance. In practice, this rarely works.
Instead choose a select group of metrics that will produce information
that can be simply analyzed, digested and used to manage the project. If
the metrics generate an inordinate amount of data, the temptation will
be to ignore the metrics, or subjectively interpret the results,
negating their value in the SLA.
- Set a proper baseline.
Defining the right metrics is only half of the
battle. To be useful, the metrics must be set to reasonable, attainable
performance levels. It may be difficult to select an initial,
appropriate setting for a metric, especially when a customer does not
have any readily available performance metrics or a historical record of
meeting those metrics. Companies with active metrics programs will have
the data needed to set a proper baseline. Others will have to perform an
initial assessment to establish that baseline. Unless strong historical
measurement data is available, be prepared to re-visit and re-adjust the
setting at a future date through a pre-defined process specified in the
SLA. Further, include a built-in, realistic tolerance level.
Consider the example of a customer that selects an outsourcing
service provider to run its building operations. An important customer
objective is to keep occupants comfortably heated and cooled. To that
end, a metric is selected requiring the service provider to achieve a
specified "comfort" level of 70 degrees. It would be tempting
to set the comfort metric so that the service provider had to meet the
threshold 100% of the time. But why require the service provider to keep
the buildings heated and cooled 24 hours a day/seven days a week, even
when unoccupied, especially since it will cost the customer money to do
so? A better way would be to define a metric that accommodated different
comfort levels at different times. In addition, since the customer has
historically been able to maintain even heating and cooling only 95% of
the time, it would be reasonable to grant the service provider the same
tolerance level. By taking the time to weigh expectations and set
reasonable, attainable performance goals, the customer is able to
achieve its goal of comfort at a lesser cost while the service provider
is motivated to do its best to meet those needs.
The Theory Behind SLA Metrics
Before discussing metrics selection in more detail, some context is
necessary. A high level understanding of the factors affecting an
outsourcing contract helps illustrate why certain categories of metrics
are needed and highlights the issues that must be considered when
developing the initial performance targets for those metrics.
Figure 1
Figure 1 illustrates the types of metrics required to
support a generalized outsourcing engagement. In its simplest form, the
engagement can be viewed as a "black box" that accepts a volume
of work requests and delivers a volume of work produced. The
length of time needed to complete the work is time-to-market or responsiveness.
The work is produced for an overall cost, and efficiency can be
calculated as cost/work product unit. Quality is defined as the
ability of a work product to pass its acceptance standards. Each of these
factors represents an interface between the client and the outsourcer and
can be manipulated as part of a SLA.
Certain factors are solely under control of the client.
The client determines the volume of work requests. These requests include
"official" requests following standard processes and "under
the table" work which passes directly between the requestor and
implementor. Identifying and quantifying the "under the
table" work is a difficult, yet important, challenge in setting
the initial SLA. Since it is not officially sanctioned, this work is
invisible to client managers and fails to be included in the SLA. This
failure is often the root of later client dissatisfaction with the
outsourcer. Another factor is pre-existing defects. Inevitably, the
client's processes, portfolio of applications to supported, etc., contain
some level of existing defects. These defects influence the ability of the
outsourcer to meet its work product quality commitments. While correction
of these defects can be negotiated into an outsourcing contract, a smart
service provider will want to quantify them before establishing delivery
commitments.
Responsiveness, efficiency and volume of work produced
are under the control of the outsourcer, yet the client typically sets the
baseline standards of acceptability. It is the outsourcer's responsibility
to determine if they can profitably meet these client-set commitments.
While the outsourcer controls efficiency (cost per unit production), it
cannot control costs unless volume is fixed. A backlog is created
when the volume of work requests exceeds the maximum volume of work
produced (capacity). Reworks are the number of outsourcer produced
work products that fail to meet quality requirements and must be redone.
Factors within the outsourcing box, including number of
tasks, task efficiency, work efficiency, internal rework levels and
staffing costs, are typically fully under the outsourcer's control. By
manipulating these items, an outsourcer influences its costs, capacity,
responsiveness and ultimately, profitability. While extremely important to
successful outsourcers, these categories of measures are generally not
included in a SLA and are not explored in this article.
Categories of SLA Metrics
When setting up a SLA to control and manage the factors
described above, there are many possible metrics from which to choose. The
simplest way to approach these metrics is to group them into categories,
decide which ones in a given category work best for the particular
project, and then construct the desired metrics.
The key factors can be managed through four major categories of SLA
metrics:
- Volume of work
Volume of work is typically the key sizing determinant of an
outsourcing project, specifying the exact level of effort to be provided
by the service provider within the scope of the project. Any effort
expended outside of this scope will usually be separately charged to the
company, or will require re-negotiation of the terms of the SLA. Broadly
defined as the number of units of a work product or the number of
deliverables produced per unit of time, volume of work metrics should be
specified for every major deliverable cited in the SLA. Pick the
simplest volume metrics possible to ensure consistent results. More
complex metrics, such as function point-based volume measures, can be
difficult and costly to obtain for many organizations, and risk
inconsistency and subjectivity. Projects that are billed on a time and
materials basis may discuss volume in terms of number of resources,
while a fixed price project will generally specify volume of
deliverables. Example metrics include number of support calls per month,
number of maintenance requests per month, etc.
- Quality of work
Quality metrics are perhaps the most diverse of all
of the SLA metrics. They cover a wide range of work products,
deliverables and requirements and seek to measure the conformance of
those items to certain specifications or standards. When deliverables
fail to meet the acceptance criteria in the specifications or standards,
quality problems arise. It is best if each major deliverable contained
in the SLA has a corresponding acceptance criteria to judge the quality
of the deliverable. When that is the case, quality of work can be
expressed positively (% of deliverables accepted) or negatively (% of
deliverables rejected).
A quality definition may contain several, individual metrics that may
form part of the deliverable's acceptance criteria, or that may serve as
standalone measurements of a single aspect of service. Briefly, these
metrics include:
Counts or percentages that measure the errors in major
deliverables, including number of production failures per month,
number of missed deadlines, number of deliverables rejected (reworks),
etc.
Internal standards for application source code, documentation,
reports and other tangible deliverables, including number of
enhancement tasks passing standards reviews, number of documented
programs, etc.
Measurements of the technical quality of application code, normally
produced by commercial tools that look at items such as program size,
degree of structure, degree of complexity and coding defects. The
actual metrics depend on the tool used, but may include items such as
McCabe Cyclomatic Complexity, McCabe Essential Complexity, average
program size, etc.
The amount of time/window of time that the services managed by the
outsourcer are available, ranging from on-line application
availability to delivery of reports by a specified time-of-day.
Measures can be reported positively or negatively, and usually
incorporate some level of tolerance. Examples include on-line
application availability 99% of the time between the hours of 08:00 AM
and 06:00 PM, etc.
The client's level of satisfaction with the perceived level of
service provided by the outsourcer captured for each major function
through internal and/or external surveys. Ideally, these surveys are
conducted periodically by a neutral third party. Although subjective,
they are a good double-check on the validity of the other SLA metrics.
For example, if an outsourcer meets all specified performance targets,
but receives a substandard satisfaction rating, the current SLA
metrics are clearly not targeting the right factors. Within the SLA,
metrics specify the minimum satisfactory ratings in key survey
categories.
- Responsiveness
Responsiveness metrics measure the amount of time that it takes for
an outsourcer to handle a client request. They are usually the most
important ones from the client's perspective, and figure heavily in its
perception of the quality of service provided by the outsourcer.
Responsiveness to requests often motivates business areas to seek
outsourcing in the first place. Metrics include:
- Time-to-market or time-to-implement
These metrics measure the elapsed time from the original receipt of
a request until the time when it is completely resolved. Sample
metrics include time to implementation of an enhancement, time to
resolve production problems, etc.
These metrics measure how responsive the outsourcer is by focusing
on when a request is acknowledged, and accessibility of status
information. Sample metrics include time to acknowledge routine
support calls, programmer response time to production problems, etc.
Another measure of responsiveness is the size of the backlog,
typically expressed as the number of requests in the queue or the
number of hours needed to process the queue. Metrics include # of
resource-months of enhancements, # of unresolved support requests,
etc.
- Efficiency
Efficiency metrics measure the engagement's effectiveness at
providing services at a reasonable cost. Pure cost metrics, while
important, miss the relationship between volume of work and
effectiveness of its delivery. For example, an outsourcer may commit to
handle 1,000 telephone support requests per day for a fixed price of $
20,000/day. Even if the outsourcer doubles its effectiveness, it still
handles 1,000 calls and charges $ 20,000. If an efficiency metric such
as average cost/call is used instead, the client will see a change from
$20/call to $10/call. For the client, increases in efficiency often
produce cost reductions. For the outsourcer, improved efficiency
translates into increased profits. Outsourcing arrangements often share
these benefits between client and outsourcer to encourage them to seek
efficiency improvements. Establishing a history of efficiency metrics
also enables volume adjustments. If the client later needs 1,500
calls/day, pricing is more straightforward. Similarly, efficiency can
translate into either the same volume of service for less money or a
greater volume of service for the same money. Examples of efficiency
metrics include:
This efficiency indicator is typically tied to an index that is
based upon cost per unit of work produced, and is used to document
cost reductions or increases in productivity. Sample metrics include
number of programs supported per person, cost per support call, etc.
These metrics track the workload of each team member to aid in wise
utilization of resources. Engagements that charge on a time and
materials basis should include metrics on staff utilization to measure
the effectiveness of staff deployment in order to encourage the
outsourcer to make staff reductions as efficiency is gained. Sample
metrics include % of time spent on support, % utilization, etc.
Although rework metrics are also quality measures, they can be
applied on a percentage basis to measure the effectiveness of
implementing quality improvements. These metrics track the percentage
of work products that returned to a previous step for additional work,
correction or completion. They track "wasted effort" and
help to measure the quality of a process and its efficiency. Metrics
measure rework rates for particular tasks, and for specific processes.
Reporting Metrics Information
After specifying all major deliverables and their
associated performance metrics in the SLA, the client and outsourcer must
agree on how the information is to be presented during the outsourcing
engagement. As always, simple is better. The key to effective reporting is
to present the results in actionable form. Rather than provide a long list
of metrics, summarize the results into trends. Methods such as balanced
scorecards, weigh the value of individual metrics against the overall
objectives of the project. Otherwise, a manager may be tempted to
overreact to a decline in one metric when overall trends are improving.
Typically, the parties will draft a prototype report(s),
reaching agreement on the selection of deliverables, the appropriateness
of the metrics, the ability to collect the data, and the frequency and
timing of reports. In some cases, this exercise will lead the parties to
re-negotiate, modify or eliminate certain metrics altogether. Generally,
if a metric is not important enough to directly contribute to the report,
it is not important enough to collect. Depending on the structure of the
project, a single report containing all metrics may suffice; other
projects may require multiple reports for the different application or
business areas involved.
Conclusion
Effective SLAs are extremely important to assure effective outsourcing
engagements. The metrics used to measure and manage performance to SLA
commitments are the heart of a successful agreement and are a critical
long term success factor. Lack of experience in the use and implementation
of performance metrics causes problems for many organizations as they
attempt to formulate their SLA strategies and select and set the metrics
needed to support those strategies. Fortunately, while reaching for
perfection is difficult and costly, most organizations can achieve their
objectives through a carefully chosen set of simple-to-collect metrics.
Hopefully, this paper provides some insights into the "whys" and
"hows" of this selection process.
