Skip to content

Editors

Harry Cassin
Publisher and Editor

Andy Spalding
Senior Editor

Jessica Tillipman
Senior Editor

Bill Steinman
Senior Editor

Richard L. Cassin
Editor at Large

Elizabeth K. Spahn
Editor Emeritus

Cody Worthington
Contributing Editor

Julie DiMauro
Contributing Editor

Thomas Fox
Contributing Editor

Marc Alain Bohn
Contributing Editor

Bill Waite
Contributing Editor

Shruti J. Shah
Contributing Editor

Russell A. Stamets
Contributing Editor

Richard Bistrong
Contributing Editor

Eric Carlson
Contributing Editor

Sean Griffith: The compliance metrics are in. And they prove . . . nothing

Recently the CCO of a major financial institution was telling me how his principal regulators now insist upon seeing compliance metrics. The regulators demand quantitative evidence that the firm makes good on its commitment to compliance. So the firm produces a stack of documents — “dashboards,” “heatmaps,” “scorecards,” “KRIs,” “KPIs,” and so on.

When he was done I asked how the metrics inform the firm’s approach to compliance.

“They don’t,” he shrugged. “At the end of the day, we sit around a big table and talk about risk.” In other words, his people quantify compliance risk for their regulators, then de-quantify it so that they can actually do their jobs.

This this may sound shocking, but I’m guessing it comes as no surprise to readers of the FCPA Blog. In fact, I had heard another top CCO confess the same once before:

We do have our metrics around surveillance and testing, but in the end, do we know if we have an effective program? We haven’t figured that out yet. We do know we have a program in size. We just don’t know if it works. We do know that for purposes of the federal sentencing guidelines we have a program that ticks all the boxes. We’ve had independent law firms come in and validate that for us. We do know how our size compares to others …. [But] in terms of … impact on the organization …? Don’t know.

Firms have lots of compliance metrics. To evaluate program effectiveness, compliance departments log internal audit findings, track hotline calls, monitor training completion rates, record the speed and outcome of internal investigations, and run self-assessments, surveys, and peer comparisons. Then they hire outside professionals to review it all and tell them that they are doing more-or-less what everyone else is doing. Yet in one study, only 52 percent of CCOs surveyed said they were “confident” or “very confident” that the metrics used by their organization gave them a true sense of the effectiveness of the compliance function.

We don’t have a good way of measuring whether compliance actually works. Part of the problem is that much of what firms measure is backward-looking rather than forward-looking, and as anyone who has been burned in the stock market knows, part performance is no guarantee of future returns. Moreover, many compliance metrics track activity rather than impact, thereby demonstrating that compliance may be busy but not necessarily that it is effective.

As I argue in Corporate Governance in an Era of Compliance, the uncertainty around effectiveness gives rise to two challenging questions.

First, why should prosecutors give firms any credit for employing compliance mechanisms whose effectiveness has not been proven?

And second, why should prosecutors impose unproven compliance mechanisms on firms?

The lack of solid, empirical knowledge should inspire humility in enforcement authorities, as indeed it has in CCOs, at least in their quiet moments. We are in a learning period with regard to compliance. Enforcement and regulatory authorities are not wrong to insist on metrics. But unless and until they can reliably sort good programs from bad, they are wrong to impose a particular vision of compliance or a particular set of reforms.

The full article is available for download here.

____

Sean J. Griffith is the T.J. Maloney Chair in Business Law at the Fordham University School of Law. He also serves as Director of the Fordham Corporate Law Center. Professor Griffith is a graduate of Sarah Lawrence College and received his law degree magna cum laude from the Harvard Law School, where he was an editor of the Harvard Law Review and a John M. Olin Fellow in Law and Economics.

Share this post

LinkedIn
Facebook
Twitter

5 Comments

  1. So True. Bean counters make themselves happy, and even get a warm fuzzy from metrics, but reliance upon metrics is misplaced in its exclusivity.
    So much is missed through such reliance, and when the crap hits the fan they wonder how it could have happened. How much valuable time is wasted checking the boxes (and drafting the boxes in the first place).
    My saying is: Too much business school and not enough business.

  2. Hi, Sean – There is a famous quote from Einstein that not everything that can be counted counts, and not everything that counts can be counted. Many in compliance and ethics count what is countable, and do not want to do the more difficult work of digging in, talking with people, and getting the real pulse of the company. To the extent they live by “dashboards” and “KPI’s” they are inviting trouble.

    One of the measurement tools I recommend is the deep dive, where the compliance person digs deeply into specific business units to get a feel for what is happening. The same point applies for measuring culture. People use surveys, but without recognizing their limits. Again, it is necessary to dig down. I fundamentally believe you can learn much about a company, including about its culture, by just going to lunch with some of the workers and listening to them.

    On the other hand, government is right to push companies to implement compliance programs, but they should not take a DIY approach. There is no reason for prosecutors and regulators to make things up out of the ether. The US Sentencing Guidelines take a well thought-out approach, and are a good model for companies to use. Unfortunately, few people actually bother to read all that is in them and apply what they actually say. In reality, they are essentially a project management approach that works if people follow them. For example, they call for the use of incentives, yet rarely if ever do companies apply this in any meaningful way.

    Like it or not, the real driver in the development of compliance programs has been the fact that government offers incentives. What is needed is for government to do this in a more serious way. Enforcers and regulators need to make sure they know what they are doing in assessing programs, be very public about what gets credit and what does not, and make it extremely clear that only strong programs that take the Sentencing Guidelines standards seriously will get credit.

    There are many more interesting points about this, but I have tried to keep this quick response short. One additional point: for companies that are looking for outside law firms to assess their programs, always ask this question first: do you, the huge, global law firm, have a compliance program yourself? If they do not, then it is tough to assume they actually know enough to assess your program.

    Cheers, Joe

  3. Good afternoon,

    From a European perspective: This very true. In my experience as CCO of two internationally operating financial institutions in Europe, I often questioned the relevance what we were doing to evidence the effectiveness of the compliance function.

    Much was dominated by template fever generated by other risk functions in the firm. Most of it involved reporting activity and process, but it inadequately provided for scenarios of emerging risks in a fast changing environment nor did it question the acceptability of the decisions the firm was making.

    It certainly did not identify root causes nor record whether there was any meaningful improvement, which is I think the key to being effective, prompting me to suspect that we were missing the point.

    I agree, part of it is what regulators require, but there is also a deeply ingrained template culture and supporting methodology which we have borrowed from other risk functions predominantly operational risk which provide with a false feeling of safety in a forest of paper. This has pushed us in a certain direction and it is time to move in a different direction.

    The bad news is that EU regulators are increasingly data driven which is fine as long as you do not throw good judgement out of the window.

    The good news is that an experienced CCO will be able to take these hurdles by adding his or her's authoritative insight and opinion.

    As a CCO you need to speak your mind.

    On one occasion a member of a Supervisory Board asked me after going through syrupy templates on an extensisive AML/Corruption file review, what I thought. My answer: The business model for these customers is unsustainable.

    Regards

    Nico Zwikker

  4. Sean,

    A story after my heart! Executives and Regulators love metrics, but understanding them, and using them to improve compliance, is a different matter. Take the classic discussion I had countless times. If there is an increase in Whistles being blown is that a bad thing or a good development? Tell me what answer you want and I can argue it for you!
    Compliance is like a good insurance policy. You don't know how solid it is until you have a problem. And then it's too late. In reality, compliance is about talking to the business continuously, finding out what they are planning or doing and making sure that they think about compliance risks UPFRONT and mitigate against these. Sometimes it's ok, sometimes it needs some tweaking and sometimes you have to convince them business that the ultimate mitigation (abandoning the plan) is the right way forward. Those cases are rare, thankfully. Therefore the only measure that really tells you in-depth about your compliance programme may be 'effective face to face time'with the business you support. And to be taken serious as a helpful business partner, this requires you to understand their business.


Comments are closed for this article!