Recently the CCO of a major financial institution was telling me how his principal regulators now insist upon seeing compliance metrics. The regulators demand quantitative evidence that the firm makes good on its commitment to compliance. So the firm produces a stack of documents — “dashboards,” “heatmaps,” “scorecards,” “KRIs,” “KPIs,” and so on.
When he was done I asked how the metrics inform the firm’s approach to compliance.
“They don’t,” he shrugged. “At the end of the day, we sit around a big table and talk about risk.” In other words, his people quantify compliance risk for their regulators, then de-quantify it so that they can actually do their jobs.
This this may sound shocking, but I’m guessing it comes as no surprise to readers of the FCPA Blog. In fact, I had heard another top CCO confess the same once before:
We do have our metrics around surveillance and testing, but in the end, do we know if we have an effective program? We haven’t figured that out yet. We do know we have a program in size. We just don’t know if it works. We do know that for purposes of the federal sentencing guidelines we have a program that ticks all the boxes. We’ve had independent law firms come in and validate that for us. We do know how our size compares to others …. [But] in terms of … impact on the organization …? Don’t know.
Firms have lots of compliance metrics. To evaluate program effectiveness, compliance departments log internal audit findings, track hotline calls, monitor training completion rates, record the speed and outcome of internal investigations, and run self-assessments, surveys, and peer comparisons. Then they hire outside professionals to review it all and tell them that they are doing more-or-less what everyone else is doing. Yet in one study, only 52 percent of CCOs surveyed said they were “confident” or “very confident” that the metrics used by their organization gave them a true sense of the effectiveness of the compliance function.
We don’t have a good way of measuring whether compliance actually works. Part of the problem is that much of what firms measure is backward-looking rather than forward-looking, and as anyone who has been burned in the stock market knows, part performance is no guarantee of future returns. Moreover, many compliance metrics track activity rather than impact, thereby demonstrating that compliance may be busy but not necessarily that it is effective.
As I argue in Corporate Governance in an Era of Compliance, the uncertainty around effectiveness gives rise to two challenging questions.
First, why should prosecutors give firms any credit for employing compliance mechanisms whose effectiveness has not been proven?
And second, why should prosecutors impose unproven compliance mechanisms on firms?
The lack of solid, empirical knowledge should inspire humility in enforcement authorities, as indeed it has in CCOs, at least in their quiet moments. We are in a learning period with regard to compliance. Enforcement and regulatory authorities are not wrong to insist on metrics. But unless and until they can reliably sort good programs from bad, they are wrong to impose a particular vision of compliance or a particular set of reforms.
The full article is available for download here.
Sean J. Griffith is the T.J. Maloney Chair in Business Law at the Fordham University School of Law. He also serves as Director of the Fordham Corporate Law Center. Professor Griffith is a graduate of Sarah Lawrence College and received his law degree magna cum laude from the Harvard Law School, where he was an editor of the Harvard Law Review and a John M. Olin Fellow in Law and Economics.