The new and unique Corruption Risk Forecast at the national level launched by the European Research Centre for Anti-Corruption and Statebuilding (ERCAS) and CIPE has started to receive feedback from various sources. One of the reactions by a researcher from TRACE, who also monitors some risk indicators related to corruption, has shown a mix of misunderstanding and misinterpretation.
Aside from opinions by the author (to which he is fully entitled), he alleges that the CRF interprets trends incorrectly across time due to a flaw in the significance test. Furthermore, he questions the CRF’s decision to add a qualitative step based on the latest political facts (including war, revolutions, or radical change through elections) that impact long-term trends. Overall, he says that the CRF is too optimistic.
Questions raised have already been answered in peer-reviewed publications. The methodology as well as the country profilesfor the 120 countries covered by the CRF show transparently in each and every case why a country is forecast to progress, stagnate or regress. Still, a short recap may be helpful for the lay reader who does not have the time to read lengthy academic texts.
First, the CRF is based on the aggregated, fact-based indicators of the Index for Public Integrity (IPI), which models corruption risk as an interaction between enablers and disablers of corruption and traces these indicators as far back as they go without missing values for a core group of 115 countries (as China, Azerbaijan, and Saudi Arabia were removed due to doubts over their influence on some World Bank indicators, pending clarification). This is a 12-year interval, as fiscal transparency measurements existed in too few countries further back. Since its initial production in 2014-2016 by a grant from the government of the Netherlands, the index has increased coverage and has been the object of many peer-reviewed publications.
Secondly, the CRF intentionally offers a more granular picture of corruption risk determinants. It also avoids the problem of lagging, nonspecific perception indicators that present all continents and income groups as flat for the past 20 years and more. This working paper describes the difficulty (or rather the impossibility) of tracing corruption over time when relying on aggregated perception indicators such as the Corruption Perceptions Index or the World Bank governance indicators.
As a result, it was the avowed goal of this new instrument to capture change. The methodological choices that were made sought to increase sensitivity to change that had previously been frustrating when using perception-based indicators. For the CRF, the ERCAS research team opted not to use the overall IPI itself in the aggregate form to trace corruption across time (although values exist since 2015), but instead its disaggregated indicators. This allowed us to consider carefully if the observed trends were consistent across time or not.
With this context, lay readers can better understand why the trend analysis and threshold test is built in the way described in the CRF methodology section.
For the forecast’s Trend Analysis and Threshold Test, it is important to eliminate changes that may just be random. To do this, we compare our sample of 120 countries with a similar theoretical group of countries where average change is zero. We rate as significant any change above or below the global standard deviation of average change for each disaggregated IPI factor, compared against a control group with zero change.
We acknowledge that this is a more positive scenario than using a null hypothesis with average global change as a baseline, but more negative than using absolute change values or just setting a threshold arbitrarily. Comparing with the mean change by continent is also offered in Compare Trends.
We then rate change as consistent if a country has progressed (or regressed) in at least two indicators out of five and has not regressed (or progressed) in any. We use only five of the IPI’s six indicators because methodology changes at the World Bank means Trade Openness is not available for the past 12 years. Though additional adjustments may be made in the future, the CRF’s core approach of following the determinants of corruption and their consistent change will remain. The other factors included in the forecast analysis – the subjective political change check and the societal demand check – are fully explained in the methodology section.
We emphasize that anyone can freely use the tools to see how the results differ and arrive at their own conclusions. We publish and share ours. There is no iron law of statistics about the decision of where to place the thresholds. All these decisions have a theoretical motivation, and ours is transparently explained. We chose to increase sensitivity to reforms in the countries, but also to take in major events like Russia invading Ukraine, which would certainly affect the forecast of both, and many more.
Comments are closed for this article!