Methodology update: adjustments to CIVICUS Monitor Ratings
5th March, 2018
The CIVICUS Monitor has just released an updated set of country civic space ratings. These new ratings are based on adjustments made to specific aspects of our research methodology. As such, they do not reflect specific improvements or declines in civic space conditions on the ground in recent months in the countries affected by ratings changes.
Most of the adjustments that have been made involve countries for which the data indicated a situation close to the boundary between two ratings, for instance open and narrowed. In summary, while a number of country ratings are now different, the overall trend is substantially similar to that shown by the previous set of ratings. A list of new country ratings is included at the end of this document.
Below, you can read more about why we did this, what the adjustments are and how they impact our overall ratings. We encourage everyone to send us their questions to email@example.com and we will be happy to provide any further information you require.
You can also read more about the details of our methodology here.
What has actually been adjusted?
Adjustments have been made to two aspects of how data is combined within our methodology:
- We have adjusted the mathematical formula used to take “raw” data inputs and apply them to a consistent scale of 1 to 100 points upon which our ratings calculations are all based (see below for the scale).
- We have also adjusted the online database to remove the rounding of figures at certain stages of our calculations. Ratings are now based on exact figures and not rounded totals.
What effect do these adjustments have on the ratings on the CIVICUS Monitor?
These adjustments mean that some countries’ ratings have changed. The before and after picture is as follows:
For a full list of countries with new ratings, please see the end of this page.
Why did we make these adjustments?
These adjustments are part of the continuing Beta phase of development of the CIVICUS Monitor. The adjustments were agreed upon following a thorough review by in-house civic space experts and an external quantitative research methodology expert which involved an in depth examination of the finer detail of our initial civic space datasets. We are confident that these adjustments now mean that our ratings more precisely reflect realities on the ground in each country and will thus be more reliable over time.
Why are we making these adjustments now?
Having had ratings for all countries since April 2017, we felt that the second half of 2017 was the right time to review how our methodology was being applied to real data. We wanted to make any necessary adjustments to calculations as early as possible in the life of the Monitor so that we can generate reliable and comparable time series data from now on.
As outlined in our guiding principles, the CIVICUS Monitor is being developed through an intentionally iterative process of conversations with experts and civil society groups from all over the world. As its development continues, CIVICUS and its research partners will continue to seek feedback and strengthen our research in line with advice and feedback received from a growing cohort of users. While we are primarily focused on generating an international dataset to monitor global trends, we are also mindful of the need to be flexible and ensure that the Monitor is useful at the country level - where civil society groups may wish to use the data to further their own advocacy initiatives. In this, the CIVICUS Monitor research team will strive to remain flexible, listen to our membership and strike a balance between the competing priorities of all users.
What verification was done?
As is the case for any ratings update, the CIVICUS Monitor research team carried out a thorough assessment of the appropriateness of these new ratings. This involves an independent verification check by our Advisory Panel, as well as a review by our research partners, with whom we work closely to continuously update country information on the Monitor.
What about the old ratings?
For the vast majority of countries, the old ratings remain the same as the new ones. 27 out of 195 countries now have different ratings. What has changed is the scores (a number between 1 and 100) which lies behind those ratings. While these adjustments mean that scores for all countries have changed, most of those changes are small - in fact the majority are within 5% of their original score.
Because we use broad ratings categories on the monitor, ratings (ie whether a country is Open, Narrowed, Obstructed, Repressed or Closed) for most countries do not change. Thus, while ratings for some countries are changing, for most, ratings remain the same.
We also see that, on the whole, the new ratings confirm the trend described by our previous set of ratings. Globally, the outlook for civic space remains bleak, with 109 countries in the obstructed, repressed and closed categories; and just 4 out of every 100 people living in countries with open civic space.
Why are there so many more open countries in this latest set of ratings?
Prior to these adjustments, there was a group of countries which had civic space scores that were close to the numerical boundary between narrowed and open. The effect of the two adjustments we are implementing is that the scores for each country get “stretched” across a more even 1 to 100 scale.
Therefore, when these adjustments were applied, it meant that a lot of countries were “pulled” from the narrowed to the open categories. Again, this does not mean we are saying that civic space has improved in these countries. Instead, we now believe that this new set of ratings represents a closer reflection of the conditions for civic space globally.
Do these adjustments mean CIVICUS believes civic space is improving?
No. This update to CIVICUS Monitor ratings is not reflective of either an improvement or a decline in overall civic space. The ratings released today are based on the same data as those released in September 2017, the only difference being that new calculations are being applied to that data.
Will CIVICUS continue to make adjustments to its methodology?
As explained above, the CIVICUS Monitor is being developed through an intentionally iterative process of conversations with experts and civil society groups from all over the world. Having said that, we believe that, fundamentally, the core structure of our methodology, and therefore the basis of our ratings, is sound and will not change.
We are confident that these adjustments are an important step forward in making sure that our analysis of civic space reflects as closely as possible the situation on the ground, particularly across varied contexts and over time. While we do not envisage any further adjustments in the immediate term, CIVICUS will continually reflect on the data being generated by the Monitor and will ensure that its methodology includes the latest thinking and data sources on civic space.
When will ratings be updated next?
CIVIUS Monitor ratings are dynamic, meaning that they can be updated at any time, based on new data on civic space that we become aware of or in response to events on the ground that merit a ratings change. These changes are always made in consultation with our independent advisory panel and with our research partners. The CIVICUS Monitor intends to develop a mechanism so that users are able to see how these ratings change over time.
How can I find out more?
The CIVICUS Monitor team would love to hear from civil society activists, academics or anyone interested in learning more about our research methodology. Please write an email to firstname.lastname@example.org and we will respond to your questions or set up a meeting if there is something you would like to discuss with us in detail.
List of countries with new ratings:
|Country||Old Rating||New Rating as of 5th March 2018|
|Antigua and Barbuda||Narrowed||Open|
|St Kitts and Nevis||Narrowed||Open|