Measuring the impact of investigative reporting
THE CASE STUDY
This case study looked at how the Organised Crime and Corruption Reporting Project (OCCRP) has evolved the way it measures the impact of its work both in terms of reporting to donors but also informing how it follows up on its investigations through further reporting. The case study also heard from one of OCCRP's donors, USAID.
Traditional monitoring and evaluation
Before setting out their current methodology, the meeting heard about how OCCRP used to focus on metrics such as whether investigations were trending on social media and website data.
These metrics can be superficial and misleading.
They also struggle to measure engagement and trust in journalism and therefore they don't equate to real impact.
These kinds of metrics also fail to acknowledge the fact that investigative reporting:
Is seen as a vehicle to address countless development goals: anti-corruption, access to information, creating an informed citizenry, democracy-building, building civil society, accountability, and many more.
Does not fit well into short term (six-month, one-year projects) and requires a long-term perspective.
“It takes 10 years to mint an investigative reporter that's really going to create change in society.”
Measuring impact by accountability
Effective monitoring and evaluation, the OCCRP case study suggests, in large part depend on having a viable, clear theory of change.
Strengths of the OCCRP approach
This approach draws on and is supported by studies - such as James T. Hamilton (2016) Democracy’s Detectives - The Economics of Investigative Journalism - which argues that every dollar invested in investigative journalism, returns about one hundred dollars in societal good.
For more resources on measuring the impact of investigative journalism see:
Limitations of the OCCRP approach
While the strengths of the OCCRP approach are considerable, the case study also identified some limitations:
You can't promise donors these kinds of results upfront -- X number of corrupt officials are fired or Y amount of illegal assets are seized -- this is why taking a long term perspective is essential.
There is no baseline or ability to know what would happen without certain investigations being published.
It obviously cannot measure intangibles, for example, the corruption that didn’t take place for fear of being caught or decisions behind prosecutions and priorities of law enforcement.
Causality: How can you say this investigative reporting created this firing or this particular change in government?
The law of unintended consequences: “Things can get worse. You get somebody thrown out and somebody worse comes in.”
Some other organisations who have adopted this approach have made claims around the impact of their work that seem hard to justify.
Opportunities from the OCCRP approach
Better systems and audits are needed to justify claims made around impact.
OCCRP follows up on its stories continues reporting on them, in order to better understand the impact of the story. This probably would not be such a focus if they were not a government and institutional donor sponsored organisation.
It was also noted in the breakout group that the way some investigative journalism organisations report results in a context where results are not expected to be achieved in a linear way has similarities to the methodology of Outcome Mapping.
Taking the long view
USAID has been supporting OCCRP since 2007 which enables them to take a long term perspective on OCCRP's growth as an organisation.
The importance of donors being patient when supporting investigative reporting was emphasised by both USAID and OCCRP.
This approach has led to OCCRP being "the most decorated investigative reporting organisation in Eastern Europe."
"Its members’ networks are regularly called upon to testify at multilateral meetings and hearings; OCCRP also advocates for safer working conditions by providing and pushing donors and policymakers to push for focussing on digital, physical and legal security; has also mentored other partners and outlets; and is an incubator of technology."
BREAKOUT GROUP
Breakout group three was attended by
6 representatives of investigative journalism organisations
3 media development donors
5 representatives of media development/journalism support organisations involved in policy and learning
and 1 independent researcher
Gender ratio: 7 Male / 8 Female
Participants selected which breakout group they wanted to attend.
Summary
The breakout group agreed that continued long-term support for cross-border, collaborative investigative journalism is essential.
The group put forward a number of areas for future research and ideas for how donors adjust how they approach supporting investigative journalism.
Advice to donors new to investigative journalism
The group welcomed the prospect of new supporters for investigative journalism but warned that considering the complexities and risks, it is essential that they collaborate with more experienced donors and listen to the advice that already exists.
Concerns were raised that the decisions of some less experienced donors have led to mistakes that are damaging to the reputation of the sector.
For example, by funding investigative reporting or programmes with media development groups and/or media who are themselves new to proper investigative reporting.
Concerns were also raised that some donors do not provide adequate support for or access to safety and security resources (including, physical, digital, legal, and psycho-social) among the grantees.
It is critical to consider such support as part of the "do no harm" principle, particularly given how risky investigative journalism, in particular, can be.
A guide for donors already exists - How to fund investigative journalism Insights from the field and its key donors - DW Akademie (2020).
And events held dedicated to this topic as recently as 2019.
Donors new to investigative reporting were encouraged to
learn from their peers and the expertise available
engage more constructively and realistically on evaluation and measuring impact
put greater priority on safety and security
Potential solutions:
Smaller or newer supporters of investigative journalism might consider teaming up with other donors to
Provide funding that is longer-term.
Pool resources and knowledge on safety, security and monitoring and evaluation.
Reduce the burden of having to provide multiple reports for multiple donors
Evaluating investigative skills
“One of the nice things about investigative reporting is that everything you need to know about an investigative reporter is in that story. And if you have an expert professional read that story, they can tell you how good this journalism is."
"The problem is many donors and many development agencies don't always have investigative reporters doing this [evaluation] and so consequently that there can be a lot of inaccurate information that's collected.”
Emphasis should be placed on evaluating the skills of the investigative journalists and organisations, looking at key indicators like accuracy, fairness, the quality and number of sources, as well as how libelous a story was.
Suggested solution:
Donors need to involve the right profile of people when evaluating investigative journalism.
As it is unlikely to be practical to bring the expertise in-house, could these skill sets be pooled between donors?
Enabling environments
Often a measure of the impact of investigative journalism is whether you are “moving the needle” or “creating enabling environments” on issues around good governance, fighting corruption, human rights and other areas in the public interest.
However, participants noted that in closed societies, this can be an unrealistic expectation considering that even measuring your audience can be problematic if your work is primarily being accessed by VPN or being shared by messaging apps.
Can a hostile action indicate impact?
One of the donors in the breakout group posed this question:
Should investigative journalists and their supporters consider negative responses to their work (SLAPPs, lawsuits, harassment) - assuming that reporting was accurate and met high editorial standards - as an indirect way to measure impact?
Some donors appear to be open to this idea but further research is needed to codify and test this.
Awards and recognition
One of the donors in the group argued that the following could, in some circumstances, be useful indicators of impact:
Receiving prizes or awards when they:
are awarded by a relevant organisation or sector
are an acknowledgement of the processes that led to the award.
Being asked to testify at meetings, hearings, or high-level conferences with the public and private sector because of their technical expertise.
Being seen by peers as a leader and an advocate for the tools and enabling environment that facilitate investigative journalism.
Developing and incubating tools and resources that are used by other investigative journalism organisations.
The multiplier effect of partnering with other IJ organisation as well as news organisations. (Because of risks involved in investigative reporting organisations only work with each other if there is a high degree of trust.)
Collaborating with civil society
Recent trends of investigative journalists collaborating with advocacy and civil society groups are to be welcomed.
It was acknowledged that there need to be strict rules of engagement to ensure that investigative journalists are not seen as activists.
However, considering the hostile environment towards investigative journalism, the benefit of collaborating with civil society is that it helps achieve some of the ultimate outcomes that donors wish to see: transparency, anti-corruption, access to information, creating an informed citizenry, democracy-building, building civil society, accountability.
Future GFMD IMPACT meetings
OCCRP will be invited to share best practice from their collaboration with Transparency International so that can be shared so others can learn from the experience.
Last updated
