Score Ukraine Methodology

What is SCORE?

Compared to the resources dedicated to war and conflict, the financial and human resources available to invest in peacebuilding is significantly limited. Effective, efficient and strategic use of limited resources to influence positive change in conflict transformation and sustaining peace necessitates a deep grasp of conflict drivers, as they manifest in each specific context, in order to design appropriately targeted and cost-effective programmatic interventions. Peacebuilding and development programs are all too often designed on the basis of limited inputs from a small number of in-country experts or conflict analysis consultants, with insufficient data-driven insights. Evidence-based decision and policy making is absolutely essential to improve peacebuilding and conflict transformation efforts, and in fostering social harmony both vertically (with governance institutions and citizens) and horizontally (between different groups in society). Yet, existing assessment approaches are often challenged when it comes to empirically linking potential drivers of conflict with desired peace outcomes in a way that can help resolve policy and programmatic dilemmas. As such, genuine efforts and investments can often miss the mark in producing the outcomes and societal change they seek to influence.  

SCORE seeks to improve the effectiveness of peacebuilding efforts based on evidence, and was designed to address the abovementioned deficits in informing sustainable processes of conflict transformation. As a customizable, flexible and evidence-based diagnostic and predictive assessment instrument, SCORE can be utilized to identify programmatic entry points which are most likely to have a positive impact on peacebuilding outcomes. From this perspective, SCORE speaks to the concerns of the peacebuilding community, often comprising national and international actors who are still uncertain about the real efficacy of their investments. SCORE offers governments, donors and peacebuilding organizations the opportunity to systematically design and test conflict transformation theories of change before sponsoring and endorsing new peace-building programs.

SCORE draws inspiration from multiple scientific disciplines such as sociology, psychology, international relations and security studies and is flexible enough to incorporate new research findings, global policy guidelines and the realities of each local and regional context. It was developed in 2012 in Cyprus by the Centre for Sustainable Peace and Democratic Development (SeeD) in partnership between UNDP and funded by USAID. The backbone of SCORE is founded upon participatory research based on mixed-methods, where multi-level stakeholder consultations, focus groups and interviews are conducted to inform the calibration of the SCORE questionnaire, which draws from the extensive SCORE library of measurement instruments and indicators. Figure 1 below illustrates the SCORE process cycle.

Figure 1. SCORE Process Cycle

The SCORE Index eschews the one-size-fits-all approach that often characterizes cross-national indices. To ensure that each SCORE Index reliably captures the societal dynamics of the specific country where it is being implemented, the process typically begins with inclusive consultations with a broad cross-section of national stakeholders such as civil society, academia, government, business leadership and grassroot communities. These contribute to an initial in-depth understanding of societal dynamics in relation to outcomes of interest, while the technique of system mapping (detailed and integrated conceptual models) is utilized to capture and validate complex inter-relations between different societal, economic, political and individual dynamics and components. With such a conceptual framework designed, using inclusive and participatory dialogue techniques, appropriate measurement instruments are then selected, adapted and designed – either from the existing library of SCORE instruments or custom-made for the specific country.

The SCORE Index can flexibly integrate different modalities of data collection as required, including surveys, discourse analysis, expert assessments and draws its strength from advanced analytical and statistical toolkits. Large SCORE sample frames are then designed in a way to ensure that results can be reported with a high level of confidence for different sub-regions within the country, but also for distinct societal groups of interest (e.g. specific ethnic communities or social demographics such as internally displaced communities). The actual fieldwork is usually conducted in collaboration with established national researchers or research agencies, who can display the needed level of cultural and contextual awareness and sensitivity to assure a reliable data collection process. Results are processed using advanced data analysis techniques (e.g. factor analysis, anova analysis, regression and structured equation modelling) from which robust metrics are designed for multiple indicators, which are translated into network analysis and predictive models that can reveal the intricate relationship between different indicators and groups. This modelling process is used to suggest effective entry points to design evidenced-based projects and policies.

Using participatory research principles, preliminary SCORE results are interpreted through multi-level stakeholder consultations and dialogue groups, which informs further data analysis and design of participatory policy briefs. Such a close cooperation and communication with the key stakeholders as well as grassroots not only adds richness, depth and value to the SCORE findings but also builds ownership and triggers constructive public debate ensuring that the final SCORE products are as contextual and as impactful as possible. The analysis and recommendations presented in this booklet is based on this extensive and inclusive process framework.  

 

The SCORE Vocabulary

  • Simplified conceptual model: Theory of change and system map of different indicators, assumptions, and societal dynamics designed based on multi-level stakeholder consultations simplified into a concise illustration. Initial conceptual models use fuzzy cognitive mapping and system mapping tools (e.g. mental modeller) to map assumptions and hypothesis about multiple theories of change that guide the design of the SCORE indicators and questionnaire.

  • Outcomes of interest: Identified desirable and high priority normative objectives that relate to assessing and fostering social cohesion in a given context (i.e. intergroup harmony, mitigating violent tendencies, fostering a human rights ethos). These outcomes are listed under the dimensions section of the SCORE Platform but should not be confused with categorical/thematic groupings. The outcomes of interest that make up the key components of social cohesion for Moldova are illustrated in the figure above. The other dimensions listed on the SCORE platform are categories that group different indicators thematically.

  • Indicators: The components of the conceptual model are translated into metrics and indicators that are quantifiable and measurable via public opinion polls. Each indicator that is measuring a particular phenomenon (e.g. economic security, discrimination towards out groups, belief in human rights, support for certain policy options, post-traumatic stress disorder, etc.) is usually assessed with minimum 3 questionnaire items, scaled following reliability tests, to ensure that the SCORE can robustly capture different dynamics underlying the given indicator.

  • Drivers/Predictors: Indicators that have a strong positive or negative impact on the outcome of interest are called drivers or predictors. They provide strategic entry points that hold the most likelihood of impact on the desired outcome of interest. These can be seen under the path analysis section.

  • Heatmaps: A score is calculated for each indicator. The scores range from 0 to 10, where 0 means that the phenomenon the indicator is measuring is not observed in the context at all, and 10 means that it is observed strongly and prevalently. Heatmaps demonstrate the regional differences of these scores in order to identify areas of concern and tailor interventions more precisely. For example, the Personal Security indicator is measured through the following questions:

    • To what extent do you feel safe from violence in your daily life?

    • To what extent do you feel confident that the police or other institutions can protect you from violence?

    • To what extent would you feel safe walking alone in the street at night?

A score of 0 for personal security would mean that no one in a given region feels secure at a personal level, while 10 would signify that every person feels absolutely secure. As such, a heatmap of personal security for Ukraine illustrates the scores for each oblast, allowing for an understanding of the regional variance in terms of personal security across the country.

  • Path analysis (predictive models): Based on advanced statistical analysis including regression, network analysis and structural equation modelling, predictive models investigate the relationship between different indicators and the outcomes of interest. Predictive models reveal those indicators that may have a reinforcing or mitigating influence on other indicators as well as the outcome. While the first wave of the SCORE can be used for identifying directional correlations and benchmarking, second and third waves of the SCORE, where temporal comparisons are possible, can help identify trends and causal relationships. The colour of the connecting lines represents the nature of the relationship - blue symbolizes a positive correlation, and red symbolizes a negative correlation. The thickness of the lines represents the strength of the correlation – the thicker the line, the stronger the relationship. This section provides a brief preview of the SCORE methodology. To explore the SCORE methodology in more detail, you can download the SCORE book or explore other SCORE publications.

 

SCORE Ukraine Phase Two Overview and Methodology

In 2015, SeeD was invited to implement the SCORE Index in Ukraine, by the USAID/OTI supported program “Ukraine Confidence Building Initiative” (UCBI). To address the issues underpinning community tensions and cohesion, SeeD in partnership with UCBI implemented the Phase One of the SCORE in Ukraine between August 2015 – December 2016. SCORE Phase One succeeded in developing evidence-based and participatory policy briefs and communication messages, and mobilized significant interest among a wide range of key stakeholders (e.g. senior policy makers, journalists, civil society).  

USAID and SeeD acknowledged that it is crucial to conduct longitudinal and comparative research that can provide temporal analysis in order to monitor situations in different settings. Therefore, a SCORE Phase Two was calibrated in the summer of 2017 which aimed to:  

  • Monitor prevailing societal and inter-group dynamics;

  • Detect human security threats that different communities are experiencing;

  • Investigate how alternative policies would enhance or undermine social cohesion, civic participation and sense of security in Ukraine;

  • Provide a deeper investigation into zones of crisis, priority issues or specific societal tensions identified in Phase One.

SCORE Ukraine Phase Two was calibrated in the summer of 2017 with stakeholder consultations and 8 regional focus groups conducted in Kherson, Kharkiv, Kramatorsk, Kyiv, Ivano-Frankivsk, Lviv, Severodonetsk and Odessa. Following the completion of the calibration process and conceptual modeling, the Phase Two questionnaire was designed and fieldwork was deployed in January 2018 by Growth from Knowledge (GfK) Survey Company. Figure 2 below summarizes the conceptual model and areas of investigation for SCORE Ukraine Phase Two process.

Figure 2. Simplified Conceptual Model for the SCORE Ukraine Phase Two

The Phase Two sample frame included a strong pane sample which allowed for robust and unique analysis of causal relationships and changes in civic attitudes. In total 9,018 face-to-face household interviews in 24 oblasts and Kyiv city were conducted via random stratified sampling. The overall reach of panel respondents amounted to 81.4% of the total sample. In other words, 6,102 respondents participated Phase One were recruited to participate in Phase Two. Furthermore, 1,042 interviews were conducted in non-government-controlled areas (NGCA) of Luhanska and Donetska oblasts via random stratified sampling. Due to accessibility challenges, 445 face-to-face interviews were conducted in NGCA of Donetska and 597 telephone interviews in NGCA of Luhanska (via random generation of phone numbers) were completed in total. In sum, the findings presented in this policy booklet is based upon focus group and stakeholder consultations and quantitative fieldwork with 10,060 respondents nationwide including NGCA of Luhanska and Donetska oblasts