Gender Equality in Academia and Research
Step 5: Monitoring progress and evaluating a Gender Equality Plan
Monitoring and evaluation are important parts of the process of change. As you know by now, a gender equality plan (GEP) will typically address several issues at once, leading to a complex set of measures. Nonetheless, effective monitoring and evaluation instruments are often lacking, which undermines the transformative potential of the planned measures. If objectives are not indexed on relevant progress, success or outreach indicators, it is difficult to assess whether the organisation is actually being transformed. This might also reduce the commitment of stakeholders to meeting those objectives. Having an appropriate monitoring and evaluation plan in place, however, can support the effective implementation of measures, ensure accountability, and enhance your knowledge and understanding of ongoing changes. This way, you also know whether your GEP requires adjusting.
Besides these logical arguments for thinking about monitoring and evaluation from the very beginning, it is also a GEP requirement for participating in Horizon Europe. You need to be aware that, in order to be eligible for Horizon Europe, ‘it is mandatory that organisations collect and publish disaggregated data on the sex and/or gender of personnel (and students, where relevant) and carry out annual reporting based on indicators’ (see Horizon Europe Guidance on Gender Equality Plans, pp. 23–27). While this step comes only after planning and implementing your GEP, as laid out in the step-by-step guide (because that is when you start monitoring the effects of your measures), you need to know that the monitoring and evaluation strategy needs to be set out beforehand.
Ideally, you considered which areas you wanted to focus on in step 2 when analysing and assessing the status quo in your organisation. In step 3, you then set out specific, measurable, attainable, realistic and time-related (SMART) targets and measures addressing these areas. In order to develop a monitoring and evaluation strategy, use the status quo assessment as a starting point. The results of this assessment will establish the baseline, which will allow you to monitor and evaluate your progress.
Understand the basics of monitoring and evaluation
In order to develop an effective monitoring and evaluation strategy, you need to differentiate between monitoring targets and evaluation targets. In order to understand the difference, consider the following definitions, used by the gender equality monitoring tool (pp. 3–8) of the EU project ‘Taking a reflexive approach to gender equality for institutional transformation’ (TARGET).
The tool defines monitoring as a continuous process, in which data is systematically collected in order to provide management and key stakeholders with regular updates on the progress and achievement of objectives and the use of allocated funds.
Evaluation, on the other hand, relates to a systematic and objective assessment of an ongoing or completed project, programme or policy based on the monitoring data, providing lessons learnt for the planning of future measures.
In other words, ‘Monitoring ensures that the right thing is done, while evaluation ensures that the right outcomes are achieved’ (TARGET gender equality monitoring tool, p. 3), which means that the two go hand in hand. But as you can see from the definitions above, their specific targets differ: the monitoring targets focus on the specific outputs and related processes (implementation level), while the evaluation targets relate directly to the targets set out in your GEP (i.e. to the impact or outcome you wish to achieve) (strategic level). This is also why the respective time frames differ: while an evaluation is a more in-depth analysis, usually conducted at the end of your GEP or funding cycle (of course, interim evaluations are possible), your monitoring targets will be assessed at much shorter intervals, to inform you about the progress. Remember that Horizon Europe requires annual monitoring reports.
The factors that need to be considered in order to identify monitoring targets also generally apply to the evaluation level. However, keep in mind that achieving the desired outputs (as shown by the monitoring) does not necessarily lead to an achievement of the desired outcome or impact (as shown by the evaluation). This might be the case, for instance, if the measured output to increase the number of teachers who participated in gender training to a certain percentage was reached, but the training was not effective enough to ensure the desired outcome of increased gender competence among teachers.
The monitoring and evaluation indicators that you will identify are concrete variables that you can measure in order to assess if a monitoring or evaluation target was reached. For more information on how the different dimensions relate to each other, consult the gender equality monitoring tool directly.
Create a monitoring and evaluation strategy
Now that you know about the basics of monitoring and evaluation (and the difference between the two), you can think about a concrete strategy. Note that it is always helpful to involve people who have experience with monitoring and evaluation in this process. You can also check how it was performed in other organisations, by looking at the examples in the (GEAR) action toolbox.
When coming up with a strategy, use the logic model you identified in step 3 (embedded in a theory of change): the impact pathway of the measures implemented will help you identify what you want to monitor and evaluate. As for the details of your strategy, you need to differentiate between monitoring and evaluation. Consider the following steps in order to come up with a monitoring strategy.
- Identify concrete output indicators. For this purpose, take a look at your GEP and your status quo assessment and identify output indicators for each of your measures. A list of potential indicators is provided below. Consider that the collection of the relevant data needs to be feasible with the resources available to you.
- Select appropriate data collection instruments. Know that, in general, these will be the same instruments used in step 2 for your status quo assessment. Broad data will be available from the funding process, other (internal) data might be available on a regular basis from the human resources department, still other data you might have to collect yourself. For the data you need to collect yourself, you could conduct an annual survey of staff to monitor change, for instance. The gender equality audit and monitoring (GEAM) tool provides ready-to-use surveys (implemented via the LimeSurvey) for this purpose, including a specific part for funding bodies.
- Come up with a time frame. Your monitoring should take place annually, including annual monitoring reports published on the organisation website to meet the Horizon Europe requirements.
- Plan regular monitoring sessions. Involve the core and/or extended team responsible for your GEP. These meetings can be crucial for reflecting on the progress by looking at the monitoring data and exchanging experiences. This way, you will be able to react to potential issues and steer your measures in the desired direction.
As for creating your evaluation strategy, the general process will be similar, but some things need to be considered in more depth.
- Think about the context. When planning the evaluation of your GEP, you need to consider the context of your organisation. Relevant context factors were discussed in step 1. As the evaluation is more extensive than your annual monitoring, you need to consider, in particular, the time and (human and financial) resources available to you. These will also depend on the type and size of your organisation.
- Identify additional impact indicators. While you will consider output indicators in your evaluation, it will also focus specifically on the impact of your implemented measures. Make sure to include both quantitative and qualitative indicators, as some measures cannot be properly assessed by looking only at quantitative figures (e.g. the participation of women in senior positions, the number of cases of sexual harassment reported to / dealt with by the relevant body).
- Use additional (qualitative) data collection instruments. In your final evaluation, you may want to add additional data collection instruments, such as individual interviews, focus groups, participatory workshops, document analyses or participant observation. These qualitative techniques will allow you to gain a deeper understanding of the impact of your measures.
- Take your monitoring results into account. You will carry out your evaluation at the end of your GEP cycle. In your final data analysis, also include the results of your monitoring process.
Keep in mind that monitoring and evaluation should not be a huge burden for you, but that some time needs to be invested to ensure the success of your measures. When drawing upon external expertise to carry out your monitoring and evaluation processes, we recommend bringing together these external evaluators and the people in charge of implementing change within the organisation to ensure that instruments are adapted to your goals and constraints.
For additional resources on how to plan your monitoring and evaluation process, take a look at the tab ‘Tools and resources’.
Identify quantitative indicators
A list of quantitative indicators (as suggested by the Horizon Europe Guidance on Gender Equality Plans) was provided in step 2. The same indicators should be considered for continuously monitoring progress in your organisation. Of course, the list is not exhaustive and indicators should be selected to fit your specific targets and objectives. Furthermore, the indicators in the list below may have to be adjusted to the context of your planned measures:
- staff numbers by gender at all levels, by disciplines, by function (including administrative/support staff) and by contractual relation to the organisation;
- average numbers of years needed for women and for men to make career advancements (per grade);
- wage gaps by gender and job;,
- numbers of women and men in academic and administrative decision-making positions (e.g. boards, committees, juries);
- numbers of women and men candidates applying for distinct job positions;
- numbers of women and men having left the organisation in the preceding years, specifying the numbers of years spent in the organisation;
- number of staff by gender applying for / taking parental leave, for how long they took leave and how many returned after taking the leave;
- numbers of absence days taken by women and by men differentiated, by absence motive (sick leave, care leave etc.);
- numbers of training hours/credits attended/received by women and men;
- numbers of women and men students at all levels and for all disciplines, and academic and employment outcomes;
- shares of women and men among employed researchers;
- shares of women and men among applicants to research positions, among people recruited and success rate, including by scientific field, academic position and contract status;
- shares of women and men in recruitment or promotion boards and as heads of recruitment or promotion boards, and shares of women and men in decision-making bodies, including by scientific field.
Based on the targets set out in your GEP, specific indicators need to be developed to establish a baseline and monitor progress. Such indicators help to build accountability for the success or failure of implemented measures. The ‘Evaluation framework for promoting gender equality in research and innovation’ (EFFORTI) toolbox can support you in identifying quantitative (and qualitative) indicators to measure the output, outcome and impact of your measures.
You may also want to consider breaking down the data even more and looking into additional dimensions besides gender: looking at intersectionality can include individual or group features, such as a migrant or minority background, disabilities, low socioeconomic status or at risk of poverty, sexual orientation, and so on. Pay special attention to data protection issues if you plan on breaking down the data into small groups. Consider also any national regulations on collecting personal data.
Look at the tab ‘Tools and resources’ for examples of quantitative indicators identified by other organisations and resources provided by structural change projects.
Identify qualitative indicators
Qualitative indicators are especially relevant to see whether your desired outcomes were reached. However, qualitative indicators can also give additional information on your progress and help you understand the dynamics of change (or lack thereof). Qualitative indicators may look at dimensions such as the following.
- Mainstreaming of gender knowledge. This can be measured, for instance, by the relevance given to knowledge creation on gender equality within the organisation, the institutionalisation of gender equality (in the form of dedicated programmes or departments), the dissemination of gender equality knowledge across disciplines, and so on.
- Awareness among different categories of staff and external stakeholders (reviewers, board/panel members, applicants). This can be measured by the attention given to gender equality by different categories of stakeholders through communication initiatives, codes of conduct and activities centred on gender-related aspects.
- The uptake of gender equality objectives set out in your GEP. This can be monitored by observing the participation in and acceptance of your implemented measures and the (human and financial) resources allocated to support these measures.
- The actual transformation towards greater gender sensitivity. This should focus on the effects on both formal and informal practices due to the implemented measures. It may, for instance, be shown by increased attention being given to women’s ideas and perspectives in decision-making mechanisms that are dominated by men. Consider different levels of hierarchy.
- The diffusion of a gender equality culture. This can be measured in terms of changing working conditions, but also verbal and non-verbal interactions and decision mechanisms (seating arrangements in panels). It could be reflected in changes regarding the management of work–life balance, awareness of sexual harassment and other aspects of gender-based violence, non-sexist communication, and so on.
For more inspiration and sample questionnaires, see the GENDER-NET report on qualitative indicators in research institutions. Know that qualitative indicators have a huge learning potential. They support self-reflexivity and may provide useful indications for a continuous enhancement of the implemented measures. They may also provide evidence that change happens and that gender equality and awareness are not out of sight. Techniques to collect qualitative data include individual interviews, focus groups, participatory workshops, document analyses and participant observation.
Implement your strategy and communicate the results
To help with the implementation of your monitoring and evaluation instruments, EU-funded structural change projects have developed a number of useful templates and ready-to-use resources. Switch to the tab ‘Tools and resources’ to view the full list. The ‘Gender equality in engineering through communication and commitment’ (GEECCO) data monitoring tool, for instance, provides an Excel template that includes indicators for a variety of targets, detailed definitions, formulas to calculate shares and other useful options. You may also want to take a look at the GEAR action toolbox for best-practice examples of monitoring and evaluation in other organisations.
Once you have collected and analysed the data, you can check if there have been any (significant) changes since your initial status quo assessment (baseline). You should also assess whether the monitoring and evaluation targets have been met. Discuss the results with your team and draw conclusions on what they mean for your GEP. During your monitoring, you might want to check whether you need to adjust some of your targets or the way that measures are implemented. During the final evaluation, you need to ask yourself what you can learn for the next GEP cycle.
Finally, you need to communicate your results.
- Provide annual monitoring reports as well as a final evaluation report. These reports should be published on the organisation website. The number of evaluation periods and reports depends on your individual strategy.
- Regularly update leadership/management about the results. This will be done through your annual monitoring and final reports. However, you might also want to involve them in meetings or update them more regularly, depending on your organisational structure.
- Inform other stakeholders within your organisation. It is not just the leadership that is interested in the progress of change in your organisation. Make sure to communicate the results to all relevant stakeholders. You might also want to keep them engaged by organising a meeting to present and discuss the results of your analysis (e.g. after the final evaluation of your GEP). Note that the monitoring and evaluation process is also an effective way to keep stakeholders (including leadership/management) engaged and to ensure their ongoing support for your measures. It also paves the way for the future by helping you design even more resolute measures for your next GEP.
- Consider external stakeholders. Your communication can also target policy stakeholders at regional or national level, professional associations or other institutional partners of your organisation.
When communicating the impact of your measures, know that there might be other positive side effects (or added value) of your implemented measures: the entire process may lead to a strengthened sense of community; more transparent recruitment, appraisal and evaluation procedures; stronger pluri-disciplinarity in research; and improved working conditions in general. All of this may be uncovered by your monitoring and evaluation process. In particular, your final evaluation will show the positive dynamics brought about by gender mainstreaming strategies and their inherent opportunities. Making these positive side effects visible can help strengthen your position and build the foundation for the next GEP cycle.
In order to view videos and webinars or further tools and resources on the topics discussed in step 5, switch between the respective tabs. Otherwise, click below to continue to the next step and read about how to ensure the sustainability of your measures. You can also go back to the previous step.