Evaluation Results

Evaluation of the Adult Education Academy 2021

External Evaluation: Katrin Riß

INTALL-Project Coordinator: Prof. Dr. Regina Egetenmeyer (Julius-Maximilian University Würzburg)

This outline summarizes the main results of the short-term evaluation of the Adult Education Academy (AEA) 2021 – until 2020 referred to as the “INTALL Winter School”. The respective evaluation step was the fifth out of six within the programme ‘INTALL - International & Comparative Studies in Adult Education & Lifelong Learning’ (2018-2021). Further details are spelled out in the evaluation report.

Since the beginning of our external evaluation in 2019, we continuously see a very positive feedback by the participants and even a rising tendency in most of the findings. All in all, the programme seems to be well geared to its target groups and successfully reaches its goals in fostering interest, knowledge and competences in the field of adult education and lifelong learning. It catches one’s eye that other than in 2020, when master students tended to rate highest, in 2021 the doctoral students seemed to be most satisfied with the AEA. Practitioners obviously were only slightly less content with the programme. Master students close the list of subgroups, still being very pleased with the AEA.

Following we will pick out some findings to illustrate this overall picture. Also, we will summarize some results regarding the effects of the AEA.

The AEA 2021 – programme and evaluation

In 2021, the worldwide coronavirus pandemic took its toll on the AEA. Due to the binding contact and travel restrictions, the programme had to be transferred from a blended learning module to a fully virtual programme. All participants and all staff attended the AEA via their digital devices at home. All lectures and group work (including excursions and socializing events) were realized via video chatting services and further online collaboration tools. Nevertheless, the organizers succeeded to implement the agenda approved during the previous year without any major changes. Some modules were further optimized (following previous evaluation results). For example, the module “Employability in adult education” was expanded and transferred to the online preparation phase.

The external evaluation continued to work with the questionnaire developed by the organizers in previous years. In 2021, the analysis is based on a total of 61 cases.

Participants’ profile

Quite some questions of the short-term evaluation deal with the participants’ profile and their run-up to the AEA. The following findings intend to give an idea of the group.

• At the time of the AEA, 46% of the respondents (28 persons) were studying for a master’s degree, 29% (18 respondents) were doctoral students, and 23% (14 respondents) were practitioners (in the field of adult education and lifelong learning).
• Again, the AEA impressed with its broad internationality. According to the registration data participants of 21 different countries of origin joined the programme, studying in 15 different countries. These included the countries of the INTALL consortium as well as others, a fact that again reveals the internationality of students studying at the INTALL partner universities.
• In general, participants apply for the AEA due to content-related reasons, due to the wish to meet student/lecturers/practitioners from other countries and due to career-related motives equally. Besides, half of the doctoral students (50% / N=18) and around one third of the practitioners (31% / N=13) indicated to have registered in the AEA 2021 because they had attended it before (respectively the predecessor winter schools). This applies to none of the master students. Also, recommendations of former participants seem to be a strong inducement to participate. Among the master and doctoral students this applies for roughly half of the group (master students: 46% / N=26, doctoral students: 56% / N=18). Also, 30% of the practitioners agreed to that (N=13).
• 59% (26 respondents / N=44) (2020: 70% / N=56) of the master and doctoral students indicated that they had received some kind of support by their home universities. 95% of them agreed that this support had been helpful (4 or 5 on the 5-point-scale, mean value 4,77 / STD 0,528 / N=22).
• Almost half of the group had not undertaken any international study mobilities before attending the AEA (49% / N=61). 38% indicated they did so once. Obviously, these shares differ between the subgroups. While for 61% of the master students (N=28) the AEA was their first study mobility, this is true for 44% of the doctoral students (N=18) and for 36% of the practitioners (N=14).
• In 2021, the share of participants who had never or only little been in touch with trans- or international topics in their study or work before was bigger than in 2020. All the same, 70% (N=54) of the respondents agreed they already had been highly interested in trans-/international and comparative adult education/lifelong learning in particular before the AEA. Also, 73% (N=51) had been highly interested in focussing on trans-/international aspects in their further studies and even 81% (N=52) in their future professional activities.
• Further, 70% of the master students (N=23) and 75% of the practitioners (N=8) stated they were highly motivated to take up PhD studies before the AEA. 81% of the doctoral students (N=16) were highly motivated to take up postdoc studies after their graduation.

Assessment of the programme

As mentioned before, the overall assessment of the AEA in 2021 turned out exceptionally well. The mean value at the question “What is your overall evaluation of the AEA?” was 4,68 (STD 0,831 / N=47) (five-point-scale, 1=poor, 5=excellent). The subgroup of doctoral students even rated with the highest possible rating at all (mean value 5,00 / STD 0 / N=15). This general picture is also confirmed by a set of questions asking how satisfied students were with the organizational, academical and didactical quality of the different programme modules. None of the modules achieved mean values lower than 4,20. Doctoral students were most satisfied. 

In the evaluation participants were also consulted, how useful they considered the different components for their own development. Except for one component all mean values lie above 4,00 and mostly range around 4,50 on the five-point-scale. Compared to 2020, most of the results turned out higher.

Effects of the Adult Education Academy

The evaluation also keeps track of the effects and benefits of the AEA as assessed by its participants. In 2021, obviously the strongest effects were achieved with the group of doctoral students, followed by the practitioners and then the group of master students.

All results regarding the effects on the thematic interest in adult education and lifelong learning as well as its international aspects turned out higher than in the previous years. All mean values ranged between 4,31 and 4,56 (N=55) on the 5-point-scale (2020: 4,04 to 4,29, 2019: 4,16 to 4,31). The effects on the motivation to undertake further study mobilities continued to be high. 90% (N=41) of the master and doctoral students ticked a 4 or 5 at the respective question. With regard to further study plans 63% of the master students and practitioners (N=32) stated that the AEA had increased their motivation to pick up doctoral studies (4 or 5 on the 5-point-scale). 87% of the doctoral students (N=15) confirmed this effect with regard to a possible post-doc research after their graduation (4 or 5 on the 5-point-scale).

Effects on the interest in adult education and lifelong learning

Further, the findings on the effects on competences related to international study and work contexts in adult education and lifelong learning resulted very high and exceeded the values of the previous year. Again, the ratings of the doctoral students turned out to be the highest among the subgroups (mean values between 4,31 to 4,75 with a relatively low spread of data points). Little lower we find the results of master students (mean values from 3,54 to 4,27).

Effects - Competences (by subgroups)

Perspectives

All in all the findings drew a very positive picture of the AEA, and it seems that the organizers successfully transferred the programme into an online concept.

In 2021 the AEA’s funding within the project ‘INTALL - International & Comparative Studies in Adult Education & Lifelong Learning’ will come to an end. Also, considering the current pandemic situation, it continues to be uncertain if the AEA – in case of further funding – will return to the face-to-face event in Wuerzburg next year. Nevertheless, before closing this report, we would like to highlight some final aspects worth mentioning or worth considering for future programmes. (1) As in the reports 2019 and 2020, at this point we will pick out the employability module. This module underwent several changes over the last years. In 2021 now, it was extended, split up in several sessions and shifted to the preparation phase. Along with these changes the participants feedback has improved since 2019 remarkably. The results in this year’s evaluation finally are on par with those of the other modules. All subgroups seem to have benefitted more than in the years before. (2) In the past evaluations, respondents often highlighted the connecting and networking effect of the Winter School/AEA. At this of course the informal parts of the programme and the socializing components did their bit, however were hard to compensate within the virtual setting. Nevertheless, the results show that the AEA obviously succeeded in its goal to foster networking among its participants. (3) On the other hand, the virtual setting also implied a number of learning opportunities. Various comments positively referred to the variety of online applications for digital teaching and learning introduced by the AEA. However, some participants also felt that this variety was too vast. (4) Some comments finally suggested to further exploit the chances of the online modus. For example, field visits to practice institutions could be extended to institutions in other countries than just Germany.

Evaluation of the Winter School 2020

Summary of the external evaluation results
Author: Katrin Riß
INTALL-Project Coordinator: Julius-Maximilian University Würzburg, Prof. Dr. Regina Egetenmeyer

The Winter School Programme 2020

In February 2020, the INTALL-Winter School was held for the second time. Compared to the previous year, three main modification were applied, which again proves the programme’s adaptability and the organizers’ constant effort to improve the Winter School according to evaluation results and observations made. First, the heterogeneous interests and needs of the target group of practitioners were taken more into account. By opening both Winter School weeks to the group of practitioners, more options were given especially to those who were interested most in dealing with theoretical approaches of adult education and lifelong learning (week 1). Second, in 2020 the organizers provided a more differentiated and extended programme regarding the topic of employability, a component that stuck out in the 2019 evaluation. And third, responding to the considerable popularity of the Winter School, the organizers increased the number of participants to a total of 83 (during week 2) compared to a total of 67 in 2019.  

Approach, data basis and perspectives of the external evaluation

Some alterations aside, the short-term survey 2020 continued to take a quantitative approach. The survey was based on the very questionnaire used in the evaluation cycle 2019, implemented as an online survey at the end of the Winter School. In total, 70 questionnaires were filled in, which sum up to a response rate of 84%. To give a more detailed picture, in 2020 the study often differentiates between the subgroups of the survey’s population, i.e. the master students, the doctoral students and the practitioners.

Sample/group characteristics

The group of participants was again particularly characterized by its great internationality. The number of countries of the students’ home universities or employers totaled to 16 in 2020, a little more than half of them were European countries (56%). Further the participants backgrounds included 33 different countries of origin and 26 different countries of present residence. This range again far exceeded the consortium of the ten INTALL partners and reflects the considerable internationality of the INTALL partners as well as the remarkable outreach of the programme.

In 2020, a little less than half of the respondents were studying for a master’s degree at the time of the Winter School (31 respondents, 44%). Around one third were PhD students (25 respondents, 36%) and 1 out of 5 belonged to the group of practitioners working in the field of adult education and lifelong learning (20%).

Regarding the gender distribution, 71,4% of the group were female, 28,6% male.

Participants’ Satisfaction

The international Winter School in Wuerzburg has received remarkably good evaluation results during the last years. The Winter School in 2020 is no exception to this. In an overall perspective, the participants rated the Winter School programme with a mean value of 4,33 (SD 0,675) on the 5-point-scale and thus nearly as high as in 2019 (4,39). While the practitioners stayed very steady in their assessment (from 4,11 in 2019 to 4,14 in 2020), the doctoral students rated a little lower (from 4,47 in 2019 to 4,04 in 2020) and the master students a little higher (from 4,42 in 2019 to 4,65 in 2020).

With regard to the organisational, academical and didactical quality of the different programme modules, the results turned out to be rather high. The respective mean values ranged between 3,70 and 4,56. Strongest modules seemed to be the Preparatory Phase and “Lifelong Learning Strategies in Europe/ International Strategies in Adult Education”. At this point we see a rising tendency since 2019. 10 out of 15 mean values (5-point-scale) turned out to be higher than in the previous Winter School.

Also, the participants were asked how useful they considered the different programme components for their own development (rating on a 5-point scale with 1 = not useful and 5 = very useful). Almost all mean values of the components turned out to be higher than the value of 4,00. 23 out 35 items were rated higher than in the 2019.

Effects of the Winter School

The results of the short-term-survey in 2020 equal those of 2019 remarkably. Again, the Winter School had strengthened the participants’ interest in adult education and lifelong learning considerably and also increased their interest and motivation to focus on its trans-/international and comparative aspects in their further studies or future professional activities. All items asked for in the questionnaire received very high mean values between 4,04 and 4,29 on the 5-point scale (1=not at all, 5=very much). Taking a closer look at the subgroups the survey shows that doctoral students tended to rate these questions a little lower.

Also, participating in the programme seemed to increase the respondent’s motivation to undertake further study mobilities considerably. 84% of the master and doctoral students (N=56) scored a 4 or 5 at this question. The respective mean value was 4,32 (STD 1,011).

Further, 66% of the master students and practitioners (N=45) ticked a 4 or 5 when being asked if their motivation to take up PhD studies had grown (2019: 56%). 64% of the doctoral students (N=25) chose a 4 or 5 to indicate that their interest in taking up postdoc research after graduation had increased due to the participation in the Winter School.

And finally, the Winter School also intends to have an effect on competencies related to international or internationally influenced working contexts. In 2020 again, a large share of respondents clearly agreed on these effects. First, the Winter School obviously has a large impact on the development of intercultural competencies. 82% of the participants indicated that the Winter School had helped to improve their “competencies in interacting with people from other cultural backgrounds” considerably by rating a 4 or 5 (mean value 4,21 / STD 1,001). Also, 87% of the respondents stated that their “understanding of adult education and lifelong learning in other countries” had improved by ticking a 4 or 5 (mean value 4,37 / STD 0,745). 74% scored a 4 or 5 when being asked in how far the Winter School had helped to “develop different perspectives on adult education and lifelong learning in their own country” (mean value 4,06 / STD 1,048).

Regarding methodological skills, analytical competencies and professional network competencies the results turn out to be a bit lower. The mean values at the respective items are 3,81 (STD 1,115), 3,78 (STD 1,104) and 3,76 (STD 1,173). The results of the different subgroups at these effects differ considerably. Except for a better understanding of adult education and lifelong learning in general, master students seem to profit most regarding the development of their competencies.

Table 1: Effects - Competencies (by subgroups)

Conclusions

The short-term evaluation of the INTALL-Winter School 2020 again shows that the programme is highly appreciated by its participants and the effects must be considered as high. In an overall perspective the results equal those of the previous year(s) and continue to be on a very high level. Again, the evaluation shows that especially the welcoming, respectful and encouraging learning atmosphere is of great value to the participants. Particularly good and noticeably higher ratings compared to 2019 were given to the module “Lifelong Learning Strategies in Europe/ International Strategies in Adult Education” which is realized during week 1 of the Winter School.

In an overall perspective, it seems that the master students are most satisfied with the Winter School experience. Looking at their motivation we saw that they started with the broadest interest in all aspects and learning opportunities of the programme and also have profited in a very broad and multiple sense. Doctoral students obviously seemed to be focused especially on the content level of the Winter School and in general gave lower ratings than the masters (and mostly practitioners) did. Practitioners finally entered the programme mostly for reasons of exchanging with other experts/fellows in the field as well as looking for expertise in a theoretical but also practical sense, and in the end seem to be fairly satisfied with the Winter School.

The evaluation 2020 produced the following conclusions for further improvement, which however only concern smaller aspects and do not show a need for greater modifications: (1) Since 2019, considerably chances have applied with regard to the practitioners and obviously already have led to various effects. The practitioners’ role seems to be clearer, the integration in the group obviously was improved and all in all we find the practitioners’ satisfaction with the programme to be quite high. Nevertheless, some (single) findings suggest that in week 2 the role of practitioners during the comparative groups could be further developed. (2) The Employability Day has been changed since 2019 considerably and in 2020 the respective evaluation results have improved. Nevertheless, this module seems to bear further developmental potential compared to the other parts of the programme. Unfortunately, the findings give no further hints in which way this component should be altered.

Mid-term Evaluation 2021 - Summary

Author: Katrin Riß
INTALL-Project Coordinator: Julius-Maximilian University Würzburg, Prof. Dr. Regina Egetenmeyer

Approach and data bases

The mid-term survey 2021 was the final step of the external evaluation of the INTALL Adult Education Academy (formerly INTALL Winter School). Based on the concept of the previous years it addressed former participants of the years 2016 to 2020. In 2021, the number of cases was relatively low (N=38) and comprised 15% of the participants addressed, 11% of all former participants. Most of the data of the survey reflect the participants’ perspective one to three years after the participation in the AEA. 39% (N=32) of the data refer to a participation in the previous programme cycle (2020). 21% refer to a participation in 2019 and further 24% reflect the perspective after three years.

Sample/Group characteristics 

Again, the survey succeeded in capturing the broad spectrum of countries the AEA’s participants are related to. In the years 2019 to 2021 the programme had gathered participants from 15 to 20 different countries each year. The mid-term survey 2021 includes 15 different countries where the participants had received or will receive their master or PhD degree. 54% (N=35) of the master’s degrees and 59% (N=17) of the PhD degrees were or will be received at European universities (Germany, Italy, Hungary, Portugal, Norway). Non-European countries represented in the survey are Brazil, Georgia, Ghana, India, Nigeria, Palestine, South Africa, Ukraine, and the USA.

58% (22 persons) of the respondents were still studying when participating in the survey. The exact same share (58% / 22 persons) indicated they were currently employed or had been employed since their participation at the AEA. 10 out of theses 22 were working in the practice or policy field of adult education and lifelong learning, 7 were in the research or academia of adult education and lifelong learning.

Results

The results of the survey 2021 perfectly fit in the picture drawn by the mid-term surveys in 2019 and 2020 and confirm and consolidate the overall picture before.

1. Regarding the effects on the participants’ further studies the data in 2021 was rather scarce. Nevertheless, 14 out of 24 persons (58%) indicated that the AEA had rather large effects on their master or PhD theses (ratings 4 and 5 on the 5-point-scale), a figure that matches the results of the previous year (55%). The mean value is 3,46 (STD 1,215). 64% (ratings 4 or 5, N=22) confirmed they got a more differentiated view on their topic while the same share (64%, N=22) stated that the AEA had encouraged them to include international (-comparative) perspectives in their theses.

2. Looking at future work contexts, 81% (N=26) of the respondents clearly confirmed a motivating effect to work on international issues (ratings 4 and 5). 58% stated that the AEA had provided them with helpful information and contacts for finding future work. Also, 46% chose a 4 or 5 when asked to what extent the AEA gives them a competitive advantage on the labour market.

3. The results concerning the AEA’s effects on the respondents’ competences turned out very clear and confirm the results of 2020. All competences asked for were rated with mean values between 4,19 and 4,59 (N=32). The highest effects obviously are achieved regarding competences in interacting with people from other cultural backgrounds. Here, 94% ticked a 4 or 5 on the 5-point scale. Further, a vast majority highly confirmed effects on their understanding of adult education and lifelong learning in other countries (88% ratings 4 or 5). Networking competences were rated a little higher than in 2020. The mean value rose from 3,98 to 4,28 and thus approached the other competences’ high level.

4. The survey finally intended to learn about the effects on the respondents’ networking. Obviously, the AEA creates a network the participants feel they can rely on. 78% (N=32) confirmed that fact by ticking a 4 or 5 on the 5-point-scale. This obviously not only concerns a professional level. More than half of the group confirmed they had made “really good friends” at the AEA (56% ratings 4 or 5). Regarding a deeper networking process, the opinions differ. Little more than half of the group agreed that they have started a very close exchange or cooperation with some participants (53 ratings 4 or 5).

Mid-term Evaluation 2020 - Summary

Author: Katrin Riß
INTALL-Project Coordinator: Julius-Maximilian University Würzburg, Prof. Dr. Regina Egetenmeyer

Approach

The mid-term survey 2020 again was based on the evaluation concept of the previous years. It follows a mainly quantitative approach realised as an online questionnaire. The call for participation again reached out for all former participants of the Winter School years from 2015 to 2019. In 2020, 49 persons fully completed the questionnaire, around 70% of the sample size of 2019 (71 cases). For the first time the sample also included practitioners, a subgroup that the Winter School was opened for in 2019.

Sample/Group characteristics 

The survey successfully addressed the large international spectrum of Winter School participants, which of course is to be considered as one of the outstanding features of the programme. The number of countries where the respondents had or will earn their degrees (master, PhD and both) amounted to a total of 15. 72,7% of the master’s degree and 56,5% of the PhD degrees were earned at European universities. The list of European countries included Germany, Greece, Hungary, Italy, Norway, Portugal, Romania, Serbia, Slovenia and Spain. Further, participants from India, Malaysia, Morocco, Nigeria and the USA have answered to the organisers request.

At the time of the survey, approximately two out of three participants were still studying or intended to continue their master studies with PhD studies (33 persons, 67,4%). About one out of four (12 persons, 24,5%) on the other hand had finished studies already. In 4 cases the study status remained unclear or was indicated as “other”. Further, 32 persons (65,3%) were or had been employed since their participation in the Winter School. This far exceeds the respective share in 2019 when 19,7% (14 persons) indicated such circumstances.

Results

In summary, the findings of the mid-term evaluation 2020 confirm the results of the previous year(s) and give evidence that the Winter School programme widely succeeds in its aims. These continuously good results surely confirm the concept and realization of the Winter School programme and prove it to be an effective learning opportunity. In detail:

(1) The Winter School obviously had remarkable effects on its participants further studies. 90,9% of the PhD students and PhD graduates agreed that the Winter School had strengthened their motivation to take up PhD studies (ratings 4 or 5 on the five-point scale). Among the  respondents who at the time of the survey were studying for their master’s degree or had finished their master’s degree 62,6% did so. The chart below illustrates that the Winter School also helped its participants to progress their studies and theses in content and methodological related terms. At this, around half of the group (55,3%) agreed that the Winter School had encouraged them to include international and/or international comparative perspectives in their theses, which of course was one of the main interests of the Winter School.

Figure 2: Effects on Thesis

(2) Regarding its work-related aims, the Winter School indeed also strengthened the participants’ motivation to work in international work contexts. At this, the data of 17 persons “currently working” and 18 persons “currently not working” was available. In both groups the majority again highly agreed that the Winter School had strengthened their motivation to work in international work contexts (83,4% / 76,4%, ratings 4 or 5). 38,9% of the participants currently working felt that the Winter School had provided them with useful information for finding their employment. 64,7% of the former participants not working at the time of the survey strongly agreed to that question.

(3) A remarkable number of participants felt that the Winter School had fostered their competencies needed in work contexts including international aspects of one kind or another. These regarded all competencies suggested in the questionnaire including intercultural competencies, knowledge and understanding of adult education and lifelong learning in other countries, English language competencies and to a slightly lower degree analytical and methodological competencies. The chart below illustrates the details.

Figure 3: Competencies - Effects of the Winter School

(4) And finally, the mid-term survey again confirmed that the Winter School successfully provides a (growing) professional network to its participants and induced close exchanges and cooperation among its participants. 75,6% of the respondents felt to be part of a network they could contact whenever needed (ratings 4 or 5). Around half of the group (51%) agreed to have established very close exchange or cooperation with other participants. The use of the LinkedIn-networks related to the Winter School seems to be rather individual. 28,6% of the respondents stated they used these networks in an active way (posting news, participating in discussions), 34,6% follow the news and discussions but contribute only occasionally.