Evaluation Results
Evaluation on the impact of the joint module – outcomes of the Adult Education Academy
For assessing the development of the joint module and for measuring the impact of the intensive programme, the Strategic Partnership has implemented a formative evaluation, both qualitative and quantitative that accompanied the whole development process and critically reflected the implementation of all intellectual outputs – within and beyond the learning activities. The evaluation is divided in an external evaluation and an internal evaluation. The external evaluation was conducted by Susanne Lattke of the DIE (German Institute of Adult Education). The evaluation design and interpretation was developed in collaboration with the COMPALL Project Coordinator, Prof. Dr. Regina Egetenmeyer (Julius-Maximilians-University of Würzburg). All citations on this webpage are part of the internal evaluation reports. The evaluation design involved both short and mid-term evaluations of the Adult Education Academies defining indicators at the participant and the organizational level.
For the short-term evaluation, participants were requested to fill in a paper questionnaire at the final session of each Adult Education Academy. The questionnaires of each year were rather identical with adopted questions on specific components. In addition, a number of interviews were conducted with selected participants during the last days of the Adult Education Academy. Information from project documents (e.g. COMPALL application, International Adult Education Academy brochure) and information orally provided by the organisers (e.g. regarding total participant numbers) have complemented the data, which was obtained through the surveys and interviews.
For the mid-term evaluation, participants of former Adult Education Academy were requested to fill in an online-survey after more than one year of their participation in the Adult Education Academy. Within the COMPALL project two mid-term surveys will be carried out. A first survey was conducted in spring 2017. It addressed all participants from the Adult Education Academy 2016 and, in addition, all participants of the previous pilot editions of the years 2014 and 2015, which took place before the COMPALL project. A second midterm survey is taking place in spring 2018 targeting all cohorts of the programme.
Both evaluation methods form the comprehensive picture on the impact of the implemented intensive programme focusing on the following targets:
“The main focus of the short-term evaluations, in a formative perspective, is on participant satisfaction and on the quality of the Adult Education Academy and its related offers. Also, the short-term evaluations focus on the academic and personal outcomes and benefits which the participants have gained from the Adult Education Academy, mainly in terms of learning outcomes and increased motivation for international activity.” (Midterm-Term Evaluation Report 2017, p.5)
“The mid-term surveys, by contrast, focus primarily the development of former participants’ professional careers and/or further studies, and the impact which the Adult Education Academy has had on these.” (cf. ibid.)
The structure of the external evaluation followed a six-step process:
· Step 1: Ascertainment of the evaluation concept for the three year, development of indicators into questionnaire items
· Step 2: Student questionnaire used for the first cohort after the first intensive phase
· Step 3: Student questionnaire used for the second cohort after the second intensive phase
· Step 4: Mid-term student questionnaire used for asking the first cohort for longer-term effects
· Step 5: Student questionnaire used for the third cohort after the third intensive phase
· Step 6: Mid-term student questionnaire used for asking all cohorts for longer-term effects.
In addition to the external evaluation, the Strategic Partnership conducted an online survey on the developed online tutorials piloted in the intensive programmes 2017 and 2018. A short summary of the results is provided in the corresponding Abstracts.
 |  |  |  |
|---|
Mid-Term Evaluation 2018
Extraction of the external evaluation results,
which was conducted by German Institute for Adult Education, Susanne Lattke
COMPALL-Project Coordinator: Julius-Maximilian University Würzburg, Prof. Dr. Regina Egetenmeyer
Participation in the Mid-Term Survey
The midterm survey 2018 was implemented as an online survey. All students who had participated in the Adult Education Academies in the years 2014-2017 (n=279) received an invitation to fill in the questionnaire which they could easily access through a link which was provided together with the invitation. The years 2014-2015, which do not belong to the ERASMUS+ COMPALL-project where included to ensure a broader database and to raise the value of the data for the analysis of the impact of the COMPALL-project. For data privacy reasons, the invitation was sent by the organisers of the Adult Education Academies and not by the evaluator (the author of this report). The questionnaire opened between 24 April and 29 May 2018. Three reminders were sent within this period to reach as many former participants as possible. Ultimately, the questionnaire was fully completed by 68 respondents, which corresponds to a response rate of 24.4%.
Labour Market
The midterm survey 2018 shows that a high proportion (three quarters) of respondents currently working are actually facing international aspects in their work, and a similar proportion of those currently not working are highly motivated to engage with international issues in their future activity. Respondents not only find an international dimension in their future work highly attractive but also consider this perspective as rather realistic for themselves. Even cross-border mobility for finding employment in another country is considered a likely option by a considerable share of the respondents, in the 2018 survey even much more (mean 4.02) than in 2017 (mean 3.48). Regarding the Adult Education Academy’s impact, respondents confirm in particular that the Adult Education Academy has strengthened their motivation for international work, but also that it has contributed to developing competencies which are considered as useful in view of labour market and career perspectives. Considerable shares of respondents, both currently working and currently still studying, are also convinced that the Adult Education Academy participation has enhanced their competitive advantage on the labour market and, in particular, within the scientific community, in case they aim for a career in that context.
Networking Impact
About two thirds of the participants have started close collaborations with other participants after the Adult Education Academy and almost all respondents have the feeling of being part in an international network which may prove useful to them in case of need. These findings were consistent in both midterm surveys, 2017 and 2018. The finding from the midterm survey 2017, that about 25% of the respondents were still rather active users of the LinkedIn-networks was also repeated in 2018. The number of passive users was with about 60% of the respondents clearly higher. The online linkedIn networks thus seem not to be the most important tool for keeping in touch, but still they offer an additional resource which is used at least passively by a considerable share of the participants.
Impact on PhD Study
Another aim of the Adult Education Academy is to increase the number of highly qualified graduates in the field of adult education in general, and in view of an international study profile in this field in particular. As in 2017 and to an even higher extent, the midterm survey results 2018 show that the Adult Education Academy has strengthened in many respondents the motivation to take up a PhD study. Those respondents who have already started their PhD research also believe that the Adult Education Academy has contributed to the methodological and analytical quality of that Research.
Impact on International Dimension of Study
The majority of respondents had actually included international perspectives in their Master or PhD thesis or were intending to do so. Respondents also stated that the Adult Education Academy had had an impact on their thesis. Especially respondents who, at the time of answering the survey, were currently studying for a PhD or who had just finished their Master studies and were planning to take up a study rated this impact as rather high. The same pattern could already be observed in the midterm 2017 survey, with even higher impact rates at that time. It seems likely that these particular groups have participated in the Adult Education Academy during a “crucial” phase in their studies; a phase, in which many issues regarding a possible follow up PhD research may not have been firmly fixed but had still to be decided and shaped, so that the Adult Education Academy could come in as an influential experience.
Personal and Professional Outcomes
Looking back from a distance, most participants had very positive views regarding their individual outcomes from the Adult Education Academy. More than three quarters of the respondents agreed that the Adult Education Academy had considerably helped them in both their professional and personal development, and respondents were nearly unanimous in saying that the Adult Education Academy had been an enjoyable experience.
Participant Quotes from open Questions:
 |  |
|---|
Evaluation of Adult Education Academy 2018
Extraction of the external evaluation results,
which was conducted by German Institute for Adult Education, Susanne Lattke
COMPALL-Project Coordinator: Julius-Maximilian University Würzburg, Prof. Dr. Regina Egetenmeyer
Participation in the 2018 Evaluation
The Adult Education Academy 2018 evaluation is based on the paper questionnaire which was distributed by the working group leaders to all participants on the final day of the Adult Education Academy. The International Adult Education Academy 2018 was attended by a total of 61 participants. The number of filled questionnaires amounted to 59, so that the response rate is 96.7%. About two thirds of the participants (42) were Master students and one third (19) were PhD students. The sample of respondents reflects the same 2:1-relation between MA and PhD students (31 Master vs. 14 PhD). 14 respondents chose not to indicate their study level in the questionnaire.
Participants’ Satisfaction
The Adult Education Academy in 2016 had set out already on a very high level, as far as both participant satisfaction and perceived benefits/impact of the Adult Education Academy are concerned. The Adult Education Academies in 2017 and 2018 not only have maintained these level but managed to achieve even further improvements in the participant ratings over the three years. The overall assessment shows that the participants’ general satisfaction with the International Adult Education Academy was very high. The mean value was 4.42 (n=48) which means an improvement compared to the already very high values of the two previous years (4.20 and 4.24 respectively).
Again, participants’ satisfaction turned out to be very high for all aspects with each single mean value being above 4.0. This, too, means a further improvement compared to the previous year(s), although values had been rather high already in 2016 and 2017. In particular, satisfaction with the preparatory phase has increased compared to 2017. This show that the new preparatory offers (video tutorials, online information tools) which have been developed by the organisers since the last year, are well accepted by the participants. Complaints from the first year regarding the academic content of the Adult Education Academy or the composition of the participants group (uneven skills levels, e.g. regarding English proficiency, or unbalanced geographical distribution) have greatly diminished over the years. This suggests that the organisers have been successful in establishing an effective recruiting policy which ensures that potential participants are well informed about what they have to expect from the Adult Education Academy and what is required from them for a successful participation. At the same time, the geographical distribution of participants’ countries of origin became more balanced over the years: the numerical dominance of a few countries was reduced and it became easier to compose the working groups within the Adult Education Academy in a way to include a broad range of different country perspectives.
Participants’ Competence Development
In the later years 2017 and 2018, the focus on academic and professional content has become clearer, whereas in 2016 participant feedback had still focused very prominently on intercultural and social outcomes, including networking opportunities. These intercultural and social aspects remained important also in 2017 and 2018, however they were no longer the dominating feature in the participant feedback. Rather, the Adult Education Academy increasingly appears as a learning offer which combines in an integral way specific subject-related competence development with the development of transversal competences and personal soft skills, besides offering a highly enjoyable experience and an excellent opportunity for building up personal and professional networks.
Impact Career on Career Chances
The career relevance of the Adult Education Academy (outcomes) have been unchangeably rated positively by the participants in all years, which suggests that the Adult Education Academy concept reflects well the competence needs in both an academic context and the labour market.
Summary of external evaluation
One can conclude that the overall concept of the Adult Education Academy seems highly sustainable and does not need to be changed in a substantial way. There will be a need to secure the necessary financial means for the running of the Adult Education Academy after the Erasmus project funding has ended. Since the Adult Education Academy can rely on a solid consortium of partners and since during the project’s lifetime a substantial basis of materials was built up which can be used also in the future, there are good prospects for the Adult Education Academy to become a sustainable offer in the long-term. An appropriate information policy to reach and support (prospective) participants seems essential for the future success of the Adult Education Academy. A substantial part of the critical remarks made by participants in all years can be attributed to wrong expectations on the side of the participants or to a (felt) lack of orientation or preparation. The Adult Education Academy organisers are aware of this issue and have already developed with some success their information and support offers over the years. Also in the future, this issue of information and preparation of the participants should receive particular attention, as its impact on the Adult Education Academy success and the satisfaction on the participants seems considerable. The preparation phase underwent substantial developments over the three years, with many new materials (videos, tutorials, guidelines) being developed to support participants in their own work prior to the intensive phase in Würzburg. These materials are clearly appreciated and found helpful by the participants. They, too, should be kept as a regular feature and be updated as needed. By way of a concluding overall appreciation, it can be stated that the International Adult Education Academy obviously provides a very attractive and highly effective opportunity to students from (potentially) all over the world to learn and develop as an individual and as a (future) professional in the field of adult education in an increasingly globalized context. Moreover, the Adult Education Academy has shown its ability to develop and improve further over the years so as to satisfy participants’ needs and expectations to an ever higher degree.
Evaluation 2017/2018
How do you asses the online-tutorials all together?
How helpful do you estimate the online-tutorials for your preparation to the Adult Education Academy?
Mid-Term Evaluation 2017
Extraction of the external evaluation results,
which was conducted by German Institute for Adult Education, Susanne Lattke
COMPALL-Project Coordinator: Julius-Maximilian University Würzburg, Prof. Dr. Regina Egetenmeyer
Participation in the Mid-Term Survey
The midterm survey 2017 was implemented as an online survey. All students who had participated in the Adult Education Academies in the years 2014-2016 (n=189) received an invitation to fill in the questionnaire which they could easily access through a link which was provided together with the invitation. The years 2014-2015, which do not belong to the ERASMUS+ COMPALL-project where included to ensure a broader database and to raise the value of the data for the analysis of the impact of the COMPALL-project. For data privacy reasons the invitation was sent by the organisers of the Adult Education Academy and not by the evaluator (the author of this report). The questionnaire was open between 25 April and 21 May. Three reminders were sent within this period to reach as many former participants as possible. Ultimately, the questionnaire was fully completed by 65 respondents which corresponds to a response rate of 34,4%.
Labour Market
Regarding work related issues, the Adult Education Academy had a particularly high impact on increasing participants’ motivation to work on international issues. This was especially true for respondents who were not (yet) working (mean 4.13 on a five-point-scale). About half of the respondents not (yet) working considered that the Adult Education Academy had notably (rating ‘4’ or ‘5’) contributed to their employment perspectives, by giving them a competitive advantage on the labour market and by providing them with information or contacts which would be useful in finding employment (ibid.). The proportion of respondents currently working who were of the same opinion with regard to their current employment was only slightly lower. It ranged between one third and almost one half of the respondent for the different items.
Networking Impact
Two thirds of the respondents stated that they had started a close exchange or cooperation with some fellow participants, and 75% of the respondents felt they had made some really good friends at the Adult Education Academy. The linkedIn networks seem to play only a subsidiary role in that. Only about one quarter of the participants stated that they were very using these networks very actively. About two thirds were at least following the news in the networks more passively. This reduced active use of the network seems however not to compromise the Adult Education Academy’s impact as such. As many as 86% of the respondents stated that they were feeling part of an international network of people whom they can contact at any time if need be, no matter if they were permanently in touch with them or not.
Impact on PhD Study
About half of the respondents who were currently preparing or studying for a PhD or who had already finished their PhD agreed to a high extent with the statements that the Winter School participation had strengthened their motivation to take up a PhD, that it had contributed to its analytical quality and that it had helped them to better cope with methodological research issues. Furthermore, at least two respondent have obviously found their (co)supervisor for their PhD research through the Adult Education Academy. Of those respondents who were not (yet) actively dealing with PhD studies (n=32), 24 stated that it was somehow likely that they would take up a PhD research in the future.
Impact on International Dimension of Study
Again, about half of these respondents confirmed that the Adult Education Academy had considerably strengthened their motivation for taking up a PhD and that the Adult Education Academy had also provided them with ideas for a possible topic. As many as nine respondents agreed that the Adult Education Academy had a role in helping them to find a prospective (co)supervisor for their thesis.
Personal and Professional Outcomes
Looking back from a distance, most participants had very positive views regarding their individual outcomes from the Adult Education Academy. When asked to rate different outcomes on a five-point scale, about 80% of the respondents agreed by rating “4” or “5” that the Adult Education Academy had considerably helped them in both their personal and professional development, and respondents were nearly unanimous in saying that the Adult Education Academy had been an enjoyable experience. Regarding improved career chances, about half of the respondents rated the Adult Education Academy’s impact “4” or “5”: 45% of the respondents saw a good impact on their labour market chances in general, and almost 60% said the same with respect to a career in the scientific community, in particular.
Evaluation of Adult Education Academy 2017
Extraction of the external evaluation results,
which was conducted by German Institute for Adult Education, Susanne Lattke
COMPALL- Project Coordinator: Julius-Maximilian University Würzburg, Prof. Dr. Regina Egetenmeyer
Participation in the 2017 Evaluation
The Adult Education Academy Evaluation 2017 is based on the paper questionnaire which was distributed by the working group leaders to all participants on the final day of Adult Education Academy.
The number of filled questionnaires also amounted to 84, so that the response rate would be 100%. Master students were with 50 respondents the biggest group compared to 32 PhD students. Two respondents chose not to indicate their study level in the questionnaire. (Master students: 59, 5 %, PhD students: 38, 1, other/no answer: 2,4%).
Participants‘ Satisfaction
The overall assessment shows that the participants’ general satisfaction with the International Adult Education Academy was very high. The mean value was 4.24 which is almost exactly the same as in 2016 (4.20).
Again, participants‘ satisfaction turned out to be very high for all aspects with means ranging between 3.8 and 4.15 for almost all aspects. There were only a few comments given in the questionnaire regarding the preparatory phase. Nevertheless, one should note, that especially the preparatory phase is still under development and new features, such as online tutorials, were newly introduced and piloted fort the first time in 2017.
Participants‘ Competence Developement
The ratings are very similar to 2016. In a general perspective, participants judged their academic and personal outcomes from the International Adult Education Academy very positively. Regarding specific competences, respondents clearly feel that the International Adult Education Academy has considerably contributed to their competence development in many different regards, including both “soft” skills” like intercultural competence and “hard” academic skills like methodological research skills.
Impact Career on Career Chances
The career relevance of the Adult Education Academy have been rated very high. It shows that the competences which the International Adult Education Academy addresses are considered by the participants as highly relevant for their career chances.
Summary of external evaluation
As in 2016, the findings from the Adult Education Academy evaluation provide a very positive overall picture. Regarding the perspectives for the future development of the International Adult Education Academy , one can conclude that the overall concept of the Adult Education Academy seems highly sustainable and should not be changed in a substantial way. By way of a concluding overall appreciation, it can be stated that the International Adult Education Academy obviously provides a very attractive and highly effective opportunity to students from (potentially) all over the world to learn and develop as an individual and as a (future) professional in the field of adult education in an increasingly globalized context. Most participants had already dealt with international topics at least to some extent, but for about one half of the participants this was their first international study mobility. Participants’ motivations for attending the International Adult Education Academy were multidimensional but the desire to meet people from other countries and to learn about other countries were particularly stressed. Regarding participant satisfaction and impact, the evaluation shows that the International Adult Education Academy has succeeded in maintaining the very high level of the previous year in all respects. The Adult Education Academy provides a multifaceted and at the same time coherent learning opportunity which makes a valuable contribution to the participants’ both personal and professional development as adult education researchers or experts.
Evaluation of Adult Education Academy 2016
Extraction of the external results,
which was conducted by German Institute for Adult Education, Susanne Lattke
COMPALL-Project Coordinator: Julius-Maximilian University of Würzburg, Prof. Dr. Regina Egetenmeyer
Participation in the 2016 Adult Education Academy
The Adult Education Academy 2016 was attended by 91 participants from 19 countries. Of these, 82 participants filled in and returned the questionnaire, corresponding to a response rate of approx. 90%. Master students (51 respondents) outnumbered PhD students (23). The proportion of Non-European participants (at least 25) was considerable, but the focus was clearly on participants from European countries (at least 50).
Participants’ Satisfaction
The overall satisfaction with the Adult Education Academy was very high. The overall means was 4.20. Participants also appreciated the opportunity to learn about international concepts and about the situation in other countries and to get new ideas and perspectives. Another aspect which was highlighted was the active involvement of students through group discussions, group/team work and self-responsible work of students.
Participants’ Competence Development
The ratings show that respondents feel the Adult Education Academy has considerably contributed to their competence development. The lowest – but still high – mean values were obtained by English language competencies” (mean 3.8) and methodological skills for comparative research (3.9). All other items – analytical competencies, being able to see adult education in one’s own country in a new light, professional networking, interacting with people from other cultural backgrounds, and understanding adult education/lifelong learning in other countries - scored above 4. Overall, participants judged their academic and personal outcomes from the Adult Education Academy very positively (means of 3.7 for academic and 4.3 for personal outcomes). The overall picture thus shows that respondents feels they have obtained a broad range of learning outcomes with the foci being very much in line with the intended aims of the Adult Education Academy.
Impact Career on Career Chances
All competencies were attributed a high importance by the respondents with the mean values ranging from 3.93 to 4.74. If one compares, how respondents rated the career relevance of the different competencies addressed by the Adult Education Academy on the one hand, and how they assessed the Adult Education Academy’s contribution to developing exactly these competencies on the other hand (see section 4.2.2) one can conclude that the Adult Education Academy made a significant contribution to enhancing the participants’ future career chances.
Summary of external Evaluation
The evaluation showed very high satisfaction levels of the participants concerning the Adult Education Academy as a whole as well as its single parts. As regards the Adult Education Academy’s impact, participants felt that the Adult Education Academy had considerably contributed to enhancing their competences in many regards. Most notably, increased competence for intercultural interaction was highlighted as outcome, but also the development of other competences, both content-related and transversal ones, was felt to have been promoted by the Adult Education Academy. Participants also considered that the competences which were promoted by the Adult Education Academy were highly relevant for their further career chances, thus contributing to their employability. After attending the Adult Education Academy, participants also felt more motivated to engage with international topics in their studies or future professional activities as well as to undertake further study mobilities.
In conclusion, the evaluation showed that the Adult Education Academy reached a high number of its target audience, involving different countries of origin, study levels, motivations and degrees of previous international experience. By all these groups the Adult Education Academy was perceived as a high quality event which had much to offer for their own academic/professional as well as personal development.
