EVALUATION REPORT FOR INSTITUTIONAL PILOT EVALUATION of Azerbaijan Technical University

Mission report of the Activity: 4.6 Analysis of the Pilots

TWINNING AZERBAIJAN

Support to the Ministry of Education of the Republic of Azerbaijan for Further Adherence of the Higher Education System to the European Higher Education Area (AZ-ad-EHEA)

AZ/14/ENP/OT/31

EU Short Term Expert Mission Report 

1. Basic Information

Component and Activity:

Component: 4. Standards and Guidelines for QA in HE

Activity: 4.6 Analysis of the Pilots

Names of the Experts: Ms Helka Kekäläinen, Ms Heli Mattisen, Ms Kirsi Hiltunen, Ms Kati Isoaho, Mr Hannu Apajalahti, Mr Tauno Otto, Mr Heikki Malinen, Mr Touko Apajalahti, Ms Tiia Bach, Ms Lagle Zobel

Dates of the Mission: 5 – 9 June 2017

Contractor: Finnish Education Evaluation Centre (FINEEC) /

   Estonian Quality Agency for Higher and Vocational Education (EKKA)

2. Relevant Background Information/State of Affairs

The objective of Activity 4.5 was to coordinate a trial/test run of the Standards and Guidelines at three higher education institutions in Azerbaijan. The main aims of the pilot evaluations was to support the strategic management of institutions, provide external feedback to the institutions’ own internal quality assurance procedures, as well as inform the internal and external stakeholders of the compliance of the institutions’ quality assurance with the ESG.

The three pilot universities (Azerbaijan State Pedagogical University, Azerbaijan State Economic University and Azerbaijan Technical University) submitted their self-evaluation reports in January – February, 2017. The site visits of the pilot universities took place in April, 2017 and the assessment reports were finalized in May – June, 2017.

The pilot evaluations had an institutional approach with the focus on teaching and learning. The evaluation reports provide the pilot institutions with information regarding their strengths and good practices, as well as recommendations for the institutions’ further development and suggestions to the MoE regarding the regulations governing higher education in Azerbaijan. The summary of main strengths and areas of improvement of UNEC, ASPU and AZTU is set out in Annex I of this report.

3. Objectives and Tasks of the Mission

The overall aim of the Component 4 is to develop Standards and Guidelines for Quality Assurance in Higher Education in Azerbaijan, in line with the European Standards and Guidelines for Quality Assurance (ESG). The main objectives of Missions of Activity 4.6 were:

• to analyze the results of the pilot evaluations and to integrate remarks into the revised proposal for the Standards and Guidelines for Quality Assurance, and
• to propose a roadmap for the implementation of an EHEA compatible QA system (including an ENQA compatible QA Agency for HE), and ensure appropriate dissemination.

4. Time Schedule of the Mission

5. Achievement of the Expected Results

The expected results for the mission were achieved: the results of the pilot evaluations were analysed and recommendations were prepared for the future activities of the Accreditation and Nostrification Office (ANO) in Azerbaijan.

From Monday, June 5, to Tuesday, June 6, feedback meetings were conducted with the Azerbaijan State Economic University, the Azerbaijan State Pedagogical University and the Azerbaijan Technical University. During the meetings, the members of evaluation panels gave a presentation outlining the main strengths and areas of improvement of each institution, followed by a discussion about possible follow-up activities by the HEIs. The HEIs also had an opportunity to give a short presentation on the impact of the evaluation and to describe some good practices identified in their university during the evaluation process.

On Wednesday, June 7, an analysis of the pilot evaluations took place in the format of a Learning Café workshop, where the representatives of the HEIs and the panel members had an opportunity to share during four respective sessions their opinions and suggestions on the topics of 1) self-evaluation, 2) criteria and manual for pilot evaluations, 3) site visit end the evaluation process, and 4) the evaluation report. The project managers of the panels served as rapporteurs to these sessions and compiled a summary of the key findings under each topic set out in Annex I to this report.

On Thursday, June 8, a discussion took place between the STEs, panel members and the representative of ANO resulting in recommendations for ANO’s further activities in the areas of assessment criteria, self-evaluation, composition of panels and training, the site visit and writing of the evaluation report.  These recommendations support the implementation of an EHEA compatible QA system in Azerbaijan and are set out in Annex II to this report.

6. Unexpected Results

On June 2, 2017, an Action Plan on the implementation of the Order №2199 of the President of the Republic of Azerbaijan on the “Additional measures with regard to increasing the efficiency of business environment and to further improving the standing of our country in international ratings” was approved by the Order of the President of Republic of Azerbaijan[1]. In Section 9.1.2.3 of this Action Plan (Quality of education system), it is stated that standards and guidelines for quality assurance in Azerbaijani higher education system are developed in line with the relevant European Standards and Guidelines within the TWINNING programme of the European Union​.

7. Issues Left Open After the Mission

No issues were left open after the mission

8. Recommendations for Future Missions

-

9. Conclusions and General Remarks Concerning the Project

The main findings of the Mission are set out in Annexes I and II to this report as follows:

ANNEX I: The summary of main strengths and areas of improvement of UNEC, ASPU and AzTU and summary of the key findings of the analysis of pilot evaluations

ANNEX II: Recommendations for ANO’s further activities supporting the implementation of an EHEA compatible QA system in Azerbaijan

Annex I to the Mission report of Activity: 4.6 Analysis of the Pilots

ANNEX I  

Main strengths and areas of improvement in the area of higher education in the Republic of Azerbaijan based on pilot evaluations

SUMMARY OF THE PILOT EVALUATION IN UNEC, ASPU AND AzTU (mark with ‘X’):

OVERALL STRENGTHS

• The roles and focuses of the universities – pedagogy, economy, technology –  are clear and well justified
• Willingness to improve, awareness of the need for change
• Orientation towards improving the management structure and create conditions for the senior management staff to take more responsibility and ownership in fulfilling their role
• Awareness of the need to improve management and leadership competences

OVERALL AREAS OF IMPROVEMENT

• The universities should work on its mission and vision statements and general objectives, which could be more academic in content and which lay down the general framework for the strategic choices.
• All the relevant stakeholders should be taken along to the strategy process in the future. The universities should pay more attention to inform all stakeholders about the strategic development plan and the objectives.
• Even if the management and staff are aware of the needs to monitor changes in the labour market, the needs of the society, and the need to involve more stakeholders, addressing those needs requires a more systematic and efficient approach.
• Management of lower level is very fragmented – very many positions and duties. There is too much focus on rector’s level – middle management level does not work properly.
• Reviewing of study programmes and syllabuses should be conducted regularly. Students and external stakeholders should be involved in these reviews.
• The improvement of teaching staff’s competencies as well as the monitoring of the improvement should be systematic.
• More emphasise should be given to the relevance of up-to-date teaching materials, technical resources should be improved and the efficient use of technical equipment in the teaching and learning process assured.
• Learning outcome based and student-centred approach in teaching and learning need to be developed, e.g. praising the first one and supporting the last ones, as the chain is as strong as the weakest link; more real elective courses students; assessment of achieved learning outcomes instead of multiple choice tests etc.
• Establishing systematic mechanisms that support the staff in conducting research activities. The change from teaching mind set to research mind set should be prioritized. Access to international research databases is integral to improving research activities. Courses and training on research methodologies and skills should be offered to students.
• Internationalization is a challenge at all levels – better English language skills, at least one full course in English, participation in exchange programmes (both, academic staff as well as students), international research projects etc.
• The distribution of full-time teaching staff by age is uneven. More younger, qualified and motivated teachers and researchers are needed in order to facilitate the improvements in the area of higher education.

RECOMMENDATIONS AT THE STATE LEVEL

• The  legal and normative framework which regulates governance of higher education institutions should be reconsidered by the Ministry of Education to allow universities to implement a more effective management system, e.g., have two different governing bodies – academic council and operational management body.
• Higher education institutions should be granted more freedom and autonomy to continuously modify and update the curricula of their study programmes according to needs of labour market and based on recent research results.
• Doctoral studies should be reorganised based on the Bologna system, as the 2-level system   of doctoral degrees is demotivating and may cause losing the interest of young generation to start their academic career.
• Access to up-to-date academic research through electronic databases should be ensured at all Azerbaijani higher education institutions.
• For internationalisation of higher education and in order to become part of international research community, the English language competences should be raised already at the level of secondary education. National curriculum should be reconsidered having stronger emphasis on acquiring English language skills.

ANNEX I

Component 4: Standards and Guidelines for QA in Higher Education

Activity 4.6: Analysis of the Pilots 5-9 June 2017

Summary of the key findings of the analysis of pilot evaluations

The Learning café workshop took place on June 7, 2017 in the Azerbaijan Technical Universities in four stations.

1. The topic of the first station was Self-evaluation and the Secretary and Rapporteur was Kati Isoaho.

1. The following themes of discussion were proposed to the pilot university representatives:

• Assess the Guidelines for Self-Evaluation document. What was useful and how could the document be improved? How do you consider the number and contents of claims included in each category? What kind of appendices would be most beneficial (e.g. statistics, organisation chart, other figures etc.) to be included in the self-evaluation report?
• Asses the process of conducting the Self-Evaluation: Was it useful? How was the cooperation with your colleagues? Did you learn anything new? How would you organize it next time? Would you need external trainers for it? Who should at least be involved in the self-evaluation process? How would you assess your university´s capacity to write evaluative text and how would you enhance/improve it?

2. The members of the evaluation groups were asked to discuss the following questions:

• Asses the Self-Evaluation documents as the guiding documents for the evaluation work: What worked well? What created most misunderstandings or difficulty? What was missing? How could the Guidelines for Self-Evaluation be improved?
• How would you develop the self-evaluation capacity of your Azerbaijani colleagues? For instance, is there need for increasing capacity in group work, stakeholder involvement, student involvement or evaluative and reflective writing, etc.

3. As a result of the discussions, three main issues were discussed: the practical side of conducting SER, the usefulness of the criteria in the Accreditation Manual and the quality of the SERs. As a result, the following key findings can be outlined:

• From a practical point of view, a lot universities felt that there were some challenges regarding the scheduling issues (which are common on institutional level) and reaching the relevant persons inside the institutions due to these scheduling issues – an examination period was taking place at the same time as the report-writing.
• It was also mentioned in the discussion that to carry out SER process, institutions need to build up a team of more than 10 people involved with someone in charge. It should be a team effort and never the responsibility of solely one person to take care of the process.
• One of the advantages of the SER process is that it creates a possibility for different chairs in the HEIs to get to know each other and each others’ activities better than before.
• One point of concern was that the use of the final report was not fully clear for all the university representatives.
• There were different opinions on the usefulness of the SER criteria: on the one hand, they cover all the key operations of the university, but on the other hand, they were not considered to be measurable enough – more figures and equals would have been needed.
• It was highlighted that because Azerbaijan cooperates internationally, the SER criteria should be applicable globally, not only in the European context.
• The quality of level and information provided in SERs was also discussed. The evaluation team members would have needed more descriptive information on the universities’ operations, and foreign experts were hoping for more information on the local HE system. Also, the general opinion seemed to be that there should have been more appendixes to the SERs – there was too much factual information in the report that made it challenging to read.
• There were some gaps in SERs, as some areas that were not discussed e.g: how the strategic plan was developed, what tools were used and who participated in the process, etc. Also, it was felt that the reports were always not analytical/informative enough. It has to be decided how to avoid gaps in the future – whether to provide more guidelines or to raise the capacity of the HEIs how to conduct SER on an institutional level.
• The local experts were commended by the foreign colleagues, as they served a necessary purpose of providing information on local context and Azerbaijani HE system.
• In conclusion, it can be said that the SER is crucial for the whole evaluation process. If the quality of SER is good, it makes the whole process successful. It is the task of ANO to stress this in the future and raise the capacity of HE institutions in this matter.

1. The topic of the second station was Criteria and Manual for Pilot Evaluations and the Secretary and Rapporteur was Kirsi Hiltunen.

1. The following discussion themes were proposed to the pilot university representatives:

• Assess the Manual for Pilot Evaluations document. What was useful and how could the document be improved?
• Assess the assessment areas and criteria: What worked well and what should be improved? Please, discuss with your group each criteria individually. Which areas were most difficult to self-assess? Where did the criteria overlap? Where were the biggest differences between the self-evaluation and the external evaluation? What would you leave out? Anything you would like to include?
• What other feedback would you like to give to develop Standards and Guidelines for Quality Assurance and future QA system in Azerbaijan?

2. The members of the evaluation groups were asked to discuss the following questions:

• Assess the functionality of the assessment areas and criteria: How could they be improved?
• Please, discuss with your group each criteria individually. Which areas were most difficult to assess? Where did the criteria overlap? Where were the biggest differences between the self-evaluation and the external evaluation? What would you leave out? Anything you would like to include?
• What other feedback would you like to give to develop Standards and Guidelines for Quality Assurance and future QA system in Azerbaijan?

3. As a result of the discussion, the following key findings can be outlined:

• The Manual for Pilot Evaluations was considered to be very good – systematic and providing good guidelines for self-evaluation. The manual provides an outline of how the learning environment and innovative practices could be applied in a university. The guidelines explain in detail how to develop management, strategic planning and research in a university. According to the university representatives, the manual outlined ways in which a quality assurance system could be established in a university. If was also found that the manual opens the way to interactive discussions both within university and with the external stakeholders. The manual also fosters the culture of reporting, establishing quality as an integral asset, and encourages communication between the students and the faculty.
• It was concluded there was not too much overlapping between different criteria – some was found in research and human resources (e.g, a question of involvement of staff in research projects and information about quality management system), but not much. The universities wouldn’t leave out anything from the criteria and believe that all of them are needed.
• It was also discussed which areas were the most difficult to self-assess/assess, and the following criteria were pointed out:
• the assessment area of management was mentioned twice;
• study programmes and their preparation (many people do not have skills yet to transfer from the traditional way to the new way);
• the prioritization of research areas and how research activities support teaching
• the human resources: it was not considered fit for purpose to have a requirement that selection procedures for administrative staff should be open for public (selected usually from amongst the graduates and university staff).
• the infrastructure (for the evaluation panel) because there was not enough data, i.e., the panel could only visit 5-10 classrooms.
• The biggest difference between self-evaluation and external evaluation was considered to be that the SER was more focused on quantitative aspects and the qualitative aspects were missing.

However, it was also mentioned that there were no so much differences between SER and the evaluation report. According to some HEIs, the experts seemed to have studied the activities so well ”as if they had had informants from the universities”.

• Also, a discussion took place wheteher something should be added to the criteria. Professional development of staff was mentioned as a separate assessment are, as well as cooperation between university and the industry and the academic ethics. It was also mentioned that there should be criteria in place how to assess study programme groups. The universities expressed an opinion that the sub-criteria included in the table should also be graded and the grading should be more measurable. They also found that the grading should be measurable (including percentage).The evaluation teams, in turn, considered there to be enough assessment areas but far too much sub-criteria.
• A question was raised by the teams wheter the data collected by MER could also be used in evaluations.

• The topic of the third station was Site-visit and the evaluation process and the Secretary and Rapporteur was Tiia Bach.

1. The following discussion themes were provided for the pilot university representatives:

• Assess the timetable of the evaluation: How could the timetable be improved?
• Assess the implementation of the site visit: What worked well and how could the site visit be improved?
• Assess the performance of the FINEEC/EKKA project manager: Where did she/he perform well? How could the activities of the project manager be improved?
• Assess the cooperation between FINEEC/EKKA and your university: What worked well and how could the cooperation be improved?
• Assess the work of the evaluation group: Where did it perform well, what could it have done better?
• Assess the whole audit process: What was the best for your university and what could be improved?

2. The members of the evaluation groups were asked to discuss the following themes:

• Assess the overall scheduling of the evaluation: How could the scheduling be improved?
• Assess the team’s preparation for the site visit: What worked well, and how could the preparation be improved?
• Assess the implementation of the site visit: What worked well, and how could the visit be improved?
• Evaluate the activities of the FINEEC/EKKA project manager: What worked well, and could the activities of the project manager be improved?
• Assess the training provided prior to the evaluation: What worked well, and how the training be improved?
• Assess the activities of your evaluation group: Where did the team succeed, what could have been done better?
• What kind of feedback would you like to give to FINEEC/EKKA in general? What did you learn from the evaluation process?

3. As a result of the discussion, the following key findings can be outlined:

• All the panel members were found to be well-prepared, professional and constructive.
• The university representatives found that the timetable for the visit was appropriate and there was a sufficient mount of time for the site-visit (three days). They had a clear overview before the visit how the visit would look like. However, some inferviewees were invited to the discussions at the last minute, which was challenging. All the important units in the HEIs were covered.
• The experts found that the preparation for the visit went very well and the tasks were clearly divided. However, more supportive documents should have been presented by the institutions.
• The experts appreciated that all the interviews were conducted in an open and transparent atmosphere. Regarding the site visit – more time was needed to see the facilities needed. This should be taken into account in the future.
• It was brouht out that there was too little time for some expert teams to prepare the evaluation reports. In the future, HEIs need more time to submit their comments to the report.
• According to the university representatives, the questions asked at the interviews were relevant, but sometimes not specific enough and not enough tailored to the profile of the university. Sometimes it was felt that the length of interviews should have been longer.
• The collaboration between universities and project management was good. HEIs appreciated the possibility to present addidional materials during the process.
• Regarding the whole process, it was mentioned that analysing all the assessment areas gave the universities an input for drafting study development plan, identify their weaknesses, etc.
• It was considered to be very important that more local experts are trained and the teams include labour market representatives in the future.

1. The topic of the fourth station was Report and the Secretary and Rapporteur was Touko Apajalahti.

1. The following themes were discussed with the pilot university representatives:

• Assess the content of the evaluation report: What was useful, what was unclear or unnecessary?
• How would you improve reporting?
• What would be the best way to disseminate the information of the pilots? Who should read the report?

2. The members of the evaluation groups discussed on the following topics:

• Assess the process of producing the evaluation report: What worked well, and how could the process be improved? You may also comment on the quality of the report.
• What would be the best way to disseminate the information of the pilots? Who should read the report?

3. As a result of the discussion, the following key findings can be outlined:

• Both the teams and universities were satisfied with the quality of the report. The HEIs were especially satisfied with the practical recommendations they can use in their development activities.
• The teams got good support from the chairs and Project Managers.
• To improve the process, there could be more guidelines for the team members to write the report (regarding the style, level of detail, special tools like Google Docs, etc.).
• It was stressed that enough time is needed for writing the report and the translation should be of high-quality to avoid any misundestandings. The responsible persons for each topic should also check the facts in the report.
• Regarding how to disseminate the results: both univesities and the panel members found that the reports should be also public in the future and published both on universities’ and ANO’s website.
• The report should be read by basically all stakeholders: the university management (who can make a change), student representatives, external partners (who can influence change), and everyone in the university (who should be committed to change.) Also, the Ministry, because sometimes system-wide changes are needed.

Annex II to the mission report of Activity: 4.6 Analysis of the Pilots

ANNEX II

Component 4: Standards and Guidelines for QA in Higher Education

Activity 4.6: Analysis of the Pilots 5-9 June 2017

Recommendations for further activities of ANO supporting the implementation of an EHEA compatible QA system in Azerbaijan

1. The Assessment Criteria

• The key terms in the Accreditation Manual (e.g the Quality Assurance system) need to be explained more so that everyone involved in the evaluations can have a joint understanding.
• According to the international experts, the number of sub-criteria under the assessment criteria is far to high. However, which of these criteria need to be removed or merged, needs further discussion. The number of sub-criteria under different criteria should be balanced out more (under some criteria there are currently 5 or 6 sub-criteria, whereas under others there are as much as 8 or 9), as within the timeframe given, it was difficult for the experts to get information about all the criteria. On the other hand, the university representatives found all the criteria to be very important. It is important to eliminate all possible overlapping between the criteria.
• The pilot universities suggested that the criteria should be graded in a more measurable way instead of „conforming“/“not conforming“, e.g in percentages. However, each criteria might have its strengths and weakesses and measuring them in such a strict manner might reduce the overall anaytical quality of the report.
• It is very important to make sure that all the stakeholders can understand the assessment  criteria in a uniform way. ANO has to remember that criteria has to be applicable to all kinds of universities specialized in humanities, social sciences, law, etc.
• The universities suggested to highlight as separate assessment areas the professional development of staff (in-house training) and the cooperation between universities and the industry/labour market. However, these topics are already covered by the existing assessment areas.
• Maybe there could be some sort of a „free assessment area“ that ANO and the HEI could decide upon together – one optional evaluation area on which the HEI could get some enhancement-based feedback.

2. Self-evaluation

• It is highly recommended to provide public self-evaluation trainings to HEIs in order to raise their capacity in how to conduct SER on an institutional level. ANO could take responsibility for that. Also, institutions themselves could appoint external experts as „critical friends“ for consulting purposes. Institutions could also benefit from benchmarking with previous self-evaluation reports.

ANO is advised to apply a road map approach to development of self-evaluation capacity of the HEIs. For the coming years it is necessary to provide concise support for the HEIs in this matter, but it might be possible to increase the level of autonomy in self-evaluation process later in future, when the institutional capacity has reach the more mature level.

• Before sending the SER to the expert panel, it should be checked whether 1) all the areas are covered, 2) all requirements set for SER are met, 3) additional data are there, and 4) the report is clear for readers. A balance has to be struck by ANO not to influence the content of the report but make sure it gives all information necessary for assessment. One option would be to include the requirements towards SER in the assessment criteria to be fulfilled as a prerequisite before checking other assessment areas. It is important to keep one eye on the ESG and the other on the local context.
• As there were some differences between the institutions how they organized the SER process internally, there were some challenges reaching the right persons and getting their input. Therefore, it is recommended to introduce some guidelines regarding the organization of the process in a HEI.
• More background data (statistical data) is needed for evaluations.Various factual information should be included as annexes (e.g job descriptions). However, appropriate context should be provided for the quantitative data. It is recommended to always provide in the SERs very general background data regarding the finances, students, teaching staff. Similar information about every single university should be available to the panels.

Another recommendation is that he institution itself should have some targets for different assessment areas (e.g student numbers). Some it could also be quantitative indicatiors about reaching targets in certain areas which should be supported by data in appendixes. Which kind of targets should be added to whic areas, should be elaborated by ANO. The data should then be analysed and connected to the targets. Standardized indicators could be used in the process (dropout rate, applications per study place). Information important of institutions’ point of view should be included.

3. Composition of panels and training

• The foreign members of the panel pointed out that they would need some more information on the local HE system. Therefore, it is recommended to have mixed teams of both foreign and local experts.
• It is suggested to adopte a code of ethics for experts: ANO should establish selection criteria for the best available experts to provide objectivity (there should be a mechanism for conflict of interest) before the evaluation process.
• As it is a very important requirement of the ESG to have a student on the evaluation panel, it is the task of ANO to find the students interested in the external evaluation process and also allocate appropriate resources for their training. Also, the need of a student perspective and the importance of the role of the student in a team should be better explained in the process. Very active cooperation with student unions is suggested to build up joint understanding of the whole assessment system.
• It is highly recommended to consider including the labor market representative in the panel. It is necessary to build trust between the society and the university. ANO should start working into the direction of involving the labor market representatives in the process.
• It is very important to have at least as the chair to the panel an international expert who can demonstrate new skills and provide a learning experience for local experts.

It is also recommended to hire international trainers to train ANO experts.

• When ANO succeeds in adopting new accreditation standards, it is important to communicate with different stakeholders to dissaminate information about the new standards.

4. The site visit

• Regarding the whole evaluation process:

• the timeframe (from the compilation of SER to writing the report) could be reconsidered: whether there should be more time for writing the report/submitting comments by the HEIs. ANO should thinking this through quite carefully: agreeing on the dates of the visit, submitting the report, deadlines/timelines should be in place.
• It could be good to have the role and tasks of the Project Managers set out in some kind of format so that it would be also clear to them – e.g, should they take notes during site visit, how much are they allowed to interfere with writing the report, etc.
• It is important to have one contact person in the HEI in order during the whole evaluation process in to ensure the continuity of the process.

• Before the site visit:

• It is crucial to have a detailed visit schedule ready well before the site visit. It is also important to get all the names of the interwiees before the site visit so that the team could check whether they shall meet all the necessary people.
• In the future, in order to improve that everything in the institution is well understood and prepared in time, the Project Manager could maybe meet/talk to the contact person before the visit so that everything is coordinated. Guidelines should be in place how to prepare for the visit. Also, all the people who come to the interviews could be informed about the panel members.

• During the site visit:

• Deep consideration should be given to how much time should be allocated to see the facilities and have a tour of the institution.
• If necessary, it is suggested to have a meeting at the end of the visit to clarify some issues that remained unclear, depending on the topic.
• It could also be considered to aski feedback right after the visit about the panel members and performance of the experts. This would also be useful feedback for the agency in deciding whether to use these experts again.

• As a general remark, it was suggested that EKKA and FINEEC could maybe support this new evaluation process in Azerbaijan at the beginning to make sure it starts successfully.

5. The report

• Although the Manual seems to provide enough information, more guidelines are needed for the experts how to write the report – e.g, what is the expected level of detail, in what style the report should be written and what kind of background information is needed. It is of essence that all the reports are written in a consistent manner.
• It is recommended to develop a method/toolkit to help the Project Manager how to work in different situations for the teams.
• The evaluation reports should also be public in the future, discussed in the universities and also with the whole sector. It is important for ANO to spread the knowledge system-wide, organize seminars, meta-analyses.
• There should be given clear guidelines to the institutions how to comment on the reports.
• After the committee receives the comments and decides which ones to consider, there could be a cover letter why some comments have/haven’t been accepted.
• It would be useful for ANO to follow up on its own activities, e.g conduct a their metasynthesis on their own operations and develop their own actions, as this is a requirement of the new ESG.

Recommendations on doctoral studies in Azerbaijan as part of Activity 3.3 Aligning Higher Education Qualifications

DEVELOPMENT OF DOCTORAL EDUCATION

BASED ON HIGH LEVEL RESEARCH

PREVIOUS TWINNING MISSION conducted by Professor Mati Heidmets and Karl-Erik Michelsen: No time should be wasted, because scientific environment today is challenged by fundamental changes in society, economy and environment. Hence, the capability to produce new knowledge is and will be tremendously valuable asset and it cannot be replaced by transfer of knowledge or any other means.

NIZAMI project: There is an urgent need for Azerbaijan universities to revise the rules and content of organizing doctoral studies in Azerbaijan within the Bologna process documents in terms of their consistency, especially in ensuring science and research provisions in relevance with EU standards, to fulfill the students' mobility - one of the basic provisions of Bologna process.

1. OVERVIEW OF THE CURRENT SITUATION FOCUSING ON MAIN AREAS OF IMPROVEMENT
2. Functioning of the system:

• Currently the doctoral education in Azerbaijan is too fragmented and provided by very diverse types of institutions. In 2015, 116 institutions were offering PhD studies. Academy of Sciences employs more than 11 000 staff members and additionally there are 53 universities (21 of them offering PhD studies) and several research institutions. In 2015, there were 145 organisations carrying out research.
• There is a lack of research infrastructure and support services.
• There are several governmental institutions that govern and micro-manage universities. This creates overlapping activities that eat up scarce resources. Higher attestation commission has strong powers to control dissertation process and to provide diplomas.
• Only about 10% of PhD students graduate in time.

2. Human resources:

• The heterogeneous system of research results in-efficient use of financial and human resources. Salaries are low and PhD studies are also suffering from low status. Universities and Academy of Science have difficulties in attracting best talents and brains.
• Supervisors for PhD candidates must have Doctor of Science degree (3.19). This is an outdated requirement, because about 70% of potential supervisors are older than 65 (2014 total 2510 academic staff members having Doctor of Science degree and involved in research, total number of persons involved in doctoral studies 2014 – around 3000)

3. Structure of doctoral programmes

• The system of two level doctoral degrees, PhD and Doctor of Sciences, is not in accordance with Bologna principles and doesn’t have a clear reasoning.
• PhD programs have no clear structure and don’t use credit system.
• For the level of Doctor of Sciences there is no “study programme” – this consists of research conducted by an academic professional individually or in a research group, comparable with the work of the post-doctoral researcher. Thus, it should be considered as the continuation of the academic career rather than part of formal education.

4. Internationalization

• There is weak international component in the PhD studies, involvement of foreign scientists/professors in teaching process, research projects or as co-supervisors is not common
• Dissertations are written in the form of monograph instead of articles in pre-reviewed international journals. Students (and the supervisors) publish mostly in national language.
• There is no free access to electronic databases and foreign scientific journals.
• Very few doctoral students participate in foreign exchange programs.
• Doctoral studies don’t provide a doctoral candidate with the skills of acting in an international work and study environment on their own and develop the ability to express themselves in their field of specialisation in at least one foreign language both in writing as well as orally.
• Recognition of doctoral qualifications from foreign countries in order to apply for the position of the professor is not possible (?)

1. RECOMMENDATIONS

• A precondition for doctoral education should be high level research capacity and internationally recognised research output of the institution in the area/specialty of the concrete doctoral programme. In order to define institutions eligible for offering doctoral studies, first, the evaluation of research output and capacity is needed.

Training of high level researchers is currently done in too many institutions (116!). Therefore, we find it necessary that doctoral programs should be concentrated in the large and well equipped universities that have passed rigorous quality assurance evaluations. Quality assurance of doctoral education has been “outcome-based” (provided by the High Attestation Committee), there are no strict requirements regarding the scope, level and capacity of research for opening/developing a doctoral programme.

An example of assessment criteria for research evaluation can be found in Annex.

• Criteria for research universities should include criteria for quality of research as well as quality of educational process, bearing in mind the main goal of integrating research and education. Publishing in High Impact Journals shall not replace but support student centred and learning outcome based teaching and learning.

• Cooperation between universities and research institutes of the Academy of Sciences and other’s needs to be enhanced and motivated with the means of funding. Higher education and research in the Republic of Azerbaijan should be able to compete at the global level, not nationally. Building strong research universities by bringing together the best talents and brains and high level infrastructure would make it possible.

• The differentiation should be made between PhD as doctoral degree acquired after the completion of doctoral studies, and Doctor of Sciences as a result of an individual research effort. Inherited from the Soviet system, candidate for Doctor of Sciences can partly be compared with the post-doctoral researcher in westerns countries, but it should be considered as the continuation of the academic career rather than part of formal education.

• The academic content of the doctoral programs should be re-design to assist PhD students to conduct high quality research. For instance, current mandatory course on philosophy should be replaced by courses that enhance skills in research design, methodology and writing a peer review article in English language.

Doctoral studies should also provide students with management, teamwork and project development skills as well as the knowledge of legal protection of intellectual property.

• Dissertation work should evolve from monographs into article based dissertations. At least four peer-review articles + intellectually stimulating summary is needed to complete doctoral studies. Article based dissertation connects PhD students in the international scientific networks. In addition, objective peer review process validates dissertations and improves supervision.

• Universities should be given full autonomy in scientific matters. They are the place where scientific research is done in the future. This process requires quality assurance system that will evaluate scientific functions in the universities. Those institutions that pass the rigorous evaluation should be granted rights to design and conduct programs that aim at BA, MA and also PhD degrees. High Attestation Commission could be reorganized into research evaluation authority offering regular research evaluation procedures rather than re-checking the quality of every single dissertation.

• Research infrastructure and environment should be improved by providing all PhD students and professional researchers a free access to electronic databases and foreign publications. In addition new laboratory and research facilities should be built and contracts with other countries signed to get access to national and transnational research infrastructure.

• PhD students should have good and open access to international exchange programs and at least six months training abroad should be required from all students. In addition supervision of PhD studies should be given to domestic as well as international scientific experts.

• To promote the internationalization of doctoral programs there should be considerable investments in improving English language skills of students and staff. Additionally foreign experts need to be invited to serve as opponents and members of the defence panels.

ANNEX: Description of assessment criteria and rating scale

Mission report of Activity: 4.6 Analysis of the Pilots

 TWINNING AZERBAIJAN

Support to the Ministry of Education of the Republic of Azerbaijan for Further Adherence of the Higher Education System to the European Higher Education Area (AZ-ad-EHEA)

AZ/14/ENP/OT/31

EU Short Term Expert Mission Report 

1. Basic Information

Component and Activity:

Component: 4. Standards and Guidelines for QA in HE

Activity: 4.6 Analysis of the Pilots

Name of the Expert: Ms Heli Mattisen

Dates of the Mission: 19 – 21 June 2017

Contractor: Finnish Education Evaluation Centre (FINEEC) /

   Estonian Quality Agency for Higher and Vocational Education (EKKA)

2. Relevant Background Information/State of Affairs

The objective of Activity 4.6 was to coordinate a trial/test run of the Azerbaijani Standards and Guidelines for QA of higher education (AzSG) at three higher education institutions in Azerbaijan. The main aims of the pilot evaluations was to support the strategic management of institutions, provide external feedback to the institutions’ own internal quality assurance procedures, as well as inform the internal and external stakeholders of the compliance of the institutions’ quality assurance with the ESG.

In April 2016 a Drafting Group was appointed by the Ministry of Education to work on a proposal for AzSG. A draft manual for the pilot evaluations was discussed with the Advisory Group in a seminar in June 2016. The draft was published on the Twinning project’s website in order to get feedback on the assessment areas and criteria. Amendments were made to the manual based on the feedback. The capacity of the three pilot universities (Azerbaijan State Pedagogical University, Azerbaijan State Economic University and Azerbaijan Technical University) to conduct a self-evaluation was supported through trainings and events in several previous missions.

The pilot institutions submitted their self-evaluation reports in January – February, 2017. The site visits of the pilot universities took place in April, 2017 and the assessment reports were finalized in May – June, 2017. Feedback seminar was organised in each university 5-6 June. International and local experts in cooperation with university representatives and ANO staff analysed the results of the pilot evaluations and prepared recommendations for the future activities of the Accreditation and Nostrification Office (ANO) in Azerbaijan in 7-9 June.

The pilot evaluations had an institutional approach with the focus on teaching and learning. The evaluation reports provide the pilot institutions with information regarding their strengths and good practices, as well as recommendations for the institutions’ further development and suggestions to the MoE regarding the regulations governing higher education in Azerbaijan.

3. Objectives and Tasks of the Mission

The overall aim of the Component 4 is to develop Standards and Guidelines for Quality Assurance in Higher Education in Azerbaijan, in line with the European Standards and Guidelines for Quality Assurance (ESG) and test them with three higher education institutions.

The special objective of the last missions in Activity 4.6 was to disseminate the experiences of the pilot evaluations and the Twinning project achievements to the larger higher education community. 

4. Time Schedule of the Mission

5. Achievement of the Expected Results

The expected results of the mission were achieved.

The Opening words of the conference were given by Mr. Jeyhun Bayramov, the Deputy Minister of Education, H.E. Ambassador Malena Mård, Head of Delegation of the European Union and Mr. Indrek Kiverik, Charge d’Affaires, Estonia. The MS project leaders gave a joint presentation. The mandatory results were reported and the main outcomes of the project were identified as mapping the situation in HE in a comprehensive way, realistic and applicable recommendations widely discussed with stakeholders, supporting collaborative learning in the HE community of Azerbaijan

and supporting some changes in mind-set regarding the autonomy of HEI-s combined with a rigorous, meaningful and improvement led approach in quality assurance. BC Project Leader Emin Amrullayev reflected the success of the project.

Each of the component leaders or their substitutes with their beneficiary counterparts presented the results of each component.

Component 1 Legislation by Heli Mattisen and Sülhaddin Gözalov.

The main partners in the component have been Department of Higher Education and Research and Accreditation and Nostrification Office (ANO) as well as Legal Department of the MoE, Universities, Academy of Sciences and the leaders of other components.

STE-s have given concrete recommendations given regarding

• Overall structure and coherence of the legal framework
• Amendments to the Statute for Higher Education Institutions – governance structure, autonomy in designing management structure clear division of responsibilities, student involvement etc.
• New State Standard and Programme for Higher Education – learning outcomes, student assessment, doctoral education etc.
• New Accreditation Standards
• New Accreditation Rules and Procedures

Most of the recommendations of the STE’s have been taken into account, some documents (like State Standard and Programme for Higher Education) are still under development.

Sülhaddin Gözalov provided some examples of new documents developed based on the recommendations of Twinning experts – like new Statutes for the Engineering University and the University on Languages have giving more autonomy to the universities in building up the management system, developing the programmes, involving students and external stakeholders in the government of the university. The new regulation on credit system will give more freedom to the university and the teachers to design the learning and teaching process.

On the whole, it has been an useful learning process - „formative assessment“ regarding compliance with Bologna and ESG has been offered by European experts, but the main drafting work has been done by local experts.

Component 2 - Co-ordinating and communication by Kauko Hämäläinen and Vusala Gurbanova

The main objectives and recommendations of seven different activities were reflected and also, how these recommendations are used. Main objectives of this component were the following:

• Coordination and networking capacity of the MoE and relevant stakeholders is enhanced on the basis of good practice examples in the EHEA.
• Assess progress and the current status of EHEA/Bologna reforms in Az.
• Assess the institutional set up within the MoE with best practices and improvements for the overall institutional architecture.
• Developing communication and networking between universities, stakeholders and MoE based on the experiences from Finland and Estonia.
• Conducting a training needs analysis and train experts.
• Create communication strategy.

The main recommendations from different activities were the following:

- Systematizing doctoral education.

- Redesigning national standards.

- Establishing networking between Az HEIs.

- Developing more lifelong learning services.

- Drafting a mobility program and removing obstacles for student mobility.

- Division of work and sharing responsibilities between the ministry, universities and external stakeholders.

- Prioritizing areas for the Ministry.

- Developing quality practices of the internal QA process.

- Establish rector´s conferences.

- Analyse needs and options to join international associations in HE.

- A newsletter as a medium of communication.

- Increase students´ and stakeholders involvement in internal QA process and administration of HEIs.

- Improve pedagogical training for university teachers.

- Ensuring coordination of EHEA related trainings.

Many of these recommendations have been implemented or started planning how to proceed. E.g. study visits and training has been organised to increase the capacity of people in central positions.  The main results in practice of component 2 are according the summary by Vusala Gurbanova the following:

1.MoE is realizing Nizami project parallel with the Twinning project where they are planning to restructure doctoral education in Az in line with requirements of EHEA. On respect of these 2 projects recommendations regulations about doctoral studies are going to be improved. The main changes that they are going to implement are the following: PhD will be the last level of HE. At the same time, doctor of science will be open for everybody who would like to continue his/her research. The third one is implementation of ECTS at PhD level.

2. MoE started to draft new standards of HE, where learning outcomes of all levels of HE will be reflected. Additionally, descriptors of all levels of HE will be included to this document. We are planning to combine 5 regulations in one and draft the general standards for HE. 12 recommendations out of 14 have been taken into consideration in the new draft.
3. The main achievement in the framework of component 2 was increasing cooperation of the representatives of different HEs. This project helped HEIs´ representatives to come and work together, to learn from each other, involve students to the decision making processes of universities and to cooperate with different stakeholders.
4. In the framework of the Twinning project the participants had an opportunity to meet with different rectors and to learn their opinions regarding to the establishment of a Rector´s conference. Their reflection were positive and supportive. MoE understand the importance of the Rector´s conference and they already drafted its statute.

Component 3 - Developing AzQF by Maiki Udam and Azad Akhunov

1.Trainings on QF-EHEA

The awareness among academic staff about the basics of QF-EHEA, including constructive alignment between learning outcomes, student assessment and teaching methods, vary notably.

Recommendation: At the national level, a more systematic approach to trainings of academic staff on various topics (learning outcomes, student assessment, teaching methods, e-learning, recognition of prior learning (RPL) etc) is needed.

2. Analysing the correspondence between EQF and AzQF

Although the draft of AzQF has some minor contradictions both concerning content and formulations, the document is ready to be approved and implemented.

3. Aligning HE Qualifications and Curricula (in Doctoral Studies)

It has been the most challenging area in Component 3 as the organisation of doctoral studies in Azerbaijan differs from the model used in most other European countries.

There are several issues that need to be solved:

• the outcomes of 2-level doctoral studies (PhD and Doctor of Science) and their position in the AzQF;
• cooperation between universities and research institutes of the Academy of Science;
• funding of research at the universities;
• international cooperation and visibility;
• quality assurance of research and doctoral studies.

4. Enhancing Practices and Procedures of Academic Recognition

It is commendable that the Accreditation and Nostrification Office (ANO) has been created and manned with dedicated staff.

The main recommendation is to move from nostrification (looking for equivalence of (mainly) quantitative indicators) to the recognition (looking at general correspondence of learning outcomes).

It is also advisable not to regulate the recognition issues too strictly on the national level but to give HEI-s more autonomy to decide about cases related to academic mobility and its recognition.

5. Further Implementation of the AzQF

The Draft Action Plan on Implementation of National Qualifications Framework for Lifelong Learning as well as The Draft Concept of Establishment of National Commission and Secretariat for Implementation of AzQF have been developed.

Recommendations:

• the future projects and national activities should focus more on RPL and respective trainings of academic staff;
• development of databases/IT system(s) for managing the information about students, staff, study progress, admission, graduation etc would support implementation of NQF and QA.

Component 4 – Pilot evaluations by Helka Kekäläinen and Tofig Ahmadov

The main acitivites of the component were reflected: Trainings for the EHEA context, ESG and EHEA trends and practices, Drafting the Standards and Guidelines for Quality Assurance in Azerbaijan, Fostering the Self-Evaluation Capacity in 3 Pilot institutions and The test run of the Standards and Guidelines with 3 universities in Azerbaijan.

The common strengths of the 3 pilot institutions were identified as

• The roles and focuses of the universities – pedagogy, economy, technology –  are clear and well justified,
• Willingness to improve, awareness of the need for change
• Orientation towards improving the management structure and create conditions for the senior management staff to take more responsibility and ownership in fulfilling their role
• Awareness of the need to improve management and leadership competences.

The recommendations for Strategic Planning and Study Programmes and their development were explained.

Tofig Ahmadov introduced in detail the new approach to external quality assurance that ANO will carry out in future.

6. Unexpected Results

No unexpected results

7. Issues Left Open After the Mission

There were no issues left open after the mission.

8. Recommendations for Future Missions

The last PSC will take place 28 August and the Closing Conference 29 August 2017. RTA office will look for a venue that would combine the event and catering in a satisfactory way.

9. Conclusions and General Remarks Concerning the Project

The Project has achieved all the mandatory results and more.

Mission report of Activity: 4.6 Analysis of the Pilots

TWINNING AZERBAIJAN

Support to the Ministry of Education of the Republic of Azerbaijan for Further Adherence of the Higher Education System to the European Higher Education Area (AZ-ad-EHEA)

AZ/14/ENP/OT/31

EU Short Term Expert Mission Report 

1. Basic Information

Component and Activity:

Component: 4. Standards and Guidelines for QA in HE

Activity: 4.6 Analysis of the Pilots

Name of the Expert: Ms Helka Kekäläinen, Ms Maiki Udam, Mr Kauko Hämäläinen

Dates of the Mission: 20 – 21 June 2017

Contractor: Finnish Education Evaluation Centre (FINEEC) /

   Estonian Quality Agency for Higher and Vocational Education (EKKA)

2. Relevant Background Information/State of Affairs

The objective of Activity 4.6 was to coordinate a trial/test run of the Azerbaijani Standards and Guidelines for QA of higher education (AzSG) at three higher education institutions in Azerbaijan. The main aims of the pilot evaluations was to support the strategic management of institutions, provide external feedback to the institutions’ own internal quality assurance procedures, as well as inform the internal and external stakeholders of the compliance of the institutions’ quality assurance with the ESG.

In April 2016 a Drafting Group was appointed by the Ministry of Education to work on a proposal for AzSG. A draft manual for the pilot evaluations was discussed with the Advisory Group in a seminar in June 2016. The draft was published on the Twinning project’s website in order to get feedback on the assessment areas and criteria. Amendments were made to the manual based on the feedback. The capacity of the three pilot universities (Azerbaijan State Pedagogical University, Azerbaijan State Economic University and Azerbaijan Technical University) to conduct a self-evaluation was supported through trainings and events in several previous missions.

The pilot institutions submitted their self-evaluation reports in January – February, 2017. The site visits of the pilot universities took place in April, 2017 and the assessment reports were finalized in May – June, 2017. Feedback seminar was organised in each university 5-6 June. International and local experts in cooperation with university representatives and ANO staff analysed the results of the pilot evaluations and prepared recommendations for the future activities of the Accreditation and Nostrification Office (ANO) in Azerbaijan in 7-9 June.

The pilot evaluations had an institutional approach with the focus on teaching and learning. The evaluation reports provide the pilot institutions with information regarding their strengths and good practices, as well as recommendations for the institutions’ further development and suggestions to the MoE regarding the regulations governing higher education in Azerbaijan.

3. Objectives and Tasks of the Mission

The overall aim of the Component 4 is to develop Standards and Guidelines for Quality Assurance in Higher Education in Azerbaijan, in line with the European Standards and Guidelines for Quality Assurance (ESG) and test them with three higher education institutions.

The special objective of the last missions in Activity 4.6 was to disseminate the experiences of the pilot evaluations and the Twinning project achievements to the larger higher education community. 

4. Time Schedule of the Mission

5. Achievement of the Expected Results

The Opening words of the conference were given by Mr. Jeyhun Bayramov, the Deputy Minister of Education, H.E. Ambassador Malena Mård, Head of Delegation of the European Union and Mr. Indrek Kiverik, Charge d’Affaires, Estonia. The MS project leaders gave a joint presentation. The mandatory results were reported and the main outcomes of the project were identified as mapping the situation in HE in a comprehensive way, realistic and applicable recommendations widely discussed with stakeholders, supporting collaborative learning in the HE community of Azerbaijan

and supporting some changes in mind-set regarding the autonomy of HEI-s combined with a rigorous, meaningful and improvement led approach in quality assurance. BC Project Leader Emin Amrullayev reflected the success of the project.

Each of the component leaders or their substitutes with their beneficiary counterparts presented the results of each component.

Component 1 Legislation by Heli Mattisen and Sülhaddin Gözalov.

The main partners in the component have been Department of Higher Education and Research and Accreditation and Nostrification Office (ANO) as well as Legal Department of the MoE, Universities, Academy of Sciences and the leaders of other components.

STE-s have given concrete recommendations given regarding

• Overall structure and coherence of the legal framework
• Amendments to the Statute for Higher Education Institutions – governance structure, autonomy in designing management structure clear division of responsibilities, student involvement etc.
• New State Standard and Programme for Higher Education – learning outcomes, student assessment, doctoral education etc.
• New Accreditation Standards
• New Accreditation Rules and Procedures

Most of the recommendations of the STE’s have been taken into account, some documents (like State Standard and Programme for Higher Education) is still under development.

Sülhaddin Gözalov provided some examples of new documents developed based on the recommendations of Twinning experts – like new Statutes for the Engineering University and the University on Languages have giving more autonomy to the universities in building up the management system, developing the programmes, involving students and external stakeholders in the government of the university. The new regulation on credit system will give more freedom to the university and the teachers to design the learning and teaching process.

On the whole, it has been an useful learning process - „formative assessment“ regarding compliance with Bologna and ESG has been offered by European experts, but the main drafting work has been done by local experts.

Component 2 - Co-ordinating and communication by Kauko Hämäläinen and Vusala Gurbanova

The main objectives and recommendations of seven different activities were reflected and also, how these recommendations are used. Main objectives of this component were the following:

• Coordination and networking capacity of the MoE and relevant stakeholders is enhanced on the basis of good practice examples in the EHEA.
• Assess progress and the current status of EHEA/Bologna reforms in Az.
• Assess the institutional set up within the MoE with best practices and improvements for the overall institutional architecture.
• Developing communication and networking between universities, stakeholders and MoE based on the experiences from Finland and Estonia.
• Conducting a training needs analysis and train experts.
• Create communication strategy.

The main recommendations from different activities were the following:

- Systematizing doctoral education.

- Redesigning national standards.

- Establishing networking between Az HEIs.

- Developing more lifelong learning services.

- Drafting a mobility program and removing obstacles for student mobility.

- Division of work and sharing responsibilities between the ministry, universities and external stakeholders.

- Prioritizing areas for the Ministry.

- Developing quality practices of the internal QA process.

- Establish rector´s conferences.

- Analyse needs and options to join international associations in HE.

- A newsletter as a medium of communication.

- Increase students´ and stakeholders involvement in internal QA process and administration of HEIs.

- Improve pedagogical training for university teachers.

- Ensuring coordination of EHEA related trainings.

Many of these recommendations have been implemented or started planning how to proceed. E.g. study visits and training has been organised to increase the capacity of people in central positions.  The main results in practice of component 2 are according the summary by Vusala Gurbanova the following:

1.MoE is realizing Nizami project parallel with the Twinning project where they are planning to restructure doctoral education in Az in line with requirements of EHEA. On respect of these 2 projects recommendations regulations about doctoral studies are going to be improved. The main changes that they are going to implement are the following: PhD will be the last level of HE. At the same time, doctor of science will be open for everybody who would like to continue his/her research. The third one is implementation of ECTS at PhD level.

2. MoE started to draft new standards of HE, where learning outcomes of all levels of HE will be reflected. Additionally, descriptors of all levels of HE will be included to this document. We are planning to combine 5 regulations in one and draft the general standards for HE. 12 recommendations out of 14 have been taken into consideration in the new draft.
3. The main achievement in the framework of component 2 was increasing cooperation of the representatives of different HEs. This project helped HEIs´ representatives to come and work together, to learn from each other, involve students to the decision making processes of universities and to cooperate with different stakeholders.
4. In the framework of the Twinning project the participants had an opportunity to meet with different rectors and to learn their opinions regarding to the establishment of a Rector´s conference. Their reflection were positive and supportive. MoE understand the importance of the Rector´s conference and they already drafted its statute.

Component 3 - Developing AzQF by Maiki Udam and Azad Akhunov

1.Trainings on QF-EHEA

The awareness among academic staff about the basics of QF-EHEA, including constructive alignment between learning outcomes, student assessment and teaching methods, vary notably.

Recommendation: At the national level, a more systematic approach to trainings of academic staff on various topics (learning outcomes, student assessment, teaching methods, e-learning, recognition of prior learning (RPL) etc) is needed.

2. Analysing the correspondence between EQF and AzQF

Although the draft of AzQF has some minor contradictions both concerning content and formulations, the document is ready to be approved and implemented.

3. Aligning HE Qualifications and Curricula (in Doctoral Studies)

It has been the most challenging area in Component 3 as the organisation of doctoral studies in Azerbaijan differs from the model used in most other European countries.

There are several issues that need to be solved:

• the outcomes of 2-level doctoral studies (PhD and Doctor of Science) and their position in the AzQF;
• cooperation between universities and research institutes of the Academy of Science;
• funding of research at the universities;
• international cooperation and visibility;
• quality assurance of research and doctoral studies.

4. Enhancing Practices and Procedures of Academic Recognition

It is commendable that the Accreditation and Nostrification Office (ANO) has been created and manned with dedicated staff.

The main recommendation is to move from nostrification (looking for equivalence of (mainly) quantitative indicators) to the recognition (looking at general correspondence of learning outcomes).

It is also advisable not to regulate the recognition issues too strictly on the national level but to give HEI-s more autonomy to decide about cases related to academic mobility and its recognition.

5. Further Implementation of the AzQF

The Draft Action Plan on Implementation of National Qualifications Framework for Lifelong Learning as well as The Draft Concept of Establishment of National Commission and Secretariat for Implementation of AzQF have been developed.

Recommendations:

• the future projects and national activities should focus more on RPL and respective trainings of academic staff;
• development of databases/IT system(s) for managing the information about students, staff, study progress, admission, graduation etc would support implementation of NQF and QA.

Component 4 – Pilot evaluations by Helka Kekäläinen and Tofig Ahmadov

The main acitivites of the component were reflected: Trainings for the EHEA context, ESG and EHEA trends and practices, Drafting the Standards and Guidelines for Quality Assurance in Azerbaijan, Fostering the Self-Evaluation Capacity in 3 Pilot institutions and The test run of the Standards and Guidelines with 3 universities in Azerbaijan.

The common strengths of the 3 pilot institutions were identified as

• The roles and focuses of the universities – pedagogy, economy, technology – are clear and well justified,
• Willingness to improve, awareness of the need for change
• Orientation towards improving the management structure and create conditions for the senior management staff to take more responsibility and ownership in fulfilling their role
• Awareness of the need to improve management and leadership competences.

The recommendations for Strategic Planning and Study Programmes and their development were explained.

Tofig Ahmadov introduced in detail the new approach to external quality assurance that ANO will carry out in future.

6. Unexpected Results

The experts did not come across to any unexpected results.

7. Issues Left Open After the Mission

No issues were left open after the mission.

8. Recommendations for Future Missions

The last PSC will take place 28 August and the Closing Conference 29 August 2017. RTA office will look for a venue that would combine the event and catering in a satisfactory way.

9. Conclusions and General Remarks Concerning the Project

The Project has achieved all the mandatory results and more.