Effectiveness of New Innovations Introduced to the Continuous Assessment Mechanism of the Postgraduate Diploma in Education Programme

Two research studies (Lekamge & Jayathilake, 2002; Jayathilake, 1996) which focused on the Continuous Assessment (CA) mechanism of the Postgraduate Diploma in Education programme of the Open University of Sri Lanka (OUSL) had identified several problems such as lack of quality of the set assignments, limited comments on assignments, discrepancies among marking examiners, ill practices among student teachers and high turnround time of assignments and suggested long term and short term measures to those problems. As a result, the Faculty introduced several innovations namely course team approach, training workshops for marking examiners, Activity Based Assignment Day Schools and appointment of Centre coordinators for improving the quality of the CA mechanism. Having implemented those innovations for nearly three years, it has necessitated an exploratory research study to examine the effectiveness in improving the quality of the CA mechanism and to reveal how it could be further modified to maximize their potentials. The sample of the study included 290 student teachers representing four regional and two study centres of the OUSL, 104 marking examiners and 21 visiting academics of those centres. Exploratory interviews, questionnaires and direct observations were used as the main data collection methods of the study. They revealed that the majority of student teachers, visiting academics and marking examiners were satisfied with the new innovations and admired the positive changes taken place in setting, marking and monitoring of assignments. Finally the study 1Correspondence should be addressed to Prof. G.D. Lekamge, Department of Secondary and Tertiary Education, The Open University of Sri Lanka, Nawala, Nugegoda.(Email: gdlek@ou.ac.lk)


Introduction
The Open University of Sri Lanka (OUSL) is a pioneering institution to adopt Open and Distance Learning (ODL) methodologies for professional development programmes in the fields of engineering, nursing and teaching. The Postgraduate Diploma in Education (PGDE) is one of the most popular and demanding programmes offered to in-service teachers by the Faculty of Education. Annually this programme is conducted in all three media (Sinhala, Tamil & English) at 15 regional and study centres for nearly 2500 student teachers. Take home assignments are considered as the main method of assessing student progress. Depending on the credit rating of the courses, student teachers have to complete either three (9 credits) or two (6 credits) assignments satisfactorily to get eligibility to sit for the final examination. Thus the total number of assignments handled annually in this programme amounts to 35000. A large number of visiting academics attached to different regional and study centres mark those assignments using the marking schemes prepared by themselves at the training workshops conducted by the central faculty.
However, as reported by Lekamge and Jayathilake (2002) the Faculty had continuously experienced problems with regard to copying of assignments, low quality of assignments submitted by students, lack of interaction between the students and teachers through assignments, discrepancies among marking examiners, unavailability of constructive comments in assignments, high turnround time of assignments etc. Therefore, a number of innovative procedures had been developed and introduced for improving the quality of the Continuous Assessment (CA) mechanism. This process was exploratory and progressive and it took a few years to complete the full cycle.
Application of the course team approach for setting assignments and preparing model answers for the assignments can be considered as one such innovation which necessitates collaboration and cooperation among senior and junior academics to maintain the expected standards in the assignments. As a way of strengthening two-way communication between the marking examiners of assignments and student teachers, a new comment sheet for writing overall comments and detailed comments has been introduced and a training workshop was conducted to improve the quality of marking among marking examiners. The appointment of centre coordinators to streamline the distribution of assignments at regional and study centres and to monitor marking of visiting academics was another new strategy adopted by the Faculty. Further, with a view to ensure active participation and the development of professional competencies of student teachers a new strategy called 'Activity Based Assignment Day Schools' from here on referred as ABADS had also been brought into the programme. Having implemented those strategies for nearly three years, an in-depth study was conducted to find out the effectiveness of those innovations introduced and to recommend necessary measures for further improvement of the CA mechanism of the PGDE Programme.

Literature Review
Assessment plays a vital role in the teaching learning process. According to Granados-García et al (2011), continuous assessments assess students throughout the course with periodic testing and they would ease their assimilation of contents and progressive development of skills. Further, Granados-García et al, (2011) pointed out several advantages and drawbacks of a continuous assessment mechanism in Open Distance Learning.
Some researchers reported that feedback generated from continuous assessment was a valuable tool in the learning process enabling the learner to assess their own progress, understand and remedy any error indicated by the assessment (Macdonald etal., 1999: Zakrzewqki & Bull, 1999. Having conducted a study on students' perceptions about the impact of continuous assessment in learning Physiology in Sudanese Faculty of Medicine and Health Sciences, Kaddam and Elnimeiri (2013) had concluded that continuous assessment was an effective method in motivating the students. Jayathilake (1997) had studied the Continuous Assessment System of the PGDE programme of the Open University of Sri Lanka with a view to identify strengths and weaknesses of the system, problems faced by students and tutors and to propose possible solutions for the existing problems. The study revealed the need for redefining the objectives of using CA as a method of evaluating student achievement, restructuring the format of assignments and establishing a supervision system to reduce turn around time of assignments.
The study conducted by Lekamge and Jayathilake (2002) on the Analysis of Tutor Comments on Essay Type Assignments of the PGDE Programme had also given insights into the problems experienced by the institution and students with regard to the CA mechanism. The study recommended several mechanisms to improve the quality of marking of assignments of the PGDE Programme which is the focus of the present study.

Overall Objective and the Research Questions
The overall objective of this study was to investigate the effectiveness of the new innovations introduced in relation to the CA mechanism of the PGDE programme. The specific research questions formulated are given below: 1. How far has the participatory approach improved the quality of the set assignments? 2. How effective is the training mechanism implemented by the Faculty to train marking examiners? 3. How effective is the conduct of Assignment-Based Interactive Day Schools to improve student learning? 4. How have the centre coordinators made an impact on improving marking of assignments at centres 5. How have the marking examiners performed the expected roles in line with the CA component? 6. What are the problems (if any) and issues relating to the implementation of new interventions? 7. What are the suggestions for further improvement of the existing mechanism of CA?

Sample of the Study
Out of the total population, 290 student teachers, 21 visiting academics, 104 marking examiners, 09 Centre coordinators, 2 course teams and 800 assignments were selected for the present study. This incorporated both Sinhala and Tamil media and four regional (Colombo, Jaffna, Matara & Kandy) and two study centres (Kurunegala & Trincomalee).

Methods Used for Data Collection and Data Analysis
The mixed approach was selected for the present study as it permitted researchers to use the following multiple methods for data collection.
1. Questionnaire for student teachers 2. Questionnaire for visiting academics 3. Observations on Activity based Assignment Day Schools using a semi-structured schedule 4. Observation of marked assignments using a criteria developed by the team 5. Focus group discussions with course teams The development of data collection instruments was done by the research team using a collaborative approach. Each instrument was piloted with a small number of participants in actual situations and revised on the basis of their responses before using them on a large scale. The members of the research team and some academics of the faculty were involved in the administration of instruments in this study. Before the commencement of the data collection process, each data collector was briefed about the purpose of the study and how data should be collected using different instruments in a workshop. Focus group discussions were initiated and monitored by a member of the research team.
Internal academics checked 500 assignments related to the same three courses marked by 19 marking examiners (25 from each) at regional and study centres using a criteria developed by the research team.
Simple statistical methods such as frequencies and percentages were used to analyze data collected through questionnaires and qualitative methods such as categorical analysis and content analysis were used to analyze data collected through observations, interviews and focus group discussions.

Findings and Discussion
The results are presented in this section in line with the research questions of the study.

The Impact of the Participatory Approach on Improving the Quality of Assignments
In the PGDE Programme, the compulsory courses (1/3 credit) have three assignments. Each one has to be set according to the format agreed by all the course teams. Marking schemes are also developed using the course team approach (working as a team) and finalized at the training workshops conducted for marking examiners.
Through the exploratory interviews conducted with two course teams and the assignment coordinator of the programme, it was revealed that the course teams had deviated from the said approach due to various reasons. The main reasons they have cited were related to the heavy workload in other programmes of the Department, difficulty in dealing with two batches of the same programme simultaneously and the practical difficulties in having discussions with all course team members on a particular day as some of them were out from the main centre at different times. Therefore, it was revealed that even though the participatory approach has been recommended, the course teams did not have the opportunity to practice it fully.
As revealed through the analysis of assignments of three main courses, the outcome had been that there were drawbacks in the format, practical nature, cognitive levels, novelty and challenging nature of assignments which could have been easily avoided by using the participatory approach.
However, according to views of student teachers, the time allocated for the completion of the assignments was sufficient (81%), sufficient information and guidance was provided in the assignments (75%) and opportunities to practice theory were available through assignments (76%). Similarly, more than half of the marking examiners had expressed their moderate satisfaction with the suitability of language (50.4%) and inclusion of challenging tasks in the assignments (55.6%) while 31.7% & 27.8% respectively had expressed their fullest satisfaction with the same aspects. However, nearly 20% of marking examiners were not satisfied with the appropriateness of the assignments for the level of the programme and the opportunities given for applying professional skills through the assignments. Therefore, it could be assumed that there is room for improvement of the level and quality of the assignments through the course team approach which would in turn facilitate the development of professional competencies of student teachers.

Effectiveness of the Training Mechanism Used by the Faculty for Marking Examiners
One major procedure adopted by the Department to improve the quality of marking is to conduct a one-day training programme annually to enhance the awareness of marking examiners about their role, writing constructive comments on assignments and to finalize the marking schemes in order to increase the consistency among marking examiners. The following tables (1 & 2) illustrate the opinion of the marking examiners.
As illustrated in Table 1, 50 out of 104 marking examiners were fully satisfied with the introduction given on the three roles namely the communicator, facilitator and assessor. It was further supported by their 'fully satisfied' and 'moderately satisfied' opinion on the detailed explanation given on how to mark assignments (41% & 45% respectively) and how to make comments on assignments (48% & 33% respectively). However, it is important to note that 21% and 17% respectively were not 'fully' or 'moderately satisfied' with those aspects.  When marking examiners were asked to reveal their satisfaction with the strategies used (Table 2) in the training workshop, 45 to 52% and 29 to 41% respectively said that they were 'fully satisfied' or 'moderately satisfied' with them. It was only 13% of them were not satisfied with all seven aspects covered in this section.
The relevance and usefulness of activities and the time allocated for the activities at the workshop were the two aspects that the majority of marking examiners were not satisfied with. This is a clear indication that the strategies used at the workshop need further improvement to suit the expectations of the marking examiners.

Effectiveness of Activity-Based Assignment Day Schools (ABADS) to Improve Student Learning
ABADS was a new strategy introduced by the Faculty to avoid problems arising due to ill-practices and lack of prior preparation of students in completing take home assignments and to provide diverse and challenging opportunities for student learning. Through the utilization of ABADSs as a platform for developing inter-personal skills, presentation skills, and thinking skills of student teachers, a positive change in the personality of student teachers was also expected. It was assumed that the effectiveness of ABADS on student learning would rely on the correct perception of the visiting academics about ABADS. Therefore, visiting academics were asked to highlight their perceptions on the difference between the normal day schools and the ABADS in the questionnaire.
The majority (47.6%) of visiting academics agreed that ABADSs have the ability to encourage student teachers to read modules and participate in activities. It was somewhat satisfying to find that 23.8% visiting academics believed that ABADSs are more powerful and effective than the normal day schools. A substantial percentage of visiting academics (38.1% & 33.3% respectively) had pointed out that more preparation and extra-reading and collection of material are needed for the conduct of ABADS satisfactorily (Table 3). Further, 19.0% said ABADSs require more involvement of visiting academics than the normal day schools. These data clearly confirms that the ABADS had made a remarkable change in the behavior of visiting academics which may have directly affected the changing learning patterns of student teachers. When evaluating the effectiveness of ABADSs it was very important to find out how visiting academics felt about student teachers' preparation at the ABADSs (Table 4).
According to Table 4, 54% stated that the student teachers had shown poor preparation. However, 42.9% highlighted that the majority of student teachers were well prepared for the ABADSs. As they further revealed, student teachers were highly motivated in the ABADSs (90.5%) though student-student interaction was limited in group activities (23.8%). Therefore, it could be concluded that only some objectives formed in line with the introduction of ABADS had been realized. 3 Students are well prepared in ABADS for some courses only 4.8 The effectiveness of ABADSs could be further assessed through student teachers opinion on how those ABADSs were organized and managed by the coordinator and the Department. As they pointed out (Table 5), activities were challenging (80% was either agreed or strongly agreed), lecturers were well prepared (85%), allocation of activities were fair (77%), opportunities for peer interaction was productive (86%), atmosphere was friendly (89%) and feedback was received on the spot (84%). According to Table 6, more than 80% of student teachers either 'agreed' or 'strongly agreed' with all the statements presented to them for assessing the effectiveness of ABADSs.
Further, student teachers agreed that the activities were having an impact on their skill development (85%), strategies used at ABADS were excellent (78%), feedback was useful (87%) and ABADS were more productive (84%), useful (84%) and effective (83%) than the written assignments. In comparison with the visiting academics view that they needed to do additional reading and other prior preparations to conduct ABADS, student teachers were having a similar opinion about their preparation. The above responses clearly confirmed that both visiting academics and student teachers have a positive opinion about the ABADS. Further they confirmed that ABADS had improved opportunities for student teachers to become active participants in the teaching-learning process and visiting academics' responses were also in agreement with it.
Direct observations of academics had revealed the following about the conduct of ABADS. The faculty staff spend a lot of time in preparing detailed guidelines for the conduct of ABADS and these guidelines are discussed at the training workshops with visiting academics. Those guidelines provide opportunities to minimize differences among visiting academics teaching at different centres. However, according to the observations, there were deviations from the guidelines at the man centre and in some other centres. Some academics had followed a different procedure without showing any concern about the guidelines provided.
According to the Table 6 one can assume that the student teachers who attended ABADSs get an opportunity to obtain challenging and motivating experiences, to participate in the activities or to interact with other colleagues and to apply what they had learned through the modules. It is somewhat encouraging to find that the majority of visiting academics had followed the format and the activities had been conducted according to the expected standard. Activity-based Assignment day schools are more effective than written assignments.

Impact of Centre Coordinators on Improving Marking of Assignments at Centres
The department had appointed centre coordinators for the regional centres and assignment coordinators for the study centres in order to achieve two main objectives:  Streamlining coordination of marking of assignments  Monitoring marking of assignments As indicated by all the seven coordinators, they had a clear understanding about their roles (Strongly Agree & Agree = 100%). It was further evident that all the coordinators were spending time with marking examiners to finalize procedures (Strongly Agree & Agree = 100%) and willing to take obligatory decisions with regard to marking of assignments (Strongly Agree & Agree = 100%). With regard to copied assignments, they were following the proper procedures. However, it was discouraging to find that a considerable number of coordinators (2 out of 7-28.6%) neither agree/nor disagree with the statements related to completion of work on scheduled dates and returning the assignments on time by marking examiners. In other words coordinators had shown little concern about reducing the turn-round time of assignments.
Having analyzed the responses of Centre coordinators in relation to monitoring of marking of assignments, a positive picture had emerged. All seven coordinators agreed that the assignment marking examiners had written the necessary comments/feedback on assignments though no one strongly agreed with this statement. It is encouraging to find that the centre/assignment coordinators had valued 'the opportunity available for them to provide service to the field of education'. A substantial number felt that it was an 'opportunity to use their knowledge and training for some useful purpose' (55%) and they were 'able to contribute to teachers' professional development (33%) ' through assignments.

Performance of Marking Examiners in line with the Expected Roles
A detailed analysis was done with regard to assignments marked by 19 marking examiners according to the criteria developed by the course team. However, in this paper only the overall observations are presented in Table 7.
According to Table 7, the majority of marking examiners had received good ratings with regard to criteria 1, 3, 4, 5 and 6. In other words, the majority of marking examiners had been performing the role of facilitator and the role of assessor up to the expected level. With regard to criteria 6 and 8 (link with the role of the communicator), the performance of the majority of marking examiners was in a moderate level. Very poor ratings were received by two marking examiners (B & G) and their assignments were related to ESP 2201 and 2202 at Matara and Kandy centres. It could be further observed that two Tamil medium marking examiners (R & S) had received the highest ratings out of all 19 marking examiners and their assignments were related to ESP 2205. It was evident that the innovations had a substantial impact on improving the quality of marking of assignments and the majority of marking examiners were focusing on both assessor and facilitator roles. However, their focus was moderate with regard to the role of communicator which needs to be improved further. It was believed that the best people to inquire about marking of assignments were the student teachers themselves. Therefore, in their questionnaire, a major part was devoted to this aspect ( Table  8). The data revealed that nearly 63% of student teachers were either fully satisfied or satisfied with the feed back given by the marking examiners on their assignments and a similar percentage felt that the comments made by them were very helpful for them to complete their future assignments. This was a substantial improvement experienced through the innovations introduced in this programme. Further, the majority (72%) of student teachers believed that grades given for the assignments were appropriate for their attempt. In addition, they had expressed their high or moderate satisfaction (77%) with the process applied for marking of assignments which could be considered as a noticeable change experienced after the study conducted by Lekamge and Jayathilake (2002). Table 8. Student teachers' opinion on the role of marking examiners However, student teachers were having problems with getting the marked assignments back. Only 55% of student teachers were in agreement with the idea that they received their assignments on time/without delay.
The organization of marking of assignments at centres and sending marked assignments back to students are duties expected from the centre coordinators and they are paid for completing this task. However, it seemed that even after appointing centre coordinators to monitor and expedite the process of marking assignments, student teachers were not happy with the turn-around time of assignments. Therefore, this situation demands urgent action from the Department. The process of evaluating assignments is satisfactory. 05% 08% 10% 41% 36%

Problems and Issues Related to the Implementation of New Innovations
a. Problems pertaining to the use of participatory approach for setting assignments As revealed in a previous section, the less concern shown in applying the course team approach for setting assignments, resulted in lowering the quality of the set assignments. Further, the analysis revealed several problems in the coverage, wording and typography of assignments which may have created a lot of problems for student teachers when answering the assignments. However, problems such as 'lack of awareness about the course-team approach'' inability to continue course-team approach due to unforeseen reasons' 'limited time gap between two intakes of students in the programme' and 'Senior academics being burdened with other issues of the Department' could be identified in line with the setting of assignments through the discussions with internal academics.

b. Problems encountered in the training mechanism
In line with the responses of the marking examiners who had participated in the training workshops, problems such as 'poor planning and organization of the training workshop' 'less challenging activities and strategies applied in the workshop' 'limited opportunities given for group work and 'limited interaction with internal academics' could be identified. Further, the discussions with internal academics had revealed that they had limited guidance within the Faculty and they were always busy with several other activities which did not permit them to concentrate fully on assignments of the PGDE Programme. However, these problems may have led to lower down the quality of the CA mechanism.
c. Problems encountered with the conduct of ABADS In the above sections, the effectiveness of the ABADS was revealed. However,it does not mean that the ABADS are conducted without any problems. The majority of visiting academics had problems in managing time (47.6%) as ABADS are overloaded with too many activities (14.3%). Further, they had problems with guidelines given (9.5% and 4.8%) and late comers (9.5%). As they further revealed, the main problem of student teachers in relation to ABADS was their inability to present answers due to their poor preparation (28.6%). Lack of time given for student presentations (23.8%), limited presentation skills (19.0%) and lack of understanding and motivation of student teachers (19.0%) were the other problems noted. Student teachers also indicated similar problems in line with ABADS.
d. Problems related to marking of assignments Handing over the assignments before having the day school was the main problem faced by student teachers. This may be due to the belief of student teachers that day schools would help them to complete the assignments in a satisfactory manner. More than 20% stated that the department did not send their first set of assignments before submitting the next set of assignments. Therefore, the outcome would be that student teachers repeat the same mistakes through out the academic year. Their next issues relate to the limitations of modules prepared by the institution (16.6% & 12.8%) and the difficulty in handing over multiple assignments simultaneously. Thus, it is questionable whether the assignments can be considered as a learning tool in this programme. Student teachers also pointed out several problems in line with the preparation for assignments, assignments set by the department and comments and grades received for the assignments.
In addition, through the observations of marked assignments, the internal academics had also identified the problems related to 'delay in sending the marked assignments to students' and 'strength of the comments to motivate students for further studies' 'lack of direction for the correct answers' 'lack of specific or personalized comments'. Therefore, the quality of the set assignments and marking schemes, coordination of assignments, turn-round time, monitoring of marking and payments for marking needs further improvement.

Conclusions
It can be concluded that the internal staff members were aware of the positive impact of the participatory approach to the development of assignments, activity-based assignment day schools and marking schemes. Inability to apply the course team approach had led to low quality assignments and unworkable and impractical guidelines for ABADS which had caused numerous problems to student teachers as well as visiting academics. Further, marking examiners were fully satisfied with the introduction given on their roles and explanation given on how to make comments on the assignments though they expressed less satisfaction about the relevance and usefulness of activities, the time allocated for the activities and the strategies used at the training workshop. Visiting academics and student teachers were in agreement with the view that ABADSs have the ability to encourage student teachers to read modules and improve their active participation in day schools. Monitoring of marking through centre /assignment coordinators had a substantial impact on improving the quality of marking of assignments and the majority of marking examiners were focusing on both Assessor and Facilitator roles. Overall, the study provided positive evidence to confirm the effectiveness of the new innovations introduced to the CA mechanism of the PGDE programme.