The Analysis of the Informatics Exam of Secondary Education in Lithuania

This article analyses the informatics exam of secondary education in Lithuania. The research evaluates the correspondence of examination tasks to the exam program and its aims. Then, it studies the equality of examination variants. The article also discusses the clarity of the exam questions and instructions as well as the organization of the exam and the objectivity of evaluation criteria.


Introduction
New information and communication technologies (ICT) that are applied in various spheres of life are developing very rapidly.Knowledge and skills to use ICT gradually become the most important prerequisite for the work and life in the Information Society.Therefore, every person must gain the basic knowledge of informatics and must learn to use ICT according to his needs before graduating from the secondary school.
A number of researches analyse the problems of the evaluation of the ICT literacy (Eisenberg and Johnson, 1996;Bruce, 1998).In 1998 year, researchers I. Kirsch and others (1998) constructed and proposed the scales for the evaluation of ICT literacy.In 1986, F. Davis (1986) constructed a model for the analysis of person's capabilities to adopt and master new technologies.J. Lee (1986) developed a questionnaire for measuring the experience of computing.
In Europe, one of the most prevalent ICT literacy rating scales is ECDL (European Computer Driving License).It increasingly claims to become the world's standard -ICDL (International Computer Driving License).
Lithuanian scientists like E. Telešius and A. Otas (1999), V. Denisovas and others (2001) analyse and try to evaluate the problems and validity of ECDL.V. Dagienė (2001), L. Markauskaitė (2001Markauskaitė ( ), D. Šaparnienė (2002)), B. Bitinas (1998) and others study various other issues of computer literacy evaluation in their works too.This research carried out by the order of the National Examination Center seeks to put a foundation for the construction of a progressive evaluation system, which would motivate to introduce and apply new information technologies in the secondary curricula.
The ICT literacy of students and teachers is defined by the state's standards.The first step to check person's ICT skills is an informatics exam at the end of secondary school.Informatics examination covers the core course of informatics curriculum, which students learned in the following grades: • 9 and 10 grades of lower secondary school; and • 11 and 12 grades of upper secondary school (only the compulsory modules of informatics curriculum, i.e., the basic level of the course).
Informatics examination consists of two parts: 1.Independent creative task; and 2. Work with a computer: (a) theoretical and (b) practical tasks.
The aim of this study is to evaluate the correspondence of informatics examination tasks to the program and aims of the exam.
The research has the following objectives: 1.To analyse the correspondence of examination tasks to the examination program.
2. To estimate the equality of examination variants.
3. To study the perspicuity of examination tasks to students.4. To discuss the organization of the examination.

To analyse the evaluation criteria.
The main object of the study is the material of the examination and all other relevant documentation.
The methods of the study are the analysis of the documents using the mathematical graph-tree technique and statistical analysis of the data.
The main hypothesis of the research is that the tasks of the informatics examination correspond to the program of the exam and realize its aims and objectives.

The Correspondence between the Examination Tasks and its Program
This part of the research studies the second part of informatics exam, i.e., work with a computer.Graph-tree technique (Ore, 1973) was applied for the analysis of the equivalency of the theoretical and practical examination tasks and the exam program.The examination program covered 37 topics (Table 1).Each task included one theoretical and two practical questions.Overall, six different variants of tasks were used in the exam.Graphs were drawn according to the topics of the questions.While evaluating the examination tasks, every question was given a coefficient of the correspondence to the particular topic of the examination program.If the question fully corresponded to the particular topic of the exam program, 100% coefficient of the correspondence was given.If only minimal knowledge of this topic was needed, only 5% or 10% coefficient was given.Nine experienced teachers accomplished the evaluation of the correspondence.The average results of the evaluation of all variants are shown in Table 1.We will analyse the results of the evaluation.
As we can see, the least attention is paid to the following topics of the exam's program: 1. Hardware of the computer.
2. Software of the computer.
3. The concept of computer-based modelling.
4. Information security in the computer.
5. Evolution of the computer.
6.The concept of algorithm and program.
8. The concept of data.9. Main controlling commands: sequence of statements, conditional and repeat statements.
10.The style and culture of algorithm.
11.The steps of solving tasks.
12. Classification of information.
15. Main possibilities of graphics editor.
16. Juridical aspects of information.
17.The advantages of text processing with the computer.
18.The concept of hypertext and multimedia.
The average percentage of their usage is 0%-10%.
Another group consists of the topics having the correspondence coefficient between 10% and 50%: 1. Computer networks.
2. Files and directories, managing and compressing files.
6.The concept of information society.
7. The concept of database.
These topics were directly associated with the subject of theoretical questions, or the knowledge of these topics were necessary to solve the tasks.Some tasks required certain practical skills.
Most common topics in the examination papers were connected to the text processing skills and effective usage of spreadsheets.In all variants of practical tasks students had to demonstrate their practical skills and theoretical knowledge of spreadsheets and word processors.The following topics had the correspondence coefficient of 100%: 1. Main elements of the text.
2. Main operations with the word processing applications.
7. Editing the digital information using the spreadsheet.
Only the topic "Main operations with the spreadsheet" had the correspondence of 55%.Considering the results and having analysed the second part of the exam, we can state that the theoretical and practical questions given in the examination papers correspond to the exam program (an average coefficient is 27,83%).However, the questions lack variety, the range of assessed knowledge and skills is narrow.The exam mostly checked students' abilities to work with word processors and spreadsheets.Knowledge on such themes as file management, information processing, e-mail, computer logic and application of other software tools is used very little or not checked at all.

Perspicuity of Exam's Instructions and Questions
The first part of the informatics exam is an independent creative task.A student must create a program or other applied work using ICT.He or she must demonstrate skills, which he/she gained during the lessons of informatics or learned independently.A creative task is done under the supervision of a teacher.
While studying comprehensibility of the instructions, we can see that a creative task is formulated clearly and reasonably.Students and teachers know clearly what they have to do.They do not have any misunderstandings.Students are given examples of creative works.
However, it is quite difficult to analyse the evaluation of this part of the exam, since there are almost no possibilities to review the topics and other aspects of creative works produced by the students.In general, a certain plan and clear definition of initial conditions are necessary to accomplish any sound and fully developed modern applied work.A work presented with the plan or logical structure and the description of initial conditions will make a student and a teacher think on presenting the work to the user in understandable and logical way.A computer program or other ICT-based application cannot be presented without such a scheme and a description of initial conditions.However, there is no such a requirement in the instructions of the independent creative task.The definition of the task and the process of the implementation are left to the teacher's competence and not assessed.As a result, the evaluation of creative task may differ in various schools.For instance, a primitive creative work may get the maximal score in one school while only the minimal score in another.
The questions in the second part of the exam (all 6 variants) are formulated clearly and without ambiguities: • Theoretical questions are easily understandable and correspond the requirements of modern information culture.• The first practical task asks to draw a simple table.It does not cause any misunderstanding.Students do not have any difficulties.• The second practical task is to create an electronic table with a spreadsheet.The task is presented clearly, some variants (i.e., 1, 2, 3 and 6) have even formulas.Some uncertainties are in variants 4 and 5.In both variants the tasks do not explain clearly where and how to use given numbers (i.e., to use them as data or write directly into formulas).
The second part of the exam required not only theoretical and practical skills but also ability to type prompt, what is not the most important skill of informatics.

Analysis of the Order of the Examination
The first part of the exam is an individual task.It is not compulsory to do this creative task under permanent supervision of informatics teacher in a computer room.This means that a student may unfairly use the abilities of other people and to present a borrowed work as his own after having learned to defend it.
Teachers who have no connection with informatics or computers supervise the second part of the exam.A student accomplishes all tasks, copies the results into a floppy disk, brings it to the computer with the printer, and prints the results.However, a student may copy its work into the computer and another student may use it.

Analysis of Evaluation Criteria
A creative task gives 30 points (10 for the description, 20 for the work itself and presentation).A thorough instruction is given concerning the evaluation of each part.It is stated how many points to give for each quality element.However, the instruction does not specify the size and complexity of the work.In addition, it is not directed what kind of ICT to use.Some works may not require structural algorithmic thinking.Thus, the complexity of a creative work may vary considerably: it could be a new serious program as well as series of plain slides or a simple report.Different teachers' attitude may cause subjective evaluation.Some teachers appreciate programming, some others the ability to use general-purpose software.So, the independent creative tasks could be evaluated not equally.
The tasks of the second part (6 variants) have a strict evaluation structure and criteria.Therefore, this part is evaluated objectively.There is only one drawback is in the evaluation of the theoretical task.This task is given 30 points.Ten points out of them is given to the shaping of the answer.However, the requirements for the layout of the text are not provided in the task.The evaluation criteria of practical questions are detailed, logical, and clear.

The Evaluation of Equality of Variants
On the basis of the correspondence of the tasks and examination program, a statistical study was accomplished.Common descriptive statistic methods together with correlation analysis were applied in order to evaluate the relation between attributes.Regression analysis was applied to predict the dependence of one variable on the others ( Čekanavičius and Murauskas, 2000;2002).A linear regression model was used.Correlation (r) and determination (r 2 ) coefficients that describe correlation and functional dependence

Conclusions
Having analysed the correspondence of informatics exam to the requirements of examination program and general aims of informatics subject we came to the following conclusions: 1.The tasks of informatics exam correspond to the program of examination.
2. The exam helps to check the main students' ICT skills.
3. Examination tasks and instructions are clear and understandable to students.
4. The organization of the examination may influence students' grades (i.e., a creative work may be fulfilled not independently; a teacher not knowing how to work with the computer may not notice students' dishonest works during the second part of the exam).
5. Evaluation criteria allow evaluating students' knowledge and skills objectively.
6.The tasks of all six variants are equal.
7. The practical tasks of the second part of the exam and the independent creative task differentiate students' grades too little.

Acknowledgements
The analysis is carried out by the order of the National Examination Center.

Table 1
Correspondence of the informatics exam theoretical and practical questions to the exam program