Framework for Educational Software Quality Assurance in Lithuania

In 2003 a working group established on the initiative of the Centre of Information Technology of Education under the Ministry of Education and Science, has been reviewing regulations for the assessment and certification of educational software. This article analizes the main aspects of the proposed orders. It discusses general structure of quality assurance system, the procedures for the certification of standard and permanently renewable digital educational resources, and the main criteria and principles of the evaluation of educational software in Lithuania. In parallel, it investigates and summarizes research results and foreign policies and practices on quality assurance. Finally, the article compares them with Lithuanian regulations.


Introduction
Since the arrival of the first computers to general schools, the introduction of information and communication technologies (ICT) into education has made substantial progress: most schools in developed and developing countries of the world are quite sufficiently provided with computers and other infrastructure; the majority of teachers are already trained to use new technologies into their daily life and professional activities, etc. Nevertheless, the situation in schools' provision of appropriate high-quality educational software and content had not improved very much. There are still many problems. For instance: First, schools lack good quality educational software (EC, 2000(EC, , 2003. The production of subject-specific software in non-English languages is still economically problematic, especially for small language communities and developing countries (EENet, 1998;Quale, 2003).
Second, a lot of non-professional developers (e.g., students, teachers) produce and disseminate educational software and content. As a result the number of educational resources steadily increases, however most of them do not satisfy essential quality requirements (OECD, 2001).
Furthermore, even professional software and content producers (e. g., programmers, publishers) do not consider all aspects of educational software and, as a result, their "professional" products fail to meet the essential content, methodological or/and technological quality standards. For instance, the software may provide inaccurate information, imitates the traditional teaching and learning methods only, and/or contains programming errors (Markauskaitė, 1996;OECD, 2001).
Finally, the proliferation of e-content and e-services within and outside the educational market creates new difficulties for teachers to find, evaluate, and select qualitative, reliable and appropriate teaching and learning information sources (Technology Counts'99, 1999). Therefore, school communities are in need of help to cope with this avalanche of irrelevant resources.
For all of these reasons, quality assurance of educational software and content is an important area of ICT policies in many countries across the world (e.g., France, Hungary, Italy, the UK) (EUN, 2003).
The aim of this paper is to analize a framework for educational software quality assurance for primary and secondary (K-12) education in Lithuania. The paper has several objectives: (1) to study international practices and to identify the most traditional ways for the assurance and evaluation of educational software quality in various countries; (2) to introduce the key aspects of a Lithuanian framework for educational software quality control and certification; and (3) to discuss the basic features of the designed framework and substantiate the appropriateness of the mechanism for particular Lithuanian needs.
Several research methods have been used for the investigation in this paper: analysis of research literature, policy documents and reports, comparative analysis, and generalization and synthesis of analytical findings.
The paper is structured into five chapters: This chapter provides the introduction. It investigates and substantiates the relevance of the issue of educational software quality.
The second chapter describes the state of the art of educational software in Lithuania. Initially, the paper investigates the present situation in the provision of general schools in the country with appropriate educational software. Then, it discusses previously adopted national practices for quality assurance. It describes quality control systems of both paperbased and digital educational tools. Finally, the chapter identifies the most important problems relevant to the assessment of educational software in Lithuania. The third chapter investigates models for educational software quality management. Initially, it studies international policies and practices for software quality assessment and control and identifies the most common patterns. Then, the chapter introduces a structure and procedures of a newly designed Lithuanian framework for the certification of educational software.
The fourth chapter briefly discusses the key techniques for the evaluation of educational software. Initially, it reviews research results and international experience. Then the paper introduces the guiding principles and criteria that are considered in the evaluation of educational software in Lithuania.
The last chapter provides the discussion. It argues about the adequacy of the established framework to the present realities of the country. In parallel, it discusses the future evolution of quality management in Lithuania and draws conclusions. Fast development of educational technologies blurred the conventional definition of software and change the meaning of "educational software" to include a large variety of digital tools and content (Buckleitner, 1999). In this paper and in the discussed Lithuanian regulations (MES, 2002a(MES, , 2003a(MES, , 2003b) the definition of "educational software" 1 is used in the broadest meaning. This term refers to a broad range of digital teaching and learning aids either on CD-ROM or other digital storage media as well as in a web-based form. It includes materials in a number of formats, such as text, data, pictures, animations, videos and sound to create a rich learning environment (TEEM, 2003). Specifically, the term covers the following four main categories of software and content: (1) content-rich information sources; (2) content-free learning software and universal learning environments; (3) subject-specific and cross-curricular learning and teaching computer-based tools; and (4) tools for educational content development and learning management.

Provision of Lithuanian Schools with Educational Software
The Educational Software Market. Various institutions and individuals participate in the process of the development and procurement of educational software for Lithuanian schools: commercial suppliers, sponsors, research organizations, governmental institutions, teachers, and other individuals (Markauskaitė, submitted). However, there is a big lack of supply of both foreign and Lithuanian educational software. 2 The majority of software producers and vendors are small little-profitable organizations and individuals. The central governmental agency -the Centre of Information Technology of Education (CITE) -is charged with the responsibility to coordinate the provision of schools (K-12) with educational software. Financial Resources. Funding for schools' provision with educational software is essentially centralized in Lithuania. 3 In 2001-2002, the average annual expenditure for 1 The terms "educational software", "educational software and content", "educational tools", "educational aids" and other terms with a similar connotation are used as synonyms and embody the same semantic in this paper. 2 For instance, in January 2004, only 3 retailers of educational software were listed on the national educational portal "eSchool" (CITE, 2004a). 3 For example, almost 95% of expenditures on educational software were centralized in 2001(EUN, 2003 educational software was 0.95 Euro per student. 4 This amounted to 6.8% of the total allocations on ICT in K-12 education (EUN, 2003). 5 This budget was enough to buy about 7-15 titles of software (the same titles for all schools) and to acquire some pieces of methodological materials annually. The government does not subsidize the creation of new software products, it buys for schools only fully developed and "available on the market" educational software.
Availability of Educational Software. There is a great lack of software for almost all subjects. 6 Both the "available on the market" and centrally provided to schools Lithuanian educational software cover only small and scattered parts of curricula. 7 The majority of available software and methodological materials are for upper grades. 8 The quality of purchased educational software varies and is not always financially reasonable. A number of available software supports traditional teaching practices (e.g., drills and tests). Some tools contain subject-matter inaccuracies and programming bugs. Software tools quickly become obsolete and are not updated. 9

Policies and Practices on Quality Assurance of Educational Tools in Lithuania
Certification of Textbooks. Lithuania has long experience and traditions in assessing and controlling quality of educational aids for general education. A centralised system for the quality control of printed educational tools (e.g., textbooks) was created in early 1990s. The regulations governing the evaluation and certification of traditional tools have been permanently updated to reflect the changing situation in the publishing industry and the educational system. Nevertheless, the basic principles of the assessment remained almost unchanged until today. Presently the centralized assessment and certification applies for textbooks only (MES, 2003e). Publishers may submit a finished manuscript or just a concept of a pre-plan to write a textbook for the evaluation. Special expert commissions 10 are responsible for the certification of textbooks. They appoint up to three independent referees (at least one of them should be a scientist and another one a teacher) who evaluate the submitted material. The referees assess the manuscript or the concept of a textbook 4 By comparison, this figure amounts to 5.1 Euro in Ireland (Meesters and Plas, 2002) and more than 15 Euro in the UK (DfES, 2003). 5 By comparison, this figure accounts for 15%, in the UK (Meesters and Plas, 2002). 6 For instance, during 1996-2003, only 43 titles of educational software had been centrally supplied to general secondary schools (CITE, 2004a). By comparison, 1386 titles of educational software were available in the Netherlands and 1106 titles in Italy in 2001 (Meesters and Plas, 2002), and more than 10000 in the UK in (DfES, 2003. 7 Moreover, the data of the international SITES-M1 study showed that less than 25% of schools possessed any software for such subjects as chemistry, biology, creative arts, history, civics and economics (Markauskaitė, 2003;Pelgrum and Anderson, 2001 against general principles for the evaluation of textbooks and complete a special report form (MES, 2003d). Then based on referees' reports, the commission draws the final decision about the appropriateness of the textbook and, in case of positive assessment, awards a special label for the manuscript or provides recommendations on the future development of the textbook. State schools are allowed to buy only those textbooks that are approved and marked with this special label.
Former Practices of the Certification of Educational Software. De jure, for many years, there were no clear procedures for the assessment and certification of educational software. De facto, during last several years, the quality of all software tools, which were purchased from the state's budget, had been evaluated and approved by a special commission for Information Technologies (IT). Subject-matter and pedagogical specialists in practice were not engaged (or were involved in a purely formal way) in the evaluation. Officially, the process and criteria of the assessment were not defined at all 11 . The members of the IT commission and referees were little experienced in the evaluation of educational software, yet they were absolutely free to apply any methods and criteria for the assessment. As a result, the assessment procedure was non-transparent and unclear, and the quality of expertise was quite poor.

Objectives of Educational Software Quality Regulation in Lithuania
In order to settle the identified problems of educational software quality, the National Strategy of Schools Provision with Educational Aids (MES, 2002a) scheduled the development of special orders for the evaluation and control of educational software quality. As a result, a special regulation on the assessment and certification of digital educational tools was elaborated (MES, 2003b). The main aim of the order is to improve and guarantee high quality of software tools, which are centrally supplied or can be purchased by schools on their own initiative. The document includes two main components and has several objectives: First, the regulation defines the general model and sets up the procedure of the assessment and certification. By doing this it seeks to make the procedure of software assessment and certification more transparent for all: experts, software developers, vendors, and teachers.
Second, the document outlines a technique -i.e., guiding principles and criteriaof educational software evaluation. It seeks to unify the methodology of assessment and improve the quality and reliability of the expertise. In parallel, the regulation aims to help referees to conduct their work professionally.
The next chapters investigate these two main components of the regulation in detail.

International Policies and Practices for the Assessment of Educational Software
The issue of quality control and assurance is an important subject in many states. Countries deal with this question in different ways. Several patterns are traditionally applied for the assessment and quality control of educational software in various countries (Markauskaitė, 2004): 1. Centralized quality control based on a permanent staf. 2. Semi-centralized quality control based on external experts. 3. Decentralized quality management and control based on teachers' peer-reviews and expertise.
The first staff-based centralized model of quality control was established in such countries like Denmark, Ireland, and Singapore. Separate departments, institutions or other permanent personnel are charged with the responsibility to evaluate educational software in these countries.
The second expert-based semi-centralized model of quality control is applied in countries like Belgium and Italy. There, the central national agencies, non-profit organizations or other institutions are charged with the responsibility to organize the assessment and certification of educational tools on a country-wide scale. The agencies hire external professionals (e.g., subject experts and teachers) for the evaluation of educational tools.
The third peer-review based decentralized model of quality assurance is applied in many states too: Canada, Hungary, the Netherlands, and others. In these countries, independent professionals (e.g., teachers, subject-experts) are teamed up into small groups and extensive networks of evaluators are created, 12 while the professional teams are charged with full responsibility to identify, evaluate, and select appropriate educational software and other sources.
The latter peer-review model may have slightly different objectives than formal certification or control. As a rule, the outputs of assessments are targeted at teachers' daily needs. For instance, a reference may include not only assessment of software quality, but provide methodological suggestions about the integration of the tool into curricula as well.
Recent research shows that fast growth of volume of in-house produced content rules out any extension of centralized controls on locally produced e-learning tools (Wood, 2003). Indeed, the explosion of various independent networks, which offer services of educational software evaluation (e.g., MENON, 2004;TEEM, 2003), supports that the control of quality is shifting towards decentralization. Various reviews of national policies also display that the third decentralized pattern of educational software quality management becomes increasingly dominant (EUN, 2003;Meesters and Plas, 2002). Nevertheless, foreign practices prove (e.g., the UK, Australia) that centralized models of quality control do not lose their relevance (DfES, 2003;EdNa and DEETYA, 2002). They are still effective and can be applied to the officially produced or officially disseminated educational materials. Because of that a number of countries (e.g., the Netherlands, Hungary) have established separate procedures for the assessment of educational software, which is created by different producers (e.g., teachers vs. publishers) and those supplied to schools in different ways (e.g., for fee vs. free of charge).
Therefore, initially each country should identify the most important objectives of software quality control. Then, depending on the distinctive situation in its national educational software market, the policy of schools' provision with software and other countryspecific needs, it should choose and establish the most fitting procedures for the management of software quality. The Lithuanian structure and procedure for the assessment and certification of educational software have been designed using exactly this way of thinking.

The Model of the Certification of Educational Software in Lithuania
The Lithuanian policy for the provision of schools with modern educational aids focuses on the systematic implementation of educational software into priority subjects' domains and targeted grades (MES, 2002a(MES, , 2003a. The goal of the policy is to select, develop, provide and implement into curricula a small but rationally selected set of educational tools, which are of high-grade subject and pedagogical quality. Thus, the national policy of schools' provision with educational software emphasizes quality rather than quantity of educational software.
In this case, tight centralized control is more appropriate than an extensive, but less quality-oriented decentralized quality management. The centralized model, which is based on a permanent staff, is also impractical and economically inefficient for the small Lithuanian market. In this situation, the second semi-centralized model fits best for the country. In addition, as should be noted, a quite similar model is already implemented and has been proven fit out for the evaluation of textbooks. Hence, the structure and procedures are obvious to the national experts' commissions and independent referees. Therefore, the semi-centralized model of software quality control has been selected in Lithuania. To avoid unnecessary "novelties", the proposed new framework for software quality assessment and assurance has been kept as close as possible to existing practices: (1) already established order for the certification of textbooks, and (2) earlier adopted "unofficial" practices of software assessment. However, assessment procedures have been carefully adapted to specific features of modern digital educational aids.
The evaluation is organized in the following way. Any educational tool, which falls into the scope of "educational software", could be presented for evaluation. All developers and distributors of educational software may submit their requests. In order to simplify the development of educational tools and improve their quality, the developers are allowed to ask for the expertise of designed, but not yet finished educational tools as well. Therefore, educational software may be presented for the assessment at any of the three stages of their production: 2. Prototype and technical specification of software. 13 3. Final version of educational software.
The evaluation is based on the dual expertise of subject-matter and information technology experts. The evaluation includes three aspects: subject-matter, methodological, and technological. Subjects' experts judge the content and methodological aspects, while IT experts evaluate technological and pedagogical features.
The CITE coordinates the overall procedure of software assessment. The applicant shall lodge in its request and provide a short description of the educational tool: subject area(s), targeted grade(s) and other important information. On the basis of applicant request, the CITE forwards copies of the application to the IT commission and to all other relevant subject-matter commissions. Each commission assigns up to three independent referees (at least one of them shall be a pure subject-matter specialist and another one a teacher) who carry out detail evaluation. Then, the subject commissions sum up the conclusions of referees and decide about the subject-matter and methodological appropriateness of intended to develop or already created educational software. Likewise, the IT commission decides about the technological and methodological suitability. Afterwards, the same IT commission summarizes the decisions of all experts groups and adopts the final resolution and/or recommendations.
The most important is the evaluation of the final version of the educational software. The commissions then decide whether the tool is: 1) certified as the main aid for learning and teaching (and labelled with a special record); 2) authorized as a recommended tool for teaching and learning; or 3) is not approved for the use at schools.
The certificate of the main aid should be granted only to those software tools that satisfy the following conditions: 1) do not contain any subject, methodological and technological shortcomings; 2) comply with the national curricula and educational standards in force (MES, 2002b; and/or 3) correspond to special, adapted for ICT use, educational curricula and standards. 14 The status of the recommended tool should be given for those aids that: 1) are suitable for extracurricular learning; 2) are appropriate for project-based activities or elective subjects; 3) can be used at schools' libraries (e.g., dictionaries, encyclopaedias); 4) are designed for teachers' use only; 5) only partly correspond to the national curricula and educational standards or because of any other minor subject, methodological or technological weaknesses cannot be certified as the main aid for teaching and learning. 13 Foreign or Lithuanian, but not fully fitting to the national curricula educational software may be submitted for the assessment instead of a prototype. Then the applicant shall present a technical specification for software adaptation.
14 According to the national strategy (MES, 2002a), the national curricula and standards shall be adapted for ICT integration in the nearest future. This special curricula and standards are not available yet.
After the final evaluation and approval, the information about certified main aids and recommended tools are respectively entered into special lists (databases) of "Valid" and "Recommended" educational software. In the future, these lists shall be periodically published in the official journal of the Ministry of Education and Science and shall be available on the CITE's website. Software producers and distributors are allowed to present the information about the granted status in the documentation and other places of the educational tools. Each updated version of software shall forego a reassessment and get a new authorization.

Certification of Permanently Renewable Educational Tools in Lithuania
Today learning activities are gradually moving from off-line to on-line (ICT-League, 2002); and various constantly renewable resources are used increasingly more in daily life and various educational activities (e.g., online databases, dictionaries and other permanently updated collections of information). Thus, the determination of appropriateness of renewable tools for education and the assurance of their quality become increasingly more important. The above described procedure of the assessment and certification fits fine for the appraisal of stationary (i.e., unchangeable) educational software. However, it does not fit well for permanently renewable resources. The majority of countries (e.g., Hungary, the Netherlands) utilize the potential of teachers and trust the selection and quality control of renewable resources to various formal and informal professional bodies. However, in general, there is still a great lack of international experience about how to guarantee the quality of changeable software.
The new Lithuanian regulation seeks to make a start in the evaluation of renewable resources. It sets up a special (i.e., adapted) procedure for the assessment and certification of permanently renewable educational software.
A developer of a periodically updateable educational tool (e.g., on-line database) may apply for a special status of "Permanently Renewable Educational Software". The applicant should ask for this status at the same time he makes a standard request for the evaluation and certification of the educational tool. The applicant should provide the following information in his request: 1. Arguments that the permanent update of the software tool offers methodological, didactical or other important advantages over unchangeable educational software. 2. Arguments that it would be impossible or irrational to assess each new version of software under the standard procedure. 3. A detail description of the process of the software update. 4. Proof that the developer is competent to guarantee the quality of each new version of software autonomously.
Initially, permanently renewable software is assessed in the standard way. In case of positive appraisal, the IT commission considers the request to grant an additional status of "Permanently Renewable Educational Software". If experts decide that a regular update is essential for the functionality of software and the applicant is capable of ensuring high quality of each updated version independently, then such tool is conferred a special status (and a special label). This status indicates that the developer is allowed to make changes in the software independently without an external supervision. Nevertheless, the developer is obliged to specify in the software that the tool is permanently renewable and to indicate clearly those parts of the software, which have been changed after the certification. In addition, the IT commission is allowed to set up other extra terms and conditions for the assurance of software quality (e.g., regular reporting about updates, expiry date of the status).

The Main Methods for the Evaluation of Educational Software
The start of systematic evaluation of educational software began in the early 1980s (Buckleitner, 1999). Almost at the same time, the first standardized evaluation methods and instruments were introduced (Squires and McDougall, 1994). The early evaluation methodologies were mainly based on various checklists. The instruments aimed to separate out and quantify the most important factors of software design, which can be associated with the effectiveness of educational tools for teaching and learning. Since that time, the techniques for the evaluation of educational software made a substantial progress. Various sophisticated methods and measures have been suggested by different researchers and evaluators. The evaluation methodologies made headway into several directions.
Second, researchers proposed various innovative techniques and measures for the assessment of the suitability of educational software to particular educational goals and learning contexts: such as the narrative approach (e.g., Mallon and Webb, 1997) and the weighing of various characteristics according to their relevance to specific conditions (Wrench, 2001).
While there is still a lack of consensus among software evaluators about what makes educational software effective, during the last years professional evaluators across the world tend to adopt more holistic techniques of software evaluation. On the one hand they try to introduce criteria and checklists that comprehensively measure all possible subjectcontent, methodological and technological features of software tools (e. g., Roberts, 2002;Semenov, 2001;Wrench, 2001). On the other hand they move the evaluation of educational software away from the "desk" to the "classroom". As a result, comprehensive evaluations now include an additional test of the appropriateness of software for specific educational aims and contexts (e. g., MENON, 2004;Roberts, 2002;TEEM, 2003). This assessment is usually carried out in real classroom settings.
However, each evaluation method has it pros and coins. For example, the MENON (2004) practice demonstrates that the desk-based evaluation is relatively cheap and can be carried out within a relatively short period, but it cannot detect whether the program will be effective in a real classroom setting. Meanwhile, the classroom-based evaluation approach allows to test and comment on the appropriateness of educational tools to concrete teaching and learning objectives, but is on average 2.6 times more expensive and takes twice the amount of time to conduct. Other researches also show that different evaluations carried out by various organizations have their strengths and weaknesses: some of them are more neutral and reliable, whereas others are more specific and useful for teachers (Buckleitner, 1999). Therefore, the assessment methodology should be adjusted to the specific objectives of the evaluation.

Guiding Principles and Criteria for the Evaluation of Educational Software in Lithuania
In the case of Lithuania, the primary aim of the evaluation of educational software is quality assurance and certification of software tools, which are purchased from the states' allocations (i.e., supplied centrally or can be acquired by schools individually). Then, such factors as neutrality, reliability, comprehensiveness, simplicity, transparency, low cost, and duration are the most important features of the evaluation; while, direct utility of assessment reports for practical teaching purposes is less essential characteristic. Thus, the desk-based evaluation technique has been selected for the certification of educational software in Lithuania. It was aimed to elaborate generic and context-free evaluation principles and criteria that could provide objective judgement of any type of software. The main precondition is that the assessment per se shall not give any preference to a particular teaching and learning methodology or a specific goal. The evaluation methodology should regard all educational styles that conform to the national curricula as equally good. In order to achieve that, guidelines for the evaluation of educational software have been developed (MES, 2003b). They delineated general principles and a comprehensive list of criteria that should be considered by experts' commissions and independent referees when they review and make final decisions about the appropriateness of educational software for teaching and learning.
While developing principles and criteria for the evaluation of education software, three main sources of information have been reviewed and generalized: (1) already existing techniques and instruments that are used in Lithuania for the evaluation of textbooks, (2) ideas from research literature about the effectiveness and evaluation of educational software, and (3) assessment criteria and categories adopted by various international and foreign agencies that offer professional services of educational software evaluation. All three sources have been collated and condensed. As a result, a quite comprehensive list of principles and criteria that describe a software tool from various perspectives has been compiled. The condensed ideas have been set down in the document (MES, 2003b).
Evaluation Criteria. First, the Lithuanian regulation introduces general criteria for the appraisal of educational software. According to the guidelines, all educational soft-ware shall fulfil the main universal requirements imposed on the textbooks and supplemental modern aids in Lithuania, such as: (1) observance of the principles of humanity, democracy, nationality, and technological novelty, (2) compliance with the national curricula and educational standards, (3) reference to up-to-date and scientifically reliable information, (4) compatibility with textbooks and other non-ICT learning aids, (5) fulfilment of integrity, perspicuity, visual lucidity and other methodological principles. In addition, those educational tools that are used for teaching and learning as the main educational aids should adhere to all statutory requirements for the textbooks (MES, 2003d).
Second, the document outlines the basic elements of software, which commonly are relevant to students, teachers and other users. It introduces twelve aspects (or components) that commonly characterize educational software: (1) subject matter, (2) psychological and pedagogical aspects, (3) interactivity and flexibility of learning management, (4) user's interface, (5) facilities for users' management, (6) instrumental tools for course and content development, (7) tools for communication and collaboration, (8) technological characteristics, (9) documentation and other supplemental materials, (10) competence of developers, (11) economic efficiency, and (12) other specific aspects of software. In order to help referees to assess each component, the key features that typically describe each aspect of educational software are shortly outlined in the document (see Annex 1). On top, each feature is exhaustively illustrated by the most common examples (Table 1).
The referees are asked to evaluate educational software against the main aspects and their features, sketched out in the guidelines. However, the document acknowledges and warns that the outlined criteria are very generic, so the relevance and importance of various aspects and their features strongly depend on the type and objectives of each educational software. In other words, some features could be critical while some others could be completely irrelevant for the quality of a specific educational tool and its effectiveness in a specific context. Therefore, the criteria should be applied very flexibly and know- Table 1 An example: a fragment from the guidelines for the assessment of educational software (

Psychological
and pedagogical aspects ingly. Each evaluator should individually identify the important aspects and features and shall judge the quality of relevant characteristics only. In addition, the referees are encouraged to keep in mind that some software may have such original functionalities that fall outside of the pre-defined set of criteria. The experts should consider these characteristics as well. Evaluation Principles. The document also describes general aspects of the application of criteria and divides the responsibilities between the subject and IT experts. According to the regulation, all educational aids are evaluated from two perspectives: (1) subject-matter and methodological, and (2) technological and methodological. Both subject and IT experts are asked to evaluate all aspects of educational software.
However, the subject experts shall focus on the examination of subject and didactical aspects, and may assess technological appropriateness only from the position of an "ordinary user" (non-specialist). For instance, from the subject-matter side, they should consider carefully such aspects as learning objectives, accurateness and appropriateness of content for target audience, conceptual integrity of a digital tool with curricula and other existing teaching and learning aids, etc. From the methodological point of view, the subject experts shall focus on the appropriateness of delivery strategies, psychological and cultural fitness for a targeted age, etc.
IT experts should concentrate on the examination of technological and didactical aspects, and may assess subject content from the perspective of an "ordinary learner". From the technological side the IT experts should scrutinize all operational aspects of software, including such features as compliance with standards for usability and accessibility, ergonomity of interface, technical interoperability of content, metadata and other resources (e.g., support to SCORM), etc. From the methodological angle, the IT professionals must pay particular attention to the novelty and efficiency embedded into software e-learning methods. They should consider whether the technological design of a program caters to constructivist practice and authentic tasks, does it support higher order thinking and collaborative learning, whether the software allows interaction, construction and creation of solutions, etc.
Although the guidelines include a standard evaluation form, which could be used by referees for reporting, this form is not considered as a standard single instrument, and the experts are not obliged to use it in their assessments. They could present their reports in any acceptable manner. The above discussed criteria and principles should be applied when evaluating all: final versions of educational software as well as prototypes and technical specifications of new tools.

Discussion, Conclusions, and Future Perspectives
The discussed framework is the first official attempt to control and assure the quality of educational software in Lithuania. The established order could give several improvements into schools' supply with educational software. First, it will guarantee high quality software tools, which are centrally supplied to schools. Second, it will help schools to select and buy qualitative teaching and learning aids on their own initiative. Finally, the regulation will unify the assessment of educational software and make the procedure of certification more transparent for software developers and vendors. As a result this could promote the evolution of the national software industry and trade.
However there are several concerns that deserve more exhaustive discussion: (1) the workability of the evaluation framework into the Lithuanian context, and (2) the validity of traditional assessment approaches for interactive digital aids.
Workability of the Framework. In the Lithuanian context, the assessment of educational software is not an easy task as there is a great shortage of professional staff who are competent to accomplish the expertise. For this reason the proposed framework of educational software quality assurance has been tailored to the present experience and capacities of the national referees, as well as other state's resources.
First, in order to avoid any "unjustified innovations", the evaluation procedures of education software were kept as close as possible to already existing orders for the evaluation and certification of textbooks. Nevertheless, they were fully adapted to the unique features of electronic tools. This made the system of educational software assessment both more natural for non-IT educational specialists and appropriate for digital educational aids.
Second, the Lithuanian system of assessment and accreditation is a trade-off between centralized and decentralized models. On the one hand, centralized management of certification can guarantee a high-grade of educational tools. This suits well for the present realities and objectives of the national educational software policy in Lithuania. On the other hand, the involvement of outside referees allows extending this system to substantially large amounts of software. In addition, in the future, when the market of educational software will be more saturated and experts will grow in experience, this semi-centralized model of quality control could be easily transformed into a less rigid decentralized system of quality management.
Third, the same independent subject experts are currently involved in the evaluation of both textbooks and educational software. Therefore, the evaluation procedure of educational software rationally employs existing capacities of the national experts. This simultaneously brings several other advantages: (1) guarantees professional judgement of a content and subject-specific didactical aspects, and (2) indirectly ensures good compliance of evaluated tools with the national curricula and warrants high integrity of software with the traditional teaching and learning aids.
Lastly, the desk-based method is cheap and affordable for Lithuanian budget. At the same time, the outputs of desk-based evaluation fulfil present needs and objectives of centralised quality management quite well.
Validity of Assessment Approaches. The established framework of software evaluation absorbs many features from the existing system for the assessment of textbooks. The research warns that the task of evaluating digital tools presents unique challenges not previously encountered in the evaluation of traditional educational tools (Buckleitner, 1999). For this reason all elements of the assessment have been accurately adapted to the peculiarities of digital aids.
First, the proposed dual-side expertise of subject and IT experts assures fully-rounded assessment. Cross-covered evaluation of methodological aspects assures that the didactical design of a tool will be appropriately evaluated from both subject-specific and elearning perspectives. Moreover, the similar operative word is given to pedagogical referees and IT experts. It is expected that this will lead to well-rounded quality of educational software.
Second, the assessment procedure is extended to permanently renewable educational resources. The present level of extension meets well the existing needs for the assessment of such type centrally supplied learning resources. In addition, the model of certification of renewable resources is a good "pilot" of less rigid decentralized pattern of quality management, which in the future could be introduced for the assessment of freeware learning aids.
Third, the proposed assessment criteria consider highly the compliance of software to the international e-learning (e.g., SCORM), accessibility, and Lithuanian standards. This shall improve both the quality of educational tools and all steps of the software life cycle (e.g., production, distribution, brokerage, maintenance, re-engineering).
Finally, the possibility of getting professional expertise in early stages of software development (i.e., to submit for evaluation a technical specification or prototype) should help developers improve the quality of software tools, too. Experts' conclusions could be a good cue for software developers about the future commercial strength of evaluated products. This may encourage private companies to invest more into software development.
Summarizing all of the above, the discussed Lithuanian framework for the assessment and certification of educational software absorbs the best present international and national attainments in this field. The framework both fully complies with the present Lithuanian needs and provides for future evolution. L. Markauskaitė is a postdoctoral fellow at the University of Sydney, Australia. She received a PhD in informatics in 2000. Her major research interests are qualitative and quantitative research and data analysis, development of ICT literacy, cross-national strategies and practices of ICT introduction into secondary school as well as all other aspects of ICT implementation into education.