Educational Robotics Applications for the Development of Computational Thinking in a Brazilian Technical and Vocational High School

. Nowadays, solving problems is substantial for the social relationship human. Computational Thinking (CT) emerges as an interdisciplinary thought process encompassing mental abilities to help students solve and understand problems. Researchers invest in the methodological proposal of activities aimed at CT stimulation, educational approaches, and the conception of technologies that support these activities’ execution. Educational Robotics (ER) is one of these technologies that stand out at different educational levels to favor teamwork, logical thinking, and creativity, skills intimately articulated with the computing paradigm. The main objective of this work is to investigate the impact of ER activities on CT development and subjects learning in the Technical and Vocational Education in High School. For this, we accomplished a study of intervention research type with students and teachers analyzing quantitative and qualitative aspects. The results indicate that the introduction of ER can favor students in the development of CT skills and learning High School subjects.


Introduction
Computational Thinking (CT) is a way of solving problems using skills based on Computer Science concepts.Wing mentions that it is essential to include CT skills training at an early age (Wing, 2006).However, there is no CT formal definition and reports on how to integrate it into High School subjects.Studies are still exploring the processes, tools, and evaluation instruments to enable effective adoption (Alexander et al., 2010;Araujo et al., 2019).
Brazilian institutions have been promoting the CT culture to improve problem-solving skills, such as the Brazilian Computer Society1 .Its computer training references for Basic Education (BE) specifies CT as one of the essential knowledge axes of Computer Science (Raabe, 2017).Aswell as, with the Reference Curriculum in Technology and Computing2 , the Innovation Center for Brazilian Education gives support to Brazilian BE institutions in order to include technology and computing subjects in their curricula (CIEB).These initiatives highlight the importance of inserting technological resources within the scope of BE and offer guidelines on how to work with them as a pedagogical tool.
Educational Robotics (ER) is a multidisciplinary approach that considers design, assembly, and use of robots based on principles of engineering, computing, mathematics, and physics.However, ER has been used for teaching Science, Technology, Engineering and Mathematics (STEM), and Computer Science at different levels (Rogers and Portsmore, 2004;Sisman et al., 2020).The inclusion of ER in education can favor the learning process, since ludic activities can stimulate CT, contributing to increase the students' engagement in STEM and, consequently, improving their problem-solving skills (Chiazzese et al., 2018;Gerosa et al., 2019).
Many studies on the use of ER make specific and short-term investigations (Chaudhary et al., 2016).At the same, time they consider the ER to teach a particular knowledge (Catlin and Woollard, 2014).However, as far as we know, there are no studies that analyze the ER effects by itself can have on CT skills of High School students.Likewise, there are no references in literature to longitudinal studies that observe the CT skills progress and their relation with the ER use.
The objective of this study is to apply ER in a Brazilian Technical and Vocational Education (TVE) in the modality integrated with High School and evaluate the impact on both developing CT skills and improving their academic performance of High School students.Two research questions guide the study: (RQ1) What impact does the introduction of ER have on CT skills on students in a Brazilian TVE High School?(RQ2) What impact does the introduction of ER have on students' academic performance in a Brazilian TVE High School?
This research is a long-term study based on a research-intervention method with a sample of 36 students and 18 teachers from the 1st year of TVE High School in the Paraíba state, considering the Computing subject during 2018.The students were divided into two groups, one was submitted to the ER courses proposed (experimental), and the other was not (control).The development of CT skills was evaluated through the Bebras Challenge, and the annual performance obtained from the students' academic performance in High School (HS) was considered to observe student development.The results showed that the experimental group outperformed the control group on the Bebras Challenge and academic performance.
This paper is organized as follows: Section 2 briefly describes the key concepts.Section 3 is dedicated to related works.Section 4 details the methodology.
In Section 5, we present the results, discussions, and threats to validity.
In section 6, we detail the main lessons learned in this study.Finally, the main conclusions and future works are presented in Section 7.

Background
This section presents the key concepts of the study.

Brazilian Basic Education
In Brazil, as established in the Law of Directives and Bases of National Education (LDB) (Brasil, 1996), BE ensures students the necessary conditions for the exercise of citizenship and their qualification for work.Brazilian BE is organized in 3 stages, such as Early Childhood Education (pre-school), Elementary School, and High School (HS).
Early Childhood Education aims to stimulate children's physical, psychological, intellectual, and social aspects up to 5 years old.Elementary School is the most extensive Brazilian education stage (nine years) and attends students between 6 and 14 years old who experience a set of physical, cognitive, affective, social, emotional transformations, among others.
Finally, the HS stage is the lasts for three years and its main objective is to enhance students' critical sense and intellectual autonomy.During HS students have access to knowledge in the areas of Languages and their Technologies (Portuguese, Art, Physical Education, and English), Mathematics, Natural Sciences and its Technologies (Biology, Physics, and Chemistry), and finally, Applied Human and Social Sciences (Philosophy, Geography, History, and Sociology).
Although the Brazilian HS education is based on the LDB, each Brazilian state has the autonomy to propose operational guidelines that meet local educational needs.Brazilian HS is organized in two categories: general upper secondary education and vocational upper secondary education (Brasil, 1996).General upper secondary education has at least three years of study and a predefined subject curriculum that can be offered in full or parttime.In the end, the upper secondary education student receives a High School diploma.Vocational upper secondary education is offered in conjunction with general upper secondary education and aims to develop student skills for a productive life.With its own curricular organization, vocational upper secondary education provides professional qualification in different work areas in three years of studies in full time.In the end, the upper secondary education vocational student receives a High School with a professional qualification.
Paraíba State offers different TVE courses that integrate science and technology.These courses aim to prepare students as working citizens, to meet the requirements of the productive sector and socio-economic development.In this context, one of the courses offered is Computing, which includes subjects such as Introduction to Informatics, Hardware Architecture, Artificial Intelligence, Operating Systems, Programming Language and Programming Logic.

Computational Thinking
CT is related to the cognitive process of reasoning or problem-solving by applying computational skills (Araujo et al., 2018).According toWingWing (2006), CT involves essential skills for all, not only computer scientists, arguing in favor of including CT in activities to develop the analytical skills of all children, as is done with writing, reading, and arithmetic.
In literature, researchers usually classify CT as concepts and skills to propose a formal definition.On the one hand, Blikstein (2018) argues that CT aims at developing several skills such as productivity, inventiveness, and creativity; and he suggests that there are two phases of this thinking: abstract the problem and program a computer to solve the problem more quickly (Blikstein, 2018).On the other hand, Wing argues that the CT characteristics can be grouped into six lines, which are: it is more related to conceptualizing and not programming, it is fundamental and not utilitarian, it refers to the way humans think and not computers, it integrates mathematics and engineering, it concerns building ideas and not building products, and it is for everyone anywhere (Wing, 2006).
Since the initiative in 2006Wing ( 2006), research on CT has been growing.However, there is still no CT formalized definition, as well as the set of associated CT skills.Thus, Araujo et al. (2018) mention that the consideration of CT in distinct and isolated skills can be evolved into classification by skill groups, since the mental capacities involved in each skill may be familiar to others.Therefore, it is observed that CT is a concept present in human development, which demonstrates its importance.However, it is a scientific field that still requires research.

Educational Robotics
Nowadays, reflections show the need for new teaching and learning practices that allow students to acquire and maintain knowledge.This need justifies the use of technologies in the educational environment, such as software applications, games, and robotics.According to Schons et al. (2004), educational technologies, if well applied, can help students to develop cognitively.In recent years, ER has been considered a powerful learning tool that helps students develop their knowledge, both in the STEM subjects (Science, Technology, Engineering, and Mathematics) and in Computer Science.
ER arises as a means to contribute to the teaching-learning process in all education system levels by using Robotics (Santos et al., 2016).ER is considered as an evolution of Papert's LOGO (Seymour, 1986).Besides, the ER can integrate knowledge to the simulation of scientific methods because the student formulates hypotheses, implements, tests, observes, and makes the changes he considers necessary for the robot objectives (Maisonnette, 2002).The ER initiatives in HS more discussed in the literature comprises several purposes: teaching programming, teaching robotics as science, interdisciplinary science teaching, CT development, participation in tournaments and Olympics (Souza et al., 2018b).Although the use of the educational practices with ER is diverse, it is possible to identify practical situations involving problem-solving, whether to teach STEM or programming logic, the student is involved in "learning by doing" (hands-on).These ER characteristics can favor the application of concepts related to problem-solving how it is observed in CT skills, such as abstraction, decomposition, simulations, data collection, and analysis.Moreover, results published in the literature indicate that ER tends to stimulate the students' performance, with the teacher being a mediator (Zilli, 2004).

Bebras Challenge
Bebras3 is an international community of Computers and Education field that seek to disseminate CT among school students at all age.The Bebras community promotes an annual workshop that come together teachers and researchers worldwide to formulate questions that explore the CT skills (Dagienė et al., 2017) and that will compose the Bebras Challenge test.The questions are written in English and are available for countries that want to join the challenge to set up their tests and apply to their schools.The questions of the Bebras Challenge are related to CT skills, but the participants do not need programming knowledge to answer the test Dagienė et al. (2017).Several initiatives have applied the Bebras to measure CT skills in different contexts Araujo et al. (2019Araujo et al. ( , 2017)); Chiazzese et al. (2018); Palts and Pedaste (2017).

RelatedWork
In Brazil, studies concerning CT demonstrate its importance for High School education.Rodrigues et al. (2016) analyzed the CT effects on the academic performance of High School students and found a correlation between the programming skills and the performance of students on the Brazilian High School Exam (ENEM, equivalent to the Scholastic Aptitude Test -SATs).Costa et al. (2017) conducted a quantitative analysis with Brazilian students (under 15 years) to assess the impact of using CT in a joint approach with Mathematics obtaining positive results on students problem-solving skills.In the context of High School robotics, Souza et al. (2016) demonstrated that teaching robotics aligned to CT skills had a positive impact on academic performance of High School students.
Considering the literature on RE, it aims at simplifying the subject teaching, enabling the development of several skills, such as teamwork, logical reasoning, and creativity (Hirst et al., 2003;Choi and Lee, 2020).Seymour (1986) highlights that by using robotics in High School education, it is possible to teach students CT skills, as the use of robots in education provides a beneficial environment for learning, encouraging the student to build his knowledge using abstraction, decomposition, simulations, data collection, and analysis.
In Chiazzese et al. (2018) the authors analyzed the effect of ER on learning CT skills in primary school students using ER activities.In this case, CT skills were evaluated using the Bebras Challenge, and the results indicated that ER could be useful for the development of CT in students.
Our study is similar that presented in Chiazzese et al. (2018); Choi and Lee (2020) in the sense of introducing ER in education to develop CT skills and consequently helping students to perform better in subjects, however we applied our study TVE in High School.Besides, we also used the Bebras Challenge as data collection instruments.However, in our study, we investigate how to introduce ER in the school environment integrated with the curriculum by conducting a long-term study during six months, differently from the proposal presented by Chiazzese et al. (2018); Choi and Lee (2020).Also, we carried out a statistical analysis of the students' performance in each subject, considering the students grades.
The results presented in Chaudhary et al. (2016) demonstrated that designing, building, and programming robots is exciting for students and increase their level of involvement with learning, improving the student computational and logical thinking skills performance after performing activities with ER.The authors also highlighted that the teaching methodology applied was useful to transmit the desired skills and knowledge to High School students.
Like Chaudhary et al. (2016), we considered the basic concepts of computational and logical thinking skills; however, we did not limit the teaching of these concepts to the programming of robots.We explored the theoretical study through the practical work building stuff with robotics kits without necessarily programming robots.In this specific case, we proposed projects, and the students had to create something using robotics kits and our orientations.For example, one of the projects involved the displacement on non-uniform surfaces In this scenario, the students created vehicle prototypes for these surfaces.At the project end, we discuss about the motor rotation, the direction of rotation, buttons combination, sensors, and energy sources applied to displacement on non-uniform surfaces.We believe that it is possible to promote several academic skills through the robotics kits involving the student in the discovery process.In this sense, ER is just an educational tool for science learning.Also, we use the Bebras Challenge to observe CT skills during the ER courses, as it does not require the student to know a specific programming language.
It is essential to highlight that the proposals applied in the educational environment,an educational project defined for ER application in the classroom, whether formal or informal, are not identified in the works mentioned above.Some works highlighted the objectives and the steps applied, however, do not present elements such as pedagogical materials, lessons plans, and methodological guidelines for the replication of the applied practices.
Finally, few studies evaluates quantitative and qualitatively the effects that an ER causes on the CT skills, as well as on the cognitive development of students in High School.This limitation may be related to the lack of validated instruments that help to measure these skills.

Methodology
This study is a long-termstudy based on a research-intervention method because it intends to identify the real difficulties, support from teachers, presentation of solutions, proposition and execution of interventions, and the judgment of the results obtained after interventions (Cohen et al., 2011).

Research Design
The development of this work was divided into five phases: Application of Self-Discovery Robotics step.

2)
Application of Computer-Based Robotics step.

3)
Application of a survey among subjects' teachers.

4)
Analysis of the effects of these ER courses on students performance.

5)
In order to achieve the objectives of this study we defined the following research questions and hypotheses: RQ1: What impact does the introduction of ER have on CT skills on students in Brazilian TVE High School?H1.0:There is no evidence that the introduction of ER can impact the CT skills of

Sample and Data Collection
In this study, we considered a sample of 36 students and 18 teachers from the 1st year of a High School in the Paraíba state, Brazil.We organized the students into two groups (control and experimental) with the distribution of students being 44.4% ( 16) from the control group and 55.6% (20) from the experimental group (see Table 2).The control group was composed of students who had no contact with ER, whereas the experimental group consisted of students who had contact with ER courses proposed in this study throughout the academic year of 2018.
In order to determine whether our sample of students was representative, we applied the sample size calculation in Equation 1, where  = calculated sample,  = population size,  = standardized normal variable associated with the confidence level,  = true probability of the event, and e = sample error.
group and 55.6% (20) from the experimental group (see Table 2).The control group was composed of students who had no contact with ER, whereas the experimental group consisted of students who had contact with ER courses proposed in this study throughout the academic year of 2018.In order to determine whether our sample of students was representative, we applied the sample size calculation in Equation 1, where n = calculated sample, N = population size, Z = standardized normal variable associated with the confidence level, p = true probability of the event, and e = sample error.
Given that the size of the students' population of 42 students, the result of the sample size calculation indicates that our sample is representative with an error (the difference between estimated and real numbers) of 6.25% with 95% of confidence (the probability that the useful sampling error is less than the admitted sampling error).Similarly, the size of the teachers' population of 32 teachers, the result of the sample size calculation indicates that our sample is representative with an error of 15.52% with 95% of confidence (the probability that the useful sampling error is less than the admitted sampling error).
We conducted the data collection between April and December 2018.This data collection contemplated quantitative and qualitative data regarding the students' cognitive development during activities with ER.These data were obtained by applying a survey, the Bebras Challenge, and collecting the students' academic performance during the period concerned.Besides, we applied a survey with the subjects' teachers from the school where this study was conducted to understand their perception about ER and their past professional experiences with ER.The subjects' teachers participated in our research to provide information about both groups (control and experimental) because they support the students during the 2018 school year in subjects learning.Our study intents are also observing the ER impact on students' academic performance.In this sense, the subjects' teachers do not participate in the ER courses proposed in this study nor taught ER to students in their class.

Participants Profile
(1) Given that the size of the students' population of 42 students, the result of the sample size calculation indicates that our sample is representative with an error (the difference between estimated and real numbers) of 6.25% with 95% of confidence (the probability that the useful sampling error is less than the admitted sampling error).Similarly, the size of the teachers' population of 32 teachers, the result of the sample size calculation indicates that our sample is representative with an error of 15.52% with 95% of confidence (the probability that the useful sampling error is less than the admitted sampling error).We conducted the data collection between April and December 2018.This data collection contemplated quantitative and qualitative data regarding the students' cognitive development during activities with ER.These data were obtained by applying a survey, the Bebras Challenge, and collecting the students' academic performance during the period concerned.Besides, we applied a survey with the subjects' teachers from the school where this study was conducted to understand their perception about ER and their past professional experiences with ER.The subjects' teachers participated in our research to provide information about both groups (control and experimental) because they support the students during the 2018 school year in subjects learning.Our study intents are also observing the ER impact on students' academic performance.In this sense, the subjects' teachers do not participate in the ER courses proposed in this study nor taught ER to students in their class.

Participants Profile
Although the focus of our study was to investigate the cognitive development of students, teachers were also involved aiming at giving qualitative information related to the student's behaviour in the classroom during the study.Therefore, the participants in this study are teachers and students in the 1st year of HS.

Students Profile
The students' control and experimental groups were from distinct school classes.The experimental group choice was considering the availability of the two classes hours together per week in Mathematical and Programming Logic subject.This study was applied in the school as part of the course program of the 1st year of HS.In this sense, all students had the task of participating in the activities proposed in this study.The students from the control group participated only in the survey application and in the Bebras Challenger, while the students from the experimental group also participated in ER courses proposed in this study.
The whole sample was composed of 36 students, where 19.4% (7) were male and 80.6% (29) female.Considering the division into groups, the number of female students was also higher than male in both groups (see Table 3 for more details).
Considering the experimental group 20% (4) were male and 80% ( 16) female while in the control group 18.8% (3) were male and 81.2% (13) female.The average age of sample students was 15.6 years.However, the age of students in the control group was slightly above this average (16.1 years) while the average age in the experimental group was 15.3.This difference in average was due to a student older than 20 years in the control group.In the analyzed sample, 2.3% (1) are students who were repeating the 1st year of HS, and 97.2% (35) are students who were taking it for the first time.
Finally, 100% (36) of students declared that before the activities with ER, applied during this study, they had no knowledge about ER; however, they were interested in learning about it.

Teachers Profile
A total of 18 High School teachers participated in this study.The teacher's sample represents 56.2% of all 32 teachers (population) related to the first year of High School in 2018 at the chosen school.Among those 18 teachers, 38.9% (7) were female and 61.11% (11) were male (see Table 4).
Considering the subjects taught by the teachers, 5.6% (1) of them taught 3 subjects, 5.6% (1) taught 2, and 88.8% ( 16) taught only 1 subject to the students in the control and experimental groups.Except for mathematics, the subjects were taught in the control and experimental groups by the same teacher (see Table 5).
Using an online form, we asked teachers about their ER experiences as shown in the Table 6.Concerning the experience with ER, only 50% (9) of teachers had some experience with this educational resource and 55.6% (10) of them knew about the ER materials available at school.While more than 50% of teachers knew about these materials, 100% Teachers who highlighted that do not use RE in the classroom showed some reasons for it.Among the teachers, 5.6% (1) did not know how to use ER in the subjects they taught, 16.7% (3) did not have time to participate in ER training classes, 55.6% (10) had no knowledge about ER or about the respective materials available at school.Finally, 22.1% (4) had other reasons for not using ER.

Robotics Course
Our ER course was based on the concepts of Papert Construction (Seymour, 1986) and in commercial ER teaching methodologies such as LEGO ® Education (ZOOM, 2010) and Fischertechnik GmbH ® (Fischertechnik, 2018).We organized the activities in two steps: Self-Discovering Robotics and Computer-Based Robotics.The first step lasted two months in the first half of 2018 (second bimester 4 ), and the objective was to promote student autonomy when interacting with robotic materials from Fischertechnik GmbH ® company.The second step also lasted two months but in the second half of 2018 (third bimester).Moreover, the second step was focused on the structured learning of robotics based on the fundamentals of computing.More details of each step are presented in the following sections.

Self-Discovering Robotics
The Self-Discovering Robotics step was developed during normal school-hours of the school and aimed to provide students with knowledge from Sciences such as Physics, Mathematics, Portuguese, Technology, and the creation of their robotic solutions to Science problems.In other words, the Self-Discovering Robotics step did not propose the robotics teacher as a science, but rather to promote interdisciplinary teaching using robotics kits to externalize scientific concepts commonly studied in Brazilian basic education schools.The idea of this step was to allow students to develop their robotic activities autonomously, understanding concepts, such as rotation and displacement from Mathematics and Physics, without programming robots. 4In Brazil, the school year is divided into two semesters and each semester has two bimesters.We considered the Physics teachers in the course planning to define the contents necessary for the students.Thus, the Physics teacher chose the Science concepts considered on the course.Also, the Physics teachers aimed at selecting the contents that could be covered in the revision, as well as those of other sciences that were the basis to learn Physics.
The scope of the Self-Discovering Robotics step is described below: Syllabus: The activities used in the Self-Discovering Robotics step are available online5 .An example of the Self-Discovering Robotics can be seen in the Appendix B.

Computer-Based Robotics
The activities of the Computer-Based Robotics step was developed during normal schoolhours of the school and aimed teaching robotics based on fundamentals of computing in order to stimulate the CT skills through practical exercises.In this context, teachers are not only a knowledge mediator as presented by Seymour (1986), but also they are responsible for "teaching" concepts that are not commonly part of the Brazilian High School curriculum.
The step content was defined in accordance with the syllabus of the second half of 2018 for the Mathematical Logic and Programming subjects being taken by the students.Teachers of these subjects were responsible to indicate the contents to be addressed in the step and those that could be included in the contents review.Moreover, they suggested those contents from other sciences that are the basis for learning Mathematical Logic and Programming.The specific contents of robotics were defined considering basic concepts that could be worked in parallel with Mathematical Logic and Programming.

Hands-on: ○
Theoretical concepts application through the robotics manipulation, from a problematic situation.
In each class, we presented the concepts of Robotics in connection with those of Computing to facilitate the understanding of the areas.The proposed activities for the Computer-Based Robotics step are available online6 .

Instruments
We conducted the data collection of this study long termly to gather quantitative and qualitative data related to the student's cognitive development during the robotics course.
We used the Bebras Challenge applied in the UK in 20157 as an instrument to measure the CT skills acquired during the robotics courses.This test consisted of 16 questions designed to students aged 13 years or over.Currently, the test is not performed in Brazil, but we translated the test into Portuguese with the help of other researchers from our research group.
We considered the student's bimonthly and annual grades in school to represent their academic performance throughout the intervention in 2018.These grades result from the evaluation of the teacher in the respective subjects.Thus, the aspects evaluated were stipulated and applied by subject teachers following the standards of the Brazilian TVE High School.
Considering the qualitative data used in this study, it concerns aspects related to the students' learning progress in classroom, reported by their teachers.When observing qualitative elements, it is necessary to maintain substantial contact with the objects and people that has been investigated (Easterbrook et al., 2008), consequently the teacher's experience with students, during the 2018 school year, was essential to achieve the objectives of this study.The instruments used in this study are available online 8 .

Research Code of Ethics
As this study involved human beings, it followed the resolution 196/96 of the National Health Council of Brazil.Thus, we registered this study proposal on "Plataforma Brasil" 9 under the number CAAE 90723918.5.0000.5182.Before starting the established activities, we explained to the participants about actions that would be carried out; each participant read and signed the appropriate terms of consent for carrying out this study.Besides that, we treat all data anonymously.All the terms of the Code of Ethics for this study are available online 10 .

Results
In order to answer the research questions defined, we analyzed the results obtained from the experiment involving the students of the control and experimental groups.Each group had a different number of students, as illustrated in Table 7.In this section, we present the data analysis procedures and results per research questions.Finally, the threats to validity are described.

Data Analysis Procedures
For the analysis, we grouped the subjects by area: Natural Sciences (Biology, Physics, and Chemistry), Languages (Portuguese, Art, Spanish, and English), Humanities (Philosophy, Geography, History, and Sociology), Mathematics, and Computing (Introduction to Informatics, Hardware Architecture, Operating Systems, Programming Language, and Programming Logic).After that, we associate the students' academic performance to each of these groups.The performance related to CT skills were analyzed based on the Bebras Challenge.Table 8 describes the steps considered in the experiment and the kind of metric (Bebras or academic performance) used in the analysis.
We analyzed the statistical assumption of normality and homoscedasticity using the Shapiro-Wilk and Levene hypothesis tests, respectively, considering a significance level of  = 0.05 in the students' academic and Bebras Challenge performance by group.
The results 11 obtained showed that the assumptions of normality and homoscedasticity are not met, then we need to adopt non-parametric statistical tests.
The Mann-Whitney U test was chosen, due to non-parametric data, to evaluate if there is a significant difference between the performance of both groups (experimental and control) in CT through the Bebras result, as well as in the academic performance.Cohen's effect size index was used to calculate and analyze the robotics course effect under the students' academic performance (Cohen et al., 2011).Finally, we considered the Spearman correlation r to analyze the correlation between CT and the subjects' performance (Spearman, 2008).
In the tests, we used the R programming language and considered a confidence level of 95%, a statistical significance of  = 0.05, and the Mann-Whitney U test since we were dealing with non-parametric data.Also, the specific statistical assumptions of the test were obeyed.It is important to emphasize that this test addresses the varying conditions of the data sample size to maintain the validity of the results.

RQ1: What Impact does the Introduction of ER Have on CT Skills on Students in a Brazilian TVE High School?
We analyzed the mean and standard deviation (SD) of the students' performance on the Bebras Challenge applied after each robotic course conclusion, named Step 01 (Post-Test I) and Step 02 (Post-Test II) in Table 7.The performance of each group is shown in Table 9.Note that the experimental group achieved an average performance of 30.3% better than the control group in the Self-Discovering Robotics step (Post-Test I) and 35.2% in the Computer-Based Robotics step (Post-Test II).
We argue that the difference in the students' performance in each group was due to the positive effect of the robotics courses.The students in the experimental group 11 The statistical assumptions: p-value of the Normality and Homoscedasticity Data is available in appendix B. .This means we found some evidence that ER is beneficial to development of CT skills in the students.In order to refute the null hypothesis H1.0:There is no evidence that the introduction of ER can impact the CT skills on students in a Brazilian TVE High School; we applied the Mann-Whitney U test with a confidence level of 95% and significance  = 0.05.Analyzing the result of the U Test (see Table 10), we observed the p-value   for Post-Test II.Thus, we assumed the alternative hypothesis H1.1:The introduction of ER can impact on CT skills on students in a Brazilian TVE High School; when it is worked with the computing support, as was done in Computer-Based Robotics step.
The U Test result did not show significant differences between the experimental and control groups' performance in Post-Test I ( p-value   ).Thus, it is only possible to confirm that the Self-Discovering Robotics step significantly impact the students CT skills.However, when calculating the Cohen's d effect size of the robotics course on the experimental and control groups, it is possible to observe a positive impact attributed to the Self-Discovering Robotics step (Post-Test I) and Computer-Based Robotics step (Post-Test II) (Cohen et al., 2011).As Cohen's d is equivalent to the Z-score, a standard normal distribution table (Coe, 2002), the query in that table shows that the Cohen's d effect size of the Self-Discovering Robotics step (Post-Test I) was medium.Thus, 77% of students in the experimental group had an average performance higher than the control group.In the Computer-Based Robotics step (Post-Test II), the Cohen's d effect sizewas larger, and the percentage of 83% was favorable to the experimental group.Therefore, it is possible to conclude that ER can positively and significantly impact the development of students' CT skills.These results are according to several studies that involve ER as a tool for CT improvement (Blancas et al., 2020;Chevalier et al., 2020;Roussou and Rangoussi, 2019).However, it is important to point out that in their majority, these studies use a semi-structured interview to get the students' data as had been done by (Blancas et al.,  2020; Roussou and Rangoussi, 2019).We believe that to measure CT skills it is necessary to consider assessment tools that favor the CT evolution.

RQ2: What Impact does the Introduction of ER Have on Students' Academic
Performance in a Brazilian TVE High School?
We analyzed the mean and standard deviation of students' academic performance after the robotics courses.In this case, the percentage of students in the experimental group who presented a higher performance than the control group, varies from 1.1% to 31.1 in Step 01, from 1% to 12.7% in Step 02, from 1.2% to 16.6% in Step 03 (see Appendix B).
Considering these results, we conclude that ER improved students' learning in the subjects analyzed, both through the Self-Discovering Robotics and Computer-Based Robotics steps.In order to refute the null hypothesis H2.0:There is no evidence that the introduction of ER can impact on students' academic performance in a Brazilian TVE High School; we applied the Mann-Whitney U test with a confidence level of 95% and significance  = 0.05.
The U Test (see Appendix B) showed a p-value   in the following groupings: Natural Sciences in Step 02, Mathematics in all Steps, and Computing in Step 01.Thus, we assume the alternative hypothesis H2.1:The introduction of ER can impact on students' academic performance in a Brazilian TVE High School.It is important to remark the evidence of impact in all steps of the study considering the Mathematics subject.
The results obtained in the U Test did not show significant differences between the students' performance in the control and experimental groups considering the other subject groupings.In these situations we have p-value   (see Appendix B), consequently we cannot say that ER has a significant impact on the on the learning of the subjects for both Self-Discovering Robotics and Computer-Based Robotics steps.However, when analyzing Cohen's d effect size of the control and experimental groups, we observed a positive effect attributed to ER for the two courses.
According to the Cohen's d metric (Cohen et al., 2011), we observed specific results for the subjects groupings.In the Natural Sciences group (see Appendix B), the insertion of ER caused a large effect in Step 02 and medium effect in Step 01 and Step 03.In Languages, ER caused a medium effect in Step 02.Human Sciences group had a small effect in Step 01 and Step 02, and a medium effect in Step 03.For the Mathematics subject, ER caused a major effect in all steps.Finally, in the Computing group, ER caused a medium effect in Step 01 and Step 03.
Finally, from the standard normal distribution table (Coe, 2002), the percentage of students in the experimental group with better performance than those in the control group are different in each step: it varies from 56% to 96% in Step 01, 58% to 91% in Step 02 and 60% and 84% in Step 03.
We also considered qualitative aspects related to the students' behaviour along the study under the perspective of their teachers.At the end of the 2018 school year, teachers were asked to complete a survey in order to gather information about their students' behaviour in terms of participation, argumentation, and learning observed in classroom.
The data collected was analyzed by comparing students in the experimental and control groups, before and after the introduction of ER in the school.
Considering the experimental group, the students effectively participated in the 70% of the subjects in Step 01 and 90% in Step 02, according to the teachers' point of view.The students of this group increased the participation in the subjects in 20%, after having contact with the ER activities.The students of the control group, who had no contact with the ER, effectively participated in the 65% of the subjects in Step 01 and 80% in Step 02, which represents an increase of participation in 15% (see Fig. 4).
Regarding the quality of the students' arguments in the subject classes, students of the experimental group presented satisfactory arguments in 50% (10) and 95% ( 19) of the subjects before and after the activities with ER, respectively.This means an increase in arguing ability of 45% from the teachers' point of view.On the other hand, students in the control group improved their arguing ability in 25% (see Fig. 2).
It is well know that external factors such as personal and health problems can interfere with the students' academic performance.We asked about the students' ease of learning the subjects taught by teachers.In the experimental group, students showed ease in learning in 60% ( 12) of the subjects, before the activities with ER, but after such activities the percentage reached 100% (20) of the subjects.In the control group, the results showed ease in learning in 60% ( 12) of the subjects for both before and after the activities with ER.In this case, according to the teachers' impressions, students of the experimental group showed an increase of 40% in learning the subjects.In contrast, there was no impact in the control group (see Fig. 3).
Answer number in percentage (%) Although the U Test did not give evidence to ensure the ER has significance in all subject groupings, we observed that the methodologies applied in this study could positively and significantly influence the students' academic development.Considering the students' performance in the Bebras and the school subjects, the experimental group showed better performance than the control group.Besides, the information provided by the teachers corroborates with these results, strengthening the alternative hypothesis H2.1:The introduction of ER can impact on students' academic performance in Brazilian TVE High School.Finally, it is essential to highlight, from the results, that the use of ER as a supporting tool in education causes a positive and significant impact on mathematics teaching and learning.

Threats to Validity
The validity of a study is related to the reliability established with the applied methodological process (Travassos et al., 2002;Jacques, 2007).Therefore, it is necessary to consider aspects such as construct, internal, external, and conclusion validity.
This study has threats to validity.Concerning the survey, there may be threats to the validity of the questions included in it.They may not have collected the desired information.Also, during the School year, the students had contact with computing subjects, so there may be other factors involved in their academic performance.Also, the Bebras Challenge was corrected and cataloged manually, as well as the data collected in the students and teachers surveys, and although with double data checking, correction errors attributed to human factors may occur.
In this study, it was not possible to control school life factors that may influence the results, such as administrative dependence of the school, elementary school quality, and extracurricular activities.Students may have appropriated the Bebras Challenge during their reapplications, which possibly helped in solving the questions.Also, like they answered the same questions, students may lose motivation throughout the Bebras reapplications.During the experiment, many students were transferred from one school to another.Thus, these students were excluded from this study.
The Bebras questions used in the study were translated from English, and although two external collaborators have reviewed the translated text, it may contain translation bias because it does not adequately represent the original questions intents.Some elements of the Bebras Challenge are not from the students' reality, which may have influenced the understanding of the problems.Besides, the Bebras test may not adequately measure CT skills.It is also important to emphasize that the students' academic performance may not represent their learning, especially in the subjects taught by different teachers in the experimental and control groups.
Finally, the sample analyzed is representative of the student in their 1st year of High School in a Brazilian public school, so we can not generalize the results achieved to other student populations.The robotics laboratory limitations may have been compromised the activities by the lack of materials and even by environmental conditions.

Non-Use of Robotics Materials
The school where we carried out this study received the first robotics tools in the year 2013 and, although it offered the Computer Science course in High School, as well as others international situations (Chevalier et al., 2020), these tools were not used in the school day-to-day.Initiatives to use the robotics tools in school until this study were motivated by exhibition events.
Several factors influenced this school not to use robotics tools.The study Souza et al. (2018a) describes that the application of ER can be influenced by physical structure, limitations of educational materials, maintenance of electronic equipment, absence or lack of continued training of professionals.These factors may have impacted the motivation and learning of students and teachers in the context of the Brink Robotic project of the school where this study was conducted.
In this sense, we extended the questions of the survey of teachers who explored contact with ER to other public schools teachers in Paraíba State, Brazil, through an online form.Thereby, we seek to confirm or identify new data that justify the non-use of ER tools by these schools.We applied the survey from October to December 2018 and collected data from 61 teachers, a value that, according to the 2018 Brazilian Census, represents 0.45% of teachers working in the Brazilian High Schools of Paraíba State in the same year.
Of the 61 teachers who answered the survey, 48.4% (29) know the school's ER tools, and 51.6% (32) did not.As for the use of ER tools, 33.9% (21) use ER in the classroom, and 66.1% (41) did not.The reasons that justify the ER tools non-use in Paraíba schools by teachers are: 56.1% (23) believe that the subject they teach can be worked with ER, but, they do not have knowledge about ER or about the educational materials available at the school; 19.51% (8) do not have time to plan or apply classes with ER; 7.32% (3) do not see ways to work with ER in the subject they teach; 4.88% (2) do not have ER tools in the school where they teach; 4.88% (2) have other reasons; 2.44% (1) highlighted that the training offered is exclusive to exact teachers; 2.44% (1) state that ER materials are retained by the school administration; and finally, 2.44% (1) highlighted that ER tools are wrecked.
As a result, other schools in Paraíba also show difficulties in making ER operational.The physical structure absence is the main factor that interferes with the use of ER in the classroom by teachers, followed by the lack, or even limitations in teacher training.Brazilian High School teachers are usually not provided with technological training, thus requiring complementary training.These limitations imply not using the ER and making use initiatives difficult.
Besides, issues such as restrictions or impediments of use by the school administration, indicate that the training limitation transcends the classroom space, and is reflected in school administrative decisions.Thus, we consider it necessary for the school com-munity to see technological resources as instruments to be used in daily life and not as equipment that must be preserved in shop windows.
In general, taking as a basis the reality of Brazilian High Schools of the Paraíba State, we learned that for the use of ER tools, it is necessary to consider physical, structural and methodological factors as a basis for the ER tools use that were distributed.

Relationships between CT and Subjects' Performance
We applied the Spearman r correlation test (Spearman, 2008) to analyze the correlation between students' performance on CT skills (Bebras Challenge) and students' academic performance.In the experimental group (see Table 11), we observed that all grouping steps have a positive correlation between the performance in CT and the academic performance.On the other hand, in the control group, we observed a predominance of negative correlations, demonstrating that the students' performance in CT and subject groupings commonly go in opposite directions.This result may evidence an excellent contribution of the ER to the students.
In the experimental group, according to the correlation level defined by Spearman (Spearman, 2008), we observed a very weak correlation in the Step 01 of the Humanities grouping.A weak correlation was observed in some cases such as Step 01 and Step 02 of Natural Sciences, Step 02 of Languages and Computing, and Step 01 of Mathematics.In other cases, the correlations were moderate.
Although the main objective of this study was to understand how the ER impacts the development of students in their 1st year of a Brazilian TVE High School, the correlation analyzes show that the relationship between the CT and students' academic perfor-

Computing
Step 01 (2nd Bimester) Step 02 (3rd Bimester) Step 03 (Annual average) -0.33 0.08 -0.27 0.67 0.5 0.68 mance who had contact with ER courses proposed in this study.Also, the results of the correlation collaborated with CT literature research Fidelis (2017); Blancas et al. (2020); Roussou and Rangoussi (2019), as well as favoring the investment of methodologies with ER for TVE High School teaching.

Long-term Studies in Brazilian TVE High School
The execution of a long-term study allowed us to observe changes in the students' performance throughout the ER Courses.However, we found that the sample size suffered wear over time; that is, there was a premature loss of respondents in the sample, a phenomenon called selective death.This phenomenon is capable of disturbing a long-term study, since there may be the disconnection of essential participants.The causes that commonly justify selective death are related to the participants' voluntary termination, who, for different reasons, choose not to continue collaborating with the study.
In this study, the causes of the occurrence of selective death are mainly due to the dynamics of High School.Naturally, in High Schools, it is common to transfer students during the school year, so all students in the control and experimental groups who left or enrolled in the school after the ER Courses beginning were removed from the sample.In addition to transfers, High School students can be absent from classes daily, in some cases, staying days away from school activities.Some of these absences occurred during the application of the survey and the Bebras Challenge, moments when data collection was carried out.Thus, students who participated in the intervention classes, but who did not take at least Bebras Challenge, were also removed from the sample.
Thus, in our long-term study, we highlighted situations related to selective death that corroborate with results presented in the literature Bonamino and Oliveira (2013); Rodrigues et al. (2016).Also, knowing the students' skills in the experimental group, who were removed from our sample, we believe that the students' performance in this group could be more expressive if the data of these students were present in the analyzed sample.

Student Relation with Robotics
The ER courses required students to produce authorial robotic assemblies with minimal guidance.Thus, we seek to stimulate problem-solving skills through the combination of robotics materials.The end of the classes was linked to the success of the robot assembly; we observed a relationship between success in the assembly phase and students' motivation.The students that did not have assembly success showed a demotivation predisposition, while those that were successful showed motivation to learn more.
With this, we seek to ensure that all student teams were able to complete the proposed assemblies for each activity.Thus, we conditioned the classes to end with the ro-bot assemblies to be finished, and the students of the teams that were reaching the objectives started to help those who showed difficulties.This decision favored the sharing of knowledge between the groups and minimized problems from the non-completion of the robot assemblies, which provided a change in the perspective of students who initially always showed difficulties.
From this experience, we learned that the classroom ER use could motivate High School students.However, it is also capable of causing apathy, making it the learning effectiveness difficult.We also learned that motivated students could assist in the collective teaching and learning.In this sense, the main challenge is to introduce significance to the ER methodological objectives.

Bebras Challenge
We used the Bebras Challenge to measure CT skills.The bebras word comes from the Lithuanian language, which means "beaver" and was used as a community name because it represents an intelligent animal that works hard to achieve its goals.For this reason, Bebras issues involve situations typical of a beavers', but beavers are not common in Brazil.
It is important to note that we translated the questions into Portuguese, as the Bebras Challenge, until 2018, was not officially applied in Brazil.Therefore, even though external reviewers have revised it, there may be distortions attributed to the translation that makes it challenging to represent the original goals.
The Bebras is designed following an international standard based on contextualization.In this sense, the questions demand from the student problem-solving skills associated with reading and interpreting text skills.During the Bebras application, we observed that the students had difficulties in solving, sometimes the problem came from reading and interpreting the text.However, in the vast majority of times, this problem was due to the lack of reading, as we noticed a tendency for the student to go directly to the question without reading the question context.
During the Bebras application, we observed too that the students were unaware of the existence of the beaver animal, which made it difficult to associate the character design with the beaver name contained in the questions.Considering that beaver is a type of semi-aquatic rodent native to North America and Europe, it is natural for Brazilian students not to know this.This fact may compromised the Bebras application.Several times, students promoted discussions about the beaver and requested clarification.At each student's intervention, we observed that those who were concentrated in the evaluation had their concentration impaired.We believe that the character change by another native of Brazil would benefit the accomplishment of the Bebras Challenge and, consequently, the student's performance.
During the reapplications, we observed cases of students without interest, but also of students having difficulty in answering the questions.We also noticed expressions of sadness for not being able to assimilate, anger for going through it without need, and joy when they were sure of the answer to a question.
With the use of the Bebras Challenge, we understand that it is necessary to propose specific instruments to assess the CT skills of Brazilian High School students.The adversities faced by students may have negatively influenced the students' performance.We also understand that Brazilian High School demands educational practices that encourage students to read and understand the text, many of the difficulties we identified are not related to the problem-solving skill, but rather to reading and understanding texts.

Concluding Remarks
The objective of this study was to stimulate the CT skills through ER Courses in the 1st year of a Brazilian TVE High School, however, to identify their relationship with learning the subjects.Thus, we performed a long-term study based on a research-intervention.
The results obtained show that the experimental students group, showed better performance in the Bebras Challenge applied after each ER course, indicating that the ER teaching can impact the CT skills.However, the results state that when ER is worked with the computing concepts support, as implemented in Computer-Based Robotics step, the impact on student CT skills is more significant.Besides, the ER Courses can influence the students' academic performance in High School, because the experimental group showed a high performance when compared to those in the control student group, factors that were confirmed with the teachers' observations.However, the Mathematics results comply with the assumptions of all the statistical resources used, demonstrating that the ER introduction in teaching in the 1st year of TVE High School significantly favors the mathematics learning.
Our study showed enhanced performance in the students when the ER is used in Brazilian TVE High School.This enhancement can be related to subject learning or CT skills improvement.In this sense, we believe that ER can offer considerable benefits to learning different sciences, and investing in the teachers' formation in ER could promote the dissemination of this technology in Brazilian schools.It is important to underscore that the ER use in schools is not simply to teach technology or computing concepts but also teach concepts from science.
The main contribution of this work was the understanding the impacts ER causes in students through a long-term study.Literature studies currently consider specific observations that may not be valid for the Brazilian context.Also, as a future work, we highlight the need to identify instruments that facilitate the application and evaluation of CT skills.Until then, we will investigate an ER methodology focused on CT skills and not only in Computer Science teaching.Besides, we will study a way to evaluate High School students regarding the CT skills considering the specificities of education with ER.

A. Appendix A: Detailed Data
Table 12 presents the obtained p-value for normality and homoscedasticity tests regarding the subjects, whereas Table 13 shows test results for Bebras Challenge.Tables 14  and 15 presents the performance of student in subjects groups (see RQ2).

Table 2
Characterization of student groups.

Table 6
Teachers' profile regarding contact with Educational Robotics • The scope of the Computer-Based Robotics step is described below:

Table 11 Spearman
Correlation coefficients between Bebras and academic performance Souza is a Ph.D. student in Computer Science from Federal University of Campina Grande (UFCG) Brazil, working in the research with emphasis on developing strategies to stimulate Computational Thinking through Educational Robotics.Isabelle was a substitute teacher at the Federal Institute of Paraíba (2019-2021).Isabelle received the Master's Degree in Computer Science from UFCG (2019), a certificate in Didactic Multimedia Design for Distance Learning from the Faculty of Administration, Sciences and Letters (2012), and a Degree in Computer Science at Paraíba State University (UEPB) (2012).Andrade received the Bachelor's, Master's, and Ph.D. Degree in Computer Science from the Federal University of Campina Grande (UFCG), Brazil, in 2005, 2007, and 2011, respectively.He worked at both Paraíba State University (UEPB) and Federal University of Paraíba (UFPB).Since 2010, he is a professor in the Systems and Computing Department at UFCG.He has experience in Computer Science, with an emphasis on Computers in Education and Software Engineering.Specifically, his research interests concern Computational Thinking, Educational Robotics, and the formal verification and validation of software systems, including formal methods, model-based testing, and software testing automation.Sampaio received the Bachelor's and Master's Degree in Computer Science and PhD Degree in Electrical Engineering (with emphasis in Distributed Systems) from the Federal University of Campina Grande (UFCG), Brazil, in 1998, 2000, and 2007, respectively.She worked at both Federal University of Paraíba (UFPB) and Paraíba State University (UEPB).Since 2009, she has been a professor in the Systems and Computing Department at UFCG.She has experience in Computer Science, with emphasis on Computers in Education and Distributed Systems.Specifically, his research interests concern Computational Thinking, Educational Robotics, Programming Teaching and Distributed Computing Applied to Data Science and Energy Efficiency. L.M.R.

Table 12
Statistical assumptions: p-value of the Normality and Homoscedasticity for subjects groups Statistical assumptions: p-value of the Normality and Homoscedasticity for Bebras Challenge tests Average, standard deviation, and the average difference in the performance of student in subjects groups

Table 15
Hypothesis testing and effect size of the students in the subject groupings