Nowadays, few professionals understand the techniques and testing criteria to systematize the software testing activity in the software industry. Towards shedding some light on such problems and promoting software testing, professors in the area have established Massive Open Online Courses as educational initiatives. However, the main limitation is the professor’s lack of supervision of students. A conversation agent called TOB-STT has been defined in trying to avoid the problem. A previous study introduced TOB-STT; however, it did not analyze its efficacy. This article addresses a controlled experiment that analyzed its efficacy and revealed it was not expressive in its current version. Therefore, we conducted an in-depth analysis to find what caused this result and provided a detailed discussion. The findings contribute to the TOB-STT since the experimental results show that improvements need to be made in the conversational agent before we use it in Massive Open Online Courses.