arXiv

Classroom chatbots can pass the tone test and fail the lesson

September 25, 2026 4:52 AM

Teacher-configured chatbots can respond promptly and sound right without carrying out the lesson they were designed to teach. A study of 27 middle-school teachers found that purpose alignment was the weakest of four measures, showing why classroom AI needs ways to test and repair its teaching behavior before students can rely on it.

A teacher can ask a general-purpose chatbot to help students, but cannot reliably tell whether it will preserve productive struggle, stay within a lesson’s scope, or give answers instead of guidance. A student may need a hint or a question, yet receive a finished answer. The teacher is still responsible for checking what the bot says and what students learn.

A chatbot can sound right and still teach wrongly

Large language models do not follow a lesson plan in the same fixed way a worksheet does. They generate a likely next response from the instructions and conversation they receive. An instruction can influence the response without guaranteeing it.

That makes “helpful” a slippery word in a classroom. A friendly, responsive reply may still defeat the lesson if the teacher wanted students to work through a problem themselves. A bot that stays on topic may still give away the answer. A teacher-configured agent is meant to turn a one-time prompt into a reusable classroom role, such as a subject tutor that works within a boundary.

The study also separates the bot’s teaching job from the practical conditions around it. Schools must consider student privacy, age-appropriate content, devices, networks, and local approval before using an AI tool. The question here is narrower but important: can a teacher tell whether the bot is doing the intended teaching?

The test was not just about a pleasant tone

Researchers studied 27 middle-school teachers in professional-development workshops in summer 2026. The teachers made and tested chatbots for science or computational-thinking activities.

The authoring tool gave them separate places to describe the bot’s Purpose, Rules, and behavioral traits, or Persona. They could also choose a language model and optionally add course files for the bot to search. Purpose was mainly used for the learning goal and subject focus. Rules were more often used for teaching behavior, limits, and adjustments for different learners.

The researchers then judged the finished bots on four separate questions: Did the bot respond? Did it serve the stated purpose? Did it follow the rules? Did it match the requested persona?

That separation matters. A bot can pass the tone test while failing the lesson. In the study, responsiveness passed for 88.9% of the chatbots. Persona alignment passed for 81.5%. But purpose alignment passed for only 59.3%, the lowest result. Rule adherence also fell short of the other measures.

Teachers said that changing the model could change whether the bot obeyed an instruction not to give a direct answer. They also asked for clearer labels, examples, templates, and guidance for the configuration controls. Writing an intention into a box was not always the same as making that intention happen.

Figure 5. Mapping Norman’s Gulfs of Execution and Evaluation to teacher-facing AI authoring. Adapted from Figure 2.1 in Norman ([Norman Donald, 2013](https://arxiv.org/html/#bib.bib56)). In our adaptation, the pedagogical Gulf of Execution captures the translation of a teacher’s pedagogical goal into chatbot configuration, while the pedagogical Gulf of Evaluation captures the interpretation of generated chatbot behavior relative to that original goal.Conceptual diagram mapping Norman's Gulf of Execution and Gulf of Evaluation to teacher-facing AI authoring. A teacher pedagogical goal is shown at the top and a teacher-facing AI authoring system at the bottom. The left side represents the pedagogical Gulf of Execution, showing how teachers translate pedagogical goals into configuration choices such as Purpose, Rules, and Persona. The right side represents the pedagogical Gulf of Evaluation, showing how teachers interpret generated chatbot behavior and assess whether it reflects their original pedagogical goals.In our study, the pedagogical Gulf of Execution captures the distance between a teacher’s intended pedagogical behavior and the configuration actions available for expressing it. Teachers had to translate instructional intentions into fields such as Purpose, Rules, and Persona, a process that was not always straightforward. The pedagogical Gulf of Evaluation captures the distance between generated behavior and the teacher’s ability to judge whether that behavior reflects the original pedagogical intention. This distinction is particularly important for generative AI, where a response may appear conversationally appropriate without fully realizing the intended instructional purpose. Together, these two gulfs show why configurable controls alone are insufficient: teacher-facing AI authoring must support both the expression of pedagogical intentions and the evaluation of whether those intentions are realized in system behavior. Source: Will It Teach as Intended? How Teachers Configure Educational AI Chatbots（arXiv）

The missing feature is proof

The study’s central finding is that configuration alone does not make a chatbot a teaching tool. Teachers need help expressing what they mean, but they also need help judging whether the bot’s replies actually match it.

That is the gap between setting a goal and checking the result. A teacher may write “guide students with questions rather than answers.” The bot may sound patient and encouraging, yet still provide the answer in a difficult case. Without a way to spot that failure, the teacher must keep policing conversations by hand.

If authoring systems can show teachers where replies depart from a learning goal, and help them revise or enforce that goal, classroom AI could become a controllable instructional scaffold rather than a generic answer machine. Teachers would have evidence about instructional purpose and constraints, not just answer quality or tone.

In a middle-school science classroom, a student stuck on the water cycle could open the class bot and receive teacher-approved questions and hints instead of a paragraph to paste into an assignment. A teacher might later see that several students needed the same hint and revisit that misconception in the next lesson, without reading every transcript. That future depends on dashboards that surface useful patterns while respecting privacy, and on the bot keeping its boundary across real conversations.

The study did not establish that yet. It mostly examined teachers testing bots during short workshops, not students using them over time in classrooms. The next step is sustained testing with real students, together with tools that diagnose and repair failures of teaching purpose.

出典

arXiv, “Will It Teach as Intended? How Teachers Configure Educational AI Chatbots”