measuring the quality of philosophical dialogue: a high-inference rating instrument for research and teacher education

Philosophical dialogue with children and adolescents has found its way into schools all over the world. The diverse teaching methods and approaches that have become established due to different authors and educational contexts are often subsumed under the term Philosophy with Children (PwC). The roots of this form of classroom discourse lie in the work of Matthew Lipman, who started the Philosophy for Children (P4C) program in the 1970s (^{Lipman, 1988}, 2009). Since then, the effects of philosophical praxis in the classroom have been examined in numerous empirical studies. These studies show that the regular implementation of PwC has a positive effect on the cognitive development of students (^{García-Moriyón et al., 2005}; ^{Yan et al., 2018}). Various studies further suggest that this method contributes to significant improvements in reasoning and argumentation skills (^{Collins, 2007}; ^{Säre et al., 2016}; ^{Walker et al., 2013}), language ability (^{Alt, 2018}; ^{Schleifer & Courtemanche, 1996}), social skills (^{Daniel, 2012}, 2021) and critical thinking skills (^{Colom et al., 2014}; ^{Siddiqui et al., 2019}, 2022; ^{Worley & Worley, 2019}). Previous research, however, has not included consideration of whether and to what extent the quality of collaborative philosophical inquiry had an impact on participants’ progress in the aforementioned areas. To facilitate such consideration, this article presents a high-inference rating instrument that measures the quality of philosophical dialogue. The instrument was developed as part of an intervention study and tested in 21 secondary-school classes in Switzerland. The paper first addresses the characteristics of philosophical dialogue and then describes the development and indicators of the high-inference rating scale using excerpts from classroom discussion. It then presents the results of the validation of the instrument carried out as part of the study. Finally, it addresses the limitations of the instrument, possible further developments and practical applications in research and teacher training.

characteristics of philosophical dialogue

This article bases its understanding of philosophical dialogue primarily on Matthew Lipman’s ideas (^{Lipman, 1988}, 2003) but also on the publications of important German-speaking authors such as Ekkehard ^{Martens (2003}), Barbara ^{Brüning (2003}) and Kerstin ^{Michalik (2013}), thus connecting to the tradition of the Socratic Method. Children and young people explore philosophical questions collaboratively and expand their insights in a co-constructive process using specific philosophical methods. Philosophical dialogue thus differs from everyday conversations and discussions because certain philosophical methods are used to promote philosophical reflection. Reviewing the philosophical practice of Socrates and Aristotle, Ekkehard Martens defined five core philosophical methods that support successful philosophising: Phenomenology, Hermeneutics, Analysis, Dialectics and Speculation. Children participating in philosophical inquiry perceive and describe facts and phenomena in a differentiated way (phenomenological method). They try to understand the statements of their peers and to grasp their perspectives (hermeneutic method). Concepts and terms are scrutinised, and arguments are examined and questioned (analytical method). This happens in a collaborative, co-constructive exchange (dialectical method). Furthermore, participants in philosophical inquiry formulate hypotheses, consider possible consequences and discuss creative ideas (speculative method) (^{Marsal, 2009}, 2014; ^{Martens, 2003}; ^{Meerwaldt et al., 2018}).

The teacher’s task is to promote critical and independent thinking in students using specific questions and statements. In the Socratic tradition, this can be called the maieutic method since the teacher helps ‘give birth’ to ideas and thoughts. The aim of such questions is to deepen the students’ philosophical reflection (^{Fisher, 2007}; ^{Haynes, 2008}; ^{Michalik, 2016}). With increasing philosophical practice, children become able to use such questions and prompts for themselves (^{Kennedy, 2013}). Table 1 presents different types of questions that can support philosophical reflection (^{Helbling, 2018}).

In addition to the depth of philosophical reflection, the type of discourse is crucial for the quality of philosophical dialogue. In a community of inquiry (^{Sharp, 2009}), students should collaboratively and co-constructively share thoughts and ideas so that the content of the dialogue evolves. Participants refer to each other and repeat statements, rephrase them, add to them, question them, contradict them, or ask questions if they are not understood (^{Daniel, 2008}; ^{Fisher, 2007}; ^{Kennedy, 2013}; ^{Michalik, 2016}). Progress becomes visible since views are not simply expressed additively (Daniel et al., 2005; ^{Jackson, 2013}). The teacher can encourage co-construction through moderation questions, as exemplified in Table 2 (^{Fisher, 2007}).

Another feature of quality is that a previously defined question becomes focused and clarified. It is easy, instead, for students to digress and get lost in detail. To counteract this, the teacher can regularly summarise the status of the dialogue with regard to the question and explicitly revisit the underlying philosophical question (^{Brüning, 2003}; ^{Jackson, 2013}). In order to be able to focus on the question, the participants must also comply with the rules of discourse so that no disturbances or interruptions occur (^{Haynes, 2008}; ^{Jackson, 2013}).

Therefore, facilitating a philosophical dialogue requires great skill and sensitivity. This becomes clear in Haynes’ description of the teacher’s role:

The teacher’s role is to prompt children into giving reasons for their ideas, making distinctions and connections, constructing arguments and developing hypotheses and analogies. A balance has to be struck between a sense of adventure and exploration and a sense of direction and progress. The teacher helps the group to clarify and build on ideas whilst allowing the discussion to follow its own course. (^{Haynes, 2008}, pp. 30-31)

Despite being expected to promote philosophical reflection and co-construction, teachers must also moderate with restraint (^{Krüger & Schick, 2012}; ^{Sharp, 2018}). As a result, they should not express any opinions or attitudes of their own and should try to limit their share of the speaking: “In a community of enquiry teachers are facilitators, holding back on their own views, tactfully supporting children’s thought and talk and encouraging the dialogue to flow between participants” (^{Haynes, 2008}, p. 35). In addition, teachers should not steer the dialogue in a direction that they have set but should follow the students’ train of thought (^{Jackson, 2013}). The teacher’s tasks in a philosophical dialogue can be summarised as follows (^{Brüning, 2003}; ^{de Boer, 2015}; ^{Kennedy, 2004}):

Philosophical dialogue can therefore be regarded as successful if a deeper philosophical reflection takes place through the application of philosophical methods and if the students gain new insights in a co-constructive and collaborative way while maintaining the focus. Teachers support these processes and act as facilitators, holding back their own views and limiting their share of the speaking.

the development of a high-inference rating instrument

As already mentioned, until now there has been no instrument for measuring the overall quality of philosophical dialogue. Studies dealing with quality have exclusively addressed individual factors. For example, Cassidy and Christie (2013) used categories to examine what types of contributions participants made to philosophical inquiry. This made it possible to determine how often children used (philosophical) talk moves such as formulating examples or hypotheses, defining or differentiating words or phrases or introducing new ideas. A Canadian research team led by Marie-France Daniel described the development of complexity and critical thinking in dialogues (^{Daniel, 2008}, 2021). Helzel and Michalik (2015) addressed, among other things, the multiple perspectives and references of students. ^{Santi (1993}) analyzed and described philosophical dialogues based on the criteria of spontaneous philosophical content, argumentative structure, epistemic categories, and the roles of teachers and peers. However, assessing the impact of philosophical dialogue requires a more comprehensive measurement tool. Therefore, we developed a high-inference rating instrument based on the rating system of Hugener et al. (2006) as well as ^{Kunter (2005}) to capture the overall quality of a philosophical dialogue. The first step consisted of a literature analysis, from which central quality characteristics emerged via a deductive procedure. These were then condensed into dimensions and operationalised with indicators. The preliminary instrument was adapted and supplemented in dialogue and based on an example class discussion, in accordance with an inductive procedure.

For this study, we used a high-inference procedure, which requires a high degree of inference and interpretation. Such procedures are used to assess the quality of instructional sequences - or, in our case, conversational sequences (^{Clausen et al., 2003}; ^{Seidel, 2005}). In contrast, low-inference methods yield statements about easily observable behaviour.

Due to the need for intensive interpretation, the conversational features to be assessed must be presented precisely. Therefore, the content of each dimension is described in the rating instrument. This basic idea is then further elaborated using detailed indicators that are as close to the behaviour level as possible. The description and the indicators reflect a typical ideal philosophical inquiry. The rater (or, in our case, the two raters) must then rate the extent to which the dialogue corresponds to this specification on a four-point scale, assessing how often a certain behaviour occurs and how pronounced it is. Usually, the rater also examines the extent to which different students are involved in the class discussion. In our case, this third feature could not be measured or rated since only audio recordings were involved. The aim is for the judgments to reflect the overall impression in every dimension (^{Hugener et al., 2006}).

Each category was supplemented by various remarks to clarify any ambiguities that arose. A complete philosophical dialogue forms the unit of analysis. The use of a highly inferential method requires intensive study of the instrument and many joint discussions for assessment (^{Rakoczy & Pauli, 2006}).

Based on the literature, we defined four dimensions of the quality of philosophical dialogue: Philosophical Richness, Co-construction, Focus and Restrained Facilitation. Table 3 provides an overview of the dimensions and indicators. The complete instrument with the corresponding literature can be found in the Appendix (Supplementary File 1: High-Inference Rating Instrument).

In the next section, we will take a closer look at the dimensions using examples from philosophical dialogues.

presentation of the instrument using excerpts from philosophical dialogue

In order to make the instrument more comprehensible, two academic excerpts from philosophical dialogues taken from the Swiss study mentioned above are presented in this section and linked to the dimensions.

The intervention study aimed to examine the influence of philosophical dialogue about the nature of science on the acceptance and understanding of evolution. In the teaching unit, philosophical dialogue was thus linked to the content of the subject of biology. The teaching approach of using philosophical reasoning in school subjects goes back to Kerstin ^{Michalik (2009}, 2013) and holds great potential for the students’ (interdisciplinary) learning. The focus of these dialogues was not on the development of questions by the students but rather on the conversation itself and its progression. Accordingly, the instrument is exclusively centred on the discussion phase.

The teaching unit on evolution was carried out in 21 secondary-school classes, with students mainly aged between 13 and 15. The classes were divided on three occasions, during which half took part in a philosophical dialogue while the other half delved further into the subject for about 25 minutes. In the philosophical dialogues, the students dealt with the following three questions: Does the theory of evolution make belief in God and religion superfluous? Can we rely on science? What distinguishes humans and animals? The first two dialogues thus focused on epistemological questions and in particular on the relationship between religion and science. These topics are part of the concept of the nature of science (^{Hofheinz, 2010}; ^{Lederman et al., 2002}; ^{McComas & Olson, 1998}) and have proven to be important factors influencing the acceptance and understanding of evolution. The third dialogue focused on anthropological and ethical questions, and its discussion required recourse to the first two dialogues. The students in the classes had generally known each other for one or two years, but this was their first time participating in a philosophical dialogue in this group or at all. Research findings indicate that experience in philosophical praxis and the duration of the group's existence have a significant impact on the quality of the dialogues (^{Daniel, 2008}).

All class discussions were assessed using the instrument described. This allowed the pursuit of the question whether the quality of the philosophical dialogue influences the acceptance or understanding of evolution. Further information on and results of the study can be found in ^{Bernhard (2022}).

In the following simplified excerpt, students consider the question of whether one can believe in God and hold the theory of evolution to be true at the same time. Due to the audio recording, the students speaking cannot be identified. The statements are thus simply numbered and assigned either to the teacher (identified as ‘T’ for ‘teacher’) or to the students.

S1: So, I think yes, that is possible. Believing in God does not only mean that he created the world and everything. It could also just help if you have a problem or something. You can still count on God. S2: So, I think you can. So, you can do everything, but you must try to separate it. You can’t just, so you must separate it. So, these are two different issues, because I think that belief is believing it is so. And the other is simply a fact, so it’s about facts. Facts that can be proven, so to speak. With God, it’s just believing. S3: I don’t know if that’s true, either. Those giraffes used to look like this.¹ There’s no proof of it, it’s just a guess. S4: You can prove that. S5: If you find dead animals, you can use the genes to look up what kind of animal it was; then you have proof that it had a short neck. S6: But we don’t know exactly how it came about. S7: You can do experiments. It’s definitely different now. It’s always changing. You can see this in all animals. S8: They also only learned about the dinosaurs from the bones they found. S9: Well, there are enough facts, I think, or evidence. S10: Yes, I think it is possible that one can be a believer and a scientist. Because if you find out something in science, for example, and believe in God and read it somewhere in the Bible and it is confirmed again, then you are actually even safer. T1: So, they can confirm each other. S11: Yes, exactly. T2: That would be great. Are there also subject areas where this is not possible?

philosophical richness

Here, students name two important reasons that one can be a scientist and religious at the same time. First, belief in God involves more than just the question of whether God created the world; it can, for example, fulfil the need for help or protection (S1). It is also emphasised that science and faith are two things that must be clearly distinguished. The important distinction between knowledge and belief is mentioned (S2). A brief discussion develops from this point, clarifying in part how science can find evidence. During the discussion, the students seek to support their views with concrete examples and thus provide justifications (S5/S7/S8). However, there is no conclusive clarification or statement of the extent to which faith and science are two different domains or of what distinguishes knowledge from faith. Here, the teacher fails to follow up specifically or summarise. She does not elaborate, merely asking about the last point mentioned (T1).

co-construction

In the excerpt, it becomes clear that the students’ statements relate to each other and are not uttered additively. The second statement is even questioned by a student so that a longer exchange develops (S3). Various participants complement each other in explaining why the idea of the development of the giraffe is based on evidence (S5/S7/S8/S9). The teacher also refers in her statements to statements made by the students. She paraphrases and sharpens the students’ statements (T1) and adapts her question to them (T2).

philosophical richness

In response to the teacher’s question (T1), a student formulates various similarities between humans and animals (S1). However, the topic of similarities is abandoned rather abruptly and not further differentiated. Various, substantial factors are proposed on the question of differences: intelligence (S4), building things (S5), using tools (T4/S7), planning one’s own future (S11), being aware of one’s own death or life (S11/S14). In this dimension, it is irrelevant whether the teacher or the students introduce or elaborate on these points. Although these key areas are mentioned, they are not further described or clarified. It remains unclear how the use of tools differs between animals and humans. It remains just as open to what extent human and animal intelligence, thinking, consciousness and planning can be differentiated. The areas are therefore only treated superficially. At various points, views are supported with examples (S5/S7/S8). Elsewhere, students find it difficult to further justify suggestions (S12/S13).

evidence of the high-inference rating instrument’s validity

According to the American Educational Research Association et al. (2014), validity describes “the degree to which evidence and theory support the interpretations of test scores for proposed uses of tests” (p. 11). There are various sources of evidence: evidence based on test content, evidence based on response processes, evidence based on internal structure, evidence based on relations to other variables and evidence for the validity and consequences of testing (American Educational Research Association et al., 2014). Since the development of the instrument and its validation were not the focus of the Swiss research project, only individual sources of evidence are addressed in this chapter.

The results of the rating of the 63 philosophical dialogues in the study in Figure 1 show that the instrument is able to record different levels of quality.

Figure 1 Histogram of the quality of philosophical dialogues, N = 63. 

The crucial question now is whether these differences can be validly measured. In the following sections, we therefore briefly discuss evidence based on test content, evidence based on internal structure and evidence based on relations to other variables. In addition, we address interrater reliability in the assessment of the dialogues as an aspect of objectivity.

evidence based on relations to other variables

If theoretically based connections with other constructs can also be determined empirically, this can provide a source of evidence for validity. In our research project, based on numerous theoretical and empirical works, we assumed that the quality of philosophical dialogue has a positive influence on acceptance of evolution. Figure 2 presents a latent change score model (^{Kievit et al., 2018}; ^{McArdle & Grimm, 2010}) of the impact of the quality of philosophical dialogue on the change in acceptance of evolution. A higher quality of dialogue leads to a greater increase in acceptance of evolution (β = .214, p = .009). The model explains 19.9% of the variance in the change in acceptance of evolution. The theoretically assumed connection was thus empirically confirmed, providing evidence of the instrument’s validity. The acceptance of evolution was measured using a questionnaire before and after the intervention (^{Bernhard, 2022})

Figure 2 Latent change score model for the philosophy group² (N = 180) with the dependent variable acceptance of evolution and the independent variable quality of philosophical dialogue. Solid lines indicate significant relationships, dashed lines non-significant relationships. AE1 = pre-test acceptance, ae1/ae2/ae3 = pre-test acceptance items, AE2 = post-test acceptance, pae1/pae2/pae3 = post-test acceptance items, ΔAE = change in acceptance, Rating: Quality = quality of the philosophical dialogue. 

interrater reliability

Two evaluators, the author and a master student, rated the philosophical dialogues from the study. The master student familiarised herself with the introductory materials for the teachers of the first dialogue as well as the rating instrument and discussed these with the author. Subsequently, various discussions on the first philosophical question were jointly evaluated, allowing the resulting judgments to be discussed and clarified. The intraclass correlation (ICC) was calculated to determine interrater reliability. Since 27 of the 63 philosophical dialogues were assessed independently by both raters, values were evaluated using a two-factor analysis of variance. The rating aimed to achieve the greatest possible agreement, so an unadjusted ICC was calculated (^{Koo & Li, 2016}; ^{Wirtz & Caspar, 2002}). The results may be seen in Table 5.

There is no precise definition of when the ICC is considered sufficient. According to ^{Koo and Li (2016}), values above 0.5 can be interpreted as moderate reliability, values above 0.75 as good reliability and values above 0.90 as excellent reliability. It can be stated that overall agreement on assessments for this instrument is very high. However, reliability is lower for the focus dimension.

conclusion

In this section, we critically discuss the potential of the developed high-inference rating instrument. We consider the limitations of our study, possible further developments and the opportunities for application in research and teacher training.

The instrument presented offers an innovative approach to holistically assessing philosophical dialogues and reflects many key features of quality. The analyses of the 63 class discussions show that the instrument enables reliable measurement. In addition, the use of the instrument is practical and comparatively simple, even if training is necessary.

Because the instrument was developed during the study, no cross-validation could take place. Consequently, it would be important to have dialogues evaluated both by different groups and by additional raters in order to obtain reliable results. Another limitation of both the instrument and the study is the omission of the distribution of student statements. Since the study analysed audio recordings, it was not possible to track which students, or how many, participated. However, with video recordings and an additional indicator in the dimension of co-construction, this factor could be taken into account in the future.

Furthermore, a sharpening and (further) differentiation of the individual dimensions, especially philosophical richness, is necessary. As can be seen in the exploratory factor analysis, this dimension is a kind of ‘super factor’ that seems to contain a lot of information from the other dimensions. Differentiation could be supported, for example, by formulating the levels of the individual indicators.

Another challenge is the distinction between statements made by the teacher and those made by the students. With the present instrument, this distinction is only partially taken into account in the dimensions of co-construction and restrained facilitation. Nevertheless, it is only partially possible to make statements about the contribution of the teacher to the quality of the dialogue. To examine this area more closely, low-inference methods could be helpful. ^{Lotz et al. (2013}) advocate a combination of the two procedures, which would allow for mutual validation and provide a more comprehensive picture of a dialogue. Thus, a possible further development would be the addition of a low-inference instrument in which, for example, the types of talk moves (introducing new content perspectives, formulating hypotheses, etc.) by teachers and students are recorded. This type of classroom discourse analysis is already widespread, as can be seen in the contributions of Cassidy and Christie (2013) or ^{Moser et al. (2022}).

The potential of a high-inference rating instrument such as the one introduced here is evident both in teacher training and in research. When pre-service teachers engage in the method of PwC, they need criteria for what constitutes a dialogue of good philosophical quality. The instrument can be used to analyse and reflect on conversations and can serve as a basis for discussion. In this way, the pre-service teachers’ ideas about philosophical dialogue can be sharpened. This is especially important because learning the competencies required to apply PwC has proven challenging (^{Akkocaoğlu Çayır, 2019}; ^{Mathis & Conrad, 2015}).

An effective means of measuring the quality of philosophical dialogue is a key contribution to the research. As with all teaching methods, the quality of implementation is decisive in determining success. The instrument offered here can be used to examine the effects of PwC in a more differentiated way. In addition, it opens up an opportunity to check the effectiveness of teacher training in the field of philosophising.