(IJCRSEE) International Journal of Cognitive Research in Science, Engineering and Education
Vol. 6, No.1, 2018.



Dr. Adem Karahoca, Bahcesehir University Engineering Faculty, Turkey
E-mail: adem.karahoca@eng.bau.edu.tr
Dr. Ilker Yengin, Institute of High Performance Computing, A*STAR, Turkey
E-mail: yengini@ihpc.a-star.edu.sg
Dr. Dilek Karahoca, Bahcesehir University Health Science Faculty, Turkey
E-mail: dilek.karahoca@hes.bau.edu.tr


Original Research
Received: January, 27.2018.
Revised: March, 06.2018.
Accepted: March, 14.2018.


Abstract. This study introduces a system design in a form of cognitive dialog game (DiaCog) to support pedagogical factors and student learning model ideas. The purpose of the study is to describe how such a design achieves tracking and adapting students’ knowledge and mastery learning levels as a cognitive assistant. Also, this study shows alternative ways for supporting intelligent personal learning, tutoring systems, and MOOCS. This paper explains method called DiaCog that uses structure for students` thinking in an online dialog by tracking student`s level of learning/knowledge status. The methodology of computing is the semantic that match between students` interactions in a dialog. By this way it informs DiaCog’s learner model to inform the pedagogical model. Semantic fingerprint matching method of DiaCog allows making comparisons with expert knowledge to detect students` mastery levels in learning. The paper concludes with the DiaCog tool and methodologies that used for intelligent cognitive assistant design to implement pedagogical and learner model to track and adapt students’ learning. Finally, this paper discusses future improvements and planned experimental set up to advance the techniques introduced in DiaCog design.
Keywords: Student modeling, Cognitive Modeling, Cognitive Assistants, Cognitive dialog game, MOOCS.


Intelligent cognitive assistants and tools can be implemented in e-learning platforms and massive open online source courses systems (MOOCS) to reduce the educational costs and improving life quality of people and to address the educational needs of people who are unable to leave their home due to their care needs. Although such users can easily access to current MOOCS platforms, they may not fully benefit from them since there are some technical challenges to overcome. For example, MOOCS platforms are usually criticized about failing to provide a social environment that enables having constructive cognitive feedback opportunities in a sustained engagement during the online courses (Nkuyubwatsi, 2013; Kop et al., 2011; Zapata, 2010).
Discussion forums are used where students have most of the social interaction to help each other and to discuss ideas related to the learning topics in MOOCS (Ezen-Can et al., 2015). Discussing ideas on learning topic have positive impacts on students learning and building knowledge in online platforms (Palmer et al., 2008).
On the other hand, we observe that discussion forums bring several issues in learning despite their positive benefits on learning. For example, a poor designed and implemented discussion forum activity may not support to discuss at all. The engagement of students may be very superficial (Bain, 2011; Hawkey, 2003; Thomas, 2002). Thus, discussion on online platforms may provide support for learning but it is not guaranteed that all the participants of the dialog may join to the ongoing discussion or the discussion is deep and meaningful. There is still have more to explore in students’ dialog in technology supported applications. Especially, we need to know which students actively and meaningfully participate to the dialogs and which of them learn from the dialog as the knowledge building activities emerges in ongoing dialogs.
Using new cognitive assistants can be supported with contemporary dialog technologies. By this way, it may be possible to improve students’ positive learning results by tracking meaningful dialog participation and interaction. To enable these functions, it is possible to design modern applications that track and adapt knowledge level of students. It may allow us to closely understand students’ dialog-based learning interactions.
To address the need of designing new applications as described here, this study discusses the design of a modern learning tool that is called as DiaCog. The remaining of the paper presents how DiaCog design enables alternative dialogic interactions and modeling learning processes in a meaningful way.

2.1. Designing Interaction and Interface

Table 1 shows a part of a hypothetical dialog. This dialogue is an example of interaction between players in a typical dialogue game. Dialog may go in a way that the players may reply to previous entries so that there is no linear order in the dialog. Therefore, the ongoing dialog cannot be read as reading a page of a book in a top to down direction because the events are non-linear. The rounds are indicated by round numbers in the first bracket and the dialog moves are coded in the second bracket at the end of players’ entries.

Table 1. An Example Dialog



DiaCog has essential functions such as a)interface interactions tied to the meaning that the users wish to communicate, b)dialog roles and specific rooms, c)structured moves categorized as interface elements, and d)dialog rules in order to carry the user’s goals (e.g. turn taking). These functions create the basic environment for successful collaborative argumentations that teachers may apply dialog-based learning scenarios to support students in practicing their argumentation skills and creative thinking skills.
Figure 1 illustrates the typical design of DiaCog’s dialog game window. In Figure 1, participants are required to select moves from the list of available speech acts (see Label # 1) and related openers (see Label#2) and type a free text of their response into the text box (see Label #3). After finishing building the expressions, participants submit their replies once they click the “Act” button (“See Label #4).


Figure 1. The typical design of DiaCog’s dialog game


2.2. Design of Pedagogical and Learner Models in DiaCog

Student state tracking and adaptation is a common application in intelligent tutoring system (ITS), which can guide us for the design decisions for cognitive assistant implementations. In the context of modern ITS design, the educational needs of students are tracked and adapted by domain model, pedagogical model and learner model. Domain model (DM) tracks the set of skills, atomic components of knowledge objects and content-based strategies. Learner model(LM) tracks cognitive, motivational, affective and other psychological states of learners during the tutoring activities. Pedagogical model (PM) decides tutoring strategies and course of action by taking input from LM and DM.

Figure 2. Presentation and relations of LM, PM. adn DM

As Figure 2 demonstrates, since LM and DM feed PM, it is a better to decide the PM design first before deciding for the LM design (DM will not be discussed here since it may dramatically change due to the topic and learning domain which is difficult to comprehend in the study). Following section describes the pedagogical strategies for designing PM.

2.3. Pedagogical Model

To tackle the design questions of selecting a suitable learning model for MOOC, we form relationships between domain model (DM), pedagogical model (PM) and learning model (LM). Then, we need to analyze the educational needs of typical students to build PM. For PM design, we list learning theories and values to track in Table 2 and Table 3 based on an extended version of Sottilare et al (2013).

Table 2. Followed theories for Pedagogical Model


Table 3. Followed values and variables for Pedagogical Model


As given in Figure 2, considering the roles of different models in system design, cognitive assistant adaptation and tracking related modules may take large number of variables to adaptn in theory. However, in practice, the numbers of variables are relatively small due to scale up issues.
PM can be designed by using different pedagogical strategies as presented in Table 3. In our design, we selected a purposeful PM strategy that is fitting the nature of the pedagogical needs of students’ that are learning by engaging interactive dialogs in discussions. For this purpose, specifically considering the dialog-based model of DiaCog, selected PM should track self-regulation and social constructivist educational needs (and linked values) since these are suitable for learning by dialogs strategies in where the learner arrives at solutions through active participation in dialog interactions. In DiaCog, tracking and adaptation of these values are achieved by LM, which feed the PM. The design process of LM and methods that LM uses explained in the following section.

2.4. Learner Model

Learner models usually infer the diagnosis like a feature to represent the current knowledge state of the students including learning difficulties and misconceptions (Conati, 2002). The learner model is dynamically updated as students build knowledge thus the model represents about latest understanding of the students (Bull, 2004). Learner models describes the (1) cognitive processes that governs students’ interactions, (2) knowledge gap levels between the student and expert, (3) students’ behavior patterns, (4) students’ characteristics and profiles (Webb, et al., 2001). Using predictive learner models, it is possible to have an insight into the nature of the dialogical interactions and learning. Using these models, each reasoning step can be traced, and the misconceptions can be detected (Johnson and Taatgen, 2005).
DiaCog uses a dialog-based learning strategy as described and modeled in PM. Thus, the learner model also should be dialog based to match with PM. In dialog based learning models, analyzing the knowledge level and learning status in ongoing dialog of students could model students learning. Basic natural language dialogue analysis techniques such as calculating student turns, number of words used in dialog turns, percentage of words per turns by students are indication of learning in an online platform (Core, 2003; Rose et al., 2003; Huang et al., 2015). More sophisticated techniques such as analyzing the different types of speech acts or dialog moves, structure of discourse and contextual meaning are also possible for tracking the learning in dialog (Sardareh et al., 2014; Prylipko et al., 2014; Vail, 2014; Rotaru, et al., 2006).
The power of learner model may change due to the sophistication of linguistic techniques that are applicable for tracking modules in MOOCS (Forbes-Riley, 2007). For instance, some techniques are easy to apply but not meaningfully informative (e.g. turn counts, dialog length) while others provide much worthwhile data for modeling (e.g. speech acts) (Bernsen, et al., 2012). Some of the techniques are well informative and largely available in the ITS literature to apply in for MOOCS but they require more effort like labeling dialog acts or designing computational algorithms for dialog analysis and labeling of corpus.
In DiaCog, we aim to tackle the issue of understanding deeper pragmatic properties of dialog by providing combination of several sophisticated methodologies that enable analyzing the dialog yielding stronger learner models. Using combining techniques in DiaCog’s proposed LM design it is possible to have sophisticated tracking functions in MOOCS systems. In DiaCog, LM may be improved further for additional applications for more advanced automatic adaptive MOOC functions that can provide more specific personalized and tailored instruction in addition to the pedagogical strategies that PM delivers as discussed above.
LM strategies for DiaCog can be constructed using machine-learning approaches (Webb, et al., 2001). However, requirement of explicitly labeled data is one of the biggest challenges of machine learning approaches for learner modeling and may be challenging for DiaCog’s LM design. Because the correct labels may not be available from simple observation of dialogs, these labels may require a manual coding or an algorithm for label clustering (Fereday et al., 2006). After determining correct labels, speech acts may represent deeper pragmatic properties in dialog, which informs the PM about students’ status. Fortunately, DiaCog interface provides a partially shortcut solution for the issue of manual labeling of speech acts. Since DiaCog interface requires users to indicate their intention using speech acts specially designed and labelled as “dialog moves” in the dialog interaction interface, DiaCog easily understand and categorize the speech acts when users build their expressions in ongoing dialogs (Yengin and Lazarevic, 2014).
Tracking students` learning (knowledge status) and comparing with expert knowledge universe handled in DiaCog as illustrated in Figure 3. To track students’ learning and knowledge status, DiaCog compute the semantic match between all ideal student interactions in a dialog and compares it with an expert knowledge domain map encoded. Comparing students’ current state of learning and knowledge of content with expert knowledge rule spaces is mapped in a QMATRIX, which enables learning-knowledge level for tracking students (Lee and Sawaki, 2009). The use of QMATRIX is relatively well-known technique in rule space learner models such as the “Additive Factor Model” (AFM) (Li et al., 2011) that is using the “Item Response Theory” (Embretson and Reise, 2013). AFM is good model for detection of students’ prior knowledge, which can be used to predict later performances. DiaCog’s LM design uses same principles for this purpose.

Figure 3. LM Design for DiaCog

To map student knowledge and compare it with expert knowledge in QMATRIX, DiaCog needs to know whether students have similar knowledge structures with the expert domain knowledge. In simple words, DiaCog needs to compare the student dialogs with the expert knowledge. If there is a match between student knowledge and expert knowledge; it is mapped in QMATRIX using variables that show the student knows a knowledge or content object. If there is no match, this is mapped, as the student has no knowledge. If there is a match leading a possible misconception, this is mapped as misconception in QMATRIX.
To understand if there is any matching or not, DiaCog must analyze the student input texts in ongoing dialogs. This function is carried in DiaCog by a “semantic matching” engine that measures semantic relatedness using a direct measure which is a stronger and sophisticated technique rather than words analysis. DiaCog “semantic matching” engine uses “cortical.io Retina API” services to make direct semantic comparison of the meanings stored in students’ dialogs. Using semantic relations, DiaCog is capable to measure whether participants of dialogs talk (know) about on the same concepts with the expert domain knowledge universe by applying natural-language processing techniques based on the distributed representations of text segments grounded in a neuro-computational model of semantics (Corticalio, 2015; Chi, 2009). This allows DiaCog to interpret students’ conceptual understanding and their meaning closeness. Figure 4 shows a conceptual dialog mapping of students.

Figure 4. A conceptual dialog mapping of students

According to the example QMATRIX in Figure 4, student knowledge of apple belongs to the knowledge space of fruits. On the other hand, the expert knowledge domain requires student to know/learn the apple as a concept of computer brand Apple (just for giving an example purpose). To detect and understand this student’s knowledge level (concept of apple) and the potential misconception, DiaCog’s LM should match the expert knowledge space with students` knowledge level. For this purpose, semantic matching engine calculates the distance scores for these knowledge objects (apple as fruit and Apple as computer brand) and compare the scores to see how close they are to each other. If the closeness score is less, this means student is very close to expert knowledge space and well knows the knowledge object. If there is a great distance than that means student doesn’t know the knowledge object. If there is an overlap like fruit domain and computer, semantic engine calculates the overlap. The degree of distance inexpert and student semantic relatedness scores are calculated using to the median score of overall population score in the system. Finally, if the overlap is not close to expert knowledge space, then it is labeled as misconception and encoded in this space.


In this study, design of DiaCog application aimed for implementation as cognitive assistant in MOOCS and discussed DiaCog’s functions for enabling tracking students’ knowledge levels and behavior in dialogs. The suggested pedagogical model in DiaCog explains strategies fitting the nature of learning with dialogs in discussions. Matching the pedagogical model, DiaCog’ learner model is demonstrated how to use the semantic match technique to compare student knowledge and learning with an expert knowledge domain map to enable tracking learners’ knowledge when they engage discussion in cognitive assistant supported MOOCS. These design decisions and techniques for pedagogical and learning models let DiaCog to be an alternative module as cognitive assistants for tracking expert knowledge and detect student mastery levels in learning.


DiaCog’s special design and analytical functions opens a door for learner modeling enabling knowledge tracking in dialogs. In future, we plan to train a student and learning model using linguistic features derived from DiaCog by applying machine-learning algorithms to create a model of students thinking processes in dialogs that also can be applied to simulate typical thinking patterns in future dialogs then. Using such models as cognitive assistant implementations in MOCCS may have adaptive pedagogical strategies to the students’ state to improve learning experience by providing additional feedback or course materials dynamically to the situations. Also predictive models based on machine learning algorithms may enable MOOCS to understand the users’ knowledge level positions in different learning scenarios. In addition, cognitive assistants enabled MOOCS can determine future skill levels of students so it can have a control and guiding mechanism for the course of learning through interactions in dialog or other content.
In future, we are also planning to examine how well different student models perform and generalize with different user populations using MOOCS. We will test the effectiveness of the DiaCog on learning by conducting experiments to validate the success of implementation and pedagogic usefulness of DiaCog in real life.


In conclusion, this paper showed and explained the design of cognitive dialog game (DiaCog) as cognitive assistants. The tracking and adapting abilities of DiaCog as a cognitive assistant was illustrated. The methodology of computing the semantic that match between student interactions in a dialog is used to inform DiaCog’s learner model. Semantic fingerprint matching method used to make comparisons with expert knowledge in order to detect student mastery levels in learning.
As a result, DiaCog tool and applied methodologies can be used for implementing pedagogical and learner modeling to track and adept students’ learning.
Finally, this paper discussed the improvements and possible experimental designs to advance the techniques.


This research is made possible through the help and suggestions of former members of A*STAR’s IHPC’s “Collaborative Thinking Technologies” research group Dr. S. Feller, and Dr. J. Herberg.

Conflict of interests
The authors declare no conflict of interest.


Bain, Y. C. (2011). Learning through online discus-sion: a framework evidenced in learners’ interactions, Research in Learning Technology, Vol 19, 29-42, DOI: 10.3402/rlt.v19s1/7779
Bernsen, N. O., Dybkjær, H., & Dybkjær, L. (2012). Designing interactive speech systems: From first ideas to user testing. Springer Science & Business Media. https://goo.gl/PZDM5v
Bull, S. (2004). Supporting learning with open learner models. Planning, 29(14), 1. https://www.researchgate.net/profile/Susan_Bull2/publication/228888082_Supporting_learning_with_open_learner_models/links/0c96052b4861a5d449000000/Supporting-learning-with-open-learner-models.pdf
Chi, M. T. (2009). Active‐constructive‐interactive: A conceptual framework for differentiating learning activities. Topics in cognitive science, 1(1), 73-105. https://doi.org/10.1111/j.1756-8765.2008.01005.x
Conati, C., Gertner, A., & Vanlehn, K. (2002). Using Bayesian networks to manage uncertainty in student modeling. User modeling and user-adapted interaction, 12(4), 371-417. https://link.springer.com/article/10.1023/A:1021258506583
Core, M. G., Moore, J. D., & Zinn, C. (2003, April). The role of initiative in tutorial dialogue. In Proceedings of the tenth conference on European chapter of the Association for Computational Linguistics-Volume 1 (pp. 67-74). Association for Computational Linguistics. https://aclanthology.info/pdf/E/E03/E03-1072.pdf
Corticalio (2015). Sparse distributed representations. http://www.cortical.io/technology_representations.html
Embretson, S. E., & Reise, S. P. (2013). Item response theory. Psychology Press. https://goo.gl/Euq1sm
Ezen-Can, A., Boyer, K. E., Kellogg, S., & Booth, S. (2015, March). Unsupervised modeling for understanding MOOC discussion forums: a learning analytics approach. In Proceedings of the fifth international conference on learning analytics and knowledge (pp. 146-150). ACM. https://www.intellimedia.ncsu.edu/wp-content/uploads/ezen-can-lak-2015.pdf
Fereday, J., & Muir-Cochrane, E. (2006). Demonstrating rigor using thematic analysis: A hybrid approach of inductive and deductive coding and theme development. International journal of qualitative methods, 5(1), 80-92. http://journals.sagepub.com/doi/abs/10.1177/160940690600500107
Forbes-Riley, K., Litman, D. J., Purandare, A., Rotaru, M., & Tetreault, J. (2007, June). Comparing linguistic features for modeling learning in computer tutoring. In Aied (Vol. 158, pp. 270-277). https://www.cs.rochester.edu/~tetreaul/aied07-ios2-final.pdf
Gagné, M., & Deci, E. L. (2005). Self‐determination theory and work motivation. Journal of Organizational behavior, 26(4), 331-362. https://doi.org/10.1002/job.322
Hawkey, K. (2003). Social constructivism and asynchronous text-based discussion: A case study with trainee teachers. Education and Information Technologies, 8(2), 165-177. https://link.springer.com/article/10.1023/A:1024558414766
Huang, Y. M., & Liang, T. H. (2015). A technique for tracking the reading rate to identify the e‐book reading behaviors and comprehension outcomes of elementary school students. British Journal of Educational Technology, 46(4), 864-876. https://doi.org/10.1111/bjet.12182
Johnson, A., & Taatgen, N. (2005). User modeling. The handbook of human factors in web design, 424-438. https://pdfs.semanticscholar.org/83d7/a8c7faf225b9ff8172693d1faa7e3cf45f77.pdf
Kolb, D.A. (1985). Learning-style inventory: Self-scoring inventory and interpretation book-let (2nd ed.). Boston: McBer & Co.
Kop, R., Fournier, H., & Mak, J. S. F. (2011). A pedagogy of abundance or a pedagogy to support human beings? Participant support on massive open online courses. The International Review of Research in Open and Distributed Learning, 12(7), 74-93. http://dx.doi.org/10.19173/irrodl.v12i7.1041
Lee, Y. W., & Sawaki, Y. (2009). Cognitive diagnosis and Q-matrices in language assessment, Journal Language Assessment Quarterly, Vol. 6, 169-171, doi.org/10.1080/15434300903059598
Li, N., Cohen, W. W., Koedinger, K. R., & Matsuda, N. (2011, July). A Machine Learning Approach for Automatic Student Model Discovery. In EDM (pp. 31-40). https://files.eric.ed.gov/fulltext/ED537453.pdf#page=44
Nkuyubwatsi, B. (2013, October). Evaluation of massive open online courses (MOOCs) from the learner’s perspective. In European Conference on e-Learning (p. 340). Academic Conferences International Limited. https://search.proquest.com/openview/2616f896b4618dba39f72b248bd6d47a/1?pq-origsite=gscholar&cbl=1796419
Palmer, S., Holt, D., & Bray, S. (2008). Does the discussion help? The impact of a formally assessed online discussion on final student results. British Journal of Educational Technology, 39(5), 847-858. https://doi.org/10.1111/j.1467-8535.2007.00780.x
Prylipko, D., Rösner, D., Siegert, I., Günther, S., Friesen, R., Haase, M., ... & Wendemuth, A. (2014). Analysis of significant dialog events in realistic human–computer interaction. Journal on Multimodal User Interfaces, 8(1), 75-86. https://link.springer.com/article/10.1007/s12193-013-0144-x
Rosé, C. P., Bhembe, D., Siler, S., Srivastava, R., & VanLehn, K. (2003). The role of why questions in effective human tutoring. In Proceedings of the 11th International Conference on AI in Education (pp. 55-62). https://www.researchgate.net/profile/Stephanie_Siler/publication/243787583_The_Role_of_Why_Questions_in_Effective_Human_Tutoring/links/00b4953bd580c7117d000000/The-Role-of-Why-Questions-in-Effective-Human-Tutoring.pdf
Rosenberg, M., & Burkert, A. (2015). Learning Styles and their Effect on Learning and Teaching. Forschende Fachdidaktik, 103-130. https://goo.gl/qMcgYh
Rotaru, M., & Litman, D. J. (2006, July). Exploiting discourse structure for spoken dialogue performance analysis. In Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing (pp. 85-93). Association for Computational Linguistics. http://www.aclweb.org/anthology/W06-1611
Sardareh, S.A.; Aghabozorgi, S.; Dutt, A. (2014). Reflective dialogues and students’ problem solving ability analysis using clustering. In: Proceedings of the 3rd International Conference on Computer Engineering & Mathematical Sciences (ICCEMS 2014), 04-05 Dec 2014, Langkawi, Malaysia. http://eprints.um.edu.my/12988/
Soloman, B. A., & Felder, R. M. (2014). Index of Learning Styles Questionnaire [Internet]. North Carolina State University http://www.engr.ncsu.edu/learningstyles/ilsweb.html
Sottilare, R. A., Graesser, A., Hu, X., & Holden, H. (Eds.). (2013). Design recommendations for intelligent tutoring systems: Volume 1-learner modeling (Vol. 1). US Army Research Laboratory. https://goo.gl/8HGjp4
Thomas, M. J. (2002). Learning within incoherent structures: The space of online discussion forums. Journal of Computer Assisted Learning, 18(3), 351-366. https://doi.org/10.1046/j.0266-4909.2002.03800.x
Vail, A. K., & Boyer, K. E. (2014, June). Identifying effective moves in tutoring: On the refinement of dialogue act annotation schemes. In International Conference on Intelligent Tutoring Systems (pp. 199-209). Springer, Cham. https://link.springer.com/chapter/10.1007/978-3-319-07221-0_24
Webb, G. I., Pazzani, M. J., & Billsus, D. (2001). Machine learning for user modeling. User modeling and user-adapted interaction, 11(1-2), 19-29. https://link.springer.com/article/10.1023/A:1011117102175
Yengin, I., & Lazarevic, B. (2014). The DiaCog: A Prototype Tool for Visualizing Online Dialog Games’ Interactions. Research in Higher Education Journal, 25. https://eric.ed.gov/?id=EJ1055317
Zapata, M. (2010). Estrategias de evaluación de competencias en entornos virtuales de aprendizaje. RED. Revista de Educación a Distancia. Sección de Docencia Universitaria en la Sociedad del Conocimiento. Número 1. Consultado https://www.um.es/ead/reddusc/1/eval_compet.pdf

Corresponding Author
Dr. Adem Karahoca, Bahcesehir University Engineering Faculty, Turkey
E-mail: adem.karahoca@eng.bau.edu.tr
This work is licensed under a Creative Commons Attribution - NonCommercial - NoDerivs 4.0
The article is published with Open Access at www.ijcrsee.com