www.ijcrsee.com
267
Rojko, K., & Erman, N. (2023). The impact of the Covid-19 pandemic on higher education students’ perceptions of educational
applications and platforms, International Journal of Cognitive Research in Science, Engineering and Education (IJCRSEE),
11(2), 267-279.
Introduction
Teachers play many roles in classrooms, including inspiring students, stimulating their interest in
the subject, and serving as a source of knowledge and information. Besides, commitment and passion are
required, while a combination of teaching paradigms is needed to maintain momentum and transform the
classroom into a learning environment full of enthusiasm and interest (Powell et al., 2012).
Teachers increasingly complain that their students do not show interest, do not prepare for lessons,
have passive attitudes toward learning activities, and limited awareness of their own learning process
(Buijs and Admiraal, 2013). This, and the contemporary competence-oriented higher education, requires
a correspondingly strong focus on pedagogy. But there is also a growing recognition that to develop
the knowledge and skills needed for the 21
st
century, we also need innovations in the eld of teaching
(Paniagua and Istance, 2018).
As explained by Heacox and Pengal (2009), the greatest challenge for today’s teacher is to respond
to the increasingly diverse learning needs, styles, and social backgrounds of students, which affect their
willingness to learn, learning styles, motivation, interests, attitudes to learning, and self-condence. He,
therefore, suggests that differentiation of instruction, which puts students at the center of teaching and
learning and also allows learning needs, learning styles, and interests to guide instructional design, can
be very helpful, including varying the pace of learning, the level of difculty and the way instruction is
delivered (Heacox and Pengal, 2009).
The traditional lecture is not an effective learning environment for many students, as many do
not actively participate during a traditional lecture (Bajpai, Biberman and Ye, 2019). The goal is to keep
students active, and approaches to learning are changing. In student-centered learning, the focus is on
students’ needs, abilities, interests, and learning styles, and the teacher is only a facilitator of learning. Due
The Impact of the Covid-19 Pandemic on Higher Education Students’
Perceptions of Educational Applications and Platforms
Katarina Rojko
1*
, Nuša Erman
1
1
Faculty of Information Studies, Novo mesto, Slovenia,
e-mail: katarina.rojko@s.unm.si, nusa.erman@s.unm.si
Abstract: The contemporary development of education is marked by the rapid expansion of online educational applications
and platforms. Consequently, it is becoming clear that teachers must consider them and in light of their changing role, know
what they can offer in their eld to be able to recommend them to students to get additional knowledge or even to integrate
them into their teaching. As students in the eld of computer science and informatics are particularly accustomed to considering
online knowledge resources, we decided to investigate their perception of educational applications and platforms. Based on
the analysis of the results of their testing of educational platforms and applications, we found that for them the most important
is content, followed by personal preferences, reason, user experience, price, etc., and only in the last place is a certicate. The
most frequent word in their research reports proved to be knowledge, which we included under the code reason, followed by
research, content, time, free, variety, and quality. It also turned out that students’ experience of testing educational applications
and platforms is predominantly positive and has even improved over the course of the last three years, which we attribute to
the effects of the Covid-19 coronavirus epidemic. The comparison of pre-pandemic and post-pandemic data also revealed that
positive sentiment came to the front, while students now prioritize user experience, reason, and quality over the content and
personal preferences compared to the pre-pandemic period, while they are still aware of the need for exploration.
Keywords: e-education, e-learning, higher education teaching, educational applications, educational web platforms.
Original scientic paper
Received: March, 30.2023.
Revised: May, 11.2023.
Accepted: May, 24.2023.
UDC:
004.9:37.011.3-057.85
37.048:[616.98:578.834
10.23947/2334-8496-2023-11-2-267-279
© 2023 by the authors. This article is an open access article distributed under the terms and conditions of the
Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
*
Corresponding author: katarina.rojko@s.unm.si
www.ijcrsee.com
268
Rojko, K., & Erman, N. (2023). The impact of the Covid-19 pandemic on higher education students’ perceptions of educational
applications and platforms, International Journal of Cognitive Research in Science, Engineering and Education (IJCRSEE),
11(2), 267-279.
to the development of the World Wide Web and search engines, ‘on-demand’ information has become a
reality (Hirsh, 2018). In this context, students need to be active and responsible participants in the learning
process, and the teacher plays a key role in the whole process.
E-learning uses various information-communication technology (ICT) tools to make the teaching
and learning process more effective (Bajpai, Biberman and Ye, 2019). As a result, classroom lectures,
whose main objective is to cover the content of the course, can be expected to become obsolete, as can
the willingness of students to attend these classroom lectures, as technology allows them to learn at
their own pace and in their own time (Moreno, 2018), expanding the possibilities for individualisation and
personalisation of learning (Urbančič, 2021). More and more people are realising the benets of learning
at a pace that suits them best, which is why e-learning is on the rise.
E-learning allows the teacher and the student to be independent in terms of location and time, making
it a form of distance learning (Bregar, Zagmajster and Radovan, 2010). Furthermore, the advantage of
online education is that many courses are very affordable for the value they offer and are aimed at almost
every individual in the world, whereas traditional education involves a limited number of students (Krastev,
2019), which is an additional reason for the rapid rise of educational apps and platforms.
While the concept of e-learning generally refers to the methods of education implemented using
ICT, ICT in the context of e-learning represents only part of the technological solutions that allow us to
complement or enrich the basic foundations and concepts of traditionally based learning (Jagodič, 2010).
Nevertheless, ICT is a potentially powerful tool for expanding educational opportunities, both formal and
non-formal (Adenusi, Adebayo and Oni, 2019), and it is the development of ICT that has brought about
signicant changes in the eld of education (Semerci and Aydın, 2018).
However, we recognise that teachers need to have certain digital competencies to be able to use
online platforms and applications for communication, teaching, and research. Therefore, national projects
to promote the use of ICT by (higher education) teachers are very welcome. Teachers play a central role in
integrating ICT into the classroom, so improving their digital competencies and attitudes is crucial for the
effective integration of ICT in education (Semerci and Aydın, 2018). In addition, it is important to be aware
that today’s students are so-called digital natives, who are expected to have the knowledge and skills that
allow them to handle ICT tools ‘naturally’ (Boh Podgornik et al., 2016), so it is not surprising that students
often expect this from teachers as well.
The introduction of the use, integration, and dissemination of ICT has ushered in a new era in
educational methodologies, radically changing traditional teaching methods and learning patterns in the
eld and offering modern learning experiences for both teachers and students (Adenusi, Adebayo and Oni,
2019). Therefore, different teaching/learning strategies need to be applied in education. This is because
digital technologies are often used to support the extension of university education into the home, social
and work environments (Castañeda and Selwyn, 2018).
As a result, ICT has a broader role in teaching and learning activities into which it can be integrated
as a tool in the learning process (Lautenbach, 2014). Furthermore, the important role of ICT can also
be found in activities such as content creation, student performance monitoring, training, knowledge
management and organisation, and formative monitoring (Margaret et al., 2018). Black and Wiliam
(2009), two of the most referenced authors, established a conceptual framework for understanding
formative monitoring through a series of publications that can be traced back at least to the mid-1990s.
Formative monitoring can be imagined as a “bridge between learning and teaching”. It is based on ve key
strategies: clarication, participation in setting and understanding learning intentions and success criteria;
designing classroom activities that provide evidence of learning; providing feedback; activating students
to become teachers to each other; and activating students to self-manage their learning (Palir Mavrič,
2017). The integration of ICT was identied as a good practice, particularly in stimulating and increasing
students’ motivation. But the use of technology in learning does not automatically result in engagement
in learning activities and improved learning outcomes (Sinha et al., 2015), and for this reason, well-
designed computer-supported collaborative learning (CSCL) environments for collaborative learning and
problemsolving might intensify learning if used based on a teacher’s instructional support (Pietarinen,
Palonen and Vauras, 2021).
Moreover, teachers and students must understand that independent learning using online
applications and platforms has a major disadvantage if not carefully addressed and that is a lack of
collaboration. Namely, students generally do not recognise the advantages of collaborative learning, and
therefore teachers should play an important role in encouraging and inuencing student participation and
achievement (Webb et al., 2019). The teacher’s role has thus changed to act as a coach and facilitator of
students’ thinking while modelling the learning process (Anderson, 2002).
ICT tools are thus playing an increasingly important role in the renewal of educational methods.
www.ijcrsee.com
269
Rojko, K., & Erman, N. (2023). The impact of the Covid-19 pandemic on higher education students’ perceptions of educational
applications and platforms, International Journal of Cognitive Research in Science, Engineering and Education (IJCRSEE),
11(2), 267-279.
This is manifested in the form of digital learning materials instead of printed books, interactive materials,
e-communication between teachers and students, etc. (Berényi and Deutsch, 2018). While the pandemic
has increased the use of and changed teachers’ attitudes toward digital learning materials, quality teaching
in the digital age would require supporting teachers with training to make working with digital materials
routine (Mohar and Kovač, 2021). ICT tools used to create and disseminate knowledge including equipment
and software solutions have become an indispensable part of modern culture, which is spreading globally
through education (Bajpai, Biberman and Ye, 2019). Software that supports the didactic process in higher
education includes internet browsers, ofce suites, communicators, specialised software suitable for
classroom use, software that supports group work, and above all e-learning applications and platforms
(Grooms, 2018), which are the focus of our research.
As the use of educational applications and online platforms is growing rapidly (e.g., Data Bridge
Market Research, 2022 or Dhawal, 2021), we decided to explore their use for study purposes. To this end,
we analysed the opinions of students from four Informatics and Computer Science degree programmes,
two rst-degree programmes and two second-degree programmes.
We started the survey in 2019 when we rst asked students enrolled in their nal year of study on
these four study programmes about which apps and online platforms, they use to support their studies.
Based on the responses we received, we found that the use of apps and online platforms is much more
widespread among students and not limited to those suggested by teachers. In addition, the students
presented in their research reports the results of the testing of the apps and online platforms they use to
complement their studies.
As today’s teachers (and especially teachers in information technology) and computer science
programmes, due to the specicity of these programmes) are increasingly aware of the need to use
and even compete with online educational opportunities, it has become clear that they need to change
their teaching methods and see their profession as adding value and upgrading the information that
students can obtain via the Internet. Our main research objective is therefore to present the ndings of a
qualitative analysis of students’ views when testing educational applications and online platforms, which
we believe could encourage higher education teachers to make greater use of them in their teaching,
not as a substitute, but as an additional source of ‘on-demand’ knowledge and motivation for learning.
Finally, this paper aims to encourage higher education institutions to reect on how encouraging the use
of educational apps and online platforms can enable students to perform even better in exams, while also
spreading awareness that they can deepen their knowledge in this way too.
Materials and Methods
Statistics (e.g., Wise, 2022) show that e-learning has grown exponentially in recent years. Studies
also show (e.g., Suwal and Singh, 2018) that the use of e-learning applications and platforms is well
accepted by students as it gives them positive experiences, and consequently, there is a growing need
for more integration of e-learning applications and platforms into existing learning activities and practices.
In the ood of different types of educational apps and online platforms, it is difcult for a teacher to decide
which app or platform is appropriate to use and recommend to students. In addition, more and more
new apps and online platforms are emerging, making it difcult to keep up with these advances and the
increasing range. In this context, we have identied the need for a systematic review of the educational
apps and platforms that students individually choose to use, as well as the way they are perceived. To do
so, we have conducted a survey in which students themselves, based on their study interests, chose an
app or online platform to test. They presented their views gained during the testing in research reports,
which we then used to achieve the aim of our research: to show that teachers need to change their role,
as the internet offers instant information on almost any topic and as educational apps and platforms are
gaining in importance. Accordingly, teachers also need to change their teaching methods to offer students
all they can gain from e-learning.
Based on the content of the students’ research reports, we identied the codes and the most
frequently written words to give their perspective on the impact and usefulness of some educational apps
and platforms in their elds of study, as the research also aims to encourage teachers to reect on how
the integration of apps and platforms in their courses can bring added value. In addition, the Covid-19
coronavirus pandemic has further stimulated the development of online educational resources. Based on
this research objective, the following research questions (RQs) were set:
RQ1: What has proved to be most important in the testing of educational apps and platforms by
students?
www.ijcrsee.com
270
Rojko, K., & Erman, N. (2023). The impact of the Covid-19 pandemic on higher education students’ perceptions of educational
applications and platforms, International Journal of Cognitive Research in Science, Engineering and Education (IJCRSEE),
11(2), 267-279.
For this question, we were looking for codes that could be extracted from a close reading of the
assignments that students had prepared based on testing educational apps and platforms. We assumed
that students would put the user experience at the center, and quality would be the decisive factor (as in
Rojko, 2020). On the other hand, although some apps and platforms provide certicates and recognition
of the knowledge acquired, we assumed that the absence of formal recognition would demotivate students
to learn online on their own initiative.
RQ2: Which words stand out in the research reports of students who have tested educational apps
and platforms?
We asked students to use different criteria to test different applications and platforms for their
research reports, mainly qualitative but sometimes also quantitative. We anticipated that the words
quality, user experience, quantity, price and certicate and their synonyms would appear most frequently.
We were also interested in whether positive or negative experiences predominated.
RQ3: Has the impact of the Covid-19 coronavirus pandemic changed the way students view
educational apps and platforms?
We expected that some new dominant words would appear in the more recent research reports
(from the second half of the academic year 2021/22) compared to the reports from the period before the
start of the epidemic (from the rst half of the academic year 2019/20). Namely, we observed that the
Covid-19 pandemic was reected in an increased motivation of students to search for additional online
education resources and in their increased willingness to try them out, as distance education had been
the only option for some time.
The survey was launched in 2019 when we asked students enrolled in the nal year of four
Informatics and Computer Science degree programmes from Slovenia, which educational apps and
platforms they use to support their studies. In addition, the majority of these students also produced a
research seminar paper based on the results of their testing.
Students chose and tested the following educational apps and/or platforms (in alphabetical order):
ArtofProblemSolving, Astra.si, Bussuu, CK-12, Code With Mosh, CodeAcademy, Codewars, Coursera,
Cybrary, Drops, Duolingo, edX, freeCodeCamp, FreeVideoLectures, FunEasyLearn, Khan-academy,
KodeKloud, Laracast, LinkedIn learning, Mathway, Memrise, Micro:bit, Mimo, MojeZnanje.si, Photomath,
Pluralsight, Programming Hub, Qlango, Scratch, Skillshare, SoloLearn, SQLZOO, Stack Overow,
Symbolab, Treehouse, Tutorialspoint, Udemy, Udacity, W3Schools, Wolfram Alpha, Yoast and YouTube.
Note that some students chose the same apps and platforms, and Udemy was the most repeated platform,
18 times.
Some of the educational apps and platforms chosen specialise in one area, others cover several
different areas, and students were instructed to focus on one specic subject area for any comparisons.
As can be seen from the list of names, some of the listed platforms cover different areas of knowledge,
although all students focused on computer science, while the listed apps focus on computer science (most
frequently tested), mathematics (medium frequency), and foreign language learning (least frequently
tested). The majority of the apps and platforms analysed were international, but two Slovenian platforms,
Astra.si and MojeZnanje.si, were also included.
Among the 140 students enrolled in the nal year of their rst (professional study programmes)
and second (masters study programmes) cycle studies in the academic years 2019/20, 2020/21 and
2021/22, 113 (81%) submitted their research seminar paper. Among these, we further excluded those
reports in which students tested other applications that they had used during their studies but were not
relevant to our research - e.g., for storing data and literature, such as Dropbox, Evernote, Google Drive,
OneNote and Zotero. Thus, the nal 90 research reports (50 reports by rst-cycle students and 40 reports
by second-cycle students) were considered, out of these 29 from the academic year 2019/20 (rst half
of the year (in continuation referred to as the pre-pandemic period) 16 reports), 36 from 2020/21, and 25
from 2021/22 (second half of the year (in continuation referred to as post-pandemic period) 10 reports).
We considered the content of research reports as transcripts, which were then subjected to qualitative
analysis to answer our research questions. We resorted to manual coding and used the Atlas.ti qualitative
analysis tool.
Manual coding (using the open axial coding methodology) required close reading and careful
consideration of the students’ opinions and observations, taking special care to avoid concepts we had
learned from studying the literature and our views and expectations. The attribution of concepts was
done both by direct conceptualisation and synonymy. The selection was made based on the conceptual
framework and within it based on recurring terms in the students’ assignments. Individual terms were
further evaluated and linked to semantically equivalent terms, which were grouped into key categories
(codes) based on their meaning.
www.ijcrsee.com
271
Rojko, K., & Erman, N. (2023). The impact of the Covid-19 pandemic on higher education students’ perceptions of educational
applications and platforms, International Journal of Cognitive Research in Science, Engineering and Education (IJCRSEE),
11(2), 267-279.
Atlas.ti was used to search for the most frequently repeated words in the assignments. Although this
tool also allows the preparation of word clouds based on Slovene texts, it only formats them appropriately
in some of the world’s leading languages. As the tool is not able to combine Slovenian words in different
conjugations, does not correctly identify word types, etc., we rst translated the 90 analysed seminar
reports, with an average size of one author eld (The author’s pole consists of 30 000 characters (letters,
numbers, punctuation, spaces) or approximately 16 typed pages.), into English. We then extracted nouns,
verbs, adjectives, and adverbs, followed by manual removal of the remaining inappropriate words. Finally,
for the most common words, we rst considered separately the terms for the full spectrum of experience
from positive to negative, and then all the remaining most commonly used words.
In addition, we also analysed the research reports from the rst half of the 2019/20 academic
year (considered as pre-pandemic period reports) and the research reports from the second half of the
2021/22 academic year (considered as post-pandemic period reports) separately and used the results
to compare the two periods, which led us to detect the changes in the testing experience as well as in
students’ attitudes towards educational apps and platforms. To do so, we applied chi-squared test at
signicance level of 0.05 to nd potential association between pre- and post-pandemic periods regarding
the testing experience and students’ attitudes. In the case of a statistically signicant chi-squared test, the
association was conrmed and was then tested for strength using the Cramer’s V contingency coefcient
at 1 degree of freedom. We used the value of the latter to determine whether the association was weak
(Cramer’s V value between 0.1 and 0.3), moderate (Cramer’s V value between 0.3 and 0.5) or strong
(Cramer’s V value between 0.5 and 1.0). To visually represent the observed associations, we used mosaic
plots in which two-dimensional frequency table is displayed using rectangles. The rectangular regions
in the mosaic plot are proportional to the cell frequencies they represent, where shadings indicate the
residuals between observed and expected frequencies. Positive value of a residual is represented by the
blue color, and negative value of a residual is represented by the red color, whereas the shade of color
indicates the site of residual: the larger the absolute value of residual, the darker the color, and vice versa,
the smaller the absolute value of residual, the lighter the color. This is also presented in the legend added
to the mosaic plot (Kabacoff, 2015).
Results
We rst present the results of the manual coding (RQ1) and then the results of the analysis using
the Atlas.ti tool, which was used to produce linear word clouds for all analysed research reports together
(RQ2) and separately based on research reports from before the Covid-19 epidemic and two and a half
years later (RQ3).
Coding results
Based on a close reading of a selection of research reports (N=90), we identied 10 key content
categories (codes) in the texts: content, personal preference, reason, user experience, price, quantity,
basic, development, quality, advanced and endorsement.
Figure 1. Comparison of paradigm models with Rojko (2020)
www.ijcrsee.com
272
Rojko, K., & Erman, N. (2023). The impact of the Covid-19 pandemic on higher education students’ perceptions of educational
applications and platforms, International Journal of Cognitive Research in Science, Engineering and Education (IJCRSEE),
11(2), 267-279.
Figure 1 shows that our analysis produced slightly different results than Rojko (2020) as interaction,
motivation, recommendation, and traditional were replaced by reason, development, and content, in
Figure 1 right, according to our ndings. Similarly, Rojko (2020) found that quality dominates over user
experience and quantity, whereas our ndings showed a different picture – as presented in the Figure 2.
Figure 2. Representation of each dominant word by code
Figure 2 shows the representation of each of the dominant words by codes (We did not add the code
area to the selection (dominant words: coding, programming, foreign, language(s), technology) because
it is not relevant, as these are the elds of study of the students whose research seminar papers are
analysed.), which were written at least 200 times in all research reports combined. It turns out that content
stands out the most (words: content, tutorial, example, material, new, video, game, help, information,
data), followed by personal preferences (words: option, year, years, time, different, decision, way,
possibility, relevant, important, choose, criterion, need, work). For content, we would highlight the words
video, game, and help as they best convey the students’ observations, while for personal preferences we
would highlight time, different, and way as they best convey the students’ thinking.
Dominant words
The dominant words were extracted using Atlas.ti tool, but despite the translation of the analysed
research reports into English, a lot of manual renement was necessary (see Appendix 1 for a full list of
removed words). The words that appear in Figure 3 appear at least 200 times in the analysed reports. The
most frequently occurring word is language (N=1476; also, languages in 16th place (N=657)), which in turn
refers to the domains of the applications and platforms tested (computer languages, foreign languages).
This is also the case for the third-ranked word programming (N=1342), the twelfth-ranked word coding,
etc. The second-ranked word is knowledge (N=1419), which we have classied under the code reason.
The results based on the most frequent individual words thus show that knowledge is the most important
for students, research is needed, followed by content, time is a constraint, and free, variety, and quality
are also among the most important.
It should also be noted that it is necessary to group certain words that appear separately in the
gure below, e.g. research (N=1318) and search (N=235), information (N=595) and data (N=508), nd in
the present and past tense (N=559 and N=371), video (N=459) and videos (N=270), year (N=238) and
years (N=204), play (N=358) and games (N=205), etc.