Visual Discrete Format: An Alternative to Likert-Type Formats of Survey Items Sensitive Enough to Measure Small Changes in Stable Constructs Such as Self-Concept in Science

Daniel Solis; David Hutchinson; Nancy Longnecker

doi:10.23947/2334-8496-2022-10-2-01-16

Authors

Daniel Solis Unidad Profesional Interdisciplinaria de Ingeniería Campus Zacatecas, Instituto Politécnico Nacional, Zacatecas, Zacatecas, México https://orcid.org/0000-0002-2717-2983
David Hutchinson The Dodd-Walls Centre for Photonic and Quantum Technologies, Department of Physics, University of Otago, Dunedin, Otago, New Zealand https://orcid.org/0000-0003-4923-0141
Nancy Longnecker Department of Science Communication, University of Otago, Dunedin, Otago, New Zealand https://orcid.org/0000-0002-2798-6389

DOI:

https://doi.org/10.23947/2334-8496-2022-10-2-01-16

Keywords:

Likert-type scale, visual discrete scale, visual discrete format, self-concept in science, scientific literacy, questionnaire

Abstract

A visual discrete format was developed for use in surveys as an alternative to a Likert-type format to measure changes in a relatively stable construct before and after an intervention. Visitors to a science centre ranging in age from 8 years old upwards were asked to complete surveys that included a seven-item questionnaire scale on self-concept in science using either the Likert-type format (n=446) or the visual discrete format (n=375), before and after their visit. A new set of statements to assess self-concept in science were developed and validated so they could be conducted on either format. Matched responses were used to calculate internal consistency, standard deviation, confidence interval and percentage of missing values; these were all similar for both formats. In the visual discrete format, text labels were replaced by visual labels with a single image of different size for each response option. While a Likert-type format describes a level of agreement/disagreement with a specific item, the visual discrete format relates more to choosing the best reflection of the self in relation to that item. The Likert-type format included a set of emoji in its descriptions to appeal to younger participants. The visual discrete questionnaire scale detected a small increase with medium effect size in self-concept in science after the visit to the science centre while the Likert-type questionnaire scale did not detect any change. This suggests the proposed new format can not only be a viable and useful alternative, but potentially more sensitive under certain conditions.

Downloads

Download data is not yet available.

References

Bong, M., & Skaalvik, E. M. (2003). Academic self-concept and self-efficacy: How different are they really? Educational psychology review, 15(1), 1-40. https://doi.org/10.1023/A:1021302408382 DOI: https://doi.org/10.1023/A:1021302408382

Borgers, N., Hox, J., & Sikkel, D. (2003). Response quality in survey research with children and adolescents: the effect of labeled response options and vague quantifiers. International Journal of Public Opinion Research, 15(1), 83-94. https://doi.org/10.1093/ijpor/15.1.83 DOI: https://doi.org/10.1093/ijpor/15.1.83

Brennan, R. L., & Lee, W.-C. (2018). True Score. In B. B. Frey (Ed.), The SAGE Encyclopedia of Educational Research, Measurement, and Evaluation (pp. 1724-1728). Thousand Oaks, CA: SAGE Publications. https://dx.doi.org/10.4135/9781506326139 DOI: https://doi.org/10.4135/9781506326139

Creswell, J. W. (2009). Research Design: Qualitative, Quantitative, and Mixed Methods Ap-proaches. Thousand Oaks, CA: SAGE Publications.

De Cruz, H. (2006). Why are some numerical concepts more successful than others? An evolu-tionary perspective on the history of number concepts. Evolution and Human Behavior, 27(4), 306-323. https://doi.org/10.1016/j.evolhumbehav.2006.02.001 DOI: https://doi.org/10.1016/j.evolhumbehav.2006.02.001

Diamond, J., Horn, M., & Uttal, D. H. (2016). Practical evaluation guide: Tools for museums and other informal educational settings. Lanham, MD: Rowman & Littlefield.

Etikan, I., & Bala, K. (2017). Sampling and sampling methods. Biometrics & Biostatistics International Journal, 5(6), 00149. https://doi.org/10.15406/bbij.2017.05.00149 DOI: https://doi.org/10.15406/bbij.2017.05.00149

Fenichel, M., & Schweingruber, H. (2010). Surrounded by Science: Learning science in informal environments. Washington, DC: The National Academies Press. https://doi.org/10.17226/12614 DOI: https://doi.org/10.17226/12614

Field, A. (2013). Discovering Statistics Using IBM SPSS Statistics. Los Angeles, CA: SAGE Publications.

Fowler, F. J., Jr. (2013). Survey research methods (5th ed.) Los Angeles, CA: SAGE Publications.

Friborg, O., Martinussen, M., & Rosenvinge, J. H. (2006). Likert-based vs. semantic differen-tial-based scorings of positive psychological constructs: A psychometric comparison of two versions of a scale measuring resilience. Personality and Individual Differences, 40(5), 873-884. https://doi.org/10.1016/j.paid.2005.08.015 DOI: https://doi.org/10.1016/j.paid.2005.08.015

Friedman, A. J. (Ed.) (2008). Framework for Evaluating Impacts of Informal Science Education Projects. National Science Foundation. Retrieved from https://www.informalscience.org/framework-evaluating-impacts-informal-science-education-projects

Friedman, H. H., & Amoo, T. (1999). Rating the rating scales. Journal of Marketing Management, 9(3), 114-123. Retrieved from https://ssrn.com/abstract=2333648

Hall, L., Hume, C., & Tazzyman, S. (2016). Five degrees of happiness: Effective smiley face likert scales for evaluating with children. Paper presented at the Proceedings of the The 15th International Conference on Interaction Design and Children, New York, NY. https://doi.org/10.1145/2930674.2930719 DOI: https://doi.org/10.1145/2930674.2930719

Hernández, S., Fernández, C., & Baptista, P. (2014). Metodología de la Investigación [Investigation Methodology] (6th ed.). Mexico City, Mexico: McGraw-Hill.

Hirschman, E. C. (1986). The effect of verbal and pictorial advertising stimuli on aesthetic, utilitarian and familiarity perceptions. Journal of Advertising, 15(2), 27-34. https://doi.org/10.1080/00913367.1986.10673002 DOI: https://doi.org/10.1080/00913367.1986.10673002

Illeris, K. (2018). Contemporary theories of learning: learning theorists... in their own words. London, UK: Routledge. https://doi.org/10.4324/9781315147277 DOI: https://doi.org/10.4324/9781315147277

International Association for the Evaluation of Educational Achievement. (1998). IEA Third International Mathematics and Science Study - Repeat: Student questionnaire main survey. Retrieved from https://timssandpirls.bc.edu/timss1999i/pdf/BM2_StudentG.pdf

International Association for the Evaluation of Educational Achievement. (2018). IEA Trends in International Mathematics and Science Study 2003: Main survey student questionnaire grade 4. Retrieved from https://nces.ed.gov/timss/pdf/T19_GR4_StudentQ_USA_Questionnaire.pdf

Jansen, M., Scherer, R., & Schroeders, U. (2015). Students’ self-concept and self-efficacy in the sciences: Differential relations to antecedents and educational outcomes. Contemporary Educational Psychology, 41, 13-24. https://doi.org/10.1016/j.cedpsych.2014.11.002 DOI: https://doi.org/10.1016/j.cedpsych.2014.11.002

Jansen, M., Schroeders, U., & Lüdtke, O. (2014). Academic self-concept in science: Multidimen-sionality, relations to achievement measures, and gender differences. Learning and Individual Differences, 30, 11-21. https://doi.org/10.1016/j.lindif.2013.12.003 DOI: https://doi.org/10.1016/j.lindif.2013.12.003

Kelley, C. M., & Lindsay, D. S. (1993). Remembering mistaken for knowing: Ease of retrieval as a basis for confidence in answers to general knowledge questions. Journal of Memory and Language, 32(1), 1-24. https://doi.org/10.1006/jmla.1993.1001 DOI: https://doi.org/10.1006/jmla.1993.1001

Kind, P., Jones, K., & Barmby, P. (2007). Developing attitudes towards science measures. International Journal of Science Education, 29(7), 871-893. https://doi.org/10.1080/09500690600909091 DOI: https://doi.org/10.1080/09500690600909091

Krishnamurthi, A., & Rennie, L. J. (2012). Informal science learning and education: definition and goals. Retrieved from https://resources.informalscience.org/informal-science-learning-and-education-definition-and-goals

Kromann, C., Bohnstedt, C., Jensen, M., & Ringsted, C. (2010). The testing effect on skills learning might last 6 months. Advances in Health Sciences Education, 15(3), 395-401. https://doi.org/10.1007/s10459-009-9207-x DOI: https://doi.org/10.1007/s10459-009-9207-x

Lantz, B. (2013). Equidistance of Likert-type scales and validation of inferential methods using experiments and simulations. The Electronic Journal of Business Research Methods, 11(1), 16-28. Retrieved from https://academic-publishing.org/index.php/ejbrm/article/view/1299/1262

Lee, J. D. (1998). Which kids can “become” scientists? Effects of gender, self-concepts, and perceptions of scientists. Social psychology quarterly, 61(3), 199-219. https://doi.org/10.2307/2787108 DOI: https://doi.org/10.2307/2787108

Likert, R. (1932). A technique for the measurement of attitudes. Archives de Psychologie [Archives of Psychology](140), 1-55.

Longnecker, N. (2016). An integrated model of science communication. Journal of Science Communication, 15, 1-12. https://doi.org/10.22323/2.15050401 DOI: https://doi.org/10.22323/2.15050401

Longnecker, N., Elliot, J., & Gondwe, M. (2014). Inspiring Australia: An evaluation tool for science engagement activities. https://doi.org/10.13140/RG.2.1.3477.9683

MacKenzie, S. B., & Podsakoff, P. M. (2012). Common method bias in marketing: causes, mechanisms, and procedural remedies. Journal of retailing, 88(4), 542-555. https://doi.org/10.1016/j.jretai.2012.08.001 DOI: https://doi.org/10.1016/j.jretai.2012.08.001

Martin, A. J., Durksen, T. L., Williamson, D., Kiss, J., & Ginns, P. (2016). The role of a muse-umbased science education program in promoting content knowledge and science motivation. Journal of Research in Science Teaching, 53(9), 1364-1384. https://doi.org/10.1002/tea.21332 DOI: https://doi.org/10.1002/tea.21332

McLeod, A., Pippin, S., & Wong, J. A. (2011). Revisiting the Likert scale: can the fast form ap-proach improve survey research? International Journal of Behavioural Accounting and Finance, 2(3-4), 310-327. https://doi.org/10.1504/IJBAF.2011.045019 DOI: https://doi.org/10.1504/IJBAF.2011.045019

Mellor, D., & Moore, K. A. (2013). The use of Likert scales with children. Journal of pediatric psychology, 39(3), 369-379. https://doi.org/10.1093/jpepsy/jst079 DOI: https://doi.org/10.1093/jpepsy/jst079

Miles, R. S., Alt, M., Gosling, D., Lewis, B., & Tout, A. (Eds.). (1988). The Design of Educational Exhibits (2nd ed.). New York, NY: Routledge. https://doi.org/10.1111/j.2151-6952.1985.tb01634.x DOI: https://doi.org/10.1111/j.2151-6952.1985.tb01634.x

Munshi, J. (2014). A method for constructing Likert scales. SSRN Electronic Journal, 1-12. http://dx.doi.org/10.2139/ssrn.2419366 DOI: https://doi.org/10.2139/ssrn.2419366

Nagengast, B., Marsh, H. W., Scalas, L. F., Xu, M. K., Hau, K.-T., & Trautwein, U. (2011). Who took the “×” out of expectancy-value theory? A psychological mystery, a substantive-methodological synergy, and a cross-national generalization. Psychological science, 22(8), 1058-1066. https://doi.org/10.1177/0956797611415540 DOI: https://doi.org/10.1177/0956797611415540

National Research Council. (2009). Learning science in informal environments: People, places, and pursuits (Vol. 32). Washington, DC: National Academies Press. https://doi.org/10.17226/12190 DOI: https://doi.org/10.17226/12190

Nenkov, G. Y., & Scott, M. L. (2014). “So cute I could eat it up”: Priming effects of cute products on indulgent consumption. Journal of Consumer Research, 41(2), 326–341. https://doi.org/10.1086/676581 DOI: https://doi.org/10.1086/676581

Organisation for Economic Cooperation and Development. (2009). PISA 2006 Technical Report. https://doi.org/10.1787/9789264048096-en DOI: https://doi.org/10.1787/9789264048096-en

Osborne, J. W., & Blanchard, M. R. (2011). Random responding from participants is a threat to the validity of social science research results. Frontiers in Psychology, 1, 220. https://doi.org/10.3389/fpsyg.2010.00220 DOI: https://doi.org/10.3389/fpsyg.2010.00220

Otago Museum. (2019). Annual report 2018-2019. Retrieved from https://otagomuseum.nz/assets/Otago-Museum-Annual-Report-2018-19-low-res-25.11.19.pdf

Paulhus, D. L., & Vazire, S. (2007). The self-report method. In R. W. Robins, R. C. Fraley, & R. A. Krueger (Eds.), Handbook of research methods in personality psychology (pp. 224-239). New York, NY: Guilford Press.

Phan, N. Q., Blome, C., Fritz, F., Gerss, J., Reich, A., Ebata, T., . . . Ständer, S. (2012). As-sessment of pruritus intensity: prospective study on validity and reliability of the visual analogue scale, numerical rating scale and verbal rating scale in 471 patients with chronic pruritus. Acta dermato-venereologica, 92(5), 502-507. https://doi.org/10.2340/00015555-1246 DOI: https://doi.org/10.2340/00015555-1246

Podsakoff, P. M., MacKenzie, S. B., Lee, J.-Y., & Podsakoff, N. P. (2003). Common method bi-ases in behavioral research: A critical review of the literature and recommended remedies. Journal of Applied Psychology, 88(5), 879-903. https://doi.org/10.1037/0021-9010.88.5.879 DOI: https://doi.org/10.1037/0021-9010.88.5.879

Podsakoff, P. M., MacKenzie, S. B., & Podsakoff, N. P. (2012). Sources of method bias in social science research and recommendations on how to control it. Annual Review of Psychology, 63, 539-569. https://doi.org/10.1146/annurev-psych-120710-100452 DOI: https://doi.org/10.1146/annurev-psych-120710-100452

Reynolds-Keefer, L., Johnson, R., Dickenson, T., & McFadden, L. (2009). Validity issues in the use of pictorial Likert scales. Studies in Learning, Evaluation, Innovation and Development, 6(3), 15-25.

Sasson, I. (2014). The role of informal science centers in science education: attitudes, skills, and self-efficacy. JOTSE, 4(3), 167-180. https://dx.doi.org/10.3926/jotse.123 DOI: https://doi.org/10.3926/jotse.123

Schwarz, N. (1999). Self-reports: how the questions shape the answers. American Psychologist, 54(2), 93-105. https://doi.org/10.1037/0003-066X.54.2.93 DOI: https://doi.org/10.1037/0003-066X.54.2.93

Scott, J., Brynin, M., & Smith, R. (1995). Interviewing children in the British household panel survey. In I. Stewart & R. Vaitilingam (Eds.), Advances in family research (pp. 259-266). Essex, UK: University of Essex.

Şentürk, E., & Özdemir, Ö. F. (2014). The effect of science centres on students’ attitudes towards science. International Journal of Science Education, Part B, 4(1), 1-24. https://doi.org/10.1080/21548455.2012.726754 DOI: https://doi.org/10.1080/21548455.2012.726754

Siersdorfer, S., Minack, E., Deng, F., & Hare, J. (2010). Analyzing and predicting sentiment of images on the social web. Paper presented at the 18th ACM international conference on multimedia, Firenze, Italy. https://doi.org/10.1145/1873951.1874060 DOI: https://doi.org/10.1145/1873951.1874060

Solis, D. H. (2020). Exploration and discovery: learning at a science centre. (Doctor of Philosophy), University of Otago, New Zealand. Retrieved from https://ourarchive.otago.ac.nz/handle/10523/10160

Solis, D. H., Hutchinson, D., & Longnecker, N. (2021). Formal Learning in Informal Settings—Increased Physics Content Knowledge After a Science Centre Visit. Frontiers in Education, 6(322). https://doi.org/10.3389/feduc.2021.698691 DOI: https://doi.org/10.3389/feduc.2021.698691

Stange, M., Barry, A., Smyth, J., & Olson, K. (2018). Effects of smiley face scales on visual processing of satisfaction questions in web surveys. Social Science Computer Review, 36(6), 756-766. https://doi.org/10.1177/0894439316674166 DOI: https://doi.org/10.1177/0894439316674166

Stocklmayer, S. M., & Bryant, C. (2012). Science and the public—What should people know? International Journal of Science Education, Part B, 2(1), 81-101. https://doi.org/10.1080/09500693.2010.543186 DOI: https://doi.org/10.1080/09500693.2010.543186

Sturm, H., & Bogner, F. X. (2010). Learning at workstations in two different environments: A museum and a classroom. Studies in Educational Evaluation, 36(1-2), 14-19. https://doi.org/10.1016/j.stueduc.2010.09.002 DOI: https://doi.org/10.1016/j.stueduc.2010.09.002

van Laerhoven, H., van der Zaag-Loonen, H., & Derkx, B. H. (2004). A comparison of Likert scale and visual analogue scales as response options in children’s questionnaires. Acta paediatrica, 93(6), 830-835. https://doi.org/10.1111/j.1651-2227.2004.tb03026.x DOI: https://doi.org/10.1111/j.1651-2227.2004.tb03026.x

Venville, G., Rennie, L. J., Hanbury, C., & Longnecker, N. (2013). Scientists reflect on why they chose to study science. Research in Science Education, 43(6), 2207-2233. https://doi.org/10.1007/s11165-013-9352-3 DOI: https://doi.org/10.1007/s11165-013-9352-3

Weijters, B., Geuens, M., & Baumgartner, H. (2013). The effect of familiarity with the response category labels on item response to Likert scales. Journal of Consumer Research, 40(2), 368-381. https://doi.org/10.1086/670394 DOI: https://doi.org/10.1086/670394

Wilkins, J. L. (2004). Mathematics and science self-concept: An international investigation. The Journal of Experimental Education, 72(4), 331-346. https://doi.org/10.3200/JEXE.72.4.331-346 DOI: https://doi.org/10.3200/JEXE.72.4.331-346

Wittkower, D. E. (2012). On the origins of the cute as a dominant aesthetic category in digital culture. In T. W. Luke & J. Huninger (Eds.), Putting Knowledge to Work and Letting Information Play (pp. 167-175). Rotterdam, The Netherlands: Sense Publishers. https://doi.org/10.1007/978-94-6091-728-8_13 DOI: https://doi.org/10.1007/978-94-6091-728-8_13

Worcester, R. M., & Burns, T. R. (1975). Statistical examination of relative precision of verbal scales. Journal of the Market Research Society, 17(3), 181-197.