Using Text Analytics to Analyze Students’ Open-Ended Evaluation of Teaching

doi:10.22716/sckt.2022.10.1.008

All Issue

2022 Vol.10, Issue 1 Preview Page Next Page

Research Article

Using Text Analytics to Analyze Students’ Open-Ended Evaluation of Teaching 텍스트 애널리틱스를 활용한 서술형 강의평가 분석: Jeonggyu Jee¹
지정규¹; ¹Professor, Department of Global Talent, Busan University of Foreign Studies

¹부산외국어대학교 글로벌인재학부 교수

31 March 2022. pp. 71-80

PDF

Abstract

학생들의 강의평가는 개방적 서술형보다는 약식의 폐쇄형 몇 문항들의 5점 척도로 된 설문 응답에 의존하고 있는 것이 현실이다. 본 연구는 학생의 서술형 강의평가를 SEEQ의 9개 차원으로 토픽을 분류하여 강의 품질 개선을 위한 중점 토픽을 파악하고 1학기와 2학기 토픽 비교를 통해 강의 평가 활용의 개선 방향을 제시한다. 이를 위해 B대학의 2020년 1학기와 2학기 서술형 강의평가 문서 각각 20,569개와 16,166개를 분석하였다. 1학기와 2학기 서술형 평가에 대한 토픽 6개를 각각 추출한 결과 SEEQ에 없는 차원인 동영상 강의와 이클래스 활용이 나타났으며, 학습가치와 학습 범위 차원은 공통으로 나타났고 나머지 차원은 서로 다르게 나타났다. 본 연구의 학술적 시사점은 SEEQ 차원이 서술형 강의 평가에도 그대로 반영되고 있음을 파악하였다. 실무적 시사점은 강의 품질 개선을 위한 중점 차원 제시 및 학생 평가에 대한 피드백을 제시해 줄 수 있는 기반 구축이다.

The reality is that students’ teaching evaluations so far have relied on a five-point scale of a few closed-ended questions rather than an open-ended questions. Therefore, this study classifies the topics of students’ open-ended evaluation of teaching into 9 dimensions of SEEQ, identifies the main topics for improving teaching quality, and suggests the improvement direction of the use of teaching evaluation by comparing the topics in the first and second semesters. For this purpose, I analyzed 20,569 and 16,166 students’ written responses, respectively, in the first and second semesters of 2020 at B University. As a result of extracting 6 topics for open-ended evaluation in the first and second semesters, ‘video lecture’ and ‘use of e-class’, dimensions that are not in the SEEQ, were found. The ‘learning value’ and ‘learning scope’ dimensions were found in common and the rest of the dimensions are different. The academic implications of this study were identified that the SEEQ dimension is reflected in the open-ended evaluation of teaching. The practical implications are to present a focus dimension for improving the quality of teaching and to build a foundation that can provide feedback on student evaluation.

Keywords

Student evaluation of educational quality

Open-ended evaluation of teaching

Topic modeling

Teaching quality

SEEQ

References

M. Sánchez, D. Carmen, M. Jose-Amelio, R. Rafael, and F. Luis, "Relationships among Relational Coordination Dimensions: Impact on the Quality of Education Online with a Structural Equations Model", Technological Forecasting and Social Change, Vol. 166, Article 120608, 2021.https://doi.org/10.1016/j.techfore.2021.120608
M. Schneider and F. Preckel, "Variables Associated with Achievement in Higher Education: A Systematic Review of Meta-Analyses", Psychological Bulletin, Vol. 143, No. 6, pp. 565-600, 2017.https://doi.org/10.1037/bul0000098PMid:28333495
T. Huybers, "Exploring the Use of Best-Worst Scaling to Elicit Course Experience Questionnaire Responses", Assessment & Evaluation in Higher Education, Vol. 42, No. 8, pp. 1306-1318, 2017.https://doi.org/10.1080/02602938.2016.1270256
G. A. Boysen, "Student Evaluations of Teaching during the COVID-19 Pandemic", DOI:10.1037/stl0000222, 2020.
S. L. Wright and M. A. Jenkins-Guarnieri, "Student Evaluations of Teaching: Combining the Meta-Analyses and Demonstrating Further Evidence for Effective Use", Assessment & Evaluation in Higher Education, Vol. 37, No. 6, pp. 683-699, 2012.https://doi.org/10.1080/02602938.2011.563279
P. Spooren, B. Brockx, and D. Mortelmans, "On the Validity of Student Evaluation of Teaching: The State of the Art", Review of Educational Research, Vol. 83, No. 4, pp. 598-642, 2013.https://doi.org/10.3102/0034654313496870
F. Zabaleta, "The Use and Misuse of Student Evaluations of Teaching", Teaching in Higher Education, Vol. 12, No. 1, pp. 55-76, 2007.https://doi.org/10.1080/13562510601102131
E. Balam and D. Shannon, "Student Ratings of College Teaching: A Comparison of Faculty and Their Students", Assessment & Evaluation in Higher Education, Vol. 35, No. 2, pp. 209-221, 2010.https://doi.org/10.1080/02602930902795901
H. W. Marsh, "SEEQ: A Reliable, Valid, and Useful Instrument for Collecting Students' Evaluations of University Teaching", British Journal of Educational Psychology, Vol. 52, No. 1, pp. 77-95, 1982.https://doi.org/10.1111/j.2044-8279.1982.tb02505.x
D. W. Jordan, "Re-Thinking Student Written Comments in Course Evaluations: Text Mining Unstructured Data for Program and Institutional Assessment", Doctoral Dissertation, California State University, Stanislaus, 2011.
T. Sliusarenko, L. Harder Clemmensen, and B. Ersbøll, "Text Mining in Students' Course Evaluations - Relationships between Open-ended Comments and Quantitative Scores", Proceedings of the 5th International Conference on Computer Supported Education (CSEDU-2013), pp. 564-573, 2013.
R. Menaha, R. Dhanaranjani, T. Rajalakshmi, and R. Yogarubini, "Student Feedback Mining System Using Sentiment Analysis", International Journal of Computer Applications Technology and Research, Vol. 6, No. 1, pp. 51-55, 2017.https://doi.org/10.7753/IJCATR0601.1009
E. Santhanam, B. Lynch, and J. Jones,"Making Sense of Student Feedback Using Text Analysis - Adapting and Expanding a Common Lexicon", Quality Assurance in Education, Vol. 26, No. 1, pp. 60-69, 2018.https://doi.org/10.1108/QAE-11-2016-0062
곽민호, 민혜리, 김리림, "토픽 모델링을 활용한 대학생의 서술형 강의평가 분석", 아시아교육연구, 제20권 제2호, pp.491-522, 2019.https://doi.org/10.15753/aje.2019.06.20.2.491
B. Leem, H. Kang, S. Eum, and K. Srijan, "Using Sentiment Analysis to Analyze the Feedback of Students with Open-Ended Questions", Studies in Humanities and Social Sciences, Vol. 63, No. 2, pp. 55-64, 2020.
임병학, "대학생 학습 불만족 결정 요인: 텍스트마이닝 접근법", 교육컨설팅코칭연구, 제5권 제2호, pp. 5-24, 2021.

Information

Publisher :The Society of Convergence Knowledge
Publisher(Ko) :융복합지식학회
Journal Title :The Society of Convergence Knowledge Transactions
Journal Title(Ko) :융복합지식학회논문지
Volume : 10
No :1
Pages :71-80
DOI :https://doi.org/10.22716/sckt.2022.10.1.008

[1] M. Sánchez, D. Carmen, M. Jose-Amelio, R. Rafael, and F. Luis, "Relationships among Relational Coordination Dimensions: Impact on the Quality of Education Online with a Structural Equations Model", Technological Forecasting and Social Change, Vol. 166, Article 120608, 2021.https://doi.org/10.1016/j.techfore.2021.120608

[2] M. Schneider and F. Preckel, "Variables Associated with Achievement in Higher Education: A Systematic Review of Meta-Analyses", Psychological Bulletin, Vol. 143, No. 6, pp. 565-600, 2017.https://doi.org/10.1037/bul0000098PMid:28333495

[3] T. Huybers, "Exploring the Use of Best-Worst Scaling to Elicit Course Experience Questionnaire Responses", Assessment & Evaluation in Higher Education, Vol. 42, No. 8, pp. 1306-1318, 2017.https://doi.org/10.1080/02602938.2016.1270256

[4] G. A. Boysen, "Student Evaluations of Teaching during the COVID-19 Pandemic", DOI:10.1037/stl0000222, 2020.

[5] S. L. Wright and M. A. Jenkins-Guarnieri, "Student Evaluations of Teaching: Combining the Meta-Analyses and Demonstrating Further Evidence for Effective Use", Assessment & Evaluation in Higher Education, Vol. 37, No. 6, pp. 683-699, 2012.https://doi.org/10.1080/02602938.2011.563279

[6] P. Spooren, B. Brockx, and D. Mortelmans, "On the Validity of Student Evaluation of Teaching: The State of the Art", Review of Educational Research, Vol. 83, No. 4, pp. 598-642, 2013.https://doi.org/10.3102/0034654313496870

[7] F. Zabaleta, "The Use and Misuse of Student Evaluations of Teaching", Teaching in Higher Education, Vol. 12, No. 1, pp. 55-76, 2007.https://doi.org/10.1080/13562510601102131

[8] E. Balam and D. Shannon, "Student Ratings of College Teaching: A Comparison of Faculty and Their Students", Assessment & Evaluation in Higher Education, Vol. 35, No. 2, pp. 209-221, 2010.https://doi.org/10.1080/02602930902795901

[9] H. W. Marsh, "SEEQ: A Reliable, Valid, and Useful Instrument for Collecting Students' Evaluations of University Teaching", British Journal of Educational Psychology, Vol. 52, No. 1, pp. 77-95, 1982.https://doi.org/10.1111/j.2044-8279.1982.tb02505.x

[10] D. W. Jordan, "Re-Thinking Student Written Comments in Course Evaluations: Text Mining Unstructured Data for Program and Institutional Assessment", Doctoral Dissertation, California State University, Stanislaus, 2011.

[11] T. Sliusarenko, L. Harder Clemmensen, and B. Ersbøll, "Text Mining in Students' Course Evaluations - Relationships between Open-ended Comments and Quantitative Scores", Proceedings of the 5th International Conference on Computer Supported Education (CSEDU-2013), pp. 564-573, 2013.

[12] R. Menaha, R. Dhanaranjani, T. Rajalakshmi, and R. Yogarubini, "Student Feedback Mining System Using Sentiment Analysis", International Journal of Computer Applications Technology and Research, Vol. 6, No. 1, pp. 51-55, 2017.https://doi.org/10.7753/IJCATR0601.1009

[13] E. Santhanam, B. Lynch, and J. Jones,"Making Sense of Student Feedback Using Text Analysis - Adapting and Expanding a Common Lexicon", Quality Assurance in Education, Vol. 26, No. 1, pp. 60-69, 2018.https://doi.org/10.1108/QAE-11-2016-0062

[14] 곽민호, 민혜리, 김리림, "토픽 모델링을 활용한 대학생의 서술형 강의평가 분석", 아시아교육연구, 제20권 제2호, pp.491-522, 2019.https://doi.org/10.15753/aje.2019.06.20.2.491

[15] B. Leem, H. Kang, S. Eum, and K. Srijan, "Using Sentiment Analysis to Analyze the Feedback of Students with Open-Ended Questions", Studies in Humanities and Social Sciences, Vol. 63, No. 2, pp. 55-64, 2020.

[16] 임병학, "대학생 학습 불만족 결정 요인: 텍스트마이닝 접근법", 교육컨설팅코칭연구, 제5권 제2호, pp. 5-24, 2021.

The Society of Convergence Knowledge Transactions ISSN:2287-8920(Print) 융복합지식학회논문지

All Issue