Assessing English Speaking and Listening Skills with the Mobile Application Telegram

Agus Wardhono; Stephen Spanos

Agus Wardhono Universitas PGRI Ronggolawe Tuban, Indonesia
Stephen Spanos Middlebury Institute of International Studies at Monterey, USA

Keywords: Assessment, Speaking, Listening, Telegram

Abstract

This paper describes an English as a Foreign Language (EFL) speaking and listening test designed and piloted through the mobile application Telegram. The test was designed to diagnose the speaking and listening skills of first-year students at the Indonesian university UNIROW Tuban before they arrive on campus. The mobile delivery of the test, which can be accessed by smart phone and computer, was aimed to evaluate if learning apps can increase communicative learner interaction with authentic materials in English. The mobile test takes advantage of the vast amount of multimedia that can be transferred online, such as video and podcasts. Test-takers respond to Speaking Section prompts by recording themselves and answer Listening Section questions by clicking on the correct multiple choice options. All test-taker responses are sent to the administrator of the test, created through an inline Telegram bot. After piloting the test with students in a first-year conversation class at UNIROW Tuban, the test underwent a series of analyses, including item facility, item discrimination, split-half reliability, inter-rater reliability, and subtest relationships. These analyses are important for verifying the testâ€™s validity, reliability, and practicality. Overall, the test seems to be an important tool not only for diagnosing students listening and speaking needs, but also to increase interest in learning and practicing conversational English. Although the test was designed for one university in particular, it (along with variations) can be used in similar contexts throughout the world.

References

Alderson, J. C. (2005). Diagnosing foreign language proficiency: The interface between learning and assessment. London, UK: Continuum International.

Alderson, J.C., Clapham, C., & Wall, D. (1995). Language test construction and evaluation. Cambridge, UK: Cambridge University Press

Bachman, L. (1990). Fundamental considerations in language testing. New York, NY: Oxford University Press.

Bachman, L., & Palmer, A. (2010). Language assessment in practice. New York, NY: Oxford University Press.

Burston, J. (2013). Mobile-assisted language learning: A selected annotated bibliography of implementation studies 1994â€“2012. Language Learning & Technology, 17(3): 157â€“225. Retrieved from http://llt.msu.edu/issues/october2013/burston.pdf

Burston, J. (2014). The reality of MALL: Still on the fringes. CALICO Journal, 31(1): 103â€“125.

Burston, J. (2015). Twenty years of MALL project implementation: A meta-analysis of learning outcomes. ReCALL, 27(1): 4â€“20.

Bailey, K. M., & Curtis, A. (2015). Learning about language assessment: Dilemmas, Decisions, and Directions (2nd ed.). Boston, MA: National Geographic Learning

Brown, J. D. (2005). Testing in language programs: A comprehensive guide to English language assessment. White Plains, NY: Pearson Education.

Brown, G., & Yule, G., (1989). Discourse analysis. Cambridge, UK: Cambridge University Press.

Buck, G. (2001). Assessing listening. Cambridge, UK: Cambridge University Press.

Chase, C. I. (1999). Contemporary assessment for educators. New York, NY: Longman.

Cumming, A., & Berwick, R., (Eds.) (1996). Validation in language testing. Clevedon, England: Multilingual Matters, Ltd.

Deng, H., & Shao, Y. (2011) Self-directed English vocabulary learning with a mobile application in everyday context. Paper presented at the Proceedings 10th World Conference on Mobile and Contextual Learning (mLearn), Beijing, China: Beijing Normal University.

Douglas, D. (2000). Assessing languages for specific purposes. New York, NY: Cambridge University Press.

Guo, Hui (2015) Analysing and Evaluating Current Mobile Applications for Learning English Speaking. Birkbeck: University of London.

Haertel, Edward H. (2013) Reliability and Validity of Inferences about Teachers based on Student Test Scores. Princeton: ETS Research and Development.

Hatch, E. M. & Farhady, H. (1982). Research design and statistics for applied linguistics.
Rowley, MA: Newbury House.

Iwashita, N., Brown, A., McNamara, T., & Oâ€™Hagan, S. (2008). Assessed levels of second
language speaking proficiency: How distinct? Applied Linguistics, 29, 24-49.

Kukulska-Hulme, A., & Shield, L. (2008). An overview of mobile assisted language learning: From content delivery to supported collaboration and interaction. ReCALL, 20(3): 271-289.

Lixun, Wang (2017) Public Lecture Series 2017: The English You Didnâ€™t Learn in School V, Mobile Assisted Language Learning. Retrieved from https://www.ied.edu.hk/ele/pls/spring 2017/seminar4.pdf on June 8th 2017.

Mertler, C. A. (2001). Designing scoring rubrics for your classroom. Practical Assessment, Research & Evaluation, 7(25). Retrieved from http://PAREonline.netgetvn.aspv=7&n=25

Mertler, C. A. (2003) Classroom assessment: A practical guide for educators. Los Angeles, CA: Pyrczak Publishers.

Nadia, Maarouf (2013) The Importance of Continuous Assessment in Improving ESP Studentsâ€™ Performance. Kasdi Merbah Ouargla University.

Nitko, A. J. (2001). Educational assessment of students (3rd ed.). Upper Saddle River, NJ:
Merrill.

Oller, J. W. (1979). Language tests at school. London: Longman Group.

Richards, J. C., Platt, J., & Platt, H. (1992). Longman dictionary of language teaching and
applied linguistics. Essex, UK: Longman Group.

SÃ rosdy, J. and Bence, T,F. Vadnay, M (2006) Applied linguistics 1 for BA students in English. Cambridge: Cambridge university press.

Swain, M. (1984). Large-scale communicative language testing: A case study. In S. J. Savignon & M. Berns (Eds.), Initiatives in communicative language teaching (pp. 185â€“201). Reading, MA: Addison-Wesley.

Understanding your TOEFL iBT test scores. (2017). Retrieved from
https://www.ets.org/toefl/ibt/scores/understand/

Vitiene, N., & Miciuliene, R. (2008). Application of criteria-referenced assessment and
qualitative feedback to develop foreign language speaking skills in the context of e- teaching/learning. Quality Of Higher Education, 5(7),:152-168.

Wesche, M. (1983). Communicative testing in a second language. Modern Language Journal, 67(1): 41-55.