A corpus of spoken learner English in which every response is linked to the task that elicited it and to the scores it received.
574 first-year university students in Japan × 9 speaking tasks = 5,166 responses — approximately 290,000 words and 74 hours of speech, each response linked to its task, its ratings, and rich speaker metadata.
Download the corpus | Search online | Terms and conditions
1. Overview
The KIT Speaking Test Corpus (KISTEC) has been developed by Katsunori Kanzawa at Kyoto Institute of Technology (KIT) in Japan and his team. It comprises transcribed and tagged speech responses to the KIT Speaking Test, a computer-based semi-direct English speaking test administered to all first-year undergraduates at KIT.
What makes KISTEC distinctive is that speech, task, and assessment are integrated at the level of the individual response. Every response in the corpus is linked to:
- the full content of the task that elicited it, published on this website (Section 5.2);
- Task Achievement and Task Delivery scores assigned to that particular response by trained raters; and
- metadata on the speaker, including equated test scores, TOEIC L&R scores, and English learning experience.
This linkage allows questions that are difficult to address with existing corpora—for example, how the same learner’s performance varies across tasks, or which features of a response are associated with higher scores.
The KISTEC is available to everyone free of charge. To download the data, see Section 5; to use the Search Interface online, see Section 6. In either case, please review the terms and conditions in Section 9.
This project has received approval from the KIT Research Ethics Review Board. Only speech responses from examinees who consented to participate in the project have been included in the corpus.
2. Why KISTEC?
Spoken corpora of Japanese learners of English remain scarce. In addition, existing corpora rarely publish the tasks used to elicit the speech—often because they are built on commercial tests—and proficiency information is usually a single level per learner rather than a score for each response. Without the task, it is hard to tell whether an observed feature reflects the learner, the task, or both; without response-level scores, features of speech cannot be related to the evaluations the speech actually received.
KISTEC was designed to fill these gaps. Its four design principles are:
- speech from an entire cohort of first-year university students, not from self-selected volunteers;
- nine task types, all publicly available;
- human ratings attached to every single response; and
- integrated metadata, from learner attributes to external proficiency scores.
3. The KIT Speaking Test
3.1 A local test
The KIT Speaking Test is a local test, developed by faculty members at KIT and used within the university. Because the test is locally owned, its tasks, rating scales, and response data can be shared with the research community—something commercial testing programs rarely permit.
3.2 Examinees
The test was administered to all first-year undergraduates across the university’s six degree programs. Because the entire cohort took the test, the data reflect average students rather than volunteers, who tend to be more motivated or proficient than their peers.
Approximately 97% of the examinees were Japanese, and the remaining 3% were international students from Asian countries, including China, Malaysia, and South Korea; a nationality header allows analyses to be restricted to Japanese nationals. The examinees’ mean TOEIC L&R score was 563.54 (SD: 133.15, range: 195–985), corresponding to roughly B1 on the CEFR. Since Japanese learners’ speaking typically lags behind their listening and reading, their speaking proficiency is estimated to center on A2.
3.3 Test format and tasks
The KIT Speaking Test is a computer-based, semi-direct speaking test: questions are presented using audio and a PC display, and responses are recorded through a headset microphone. Each examinee took one of three parallel versions (Ver. 1–3) and responded to nine questions in three parts:
- Part 1: Examinees respond based on the pictures presented (no planning time).
- Part 2: Examinees listen to a conversation, then summarize it and express their own opinions about it (no planning time).
- Part 3: Examinees organize their thoughts and opinions and discuss them logically (with one-minute planning time).
The following table lists the test items for Ver. 1 to Ver. 3. The full version of the items can be downloaded in Section 5.2.
| Question NO. | Task type | Rehearsal time(sec.) | Response time(sec.) | Ver. 1 | Ver. 2 | Ver. 3 | |
|---|---|---|---|---|---|---|---|
| Part 1 | Q1 | Imagine | 0 | 45 | Imagine why the bicycle is left here | Imagine why the cup is left here. | Imagine why the bags are left here. |
| Q2 | Imagine | 0 | 45 | Imagine what the man is thinking. | Imagine what the man is thinking. | Imagine what the boy is thinking. | |
| Q3 | Compare | 0 | 45 | Which of these things would you buy for a 5-year-old child to play with, building blocks or a computer? Explain the reasons for your choice, comparing the advantages and disadvantages of both. | Which of these places would you prefer to live in, a large house or a high-rise apartment? Explain the reasons for your choice, comparing the advantages and disadvantages of both. | Where would you take your guests from abroad, to the countryside or to a big city? Explain the reasons for your choice, comparing the advantages and disadvantages of both. | |
| Part 2 | Q4 | Identify different values | 0 | 45 | How are Bill’s and Kyoko’s opinions different? | How are Bill’s and Mariam’s opinions different? | How are Kenji’s and Susan’s opinions different? |
| Q5 | Take position | 0 | 60 | Which way of thinking do you support? Explain your position and give reasons. | Which way of thinking do you support? Explain your position and give reasons. | Which way of thinking do you support? Explain your position and give reasons. | |
| Q6 | Identify problem | 0 | 45 | What is the problem Kate is facing? | What is the problem Susan is facing? | What is the problem Mariam is facing? | |
| Q7 | Problem solving | 0 | 60 | If you were Kate, what would you do to solve the problem? | If you were Susan, what would you do to solve the problem? | If you were Mariam, what would you do to solve the problem? | |
| Part 3 | Q8 | Plan and organise | 60 | 60 | You have been asked to make a promotional video of your university. Explain how you would organize it. | You want to establish a new cycling club in your university. Explain how you would organize it. | You want to organize a party for your high school classmates. Explain how you would organize it. |
| Q9 | Persuade | 60 | 60 | You are talking with friends from other countries about holidays. Explain to them why they should visit your country. | Some friends from another country are visiting you for one week. Choose a place for them to go and explain why they should go there. | You are talking with friends about hobbies. Explain to them why your hobby is interesting and why they should try it. |
3.4 Rating
Each question was rated by a pair of trained raters, one native and one non-native speaker of English, with different pairs assigned to different questions (18 raters in total). Responses were rated on two scales from 0 to 5: Task Achievement (TA), which evaluates how well the response accomplished the task, and Task Delivery (TD), which evaluates how effectively the response was delivered. When the two ratings differed by 2 points or more, an experienced senior rater reassessed the response; otherwise, the two scores were averaged. Raw scores therefore run from 0 to 5 in increments of 0.5.
| Score | Task Achievement (80% weighting) | Task Delivery (20% weighting) |
|---|---|---|
| 5 | The task is achieved, being developed with a satisfactory level of detail. | The delivery is mostly confident. Given time is well used without obvious problems with delivery such as intrusive pauses, hesitations, or repetitions. |
| 4 | The task is mostly achieved, with some supporting detail in places. | Given time is quite well used despite some problems with delivery such as slow rate of speech, pauses, hesitations, or repetitions. |
| 3 | The task is minimally or partially achieved, being supported with some basic detail. | General meaning comes across, but given time is not effectively used because of problems with delivery such as slow rate of speech, pauses, hesitations, or repetitions. |
| 2 | The task is addressed, but there is no or very little supporting detail. | The speaker keeps trying, but problems with delivery (e.g. slow rate of speech, pauses, hesitations or repetitions) allow a very limited amount of meaning to be conveyed. |
| 1 | The task remains essentially unachieved, though there may be some relevant words. | The speaker gives up trying, or problems with delivery (e.g. slow rate of speech, pauses, hesitations, repetitions) are fatal to meaning coming across. |
| 0 | There is no relevant contribution (e.g. content is entirely unconnected to topic). | The speaker does not start the task (e.g. s/he is silent, utters only fillers, or just says, ‘I don’t know’). |
After rating, examinee proficiency was estimated using Item Response Theory (IRT), yielding three equated scores that are comparable across test versions: Overall score (0–100), TA rank (1–5), and TD rank (1–5). Since the test places a strong emphasis on task achievement, the Overall score weights TA at 80% and TD at 20%. The equated scores are distributed as follows:
| Score type | Average (Standard deviation) | Max | Min |
|---|---|---|---|
| Overall score | 48.22 (10.45) | 90 | 21 |
| TA rank | 2.97 (1.42) | 5 | 1 |
| TD rank | 2.98 (1.40) | 5 | 1 |
The figure below plots the KIT Speaking Test scores on the horizontal axis and TOEIC L&R scores on the vertical axis. The correlation coefficient between the two is 0.59.
4. The corpus
4.1 Corpus size
The corpus comprises transcribed responses from 574 examinees, each of whom was allocated 7 minutes and 45 seconds of response time across the nine tasks. In total, the corpus contains approximately 290,000 words and 74 hours of speech:
| No. of examinees | No. of words | No. of words per examinee | Total response time | Words per minute | |
|---|---|---|---|---|---|
| Ver. 1 | 193 | 98,507 | 510.40 | 24:55:45 | 65.86 |
| Ver. 2 | 190 | 96,945 | 510.24 | 24:32:30 | 65.84 |
| Ver. 3 | 191 | 95,002 | 497.39 | 24:40:15 | 64.18 |
| Overall | 574 | 290,454 | 506.02 | 74:08:30 | 65.29 |
4.2 Corpus files
The corpus is distributed in plain-text (.txt) format in two versions: a tagged version, which contains header information (metadata) and speech tags, and an untagged version, which contains plain transcriptions only.
The file format and the transcription and tagging conventions follow those of the NICT JLE Corpus as closely as possible. Tools and scripts developed for that corpus can therefore be reused with KISTEC, and results can be compared across the two corpora. Detailed guidelines are provided in the transcription/tagging manual (Section 5.3).
4.2.1 Header information
| Header information | Meaning |
|---|---|
| <grade> | Year of study |
| <nationality> | Nationality |
| <sex> | Sex (1 = male, 2 = female) |
| <version> | Version of the speaking test taken (1–3) |
| <total_score> | Equated overall score on the speaking test (0–100) |
| <ta_rank> | Equated TA rank (1–5) |
| <td_rank> | Equated TD rank (1–5) |
| <ta1-9_score> | Raw TA score of the response to each of the nine tasks (0–5); not equated |
| <td1-9_score> | Raw TD score of the response to each of the nine tasks (0–5); not equated |
| <toeic_score> | TOEIC L&R total score (10–990) |
| <toeic_rscore> | TOEIC Reading score (5–495) |
| <toeic_lscore> | TOEIC Listening score (5–495) |
| <experience1> | The following are the responses to the survey by IIBC (a TOEIC developer in Japan). How many years have you studied English? A=4 years or fewer/B=4–6 years/C=6–10 years/D=10 years or more/Blank=No answer |
| <experience2> | Which of the following language skill(s) is/are the most important for you? A=Listening/B=Reading/C=Speaking/D=Writing/E= Listening and Speaking/F= Reading and Writing /G=All of them/Blank=No answer |
| <experience3> | What percentage do you use English in your daily life? A=0%/B=1–10%/C=11–20%/D=21–50%/E=51–100%/Blank=No answer |
| <experience4> | Which of the following language skills do you use the most? A=Listening/B=Reading/C=Speaking/D=Writing/E= Listening and Speaking/F= Reading and Writing/G=All of them/Blank=No answer |
| <experience5> | How often does your lack of English proficiency prevent your communication? A=hardly ever/B=occasionally/C=sometimes/D=often/E=usually/ Blank=No answer |
| <experience6> | Have you ever stayed in a country where English is the primary language? A=No/B=6 months or fewer/C=6–12 months /D=1–2 years/E=1 or more years/Blank=No answer |
| <experience7> | What was the purpose of your stay in a country where English is the primary language? A=Study (excluding learning English)/B=To participate in an English language learning program/C=Travel (excluding business)/D=Business/E=Others/Blank=No answer |
4.2.2 Speech tags
| Tags | Meaning |
|---|---|
| <F> </F> | Filler |
| <R> </R> | Repetition |
| <R?> </R?> | Repetition (transcriber not confident in listening) |
| <SC> </SC> | Self-correction |
| <SC?> </SC?> | Self-correction (transcriber not confident in listening) |
| <TO> </TO> | Timeout (speech cut off by the response time limit) |
| <RE> </RE> | Recording error |
| <nvs> </nvs> | Nonverbal sound (laughter, sigh, cough, yawn) |
| <CO> </CO> | Cutoff (speech suspended by the speaker) |
| <?> </?> | Not confident in listening |
| <??> </??> | Completely unintelligible |
| <H pn=“X”> </H> | Proper nouns, discriminatory terms, and others (content removed) |
| <JP> </JP> | Japanese |
| <.> </.> | Pause (2–3 s) |
| <..> </..> | Pause (3 s or more) |
| <laughter> </laughter> | Laughing while speaking |
4.3 Notes on using the data
- The speech was produced under test conditions: time-limited monologues addressed to a computer. Features such as pauses and fillers may pattern differently in dialogue or everyday conversation.
- The examinees are a single cohort at one university, with speaking proficiency centered on CEFR A2. On its own, the corpus is not suited to studying highly proficient speakers, and findings should not be generalized to all Japanese learners of English.
5. Downloads
5.1 Corpus files
Both the tagged and untagged versions of the corpus are available for download in zip format, secured with a password. To obtain the password, please fill out the registration form; the password will be sent to the registered email address.
Notes:
- The file ‘316’ has been removed from the corpus due to silence (recording error).
- The untagged version excludes header information; please refer to the tagged version for this data.
5.2 Test tasks
The complete version of the test tasks (paper format) can be downloaded below.
Note: In Part 2 of the actual test, the conversation scripts are not presented to the examinees; they only listen to the audio of the conversation.
5.3 Transcription/tagging manual
The transcription/tagging manual can be downloaded below.
6. Search Interface (beta)
The Search Interface for the KISTEC, developed and owned primarily by Yusuke Tanaka at Kumamoto Gakuen University, allows the corpus to be searched online without downloading:
It supports KWIC search, collocation analysis, wordlist generation, characteristic-word extraction, and N-gram analysis; results can be filtered by task, gender, and test scores, and exported as CSV files.
The interface is currently in beta. Reports of bugs, comments on usability, and feature requests are all welcome: please contact Yusuke Tanaka at yusuke.tanaka.07[at mark]gmail.com.
7. Our team
Researchers
- Katsunori Kanzawa (Kyoto Institute of Technology)
- Yuichiro Kobayashi (Nihon University)
- Jaeho Lee (Waseda University）
- Haruhiko Mitsunaga (Nagoya University)
- Masayuki Mori (Kyoto Institute of Technology)
- Yusuke Tanaka (Kumamoto Gakuen University)
- Taishi Chika (University of Toyama)
Transcribers (Graduate students)
- Taishi Chika (Kyoto University), 2019–2020
- Mitsuyuki Kato (Kyoto Prefectural University), 2019
- Takumi Kitahara (Kyoto University), 2019–2020
- Toshifumi Taniwaki（Ritsumeikan University), 2019
- Taku Motozawa (Kyoto University), 2020
Note: Affiliations are as of the time of the work.
8. Grants
This project has received support from the following grants:
- JSPS KAKENHI No.19K00849 “Corpus development based on speech responses by Japanese university students to the computer-based English speaking test” (AY 2019–2021)
- JSPS KAKENHI No.22K00736 “Exploring Task Achievement for improving scoring efficiency in English speaking tests” (AY 2022–2025)
9. Terms and conditions
By downloading the KISTEC or using the Search Interface for the KISTEC, you agree to the following terms and conditions. Any violations will result in a warning. If you do not comply with this warning in a timely manner, legal action may be taken. Please be aware that these terms and conditions may be updated without prior notice, and the most recent version will apply.
- Below is an example of how to reference the KISTEC in APA 7th edition style. When you download and use the KISTEC, please include (1) in your reference list. When you use the Search Interface for the KISTEC, please include both (1) and (2) in your reference list.
- (1) Kanzawa, K., Kobayashi, Y., Lee, J., Mitsunaga, H., Mori, M., Tanaka, Y., & Chika, T. (n.d.). The KIT Speaking Test Corpus (KISTEC). https://kitstcorpus.jp
- (2) Tanaka, Y., Setoguchi, A., Chika, T., & Kanzawa, K. (n.d.). Search Interface for the KIT Speaking Test Corpus. https://www.kistecsearch.org/
- Use the corpus at your own risk. The KISTEC team is not liable for any damage, loss, or other disadvantages that may arise from its use.
- In principle, original speech responses to the KIT Speaking Test are included in this corpus without alteration. The KISTEC team is not responsible for the content of these utterances, which may contain inappropriate material.
- Despite the KISTEC team’s best efforts to build the corpus, it may still contain errors, for which the team assumes no responsibility. If you encounter any errors, please contact the project leader, Katsunori Kanzawa, at kanzawa[at mark]kit.ac.jp.
- The corpus and website may be updated or removed without notice.
- You are allowed to modify and redistribute the corpus; however, please indicate which parts have been altered.
10. Acknowledgments
We would like to express our gratitude to everyone at Kyoto Institute of Technology who was involved in the development and implementation of the KIT Speaking Test, including English faculty members, faculty and staff members of the Center for ICT Services, and all others who contributed to this project.