1. Outline
The KIT Speaking Test Corpus (KISTEC) has been developed by Katsunori Kanzawa at Kyoto Institute of Technology (KIT) in Japan and his team. It comprises transcribed speech responses, along with specific speech tags, from the KIT Speaking Test—a computer-based semi-direct English speaking test administered to all first-year undergraduates at KIT.
The KISTEC is available for free to everyone. For downloading data, please refer to Section 5. If you are using the Search Interface online, please refer to Section 6. In either case, kindly review the terms and conditions outlined in Section 9
This project has received approval from the KIT Research Ethics Review Board. Only speech responses from examinees who consented to participate in the project have been included in the corpus.
2. Background
English education in Japan is increasingly prioritizing communication skills. To support evidence-based English education in this context, it is vital to assess the current speaking abilities of Japanese learners of English and to understand the characteristics of their developmental processes.
To this end, KIT developed an original English speaking test and regularly administered it to all first-year undergraduates. This project aims to create a comprehensive corpus of students’ speech responses, allowing for analysis that reveals the speaking abilities of Japanese learners of English. We believe that the insights gained from analyzing the corpus can make a significant contribution to a wide range of fields, including English education, language testing, and NLP research. Furthermore, we anticipate that its applications may extend to areas we have not yet envisioned.
3. Key features
(1) The KISTEC provide speech data from Japanese learners of English.
There are relatively large corpora that focus on the spoken language of Japanese learners of English, such as the NICT Japanese Learners of English (JLE) Corpus and the International Corpus Network of Asian Learners of English (ICNALE). However, the limited number of such resources remains a challenge. The KISTEC provides speech data from Japanese university students. Analyzing this corpus would allow us to answer questions such as, “What is the level of Japanese university students’ speaking proficiency?” or “What can they do and what can’t they do?”
(2) Speech tags are provided.
The speech transcriptions include specific tags that represent the characteristics of the utterances. Analyzing the utterances using these tags would allow us to answer questions such as, “What do the examinees struggle with?” or “What communication strategies do they use to overcome limitations in grammar and/or vocabulary?”
(3) Metadata is attached.
The KISTEC includes metadata—such as examinees’ attributes and their English learning experiences—as header information. Analyzing the relationship between this metadata and the speech enables us to address questions such as, “Are there specific attributes or learning experiences that impact students’ performance?”
(4) Scores from the KIT Speaking Test and TOEIC L&R are attached.
Additionally, the header information includes scores from the KIT Speaking Test and TOEIC L&R. For the KIT Speaking Test scores, not only is an overall score provided, but scores for each response are also included. The feature of having scores for each response is unique compared to existing corpora, allowing for more elaborate analysis.
Analyzing the relationship between the test scores and the speech can help us explore questions such as, “Which language features significantly contribute to test scores?” and “Are other skills, such as listening and reading, related to speaking ability?” Additionally, the speaking test scores can serve as indicators of students’ approximate speaking proficiency levels. This prompts further investigation into questions like, “How do students’ performances vary across different proficiency levels?” and “How does their performance evolve as their proficiency increases?”
(5) Test tasks (questions) are publicly available.
The test tasks of the KIT Speaking Test are also available on this website. Analyzing the KISTEC alongside the test tasks would allow us to answer questions such as, “How do test tasks affect examinees’ performance?” or “What tasks are effective in eliciting examinees’ performance?”
(6) The KISTEC is compatible with the NICT JLE Corpus.
The file format and transcription/tagging methods of this corpus conform, as much as possible, to those of the NICT JLE Corpus. This allows us to repurpose analysis programs from the NICT JLE Corpus, facilitating easier analysis. Additionally, this enables us to compare data with the NICT JLE Corpus and answer questions such as, “How consistent are the analysis results across the two corpora?” or “What causes any observed similarities?”
4. Overview of the KISTEC
4.1 Target data
Test administration
The KISTEC is based on the KIT Speaking Test, which is a computer-based semi-direct English speaking test. Questions are presented using audio and a PC display, and examinees’ responses are recorded through their headset microphones. There are three versions of the test, which we have tentatively named Ver. 1, Ver. 2, and Ver. 3, and one of them was administered to each student.
Examinees
All first-year graduates of KIT took the test. Approximately 97% of the examinees were Japanese, while the remaining 3% were international students from Asian countries, including China, Malaysia, and South Korea. Their average TOEIC L&R score was 563.6 (Max: 985, Min: 195, SD: 133.15), indicating a proficiency level around lower B1 in Common European Framework of Reference for Languages (CEFR).
Test tasks
The KIT Speaking Test consists of three parts with a total of nine questions. A summary of each part is as follows:
- Part 1: Examinees respond based on the pictures presented (no planning time).
- Part 2: Examinees listen to a conversation, then summarize it and express their own opinions about it (no planning time).
- Part 3: Examinees organize their thoughts and opinions and discuss them logically (with one-minute planning time).
The following table provides a quick list of the test items for Ver. 1 to Ver. 3. The full version of the items can be downloaded in Section 5.2.
| Question NO. | Task type | Rehearsal time(sec.) | Response time(sec.) | Ver. 1 | Ver. 2 | Ver. 3 | |
|---|---|---|---|---|---|---|---|
| Part 1 | Q1 | Imagine | 0 | 45 | Imagine why the bicycle is left here | Imagine why the cup is left here. | Imagine why the bags are left here. |
| Q2 | Imagine | 0 | 45 | Imagine what the man is thinking. | Imagine what the man is thinking. | Imagine what the boy is thinking. | |
| Q3 | Compare | 0 | 45 | Which of these things would you buy for a 5-year-old child to play with, building blocks or a computer? Explain the reasons for your choice, comparing the advantages and disadvantages of both. | Which of these places would you prefer to live in, a large house or a high-rise apartment? Explain the reasons for your choice, comparing the advantages and disadvantages of both. | Where would you take your guests from abroad, to the countryside or to a big city? Explain the reasons for your choice, comparing the advantages and disadvantages of both. | |
| Part 2 | Q4 | Identify different values | 0 | 45 | How are Bill’s and Kyoko’s opinions different? | How are Bill’s and Mariam’s opinions different? | How are Kenji’s and Susan’s opinions different? |
| Q5 | Take position | 0 | 60 | Which way of thinking do you support? Explain your position and give reasons. | Which way of thinking do you support? Explain your position and give reasons. | Which way of thinking do you support? Explain your position and give reasons. | |
| Q6 | Identify problem | 0 | 45 | What is the problem Kate is facing? | What is the problem Susan is facing? | What is the problem Mariam is facing? | |
| Q7 | Problem solving | 0 | 60 | If you were Kate, what would you do to solve the problem? | If you were Susan, what would you do to solve the problem? | If you were Mariam, what would you do to solve the problem? | |
| Part 3 | Q8 | Plan and organise | 60 | 60 | You have been asked to make a promotional video of your university. Explain how you would organize it. | You want to establish a new cycling club in your university. Explain how you would organize it. | You want to organize a party for your high school classmates. Explain how you would organize it. |
| Q9 | Persuade | 60 | 60 | You are talking with friends from other countries about holidays. Explain to them why they should visit your country. | Some friends from another country are visiting you for one week. Choose a place for them to go and explain why they should go there. | You are talking with friends about hobbies. Explain to them why your hobby is interesting and why they should try it. |
Rating
Raters
For each question, a pair consisting of one native speaker (NS) and one non-native speaker (NNS)—an English instructor from the Philippines with a TESOL qualification—evaluated the responses. Different raters were assigned to each question, totaling 18 raters (9 NSs and 9 NNSs). The raters underwent sufficient training beforehand and only began scoring once they reached a certain standard.
Rating scales
The rating scales included two dimensions: Task Achievement (TA), which evaluates how well the examinee accomplished the task set by the question, and Task Delivery (TD), which assesses how effectively the examinee conveyed their speech. Both dimensions were rated on a scale of 0 to 5. If there was a difference of 2 points or more between the two ratings, a senior rater (an NS experienced in scoring) conducted a reassessment. In cases where the difference was within 1 point, the average of the two scores was taken. That means the raw scores assigned to each response range from 0 to 5 in increments of 0.5, resulting in a total of 11 levels.
| Score | Task Achievement (80% weighting) | Task Delivery (20% weighting) |
|---|---|---|
| 5 | The task is achieved, being developed with a satisfactory level of detail. | The delivery is mostly confident. Given time is well used without obvious problems with delivery such as intrusive pauses, hesitations, or repetitions. |
| 4 | The task is mostly achieved, with some supporting detail in places. | Given time is quite well used despite some problems with delivery such as slow rate of speech, pauses, hesitations, or repetitions. |
| 3 | The task is minimally or partially achieved, being supported with some basic detail. | General meaning comes across, but given time is not effectively used because of problems with delivery such as slow rate of speech, pauses, hesitations, or repetitions. |
| 2 | The task is addressed, but there is no or very little supporting detail. | The speaker keeps trying, but problems with delivery (e.g. slow rate of speech, pauses, hesitations or repetitions) allow a very limited amount of meaning to be conveyed. |
| 1 | The task remains essentially unachieved, though there may be some relevant words. | The speaker gives up trying, or problems with delivery (e.g. slow rate of speech, pauses, hesitations, repetitions) are fatal to meaning coming across. |
| 0 | There is no relevant contribution (e.g. content is entirely unconnected to topic). | The speaker does not start the task (e.g. s/he is silent, utters only fillers, or just says, ‘I don’t know’). |
Equating
After rating, the proficiency levels of the examinees were calculated using Item Response Theory (IRT), resulting in three standardised scores: Overall score (0–100), TA rank (0–5), and TD rank (0–5). These scores are comparable across different test versions. Since the KIT Speaking Test places a strong emphasis on task achievement, the Overall score was weighted with 80% for TA and 20% for TD.
Score distribution
The scores for the KIT Speaking Test are shown in the table below:
| Score type | Average (Standard deviation) | Max | Min |
|---|---|---|---|
| Overall score | 48.22 (10.45) | 90 | 20 |
| TA rank | 2.97 (1.42) | 5 | 1 |
| TD rank | 2.98 (1.40) | 5 | 1 |
The figure below plots the KIT Speaking Test scores on the horizontal axis and TOEIC L&R scores on the vertical axis. The correlation coefficient between the two is 0.59.
4.2 Corpus size
The corpus comprises transcribed data from 574 examinees. Each examinee was allocated 7 minutes and 45 seconds to complete nine tasks, resulting in a total of approximately 75 hours of recorded data. The total word count of the transcriptions is around 300,000 words.
Below is a summary table illustrating the corpus size:
| No. of examinees | No. of words | No. of words per examinee | Total response time | Words per minute | |
|---|---|---|---|---|---|
| Ver. 1 | 193 | 98,507 | 510.40 | 24:55:45 | 65.86 |
| Ver. 2 | 190 | 96,945 | 510.24 | 24:32:30 | 65.84 |
| Ver. 3 | 191 | 95,002 | 497.39 | 24:40:15 | 64.18 |
| Overall | 574 | 290,454 | 506.02 | 74:08:30 | 65.29 |
4.3 Corpus files
We offer two types of the corpus: a tagged version and an untagged version, both available in .txt format.
The tagged version features a header information containing metadata, including examinee attributes, English learning experiences, and test scores, followed by a section with transcribed speech responses that incorporate specific speech tags. For detailed information on the header information, please refer to Section 4.3.1, and for the speech tag, consult Section 4.3.2.
Both the file format and the transcription and tagging methods closely follow the standards of the NICT JLE Corpus. Detailed guidelines can be found in the transcription/tagging manual in Section 5.3.
The untagged version omits both header information and speech tags. Therefore, please refer to the tagged version for this data.
4.3.1 Header information
| Header information | Meaning |
|---|---|
| <grade> | University grade |
| <nationality> | Nationality |
| <sex> | Sex (1 for male, 2 for female) |
| <version> | Version of the speaking test |
| <total_score> | Score in the speaking test (0–100) |
| <ta_rank> | Task Achievement rank in the speaking test (0–5) |
| <td_rank> | Task Delivery rank in the speaking test (0–5) |
| <ta1-9_score> | Task Achievement raw score for each response (0-5*) *Average of two raters |
| <td1-9_score> | Task Delivery raw score for each response (0-5*) *Average of two raters |
| <toeic_score> | TOEIC score（10–990） |
| <toeic_rscore> | Score of Reading Section in TOEIC（5–495） |
| <toeic_lscore> | Score of Listening Section in TOEIC（5–495） |
| <experience1> | The following are the responses to the survey by IIBC (a TOEIC developer in Japan). How many years have you studied English? A=4 years or fewer/B=4–6 years/C=6–10 years/D=10 years or more/Blank=No answer |
| <experience2> | Which of the following language skill(s) is/are the most important for you? A=Listening/B=Reading/C=Speaking/D=Writing/E= Listening and Speaking/F= Reading and Writing /G=All of them/Blank=No answer |
| <experience3> | What percentage do you use English in your daily life? A=0%/B=1–10%/C=11–20%/D=21–50%/E=51–100%/Blank=No answer |
| <experience4> | Which of the following language skills do you use the most? A=Listening/B=Reading/C=Speaking/D=Writing/E= Listening and Speaking/F= Reading and Writing/G=All of them/Blank=No answer |
| <experience5> | How often does your lack of English proficiency prevent your communication? A=hardly ever/B=occasionally/C=sometimes/D=often/E=usually/ Blank=No answer |
| <experience6> | Have you ever stayed in a country where English is the primary language? A=No/B=6 months or fewer/C=6–12 months /D=1–2 years/E=1 or more years/Blank=No answer |
| <experience7> | What was the purpose of your stay in a country where English is the primary language? A=Study (excluding learning English)/B=To participate in an English language learning program/C=Travel (excluding business)/D=Business/E=Others/Blank=No answer |
4.3.2 Speech tags
| Tags | Meaning |
|---|---|
| <F> </F> | Filler |
| <R> </R> | Repetition |
| <R?> </R?> | Repetition (not confident in listening) |
| <SC> </SC> | Self-correction |
| <SC?> </SC?> | Self-correction (not confident in listening) |
| <TO> </TO> | Timeout |
| <RE> </RE> | Recording error |
| <nvs> </nvs> | Non-verbal sound |
| <CO> </CO> | Cutoff (suspended speech) |
| <?> </?> | Not confident in listening |
| <??> </??> | Completely inaudible |
| <H pn=“X”> </H> | Proper nouns, discriminatory terms, etc. |
| <JP> </JP> | Japanese |
| <.> </.> | Pause (2–3 sec.) |
| <..> </..> | Pause (3 or more sec.) |
| <laughter> </laughter> | Laughing while speaking |
5. Downloads
5.1 Corpus files
Both the tagged and untagged versions of the corpus are available for download in zip format, secured with a password. To obtain the password, please fill out the registration form. This way, the password will be sent to the registered email address.
Notes:
- The file ‘316’ has been removed from the corpus due to silence (recording error).
- The untagged version excludes header information; please refer to the tagged version for this data.
5.2 Test tasks
The complete version of the test tasks for the paper format can be downloaded from the link below.
Note: In Part 2 of the actual test, the conversation scripts are not presented to the examinees; they only listen to the audio of the conversation.
5.3 Transcription/tagging manual
The transcription/tagging manual can be downloaded from below.
- Ver. 1 (20/04/22 updated)
6. Search Interface for the KISTEC – Beta Version Now Available
The Search Interface for the KIT Speaking Test Corpus has been developed and owned primarily by Yusuke Tanaka at Kumamoto Gakuen University in Japan.
Announcement: Beta Version Release
We are excited to announce the beta release of the Search Interface for the KIT Speaking Test Corpus!
This beta version is part of our ongoing development and is being released to collect feedback from real users before the official launch. While most core features are functional, you may encounter some bugs or incomplete elements during use.
We would greatly appreciate your feedback to help us improve the app.
How You Can Help:
- Report any bugs or errors you encounter
- Share your thoughts on the usability and design
- Let us know any suggestions or feature requests
Feedback Contact:
Please send your feedback to: Yusuke Tanaka at yusuke.tanaka.07[at mark]gmail.com
Your input is invaluable in helping us create a better user experience. Thank you for your support and for being part of our beta testing community!
7. Our team
Researchers
- Katsunori Kanzawa (Kyoto Institute of Technology)
- Yuichiro Kobayashi (Nihon University)
- Jaeho Lee (Waseda University）
- Haruhiko Mitsunaga (Nagoya University)
- Masayuki Mori (Kyoto Institute of Technology)
- Yusuke Tanaka (Kumamoto Gakuen University)
- Taishi Chika (Kyoto University)
Transcribers (Graduate students)
- Taishi Chika (Kyoto University), 2019–2020
- Mitsuyuki Kato (Kyoto Prefectural University), 2019
- Takumi Kitahara (Kyoto University), 2019–2020
- Toshifumi Taniwaki（Ritsumeikan University), 2019
- Taku Motozawa (Kyoto University), 2020
Note: Affiliations are as of the time of the work.
8. Grants
This project has received support from the following grants:
- JSPS KAKENHI No.19K00849 “Corpus development based on speech responses by Japanese university students to the computer-based English speaking test” (AY 2019–2021)
- JSPS KAKENHI No.22K00736 “Exploring Task Achievement for improving scoring efficiency in English speaking tests” (AY 2022–2025)
9. Terms and conditions
By downloading the KISTEC or using the Search Interface for the KISTEC, you agree to the following terms and conditions. Any violations will result in a warning. If you do not comply with this warning in a timely manner, legal action may be taken. Please be aware that these terms and conditions may be updated without prior notice, and the most recent version will apply.
- Below is an example of how to reference the KISTEC in APA 7th edition style. When you download and use the KISTEC, please include (1) in your reference list. When you use the Search Interface for the KISTEC, please include both (1) and (2) in your reference list.
- (1) Kanzawa, K., Kobayashi, Y., Lee, J., Mitsunaga, H., Mori, M., Tanaka, Y., & Chika, T. (n.d.). The KIT Speaking Test Corpus (KISTEC). https://kitstcorpus.jp
- (2) Tanaka, Y., Setoguchi, A., Chika, T., & Kanzawa, K. (n.d.). Search Interface for the KIT Speaking Test Corpus. https://www.kistecsearch.org/
- Use the corpus at your own risk. The KISTEC team is not liable for any damage, loss, or other disadvantages that may arise from its use.
- In principle, original speech responses to the KIT Speaking Test are included in this corpus without alteration. The KISTEC team is not responsible for the content of these utterances, which may contain inappropriate material.
- Despite the KISTEC team’s best efforts to build the corpus, it may still contain errors, for which the team assumes no responsibility. If you encounter any errors, please contact the project leader, Katsunori Kanzawa, at kanzawa[at mark]kit.ac.jp.
- The corpus and website may be updated or removed without notice.
- You are allowed to modify and redistribute the corpus; however, please indicate which parts have been altered.
10. Acknowledgments
The development and implementation of the KIT Speaking Test were carried out by all English faculty members at Kyoto Institute of Technology. We extend our gratitude to everyone involved.