Home Previous Columns Book Reviews Other Articles Reader Response Links






The Evaluation of Computer Assisted Kanji Study Materials - Lexikan 2.0
by Howard Brown
M.A. Applied Linguistics, M.Ed Language Teaching --University of Southern Queensland

Introduction

The description and evaluation of CALL resources is an important and complex undertaking. There are a wide variety of issues that need to be considered ranging from technical issues of software compatibility to pedagogical issues of learner fit. Without a guiding principle, CALL resource evaluation could easily become an arbitrary and not very valuable endeavor. For the purposes of this paper, the software package in question will be described in terms of Hubbardfs (1992, cited in Levy, 1997 and Healy, 1996) application of Richards and Rodgersf (1987, 2001) approach design and procedure framework to a CALL context. The software will then be evaluated using the 6-point breakdown of criteria proposed by Chapelle (2001). She advocates the judgmental evaluation of CALL resources based on language learning potential, learner fit, meaning focus, authenticity, positive impact, and practicality.

Software Description

Lexikan 2.0 (See Appendix A for availability information and system requirements) fits into what Levy (1997) calls the ecomputer as tutorf model. According to the developersf promotional material, it is designed to help learners develop proficiency in reading and writing Japanese characters, both kana and kanji (Lexikan Software Company, 2003). The program is based on a series of flashcards containing the 1945 standard kanji required for secondary school level reading proficiency in Japan. The flashcards can be presented in random groups as well as being grouped by grade level or by content domain such as animal names, geographic terms, travel verbs, and so on. The cards can be simply viewed, or can be accompanied by one of two quiz formats -- multiple choice and fill in the blank. In both cases, the users have to provide the reading of the character. Following the question, there is a screen of detailed information about the kanji including a variety of readings, some example words using the kanji and references to popular kanji texts.

The program also has a writing element. The learners can use the mouse to write kanji characters on the screen. The program can recognize and correct both stroke placement and stroke order. The learner also has help options for writing. A prompt for the next stroke, an outline of the entire character, and a full animation of the character being drawn are available.

Looking at the approach level of analysis, Richards and Rodgers (1982) suggest that one needs to consider the view of language and language learning that underlie the design decisions. To this Hubbard (1992) adds the CALL specific aspect of computer use. What strengths of the computer are being taken advantage of? At this level we see that Lexikan seems to be based on the view that learning to read and write is a question of learning to recognize and produce discrete elements. The program presents kanji characters in isolation for recognition and reproduction without significant context. The kanji are used in compounds but this is still at the single word level. There is also a reliance on the view that language learning is a question of habit formation. As the companyfs own promotional materials say gWith repetition, your mastery of both writing and reading is significantly enhancedh (Lexikan Software Company, 2003, para. 3). The computer is used because of its ability to repeat, recognize errors, and randomize. The multimedia potential of the computer is not taken advantage of.

The design level concerns itself with learner variables, content issues and questions of program control. The program seems to be designed for independent learners. The promotional materials do not make any mention of potential classroom use. Rather, the product is presented as something to use at home in addition to or instead of classroom work. It seems to be based on a visual learning preference though some kinesthetic activity takes place through the use of the mouse to write kanji. The level of difficulty is variable. The program contains all of the kanji covered in the public school curriculum and thus the difficulty can be controlled. At a basic level, the learner has a fair level of control over the program. Options for repetition, type of error correction, and type of hints are all easily available and fairly intuitive to use. Advanced control options such as grouping controls and the option to add new vocabulary items are available but they are not easily mastered. A fair level of comfort with the program and significant computer literacy are needed to fully control and customize the program. The developersf web site includes a link to a user review / testimonial, which states that it takes up to an hour per week to set the software for customized groupings of kanji (Coffee, n.d.).

The procedure level includes questions like how the information is presented, what kinds of activities are included, how learner performance is judged, and what kind of feedback is provided. CALL specific criteria include screen layout, and help options. Lexikan is based on recognition and repetition drills. A kanji is presented visually and the leaner can choose the correct reading from four text options. Alternately the leaner can enter the reading by typing it. Since kanji have multiple possible readings, this quiz format accepts multiple correct answers. Also, the advanced customization controls allow users to add readings to the database. The screen layout is clear and uncluttered. Though the toolbar control icons are not entirely clear intuitively, one quickly becomes accustomed to them. Help information is readily available at any point in the program.

Software Evaluation

Chapelle (2001) advocates the use of a 6-point evaluation of CALL materials. Combining this with advice from Son (2003) a checklist was developed for the evaluation of CALL resources (see Appendix B). This checklist formed the basis of the following evaluation. In addition, two groups of Japanese learners were asked to use the demo version of Lexikan at home for one week and record their impressions on a survey (see Appendix C for the survey questions and Appendix D for a summary of the results).

Language Learning Potential

Chapelle (2001) describes language learning potential as the level of beneficial focus on form that the materials call for. Focus on form is seen as one of the key elements of second language acquisition (Vale, Scarino, & McKay, 2002). Lexikan promotes focus on form exclusively. In reading practice, the learner is focused entirely on the form of a given kanji and how one is to read it. In writing practice, the leaner is tested on and prompted for the correct stroke placement and order - important features of Japanese penmanship. However, excessive focus on form, at the expense of meaning, is not necessarily always beneficial to the learners. This is an unfortunate weakness of the Lexikan system. The focus is strictly on the form of individual kanji or individual words made up of two or three kanji. There is no attempt made to provide a deeper context for the learner, thus the claim that Lexikan helps improve reading skills may be overstated. Also, while the program does allow for variable response times promoting considered output, it is not meaningful output. The learners are only expected to provide the reading of the kanji. Neither its meaning, nor use are involved. This exclusive focus on form is not considered an effective way to present vocabulary (Descarrico, 2001). This having been said, the survey groups both reported being able to learn with this program. The repetition factor was useful for them as were the various prompts in the writing practice.

Feedback in the reading practice tasks is somewhat inappropriate. The program simply gives the correct answer if a mistake has been made. Then, a detailed reference screen is shown. While the information on this screen is of value (see below) it is generally better to have the default option lead to further practice, rather than reference information (Colvin Clark & Mayer, 2003). There is no opportunity to try again until all of the kanji in a given group have been tested. There is then a chance to retry the mistaken cards.

The feedback options in the writing exercises are more appropriate. After each stroke of the mouse, learners are told if they are on the right track. Also, they are told specifically if they have made a mistake of placement, stroke type, or order. This is a real strength of the program.

Another strength of the program is the availability of reference information. Reading flashcards are accompanied by detailed references to other kanji learning materials, as well as sample vocabulary items, meanings and alternate readings. Writing flashcards are accompanied by optional prompts showing the location of the next stroke, practice outlines to trace, and full motion animations of the kanji being written.

Learner Fit

Chapelle (1991) describes learner fit in terms of language level and learner characteristics. The language level of CALL resources must be appropriate to the target learner. If it is too easy or too difficult, nothing will be learned. This is one of the strengths of Lexikan 2.0. Containing the entire list of normally required kanji, and being expandable beyond that, Lexikan is appropriate, in terms of content, for a wide range of learners from those just starting to learn kanji to those nearly fluent.

Another strength is the level of control available to the learner. At the basic level, the learner is in control of the number of repetitions of writing samples, whether or not to be prompted for the next stroke, the method and time allowed for responses, whether to simply view the cards or be quizzed on them, and the number of times a given set of flashcards is practiced. They can also choose to see the relevant information and quizzes displayed in Roman letters or in kana characters. These are all important decisions likely to give the learner a sense of control over the program.

Another, one might say more advanced, set of learner control options is also part of the Lexikan system. Learners can take control over the order flashcards are presented in and the grouping of cards. They can also add or delete kanji and/or kanji readings from the database and add new words to the vocabulary list for any given kanji. These control features are not likely to appeal to novice users because they require a great deal of familiarity with the program and a considerable knowledge of the kanji involved. This two-tiered system of learner control is consistent with research findings in computer-based instruction. Research has shown greater degrees of control are beneficial to learners with greater background knowledge in the area. However, novice learners, or those with low achievement levels can be hampered by having to make too many control decisions (Colvin Clark & Mayer, 2003). Novice Lexikan users can leave the advanced control options alone and concentrate on the preexisting database and automatic groupings. Those with some kanji learning experience and understanding can customize the system to suit themselves. It is the best of both worlds.

Another issue in the area of learner fit is the appeal of the program. It should be appealing to and suitable for the target audience, in this case a wide variety of home based learners. Here Lexikan has some drawbacks. Firstly, it is very much visually and sequentially based. Those more comfortable with a broader picture may have trouble with it. Also, being based on a very inductive teaching plan, it is unlikely to appeal to those with a deductive cognitive style. Also, it must be said that Lexikan is somewhat repetitious, dull and unappealing. The lack of context for individual kanji items adds to this problem. Several members of the survey group commented that they found the program boring.

One other facet of Lexikan that is likely to limit its appeal it that the menu options, help information, manual, and instructions are all in English. The company producing Lexikan is based in The United States, so this is a logical choice for them but it has some drawbacks. While English is something of a lingua franca, it is not the native language of the majority of people learning Japanese (R. Odake, personal communication, September 28, 2003). The two groups who piloted the program for this review were made up of native speakers of various languages. Several participants noted that they had trouble controlling the program because the menus and help screens are in English only. These same participants noted that the default setting of 3 seconds to read and respond to the flashcards was too fast. They did not take advantage of the control options to change this setting. The program would have a wider appeal and fit a greater number of learners if it were available in alternate languages such as Korean or Chinese.

Meaning focus

Chapelle (2001, p.56) says gthe importance of meaning focus in language learning may go without sayingh. It is widely accepted in SLA research and language teaching that a focus on meaning is key to language learning (Vale, Scarino, & McKay, 2002). Krashenfs (1982) idea of comprehensible input, and the need to negotiate meaning are central ideas not present in Lexikan. The program has no context whatsoever for the kanji being studied. There are no example sentences or longer reading texts. There is no opportunity for the students to negotiate meaning, and there is no task beyond memorizing the kanji. The learners do not even have the challenge of operating the system in Japanese. The learners are in no way focused on a larger meaning.

The program does not even ask the learner to focus on the meaning of the isolated kanji they are working with. The meaning is given on the reference screen but there are no questions dealing with meaning. This can be especially problematic in Japanese due to the very large number of homonyms. For example. all of the following kanji –åA–äA•¶A•· and –ã can be read as gmonh. Without focusing the learner on the meaning the program does nothing to help differentiate them. Using the preexisting input method (multiple choice quizzes) it would be simple to add meaning questions to the program. The user could be presented with a kanji and 4 possible meanings, or with an English word and four possible kanji to choose from. Granted, this would decrease even further the usability of Lexikan for non-native English speakers. But assuming that the target market is made up of English speakers it would greatly improve the programs value as a study tool.

This lack of focus on meaning is by far the most serious drawback of the Lexikan program and it was also noted by the test groups. On item 5 gI found the language information / models helpfulh the average score on the Likert scale was quite low (1.9). Comments included gThere are no models are there?h and gItfs only readings, no readingh. Clearly, Lexikan should add, at the very least, some example sentences to its database.

Authenticity

Another serious drawback for Lexikan is in its lack of authenticity. Chapelle (2001) refers to authenticity as the degree of resemblance of a CALL task to real world language and situations. This is an important issue in the transfer of learning. It is well established in CBI and general education that learning is situation specific. That is, learning transfers better to real world tasks if the learning tasks closely resemble the real world tasks (Colvin Clark & Mayer, 2003). It is also well established in language teaching that students should develop reading skills using materials as authentic as practically possible for their language level (Tribble, 1996) and develop writing skills in tasks that are as situationally authentic as possible (Wallace, 1992). This is another important failing of Lexikan. While the material is accurate (no factual mistakes were found in reviewing the materials) the lack of context eliminates any opportunity for authentic practice. It is hard to imagine an authentic situation where the learner will be called on to read or write a single kanji. Adding example sentences would be a great improvement for this program. Longer comprehension texts based on the kanji being studied would be technically difficult given the users ability to customize the groupings and compounds the kanji are found in. However, a supplementary dictionary file is available for free downloading. It contains over 100 000 compound words made up of 2 or more of the kanji contained in Lexikanfs database. Example sentences could be added to this dictionary database. Also, manipulation exercises using the current data input methods (typing the reading or selecting the reading from 4 options) could be based on example sentences, not simply one word flashcards.

Positive Impact

Positive impact can be thought of as the beneficial impact that CALL resources may have on learners above and beyond the language learning itself. It includes things like increasing motivation or interest in the L2 culture and developing appropriate metacognitive skills such as goal setting and accountability (Chapelle, 2001). Not everyone has the metacognitive skills necessary to be an effective and self-directed learner but these can be learned (Williams & Burden, 1997). In a CALL context it has been suggested that appropriate metacognitive strategies can be taught using a slow build up of learner control (Jamieson & Chapelle, 1988) or a program with adaptive feedback and advisement (Pederson, 1987, Colvin Clark & Mayer, 2003).

Here again, Lexikanfs lack of focus on meaning and authenticity work against it. Without more context, it is unlikely that motivation and interest in Japanese culture are going to be fostered.

As for program control, as was discussed above, Lexikan does boast an impressive array of user control options that could be exploited by learners operating with higher order metacognitive skills. And for those without a background in kanji or language learning, the program can be used in the default mode with relatively few choices to be made. However, the program lacks a slow build up of control and advisement. While allowing for the use of higher order metacognitive skills, it is unclear that Lexikan promotes their development.

The program received somewhat low marks from the survey groups on positive impact as well. Item 7 (gI found it easy to set goalsh) was rated 2 out of a possible 5 and item 8 (gUsing this program made me more interested in learningh) was rated 2.5. Only one user noted that it was fun and engaging to use a computer for kanji practice and expressed interest in buying the full version of the program.

Practicality

Practicality is the overall ease of implementation of a program (Chapelle, 2001). This is one of the great strengths of Lexikan. It is easy to install and does not require a great deal of computer literacy to operate at a basic level, though the advanced control options are somewhat complicated. Its system requirements (see appendix A) are rather basic so it is likely to run well on most IBM compatible computers currently in use. The on-screen instructions are all clear, with the caveat that they are in English only, and the help information is clear and readily accessible. The programfs on screen layout is also effectively designed. It does not have extraneous graphics, and key information is presented with sufficient white space surrounding it to allow it to be the clear focus of attention. The methods of input for both the reading and writing flashcards are easy to use. The program received a favorable rating from the survey groups on practicality. Comments included gThe program is easy to useh and gThe interface is clearh.

Conclusions - Implications for Use

Lexikan 2.0 has some serious drawbacks as well as some genuine strengths. To say that it is a good or bad program is however to forget that CALL resource analysis is necessarily situation specific (Chapelle, 2001). Materials cannot be evaluated in isolation. The learners and the tasks that they perform are a key element of the analysis. It is better to say for whom and when the materials would be appropriate or inappropriate.

Lexikan sells itself as a self-study program to teach reading and writing in Japanese. As such, it is claiming to be a process-oriented program. That is, a set of materials providing a framework for communicative use of the language (Levy, 1997). This may not be the most appropriate use. The lack of meaning focus and authenticity would seriously hamper the development of genuine reading and writing skills and may not foster motivation if the program is used in isolation. Also, the advanced control options might not be fully exploited without some expertise in language learning in general and kanji in particular. However, if used as part of a wider program Lexikan would be an excellent supplement to more authentic reading and writing materials. The advanced control features allow users (or teachers) to group together kanji based on a text being studied or a theme being written about. The learners could than use Lexikan to improve their character recognition and recall as well as their stroke order and other penmanship skills. In this supplementary resource role, known as content materials (Levy, 1997), the repetition and control features of Lexikan would be of great value to the learners.

In short, Lexikan as a stand-alone program is flawed but it has great potential to play a part in a balanced reading and writing curriculum.

References

Chapelle, C. A. (2001). Computer applications in second language acquisition: Foundations for teaching testing and research. Cambridge: CUP

Coffee, S. (n.d.). Lexikan, version 2. Retrieved 10/9/2003 from
http://www.kanjiclinic.com/reviewlexikan.htm

Colvin Clark, R., Mayer, R.E., (2003). e-learning and the science of instruction. San Francisco: Jossey-Bass Pfiffer.

DeCarrico, J.S. (2001). Vocabulary learning and teaching. In M. Celce-Murcia (Ed.), Teaching English as a second or foreign language. (pp. 251-266). Boston: Heinle & Heinle.

Healey, D., (1996). Deborah Healey's remarks on Hubbard's framework for CALL evaluation. Retrieved 22/09/2003 from
http://oregonstate.edu/Dept/eli/vstevens/day1.htm/healey

Hubbard, P. (1992). A methodological framework for CALL courseware development. In M. Pennington & V. Stevens (Eds.), Computers in Applied Linguistics (pp. 39-66). Philadelphia: Multilingual Matters.

Jamieson, J., Chapelle, C. (1988). Using CALL effectively: What do we need to know about students?. System, 16, 151-162.

Krashen, S. (1982). Principles and practices in second language acquisition. Oxford: Pergamon.

Levy, M. (1997). Computer assisted language learning: Context and conceptualization. Oxford: Oxford University Press

Lexikan Software Company. (2003). What is Lexikan 2.0?. Retrieved 10/09/2003 from http://www.lexikan.com/about_lexikan.htm


Pederson, K.M. (1987). Research on CALL. In W.F. Smith (Ed.), Modern media in foreign language education: Theory and implementation (pp. 99-131). Lincolnwood, Il.: NTC.

Richards, J.C., & Rodgers, T. (1987). Method: approach, design, and procedure. In M. Long & J.C. Richards (Eds.), Methodology in TESOL: A book of readings (pp. 133 - 158).Boston: Heinle & Heinle.

Richards, J.C., & Rodgers, T. (2001). Approaches and methods in language teaching (2nd ed.). Cambridge: CUP.

Son, B.J. (2003). Computer assisted language learning: Study book. Toowoomba: DEC

Tribble, C. (1996). Writing. Oxford: Oxford University Press.

Vale, D., Scarino, A., McKay, P., (2002). Pocket ALL. Carlton: Curriculum Corporation.

Wallace, C. (1992). Reading. Oxford: Oxford University Press.

Appendix A

A Lexikan 2.0 demo version is available for download from www.lexikan.com. The full version is available on CD-Rom for $35(USD) for the standard edition or $45 (USD) for the advanced edition.

System Requirements

* Windows 98 or better
* 4MB of hard disk space
* 32MB of RAM


Appendix B

CALL Resources Evaluation Questionnaire

Program Name:

Rating Scale (1-5)

1 - unacceptable
2 - acceptable
3 - fair
4 - good
5 - excellent

Language Learning Potential

* The learnersf attention is focused on form.
* The program has mechanisms to prompt/support learning.
* The program has potential for modified output.
* The program allows for sufficient time for considered output.
* The feedback is appropriate and helpful.
* Reference material is readily available.

Learner Fit

* The language level of the tasks is appropriate for the learner.
* The language level of the instructions is appropriate for the learner.
* There is an appropriate level of learner control.
* The program is enjoyable.
* The program appeal to a wide range of learning styles.

Meaning Focus

* The learners are focused on meaning during the task.
* The learners have an opportunity to negotiate meaning during the task.
* The students need to understand the language in order to complete the task.
* The tasks have a non-linguistic goal.

Authenticity

* The content is accurate.
* The content has a rich authentic context.
* The material is culturally authentic.
* The content is presented at a natural pace and length.

Positive Impact

* The program is likely to encourage accountability.
* The program is likely to encourage the use of metacognitive strategies.
* The program is likely to encourage interest in L2 culture.
* The program is likely to encourage motivation.

Practicality

* The onscreen instructions are clear and easy to follow.
* Help is available at appropriate times.
* The program is free of bugs and breaks.
* The program is easy to install and run.
* The methods of input are easy to use.
* The program makes effective use of screen layout.
* The program makes effective use of graphics.
* The program makes effective use of color.
* The program makes effective use of sound.

Evaluated by: Date:


Appendix C

Program Evaluation Survey

Program Title: Lexikan 2.0 Date:

Please answer each item below on a 1 to 5 scale. 1= totally disagree 5= totally agree
Also, please feel free to add any comments you may have.

1) The point of the program was clear from the beginning.

2) I was able to learn using this program. .

3) This program was fun to use. .

4) I found the feedback helpful. .

5) I found the language information / models helpful. .

6) I found the pace of the program appropriate. .

7) I found it easy to set goals using this program. .

8) Using this program made me more interesting in learning. .

9) This program is user friendly. .

10) I am likely to continue using this program. .


Appendix D

Survey Results

The survey was administered to two groups of Japanese learners. Group A consisted of 7 learners at a lower intermediate level while group B consisted of 5 learners at the upper intermediate and advanced levels. Both groups are residents of Japan and both groups are multicultural classes. The participants were provided with a demonstration copy of Lexikan 2.0 and asked to use it at home for one week before completing the survey. The survey was printed in English but presented to the groups with a running oral translation to Japanese. Where necessary, the sample comments have been translated.

Item number
A
B
Comments

1)
4
4.2
It is self-explanatory. I understood what I should do easily. The way to use it is easy.

2)
4
3.5
I can remember the kanji. I can see the kanji as many times as I want.

3)
3
2
It is very fun. It is OK. It was fun to start.

4)
2
3
Drawing feedback is good. Too fast.

5)
1.8
1.9
Itfs only readings, no reading. There are no models are there?
I want examples in a sentence.

6)
2.2
3.1
Too fast. One question doesnft have enough time to think.

7)
2
2.4
It is hard to choose kanji. Itfs OK.

8)
2
3
It is kind of boring. It was fun at the beginning but it is all the same.

9)
3
2.5
Easy to use. Too fast. It is all English.

10)
3
3
Itfs OK but not so different from my kanji book. It is fun I will continue. I want to buy the full version.