Read article_626.pdf text version

Gabriel Jacobs and Catherine Rodgers

Treacherous Allies: Foreign Language Grammar Checkers

Gabriel Jacobs and Catherine Rodgers

University of Wales Swansea

ABSTRACT This article discusses the use of a French computerized grammar checker as a learning and teaching resource. As is well known, foreign language grammar checkers can give unreliable, even farcical results, but that does not necessarily mean that they cannot be effectively used by foreign language teachers. After some discussion of the nature of foreign language grammar checkers, the article presents the results of a controlled series of experiments in which groups of students were given the task of correcting French texts containing grammatical, lexical, and orthographical errors by using either an on-screen grammar checker or grammar books and dictionaries. The study draws some conclusions about the advantages and pitfalls of using foreign language grammar checkers for teaching and learning.

KEYWORDS Grammar Checker, Spelling Checker, Proofing Tool, French Grammar, Foreign Language Teaching Resources

INTRODUCTION: THE GRAMMAR CHECKER PROBLEM It is almost a commonplace among teachers of French to deride computerized French grammar checkers, and it is not difficult to understand why.1 The Microsoft French Proofing Tool, which we used for the experiments described later in this paper, picks up the sentence La peine que cela m'a coûtée `The trouble that that cost me' and comments

© 1999 CALICO Journal

Volume 16 Number 4


Treacherous Allies

D'habitude, le participe passé de coûter reste invariable sauf au sens figuré `Usually, the past participle of coûter remains invariable [does not agree with the preceding direct object] except in the figurative sense' It then gives the same sentence as an example of a figurative use: la peine que cela m'a coûtée, and suggests coûté as a solution! It is no wonder that grammar checkers have gained their reputation for generating farcical results, and it can certainly be fun to play games with them. For instance, if a message such as Si ceux-là est le sujet de est, il y a une faute d'accord `If those is the subject of is, there is an agreement error' is itself sent through a checker, a self-referential result will almost inevitably be generated: Si ceux-là est le sujet de est, il y a une faute d'accord. Less laughable, and therefore more disturbing, is the following correct sentence: Je ne crois pas que ce soit vrai for which the Microsoft Tool generates the erroneous message Temps et mode: Mettez le verbe soit à l'indicatif: crois que demande l'indicatif. `Tense and mood: Put the verb soit in the indicative: crois que requires the indicative.' Here, it would have been a comparatively elementary task to add to the checker's database for the entry under croire the fact that when a form of this verb is surrounded by ne ... pas [negative marker] and followed by que `that', the subjunctive mood is to be expected. More generally however, taking a negative into account in determining whether or not a subjunctive should be used in French represents no mean feat of programming in view of the fact that the use of the subjunctive is often conditional on sense and interpretation and sometimes purely subjective. Even seemingly mechanistic grammatical rules governing, say, the agreement of past participles are no less complex and can in certain instances be no less dependent on the sense intended. It is therefore hardly surprising that French grammar checkers indicate a probable error in each of the following grammatically correct cases: 1. Les cinquante kilos qu'elle a pesé. `The fifth kilos that she weighed.' 2. La chaleur qu'il a fait. `The heat that it was [literally `made'].' 3. Une maison qu'on aurait dit récente. `A house that we would have thought [literally `said'] recent.' While such examples (taken from Le Petit Robert) admittedly do not oc510 CALICO Journal

Gabriel Jacobs and Catherine Rodgers

cur frequently in normal communicative French, they nevertheless point up the so far insurmountable problem of parsing natural language, a problem which lies at the heart of producing a reliable computerized grammar checker. Sentence 1 above causes difficulty because peser `to weight' is usually a transitive verb (e.g., Les cinquante pommes qu'elle a pesées `The fifty apples that she weighed'). Sentence 2 is problematic because the il `it' is impersonal--compare the correct (if unusual) La chaleur qu'il a faite `The heat that he/it made,' where il refers to an agent. Sentence 3 causes difficulty because of the similarity with sentences such as the correct Une phrase qu'on aurait dite `A sentence that one could have said.' In all three cases, grammatical rules cannot be applied mechanically until the meaning of the sentence has been grasped. A good example of this phenomenon is provided by what the Microsoft product makes of Malgré la tentation qu'elle a ressentie la veille, Marie décide ... `Despite the temptation that she felt the day before, Marie decides ...' It correctly allows Malgré la tentation qu'elle a ressentie, but the addition of la veille generates the comment Mettez le participe passé au masculin singulier. L'accord du participe ne se fait pas ici: il n'y a pas d'objet direct ou l'objet direct suit le verbe. `Put the past participle in the masculine singular. The agreement of the particle is not made here: there is no direct object or the direct object follows the verb.' The program takes veille to be the direct object of a ressentie, the construction elle a ressenti la veille making perfect grammatical sense to the checker. Once again, one could imagine adding the phrase la veille to the checker's database and attaching an adverbial flag to it, but how many more examples would have to be coded to cover the entire language? Even a rule apparently as elementary as `Transitive verbs require a direct object' (ressentir is always transitive) fails to cover myriad cases where grammatically and semantically incorrect sentences fall within it. The most upto-date and advanced French grammar checkers, including Antidote, Cordial, Correcteur 101, Grammatique, GramR and Hugo Plus, allow complete sentences of the type Il met le couteau `he puts the knife' as correct. One might incorporate a rule that mettre requires both a direct object and a prepositional phrase (Il met le couteau sur la table `He puts the knife on the table'), but then how does one account for Il met le couvert `He is setting the table?' No computerized grammar checker is currently able to parse a language well enough to avoid frequent misreadings when meaning enters into the equation. Vanneste (1994), having briefly compared some French grammar checkers, gives an example of an error of meaning

Volume 16 Number 4 511

Treacherous Allies

which no checker could handle now or in the foreseeable future, La banane a mangé le singe `The banana ate the monkey.' The point is put forcefully by Bolton (1993). In the crucial issues of rhetorical cogency, expressive adequacy, and honest intent, such tools have no place. On the other hand, while grammar checkers are ready to accept meaningless but grammatically correct sentences, they refuse--arguably rightly so--meaningful but grammatically incorrect sentences. Meaning may be unambiguously conveyed by ungrammatical sentences such as Cette phrase pas de verbe `This sentence no verb,' an error which is identified as a problem but which is not properly corrected by the Microsoft Tool, Correcteur 101, or Cordial. Pinker (1994) gives "Drum vapor worker cigarette flick boom" as a splendid example. However, while it may be argued that all one has the right to expect of a grammar checker is that it checks grammar, not meaning, the difficulty remains. In order to check grammar efficiently, meaning must be inferred, and the problem of writing parsing rules which take meaning fully into account is one of unfathomable complexity. Nevertheless, computerized grammar checkers are not to be too thoughtlessly ridiculed. Early versions of French checkers worked by merely verifying agreements between pairs of adjacent words (noun and adjective, subject and verb, finite verb plus infinitive, etc.). This limitation meant that the software was incapable of coping with any intercalated words (even in expressions such as il vous aime `he loves you,' which would have been flagged as a mismatch between subject vous and verb aime) and therefore of any real attempt at parsing. Present day grammar checkers use Artificial Intelligence techniques to analyse the structure of sentences. They too fall into numerous intercalated-word and other syntactic traps, but much progress has been made. In the hands of a user with a good knowledge of the grammar of a given language--one who is therefore able to discount obviously erroneous suggestions and obviously misguided instructions--grammar checkers can be very useful tools. They can make all but the most confident and expert grammarians reconsider. But what of users whose knowledge of grammar is comparatively limited? Herein assuredly lies a potential danger. In the days before word processors able to produce documents virtually indistinguishable from professionally typeset material, there was a tendency for the average person to believe anything which appeared in print. Has one now reached a stage at which a computer screen is taking the place of print as an apparently infallible source of information? If so, the effects could be far reaching. We have been unable to identify any specific research into the trust people afford to grammar checkers, but many studies have been carried

512 CALICO Journal

Gabriel Jacobs and Catherine Rodgers

out on the levels of trust people afford to computers in general, usually with predictable results. For example, Gueutal (1989) has shown that a positive attitude towards computers resulting from their previous successful use is highly correlated with confidence in their decisions. Muir (1994) has proven that the level of trust in a computer is directly related to the extent to which it has consistently demonstrated superior performance. Will (1991) devised a rule-based computer system which deliberately gave defective advice and found that novice subjects were more likely to have confidence in it than expert subjects. One might therefore assume that since spelling checkers have become essential word-processing tools for many language students which consistently display effective and reliable performance, these students will tend to generalize the confidence they have in spelling checkers to include grammar checkers. Yet the algorithms used in spelling checkers cannot be fairly compared with the complex parsing algorithms required for grammar checkers. Spelling checkers rarely make a mistake because their primary function is merely to search for strings of characters which match those in their stored word lists. Likewise, electronic thesauri, which use an admittedly complicated but (in overall design terms) elementary system of hyperlinks, depend on a comparatively simple algorithm. Grammar checkers have an infinitely more difficult task to perform and are thus much more likely to show the computer for what it is: a machine capable of formidable feats but also of brazen stupidity. The danger in using grammar checkers is one of more or less blindly accepting screen instructions as a result of having had long experience with the power and seeming infallibility of other editing tools that are much simpler in concept. The problem is magnified when a grammar checker is used to proof text in a foreign language.

THE EXPERIMENTS Given all of the above, one may wonder if French grammar checkers, with all their potential failings, can be usefully used as teaching and learning tools and what level of trust students might afford them. We decided to address these two questions in a perhaps limited way by conducting a series of controlled experiments. We first constructed four written texts in business French of around 300 words each. Each text contained a number of grammatical, lexical, and orthographical errors made by previous generations of second-year UK university students.2 Grammatical mistakes in the texts--not including mistakes of gender--were those of agreements, word order, verb form, tense, mood, and so on (average number of errors per text = 21). Gender mistakes averaged 10 errors per text. Lexical errors (average number of errors per text = 10) included anglicisms and incorrect context.

Volume 16 Number 4 513

Treacherous Allies

Orthographical errors (average number of errors per text = 12) included absent or incorrect accents where meaning was not affected (if affected, the error was deemed to be grammatical). Exact equivalence between the four texts in terms of linguistic difficulty, tone, register, and categories of mistakes would have been ideal, but we were not able to establish such formal equivalencies and had to rely to some extent on subjective judgment. We nevertheless believe that the four texts presented much the same profile to students. (Short extracts from the texts can be found in Appendix A.) We loaded the first text on workstations in a PC laboratory where the Microsoft Tool was installed. We chose the Microsoft Tool because it was easily available to us and because the texts were written in Microsoft Word, thereby guaranteeing maximum compatibility. Most available grammar checkers claim to be compatible with Word on the PC, but we saw no need to conduct tests on compatibility since the Microsoft product was so readily usable.3 The Microsoft Tool was more than sufficient for our purposes since a good part of what we wished to investigate was student reaction to false positives (i.e., correct French expressions erroneously flagged as incorrect), false negatives (errors that the checker failed to identify), and gray areas (expressions in which, for example, the checker flagged an error but for the wrong reason). We did not seek to assess the Microsoft product to determine its level of functionality, its competence in not violating grammatical constraints, or other criteria which can be used to evaluate grammar checkers. (See EAGLES, 1995.) Next, we divided alphabetically by surname 42 second-year students taking business French modules into two groups of equal number. All the students were familiar with using computers but none had ever used a grammar checker. We explained to the members of Group A how to use the Microsoft Tool and asked them to use it to check the grammar of Text One and then to produce, to the best of their ability, a version in correct grammatical French. Each student worked individually at a workstation under supervised examination conditions. Students in this group did not have access to any other resources (e.g., grammar books or dictionaries). Meanwhile, the students in Group B, again under supervised conditions, were asked to correct the same Text with grammar books and dictionaries. Students in this group did not have access to computers. We could have used other means of comparison, such as testing the output of the grammar checker against an on-line grammar resource, but we chose to compare the results obtained with a grammar checker against the results obtained with the tools that were the most familiar to our students and that were widely available to them. The students in Group B were given 25% more time to complete the task than those in Group A. This figure may seem low, but we noticed in a pilot trial that students tended to use dictionaries more extensively than grammar books and required less time

514 CALICO Journal

Gabriel Jacobs and Catherine Rodgers

than we had expected to find information in the printed resources. The following week, the roles were reversed under conditions identical to those of the previous week; students in Group B attempted to correct Text Two in the computer laboratory, whereas students in Group A corrected this same text using printed reference material.

RESULTS At the end of this first trial, we marked all the computer and paper texts produced by the students. Once again, some subjective judgment was involved here since we had to decide (as language teachers regularly have to) what constituted a serious error, a minor error, and an error that fell in between the two. For the sake of simplicity, we used a negative marking scheme, ranging from a double underline (minus 2) to a dotted underline (minus 1/2). (See examples of the use of the marking scheme in Appendix A.) We are well aware that our judgments of the gravity of errors may be open to some discussion (one of us is a native speaker of English and the other one a native speaker of French). Many of those who are involved in marking foreign language papers will be aware of the differences of opinion which tend to arise between English-speaking teachers of a foreign language and their native-speaking counterparts. In cases in which we could not resolve our differences of opinion, we merely compromised. Mistakes made by the students were, as expected, both errors of omission and commission. We did not distinguish between these two kinds of errors in our quantitative analysis. Table 1 sets out the mean scores of the two groups on each text. Table 1 Mean Scores of Group A and Group B for Text One and Text Two Text One Text Two Group A (N = 21) 43 37 Group B (N = 21) 41 56 Notes: Students' scores represent the number of points deducted for errors in their corrected French texts. Students in group A used the grammar checker for Text One and printed reference materials for Text Two; students in group B followed just the opposite schedule. Differences in the mean scores of the groups between Text One and Text Two may be attributed to differences in the level of difficulty of the two texts. (Recall that we were not able to establish formal text equivalencies.) Nevertheless, the critical outcome here is that the computer group (Group A for Text One and Group B for Text Two) performed less well in

Volume 16 Number 4 515

Treacherous Allies

both cases, especially in Text Two. We also examined the types of mistakes made by the students, comparing those made by the computer groups to those made by the paper groups, again including both errors of omission and commission. The general results of our investigation showed that blatant mistakes were corrected by both groups. Such mistakes included disagreement in number of verb and subject (e.g., les annonceurs vise `the announcers aims' [corr: les annonceurs visent]) or of gender between noun and adjective (e.g., son déclin est mondiale `its decline is worldwide' [corr: son déclin est mondial]), and in gender for which basic rules had supposedly already been learned (e.g., la gaspillage `squandering' and la message `message' [corr: le gaspillage and le message]), all of which were appropriately corrected by the grammar checker. However even when the grammar checker failed to identify the real problem, as in commence a, where a was flagged as a verb without a subject, the students in the computer group correctly added the accent mark (commence à). For this type of mistake, the grammar checker is obviously of limited use except as a safety net. Other cases in which the use of a grammar checker presents little interest for teaching purposes involve texts which are grammatically correct and in which the checker rightly makes no comment. With one or two exceptions, neither students in the paper group nor those in the computer group created additional mistakes. More interesting from a pedagogical point of view are the following four scenarios: 1) A true positive (An error exists in the text, and the grammar checker rightly suggests a correction.) True positives include spelling errors (e.g., déringoler `to tumble, to fall down' and verité `truth'[corr: dégringoler and vérité])-- which were corrected by the computer groups because the grammar checker also performs a spelling check, unambiguous grammatical mistakes related to spelling (e.g., quatre-vingt `eighty' [corr: quatre-vingts]) and vocabulary mistakes (e.g., grâce à `because of' [corr: à cause de]. Both of the latter mistakes tended to be corrected by the computer groups who clearly showed their overwhelming advantage here. 2) A false positive, type one (The grammar checker identifies a nonexistent error in the text.) An example of a false positive is Les publicités sont en général très coûteuses `Ads are in general very costly.' Here, the Microsoft Tool saw a mismatch in number and gender between général and what it takes to be its adjective coûteuses. One fourth of all stu516 CALICO Journal

Gabriel Jacobs and Catherine Rodgers

dents, whether members of a computer group or a paper group, changed something in this sentence. With respect to the paper students, this finding appears to contradict what we have said above about students' not creating additional mistakes. In this case, the finding can be explained by the fact that this sentence was one of the very rare sentences which did not contain a mistake but that the paper students felt must contain an error somewhere. If we compare the changes made by computer and paper groups, we find that while about a third of changes made by the paper groups produced incorrect French, more than two thirds of changes made by the computer groups resulted in incorrect French because the computer students blindly followed the grammar checker. It is clear that students are not completely impervious to erroneous advice when that advice has been generated by a computer. 3) A false positive, type two (The grammar checker flags a genuine error in the text but for the wrong reason.) This type of false positive includes sentences such as si le publicitaire sut ses consommateurs ... `if the plublicist knew his/ her consumers ...' [corr: si le publicitaire connaissait ses consommateurs ...]. Here, there are at least two errors in the same word. The Microsoft Tool did not spot the lexical problem (savoir vs. connaître). It flagged a tense problem but only to inform the students that the past historic was not an appropriate tense for this kind of text. All but one computer student tried to correct the problem, but only one managed to find a solution by completely recasting the sentence. Among the others, about 80% attempted to correct the tense and on the whole managed to do so, but only one spotted and rectified the vocabulary problem. The paper students fared much better; all understood that there was a problem, and over two thirds correctly corrected both mistakes, while less than a fourth left the lexical error unchanged. 4) A false negative (The grammar checker misses an error in the text.) In our experiments, false negatives turned out to be a fairly common occurrence. Anglicisms fall into this category such as penser de (for penser à), sur la télé (for à la télé), and expériencer des problèmes (for rencontrer or connaître des problèmes). No computer student caught the errors. Approximately 80% of the paper students corrected penser de and sur la télé, about two thirds spotted the error in the use of expériencer, and about half of these

Volume 16 Number 4 517

Treacherous Allies

students correctly changed it. One must assume that the computer students believed that if the grammar checker had not flagged a mistake, there must not be one. In this context, it should be mentioned that no French grammar checker is aimed at native English speakers, although some French producers of proofing software are said to be considering versions for native English speakers. After this first trial, the limitations of grammar checkers and the Microsoft Tool in particular were discussed in class. We pointed out to the students that no grammar-checking program could interpret meaning and that all of them could be fooled by such things as word order. We reinforced the point that grammar checkers were capable of making absurd suggestions and missing obvious errors. In addition, we went through the two test texts to demonstrate how the checker might have been used to best advantage. Four weeks later, we conducted a second trial using the same procedures with two different texts. We did not wish to involve any students in the second trial who had not been involved in the first, and Groups A and B were reduced to 20 and 18 students, respectively, because of absences. The quantitative results of this trial are shown in Table 2.

Table 2 Mean Scores of Groups A and B for Text One and Text Two Text Three Text Four Group A (N = 20) 35 37 Group B (N = 18) 28 41 Notes: Students' scores represent the number of points deducted for errors in their corrected French texts. Students in group A used printed reference materials for Text Three and the grammar checker for Text Four; students in group B followed just the opposite schedule. Although the students who used the computer had better scores than those who used printed materials, the differences in mean scores is not very great. Nevertheless, it seems that teaching the students how best to use a grammar checker was in itself a worthwhile exercise. In order to gain a better understanding of the help that the grammar checker offered to students, we classified the results of the trials by type of error (grammatical, gender, lexical, and orthographical). On average for each text, the Microsoft Tool was able to correct, directly or indirectly, 75% of the gender mistakes, 46% of the grammatical mistakes, and 91% of the orthographical mistakes. It did not flag any of the lexical mistakes in the texts. The results, expressed as percentages of uncorrected errors, for the computer and paper groups in each of the two trials are set out in

518 CALICO Journal

Gabriel Jacobs and Catherine Rodgers

Table 3. The Percentage of Uncorrected Errors for the Microsoft Tool are listed as points of reference. Table 3 Percentages of Uncorrected Errors by Trial and Group Microsoft Tool Trial 1 PG CG Grammatical 54 47 57 Gender 25 17 8 Lexical 100 64 80 Orthographical 9 33 8

Trial 2 PG CG 46 43 15 7 67 72 27 8

Notes: The Microsoft Tool percentages are average percentages per text. PG = Paper Groups and CG = Computer Groups. Some general conclusions can be drawn from Table 3 and the examination of students' scripts. 1) In the first trial, the computer groups tended to perform rather poorly on correcting grammatical errors and lexical errors and fairly well on gender and orthographical errors. The strengths and weaknesses of the grammar checker are broadly mirrored in the students' performance, which suggests that they leaned very heavily on the checker. 2) In the first trial, the paper groups performed better than the computer groups on grammatical and lexical errors but worse on gender and orthographical errors. This finding corroborates our view expressed above that the paper students tended to think more for themselves but of course did not benefit from the strengths of the grammar checker. 3) The variations observed in the rates of correction in the second trial (i.e., after the training period) seem to us to be less explicable in terms of conventional categories of mistakes than in terms of whether the mistakes had been flagged or not by the computer. When one looks at the types of mistakes the students corrected by means of the grammar checker in this perspective, certain clear points emerge. i) As expected, the spell checker still produced a more correctly spelled version than the version for which a printed dictionary had been used, particularly with respect to accented characters. The same is true of certain unambiguous errors. For example, all the computer students corrected soixantes `sixty' [corr: soixante], whereas three of the paper


Volume 16 Number 4

Treacherous Allies

students did not. In addition, whereas all the computer students corrected quatre-vingt `eighty' [corr: quatre-vingts], six of the paper students still did not correct it despite the fact that this error had occurred during the previous trial. ii) With respect to anglicisms (anglicisms were not flagged by the grammar checker), both sets of students once again gave a relatively poor performance. For example, very few students, whether members of the computer groups or the paper groups corrected par les années 80 `by the 1980s' [corr: à partir des années 80], or adhérer `to belong to' [corr: devenir membre de] in relation to one's membership in a trade union. The explanation is that for this type of error, if it is not flagged by the grammar checker, improvement is obviously limited by the knowledge of the student. iii) The computer groups performed better at correcting errors in word order flagged for the wrong reason by the grammar checker and spotted them more regularly than their paper counterparts. In the first trial, all of the computer students attempted a correction of si l'on bien cible `if one aims well' [corr: si l'on cible bien] of whom 62% made the right correction, whereas only 66% of paper students spotted an error of whom 83% made the right correction. In the second trial, the sentence On pensait qu'on toujours travaillerait `they thought they would always work' [corr: On pensait qu'on travaillerait toujours] represents a fair point of comparison. All of the computer students attempted to correct the sentence in some way, versus 80% of paper students. It should be noted however that more of the computer students (83%) correctly corrected it, an improvement which is better than the marginal improvement of the paper students (85% correctly correcting it). (iv) The computer students showed clear improvement in correcting errors flagged by the grammar checker but for the wrong reasons (e.g., tous le temps `all the time' [corr: tout le temps] flagged as incorrect because the singular le appears not to agree with temps which is taken to be plural for les emploies à temps partiels `part time employees' [corr: les employés à temps partiel] flagged as incorrect because emploies is taken to be a verb).


CALICO Journal

Gabriel Jacobs and Catherine Rodgers

FEEDBACK As well as discussing the texts in class and conducting some individual interviews, we asked all the students to complete a questionnaire immediately after the first and second trials. (See the student questionnaire in Appendix B.) Analysis of students' responses to questionnaire items leads to a number of conclusions. 1) Most students found the Microsoft Tool easy to use, though most also found the messages it generated rather difficult to understand. 2) Trust in the grammar checker appears to have been increasingly limited, but students on the whole thought that it helped them correct mistakes. Most students realized that the grammar checker was capable of making mistakes but believed that they had not accepted many of its incorrect suggestions. Here, one should distinguish between a prescriptive message generated by a grammar checker and a tentative suggestion. Most recent grammar checkers (especially Antidote) avoid prescriptive messages almost entirely, whereas the Microsoft Tool is full of categorical comments. Nevertheless, even when no distinction is made between prescription and suggestion, the students were right in thinking that they had been discerning. On average, over 90% of the incorrect suggestions made by the grammar checker were refused by the students in the second trial. This result appears to suggest that the use of spelling checkers had little influence on the trust generally afforded to the grammar checker. The difference may be explained by the fact that students learned to identify the strengths and weaknesses of the grammar checker, understanding that they could trust it for spell-checking purposes as well as for its suggestions regarding gender and straightforward agreements, but that they should be wary of its suggestions regarding other types of mistakes. They appear to have understood that a grammar checker is a beast quite different from a spelling checker, something which we had pointed out in our training. 3) The students were fairly sharply divided in the comparison of paper resources with the computerized grammar checker. This finding was apparent in the answers to a question concerning whether the French Department at the University of Wales Swansea, given its limited resources, should invest in one or more computerized grammar checkers or in more grammar books and dictionaries. In class and in individual interviews, we pointed

Volume 16 Number 4 521

Treacherous Allies

out that grammar checkers more advanced than the Microsoft product were available and demonstrated the use of Le Correcteur 101 and Cordial on one of the test texts. Both of these programs clearly performed better than the Microsoft Tool, but the division among students generally remained strong. We should mention that students in the French Department are not obliged to submit work as word-processed (i.e., documents which can be passed through a computerized grammar checker). It was clear from comments that this policy had an influence on students' opinions. 4) Most students were nonetheless convinced that the systematic use of a grammar checker could improve their written French, despite the fact that their responses on the questionnaire clearly showed that they did not make much use of the Microsoft Tool's explanation files. Students' lack of use of the explanation files may have been due to the fact that these files did not offer thorough grammatical explanations. The equivalent files in some other products provide better and more extensive explanations with clearer examples. 5) The discussions in class and the individual interviews revealed that when students used the grammar checker, they at first tended to jump from one flagged problem to another without bothering to check text in between. When they had dealt with the last flagged problem, they felt that their task was complete.

CONCLUSION As a result of our experiments, we are able to make a number of recommendations to teachers considering the use of a foreign language grammar checker as an educational resource. In the first place, it should be noted that since French grammar checkers are aimed at native French speakers, their usefulness as learning and teaching tools may conceivably differ with native speakers of other languages. The texts used in our experiments were constructed specifically with the kinds of mistakes English speakers tend to make in French. Quite different results might be obtained with Spanish- or Japanese-speaking students. Regardless of their native tongue, students may legitimately be encouraged to afford grammar checkers a good level of trust in matters of spelling, gender, and straightforward agreements--with the proviso that suggestions about agreements be treated with a measure of circumspection if students are not sure whether the agreement is straightforward or not. We recognize of course that such circumspection already requires some expertise on the

522 CALICO Journal

Gabriel Jacobs and Catherine Rodgers

part of students. It is important to inform students that they should not implicitly trust grammar checkers in other matters but, rather, should always try to evaluate their advice. Students also need to understand that they should not assume grammar checkers detect all errors. One type of mistake on which teachers should concentrate when analyzing students' corrected texts is the type two false-positive category, that is, when the grammar checker flags a genuine error but for the wrong reason. In general, an excellent way to use foreign language grammar checkers as a learning resource is the way in which native speakers tend to use them, that is to say, as a flagging tool which brings possible errors to their attention. Students reported to us that the use of the grammar checker encouraged them to consult grammar books because of messages generated by the checker. Once such errors have been flagged (at least in French), it is worth considering one of the fairly wide range of reference programs covering French grammar instead of relying on the generally comparatively limited explanation files in most grammar checkers. For example, students can call up the Système-D application while using a word processor in order to look up a particular point of grammar, to see examples, to check the conjugation of a verb, and so forth. A number of shareware and public domain verb conjugators are available which provide extensive databases. It cannot be long before we see a digital Grevisse Sans Faute/Grammaire. An enhanced spelling checker which does not make use of sophisticated AI techniques, and consequently more limited in scope (and which also makes fewer mistakes), could be a useful tool for simple proof reading. This kind of program could also be used by advanced students of French for teaching and learning purposes. Our general conclusion about the value of using French grammar checkers with native English-speaking students is, on the whole, positive. We are so confident of their value that we intend to extend the experiments described in this article by incorporating them into the curriculum proper, thus allowing us to conduct longitudinal tests, possibly with Antidote because of its high scores in rigorous comparative tests (Katz & Villaudy, 1997). We believe that computerized foreign language grammar checkers can be formidable teaching allies if their tendency to betray is constantly borne in mind. It would be interesting to discover whether the regular use of a grammar checker alters students' attitudes towards the foreign language texts they produce, for example, whether it teaches them to be more suspicious of their own possible failings. Such matters will be the subject of future investigations.

Volume 16 Number 4


Treacherous Allies

APPENDIX A Brief extracts from the texts showing examples of the marking system Partout des entreprise deviennent plus petites, resultat: quelques employées ... permanentes® qui sont soutenu par beaucoup des employées® de temps partielles. [...] Il y a 20 ans, on a su que le travail serait disponible toujours. Ceci n'est plus vraie. [...] La publicité a commencé a déringoler au debut de 1989, avant la reces... ... sion s'est mise à devenir grave. Grâce à sa incapacité de réagir au changements ... economiques, elle expérience des problèmes et son déclin est mondiale.

APPENDIX B Questionnaire on the Use of the French Grammar Checker: NAME: Please tick the relevant box for each question (most are on five-point scales, with the extremes of the scale described): 1. Did you find the grammar checker easy to use o o o o o difficult to use

2. Did you find its messages easy to understand o o o o o hard to understand

3. Did you trust its analysis on every occasion o o o o o never


CALICO Journal

Gabriel Jacobs and Catherine Rodgers

4. What percentage of mistakes do you think the grammar checker helped you correct: o 100-90% o 90-70% o 70-50% o 50-30% o less than 30%

5. Do you think the grammar checker is better at picking up some types of mistakes? o Yes o No

If Yes, please indicate here which types of mistake:

7. Did the grammar checker suggest any incorrect amendments? o Yes o No If No, go to 8

If Yes, to what extent did this happen? a lot o o o o o very little

If Yes, to what extent do you think you might have accepted incorrect amendments? a lot o o o o o very little

8. How useful was the grammar checker compared with a grammar book and a dictionary? much more useful o o o o o much less useful

9. In comparison with the combination of a grammar book and dictionary, is the grammar checker o easier to use o more difficult to use

10. Did you use the Explain button o systematically o often o from time to time o rarely o never If never, go to 11 If you used it, did you find its explanations on the whole: extremely useful o o o o o completely useless

Volume 16 Number 4


Treacherous Allies

11. If you had free, easy access to a grammar checker, do you think you would use it: o systematically o often o from time to time o rarely o never

12. If you used a grammar checker regularly, do you think you would make more or fewer mistakes in your written French: far fewer o o o o o many more

13. For student use, do you think it would be better for the French Department to invest in a grammar checker or in a dozen or so dictionaries/grammar books? o Grammar checker Please justify your answer: o Dictionaries/grammar books

14. If you wish, please add any further comments on your feelings about using a grammar checker, in particular a) why you might or might not use one in the future b) why you think the French Department should or should not make a grammar checker available. Please also state how much you think you learned by using the grammar checker. Continue on the back of this sheet if necessary.


CALICO Journal

Gabriel Jacobs and Catherine Rodgers



Grammar checkers and related products referred to in this paper are listed below with the names and addresses of their producers/distributors. For a listing of many available foreign language grammar checkers, see The Multilingual PC Directory, 1998. The reviews mentioned for the individual grammar checkers below have been selected both for the level of description they provide for the products in question and for judgements of them. For a comparison of Antidote, Correcteur 101, Cordial, Hugo Plus, and Pro Lexis, see Katz and Villaudy (1997) and SVM Micro (1998). For a comparison of Le Correcteur 101, GramR, Hugo Plus, and the French Proofing Tools for Word 6 for Windows, see Burston (1996). [Editor's note: This article was submitted before two other important grammar checker reviews were published. See Mogilevski (1998) and Burston (1998).] Antidote, Druide Informatique, 5515 chemin de la Côte-Saint-Luc, Montreal, Quebec, Canada H3X 2CS. See reviews by Fortin (1997), Tajtelboom (1997), and Tremblay (1997) [comparison with Correcteur 101]. Cordial, Synapse Développement, 23 rue Stalingrad, F­31000 Toulouse, France. (Le) Correcteur 101, Machina Sapiens, 3535 chemin de la Reine-Marie, Suite 420, Montreal, Quebec, Canada H3V 1H8. See reviews by Tremblay (1997) [comparison with Correcteur 101] and Mogilevski (1998) [comparison of Version 2.2 and 3.5 Pro]. Grammatique (WordPerfect Language Module), Corel Corporation, 567 East Timpanogos Parkway, Orem, Utah, 84097-6209, USA. See review by Vanneste (1994). GramR, Software Technologies International (Distribution Lurciel), Les Ulis, France. Distributed in the USA by Schuller & Associates, 1103 Dolores Street, San Francisco, CA 94110. See reviews by Gray (1996) and Vanneste (1994). Hugo Plus (Softissimo), distributed by Logidisque, 1225 rue de Condé, Montreal, Quebec, Canada H3K 2EE. See review by Vanneste (1994). Microsoft French Proofing Tool. For distributors, updates, etc. see http:// At the time of writing (September 1998), the latest version (for PC) was called the Microsoft Natural Language Grammar Checker. Pro Lexis, Editions Diagonal, 719 route de Bordinas, 06340 Cantaron, France. Sans Faute/Grammaire (BCDL), distributed by Hexacom, 2 route de Bergues, BP 73, 59412 Coudekerque-Branche Cédex, France. See review by Le Du (1996). Système-D, Heinle & Heinle, 20 Park Plaza, Boston MA 02116, USA. See review by Gray (1995).


According to the largest and most recent study of UK language students (Coleman, 1996), typical second-year students of French are in their tenth year of study of the language. They follow a curriculum that is very narrow compared to that of most other countries, taking only one or two subjects at university. They have covered much formal French grammar, though without complete mastery. French language makes up 30% to 40% of their study time for French, the rest consisting Volume 16 Number 4 527

Treacherous Allies

of `content' (literature, politics, geography, etc. of France). However, the students who took part in our trials were following a degree course consisting of two languages and Business Studies in which French formed only a third of their overall studies but with language study accounting for about 70% of their French study time.


The Microsoft product is not, in our view, the best French proofing tool. Antidote seems to us to offer the best tools, followed by Cordial and Le Correcteur 101 (see Burston, 1996; Katz & Villaudy, 1997).


Bolton, W. (1993). Checking the spellers. English Today, 9 (2), 45-50. Burston, J. L. (1996). A comparative evaluation of French grammar checkers. CALICO Journal, 13 (2-3), 104-111. Burston, J. L. (1998). Antidote 98. CALICO Journal, 16 (2), 197-212. Coleman, J. A. (1996). Studying languages: A survey of British and European students. The proficiency, background, attitudes and motivations of students of foreign languages in the United Kingdom and Europe. London: Centre for Information on Language Teaching and Research. EAGLES (Expert Assessment Group on Language Engineering Standards) (1995). Evaluation of natural language processing systems: Final report. Geneva: ISSCO, University of Geneva. Fortin, M. (1997, February and March). La potion magique d'Antidote. CLIC: Bulletin collégial des technologies de l'information et des communications, 14/15. Gray, E. F. (1996). Automatic checking of French grammar and spelling. French Review, 67, 543-545. Gueutal, H. G. (1989). Utilizing high technology: Computer-aided design and user performance. Information and Management, 17, 13-21. Katz, L., & Villaudy, B. (1997, October). Outils d'aide à la rédaction: Dossier et comparatif. PC Mag, 114. Le Du, B. (1996, July). Review of Sans Faute/Grammaire. Univers Mac, 58. Mogilevski, E. (1998). Le Correcteur 101: A comparative evaluation of version 2.2 and 3.5 pro. CALICO Journal, 16 (2), 183-196. Muir, B. M. (1994). Trust in automation. Part I. Theoretical issues in the study of trust and human intervention in automated systems. Ergonomics, 37 (11), 1905-1922. SVM Micro, 156 (1998, January) The Multilingual PC Directory. (1998). London: Knowledge Computing. Pinker, S. (1994). The language instinct. London: Penguin Books. Tajtelboom, J. (1997, December). Un traitement pour le texte. Génération PC. 528 CALICO Journal

Gabriel Jacobs and Catherine Rodgers

Tremblay, M. (1997, 14 April-12 May). Rectitude apolitique. Québec Micro, 3.04. Vanneste, A. (1994). Checking the grammar checkers. In L. Van Waes, E. Woudstra, & P. Van den Hove (Eds.), Functional communication quality, Utrecht Studies in Language and Communication 4 (pp. 208-219). Amsterdam/ Atlanta: Rodopi. Will, R. P. (1991). True and false dependence on technology: Evaluation with an expert system. Computers in Human Behavior, 7, 171-183.

AUTHORS' BIODATA Gabriel Jacobs is Professor of Informatics in the European Business Management School, University of Wales Swansea. He has published extensively in the CALL field and is Executive Editor of the Journal of the Association for Learning Technology (ALT-J). Catherine Rodgers is a Lecturer in French language and literature in the University of Wales Swansea. She has researched and published in the area of CALL, especially on aspects of videoconferencing for foreign language teaching.

AUTHORS' ADDRESSES Gabriel Jacobs European Business Management School University of Wales Swansea SA2 8PP, UK Phone: +44 (0)1792 295577 Fax: +44 (0)1792 295626. E-Mail: [email protected] Catherine Rodgers Department of French University of Wales Swansea SA2 8PP, UK Phone: +44 (0)1792 295973 Fax: +44 (0)1792 295710. E-Mail: [email protected]

Volume 16 Number 4


Treacherous Allies


CALICO Journal


22 pages

Report File (DMCA)

Our content is added by our users. We aim to remove reported files within 1 working day. Please use this link to notify us:

Report this file as copyright or inappropriate


You might also be interested in

Developing a new grammar checker for English as a second language