A COMPARATIVE EVALUATION OF FRENCH GRAMMAR CHECKERS

Jack Burston, Monash University

DESCRIPTION Four grammar checkers, all of French Canadian origin, were evaluated in this study: Le Correcteur 101 (Le Correcteur), GramR, Hugo Plus (Hugo), and the French Proofing Tools for Word for Windows Ver. 6 (French Proofing Tools). All the programs except GramR exist in both IBM-PC and Macintosh versions. The versions tested in this study are indicated in Table 1 (other available versions are indicated in parentheses) IBM-PC software was tested on a 486/66; Macintosh software was run on a Quadra 650. These programs will operate on lesser platforms, but with considerably slower response times. DOCUMENTATION All products include both printed and on-line documentation. With the exception of French Proofing Tools, which is in English, all are written in French. Comprehensiveness varies considerably. The Guide d'Utilisation for GramR is a cursory 27 pages and its on-line help is equally limited. The user's guide for French Proofing Tools is in reality a generic text used for all the foreign language versions of Proofing Tools for Word.1 Of a total 33 pages, a scant 16 are devoted to the Proofing Tool. Extensive on-line support, however, is available through the standard help menus which accompany the English grammar/spell checker, the operation of which is identical to its French counterpart. Both Le Correcteur (93 pp) and Hugo (140 pp) provide substantial printed documentation. On-line help for both is likewise quite satisfactory. It is to be noted that Hugo is unique in offering a choice between French or English f or the user interface language.2 The interface in all the other grammar checkers is monolingual: French in the case of GramR and Hugo, English in the French Proofing Tools.

CALICO Journal, Volume 13 Numbers 2 & 3

104

Lexical Resources Au x Price Correcteur 101 Mac Ver 2 (Windows) GramR DOS Ver 2 (Windows) Hugo Plus Mac Ver 7 (DOS/Win dows) Fren. Proof Tools Word Windows 6 (Mac) $160 Dic t 4 Dict Looku p 4 Thesaur Verb Conjug 4

Control Parameters Ling Levels 4 Typ o Feat 4 Style Feat 4 Aut o 4

Error Correction/Analysis Context Intera ct 4 Sensitive 4 Trace 4 Phra se Leve l 4

Gramm ar Help Context Sensitiv e 4

Word processing Edi t Prin t

$60

$60

$90

Table 1. French Grammar Checkers

FEATURES The features of the four programs are summarized in Table 1. As should be immediately apparent, Le Correcteur easily tops the list in terms of available features (and cost). For purposes of comparison, features are grouped into five main categories: 1.Lexical Resources All the programs incorporate spell checkers, of course, and have facilities for the creation of auxiliary word lists. Both Hugo and Le Correcteur permit dictionary word searches, but only Le Correcteur allows control over the exactness of matching. The French Proofing Tools is unique in offering a thesaurus. On the other hand, it is also the only program not to include a verb conjugator. 2. Control Parameters With the exception of GramR, all the grammar checkers allow control over the level of comprehensiveness of corrections (e.g. familiar, standard, formal), as well as whether or not to ignore typological features (e.g. capitalization, punctuation) and stylistic constraints (e.g. passive construction warnings). 3. Error Correction and Analysis All the grammar checkers operate interactively, i.e. they identify probable errors, then await user input before making any corrections. Le Correcteur and the French Proofing Tools also optionally perform completely automatic corrections. All the programs except GramR identify errors with specific contextual reference, e.g. une petite

CALICO Journal, Volume 13 Numbers 2 & 3

105

magasin": noun/ adjective agreement error, "magasin" is masculine; "petite" is feminine. All programs except the French Proofing Tools allow the tagging of locations where corrections have been made. Of all the grammar checkers, Le Correcteur is unique in being able to assess the well-formedness of texts below sentence level, i.e. clauses, phrase structures. In fact, it will attempt to analyse any highlighted sequence of words. 4. Grammatical Explanations All the programs incorporate an on-line reference grammar. With the exception of Le Correcteur, grammatical rules are presented generically and recycled as needed. Le Correcteur is the only grammar checker to provide detailed context sensitive grammar explanations. 5. Word Processing All the programs allow texts to be imported from a number of formats, e.g. ASCII, ANSI, WordPerfect. GramR, however, is particularly restricted in limiting text editing to the correction of identified errors. The French Proofing Tools, of course, can only be used within Word for Windows. Only The French Proofing Tools and Le Correcteur offer print out facilities. Texts corrected with GramR and Hugo must be exported to an external word processor for printing. PERFORMANCE EVALUATION Before going into details, it is worthwhile to comment first in general terms upon what a present-day grammar checker can, and cannot, do. As should be expected, the primary constraint limiting error correction is the scope of semantic analysis. Even the most sophisticated language parsers are restricted to sentence internal relationships. Discourse level features such as deixis (demonstrative pronouns and their antecedents), pronominal reference and tense usage, and errors associated therewith, are simply not within the power of a desk top grammar checker to detect. Lexical appropriateness can, at best, only be queried in the context of troublesome homonyms (e.g. voie / voix), paronyms (e.g. conjecture / conjoncture) and anglicisms (e.g. actuellement / actually). Within a sentence, only orthography, inflectional morphology, head/modifier agreement phenomena, and the most nuclear syntactic dependencies (e.g., verb/subject/object) can be treated. In the virtual absence of semantic analysis, even this requires an algorithm capable of correctly applying several thousand rules3 (whence the need for substantial RAM and very fast computer CPU). Table 2 contains

CALICO Journal, Volume 13 Numbers 2 & 3

106

examples representative of the kinds of sentence internal mistakes which passed undetected through all the French grammar checkers. Not surprisingly, because of its inherent semantic nature, pronoun usage goes unmonitored. So, too, given their intrinsically non-nuclear character, errors involving prepositional phrase structure are invisible to the grammar checkers. Dans ma famille ce sont cinq personnes Jattends de ta lettre prochaine. On a besoin de solitude, et le silence. Sur le weekend, jcoute la musique classique et le jazz. Il y a beaucoup de choses ce que jaime faire. Je frquente une librairie, o je passe le temps lire les nouvelles sorties. Jaime relcher en famille. Nous coutons la musique populaire, le jazz et la musique classique. Ils sont en vacances en le centre dAustralie. cris moi bientt. Nous avons fair du camping pendant une semaine, et nous avons eu un bon temps. Jaime jouer des sports pendant mon temps libre. Jadore mes soeurs et ses maris. Je voyage souvent en Australie ou en outre-mer. Je vais au gymmnase pour faire lexercice. Nous parlons autour de nos amis, de nos vies et de tout le monde. Jai oubli dire que jaime toutes varits des musiques et des films. Table 2. Undetectable Error Types TEST DATA The French grammar checkers were tested against 30 compositions written by 10 first semester intermediate level university students (i.e. third semester ab initio). The essays, each about 200 words in length, consisted of a pen-pal letter, a personal portrait, and a description of leisure activities. The texts were originally composed on a word processor which incorporated the French Assistant suite of writing tools.4 All compositions were final drafts which had previously undergone peer group correction in class. Despite this prior revision, there was, alas, no shortage of errors of the types the grammar checkers could reasonably be expected to handle. By far, the greatest number of residual errors awaiting detection were those involving low level morphology (spelling, diacritics, hyphenation, capitalization, elision, preposition+article contraction) and gender-based form misassignments (e.g. le

CALICO Journal, Volume 13 Numbers 2 & 3

107

maison). Out of a total 164 errors detected, two thirds (108) fell into this category. Verb form errors (conjugation, subject agreement, auxiliary verb in compound tenses, past participle agreement, indicative/subjunctive mood) accounted for another 32 mistakes. Misagreement of nouns and their modifiers added another 24 errors to the tally. A summary of the performance of the four grammar checkers is presented in tables 3 and 4.
Low Level Morphology (n=108) Detected Selection Flag MisFirst Man Only Corr 87 9 5 3 Verbal Forms )n=32) Detected Selection Flag MisFirst Man Only Corr 18 2 6 2 Nominal Agreements (n=24) Detected Selection Flag MisFirst Man Only Corr 22 0 0 1

NonDect 4

NonDect 4

NonDect 1

Correcteur 101 GramR Hugo Plus Fren. Proof Tools

False Warnings 12

31 70 83

20 14 9

28 6 1

0 3 1

29 15 14

2 7 0

3 9 21

11 1 8

1 2 0

15 13 3

6 11 1

0 4 18

12 0 0

0 2 0

6 7 5

5 10 6

Table 3. Error Detection/Correction

Detectable Errors (N=164) Total Detected N Correcteur 101 GramR Hugo Plus Fren. Proof Tools 155 114 129 142 % 95 70 79 87 First Selection N 127 39 88 84 % 82 34 68 59 Detected Manual Selection N % 11 23 25 48 7 20 20 34 Non-detected Flag Only N 11 51 7 9 % 7 45 6 6 Mis-corrected N 6 1 7 1 % 4 1 6 1 N 9 50 35 22 % 5 30 21 13 False Warnings N 12 5 10 6 % of detects 8 4 8 4

Table 4. Detectable Errors In evaluating the grammar checkers, the first consideration was whether or not they detected the errors within each category. If so, their success in dealing with the mistake was then noted, of which there were four possibilities. The most efficient form of correction, labelled "Selection/First" in the tables, was that in which the first (or only) suggestion in a list of possibilities was the appropriate one. Correction termed "Selection/Man" required either manually choosing a non-initial suggestion or entering the required form on the basis of an explicit instruction (e.g. "the verb tre must agree with its subject vous"). The least helpful form of error flagging simply gave a general warning with no indication of what to do about it (e.g. "noun/adjective agreement

CALICO Journal, Volume 13 Numbers 2 & 3

108

error"). Miscorrections, which resulted from tl-le misdiagnosis of problems, constituted the fourth type of error identification. Lastly, besides the genuine errors detected, the programs also on occasion issued false warnings indicating the presence of mistakes where in reality none existed. As was the case in the comparison of program features, the summary of actual grammar checking performance presented in Tables 3-4 unequivocally demonstrates the superiority of Le Correcteur over its competitors. Of the 164 sentence internal morphosyntactic errors contained in the student essays, Le Correcteur was able to detect 95%. Its ability to correct the errors it detected was equally impressive, with 82% rectified through an appropriate first selected option and another 7% through manual intervention. As will also be observed, such performance comes at the expense of a higher rate of miscorrections and false warnings. It is to be noted, however, that suggested modifications in Le Correcteur are always accompanied by a detailed syntactic parse, consultation of which clearly indicates where it has gone astray. Needless to say, the ability to profit from such grammatical analysis will vary in proportion to the user's comprehension of traditional metalinguistic terminology. CONCLUSION On the basis of documentation, features and performance, LeCorrecteur 101 is unquestionably the best French grammar checker on the market and certainly worth its premium cost. Where funds are limited, and Word for Windows Ver. 6 is already installed, the French Proofing Tools also merits consideration. While it requires users to perform many more corrections themselves, its error detection rate is nonetheless quite respectable and it includes a very useful thesaurus. Though much less costly than their competitors, Hugo Plus and GramR lag considerably behind in their ability to detect errors and thus cannot be recommended. TECHNICAL INFORMATION Le Correcteur 101 Publisher: Machina Sapiens 3290, avenue Lacombe Montral, Qubec Canada H3T lL7

Telephone (514) 733-1095 Fax (514) 733-2774

CALICO Journal, Volume 13 Numbers 2 & 3

109

GramR Publisher:

EDIT inc. 1253, avenue McGill College Telephone (514) 877-4200 Bureau 450 Fax (514) 877-9890 Montral, Qubec Canada H3B 2Y5

Hugo Plus Publisher:

Logidisque C.P. 10, Succursale D Montral, Qubec Canada H3K 2E4

Fax (514) 933-2182

French Proofing Tools for Word for Windows Version 6 Publisher: (under license from Microsoft Corporation) Alki Software Corporation 300 Queen Anne Ave N. Telephone (206) 286-2600 Suite 410 Fax (206) 286-2785 Seattle, Washington USA 98109 NOTES
1

It should be noted that only the French version includes a grammar checker; only a spell checker and thesaurus is offered with other languages. Hugo also senses the presence of English text and automatically switches over to spell/grammar check in English. Le Correcteur 101 claims to make use of over 3500 syntactic and 500 agreement rules. Cf. Burston (1991), Burston, et al. (1995)

REFERENCES Burston, J. (1991). "Software Report: French Language Assistant (Ver 4.04)." CALICO Journal 9, 2, 69-75. Burston, J., et al. (1995). "Software Report: Language Assistant Re-Viewed (DOS Version 5.1/Windows)." CALICO Journal 12, 2 & 3, 95-113.

CALICO Journal, Volume 13 Numbers 2 & 3

110

AUTHOR'S BIODATA Jack Burston (B.A./M.A. in French Ph.D. in Linguistics) is Senior Lecturer in French, Department of Romance Languages, Monash University. The author teaches a postgraduate course in CALL and supervises M.A. and Ph.D. candidates in this area. He is particularly interested in authoring languages and systems for courseware development and is currently Chair of the CALICO Hypermedia SIG. AUTHOR'S ADDRESS Department of Romance Languages Monash University Clayton, Victoria 3168 AUSTRALIA Phone: (61-3) 905-2222 Fax: (61-3) 905-2137 E-Mail: jburston@arts.cc.monash.edu.au

CALICO Journal, Volume 13 Numbers 2 & 3

111