Automatic Genre-Specific Text Classification
each class, the definitions of the measures are as fol-
lows. A higher F1 value indicates better classification
performance.
• Precision is the percentage of the correctly clas-
sified positive examples among all the examples
classified as positive.
• Recall is the percentage of the correctly classi-
fied positive examples among all the positive
examples.
2 × Pr ecision × Re call
• F1 =
Pr ecision + Re call
(2) Findings and Discussions – The following are
the main four findings from our experiments.
First, SVM outperforms NB in syllabus classifica-
tion in the average case (Figure 2). On average, SMO
performed best at the F1 score of 0.87, 15% better than
NB in terms of the true syllabus class and 1% better in
terms of the false syllabus class. The best setting for our
task is SMO with the genre feature selection method,
which achieved an F1 score of 0.88 in recognizing true
syllabi and 0.71 in recognizing false syllabi.
Second, the kernel settings we tried in the experi-
ments were not helpful in the syllabus classification
task. Figure 2 indicates that SMO with kernel settings
perform rather worse than that without kernels.
Third, the performance with genre features settings
outperforms those with general features settings and
hybrid feature settings. Figure 3 shows this performance
pattern in the SMO classifier setting; other classifiers
Figure 2. Classification performance of different classifiers on different classes measured in terms of F1

show the same pattern. We also found that the perfor-
mance with hybrid features settings is dominated by the
general features among them. It is probably because the
number of the genre features is very small, compared
to the number of general features. Therefore, it might
be useful to test new ways of mixing genre features
and general features to take advantage of both of them
more effectively.
Finally, at all settings, better performance is achieved
in recognizing true syllabi than in recognizing false
syllabi. We analyzed the classification results with
the best setting and found that 94 of 313 false syllabi
were classified as true ones mistakenly. It is likely that
the skewed distribution in the two classes makes clas-
sifiers favor true syllabus class given an error-prone
data point. Since we probably provide no appropriate
information if we misclassify a true syllabus as a false
one, our better performance in the true syllabus class
is satisfactory.
FUTURE WORk
Although general search engines such as Google
meet people’s basic information needs, there are still
possibilities for improvement, especially with genre-
specific search. Our work on the syllabus genre suc-
cessfully indicates that machine learning techniques can
contribute to genre-specific search and classification.
In the future, we plan to improve the classification
accuracy from multiple perspectives such as defining