Word Sense Diambiguation

Mentor - Prof. Harish Karnick

Shanu Kumar Harshit Saini Praphul Singh Robin Singh

150659 150279 150513 150591

1 Problem Statement
In Natural Language Processing, Word Sense Disambiguation is the problem of determining which
”sense” (meaning) of a word being used in a particular context, a process that works unconsciously
in humans majority of the time. Given a word with its possible senses, classify a word in context
into one or more of its sense classes.

2 Models
2.1 Model-1

Embedding Layer In each model, the words are represented in the forms of float vectors which are
called word embeddings. In the embedding layer each word of the sentence is represented in the
form of word embeddings.

Bidirectional LSTM Layer The word embeddings of a sentence is then fed into a bidirectional
two-layered LSTM to obtain sequential informations.

Attention Layer We obtain N vectors h1 , h2 · · · hN − 1, hN

3 Four Word Dataset

Word #Senses Total # of examples Distribution across Senses

hard 3 4333 (3455, 502, 376)
serve 4 4378 (1814, 1272, 853, 439)
interest 6 2368 (1252, 500, 361, 178, 66, 11)
line 6 4146 (2217, 429, 404, 374, 373, 349)

Table 1: Senseval-2 Four-Words Dataset [Senseval 2, 2001]

F1-Score Accuracy
Target Word
Train Val Test Train Val Test
Hard 89.45% 78.66% 78.11% 94.85% 89.37% 89.78
Serve 95.75% 89.80% 89.84% 96.49% 95.5% 91.94
Interest 84.32% 80.50% 72.33% 92.06% 89.06% 86.16
Line 87.98% 82.45% 78.73% 92.08% 88.75% 86.33

Table 2: Four Word Dataset Results

1
 Figure 1: One Word Model (Model 1)

4 One Million Data

• Model-1: Basic Model with two bidirectional LSTMs same
• Model-2: Classify POS Tags for all the word in the sentence to improve grammatical
context
• Model-3: Used Conditional Random Field Classifier instead of Softmax Classifier for clas-
sifying POS tags
• Model-4: Hierarchical Model(POS Tags)

Word #Senses # of examples Distribution across Senses

serve 4 3421 (1941#V1, 1272#V2, 853#V3, 439#V4)
place 6 3511 (1149#N1, 623#V1, 490#V2, 488#N2, 479#V3, 282#N3)
place 6 3511 (1149#N1, 623#V1, 490#V2, 488#N2, 479#V3, 282#N3)

Table 3: One Million Dataset: SemCor+OMSTI

2
Figure 2: Hierarchical Model (Model 4)

Figure 3: One Word Model (Model 2)

3
Figure 4: One Word Model using CRF (Model 3)

4
 F1-Score
Model
Train(POS) Val(POS) Train(Sense) Val(Sense)
Model-1 — — 90.812 77.19%
Model-2 83.84 82.45 88.162 80.29%
Model-3 82.99 83.18 91.14 77.45%

Table 4: Sense word: ”Serve” results(F1-Score)

Accuracy
Model
Train(POS) Val(POS) Train(Sense) Val(Sense)
Model-1 — — 91.034% 79.69%
Model-2 91.79% 91.45% 88.021% 81.563
Model-3 91.64% 91.73% 92.26% 80.78%

Table 5: Sense word: ”Serve” results(Accuracy)

F1-Score
Model
Train(POS) Val(POS) Train(Sense) Val(Sense)
Model-2 84.37 84.65 86.80 82.26%
Model-3 90.20 90.31 86.78 79.06%
Model-4 97.76 95.27 60.11 57.84%

Table 6: Sense word: ”Place” results(F1-Score)

Accuracy
Model
Train(POS) Val(POS) Train(Sense) Val(Sense)
Model-2 86 — 91.034% 79.69%
Model-3 91.79% 91.45% 88.021% 81.563
Model-4 97.78% 95.31% 64.28% 61.56%

Table 7: Sense word: ”Place” results(Accuracy)

F1-Score
Model
Train(POS) Val(POS) Train(Sense) Val(Sense)
Model-2 84.89 74.32 77.61 74.32%
Model-3 84.76 82.87 93.92 89.07%
Model-4 96.85 94.14 72.51 69.33%

Table 8: Sense word: ”Force” results(F1-Score)

Accuracy
Model
Train(POS) Val(POS) Train(Sense) Val(Sense)
Model-2 92.17 91.91 93.27% 89.48%
Model-3 91.80% 91.48% 94.37% 89.91
Model-4 98.37% 97.01% 87.00% 84.09%

Table 9: Sense word: ”Force” results(Accuracy)

5
Sense F1 Score Accuracy
Model
Word Train Val Train Val
Model-1 91.97% 88.35% 92.43% 88.92%
Model-2 77.71% 74.34% 93.24% 89.20%
Force
Model-3 95.39% 90.34% 95.62% 91.19%
Model-4 93.00% 88.87% 93.26% 89.49%
Model-1 -% -% -% -%
Model-2 86.80% 82.26% 88.12% 83.12%
Place
Model-3 86.79% 79.06% 88.26% 80.94%
Model-4 87.36% 80.43% 88.66% 82.18%
Model-1 -% -% -% -%
Model-2 78.31% 72.68% 86.81% 82.81%
Point
Model-3 81.55% 69.74% 89.38% 82.81%
Model-4 82.41% 70.67% 88.55% 81.05%
Model-1 -% -% -% -%
Model-2 86.80% 82.26% 88.12% 83.12%
Open
Model-3 86.79% 79.06% 88.26% 80.94%
Model-4 87.36% 80.43% 88.66% 82.18%
Model-1 -% -% -% -%
Model-2 86.80% 82.26% 88.12% 83.12%
Support
Model-3 86.79% 79.06% 88.26% 80.94%
Model-4 87.36% 80.43% 88.66% 82.18%

Table 10: Results on One Million Dataset: SemCor+OMSTI