Endometrium segmentation on transvaginal ultrasound image using

key-point discriminator
Hyenok Park*, Hong Joo Lee*, Hak Gu Kim, and Yong Man Roa)
School of Electrical Engineering, KAIST, Daejeon 34141, Republic of Korea
Dongkuk Shin
Medical Image Development Group, R&D Center, Samsung Medison, Seongnam 13530, Republic of Korea
Sa Ra Lee
Department of Obstetrics and Gynecology, Ewha Womans University School of Medicine, Seoul 07985, Republic of Korea
Sung Hoon Kim
Department of Obstetrics and Gynecology, University of Ulsan College of Medicine, Asan Medical Center, Seoul 05505, Republic of
Korea

Mikyung Kong
Department of Obstetrics and Gynecology, Yonsei University College of Medicine, Seoul 03722, Republic of Korea
(Received 25 November 2018; revised 6 June 2019; accepted for publication 6 June 2019;
published 31 July 2019)
Purpose: Transvaginal ultrasound imaging provides useful information for diagnosing endometrial
pathologies and reproductive health. Endometrium segmentation in transvaginal ultrasound (TVUS)
images is very challenging due to ambiguous boundaries and heterogeneous textures. In this study,
we developed a new segmentation framework which provides robust segmentation against ambiguous
boundaries and heterogeneous textures of TVUS images.
Methods: To achieve endometrium segmentation from TVUS images, we propose a new segmenta-
tion framework with a discriminator guided by four key points of the endometrium (namely, the
endometrium cavity tip, the internal os of the cervix, and the two thickest points between the two
basal layers on the anterior and posterior uterine walls). The key points of the endometrium are
defined as meaningful points that are related to the characteristics of the endometrial morphology,
namely the length and thickness of the endometrium. In the proposed segmentation framework, the
key-point discriminator distinguishes a predicted segmentation map from a ground-truth segmenta-
tion map according to the key-point maps. Meanwhile, the endometrium segmentation network pre-
dicts accurate segmentation results that the key-point discriminator cannot discriminate. In this
adversarial way, the key-point information containing endometrial morphology characteristics is
effectively incorporated in the segmentation network. The segmentation network can accurately find
the segmentation boundary while the key-point discriminator learns the shape distribution of the
endometrium. Moreover, the endometrium segmentation can be robust to the heterogeneous texture
of the endometrium. We conducted an experiment on a TVUS dataset that contained 3,372 sagittal
TVUS images and the corresponding key points. The dataset was collected by three hospitals (Ewha
Woman’s University School of Medicine, Asan Medical Center, and Yonsei University College of
Medicine) with the approval of the three hospitals’ Institutional Review Board. For verification, five-
fold cross-validation was performed.
Result: The proposed key-point discriminator improved the performance of the endometrium seg-
mentation, achieving 82.67 % for the Dice coefficient and 70.46% for the Jaccard coefficient. In
comparison, on the TVUS images UNet, showed 58.69 % for the Dice coefficient and 41.59 % for
the Jaccard coefficient. The qualitative performance of the endometrium segmentation was also
improved over the conventional deep learning segmentation networks. Our experimental results indi-
cated robust segmentation by the proposed method on TVUS images with heterogeneous texture and
unclear boundary. In addition, the effect of the key-point discriminator was verified by an ablation
study.
Conclusion: We proposed a key-point discriminator to train a segmentation network for robust seg-
mentation of the endometrium with TVUS images. By utilizing the key-point information, the pro-
posed method showed more reliable and accurate segmentation performance and outperformed the
conventional segmentation networks both in qualitative and quantitative comparisons. © 2019 Ameri-
can Association of Physicists in Medicine [https://doi.org/10.1002/mp.13677]

Key words: adversarial learning, endometrial region, key-point-guided discriminator, medical image
segmentation, transvaginal ultrasound (TVUS) image

3974 Med. Phys. 46 (9), September 2019 0094-2405/2019/46(9)/3974/11 © 2019 American Association of Physicists in Medicine 3974
3975 Park et al.: Endometrium segmentation on TVUS image
1. INTRODUCTION
Ultrasonography is a valuable noninvasive imaging modality.
It provides real-time high-resolution images that can be used
for diagnosis.1–4 In particular, as one type of pelvic ultra-
sound, transvaginal ultrasonography (TVUS) is widely used
to examine female reproductive organs including the uterus,
cervix, and endometrium.2 TVUS images of the uterus pro-
vide gynecologist with useful information such as the length
and thickness of the uterus and endometrium, the texture of
the endometrium, the type of uterus position (e.g., flexion,
versions), and so on. For example, gynecologists can detect
uterine myomas or polyps from a texture analysis of the endo-
metrium and diagnose infertility by considering the thickness
of the endometrium.4–9
To examine the endometrial region, the general exami-
nation is based on manual segmentation. It is labor inten-
sive and time-consuming. Moreover, delineating the
borders of the endometrial region might be different from
doctor to doctor due to unclearness of boundary of the
TVUS image. Thus, automated and consistent guidance in
identifying the endometrial region is beneficial for gyne-
cologists. Moreover, the segmentation needs to automati-
cally provide pixel-wise annotation of the region of the
endometrium.
Recently, deep learning-based approaches have shown
great success in image processing and computer-aided
applications.10–13 In particular, many studies on automatic
segmentation with natural images using deep convolutional
neural networks have achieved outstanding performances.
Long et al. applied a fully convolutional network (FCN) to
powerful existing CNNs (e.g., AlexNet and VGG16) for
object segmentation by replacing fully connected layers
with fully convolutional layers.14 The FCN achieved nota-
ble performance in natural image segmentation. U-Net is
an encoder–decoder network in which features on the
encoder are transferred to the decoder layers through skip
connections.15 The skip connection maintains the details of
the prediction by concatenating the features of the encoder
and the decoder.16 SegNet17 and DeconvNet18 also utilize
features from low-level layers of the encoder in a similar
manner. Chen L-C et al. applied atrous convolution to
increase receptive field without decreasing the resolution
of the feature map.19 They also tried to aggregate contex-
tual information by utilizing atrous convolution or pooling
at different rates.20,21 Some papers reported organ segmen-
tation using medical images.22–24 Sahiner et al.22 investi-
gated the deep learning in medical image segmentation.
Dalmısß et al.23 developed an automated segmentation
method on breast and fibroglandular tissue in MRI vol-
ume. They developed two and three consecutive U-nets.
Jin et al.24 proposed a fast segmentation algorithm for kid-
neys by combining the random forests and ferns.
The deep learning-based segmentation methods reported
good performances in natural image and some organ segmen-
tation tasks. However, as shown in Fig. 1, the endometrium
boundaries on the TVUS images are ambiguous and the
Medical Physics, 46 (9), September 2019
3975
image has a heterogeneous texture. Therefore, it is difficult to
get precise segmentation results on the TVUS image.
In this paper, we utilize key-point information in order
to segment precisely. The key points are the points con-
sidered to be meaningful to the endometrial morphology
characteristics, such as the length and thickness of the
endometrium. Gynecologists annotate key points to deter-
mine the thickness and length of the endometrium as
shown in the third column of Fig. 1. To utilize the key-
point information of the endometrium, we propose a new
deep learning framework to accurately identify the bound-
ary of the endometrium. The proposed deep learning
framework consists of a segmentation network and a dis-
criminator network. The proposed discriminator is guided
by key-point maps of the endometrium and spatial
encoded features of the ultrasound image to be seg-
mented. The key-point discriminator determines whether
or not the predicted endometrium region coincides with
the ground-truth endometrium key points. At the same
time, the key-point discriminator determines whether the
predicted endometrium region corresponds closely to that
region of the ultrasound image. Through the adversarial
learning scheme,25,26 the proposed segmentation network
tries to predict the endometrium region more precisely.
Then, the key-point discriminator tries to distinguish the
predicted region from the ground-truth. This adversarial
learning scheme allows the segmentation network to be
robust to the unclear edges and heterogeneous texture of
the endometrium and generates an improved segmentation
results.
Another challenge in endometrium segmentation is that
the endometrium has various sizes. The size of the endome-
trium on TVUS images varies in a large range depending on
the patients and the menstrual phase. For example, the size of
the endometrium on early-proliferative phase is significantly
thinner than in the other phase. To address this issue, we uti-
lize atrous convolution to maintain the resolution of the
encoded feature and, yet enlarge the receptive field.19,27 We
also incorporate an atrous spatial pyramid pooling (ASPP)
module on the top of the extracted feature to segment various
size of the endometrial region.20,21
To sum up, in this paper, we propose a new deep learning
framework consisting of a novel discriminator that is guided
by key points of the endometrium for the precise and robust
endometrium segmentation in TVUS images. For the design
of the segmentation network, we utilize atrous convolution
and an ASPP module to deal with the various sizes of the
endometrium.
Our experimental results show that the proposed segmen-
tation method provides more accurate endometrium segmen-
tation results than the conventional segmentation networks.
The rest of this paper is structured as follows. In Section 2,
we explain the details of the proposed endometrium segmen-
tation network and discriminator. Section 3 describes our
experiments and provides the results. In Section 4, we discuss
the results, strength, and limitation of the proposed scheme.
Section 5 draws conclusions.
3976 Park et al.: Endometrium segmentation on TVUS image 3976

FIG. 1. TVUS images of uterus. Note that the endometrium region is highlighted on the corresponding image for easier understanding. The key points define the
length and thickness of the endometrium. For simple visualization, the key points are on the same channel.

2. MATERIALS AND METHODS 2.A. Materials

In the proposed method, the endometrium segmentation
network is trained with the key-point discriminator in an
adversarial way. Fig. 2 shows an overview of the proposed
deep learning framework for the endometrium segmentation
on TVUS images. The endometrium segmentation network is
shown in a gray dotted box. The input ultrasound image, I, is
fed into the endometrium segmentation network, which con-
sists of the encoder, ASPP module,20,21 and decoder. The
encoder extracts features from the input image. The endome-
trium segmentation network predicts the region of the endo-
metrium, Y.^ The key-point discriminator takes the predicted
endometrium map, Y, ^ or the ground-truth endometrium map,
Y, in turns. The key-point discriminator is designed to distin-
guish real or fake segmentation map based on two guidelines.
The first guideline is the encoded spatial features, fconv3, from
the encoder. The second guideline is the ground-truth key-
point map, P, which is fed into the key-point discriminator.
These guidelines are devised to assist the discriminator to
examine the predicted endometrium region more precisely.
We performed experiments to evaluate the proposed endo-
metrium segmentation on a TVUS dataset. The dataset was
collected by three hospitals under the approval of the three
hospitals’ Institutional Review Board. The TVUS dataset
contains endometrium images and the corresponding pixel-
level annotations of the endometrium. The TVUS image reso-
lution is 256 9 320. The dataset also contains four key
points (the endometrium cavity tip, the internal os of the cer-
vix, and the two thickest points between the two basal layers
on the anterior and posterior uterine walls). The key points
are related to endometrial morphology characteristics,
namely, the length of the endometrial cavity and the endome-
trial thickness. The dataset was manually annotated by the
expert gynecologists. The dataset contains 3,372 images. Fur-
thermore, data augmentation is employed to generate a larger
number of effective images. Each image in the training set is
flipped horizontally and vertically, scaled by a random ratio
from 0.9 to 1.2 and rotated by a random degree between

FIG. 2. Overview of the proposed network for endometrium segmentation with the proposed discriminator guided by key-point maps. [Color figure can be viewed
at wileyonlinelibrary.com]

Medical Physics, 46 (9), September 2019

3977 Park et al.: Endometrium segmentation on TVUS image 3977

20° and 20°. In sum, data augmentation generates about TABLE I. The detailed structure of the endometrium segmentation network.
380,000 augmented training images in a fold.
Filter/stride/dilation
Layer name rate Output shape
2.B. Endometrium segmentation network Input image Input size: 256*320*1 -
The endometrium segmentation network consists of an Encoder Conv conv [3, 3]/ 1 128*160*64
encoder, an ASPP module, and a decoder as shown in Fig. 3. Block 1 conv [3, 3]/ 1
max pool[2, 2]/ 2
The detail structure of the endometrium segmentation net-
Encoder Conv conv [3, 3]/ 1 64*80*128
work is shown in Fig. 3. Block 2 conv [3, 3]/ 1
The encoder is composed of five convolutional blocks. max pool[2, 2]/ 2
Max-pooling is applied after each convolutional block except Encoder Conv conv [3, 3]/ 1 32*40*256
for the last two convolutional blocks. Atrous convolution is Block 3 conv [3, 3]/ 1
applied with a rate of 2 at the fourth convolutional block to max pool[2, 2]/ 2
maintain the resolution of the feature for a thin, small Skip connection Conv [1, 1]/ 1 64*80*256
endometrial region. Therefore, the size of the extracted fea- Encoder Conv conv [3, 3]/ 1/ 32*40*512
ture from the encoder is one-eighth of that of the input image. Block 4 rate = 2
conv [3, 3]/ 1/
To deal with the various sizes of the endometrium, an ASPP rate = 2
module is incorporated on top of the encoded feature to con-
Encoder Conv conv [3, 3]/ 1 32*40*512
sider multiscale information effectively.20,21 For the decoder Block 5 conv [3, 3]/ 1
part, the multiscale features from the ASPP module are con- ASPP module Concat[Conv [1, 1], 32*40*256
catenated and fed into an 1 9 1 convolutional layer. The Conv [3, 3]/ 1/
multiscale information is adaptively aggregated through the rate = 8,
1 9 1 convolution layer. The aggregated feature is then Conv [3, 3]/ 1/
rate = 12,
upsampled two times and concatenated with the feature from Conv [3, 3]/ 1/
the third convolutional block to recover the details. Finally, rate = 24,
two convolutional layers are applied, and then, the feature is Image pooling]
upsampled four times to obtain the predicted endometrium Conv [1, 1]/ 1
map, Y.^ The input image I 2 R256932091 is fed to the endo- Decoder Conv Upsample by 2 64*80*256
metrium segmentation network. The network predicts the Block 1
probability map of the endometrium, Y ^ 2 R256932091. It is Decoder Conv Concat[Decoder Conv 64*80*256
Block 2_1 Block 1, Skip
written as connection]
^ ¼ sigmoidðf ðIjhÞÞ;
Y (1) Conv [3, 3]/ 1
Conv [3, 3]/ 1
where f and h are the function of the endometrium segmenta- Decoder Conv Conv [3, 3]/ 1 64*80*1
tion network and corresponding parameters, respectively. Block 2_2
Table I summarizes the detailed structure of the endometrium Decoder Conv Upsample by 4 256*320*256
Block 2_3
segmentation network.

2.C. Discriminator guided by key-points

applied at the last convolutional layer of each block except for
Fig. 4 shows the structure of the proposed discriminator the last layer. The global average pooling and fully connected
guided by the key points of the endometrium. The discrimi- layer are employed to get a score that discriminates the input
nator consists of five convolutional blocks. Stride size = 2 is as real or fake.28

FIG. 3. The structure of the proposed endometrium segmentation network. [Color figure can be viewed at wileyonlinelibrary.com]

Medical Physics, 46 (9), September 2019

3978 Park et al.: Endometrium segmentation on TVUS image 3978

FIG. 4. The structure of the proposed key-point discriminator [Color figure can be viewed at wileyonlinelibrary.com]

The discriminator is guided by the two constraints, the For guiding the discriminator with the key points, the
ground-truth endometrium key-point maps P and the spatial ground-truth key-point maps are concatenated with the input
encoded feature fconv3, from the encoder of the endometrium endometrium segmented map which can be either the pre-
segmentation network. The ground-truth endometrium key- dicted or the ground-truth region of the endometrium. Fur-
point maps P 2 R2563204 has four channels as shown in thermore, the spatial encoded features from the third
Fig. 4. Each channel corresponds to each key point of the convolutional layer, fconv3, whose spatial resolution is one-
endometrium, which is on the boundary of the endometrium fourth of the original input resolution, are fed into the dis-
region and defines the length and thickness of the endome- criminator. The spatial encoded feature helps the discrimina-
trium. As shown in Fig. 4, we put a Gaussian blob on each tor to determine whether the predicted endometrium region
point. The Gaussian blob is defined as follows. Let (xi, yi) corresponds to the input ultrasound image. Table II shows the
denote the coordinates of each key point, where detailed structure of the key-point discriminator.
i 2 f1; 2; 3; 4g. Let r denotes the standard deviation of the
Gaussian blob in a pixel unit. The intensity of the Gaussian
blob on a certain point (x, y) is written as
TABLE II. The detailed structure of the key-point discriminator.
2 2
ðx
xi Þ þ ðy
yi Þ
f ðx; yÞ ¼ expð
Þ: (2) Layer name Filter/stride Output shape
2r2
Input Input size: 256*320*1
The intensity indicates the probability of each key point. It
Discriminator Conv Concat[Endometrium segmented 128*160*64
is effective to provide localization information of the key block 1 map,
points using Gaussian blobs rather than fixed coordinates as GT endometrium Key-Point
shown in Fig. 5, which could allow boundary ambiguity in map]
the training phase. The ground-truth key-point maps have conv [3, 3]/ 1
four channels, which is the number of key points of the endo- conv [3, 3]/ 1
conv [3, 3]/ 2
metrium.
Discriminator Conv conv [3, 3]/ 1 64*80*128
block 2 conv [3, 3]/ 1
conv [3, 3]/ 2
Discriminator Conv Concat[Discriminator 32*40*256
block 3 Conv block 2, Spatial encoded
feature]
conv [3, 3]/ 1
conv [3, 3]/ 1
conv [3, 3]/ 2
Discriminator Conv conv [3, 3]/ 1 16*20*512
block 3 conv [3, 3]/ 1
conv [3, 3]/ 2
Discriminator Conv conv [3, 3]/ 1 16*20*512
block 3 conv [3, 3]/ 1
FC Global Average Pooling 1
FIG. 5. Example of a key point (left) and corresponding Gaussian blob (right) Fully Connected Layer
(r = 2).

Medical Physics, 46 (9), September 2019

3979 Park et al.: Endometrium segmentation on TVUS image
2.D. Objective functions for training proposed
networks
The objective functions for training the endometrium seg-
mentation network are as follows. The Dice loss is used as
the first objective function.29 N denotes the total number of
the pixels of predicted probability map. yi and ^yi denote the
ground-truth and predicted probability at the ith pixel, respec-
tively. i is 1, 2, . . ., N. The Dice loss is written as
P tor, we compared the performance of the proposed method
2 Ni¼1 yi  ^y
Ldice ¼
PN PN : (3)
i¼1 yi þ i¼1 ^
yi
The second objective function for training the endome-
trium segmentation network is an adversarial loss derived
from the discriminator.25,26,28 It is written as follows.
^ f conv3 ÞÞ;
Ldis;seg ¼
E^y  p^y ½logðDðYjP; (4)
where DðÞdenotes the discriminator. By minimizing Ldis,seg,
the endometrium segmentation network tries to predict accu-
rate map that corresponds to key-point map. To sum up, the
total loss, Lseg for training the endometrium segmentation
network is written as
Lseg ¼ Ldice þ Ldis;seg (5)
For training the discriminator, the objective functions are
written as.
Ldis ¼
Ey  py ½logðDðYjP; f conv3 ÞÞ
E^y  p^y ½logð1
^ f conv3 ÞÞ:

DðYjP; (6)
By minimizing Ldis, the discriminator strictly evaluates the
predicted map based on the key-point maps and the spatial
encoded features. To sum up, the discriminator minimizes
Ldis and the endometrium segmentation network minimizes
Lseg alternatively.
After the endometrium segmentation network and key-
point discriminator are trained in the training phase, only the
endometrium segmentation network is used for the segmenta-
tion inferring phase.
3. RESULTS AND DISCUSSION
3.A. Experimental setting
During the training, TVUS images were fed into the endo-
metrium segmentation network with a batch size of 8. The
endometrium segmentation network was trained with the key-
point discriminator. We employed the Adam optimizer30 for
training the endometrium segmentation network and the dis-
criminator. Similar to Pezeshk et al., we determined the
parameters of the network, such as the number of iterations
and learning rates on a validation set in a single fold. Then,
we fixed the parameters of the network for each fold.31 We
used 0.0001 for the endometrium segmentation network
learning rate and 0.0005 for the key-point discriminator net-
work learning rate. We trained the key-point discriminator
three times and the segmentation network eight times. For
Medical Physics, 46 (9), September 2019
3979
training the key-point discriminator, the weights of the seg-
mentation network were fixed. We obtained convergence after
around 44,000 iterations in total with a batch size of 8. We
set r as 2 which was acceptable considering the size of the
endometrium. In the experiments, fivefold cross-validation
was performed. Each fold contains 672 images. We used each
fold as a test set and the other folds (4 9 672 = 2688
images) for the training.
To verify the effectiveness of the proposed discrimina-
with that of conventional deep learning segmentation net-
works such as UNet15 and FCN8s14 on the same dataset.
These baseline networks took the ultrasound images as
input, with the same resolution of 256 9 320 and pre-
dicted the segmented probability map of the endometrium
as output. For a fair comparison, we used the same aug-
mentation method, batch size, and optimizer for training
the baseline networks.
3.B. Quantitative performance evaluation
For the quantitative performance comparison, we com-
pared our proposed method with conventional deep learning
segmentation network, namely, UNet and FCN8s. Table III
shows the quantitative results on the Dice coefficient and Jac-
card coefficient. Note that values in Tables III and IV are
average of results on a fivefold cross-validation test set.
The proposed endometrium segmentation network trained
with the key-point discriminator achieved 82.67% for the
Dice coefficient and 70.46% for the Jaccard coefficient,
exceeding the performance of the FCN8s by 4.28% and
5.99% for the Dice coefficient and for the Jaccard coefficient,
respectively.
The effect of the proposed key-point discriminator is
reported in Table IV. We also conducted an experiment on
the effect of key-point information to guide the key-point dis-
criminator. To evaluate this, we first trained the stand-alone
endometrium segmentation network (without discriminator)
to obtain the performance without the discriminator. Second,
we trained the endometrium segmentation network using the
discriminator but excluding the key-point maps. Finally, we
trained the endometrium segmentation network using the dis-
criminator guided by the key-point map, which is the pro-
posed method.
When we trained the endometrium segmentation using the
discriminator without key-point maps, we found that the per-
formance marginally improved. In contrast, we found that
TABLE III. Performance comparison of endometrium segmentation. Best
results are shown in bold.
Method Dice coefficient (%) Jaccard coefficient (%)
UNet 15
58.69 41.59
FCN8s14 78.39 64.47
Proposed method 82.67 70.46
3980 Park et al.: Endometrium segmentation on TVUS image 3980

TABLE IV. Effect of the key-point map in discriminator. shows the results of the paired t-test which estimates the sta-
tistical significance of using key point maps in the discrimi-
Training way for endometrium Dice Jaccard
nator. The mean difference and 95% confidence interval
segmentation network Coefficient Coefficient
(95% CI) are shown in the Dice coefficient in percentage
Without discriminator 82.38 70.04 units. In the experiment, we calculated the Dice coefficient
With discriminator excluding key-point map 82.41 70.08 for each image on all five test folds. Therefore, the sample
With discriminator including key-point maps 82.67 70.46 size was 3356 images, which is the same size of the entire
(proposed method) dataset. We used this per-image Dice coefficient to estimate
the statistical significance of the difference between the
Bold value indicates the effect of the key-point map in discriminator.
results.
training the endometrium segmentation network with key- First, we estimated the statistical significance of using the
point discriminator improved the performance to 82.67% for proposed discriminator with key-point maps compared to the
the Dice coefficient and 70.46% for the Jaccard coefficient. method without the discriminator. As shown in the second
This shows that key-point information is useful for detailed row, fifth column of Table V, the calculated P-value is less
segmentation of the endometrium. than 0.05 (P-value = 0.0132), which means that the arith-
metic mean of difference is statistically significant. Therefore,
the proposed discriminator including key-point maps
3.C. Statistical analysis improves the performance by a statistically significant differ-
To estimate the statistical significance in the difference of ence compared to the method that does not use the discrimi-
the reported results, we conducted a paired t-test. Table V nator.

TABLE V. Statistical significance of using the key-point maps in the discriminator using paired t-test method. The mean difference, the 95% confidence interval
(CI) of the differences, and the corresponding P-value reported.

Mean difference  standard

Method Compared to error of mean difference 95% CI P-value

With discriminator including key-point maps (proposed method) Without discriminator 0.2278  0.09184 [0.0477, 0.4078] 0.0132
With discriminator excluding key-point maps Without discriminator 0.1176  0.08916 [
0.0572, 0.2925] 0.1871

FIG. 6. Qualitative performance evaluation of the endometrium segmentation. Note that the highlighted region on the image is the endometrial region. (a)-(c) are
the cases that UNet and FCN8s are failed. (d) and (c) are the cases that the TVUS image have heterogeneous texture and unclear boundary.

Medical Physics, 46 (9), September 2019

3981 Park et al.: Endometrium segmentation on TVUS image 3981

FIG. 7. Qualitative effect of the proposed discriminator. (a)-(c) are the cases that the TVUS image have heterogeneous texture. (d)-(f) are the cases that the key-
point discriminator helps to predict endometrium region. (g)-(i) are the cases that the TVUS image have unclear boundary.

Medical Physics, 46 (9), September 2019

3982 Park et al.: Endometrium segmentation on TVUS image
FIG. 8. The examples where the proposed method does not work.
Second, we estimated the statistical significance of using
the discriminator without key-point maps compared to the
method without the discriminator. The mean difference was
not significantly different because the performance is margin-
ally increased as reported in Table V. The results show that
the key-point maps played an important role in the proposed
method.
3.D. Qualitative performance evaluation
In order to illustrate the importance of the proposed
method, Fig. 6 shows segmentation results of the proposed
method and those of the conventional segmentation networks.
FIG. 9. Examples of key-point maps with various standard deviations (r). Note that the key-point maps are in one channel for simple visualization.
Medical Physics, 46 (9), September 2019
3982
As shown in Figs. 6(a)–6(c), the proposed method predicted
the region of the endometrium more accurately than UNet
and FCN8s. Fig. 6(d) and 6(e) show that more accurate per-
formance of the proposed method where the texture was
heterogeneous and the boundary was unclear.
Fig. 7 shows the qualitative effect of employing the pro-
posed key-point discriminator compared with the performance
of the stand-alone endometrium segmentation network. Fig. 7
also shows the effect of key-point maps for guiding the dis-
criminator. As shown in the results of the stand-alone endome-
trium segmentation network in Fig. 7, it is still challenging to
predict the region of the endometrium in cases where the tex-
ture is heterogeneous and the boundary is unclear. In particu-
lar, in Figs. 7(a)–7(c), the texture of the endometrium is
heterogeneous. Therefore, without the discriminator, it is diffi-
cult to accurately segment the region of the endometrium.
When we trained the endometrium segmentation network
using the discriminator without key-point maps, the results
showed little improvement. The proposed discriminator, how-
ever, obtained more detailed and accurate results. Therefore,
we found that the key-point maps allow the discriminator to
more strictly distinguish a predicted map from a ground-truth
map. We also found that utilizing key-point information was
helpful for detailed and accurate segmentation in other results.
In Figs. 7(d)–7(f), we discovered that the key-point discrimina-
tor helps the endometrium segmentation network to accurately
find the thickness and length of the endometrium. We also
noticed that utilizing the discriminator reduced mis-segmented
pixels caused by the confusing texture of the region as shown
in Figs. 7(g)–7(i). In summary, the key-point discriminator
improved the performance of the endometrium segmentation
even in challenging cases.
Fig. 8 shows the examples where the proposed method did
not work. The segmentation examples that showed bad results
were the cases where the key points and endometrium bound-
aries were hardly seen in the input TVUS images.
4. DISCUSSION
In the proposed method, we utilized the key points in the
discriminator in the training phase. These points are
3983 Park et al.: Endometrium segmentation on TVUS image
TABLE VI. The performance of the proposed method according to the stan-
dard deviation of the Gaussian blob.
Standard deviation of
Method Gaussian blob r [pixel] Dice coefficient [%]
Proposed method 1 83.51
2 83.64
3 83.45
4 83.36
supposed to be placed on the endometrium boundary in the
training TVUS dataset. However, to allow the tolerance of
their points position in the training phase, we used a Gaussian
blob on each key-point map. Fig. 9 shows examples of the
key-point maps with various standard deviations. According
to Fig. 9, with a small size of the Gaussian blob, the key-
point maps provided more exact localization to the discrimi-
nator. In contrast, with a bigger size of the Gaussian blob, it
allowed more tolerance of the position of the points.
To investigate the effect of the size of the Gaussian blob
on the performance of the proposed method, we performed
experiments according to different sizes of the intensity of
the blobs (i.e., different standard deviations of Gaussian
blobs, r). Table VI shows the performance of the proposed
method with different values of r of the Gaussian blob on a
single fold. As seen in Table VI, the segmentation perfor-
mance of ‘r = 2’ is better than those of other sizes of Gaus-
sian blobs. Comparing to a method without a key-point map
(Dice coefficient = 82.35%), the method with a key-point
map improved the performance of the endometrium segmen-
tation network. Then, allowing for the tolerance of the posi-
tion of the points improved the performance of the
endometrium segmentation network.
To verify the effectiveness of the proposed method, we
compared with other methods. As described in Section 3.B.,
the proposed network achieved 82.67% in Dice coefficient
and 70.46% in Jaccard coefficient. The results show that it
outperforms the FCN8s by 4.28% and 5.99% in Dice coeffi-
cient and Jaccard coefficient, respectively. Also, our network
outperformed the UNet by 23.98% and 28.87% in Dice coef-
ficient and Jaccard coefficient, respectively.
To estimate the statistically superior performance of the
proposed method, we conducted a paired t-test. As described
in Section 3.C., we compared the performance difference
between the performance with the key-point discriminator
and without the key-point discriminator. The experimental
result showed that there was a significant performance differ-
ence between the two methods with a P-
value = 0.0132 < 0.05. Therefore, the proposed discrimina-
tor that included key-point maps improved the performance
by a statistically significant difference compared to the
method without discriminator.
For inferring the segmentation, we used only the endome-
trium segmentation network. When inferring the endometrium
segmentation on one TVUS image, it took 0.0114 seconds per
image with NVIDIA GeForce GTX 1080 TI.
Medical Physics, 46 (9), September 2019
3983
The strength of our proposed method is that the endome-
trium segmentation network precisely predicts the endome-
trium region thanks to the key-point discriminator used in the
training phase. The proposed network is useful even in TVUS
images that have ambiguous boundary or a heterogeneous
texture. The proposed method requires the discriminator to
train the endometrium segmentation network.
5. CONCLUSIONS
We propose a key-point discriminator to train the seg-
mentation network for more accurate and reliable segmen-
tation of the endometrium on the sagittal TVUS images of
the uterus. We design the endometrium segmentation net-
work employing atrous convolution and ASPP module for
dealing with the various size of the endometrium. In order
to utilize key-point information efficiently, we train the
endometrium segmentation network with the proposed
key-point discriminator in an adversarial way. The key-
point discriminator is guided by key-point maps and the
spatial encoded feature to distinguish predicted map from
the ground-truth strictly. By utilizing key-point informa-
tion, the proposed method shows more reliable and accu-
rate segmentation performance and outperformed the
conventional segmentation networks in qualitative and
quantitative comparisons.
ACKNOWLEDGMENT
This work was supported partly by the Industrial R&D
Project (G01170441), and partly by the ICT R&D Program of
MSIT/IITP (2017-0-01779, A Machine Learning and Statisti-
cal Inference Framework for Explainable Artificial Intelli-
gence).
*Both authors contributed equally to this paper
a)
Author to whom correspondence should be addressed. Electronic mail:
ymro@ee.kaist.ac.kr; Telephone: (+82) 42-350-8094; Fax: (+82) 42-350-
7619.
REFERENCES
1. Koyama T, Tamai K, Togashi K. Staging of carcinoma of the uterine cer-
vix and endometrium. Eur Radiol. 2007;17:2009–2019.
2. Widrich T. Role of Ultrasonography in Infertility. In: Office-Based Infer-
tility Practice. Springer: Berlin, Germany; 2002:39–48.
3. Gong X-H, Lu J, Liu J, et al. Segmentation of uterus using laparoscopic
ultrasound by an image-based active contour approach for guiding gyne-
cological diagnosis and surgery. PLoS ONE. 2015;10:e0141046.
4. Machtinger R, Korach J, Padoa A, et al. Transvaginal ultrasound and
diagnostic hysteroscopy as a predictor of endometrial polyps: risk factors
for premalignancy and malignancy. Int J Gynecol Cancer. 2005;15:325–
328.
5. Gonen Y, Casper RF, Jacobson W, Blankier J. Endometrial thickness
and growth during ovarian stimulation: a possible predictor of implanta-
tion in in vitro fertilization. Fertil Steril. 1989;52:446–450.
6. Alam V, Bernardini L, Gonzales J, Asch RH, Balmaceda JP. A prospec-
tive study of echographic endometrial characteristics and pregnancy rates
during hormonal replacement cycles. J Assist Reprod Genet.
1993;10:215–219.
3984 Park et al.: Endometrium segmentation on TVUS image
7. Israel R, Isaacs JD, Wells CS, et al. Endometrial thickness is a valid
monitoring parameter in cycles of ovulation induction with menotropins
alone. Fertil Steril. 1996;65:262–266.
8. Weissman A, Gotlieb L, Casper RF. The detrimental effect of increased
endometrial thickness on implantation and pregnancy rates and outcome
in an in vitro fertilization program. Fertil Steril. 1999;71:147–149.
9. Levine DJ, Berman JM, Harris M, Chudnoff SG, Whaley FS, Palmer
SL. Sensitivity of myoma imaging using laparoscopic ultrasound com-
pared with magnetic resonance imaging and transvaginal ultrasound. J
Minim Invasive Gynecol. 2013;20:770–774.
10. Kim ST, Lee J-H, Lee H, Ro YM. Visually interpretable deep network
for diagnosis of breast masses on mammograms. Phys Med Biol.
2018;63:235025.
11. Kim DH, Kim ST, Chang JM, Ro YM. Latent feature representation with
depth directional long-term recurrent learning for breast masses in digital
breast tomosynthesis. Phys Med Biol. 2017;62:1009.
12. Kim ST, Kim DH, Ro YM. Detection of masses in digital breast
tomosynthesis using complementary information of simulated projec-
tion. Med Phys. 2015;42:7043–7058.
13. Kim DH, Kim ST, Ro YM. Improving mass detection using combined
feature representations from projection views and reconstructed volume
of DBT and boosting based classification with feature selection. Phys
Med Biol. 2015;60:8809.
14. Long J, Shelhamer E, Darrell T. Fully convolutional networks for seman-
tic segmentation. Paper presented at: IEEE conference on Computer
Vision and Pattern Recognition; 2015.
15. Ronneberger O, Fischer P, Brox T. U-net: convolutional networks for
biomedical image segmentation. Paper presented at: International Con-
ference on Medical image computing and computer-assisted interven-
tion. 2015.
16. Drozdzal M, Vorontsov E, Chartrand G, Kadoury S, Pal C. The impor-
tance of skip connections in biomedical image segmentation. In: Deep
Learning and Data Labeling for Medical Applications. Springer: Berlin,
Germany; 2016:179–187.
17. Badrinarayanan V, Kendall A, Cipolla R. Segnet: a deep convolutional
encoder-decoder architecture for image segmentation. IEEE Trans Pat-
tern Anal Mach Intell. 2017;39:2481–2495.
18. Noh H, Hong S, Han B. Learning deconvolution network for semantic
segmentation. Paper presented at: Proceedings of the IEEE international
conference on computer vision 2015.
Medical Physics, 46 (9), September 2019
3984
19. Chen L-C, Papandreou G, Kokkinos I, Murphy K, Yuille AL. Deeplab:
Semantic image segmentation with deep convolutional nets, atrous con-
volution, and fully connected crfs. IEEE Trans Pattern Anal Mach Intell.
2018;40:834–848.
20. Chen L-C, Papandreou G, Schroff F, Adam H. Rethinking atrous
convolution for semantic image segmentation. arXiv preprint
arXiv:170605587. 2017.
21. Chen L-C, Zhu Y, Papandreou G, Schroff F, Adam H. Encoder-decoder
with atrous separable convolution for semantic image segmentation.
Paper presented at: European Conference on Computer Vision 2018.
22. Sahiner B, Pezeshk A, Hadjiiski LM, et al. Deep learning in medical
imaging and radiation therapy. Med Phys. 2019;46:e1–e36.
23. Dalmısß MU, Litjens G, Holland K, et al. Using deep learning to segment
breast and fibroglandular tissue in MRI volumes. Med Phys.
2017;44:533–546.
24. Jin C, Shi F, Xiang D, Zhang L, Chen X. Fast segmentation of kidney
components using random forests and ferns. Med Phys. 2017;44:6353–
6363.
25. Goodfellow I, Pouget-Abadie J, Mirza M, et al. Generative adversarial
nets. Paper presented at:Neural Information Processing Systems; 2014.
26. Pfau D, Vinyals O. Connecting generative adversarial networks and
actor-critic methods. Paper presented at: Neural Information Processing
Systems Workshop; 2016.
27. Yu F, Koltun V. Multi-scale context aggregation by dilated convolutions.
Paper presented at: International Conference on Learning Representa-
tions. 2016.
28. Radford A, Metz L, Chintala S. Unsupervised representation learning
with deep convolutional generative adversarial networks. Paper pre-
sented at: International Conference on Learning Representations 2015.
29. Milletari F, Navab N, Ahmadi S-A. V-net: Fully convolutional neural
networks for volumetric medical image segmentation. Paper presented
at: International Conference on 3D Vision 2016.
30. Kingma DP, Ba J. Adam: a method for stochastic optimization. Paper
presented at: International Conference for Learning Representations
2015.
31. Pezeshk A, Hamidian S, Petrick N, Sahiner B. 3D convolutional neural
networks for automatic detection of pulmonary nodules in chest CT.
IEEE J Biomed Health Informat. 2018;1–1.