-
Notifications
You must be signed in to change notification settings - Fork 0
/
index.Rmd
766 lines (589 loc) · 50 KB
/
index.Rmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
---
title: R Markdown Notebook for *Vector Space Models and the usage patterns of Indonesian
denominal verbs*
author: '[Gede Primahadi Wijaya Rajeg](https://figshare.com/authors/Gede_Primahadi_Wijaya_Rajeg/1234749) <a itemprop="sameAs" content="https://orcid.org/0000-0002-2047-8621" href="https://orcid.org/0000-0002-2047-8621" target="orcid.widget" rel="noopener noreferrer" style="vertical-align:top;"><img src="https://orcid.org/sites/default/files/images/orcid_16x16.png" style="width:1em;margin-right:.5em;" alt="ORCID iD icon"></a>, [Karlina Denistia](http://uni-tuebingen.academia.edu/karlinadenistia) <a itemprop="sameAs" content="http://orcid.org/0000-0002-1060-3548" href="http://orcid.org/0000-0002-1060-3548" target="orcid.widget" rel="noopener noreferrer" style="vertical-align:top;"><img src="https://orcid.org/sites/default/files/images/orcid_16x16.png" style="width:1em;margin-right:.5em;" alt="ORCID iD icon"></a>, [Simon Musgrave](http://profiles.arts.monash.edu.au/simon-musgrave/) <a itemprop="sameAs" content="https://orcid.org/0000-0003-3237-9943" href="https://orcid.org/0000-0003-3237-9943" target="orcid.widget" rel="noopener noreferrer" style="vertical-align:top;"><img src="https://orcid.org/sites/default/files/images/orcid_16x16.png" style="width:1em;margin-right:.5em;" alt="ORCID iD icon"></a>'
date: "Created 26 January, 2019; Last Update `r format(Sys.time(), '%d %B, %Y')`"
output:
html_notebook:
fig_caption: yes
fig_width: 6
theme: "united"
highlight: "pygments"
number_sections: yes
toc: yes
toc_float: yes
bookdown::html_document2:
toc: TRUE
toc_float: TRUE
theme: "united"
highlight: "pygments"
fig_caption: yes
fig_width: 6
number_sections: TRUE
code_folding: show
keep_md: FALSE
link-citations: yes
csl: unified_stylesheet_linguistics.csl
bibliography: index_ref.bib
---
<a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/"><img alt="Creative Commons License" style="border-width:0" src="https://i.creativecommons.org/l/by-nc-sa/4.0/88x31.png" /></a><br />This work is licensed under a <a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/">Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License</a>.
[![DOI](https://img.shields.io/badge/doi-10.6084/m9.figshare.9970205-blue.svg?style=flat&labelColor=whitesmoke&logo=data%3Aimage%2Fpng%3Bbase64%2CiVBORw0KGgoAAAANSUhEUgAAAB8AAAAfCAYAAAAfrhY5AAAJsklEQVR42qWXd1DTaRrHf%2BiB2Hdt5zhrAUKz4IKEYu9IGiGFFJJQ0gkJCAKiWFDWBRdFhCQUF3UVdeVcRQEBxUI3yY9iEnQHb3bdW1fPubnyz%2F11M7lvEHfOQee2ZOYzPyDv%2B3yf9%2Fk95YX4fx%2BltfUt08GcFEuPR4U9hDDZ%2FVngIlhb%2FSiI6InkTgLzgDcgfvtnovhH4BzoVlrbwr55QnhCtBW4QHXnFrZbPBaQoBh4%2FSYH2EnpBEtqcDMVzB93wA%2F8AFwa23XFGcc8CkT3mxz%2BfXWtq9T9IQlLIXYEuHojudb%2BCM7Hgdq8ydi%2FAHiBXyY%2BLjwFlAEnS6Jnar%2FvnQVhvdzasad0eKvWZKe8hvDB2ofLZ%2FZEcWsh%2BhyIuyO5Bxs2iZIE4nRv7NWAb0EO8AC%2FWPxjYAWuOEX2MSXZVgPxzmRL3xKz3ScGpx6p6QnOx4mDIFqO0w6Q4fEhO5IzwxlSwyD2FYHzwAW%2BAZ4fEsf74gCumykwNHskLM7taQxLYjjIyy8MUtraGhTWdkfhkFJqtvuVl%2F9l2ZquDfEyrH8B0W06nnpH3JtIyRGpH1iJ6SfxDIHjRXHJmdQjLpfHeN54gnfFx4W9QRnovx%2FN20aXZeTD2J84hn3%2BqoF2Tqr14VqTPUCIcP%2B5%2Fly4qC%2BUL3sYxSvNj1NwsVYPsWdMUfomsdkYm3Tj0nbV0N1wRKwFe1MgKACDIBdMAhPE%2FwicwNWxll8Ag40w%2BFfhibJkGHmutjYeQ8gVlaN%2BjO51nDysa9TwNUFMqaGbKdRJZFfOJSp6mkRKsv0rRIpEVWjAvyFkxNOEpwvcAVPfEe%2Bl8ojeNTx3nXLBcWRrYGxSRjDEk0VlpxYrbe1ZmaQ5xuT0u3r%2B2qe5j0J5uytiZPGsRL2Jm32AldpxPUNJ3jmmsN4x62z1cXrbedXBQf2yvIFCeZrtyicZZG2U2nrrBJzYorI2EXLrvTfCSB43s41PKEvbZDEfQby6L4JTj%2FfIwam%2B4%2BwucBu%2BDgNK05Nle1rSt9HvR%2FKPC4U6LTfvUIaip1mjIa8fPzykii23h2eanT57zQ7fsyYH5QjywwlooAUcAdOh5QumgTHx6aAO7%2FL52eaQNEShrxfhL6albEDmfhGflrsT4tps8gTHNOJbeDeBlt0WJWDHSgxs6cW6lQqyg1FpD5ZVDfhn1HYFF1y4Eiaqa18pQf3zzYMBhcanlBjYfgWNayAf%2FASOgklu8bmgD7hADrk4cRlOL7NSOewEcbqSmaivT33QuFdHXj5sdvjlN5yMDrAECmdgDWG2L8P%2BAKLs9ZLZ7dJda%2BB4Xl84t7QvnKfvpXJv9obz2KgK8dXyqISyV0sXGZ0U47hOA%2FAiigbEMECJxC9aoKp86re5O5prxOlHkcksutSQJzxZRlPZmrOKhsQBF5zEZKybUC0vVjG8PqOnhOq46qyDTDnj5gZBriWCk4DvXrudQnXQmnXblebhAC2cCB6zIbM4PYgGl0elPSgIf3iFEA21aLdHYLHUQuVkpgi02SxFdrG862Y8ymYGMvXDzUmiX8DS5vKZyZlGmsSgQqfLub5RyLNS4zfDiZc9Edzh%2FtCE%2BX8j9k%2FqWB071rcZyMImne1SLkL4GRw4UPHMV3jjwEYpPG5uW5fAEot0aTSJnsGAwHJi2nvF1Y5OIqWziVCQd5NT7t6Q8guOSpgS%2Fa1dSRn8JGGaCD3BPXDyQRG4Bqhu8XrgAp0yy8DMSvvyVXDgJcJTcr1wQ2BvFKf65jqhvmxXUuDpGBlRvV36XvGjQzLi8KAKT2lYOnmxQPGorURSV0NhyTIuIyqOmKTMhQ%2BieEsgOgpc4KBbfDM4B3SIgFljvfHF6cef7qpyLBXAiQcXvg5l3Iunp%2FWv4dH6qFziO%2BL9PbrimQ9RY6MQphEfGUpOmma7KkGzuS8sPUFnCtIYcKCaI9EXo4HlQLgGrBjbiK5EqMj2AKWt9QWcIFMtnVvQVDQV9lXJJqdPVtUQpbh6gCI2Ov1nvZts7yYdsnvRgxiWFOtNJcOMVLn1vgptVi6qrNiFOfEjHCDB3J%2BHDLqUB77YgQGwX%2Fb1eYna3hGKdlqJKIyiE4nSbV8VFgxmxR4b5mVkkeUhMgs5YTi4ja2XZ009xJRHdkfwMi%2BfocaancuO7h%2FMlcLOa0V%2FSw6Dq47CumRQAKhgbOP8t%2BMTjuxjJGhXCY6XpmDDFqWlVYbQ1aDJ5Cptdw4oLbf3Ck%2BdWkVP0LpH7s9XLPXI%2FQX8ws%2Bj2In63IcRvOOo%2BTTjiN%2BlssfRsanW%2B3REVKoavBOAPTXABW4AL7e4NygHdpAKBscmlDh9Jysp4wxbnUNna3L3xBvyE1jyrGIkUHaqQMuxhHElV6oj1picvgL1QEuS5PyZTEaivqh5vUCKJqOuIgPFGESns8kyFk7%2FDxyima3cYxi%2FYOQCj%2F%2B9Ms2Ll%2Bhn4FmKnl7JkGXQGDKDAz9rUGL1TIlBpuJr9Be2JjK6qPzyDg495UxXYF7JY1qKimw9jWjF0iV6DRIqE%2B%2FeWG0J2ofmZTk0mLYVd4GLiFCOoKR0Cg727tWq981InYynvCuKW43aXgEjofVbxIqrm0VL76zlH3gQzWP3R3Bv9oXxclrlO7VVtgBRpSP4hMFWJ8BrUSBCJXC07l40X4jWuvtc42ofNCxtlX2JH6bdeojXgTh5TxOBKEyY5wvBE%2BACh8BtOPNPkApjoxi5h%2B%2FFMQQNpWvZaMH7MKFu5Ax8HoCQdmGkJrtnOiLHwD3uS5y8%2F2xTSDrE%2F4PT1yqtt6vGe8ldMBVMEPd6KwqiYECHDlfbvzphcWP%2BJiZuL5swoWQYlS%2Br7Yu5mNUiGD2retxBi9fl6RDGn4Ti9B1oyYy%2BMP5G87D%2FCpRlvdnuy0PY6RC8BzTA40NXqckQ9TaOUDywkYsudxJzPgyDoAWn%2BB6nEFbaVxxC6UXjJiuDkW9TWq7uRBOJocky9iMfUhGpv%2FdQuVVIuGjYqACbXf8aa%2BPeYNIHZsM7l4s5gAQuUAzRUoT51hnH3EWofXf2vkD5HJJ33vwE%2FaEWp36GHr6GpMaH4AAPuqM5eabH%2FhfG9zcCz4nN6cPinuAw6IHwtvyB%2FdO1toZciBaPh25U0ducR2PI3Zl7mokyLWKkSnEDOg1x5fCsJE9EKhH7HwFNhWMGMS7%2BqxyYsbHHRUDUH4I%2FAheQY7wujJNnFUH4KdCju83riuQeHU9WEqNzjsJFuF%2FdTDAZ%2FK7%2F1WaAU%2BAWymT59pVMT4g2AxcwNa0XEBDdBDpAPvgDIH73R25teeuAF5ime2Ul0OUIiG4GpSAEJeYW9wDTf43wfwHgHLKJoPznkwAAAABJRU5ErkJggg%3D%3D)](http://dx.doi.org/10.6084/m9.figshare.9970205)
# How to cite this R Notebook
Please cite this R Notebook as follows (in Unified Style Sheet for Linguistics):
> Rajeg, Gede Primahadi Wijaya, Karlina Denistia & Simon Musgrave. 2019. R Markdown Notebook for *Vector space model and the usage patterns of Indonesian denominal verbs*. figshare. https://doi.org10.6084/m9.figshare.9970205. https://figshare.com/articles/R_Markdown_Notebook_for_i_Vector_space_model_and_the_usage_patterns_of_Indonesian_denominal_verbs_i_/9970205.
# Preface
```{r setup, message = FALSE}
# global option chunck
knitr::opts_chunk$set(fig.width = 7,
fig.asp = 0.618,
dpi = 300,
dev = "pdf",
echo = TRUE,
message = FALSE,
warning = FALSE,
fig.path = "figures/")
```
This is an [R Markdown](http://rmarkdown.rstudio.com) [Notebook](https://bookdown.org/yihui/rmarkdown/notebook.html) [@rajeg_r_2019] for the statistical analyses accompanying our [paper](http://repository.tufs.ac.jp/handle/10108/94452) [@rajeg_vector_2019] on vector space models and Indonesian denominal verbs (published in [*NUSA*](http://www.aa.tufs.ac.jp/en/publications/nusa)'s special issue titled [*Linguistic studies using large annotated corpora*](http://repository.tufs.ac.jp/handle/10108/94450), edited by [Hiroki Nomoto](http://www.tufs.ac.jp/ts/personal/nomoto/) and [David Moeljadi](http://compling.hss.ntu.edu.sg/who/david/)) [@nomoto_linguistic_2019]. The Notebook, however, does not provide detailed exposition and discussion for each points, including English glossing of the Indonesian words in the tables. Readers are referred to our [paper](http://repository.tufs.ac.jp/handle/10108/94452) for details. Check the README page [here](https://github.com/gederajeg/vector_space_model_indonesian) for the tutorial to download this Notebook and the data [@rajeg_dataset_2019] for the paper.
The following R packages have to be installed and loaded to run all codes in this Notebook:
- cluster [@maechler_cluster_2018]
- tidyverse [@wickham_r_2017]
- dendextend [@galili_dendextend_2015]
- wordVectors [@schmidt_wordvectors_2017]
- Rling [@levshina_how_2015]
Click the `Code` button to hide/reveal the R codes.
```{r utilised-packages, echo = TRUE}
# load the required packages
## make sure all packages are installed!
library(cluster)
library(tidyverse)
library(dendextend)
library(wordVectors) # should be installed from Ben Schmidt's GitHub
library(Rling) # <- `Rling` package from Natalia Levshina's (2015) book 'How to do linguistics with R' published by John Benjamins.
```
# Illustration for generating Vector Space Models (VSM)
For illustration, we used six deverbal nouns with the suffix -*an*. They are *bacaan* 'reading' (from the root *baca* 'to read'), *tulisan* 'writing' (from *tulis* 'to write'), *lukisan* 'painting' (from *lukis* 'to paint/draw'), *masakan* 'cooking' (from *masak* 'to cook'), *makanan* 'food' (from *makan* 'to eat'), and *minuman* 'beverages' (from *minum* 'to drink'). We retrieved collocates within three-word window to the either sides (left and right) of each noun. We used one of the _Indonesian_ [_Leipzig Corpora_](http://corpora2.informatik.uni-leipzig.de/download.html) files, namely *ind_mixed_2012_1M-sentences.txt* [see @biemann_leipzig_2007; @quasthoff_indonesian_2013].
## Creating the co-occurrence frequency table
The following codes process the retrieved collocates into co-occurrence frequency table and print a subset of this table. The collocates data is available in the `"vsm_creation_data.rds"` file. This `.rds` file contains a `List` data-structure; the table version is available as tab-delimited `.csv` and `.txt` files (`vsm_creation_data_tb.txt` & `vsm_creation_data_tb.csv`).
```{r vsm-xmpl-collfreq}
df_list <- readRDS(file = "data/vsm_creation_data.rds")
df_ex <- purrr::map_df(df_list, dplyr::bind_rows); rm(df_list)
df_ex1 <- dplyr::filter(df_ex,
!is.na(w), # remove NAs
nchar(w) > 1, # remove one-character tokens
stringr::str_detect(w, "^[a-z-]+$"),
# remove incomplete words
!stringr::str_detect(w, "^[a-z]+-$"),
# remove incomplete words
!stringr::str_detect(w, "^-[a-z]+$"))
df_ex1_count <- dplyr::count(df_ex1, node, w, sort = TRUE)
df_ex1_count_spread <- tidyr::spread(df_ex1_count, w, n, fill = 0)
df_ex1_count_spread <- as.data.frame(df_ex1_count_spread)
df_ex1_count_spread <- df_ex1_count_spread[, colSums(df_ex1_count_spread[, -1]) > 0]
rownames(df_ex1_count_spread) <- df_ex1_count_spread$node
df_ex1_count_spread <- df_ex1_count_spread[, -1]
selected_column <- c("alkitab",
"dituangkan", "halal", "keras",
"mengandung", "penutup")
df_ex1_print <- df_ex1_count_spread[, selected_column]
rownames(df_ex1_print) <- paste("*", rownames(df_ex1_print), "*", sep = "")
colnames(df_ex1_print) <- paste("*", colnames(df_ex1_print), "*", sep = "")
df_ex1_print$`...` <- rep("...", nrow(df_ex1_print))
knitr::kable(df_ex1_print, caption = "Raw co-occurrence frequency for the target words with the context words")
```
## Performing the *Positive Pointwise Mutual Information*
Current approach in VSM adopts a more principled method of weighting the initial raw-frequency vectors into statistical measures of collocation strength before computing (dis)similarity measure between the target words. The goal of the weighting is "to give a higher weight to context words that co-occur significantly more often than expected by chance" [@heylen_monitoring_2015, 156; cf. @lappin_vector_2015, 503-504; @perek_recent_2016, 12]. These significantly associatied context words are assumed to be more informative for the semantics of the target words [@heylen_monitoring_2015, 156]. The popular weighting measure used in VSM is the *Pointwise Mutual Information* (PMI) [see @levshina_how_2015, 327-328 for computing PMI in R]:
$$\text{PMI } (x, y) = log_{2} \frac {O_{xy}} {E_{xy}}$$
where $O_{xy}$ represents the observed co-occurrence frequency between *x* and *y*, while $E_{xy}$ is their expected co-occurrence frequency, which is the frequency expected under the chance distribution between *x* and *y* given the overall distribution of *x* and *y* in the corpus. Negative PMI values is normally replaced with zero, resulting in the *Positive PMI* (PPMI) [@levshina_how_2015; @hilpert_meaning_2015].
```{r vsm-xmpl-ppmi, warning = FALSE}
mtx <- as.matrix(df_ex1_count_spread)
mtx_exp <- chisq.test(mtx)$expected
mtx_pmi <- log2(mtx/mtx_exp); rm(mtx)
mtx_ppmi <- ifelse(mtx_pmi < 0, 0, mtx_pmi); rm(mtx_exp)
ppmiprint <- as.data.frame(round(mtx_ppmi, digits = 2)); rm(mtx_pmi)
ppmiprint <- ppmiprint[, selected_column]
rownames(ppmiprint) <- paste("*", rownames(ppmiprint), "*", sep = "")
colnames(ppmiprint) <- paste("*", colnames(ppmiprint), "*", sep = "")
ppmiprint$`...` <- rep("...", nrow(df_ex1_count_spread))
knitr::kable(ppmiprint, caption = "Weighted co-occurrence frequency with *Positive Pointwise Mutual Information*")
```
Further analysis can be performed. The most common one is determining the semantic (dis)similarity between the target words (i.e., which word is more similar and different among each other). The following section briefly discusses the *Cosine Similarity* and *Hierarchical Agglomerative Cluster* (HAC) analyses as the exploratory tools [@levshina_geographic_2014; @levshina_how_2015].
## Exploring VSM with *Cosine Similarity* and *Hierarchical Agglomerative Cluster* (HAC) analysis
In VSM, *cosine similarity* is the popular measure for computing pairwise (dis)similarity between the target words. Cosine similarity computes the cosine of angles between the words' vectors to capture their (dis)similarity. The cosine value between a pair of word is close to 1 when they are semantically more similar, and close to 0 when otherwise (see [\@ref(tab:vsm-xmpl-cossim)](#vsm-xmpl-cossim)) [@levshina_how_2015, Ch. 16].
The following codes perform cosine similarity using the `cossim()` function from the Rling package [@levshina_how_2015, 329].
```{r vsm-xmpl-cossim}
# Cosine Similarity
mtx_cossim <- Rling::cossim(mtx_ppmi); rm(mtx_ppmi)
# Generate CosSim table output
attr(mtx_cossim, "dimnames")[[1]] <- paste("*",
attr(mtx_cossim, "dimnames")[[1]],
"*",
sep = "")
attr(mtx_cossim, "dimnames")[[2]] <- paste("*",
attr(mtx_cossim, "dimnames")[[2]],
"*",
sep = "")
knitr::kable(round(mtx_cossim, digits = 2),
caption = "Cosine similarity matrix between the deverbal nouns")
```
For the cluster analysis, such as HAC [@levshina_geographic_2014; see also @gries_statistics_2013, 336; @levshina_how_2015, Ch. 15; and @desagulier_corpus_2017, 276 for R implementations on HAC], the similarity matrix/table as above needs to be converted into distance matrix as input for the cluster analysis [see the codes below adapted from @levshina_how_2015, 330].
```{r vsm-xmpl-hac}
# Distance matrix computation
mtx_dist <- 1 - (mtx_cossim/max(mtx_cossim[mtx_cossim != 1]))
mtx_dist <- as.dist(mtx_dist)
# Cluster analysis
mtx_hcl <- hclust(mtx_dist, method = "ward.D2")
```
The output of HAC can be visualised into a dendrogram tree (see [Figure \@ref(fig:vsm-xmpl-dendrogram)](#vsm-xmpl-dendrogram)). To determine the optimal number of cluster solution for grouping the nouns, we used the *Average Silhouette Width* (ASW) scores and tested two up to five-cluster solutions. The two-cluster solution produces the highest ASW score. See the codes below.
```{r vsm-xmpl-dendrogram, fig.asp = 0.75, fig.cap = "VSM-based cluster dendrogram for the target words"}
# ASW calculation
mtx_asw <- sapply(2:(nrow(mtx_cossim) - 1), function(x) summary(cluster::silhouette(stats::cutree(mtx_hcl, k = x), mtx_dist))$avg.width)
# names the ASW scores representing the tested cluster-solutions
names(mtx_asw) <- 2:(nrow(mtx_cossim) - 1)
# identify the maximum ASW score
max_asw <- as.numeric(names(mtx_asw[mtx_asw == max(mtx_asw)]))
# plotting dendrogram
mtx_hcl$labels <- gsub('\\*', '', mtx_hcl$labels)
plot(mtx_hcl, hang = -1)
rect.hclust(mtx_hcl, k = max_asw)
# generate texts annotation within the plot
added_info <- paste("The ", max_asw, "-cluster solution is based on the highest Average Silhouette Width (ASW) score of ", round(max(mtx_asw), 3), ".\n(ASW ranges from 0 to 1; min. ASW score for assuming substantial cluster is 0.2)", sep = "")
mtext(added_info, side = 1, line = 1.75, cex = .8)
```
The following codes produce the plot visualising the number of cluster-solutions tested and their corresponding ASW scores.
```{r vsm-xmpl-asw-plot, fig.asp = .75, fig.cap = "Average Silhouette Widths for the tested cluster solutions"}
plot(x = 2:5, y = mtx_asw,
type = "b", xaxt = "n",
ylab = "Average Silhouette Widths (ASW)",
ylim = c(0, 1),
xlab = "Number of tested cluster solutions")
axis(side = 1, at = 2:5)
text(2:5,
mtx_asw,
labels = round(mtx_asw, digits = 3),
pos = 3,
cex = .8)
text(x = 3.25,
y = 0.2,
labels = "The 2-cluster solution produces the highest ASW score")
```
# Methodology
## Data source
The paper uses thirteen corpus files from the Indonesian Leipzig Corpora to be trained into a vector space models. The following codes show how to load the data for the information of corpus size (i.e. `"wordcount_leipzig_allcorpus.RData"`) and to print them.
```{r leipzig-corpus-size}
# load the corpus size data
load("data/wordcount_leipzig_allcorpus.RData")
# print the table
leipzig.word.count %>%
select(-Size) %>%
rename(`Size (in word-tokens)`= Size_print,
`Corpus files` = Corpus) %>%
as.data.frame() %>%
knitr::kable(row.names = TRUE, caption = "Indonesian Leipzig Corpora and their sizes")
```
In total, the thirteen corpus files amount to `r prettyNum(sum(leipzig.word.count$Size), big.mark = ",")` million word-tokens.
## The studied denominal verbs
The database for the studied verbs is available in the `.rds`, `.csv`, and `.txt` tab-delimited files named `"me_parsed_nountaggedbase"`. The verbs have been parsed and tagged using *MorphInd* [@larasati_indonesian_2011]. We limited the study on the verbs with noun-tagged roots that occur over 20 tokens. For each root, all the verb forms must occur in the three morphological schemas, namely *meN*-, *meN*-/-*kan*, and *meN*-/-*i*. For instance, we take _**men**dasar_ 'to be basic', _**men**dasar**i**_ 'to underlie sth.', and _**men**dasar**kan**_ 'to base sth. on' that are all derived from the nominal root *dasar* 'base; foundation'.
The following codes show the filtering processes to retrieve the relevant verbs and [Table \@ref(tab:studied-verbs-retrieval)](#studied-verbs-retrieval) shows the snippet of the database.
```{r studied-verbs-retrieval}
# load the database
parsed_me_noun <- readRDS("data/me_parsed_nountaggedbase.rds")
# filtering parameters
min_freq <- 20 # freq. threshold
base_type <- "noun" # root category
# filtering process
df_noun <- parsed_me_noun %>%
filter(n > min_freq) %>% # retrieve only verbs occurring over 20 tokens
group_by(base) %>%
mutate(n_affix_new = n_distinct(affix)) %>%
filter(n_affix_new == 3) %>% # make sure each root occurs in the 3 schemas
ungroup()
# retrieve the target-word character vectors
me_words <- df_noun$word
# print the database snippet
df_noun %>%
select(-n_affix_new) %>%
filter(base %in% c("dasar")) %>%
rename(base_pos = base.pos,
token_freq = n) %>%
mutate(word = paste("*", word, "*", sep = ""),
base = paste("*", base, "*", sep = ""),
affix = paste("*", affix, "*", sep = ""),
morphind = gsub("(<|>)", "\\\\\\1", morphind, perl = TRUE)) %>%
knitr::kable(caption = "Snippet of the analysed denominal verbs")
```
In total, we analysed `r length(me_words)` denominal verbs based on `r length(unique(df_noun$base))` root types occurring in three morphological schemas.
We mention in the paper about checking the existence of the studied verbs with *MALINDO Morph*, a morphological dictionary for Indonesian and Malay [@nomoto_malindo_2018]. We checked it with the latest version of MALINDO Morph from the file named `"malindo_dic_20181125.tsv"` that has been saved as `"malindo_dbase.rds"` (an R type of data) and tab-delimited `.csv` file (`"malindo_dbase.csv"`). The following codes document the checking processes. The printed verbs are those absent from *MALINDO Morph* but available in our corpus occurring over 20 tokens overall.
```{r malindo-check}
# load MALINDO Dictionary data
malindo <- readRDS("data/malindo_dbase.rds")
# print the verbs that are available in the dataset but absent in MALINDO
(absent_in_malindo <- setdiff(me_words, malindo$word_form))
```
As *MALINDO Morph* is also based on the Leipzig Corpora, it only takes into account words occurring over ten tokens in all the 300K-sentences version of the corpus [@nomoto_malindo_2018]. Our frequency check of these absent verbs confirms that they all occur less than ten tokens in the 300K-sentence files that we use. The following codes show these information.
```{r malindo-freq-check, message = FALSE, warning = FALSE}
# load the freqlist per corpus and
# check freq. of words absent in MALINDO 300K corpus
df_all_pref <- read_tsv("data/wordlist_leipzig_ME_DI_TER_percorpus.tsv", progress = FALSE)
df_all_pref %>%
filter(word %in% absent_in_malindo,
grepl("300K", corpus, perl = TRUE)) %>%
mutate(word = paste("*", word, "*", sep = "")) %>%
arrange(word, corpus) %>%
knitr::kable(row.names = TRUE, caption = 'Token frequency of the absent verbs in the 300K-sentence files')
```
## Load the vector space models for the Indonesian Leipzig Corpora
Detailed information concerning the training parameters is available in our paper. In short, we trained the Leipzig Corpora on the [*MonARCH HPC*](https://confluence.apps.monash.edu/display/monarch/MonARCH+Home) using the *skip-gram* learning algorithm from the *word2vec* model [@mikolov_efficient_2013; @mikolov_distributed_2013; @mikolov_linguistic_2013] via the wordVectors R package [@schmidt_wordvectors_2017]. The output model is available as .bin file named `"leipzig_w2v_vector_full.bin"`. The following codes show how the model is loaded into R using `read.binary.vectors()` from wordVectors package.
```{r load-model-show, message = FALSE, warning = FALSE, error = FALSE, include = TRUE, eval = FALSE}
vsm <- read.binary.vectors("data/leipzig_w2v_vector_full.bin")
```
```{r load-model-eval, include = FALSE}
vsm <- read.binary.vectors("data/leipzig_w2v_vector_full.bin")
```
To retrieve the vector space model (VSM) of the target denominal verbs, use the following codes.
```{r vsm-denominal}
# get the VSM for the studied denominal verbs
vsm_tgt <- vsm[rownames(vsm) %in% me_words, ]
# print the subset of the model
vsm_tgt
```
# Results and discussion
The following codes run the _Hierarchical Agglomerative Analysis_ (HAC) on the target verbs vector space model as well as the _Average Silhouette Width_ (ASW) statistics.
```{r hac-analysis}
# A wrapper function for HCA and ASW computations and gathering the relevant results
svs_hca <- function(vect = NULL,
clust_method = c("complete", "ward.D", "ward.D2",
"single", "average", "mcquitty",
"median", "centroid")) {
# 1 cosine distance using function from `wordVectors`
cosdist <- wordVectors::cosineDist(vect, vect)
cosdist <- as.dist(cosdist)
# 2 Hierarchical Cluster Analysis
hca <- hclust(cosdist, method = clust_method)
# 3 Compute the 'average silhouette width' (ASW)
# using the 'cutree' function for HCA partitioning
asw_f <- function(x) {
summary(cluster::silhouette(stats::cutree(hca, k = x), cosdist))$avg.width
}
asw_cutree <- sapply(2:(dim(vect)[1] - 1), asw_f)
names(asw_cutree) <- 2:(dim(vect)[1] - 1)
# 3.2 get the cluster number with highest ASW score
n_cluster <- as.numeric(names(asw_cutree[asw_cutree == max(asw_cutree)]))
max_asw <- asw_cutree[asw_cutree == max(asw_cutree)]
# 4 Put the results into a list
res <- list(cosdist, hca, n_cluster, max_asw, asw_cutree)
names(res) <- c("cosine_dist", "hcluster", "n_cluster", "asw", "asw_all")
return(res)
}
# Hierarchical Agglomerative Cluster (HAC) analysis for the denominal verbs
clust_method <- "ward.D2"
hca_res <- svs_hca(vect = vsm_tgt,
clust_method = clust_method)
```
The following codes generate the plot for the ASW scores in the paper.
```{r hca-asw-plot, fig.asp = .9, fig.width = 6.25, fig.cap = "Average Silhouette Widths for the tested cluster solutions."}
# preparing data frame for the ASW and tested clusters
asw_df <- hca_res$asw_all
asw_df <- tibble::tibble(tested_cluster = names(asw_df),
asw = unname(asw_df))
asw_df_sort <- dplyr::arrange(asw_df, dplyr::desc(asw))
# plot with ggplot2
asw_df_sort %>%
ggplot(aes(x = reorder(tested_cluster, asw), y = asw, group = 1)) +
geom_step() +
ylim(c(0, 1)) +
coord_flip() +
theme_light() +
geom_text(aes(label = round(asw, 3)), hjust = -0.2, size = 2.5) +
labs(x = "Number of tested cluster solutions",
y = "Average Silhouette Width (ASW)",
caption = paste("The tested cluster solution ranges from 2 to ",
length(me_words) - 1,
" (i.e., the length of the analysed words (",
length(me_words), ") - 1).\nThe highest ASW of ",
round(hca_res$asw, 3),
" is for ", hca_res$n_cluster,
"-cluster solution",
sep = ""))
```
The dendrogram tree is generated using the following codes:
```{r hca-plot, fig.asp = 1.3, fig.cap = "Cluster dendrogram for the studied denominal verbs based on *Hierarchical Agglomerative Clustering* (Distance measure = *Cosine Distance*; Clustering method = *Ward.D2*); The 21-cluster solution is based on the highest ASW score of 0.304 and is indicated by the red boxes."}
opar <- par(no.readonly = TRUE)
par(mar = c(2, 2, 2, 10))
dend <- as.dendrogram(hca_res$hcluster)
dend <- dendextend::set(dend = dend, "labels_cex", .9)
plot(dend, horiz = TRUE)
dendextend::rect.dendrogram(dend,
k = hca_res$n_cluster,
horiz = TRUE,
border = 2,
lwd = 0.35)
par(opar)
```
```{r get-word-frequency}
# A function to retrieve the token frequency of the studied verbs
wfreq <- function(df = df_noun, w, formatted = TRUE) {
freq <- dplyr::pull(dplyr::filter(df, word == w), n)
if (formatted) {
freq <- format(freq, big.mark = ",")
}
return(freq)
}
```
## Semantic cluster {#semclust}
The subcluster of the <span style = "font-variant:small-caps;">motion</span> verbs is extracted using the following codes:
```{r subclust-motion, fig.cap = 'Cluster for the <span style = "font-variant:small-caps;">motion</span> verbs'}
clusters <- dendextend::cutree(hca_res$hcluster, k = hca_res$n_cluster)
opar <- par(no.readonly = TRUE)
par(mar = c(2, 2, 2, 7))
dend <- as.dendrogram(hca_res$hcluster)
to_prune <- names(clusters[!names(clusters) %in% c("melangkahkan", "melangkah", "menapak", "menapaki", "menapakkan", "menjejakkan", "menjejak", "menjejaki")])
plot(dendextend::prune(dend, to_prune), horiz = TRUE)
par(opar)
```
In this section, we present three tables for n-grams data of verbs with the root *tapak* 'sole of the foot'. The data can be loaded as follows:
```{r tapak-gram, message = FALSE, warning = FALSE}
tapak <- readr::read_tsv("data/ngramexampl_3gr_menapak.txt")
tapaki <- readr::read_tsv("data/ngramexampl_3gr_menapaki.txt")
tapakkan <- readr::read_tsv("data/ngramexampl_3gr_menapakkan.txt")
```
The following codes generate the ten most frequent right-side 3-gram for *menapak*.
```{r tapak-gram-table}
tapakgr <- tapak %>%
filter(w1 == "menapak") %>%
count(ngrams, sort = TRUE) %>%
.[1:10, ] %>%
mutate(ngrams = paste("*", ngrams, "*", sep = "")) # make ngrams italics
knitr::kable(tapakgr, caption = "The ten most frequent 3-gram for *menapak*", row.names = TRUE)
```
*Menapak* can be used as transitive (item 1, 5, and 7) and intransitive verbs (the remaining items in [Table \@ref(tab:tapak-gram-table)](#tapak-gram-table)). Its transitive usage shares similar right-side collocation patterns with the *MeN*-/-*i* form *menapaki*, especially their direct object with *masa depan* 'future' and *karir* 'career' ([Table\@ref(tab:tapaki-gram-table) below](#tapaki-gram-table)).
```{r tapaki-gram-table}
tapakigr <- tapaki %>%
filter(w1 == "menapaki") %>%
count(ngrams, sort = TRUE) %>%
.[1:10, ] %>%
mutate(ngrams = paste("*", ngrams, "*", sep = ""))
knitr::kable(tapakigr, caption = "The ten most frequent 3-gram for *menapaki*", row.names = TRUE)
```
This is different from the transitive usage with *MeN*-/-*kan* schema ([Table \@ref(tab:tapakkan-gram-table) below](#tapakkan-gram-table)), which predominantly has *kaki* 'foot' as its direct object, followed by either locational/directional prepositional phrses or motion verb complements (e.g., *memasuki* 'to enter' [item 5] and *maju* 'to move forward' [item 8]).
```{r tapakkan-gram-table}
tapakkangr <- tapakkan %>%
filter(w1 == "menapakkan") %>%
count(ngrams, sort = TRUE) %>%
.[1:10, ] %>%
mutate(ngrams = paste("*", ngrams, "*", sep = ""))
knitr::kable(tapakkangr, caption = "The ten most frequent 3-gram for *menapakkan*", row.names = TRUE)
```
Codes for extracting the subset of the mixture of <span style = "font-variant:small-caps;">communication</span> and <span style = "font-variant:small-caps;">psych</span> verbs are as follows.
```{r subcluster-psychs-1, fig.cap = 'Cluster of <span style = "font-variant:small-caps;">communication</span> and <span style = "font-variant:small-caps;">psych</span> verbs (a)'}
clusters <- dendextend::cutree(hca_res$hcluster, k = hca_res$n_cluster)
opar <- par(no.readonly = TRUE)
par(mar = c(2, 2, 2, 7))
dend <- as.dendrogram(hca_res$hcluster)
to_prune <- names(clusters[!names(clusters) %in% c("membayangkan", "menyesal", "menyesali", "mengatai")])
plot(dendextend::prune(dend, to_prune), horiz = TRUE)
par(opar)
```
The above dendrogram is from the top cluster in [Figure \@ref(fig:hca-plot)](#hca-plot), consisting of (i) *menyesal* 'to be regretful' (N = `r wfreq(df_noun, "menyesal")`), (ii) *menyesali* 'regret sth.' (N = `r wfreq(df_noun, "menyesali")`), and (iii) *membayangkan* 'imagine; visualise' (N = `r wfreq(df_noun, "membayangkan")`). The other subset of <span style = "font-variant:small-caps;">communication</span> and <span style = "font-variant:small-caps;">psych</span> verbs (i.e., *mengatakan* 'to say sth.' (N = `r wfreq(df_noun, "mengatakan")`), *mencontohkan* 'to exemplify' (N = `r wfreq(df_noun, "mencontohkan")`), and *menyesalkan* 'regret sth.' (N = `r wfreq(df_noun, "menyesalkan")`)) is extracted as follows.
```{r subcluster-psychs-2, fig.cap = 'Cluster of <span style = "font-variant:small-caps;">communication</span> and <span style = "font-variant:small-caps;">psych</span> verbs (b)'}
clusters <- dendextend::cutree(hca_res$hcluster, k = hca_res$n_cluster)
opar <- par(no.readonly = TRUE)
par(mar = c(2, 2, 2, 7))
dend <- as.dendrogram(hca_res$hcluster)
to_prune <- names(clusters[!names(clusters) %in% c("mencontohkan", "mengatakan", "menyesalkan")])
plot(dendextend::prune(dend, to_prune), horiz = TRUE)
par(opar)
```
What interesting between verbs in these last two clusters is that the *MeN*-/-*i* (i.e. *mengatai*) and *MeN*-/-*kan* verbs (i.e. *mengatakan*) with the root *kata* 'word' are way apart in the dendrogram. Similar case is apparent between *menyesalkan* separated with *menyesal* and *menyesali*, where the latter two verbs cluster together and are merged first in [Figure \@ref(fig:subcluster-psychs-1)](#subcluster-psychs-1) (see §[\@ref(cluster-split)](#cluster-split) for further discussion on this split).
## Root-based clustering {#root-cluster}
This cluster type captures denominal verbs of a given root with the three different morphological schemas that cluster together ([Figure \@ref(fig:subcluster-root-based)](#subcluster-root-based)). We have seen few examples of these in the <span style = "font-variant:small-caps;">motion</span> cluster with the root *tapak* 'sole of the foot' and *jejak* 'footprint', the derived forms of which fall into one cluster but differ in terms of their within-cluster branching ([Figure \@ref(fig:subclust-motion)](#subclust-motion)). The other examples are based on the following roots:
a. *susu* 'milk' -> *menyusu* '(of a baby/young animal) to suckle' (N = `r wfreq(df_noun, "menyusu")`); *menyusui* 'to breast-feed sb.' (N = `r wfreq(df_noun, "menyusui")`); *menyusukan* 'to let sb. suckle; to breast-feed sb.' (N = `r wfreq(df_noun, "menyusukan")`)
b. *dasar* 'base' -> *mendasar* 'basic; foundational' (N = `r wfreq(df_noun, "mendasar")`); *mendasari* 'to underlie sth.' (N = `r wfreq(df_noun, "mendasari")`); *mendasarkan* 'to base X (on Y)' (N = `r wfreq(df_noun, "mendasarkan")`)
c. *tempat* 'place; location' -> *menempat* 'to place/position (sth.) at' (N = `r wfreq(df_noun, "menempat")`); *menempati* 'to occupy; to site at' (N = `r wfreq(df_noun, "menempati")`); *menempatkan* 'to put/place/position X at Y' (N = `r wfreq(df_noun, "menempatkan")`)
d. *wakil* 'vice; representative; deputy' -> *mewakil* 'to (be a) represent(ative of) X' (N = `r wfreq(df_noun, "mewakil")`); *mewakili* 'to (be a) represent(ative of) X' (N = `r wfreq(df_noun, "mewakili")`); *mewakilkan* 'to assign sb. as a representative' (N = `r wfreq(df_noun, "mewakilkan")`)
The cluster subsets of these verbs are extracted from [Figure \@ref(fig:hca-plot)](#hca-plot) into [Figure \@ref(fig:subcluster-root-based)](#subcluster-root-based) with the codes below.
```{r subcluster-root-based, fig.cap = 'Root-based clustered denominal verbs'}
clusters <- dendextend::cutree(hca_res$hcluster, k = hca_res$n_cluster)
opar <- par(no.readonly = TRUE)
par(mar = c(2, 2, 2, 7))
dend <- as.dendrogram(hca_res$hcluster)
to_prune <- names(clusters[!names(clusters) %in% c("mewakili", "mewakilkan", "mewakil", "menyusu", "menyusui", "menyusukan", "menempati", "menempatkan", "menempat", "mendasari", "mendasarkan", "mendasar")])
plot(dendextend::prune(dend, to_prune), horiz = TRUE)
dendextend::rect.dendrogram(dendextend::prune(dend, to_prune),
k = 4,
horiz = TRUE,
border = 2,
lwd = 0.35)
par(opar)
```
In the paper, we present n-grams data contrasting *mewakili* and *mewakilkan* in terms of their ride-side collocates in their 3-grams data. First, the codes for generating n-grams for *mewakili* 'to (be a) represent(ative of) X' are shown below.
```{r mewakili-gr, message = FALSE, warning = FALSE, include = TRUE}
mewakili <- readr::read_tsv("data/ngramexampl_5gr_mewakili.txt", progress = FALSE)
mewakili_mini <- dplyr::filter(mewakili, w1 == "mewakili")
mewakili_count <- dplyr::count(mewakili_mini, w1, w2, w3, sort = TRUE)
mewakili_print <- mewakili_count[1:10,]
mewakili_print <- dplyr::mutate(mewakili_print,
ngrams = paste(w1, "_", w2, "_", w3, sep = ""),
ngrams = paste("*", ngrams, "*", sep = ""))
mewakili_print <- dplyr::select(mewakili_print, ngrams, n)
knitr::kable(mewakili_print, caption = "10 most frequent 3-gram for *mewakili* 'to (be a) represent(ative of) X'", row.names = TRUE)
```
Then, the codes for 3-grams of *mewakilkan* 'to make X as the representative (of Y)'.
```{r mewakilkan-gr, message = FALSE, warning = FALSE, include = TRUE}
mewakilkan <- readr::read_tsv("data/ngramexampl_5gr_mewakilkan.txt", progress = FALSE)
mewakilkan_mini <- dplyr::filter(mewakilkan, w1 == "mewakilkan")
mewakilkan_count <- dplyr::count(mewakilkan_mini, w1, w2, w3, sort = TRUE)
mewakilkan_print <- mewakilkan_count[1:10,]
mewakilkan_print <- dplyr::mutate(mewakilkan_print,
ngrams = paste(w1, "_", w2, "_", w3, sep = ""),
ngrams = paste("*", ngrams, "*", sep = ""))
mewakilkan_print <- dplyr::select(mewakilkan_print, ngrams, n)
knitr::kable(mewakilkan_print, caption = "10 most frequent 3-gram for *mewakilkan* 'to appoint/select X as the representative of Y'", row.names = TRUE)
```
## Clustering-split between morphological schemas for a given root {#cluster-split}
This sub-section addresses in more details split cases between morphological schemas for a given root. The split, especially between *MeN*-/-*kan* and *MeN*-/-*i* verbs, reflects Sneddon et al's [-@sneddon_indonesian_2010, 100-101] hypothesis concerning clear semantic difference between some of a pair of *MeN*-/-*kan* and *MeN*-/-*i* verbs with the same root. Our VSM-based approach allows us to *visualise* such split through the dendrogram based on large-scale usage data. The sub-section also demonstrates further enrichment in charactersing the difference between morphologically related verb pairs using the technique of _nearest neighbours_ based on the VSM data.
In §[\@ref(semclust)](#semclust), we have mentioned the clear split between the transitive *melangkahkan* 'to move the foot forward' and *melangkahi* 'to step over'. Looking at the 2-gram data for each verb shows that they have different semantic orientation. The following codes generate the 2-gram data for *melangkahi*.
```{r langkahi-gr, message = FALSE, warning = FALSE, include = TRUE}
langkahi <- readr::read_tsv("data/ngramexampl_3gr_melangkahi.txt")
langkahi_print <- langkahi %>%
filter(w1 == "melangkahi") %>%
mutate(ngrams = paste(w1, "_", w2, sep = ""),
ngrams = paste("*", ngrams, "*", sep = "")) %>%
count(ngrams, sort = TRUE) %>%
.[1:10, ]
knitr::kable(langkahi_print, caption = "Ten most frequent 2-gram for right-side collocates of *melangkahi*", row.names = TRUE)
```
*Melangkahi* predominantly conveys metaphorical sense related to disobeying/disregarding certain (i) rules/protocols (i.e., *aturan*, *batas-batas*/*batasan* 'limits; restriction', *mekanisme* 'mechanism'), (ii) foundation (*dasar-dasar*), or (iii) authority (*kewenangan*).
In contrast, *melangkahkan* predominantly collocates with *kaki* 'foot' as its direct object collocates, which can be used in the literal, translational motion and metaphorical motion (see our paper for the example sentences).
```{r langkahkan-gr, message = FALSE, warning = FALSE, include = TRUE}
langkahkan <- readr::read_tsv("data/ngramexampl_3gr_melangkahkan.txt")
langkahkan_print <- langkahkan %>%
filter(w1 == "melangkahkan") %>%
mutate(ngrams = paste(w1, "_", w2, sep = ""),
ngrams = paste("*", ngrams, "*", sep = "")) %>%
count(ngrams, sort = TRUE) %>%
.[1:10, ]
knitr::kable(langkahkan_print, caption = "Ten most frequent 2-gram for right-side collocates of *melangkahkan*", row.names = TRUE)
```
Observation on the n-gram can be enriched using information from the VSM of words. Given that the *skip-gram* algorithm of *word2vec* learns to predict the contextual environments given a target word [cf. @mikolov_efficient_2013], one can retrieve from the VSM a set of words that have similar contextual-vector distribution to a given target verb on the basis their cosine similarities; these words can be metaphorically referred to as the verb's *nearest neighbours*. [Table \@ref(tab:nearest-to-melangkahi)](#nearest-to-melangkahi) illustrates the idea for *melangkahi* 'to step over'.
```{r nearest-to-melangkahi}
# function to retrieve and print top-10 nearest neighbours
neighbours <- function(model, seed_word, n = 10) {
topn <- n + 1
df <- wordVectors::closest_to(model, seed_word, topn)
# remove the seed words because it is similar to itself and become the first nearest word. We thus count top-10 nearest neighbours from the rank 2 to 11
df <- dplyr::filter(df, word != seed_word)
df <- dplyr::mutate(df, word = paste("*", word, "*", sep = ""))
df <- as.data.frame(df)
colnames(df)[2] <- paste('similarity to "*', seed_word, '*"', sep = "")
return(df)
}
# get the nearest neighbours for *melangkahi*
near_langkahi <- neighbours(vsm, seed_word = "melangkahi")
knitr::kable(near_langkahi,
caption = "10 closest words to *melangkahi* 'to step over'",
row.names = TRUE)
```
The closest words may not necessarily similar in meaning (e.g., near-synonyms), but may exhibit different kind of relationships, such as antonyms. Words in [Table \@ref(tab:nearest-to-melangkahi)](#nearest-to-melangkahi) conveying more or less antonymous sense to *melangkahi* 'to step over; to disregard' include *mengacuhkan* 'to care about/heed sth.', *mematuhi* 'to obey', and (to a degree) *memperhitungkannya* 'to take sth. into account'. *Mengangkangi* 'to straddle sth.' is the closest in meaning with *melangkahi* as it can be extended into 'disregarding' sense from its physical, posture sense: the 2-gram data for *mengangkangi* across the whole corpus ([Table \@ref(tab:kangkangi-gr)](#kangkangi-gr)) reveals that it does co-occur with rules-related direct objects, such as *hukum* 'law' (3 tokens), *peraturan* 'regulation' (3), *kebenaran* 'the truth' (2), *prinsip* 'principles' (2), *undang-undang* 'constitution' (2), *aturan* 'rules' (1), *inter alia*.
```{r kangkangi-gr, message = FALSE, warning = FALSE, include = TRUE}
ngangkangi <- readr::read_tsv("data/ngramexampl_3gr_mengangkangi.txt")
ngangkangi_print <- ngangkangi %>%
filter(w1 == "mengangkangi") %>%
mutate(ngrams = paste(w1, "_", w2, sep = ""),
ngrams = paste("*", ngrams, "*", sep = "")) %>%
count(ngrams, sort = TRUE)
knitr::kable(ngangkangi_print[ngangkangi_print$n > 1, ], caption = "The 2-gram data for right-side collocates of *mengangkangi* (n >= 2)", row.names = TRUE)
```
The following codes retrieve words nearest to *melangkahkan*.
```{r nearest-to-melangkahkan}
knitr::kable(neighbours(vsm, "melangkahkan", n = 10), caption = "Ten closest words to *melangkahkan*", row.names = TRUE)
```
The codes to retrieve the nearest words of *mengatai* 'to rebuke; speake of one's badness'.
```{r nearest-to-mengatai}
knitr::kable(neighbours(vsm, "mengatai", n = 10), caption = "Ten closest words to *mengatai*", row.names = TRUE)
```
In contrast, *mengatakan* mostly appears as communication verb with similar distribution with other reported speech verbs.
```{r nearest-to-mengatakan}
knitr::kable(neighbours(vsm, "mengatakan", n = 10), caption = "Ten closest words to *mengatakan*", row.names = TRUE)
```
Nearest words to *membuahi* 'to breed sth.' are retrieved as follows.
```{r nearest-to-membuahi}
knitr::kable(neighbours(vsm, "membuahi", n = 10), caption = "Ten closest words to *membuahi*", row.names = TRUE)
```
Codes for extracting closest words to *membuahkan* 'to bear a fruit; to result in sth.'.
```{r nearest-to-membuahkan}
knitr::kable(neighbours(vsm, "membuahkan", n = 10), caption = "Ten closest words to *membuahkan*", row.names = TRUE)
```
Notes on the usage sentences for *membuah* are below. But first load the sentence citations for *membuah* into R and print them into the console. After that the notes are manually created by manually inspecting all the usage sentences. The sentence data is available as `.rds` file of `list` (`sentence_membuah.rds`) and `.txt` file as plain text of sentences (`sentence_membuah.txt`).
```{r load-membuah-sentence}
membuah <- readRDS("data/sentence_membuah.rds")
membuah$ind_mixed_2012_1M # for instance, retrieve sentences found in `ind_mixed_2012_1M-sentences.txt`
```
The sentence format is `corpus-file-name_sentence-id-number_sentence-citation`. The `<m>...</m>` tag indicates the corresponding match/word/verb.
- *membuah* (56 citations; Summary: analogy to *membuahkan* = 24 citations; split with *-kan* = 1; mispell for *membuat* = 25; mispell for *membuang* = 5; unclear = 1)
- ind_mixed_2012_1M: 4 citations (all FULL FORM)
- ind_news_2008_300K: 5 citations (2 mispell for *membuat* [id 53666 & 155569]; 3 FULL FORM analogy to *membuahkan*)
- ind_news_2009_300K: 3 citations (1 mispell for *membuat* [id 200027] 2 FULL FORM analogy to *membuahkan*)
- ind_news_2010_300K: 1 citation (all mispell for *membuat*)
- ind_news_2011_300K: 2 citations (1 mispell for *membuat* [id 266814]; 1 FULL FORM analogy to *membuahkan*)
- ind_news_2012_300K: 5 citations (4 mispell for *membuat* [id 200952, 206980, 268407, 269898]; 1 FULL FORM analogy to *membuahkan*)
- ind_newscrawl_2011_1M: 6 citations (3 FULL FORM analogy to *membuahkan*; 3 mispell for *membuat* [id 118419, 151095, 205239])
- ind_newscrawl_2012_1M: 3 citations (1 mispell for *membuat* [id 201217]; 1 mispell for *membuang* [id 401158]; 1 got split with *-kan* [id 595662])
- ind_web_2011_300K: 1 citation (mispell for *membuat*)
- ind_web_2012_1M: 4 citations (1 mispell for *membuat* [id 602274]; 1 unclear for mispell for *membuat* [id 365510]; 2 FULL FORM analogy to *membuahkan*)
- ind_wikipedia_2016_1M: 5 citations (2 mispell for *membuat* [id 320981, 596205]; 3 FULL FORM analogy to *membuahkan*)
- ind-id_web_2013_1M: 5 citations (1 mispell for *membuang* [id 886076]; 4 FULL FORM as *membuahkan*)
- ind-id_web_2015_3M: 12 citations (1 FULL FORM as *membuahkan* [id 1396285]; 3 mispell for *membuang* [id 1362223, 1549653, 1564536], the remaining 8 mispell for *membuat*)
The following codes retrieve the nearest words to *mengakhiri* (N = `r wfreq(df_noun, "mengakhiri")`).
```{r nearest-to-mengakhiri}
knitr::kable(neighbours(vsm, "mengakhiri", n = 10), caption = "Ten closest words to *mengakhiri*", row.names = TRUE)
```
If reader wishes to check all usage sentences for *mengakhir* to confirm that it occurs as a full word-form and has similar usage patterns with *meN-*/-*i* form *mengakhiri* (i.e. in transitive constructions), use the following code. It will print all sentences for *mengakhir* into R console.
```{r load-mengakhir-sentence, eval = FALSE}
mengakhir <- readRDS("data/sentence_mengakhir.rds")
unlist(unname(mengakhir)) # print all 57 sentences in the console
```
Next, the nearest words to *mengakhirkan* (N = `r wfreq(df_noun, "mengakhirkan")`) are shown below.
```{r nearest-to-mengakhirkan}
knitr::kable(neighbours(vsm, "mengakhirkan", n = 10), caption = "Ten closest words to *mengakhirkan*", row.names = TRUE)
```
## Issues and challenges {#issues}
Codes to extract the nearest words to *mengantung* 'to hang' (N = `r wfreq(df_noun, "mengantung")`), which is a misspelling of *menggantung* 'to hang' (with double *g*) (N = `r format(sum(subset(df_all_pref, word == "menggantung")$n), big.mark = ",")`). Nearest words output shows that *mengantung* is indeed the closest word to *menggantung* in their usage co-occurrence.
```{r nearest-to-mengantung}
knitr::kable(neighbours(vsm, "mengantung", n = 10), caption = "Ten closest words to *mengantung*", row.names = TRUE)
```
The codes below extract the nearest words for *mengantungkan* (N = `r wfreq(df_noun, "mengantungkan")`), a misspelling for *menggantungkan* 'to hang sth. (onto sth.)' (N = `r format(sum(subset(df_all_pref, word == "menggantungkan")$n), big.mark = ",")`).
```{r nearest-to-mengantungkan}
knitr::kable(neighbours(vsm, "mengantungkan", n = 10), caption = "Ten closest words to *mengantungkan*", row.names = TRUE)
```
Then, the nearest words for *mengantungi* (N = `r wfreq(df_noun, "mengantungi")`) which is closest to the common spelling with _mengant**o**ngi_ (N = `r format(sum(subset(df_all_pref, word == "mengantongi")$n), big.mark = ",")`) based on the root *kantong* 'pocket'.
```{r nearest-to-mengantungi}
knitr::kable(neighbours(vsm, "mengantungi", n = 10), caption = "Ten closest words to *mengantungi*", row.names = TRUE)
```
Finally, the case where part of the complex word is split. The paper illustrates this with *menanda* which is part of *menandatangani* 'to give signature; to sign' but is written separately with whitespace, thus, *menanda tangani*, where the whitespace got tokenised.
```{r nearest-to-menanda}
knitr::kable(neighbours(vsm, "menanda", n = 10), caption = "Ten closest words to *menanda*", row.names = TRUE)
```
- *menanda* (N = `r wfreq(w = "menanda")`; Summary: split = 113; intransitive usage = 5; transitive usage analogy = 2; ambiguous = 1)
- ind_mixed_2012_1M (22 citations: 18 split from *menandatangani*; 4 citations as intransitive with *pada* (138307, 278193, 542191, 754529))
- ind_news_2008_300K (1 citation: 1 split from *menandatangani*)
- ind_news_2009_300K (2 citations: 2 split from *menandatangani*)
- ind_news_2010_300K (2 citations: 2 split from *menandatangani*)
- ind_news_2011_300K (3 citations: 3 split from *menandatangani*)
- ind_newscrawl_2011_1M (10 citations: 10 split from *menandatangani*)
- ind_newscrawl_2012_1M (9 citations: 9 split from *menandatangani*)
- ind_web_2011_300K (4 citations: 4 split from *menandatangani*)
- ind_web_2012_1M (18 citations):
- 14 citations split from *menandatangani*
<!-- - 1 citation of reduplication *tanda- menanda* (55270): Ancangan Semiotika dan Pengkajian Susastra adalah buku teori dan kritik sastra yang memumpunkan telaahnya pada masalah ilmu tanda- <m>menanda</m> dalam hubungannya dengan komunikasi dan pemaknaan sastra ." -->
- 1 citation of transitive usage in analogy to *menandai* (44043): "II . mangarus rupa , menerka , mencoba <m>menanda</m> siapa orang ini ."
- 1 citation of transitive usage in analogy to *menandakan* 'to mark/signal (the existence of) sth.' (368004): "Majlis tamat , hadir berduyun sekumpulan penari dengan meriahnya , <m>menanda</m> temasya kecil kampung bakal bermula , sedikit demi sedikit sahutan penduduk kampung menggila sepenuh nafsu menari sekali bersama penari ." (Malaysian Malay?)
- 1 citation of intransitive usage (544395): "Bagi aku , kalau tidak sempat <m>menanda</m> di sekolah ( memang tak cukup masa ) , terpaksalah bawa di rumah ."
- 1 citation of ambiguous usage (981384): "Nama dan No Kad Pengenalan disebut , dan mereka <m>menanda</m> ."
- ind_wikipedia_2016_1M (16 citations: 16 split from *menandatangan(i)*)
- ind-id_web_2013_1M (16 citations: 16 split from *menandatangan(i)*)
- ind-id_web_2015_3M (18 citations: 18 split from *menandatangan(i)*)
# Session Info {-}
```{r sessinfo}
devtools::session_info()
```
# References {-}