JSAI2020
Jun 9 - Jun 12, 2020Virtual Meetings
JSAI2020
Jun 9 - Jun 12, 2020Virtual Meetings
[1E3-GS-9-03]Automatic punctuation completion using LSTM
〇Katsuhiko Utsubo1(1. Kyodo News)
Keywords:
NLP,LSTM
In Japanese sentences, the meaning of the context may differ depending on the insertion point of punctuation, so the position of the punctuation is very important.
In this research, we create a general method that automatically complements punctuation from text information using deep learning.
The proposed method is that the corpus is split using morphological analysis and replaced infrequent words with parts of speech and performs classification of exists of a period or comma using LSTM from word strings before and after the target position.
The accuracy of classification has been improved by setting a threshold for the probability output by the model.
Furthermore, by limiting the number of input words and replacing them with parts of speech, the calculation time can be reduced without reducing the calculation accuracy.
Using this method, experiments using broadcast manuscripts as text corpora have confirmed the effectiveness of this method.
In this research, we create a general method that automatically complements punctuation from text information using deep learning.
The proposed method is that the corpus is split using morphological analysis and replaced infrequent words with parts of speech and performs classification of exists of a period or comma using LSTM from word strings before and after the target position.
The accuracy of classification has been improved by setting a threshold for the probability output by the model.
Furthermore, by limiting the number of input words and replacing them with parts of speech, the calculation time can be reduced without reducing the calculation accuracy.
Using this method, experiments using broadcast manuscripts as text corpora have confirmed the effectiveness of this method.