A Theoretical Analysis of the Repetition Problem in Text Generation
This repository share the code for the paper “A Theoretical Analysis of the Repetition Problem in Text Generation” in AAAI 2021. The repetition problem has been observed in nearly all text generation models. We theoretically prove that this problem is, unfortunately, caused by the traits of our language itself. There exists too many words predicting the same word as the subsequent word with high probability. Consequently, it is easy to go back to that word and form repetitions. We dub this problem as the high inflow problem. Based on the theoretical analysis, we propose a novel rebalanced encoding approach to alleviate the high inflow problem.
Requirements
Install