Knowledge Inheritance for Pre-trained Language Models
Knowledge-Inheritance Source code paper: Knowledge Inheritance for Pre-trained Language Models (preprint). The trained model parameters (in Fairseq format) can be downloaded from Tsinghua Cloud. You can use convert_fairseq_to_huggingface.py to convert the Fairseq format into Huggingface’s transformers format easily. We refer the downstream performance evaluation to the implementation of Fairseq (GLUE tasks) and Don’t Stop Pre-training (ACL-ARC / CHEMPROT). If you have any question, feel free to contact us ([email protected]). 1. Available Pretrained Models WB domain: Wikipedia + BookCorpus; CS domain: […]
Read more