We publish the Pre-trained Language Models for Datasets in Pretraining Language Models with Text-Attributed Heterogeneous Graphs. These models include Patents_THLM, GoodReads_THLM, and OAG_Venue_THLM. The source codes and datasets are publicly available on GitHub