An Efficient and Scalable Technique for Clustering Comorbidity Patterns of Diabetic Patients from Clinical Datasets

Clustering diabetic patients with comorbidity patterns are necessary to learn relationships between diabetes patients’ clinical profiles and as an essential pre-processing stage for analysis tasks, like classification and categorization. Nevertheless, the heterogeneity of these data makes traditional clustering methods more difficult to apply, necessitating the development of novel clustering algorithms. In this paper, we recommend an effective and scalable clustering technique suitable for datasets made up of attributes which are atomic and set-valued. In these datasets, each record corresponds to a different diagnosis detail of a diabetic patient based on his or her hospital visit, where diagnosis details in each record are represented using the International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) codes. Our proposed technique involves three main stages. In the first stage, we selected the top-k diabetes-specific comorbidities patterns. In the second stage, we ensured that the co-occurring conditions in the selected top-k diabetes-specific comorbidities patterns really co-occur together or not using topic modeling and in the last stage, we constructed high quality clusters efficiently using average linkage agglomerative clustering with cosine similarity. Also, based on silhouette analysis, we assessed the efficiency and effectiveness of our proposed technique using a large, freely available MIMIC dataset (MIMIC-III and MIMIC-IV), comprised of over 14,222 and 68,118 distinct records, respectively. Our findings reveal that our technique finds clusters that: (i) preserve interrelations between demographics (age, gender) and diagnosis codes (ICD-9-CM codes), and (ii) are well-separated and compact. Finally, the founded clusters are beneficial for numerous investigative tasks like query answering, visualization, anonymization, classification etc.