We proposed a machine learning approach to identify and distinguish dusty stellar sources employing supervised methods and categorizing point sources, mainly evolved stars, using photometric and spectroscopic data collected over the IR sky. Spectroscopic data is typically used to identify specific infrared sources. However, our goal is to determine how well these sources can be identified using multiwavelength data. Consequently, we developed a robust training set of spectra of confirmed sources from the Large and Small Magellanic Clouds derived from SAGE-Spec Spitzer Legacy and SMC-Spec Spitzer Infrared Spectrograph (IRS) spectral catalogs. Subsequently, we applied various learning classifiers to distinguish stellar subcategories comprising young stellar objects (YSOs), C-rich asymptotic giant branch (CAGB), O-rich AGB stars (OAGB), Red supergiant (RSG), and post-AGB stars. We have classified around 700 counts of these sources. It should be highlighted that despite utilizing the limited spectroscopic data we trained, the accuracy and models' learning curve provided outstanding results for some of the models. Therefore, the Support Vector Classifier (SVC) is the most accurate classifier for this limited dataset.