A number of deep neural network (DNN)-based models have been applied to help classify and detect severe cardiovascular diseases using 12-lead electrocardiogram (ECG) signals. These models, however, suffer from poor performance in detecting one or two specific cardiac abnormalities, like ST-segment abnormalities, with their accuracy lingering at only around 60%, which in turn limits their applicability in clinical practice. In this paper, we show that the convolution layers of these DNN models can cause the diminishment of the key features of ST-segment abnormalities, making it, compared to cardiac arrhythmia and normal sinus rhythm (NSR), hard to be classified out from the ECG. Correspondingly, we, in this paper, propose a novel DNN-based model that is able to achieve high accuracy of detecting 9 classes of rhythms. In specific, the moving averages of the ECG signals are used as the second input to the proposed model so that it can help fully mine the features relevant to the ST abnormalities. In addition, as opposed to the convolution layers, our model utilizes the inception-residual layers to preserve features from the shallow layers for reuse in the deep layers. On top of these network architecture improvements, we further introduce a customized differential layer so that all the relevant features can be preserved and/or amplified for classification purposes. Trained with CPSC 2018 dataset, our proposed model is able to accurately classify the 9 rhythm classes, with an F1 score as high as 89.7%, up by merely 4.2% from the best result reported in the literature. As far as the ST segment abnormalities are concerned, the proposed model achieves an 89.6% F1 score for STD and an 80.8% F1 score for STE, which is 7.8% and 13.1% respectively higher than that of the best result. The proposed method is thus poised to become a viable solution for cardiovascular health monitoring with the increasing availability of portable and home-based ECG devices.