Neural networks have become increasingly popular for language modelling and within these large and deep models, overfitting, and gradient remains an important problem that heavily influences the model performance. As long short-term memory (LSTM) and bidirectional long short-term memory (BILSTM) individually solve long-term dependencies in sequential data, the combination of both LSTM and BILSTM in hierarchical gives added reliability to minimise the gradient, overfitting, and long learning issues. Hence, this paper presents four different architectures such as the Enhanced Deep Hierarchal LSTM & BILSTM (EDHLB), EDHBL, EDHLL & EDHBB has been developed. The experimental evaluation of a deep hierarchical network with spatial and temporal features selects good results for four different models. The average accuracy of EDHLB is 92.12%, EDHBL is 93.13, EDHLL is 94.14% & EDHBB is 93.19% and the accuracy level obtained for the basic models such as the LSTM, which is 74% and BILSTM, which is 77%. By evaluating all the models, EDHBL performs better than other models, with an average efficiency of 94.14% and a good accuracy rate of 95.7%. Moreover, the accuracy for the collected Tamil emotional dataset, such as happiness, fear, anger, sadness, and neutral emotions indicates 100% accuracy in a cross-fold matrix. Emotions such as disgust show around 80% efficiency. Lastly, boredom shows 75% accuracy. Moreover, the training time and evaluation time utilised by EDHBL is less when compared with the other models. Therefore, the experimental analysis shows EDHBL as superior to the other models on the collected Tamil emotional dataset. When compared with the basic models, it has attained 20% more efficiency.

• Alías, F., Socoró, J. C., & Sevillano, X. (2016). A review of physical and perceptual feature extraction techniques for speech, music and environmental sounds. Applied Sciences, 6(5), Article 143. https://doi.org/10.3390/app6050143.

• Chen, Z., Watanabe, S., Erdogan, H., & Hershey, J. (2015). Speech enhancement and recognition using multi-task learning of long short-term memory recurrent neural networks. In Sixteenth Annual Conference of the International Speech Communication Association (pp. 3274-3278). IEEE Publishing. https://doi.org/10.1109/SLT.2016.7846281

• Cummins, N., Amiriparian, S., Hagerer, G., Batliner, A., Steidl, S., & Schuller, B. W. (2017). An image-based deep spectrum feature representation for the recognition of emotional speech. In Proceedings of the 25th ACM international Conference on Multimedia (pp. 478-484). ACM Publishing. https://doi.org/10.1145/3123266.3123371

• Erdogan, H., Hershey, J. R., Watanabe, S., & Roux, J. L. (2015). Phase-sensitive and recognition-boosted speech separation using deep recurrent neural networks. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 708-712). IEEE Publishing. https://doi.org/10.1109/ICASSP.2015.7178061.

• Eyben, F., Weninger, F., Squartini, S., & Schuller, B. (2013). Real-life voice activity detection with LSTM recurrent neural networks and an application to Hollywood movies. In 2013 IEEE International Conference on Acoustics, Speech and Signal Processing (pp. 483-487). IEEE Publishing. https://doi.org/10.1109/ICASSP.2015.7178061.

• Graves, A., Jaitly, N., & Mohamed. A. (2013). Hybrid speech recognition with deep bidirectional LSTM. In 2013 IEEE Workshop on Automatic Speech Recognition and Understanding (pp. 273-278). IEEE Publishing. https://doi.org/10.1109/ASRU.2013.6707742.

• Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9(8), 1735-1780. https://doi.org/10.1162/neco.1997.9.8.1735.

• Huang, J., Chen, B., Yao, B., & He, W. (2019). ECG arrhythmia classification using STFT-based spectrogram and convolutional neural network. IEEE Access, 7, 92871-92880. https://doi.org/10.1109/ACCESS.2019.2928017

• Hussain, T., Muhammad, K., Ullah, A., Cao, Z., Baik, S. W., & de Albuquerque, V. H. C. (2019). Cloud-assisted multiview video summarization using CNN and bidirectional LSTM. IEEE Transactions on Industrial Informatics, 16(1), 77-86. https://doi.org/10.1109/TII.2019.2929228

• Ioffe, S., & Szegedy, C. (2015). Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning (pp. 448-456). MLResearchPress. https://doi.org/10.5555/3045118.3045167.

• Karim, F., Majumdar, S., & Darabi, H. (2019). Insights into LSTM fully convolutional networks for time series classification. IEEE Access, 7, 67718-67725. https://doi.org/10.1109/ACCESS.2019.2916828

• Khan, S. U., Haq, I. U., Rho, S., Baik, S. W., & Lee, M. Y. (2019). Cover the violence: A novel Deep-Learning-Based approach towards violence-detection in movies. Applied Sciences, 9(22), Article 4963. https://doi.org/10.3390/app9224963

• Kishore, P. V. V., & Prasad, M. V. D. (2016). Optical flow hand tracking and active contour hand shape features for continuous sign language recognition with artificial neural networ. International Journal of Software Engineering and its Applications, 10(2), 149-170. https://doi.org/10.1109/IACC.2016.71

• Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 1097-1105. https://doi.org/10.1145/3065386.

• Kumar, K. V. V., Kishore, P. V. V., & Kumar, D. A. (2017). Indian classical dance classification with adaboost multiclass classifier on multi feature fusion. Mathematical Problems in Engineering, 20(5), 126-139. https://doi.org/10.1155/2017/6204742.

• Li, J., Deng, L., Gong, Y., & Haeb-Umbach, R. (2014). An overview of noise-robust automatic speech recognition. IEEE/ACM Transactions on Audio, Speech and Language Processing, 22(4), 745-777. https://doi.org/10.1109/TASLP.2014.2304637

• Liu, B., Qin, H., Gong, Y., Ge, W., Xia, M., & Shi, L. (2018). EERA-ASR: An energy-efficient reconfigurable architecture for automatic speech recognition with hybrid DNN and approximate computing. IEEE Access, 6, 52227-52237. https://doi.org/10.1109/ACCESS.2018.2870273

• Liu, Y., Zhang, P., & Hain, T. (2014). Using neural network front-ends on far field multiple microphones based speech recognition. In 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 5542-5546). IEEE. https://doi.org/10.1109/ICASSP.2014.6854663.

• Mannepalli, K., Sastry, P. N., & Suman, M. (2016a). MFCC-GMM based accent recognition system for Telugu speech signals. International Journal of Speech Technology, 19(1), 87-93. https://doi.org/abs/10.1007/s10772-015-9328-y

• Mannepalli, K., Sastry, P. N., & Suman, M. (2016b). FDBN: Design and development of fractional deep belief networks for speaker emotion recognition. International Journal of Speech Technology, 19(4), 779-790. https://doi.org/10.1007/s10772-016-9368-y

• Mustaqeem, & Kwon, S. (2020). A CNN-assisted enhanced audio signal processing for speech emotion recognition. Sensors, 20(1), Article 183. https://doi.org/10.3390/s20010183

• Park, D. S., Chan, W., Zhang, Y., Chiu, C., Zoph, B., Cubuk, E. D., & Le, Q.V. (2019). SpecAugment: A simple data augmentation method for automatic speech recognition. ArXiv Publishing.

• Pascanu, R., Mikolov, T., & Bengio, Y. (2013). On the difficulty of training recurrent neural networks. In International conference on machine learning (pp. 1310-1318). MLResearchPress. https://doi.org/10.5555/3042817.3043083.

• Rao, G. A., & Kishore, P. V. V. (2016). Sign language recognition system simulated for video captured with smart phone front camera. International Journal of Electrical and Computer Engineering, 6(5), 2176-2187. https://doi.org/10.11591/ijece.v6i5.11384

• Rao, G. A., Syamala, K., Kishore, P. V. V., & Sastry, A. S. C. S. (2018). Deep convolutional neural networks for sign language recognition. International Journal of Engineering and Technology (UAE), 7(Special Issue 5), 62-70. https://doi.org/10.1109/SPACES.2018.8316344

• Ravanelli, M., Brakel, P., Omologo, M., & Bengio, Y. (2016). Batch-normalized joint training for dnn-based distant speech recognition. In 2016 IEEE Spoken Language Technology Workshop (SLT) (pp. 28-34). IEEE Publishing. https://doi.org/10.1109/SLT.2016.7846241.

• Ravanelli, M., Brakel, P., Omologo, M., & Bengio, Y. (2017). A network of deep neural networks for distant speech recognition. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 4880-4884). IEEE Publishing. https://doi.org/10.1109/ICASSP.2017.7953084.

• Sak, H., Senior, A. W., & Beaufays, F. (2014, September 14-18). Long short-term memory recurrent neural network architectures for large scale acoustic modeling. In Fifteenth Annual Conference of the International Speech Communication Association (pp. 338-342). Singapore.

• Sastry, A. S. C. S., Kishore, P. V. V., Prasad, C. R., & Prasad, M. V. D. (2016). Denoising ultrasound medical images: A block based hard and soft thresholding in wavelet domain. Medical Imaging: Concepts, Methodologies, Tools, and Applications, 761-775. https://doi.org/10.1016/j.procs.2015.08.040

• Schwarz, A., Huemmer, C., Maas, R., & Kellermann, W. (2015). Spatial diffuseness features for DNN-based speech recognition in noisy and reverberant environments. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 4380-4384). IEEE Publishing. https://doi.org/10.1109/ICASSP.2015.7178798.

• Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. (2014). Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15, 1929-1958. https://doi.org/10.5555/2627435.2670313.

• Weninger, F., Erdogan, H., Watanabe, S., Vincent, E., Roux, J. L., Hershey, J. R., & Schuller, B. W. (2015). Speech enhancement with LSTM recurrent neural networks and its application to noise-robust ASR. In International conference on latent variable analysis and signal separation (pp. 91-99). Springer. https://doi.org/10.1007/978-3-319-22482-4_11

• Zhang, Y., Chen, G., Yu, D., Yao, K., Khudanpur, S., & Glass, J. R. (2016). Highway long short-term memory RNNS for distant speech recognition. In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 5755-5759). IEEE Publishing. https://doi.org/10.1109/ICASSP.2016.7472780

• Zhou, G., Wu, J., Zhang, C., & Zhou, Z. (2016). Minimal gated unit for recurrent neural networks. International Journal of Automation and Computing, 13(3), 226-234. https://doi.org/10.1007/s11633-016-1006-2.