Dr Qing Song's Retirement Home

AI blog Series 1: What can neural networks do with data?

Pattern recognition is a very good start point to undersatnd AI fundamentals. No matter you are interested for machine learning, neural networks or deep learning network theories or AI application like image classification, speech recognition and autopilot or autonomous car driving systems etc ...

The following is one of favariable examples of my AI courses deveoped in the course of my long academic careers.

This is a two class classification in general pattern recognition problem (interestingly, it is not only limited to AI). We want to classify the input patterns, patterns 1 (apple) and pattern 2 (orange) into two classes. In mathemetical model, the two input patterns can be described as two input vectors as in Figure 1.

A simple neural network with two-dimensional (2D) input vectors can be used to classify the input patterns into two classes as in Figure 2. Note that this network uses only a single activation function, threshold logic unit (TLU), with three weights (two for the input pattern and one for the bias input). Output of the network can be used to classify the two classes, i.e. apple and orange. Each pattern is described by two features, i.e. feature 1 = width/height and feature 2 = spectum color, corresponding to the first and second components of the input pattern vector respectively.

This two class cassificaiton problem can be solved by neural network via a hyperplane or decision plane. It is induced when the TLU input is set to zero as shown in Figure 3. It is interesting to note that this hyperplane can successfully classify the two classes with some proper values of the weights. However, there are infinite possibilities of the hyperplane as change of the weights. For multi-layred neural network, the hyperplane can be very complicated nonlinear function to seperate many more input patterns, rather than this simple two-pattern case (apple and orange). This is one of the basic reasons why deep learning has been gaining popularities with big data. It can also be integrated with some conventional signal processing methods, like that of convolutional neural network. However, if the data set is not pure and mixed with some wrong data in almost all real appilcations, called outlier, the hyperplane can be fundamentally wrong positioned, no matter how deep the network is!

AI related referred journal publications

1. Sumit Bam Shrestha* and Qing Song##, “Robustness to Training Disturbances in SpikeProp Learning”, IEEE Transactions on Neural Networks and Learning Systems, pp.3126-3139, Vol. 29, No.7, July, 2018. 2. Sumit Bam Shrestha* and Qing Song##, “Robust spike-train learning in spike-event based weight update”, Neural Networks, Volume 96, December 2017, Pages 33-46. 3. Qing Song##, X. Zhao*, H. J. Fan* and D.W. Wang, “Robust Recurrent Kernel Online Learning", IEEE Transactions on Neural Networks and Learning Systems, pp. 1068-1081, Vol. 28, No. 5, May, 2017. 4. Sumit Bam Shrestha*, Qing Song##, “Robust learning in SpikeProp”, Neural Networks, Volume 86, February 2017, Pages 54-68. 5. Xulei Yang*, Qing Song and Yi Su, “Automatic segmentation of left ventricle cavity from short-axis cardiac magnetic resonance images”, Medical & Biological Engineering & Computing, Feb. 2017. 6. Sumit Bam Shrestha*, Qing Song##, “Adaptive learning rate of SpikeProp based on weight convergence analysis”, Neural Networks, 03/2015. 7. Haijin Fan*, Qing Song, Zhao Xu*, “An information theoretic sparse kernel algorithm for online learning”, Expert Systems with Applications 07/2014; 41(9):4349–4359. 8. Haijin Fan*, Qing Song#, Sumit Bam Shrestha*, “Online Learning with Kernel Regularized Least Mean Square Algorithms”, Knowledge-Based Systems 03/2014. 9. Haijin Fan*, Qing Song#, “A linear recurrent kernel online learning algorithm with sparse updates”, Neural networks, 11/2013. 10. Zhimin Wang*, Qing Song#, Y.C. Soh, Sim Kang, “An adaptive spatial information-theoretic fuzzy clustering algorithm for image segmentation”, Computer Vision and Image Understanding 10/2013; 117(10):1412-1420. 11. Qing Song#, “Robust initialization of a Jordan Network with Recurrent Constrained Learning”, Special issue of White Box Nonlinear Prediction Models, IEEE Transactions on Neural Networks, Vol. 22, No. 12, Part II, pp. 2460-2473, 2011. 12. H.J. Fan*, Qing Song and Z. Xu*, “A Robust Information Theoretic Sparse Kernel Algorithm For Online Learning”, accepted for publication on Neurocomputing, 2011. 13. Z. Xu*, Qing Song and D.Wang, “Recurrent neural tracking control based on multivariable robust adaptive gradient-descent training algorithm”, Neural Computing & Applications, No. 11, Vol. 20, 2011. 14. Qing Song#, “On the Weight Convergence of Elman Networks”, IEEE Transactions on Neural Networks, Vol. 21, No.3, pp.463-480, 2010. 15. Z.M. Wang*, Qing Song#, Y.C. Soh, and Sim Kang, “A Robust Curve Clustering based on a Multivariate Model”, IEEE Transactions on Neural Networks, Vol. 21, No. 12, 2010, pp.1976-1984. 16. Qing Song#, X.L. Yang* and Y.C. Soh, “An Information Fuzzy C-Spherical Shells Clustering Algorithm”, Fuzzy Sets and Systems, Vol. 161, No. 13, pp.1755-1773, 2010. 17. Y.L. Wu*, Sun, FC, Zheng, JC and Qing Song, “ A robust training algorithm of discrete-time MIMO RNN and application in fault tolerant control of robotic system”, Neural Computing & Applications Vol. 19, No. 7, pp. 1013-1027, 2010. 18. X. L. Yang*, Qing Song, Y.L. Wu* “A Novel Pruning Approach for Robust Data Clustering”, Neural Computing & Applications, Vol. 18, No.7, pp.759-768, 2009. 19. Z.M. Wang, Y.C. Soh, Qing Song# and Sim Kang, “Adaptive Spatial Information-theoretic Clustering for Image Segmentation”, Pattern Recognition, Vol. 42., 2009, pp. 2029-2044. 20. C.Y. Guo* and Qing Song#, “Real-time control of variable air volume system based on a robust neural network associated PI controller”, IEEE Transactions on Control Systems Technology, Vol.17, No. 3, 2009, pp.600-607. 21. Qing Song#, Y.L. Wu* and Y.C. Soh "Robust Adaptive Gradient Descent Training Algorithm for Recurrent Neural Networks in Discrete Time Domain", IEEE Transactions on Neural Networks, Vol.19, No. 11, 2008, pp.1841-1853. 22. Qing Song#, J. Spall, Y.C. Soh and J. Ni*, "Robust Neural Network Tracking Controller Using Simultaneous Perturbation Stochastic Approximation”, IEEE Transactions on Neural Networks, Vol.19, No.5, 2008, pp. 817-835. 23. Y.L. Wu*, Qing Song#, and S. Liu, "A Normalized Adaptive Training of Recurrent Neural Networks with Augmented Error Gradient" , IEEE Transactions on Neural Networks, Vol.19, No.2, 2008, pp.351-356. 24. A. Z. Cao* and Qing Song#, “Robust Information Clustering for Automatic Breast Mass Detection in Digitized Mammograms”, Computer Vision and Image Understanding, No. 1, Vol. 109, 2008, pp.86-96. 25. X. L. Yang*#, Qing Song and A. Z. Cao*, “A Modified Deterministic Annealing Algorithm for Robust Image Segmentation”, Journal of mathematical imaging and vision, Vol. 30, No. 3, 2008, pp. 308-324. 26. X.L. Yang*, Qing Song, "A weighted support vector machine for data classification", International Journal of Pattern Recognition and Artificial Intelligence, No. 5, Vol. 21, 2007, pp.961-976. 27. J. Ni* and Qing Song, “Pruning Based Robust Backpropagation Training Algorithm for RBF Network Tracking Controller”, Journal of Intelligent and Robotics Systems, Vol. 48, No.3, 2007, pp.375-396. 28. Y.L. Wu* and Qing Song, “Robust Recurrent Neural Control of Biped Robot”, Journal of Intelligent and Robotics Systems, No. 2, Vol. 49, 2007, pp.151-169. 29. X.L. Yang* and Qing Song#, “A Robust Deterministic Annealing Algorithm for Data Clustering”, Data & Knowledge Engineering, 62 (1), 2007, pp.84-100. 30. C.Y. Guo*, Qing Song#, and W.J. Cai, “Neural Network Assisted Cascade Control System for Air Handling Unit”, IEEE Transactions On Industrial Electronic,Vol. 54, No.1, 2007, pp.620-628. 31. J. Ni* and Qing Song. ”Dynamic Pruning Algorithm for Multilayer Perceptron Based Neural Control Systems”, Journal of Neurocomputing, Vol. 69, Issues 16-18, 2006. 32. X.L. Yang* and Qing Song, Kernel-based deterministic annealing algorithm for data clustering, IEE Proceedings: Vision, Image and Signal Processing 153 (5), pp. 557-568, 2006. 33. X.L. Yang*, Qing Song, A.Z. Cao*, " A weighted deterministic annealing algorithm for data clustering", International Journal of Computational Intelligent Research, Vol.2, No.1, Pages:81-85, 2006. 34. X.L. Yang*, Qing Song# and A. Z. Cao*, “A New Cluster Validity for Data Clustering”, Neural Processing Letter, 23 (3), 2006, pp. 325-344. 35. Qing Song#, “A Robust Information Clustering Algorithm”, Neural Computation, MIT, Vol.17, No.12, 2005, pp.2672-2698. 36. W.J.Hu* and Qing.Song “An Accelerated Training Algorithm for Robust Support Vector Machine”, IEEE Transactions on Circuits and Systems II, Vol. 51, No.5, 2004. 37. Qing. Song#, W. J. Hu*, “Robust Neural Controller for VAV System”, IEE Proceedings Part D - Control Theory and Applications, Vol. 150, No. 2, 2003. 38. Qing Song, W. J. Hu*, W. F. Xie**, “Robust Support Vector Machine for Bullet Hole Image Classification”, IEEE Transactions on Systems, Man and Cybernetics, Part C, Vol. 32, 2002. 39. Qing Song, L. Yin, Y. C. Soh, “Robust Identification of Nonlinear Plant using Neural Networks”, Asian Journal of Control, Vol. 3, No. 2, 2001, Special Issue on Advances in Neural and Fuzzy Controllers. 40. Qing Song, L. Yin, Y. C. Soh, “Robust Adaptive Dead Zone Technology for Fault Diagnosis and Control of Robot Manipulators using Neural Networks”, Journal of Intelligent and Robotics Systems, the Netherlands, 2001. 41. Qing Song, Lin Yin, “Robust Adaptive Fault Accommodation for Robot System using RBF Neural Network”, International Journal of Systems Science, UK, Vol. 32, No. 2, 2001. 42. Qing Song, “Design of Robust Neural Tracking Controller”, Journal of Intelligent and Robotics Systems, the Netherlands, Vol. 20, 2000. 43. D. J. Hou, Qing Song, “Computerised Auto-Scoring System Based upon Feature Extraction and Neural Network Technologies”, Journal of Intelligent and Robotics Systems, the Netherlands, Vol. 29, 2000, Special Issue of Neural Network and Image Processing. 44. Qing Song#, J. Xiao, Y. C. Soh, “Robust Back-Propagation Algorithm for Multi-Layered Neural Tracking Controller”, IEEE Transactions on Neural Networks, USA, Vol. 10, No. 5, 1999, pp. 1133-1141. 45. Qing Song, “Implementation of Two Dimensional Systolic Algorithms for Multilayered Neural Networks”, Journal of Systems Architecture, Italy, Vol. 48, pp.1209-1218, 1999. 46. Qing Song, L. Yin, Y. C. Soh, “Shifting of the Center of Radial Basis Function Neural Network in the Presence of Disturbance”, Neural and Parallel Computation, USA, Vol. 7, No. 1, pp. 121-131, 1999. 47. Qing Song, “Robust Training Algorithm of Multi-Layered Neural Network for Identification of Nonlinear Dynamic Systems”, IEE Proceedings Part D - Control Theory and Applications, UK, Vol. 145, No. 1, 1998. 48. Qing Song, M. J. Grimble, “Design of a Multivariable Neural Controller and its Application to Gas Turbine”, ASME Journal of Dynamic Systems, Measurement and Control, USA, Vol. 119, No. 3, 1997. 49. Qing Song, J. Xiao, “On the Convergence Performance of Multi-Layered NN Tracking Controller”, Neural and Parallel Computation, USA, Vol. 5, No. 3, 1997. 50. Qing Song, “Training of a Single Perceptron Neuron using Robust Projection Algorithm”, Neural and Parallel Computation, USA, Vol. 4, No. 2, pp. 193-203, 1996. 51. Qing Song, E. K. Teoh, D. P. Mital, “Multilayered Neural Network Implementation on Transputer Systolic Array”, Journal of Microprocessing and Microprogramming, the Netherlands, Vol. 41, , pp. 289-299, 1995. 52. Qing Song, M. R. Katebi, M. J. Grimble, “Robust Multivariable Implicit Adaptive Control”, IMA Journal of Mathematical Control and Information, UK, Vol. 10, pp. 49-70, 1993. 53. Qing Song, J. Wilkie, M. J. Grimble, “Robust Controller for Gas Turbines Based Upon LQG/LTR Design with Self-Tuning Features”, ASME Journal of Dynamic Systems, Measurement and Control, USA, Vol. 115, No. 3, pp. 569-571, 1993.