Hello pupils! Welcome to the next session of the neural network series. I hope you are doing good. In the previous part of this series, I showed the double deep Q networks and discussed their differences from the deep Q network to make things clear. Today, I am going to visit a very popular neural network with you. This is the spiking neural network that mimics the functionality of the biological neurons with the help of spikes. This is a different neural network than the traditional networks and you will see the details of each point.
In this lecture, we’ll understand the introduction of the spiking neural network. We’ll discuss all the basic terms that are used while studying the SNN. After that, we’ll move on to the steps of using SNN in detail. In the end, we’ll move towards the applications of the SNN and understand how its similar structure to the brain helps to improve different applications.
Introduction to Spiking Neural Networks
The spiking neural networks (SNN) show a unique and inspiring neural network approach that is a perfect combination of deep learning neural networks, biological structure, and computational neuroscience. For their performance, the SNN uses spikes or pulses of electrical conductivity to communicate the information from one place to another. It is defined as:
"The spiking neural networks (SNN) are deep learning artificial neural networks that are inspired by biological structure and mechanisms and work with the help of discrete and precisely designed events known as spikes."
In traditional neural networks, continuous values are used to represent the activation functions but here, the continuous values are smooth and easy to implement with better performance.
History of Spiking Neural Networks
The last decade has witnessed the seamless applications and features of artificial neural networks. But the history of these networks is older than this. The spiking neural networks can be traced back to the early neural networks. Here are some important highlights of the introduction and growth of SNN:
In 1952, Alan Hodgkin and Andrew Huxley were the first to publish their thoughts in research about squid giant axons’s action potential. This helped others understand the biophysical basis and this was the foundation for the idea of spiking.
In the same decade, Warren McCulloch and Walter Pitts presented the McCulloch-Pitts neuron, which is the first mathematical neuron model. This model is the foundation of early artificial neural networks. It utilizes the binary activation values.
In the 1960s, Frank Rosenblatt was successful in developing the perceptrons. It is a single-layer artificial neural network that is able to perform simple and basic tasks. This was first appreciated well but after that, people started criticizing it because it was useful on a very small level.
In 1970, Bernard Widrow and Ted Hoff presented Adaptive Linear Neuron (ADALINE). It is also a single-layer neural network but it works on continuously valued activation functions. Other people worked more on its improvements and as a result, better networks and outputs were seen during this time.
In the 2000s, research was performed on the neurons and this gave rise to mimicking structure in SNN. It resulted in the interest of other scientists in these techniques and the work on the spikings was boosted. This was the time when new algorithms and techniques were introduced for the SNN, and the improved performance not only showed more interest among the people but also broadened the domains of the SNN.
Currently, SNN is being used in different fields such as robotics, healthcare, artificial intelligence, etc. You will see the details of applications at the end of this article.
Basic Concept of Spiking Neural Networks
It's better to understand the basic concepts to understand the working principles and applications of SNN. These are the terms often used when dealing with spiking neural networks:
What are Spikes in SNN?
The spikes are the fundamental unit of communication in the spiking neural networks. These are also known as action potentials and are the brief pulses of electrical activity.
A spike is a sudden, rapid, and transient change that represents the output of the neuron.
These are in the form of firing neurons and are responsible for the transition of the neurons in the whole network.
The SNN relies on the spikes for the transmission of the data. This point is different from the traditional neural network where continuous activation functions are required for this purpose.
The information on the spikes like the timing and frequency are important factors of the network.
If the spikes have a precise relative timing to each other then these can encode the temporal information. Hence the SNN capture the dynamic nature of the biological neural system.
Spikes also play a fundamental role in the computational capabilities. They have multiple features related to computational capabilities such as:
Temporal data more effectively
Handle the complex spatiotemporal pattern
Potentially operate in a more energy-efficient manner (as compared to traditional artificial neural networks)
The advancement in the spikes research is resulting in more powerful SNNs.
Membrane Potential in Spiking Neural Network
In biological neurons, the cell membrane is responsible for maintaining the difference between the intracellular and extracellular environments. A similar concept is also present in the membrane potential of the spiking neural networks. Usually, the membrane potential is different in both these environments.
The membrane potential is the key concept in SNN that describes the electric potential difference across the cell membrane.
This is the dynamic quantity therefore, it changes with time and determines if the neuron has to generate the spike or not.
The neuron in SNN has the threshold membrane potential (discussed below). If the potential is less than this, no change occurs in it, Otherwise, the spike is generated.
What is the Threshold Potential in SNN?
The threshold potential is a specific minimum voltage level that a neuron must reach to generate the action potential (spike). Hence, it can be considered as a border of potential values and this is described as:
If
Potential values Then Neuron does not produce a spike If Potential values>=threshold value Then Neuron produces spike In SNN, the synaptic Weight is the measure of the connection strength of two neurons. This has an effect on the influence of one neuron on the other. Strong synaptic weight means a more substantial effect on the receiving neuron. As a result, there are more chances of firing the spike because of the incoming signal from such a neuron. The opposite case is in the weak neuron. As the name suggests, the excitatory input of the SNN is the type of input signal that results in more firing of spikes. The excitatory input results in the following processes in SNN: The input results in the depolarization of the neuron The membrane potential increases because of depolarization The potential may reach the threshold potential value The result of this value can be in the firing of a spike The inhibitory input is the opposite of the excitatory input. This results in the inhibition of the firing of spikes. The following processes occur in neurons when inhibitory input is added: The inhibitory input results in the hyperpolarization of the neuron The overall membrane potential decreases The neuron moves far from the threshold potential value There are less chances of spike firing A better understanding of this concept will be achieved when you know the following terms: A presynaptic neuron is one that sends the signal to the other neuron. The neuron that receives the signal from the presynaptic neuron. A port synaptic potential is any change in the membrane potential caused by the presynaptic neuron. It is the combinational effect of the excitatory input and inhibitory input. The collective effect of both of these changes the values of the membrane potential and if it touches the threshold potential, it results in the spike generation and vice versa. Temporal coding is the process of encoding the information in the neuron of SNN. Temporal coding is a more reliable method in SNN because it does not just rely on the firing rate of spikes but it also involves the information of the occurrence of spikes. In this way, the more precise and detailed information of the data. The rate coding is another type of coding where the average timing of neuron firing is involved. It involves information on the average firing rate of spikes. Other related information such as spikes in frequency over a given time. It is a different coding method from the temporal coding. The synapses are an important concept in SNN and it is defined as: "The synapses in SNN are the specialized junctions between two neurons and these play a crucial role in the communication between these two." In synapses, the synaptic plasticity is their ability to change their strength according to the experience in the SNN. it is done by making changes in the weights of synapses and as a result, the connection is modified to a stronger or weaker force according to the case. This is an important feature to understand. Just like the biological learning principles, that move towards the optimization of the whole system according to environment, the learning process of SNN is intelligent enough to provide the best performance. It means the modification of the synaptic weights according to the current condition of the network. As a result, the system of SNN works to move towards stability and optimization according to the environment. Through the basic concepts of the spiking neural network, the working principle of the spiking neural network is clear to you. Now, there is a need to discuss the flow of all the processes occurring in SNN. The working in SNN is accomplished in five steps given next: The setting of input and Synaptic Weights Membrane Potential Update process Spike Generation in SNN Spike Propagation in SNN Learning and Plasticity for the final results in SNN Here are the details of each step that will be easy for you to understand: The first step is to initialize the neurons to create the network. Each neuron has its specific features such as membrane potential, threshold values, etc. The information of a specific neuron is based on the spikes. These have synaptic weights that determine the strength of the presynaptic neuron to the postsynaptic neuron. Once the network is arranged successfully according to the requirements, the firing of the spikes occurs. Here, when the presynaptic neuron generates spikes, it transmits the signals. There is an effect on the potential difference of postsynaptic neurons. The nature of synapses decides if the signal is an inhibitory input or an excitatory input (as discussed above). The membrane potential continuously updates throughout the whole process. The overall effect of both these inputs results in the final membrane potential of neurons at a specific point. The membrane potential has a specific threshold value. If the potential reaches this value, the postsynaptic neuron fires the spikes. The inhibitory and excitatory inputs collectively influence the timing of the spikes. Every neuron can encode information like spiking frequency, etc. The firing of spikes results in the propagation of the signal to the next neuron in the network. This process is continuous throughout the network and results in the influence of the signal on sending and receiving neurons. The propagation of the spikes occurs throughout the network and after some time, the weight of the neuron is modified in the process of synaptic plasticity. This process depends on the multiple values in neurons and it affects the learning process of the network. This not only helps in the growth and learning of the network but allows it to adopt new information and stimulate multiple processes throughout the network. Spiking neural networks are one of the most popular emerging techniques in deep learning. The working of these networks is different from that of traditional neural networks; therefore, they have a little bit different and complex applications. Here are some of the main domains where SNN is being used along with other neural networks but the output of the SNN is different from others: In neuromorphic computations, the SNN is used for the development of specialized hardware and software systems. These are the copies or mimicry of the structure and features of the human brain. These computing chips are used for different purposes where memory and related features are required. For instance, the SNN is used in neuromorphic chips that offer high processing speed and efficiency in energy usage. The SNN plays a role in areas where sensory information is required to get better output. For instance, in fields where vision or audio recognition is required for the output, SNN is used for better processing because these can work on the spatiotemporal patterns. As a result, SNN has major applications in speech, voice, and vision recognition systems. The spiking neural networks are used in the specialized cameras. These are called event-based cameras and are designed to capture the changes of the event in the frame, unlike traditional cameras. These cameras have applications such as: Object tracking Motion analysis Gesture recognition Motion detection There are different processes in the field of brain-computer interfaces that can be improved with the help of SNN. For instance, communication or control processes are made better using this neural network because it has the feature of temporal dynamics. This allows it to do better with spiking behaviours, just like the human brain. The brain-like working of SNN is suitable for cognitive modeling. Usually, the researchers use SNN to understand the functionality and working of the neural networks and learn how they deal with cognitive mechanisms and learning tasks. SNN can work on the temporal aspects that help them in processes like: Information processing Decision making Human cognition This helps to improve the functionality of the system. One of the important applications of SNN is in neuroprosthetics, where it is implemented on specialized hardware chips. These chips are designed to be used in processes like edge computation and processing using sensors. As a result, these present parallelism and efficiency. Hence, today we have seen the details of spiking neural networks. These are the modern networks that are based on a similar structure of the brain. We started with the basic definition of SNN and saw the core concept that helped us understand the flow of the spiking neural network. After that, we have seen the details of the application of SNN to understand that it is widely used in domains where human brain-like behavior is required. I hope you find this article useful. If you have any questions, you can ask them in the comment section.Synaptic Weight in Spike Neural Network
Excitatory Input in SNN
Inhibitory Input in SNN
Post-Synaptic Potential (PSP) in SNN
Temporal Coding in SNN
Rate Coding in SNN
Synaptic Plasticity
Learning in SNNs
Working of Spiking Neural Networks
Initialization of Neurons in SNN
Update in the Membrane Potential of SNN
Spike Generation in SNN
Spike Propagation in SNN
Learning and Plasticity for the final results in SNN
Applications of Spiking Neural Networks
Neuromorphic Computation with SNN
Sensory Processing Using SNN
Spiking Neural Networks in Event-based Cameras
Brain-Computer Interface (BCI) and SNN
Cognitive Modeling Process using SNN
Use of SNN in Neuroprosthetics
