chapter 2 biological neural networks - wordpress.com · chapter 2 biological neural networks ......

19
Chapter 2 Biological neural networks How do biological systems solve problems? How does a system of neurons work? How can we understand its functionality? What are dierent quantities of neurons able to do? Where in the nervous system does information processing occur? A short biological overview of the complexity of simple elements of neural information processing followed by some thoughts about their simplification in order to technically adapt them. Before we begin to describe the technical side of neural networks, it would be use- ful to briefly discuss the biology of neu- ral networks and the cognition of living organisms – the reader may skip the fol- lowing chapter without missing any tech- nical information. On the other hand I recommend to read the said excursus if you want to learn something about the underlying neurophysiology and see that our small approaches, the technical neural networks, are only caricatures of nature – and how powerful their natural counter- parts must be when our small approaches are already that eective. Now we want to take a brief look at the nervous system of vertebrates: We will start with a very rough granularity and then proceed with the brain and up to the neural level. For further reading I want to recommend the books [CR00, KSJ00], which helped me a lot during this chapter. 2.1 The vertebrate nervous system The entire information processing system, i.e. the vertebrate nervous system, con- sists of the central nervous system and the peripheral nervous system, which is only a first and simple subdivision. In real- ity, such a rigid subdivision does not make sense, but here it is helpful to outline the information processing in a body. 2.1.1 Peripheral and central nervous system The peripheral nervous system (PNS ) comprises the nerves that are situated out- side of the brain or the spinal cord. These nerves form a branched and very dense net- work throughout the whole body. The pe- 13

Upload: trinhnhi

Post on 19-Mar-2019

230 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Chapter 2 Biological neural networks - WordPress.com · Chapter 2 Biological neural networks ... brainstem or the (truncus cerebri)re-spectively is phylogenetically much older. Roughly

Chapter 2

Biological neural networksHow do biological systems solve problems? How does a system of neurons

work? How can we understand its functionality? What are di�erent quantitiesof neurons able to do? Where in the nervous system does information

processing occur? A short biological overview of the complexity of simpleelements of neural information processing followed by some thoughts about

their simplification in order to technically adapt them.

Before we begin to describe the technicalside of neural networks, it would be use-ful to briefly discuss the biology of neu-ral networks and the cognition of livingorganisms – the reader may skip the fol-lowing chapter without missing any tech-nical information. On the other hand Irecommend to read the said excursus ifyou want to learn something about theunderlying neurophysiology and see thatour small approaches, the technical neuralnetworks, are only caricatures of nature– and how powerful their natural counter-parts must be when our small approachesare already that e�ective. Now we wantto take a brief look at the nervous systemof vertebrates: We will start with a veryrough granularity and then proceed withthe brain and up to the neural level. Forfurther reading I want to recommend thebooks [CR00, KSJ00], which helped me alot during this chapter.

2.1 The vertebrate nervoussystem

The entire information processing system,i.e. the vertebrate nervous system, con-sists of the central nervous system and theperipheral nervous system, which is onlya first and simple subdivision. In real-ity, such a rigid subdivision does not makesense, but here it is helpful to outline theinformation processing in a body.

2.1.1 Peripheral and centralnervous system

The peripheral nervous system (PNS)comprises the nerves that are situated out-side of the brain or the spinal cord. Thesenerves form a branched and very dense net-work throughout the whole body. The pe-

13

Page 2: Chapter 2 Biological neural networks - WordPress.com · Chapter 2 Biological neural networks ... brainstem or the (truncus cerebri)re-spectively is phylogenetically much older. Roughly

Chapter 2 Biological neural networks dkriesel.com

ripheral nervous system includes, for ex-ample, the spinal nerves which pass outof the spinal cord (two within the level ofeach vertebra of the spine) and supply ex-tremities, neck and trunk, but also the cra-nial nerves directly leading to the brain.

The central nervous system (CNS),however, is the "main-frame" within thevertebrate. It is the place where infor-mation received by the sense organs arestored and managed. Furthermore, it con-trols the inner processes in the body and,last but not least, coordinates the mo-tor functions of the organism. The ver-tebrate central nervous system consists ofthe brain and the spinal cord (Fig. 2.1).However, we want to focus on the brain,which can - for the purpose of simplifica-tion - be divided into four areas (Fig. 2.2on the next page) to be discussed here.

2.1.2 The cerebrum is responsiblefor abstract thinkingprocesses.

The cerebrum (telencephalon) is one ofthe areas of the brain that changed mostduring evolution. Along an axis, runningfrom the lateral face to the back of thehead, this area is divided into two hemi-spheres, which are organized in a foldedstructure. These cerebral hemispheresare connected by one strong nerve cord("bar") and several small ones. A largenumber of neurons are located in the cere-bral cortex (cortex) which is approx. 2-4 cm thick and divided into di�erent cor-tical fields, each having a specific task to Figure 2.1: Illustration of the central nervous

system with spinal cord and brain.

14 D. Kriesel – A Brief Introduction to Neural Networks (ZETA2-EN)

Page 3: Chapter 2 Biological neural networks - WordPress.com · Chapter 2 Biological neural networks ... brainstem or the (truncus cerebri)re-spectively is phylogenetically much older. Roughly

dkriesel.com 2.1 The vertebrate nervous system

Figure 2.2: Illustration of the brain. The col-ored areas of the brain are discussed in the text.The more we turn from abstract information pro-cessing to direct reflexive processing, the darkerthe areas of the brain are colored.

fulfill. Primary cortical fields are re-sponsible for processing qualitative infor-mation, such as the management of di�er-ent perceptions (e.g. the visual cortexis responsible for the management of vi-sion). Association cortical fields, how-ever, perform more abstract associationand thinking processes; they also containour memory.

2.1.3 The cerebellum controls andcoordinates motor functions

The cerebellum is located below the cere-brum, therefore it is closer to the spinalcord. Accordingly, it serves less abstractfunctions with higher priority: Here, largeparts of motor coordination are performed,i.e., balance and movements are controlled

and errors are continually corrected. Forthis purpose, the cerebellum has directsensory information about muscle lengthsas well as acoustic and visual informa-tion. Furthermore, it also receives mes-sages about more abstract motor signalscoming from the cerebrum.

In the human brain the cerebellum is con-siderably smaller than the cerebrum, butthis is rather an exception. In many ver-tebrates this ratio is less pronounced. Ifwe take a look at vertebrate evolution, wewill notice that the cerebellum is not "toosmall" but the cerebum is "too large" (atleast, it is the most highly developed struc-ture in the vertebrate brain). The two re-maining brain areas should also be brieflydiscussed: the diencephalon and the brain-stem.

2.1.4 The diencephalon controlsfundamental physiologicalprocesses

The interbrain (diencephalon) includesparts of which only the thalamus will thalamus

filtersincomingdata

be briefly discussed: This part of the di-encephalon mediates between sensory andmotor signals and the cerebrum. Particu-larly, the thalamus decides which part ofthe information is transferred to the cere-brum, so that especially less importantsensory perceptions can be suppressed atshort notice to avoid overloads. Anotherpart of the diencephalon is the hypotha-lamus, which controls a number of pro-cesses within the body. The diencephalon

D. Kriesel – A Brief Introduction to Neural Networks (ZETA2-EN) 15

Page 4: Chapter 2 Biological neural networks - WordPress.com · Chapter 2 Biological neural networks ... brainstem or the (truncus cerebri)re-spectively is phylogenetically much older. Roughly

Chapter 2 Biological neural networks dkriesel.com

is also heavily involved in the human cir-cadian rhythm ("internal clock") and thesensation of pain.

2.1.5 The brainstem connects thebrain with the spinal cord andcontrols reflexes.

In comparison with the diencephalon thebrainstem or the (truncus cerebri) re-spectively is phylogenetically much older.Roughly speaking, it is the "extendedspinal cord" and thus the connection be-tween brain and spinal cord. The brain-stem can also be divided into di�erent ar-eas, some of which will be exemplarily in-troduced in this chapter. The functionswill be discussed from abstract functionstowards more fundamental ones. One im-portant component is the pons (=bridge),a kind of transit station for many nerve sig-nals from brain to body and vice versa.

If the pons is damaged (e.g. by a cere-bral infarct), then the result could be thelocked-in syndrome – a condition inwhich a patient is "walled-in" within hisown body. He is conscious and awarewith no loss of cognitive function, but can-not move or communicate by any means.Only his senses of sight, hearing, smell andtaste are generally working perfectly nor-mal. Locked-in patients may often be ableto communicate with others by blinking ormoving their eyes.

Furthermore, the brainstem is responsiblefor many fundamental reflexes, such as theblinking reflex or coughing.

All parts of the nervous system have onething in common: information processing.This is accomplished by huge accumula-tions of billions of very similar cells, whosestructure is very simple but which com-municate continuously. Large groups ofthese cells send coordinated signals andthus reach the enormous information pro-cessing capacity we are familiar with fromour brain. We will now leave the level ofbrain areas and continue with the cellularlevel of the body - the level of neurons.

2.2 Neurons are informationprocessing cells

Before specifying the functions and pro-cesses within a neuron, we will give arough description of neuron functions: Aneuron is nothing more than a switch withinformation input and output. The switchwill be activated if there are enough stim-uli of other neurons hitting the informa-tion input. Then, at the information out-put, a pulse is sent to, for example, otherneurons.

2.2.1 Components of a neuron

Now we want to take a look at the com-ponents of a neuron (Fig. 2.3 on the fac-ing page). In doing so, we will follow theway the electrical information takes withinthe neuron. The dendrites of a neuronreceive the information by special connec-tions, the synapses.

16 D. Kriesel – A Brief Introduction to Neural Networks (ZETA2-EN)

Page 5: Chapter 2 Biological neural networks - WordPress.com · Chapter 2 Biological neural networks ... brainstem or the (truncus cerebri)re-spectively is phylogenetically much older. Roughly

dkriesel.com 2.2 The neuron

Figure 2.3: Illustration of a biological neuron with the components discussed in this text.

2.2.1.1 Synapses weight the individualparts of information

Incoming signals from other neurons orcells are transferred to a neuron by specialconnections, the synapses. Such connec-tions can usually be found at the dendritesof a neuron, sometimes also directly at thesoma. We distinguish between electricaland chemical synapses.

The electrical synapse is the simplerelectricalsynapse:

simplevariant. An electrical signal received bythe synapse, i.e. coming from the presy-naptic side, is directly transferred to thepostsynaptic nucleus of the cell. Thus,there is a direct, strong, unadjustableconnection between the signal transmitterand the signal receiver, which is, for exam-ple, relevant to shortening reactions thatmust be "hard coded" within a living or-ganism.

The chemical synapse is the more dis-tinctive variant. Here, the electrical cou-pling of source and target does not takeplace, the coupling is interrupted by thesynaptic cleft. This cleft electrically sep-arates the presynaptic side from the post-synaptic one. You might think that, never-theless, the information has to flow, so wewill discuss how this happens: It is not anelectrical, but a chemical process. On thepresynaptic side of the synaptic cleft theelectrical signal is converted into a chemi-cal signal, a process induced by chemicalcues released there (the so-called neuro-transmitters). These neurotransmitterscross the synaptic cleft and transfer theinformation into the nucleus of the cell(this is a very simple explanation, but lateron we will see how this exactly works),where it is reconverted into electrical in-formation. The neurotransmitters are de-graded very fast, so that it is possible to re-

D. Kriesel – A Brief Introduction to Neural Networks (ZETA2-EN) 17

Page 6: Chapter 2 Biological neural networks - WordPress.com · Chapter 2 Biological neural networks ... brainstem or the (truncus cerebri)re-spectively is phylogenetically much older. Roughly

Chapter 2 Biological neural networks dkriesel.com

lease very precise information pulses here,too.

In spite of the more complex function-cemicalsynapseis more

complexbut also

morepowerful

ing, the chemical synapse has - comparedwith the electrical synapse - utmost advan-tages:

One-way connection: A chemicalsynapse is a one-way connection.Due to the fact that there is no directelectrical connection between thepre- and postsynaptic area, electricalpulses in the postsynaptic areacannot flash over to the presynapticarea.

Adjustability: There is a large number ofdi�erent neurotransmitters that canalso be released in various quantitiesin a synaptic cleft. There are neuro-transmitters that stimulate the post-synaptic cell nucleus, and others thatslow down such stimulation. Somesynapses transfer a strongly stimulat-ing signal, some only weakly stimu-lating ones. The adjustability variesa lot, and one of the central pointsin the examination of the learningability of the brain is, that here thesynapses are variable, too. That is,over time they can form a stronger orweaker connection.

2.2.1.2 Dendrites collect all parts ofinformation

Dendrites branch like trees from the cellnucleus of the neuron (which is calledsoma) and receive electrical signals from

many di�erent sources, which are thentransferred into the nucleus of the cell.The amount of branching dendrites is alsocalled dendrite tree.

2.2.1.3 In the soma the weightedinformation is accumulated

After the cell nucleus (soma) has re-ceived a plenty of activating (=stimulat-ing) and inhibiting (=diminishing) signalsby synapses or dendrites, the soma accu-mulates these signals. As soon as the ac-cumulated signal exceeds a certain value(called threshold value), the cell nucleusof the neuron activates an electrical pulsewhich then is transmitted to the neuronsconnected to the current one.

2.2.1.4 The axon transfers outgoingpulses

The pulse is transferred to other neuronsby means of the axon. The axon is along, slender extension of the soma. Inan extreme case, an axon can stretch upto one meter (e.g. within the spinal cord).The axon is electrically isolated in orderto achieve a better conduction of the elec-trical signal (we will return to this pointlater on) and it leads to dendrites, whichtransfer the information to, for example,other neurons. So now we are back at thebeginning of our description of the neuronelements. An axon can, however, transferinformation to other kinds of cells in orderto control them.

18 D. Kriesel – A Brief Introduction to Neural Networks (ZETA2-EN)

Page 7: Chapter 2 Biological neural networks - WordPress.com · Chapter 2 Biological neural networks ... brainstem or the (truncus cerebri)re-spectively is phylogenetically much older. Roughly

dkriesel.com 2.2 The neuron

2.2.2 Electrochemical processes inthe neuron and itscomponents

After having pursued the path of an elec-trical signal from the dendrites via thesynapses to the nucleus of the cell andfrom there via the axon into other den-drites, we now want to take a small stepfrom biology towards technology. In doingso, a simplified introduction of the electro-chemical information processing should beprovided.

2.2.2.1 Neurons maintain electricalmembrane potential

One fundamental aspect is the fact thatcompared to their environment the neu-rons show a di�erence in electrical charge,a potential. In the membrane (=enve-lope) of the neuron the charge is di�erentfrom the charge on the outside. This dif-ference in charge is a central concept thatis important to understand the processeswithin the neuron. The di�erence is calledmembrane potential. The membranepotential, i.e., the di�erence in charge, iscreated by several kinds of charged atoms(ions), whose concentration varies withinand outside of the neuron. If we penetratethe membrane from the inside outwards,we will find certain kinds of ions more of-ten or less often than on the inside. Thisdescent or ascent of concentration is calleda concentration gradient.

Let us first take a look at the membranepotential in the resting state of the neu-

ron, i.e., we assume that no electrical sig-nals are received from the outside. In thiscase, the membrane potential is ≠70 mV.Since we have learned that this potentialdepends on the concentration gradients ofvarious ions, there is of course the centralquestion of how to maintain these concen-tration gradients: Normally, di�usion pre-dominates and therefore each ion is eagerto decrease concentration gradients andto spread out evenly. If this happens,the membrane potential will move towards0 mV, so finally there would be no mem-brane potential anymore. Thus, the neu-ron actively maintains its membrane po-tential to be able to process information.How does this work?

The secret is the membrane itself, which ispermeable to some ions, but not for others.To maintain the potential, various mecha-nisms are in progress at the same time:

Concentration gradient: As describedabove the ions try to be as uniformlydistributed as possible. If theconcentration of an ion is higher onthe inside of the neuron than onthe outside, it will try to di�useto the outside and vice versa.The positively charged ion K+

(potassium) occurs very frequentlywithin the neuron but less frequentlyoutside of the neuron, and thereforeit slowly di�uses out through theneuron’s membrane. But anothergroup of negative ions, collectivelycalled A≠, remains within the neuronsince the membrane is not permeableto them. Thus, the inside of theneuron becomes negatively charged.

D. Kriesel – A Brief Introduction to Neural Networks (ZETA2-EN) 19

Page 8: Chapter 2 Biological neural networks - WordPress.com · Chapter 2 Biological neural networks ... brainstem or the (truncus cerebri)re-spectively is phylogenetically much older. Roughly

Chapter 2 Biological neural networks dkriesel.com

Negative A ions remain, positive Kions disappear, and so the inside ofthe cell becomes more negative. Theresult is another gradient.

Electrical Gradient: The electrical gradi-ent acts contrary to the concentrationgradient. The intracellular charge isnow very strong, therefore it attractspositive ions: K+ wants to get backinto the cell.

If these two gradients were now left alone,they would eventually balance out, reacha steady state, and a membrane poten-tial of ≠85 mV would develop. But wewant to achieve a resting membrane po-tential of ≠70 mV, thus there seem to ex-ist some disturbances which prevent this.Furthermore, there is another importantion, Na+ (sodium), for which the mem-brane is not very permeable but which,however, slowly pours through the mem-brane into the cell. As a result, the sodiumis driven into the cell all the more: On theone hand, there is less sodium within theneuron than outside the neuron. On theother hand, sodium is positively chargedbut the interior of the cell has negativecharge, which is a second reason for thesodium wanting to get into the cell.

Due to the low di�usion of sodium into thecell the intracellular sodium concentrationincreases. But at the same time the insideof the cell becomes less negative, so thatK+ pours in more slowly (we can see thatthis is a complex mechanism where every-thing is influenced by everything). Thesodium shifts the intracellular equilibriumfrom negative to less negative, compared

with its environment. But even with thesetwo ions a standstill with all gradients be-ing balanced out could still be achieved.Now the last piece of the puzzle gets intothe game: a "pump" (or rather, the proteinATP) actively transports ions against thedirection they actually want to take!

Sodium is actively pumped out of the cell,although it tries to get into the cellalong the concentration gradient andthe electrical gradient.

Potassium, however, di�uses strongly outof the cell, but is actively pumpedback into it.

For this reason the pump is also calledsodium-potassium pump. The pumpmaintains the concentration gradient forthe sodium as well as for the potassium,so that some sort of steady state equilib-rium is created and finally the resting po-tential is ≠70 mV as observed. All in allthe membrane potential is maintained bythe fact that the membrane is imperme-able to some ions and other ions are ac-tively pumped against the concentrationand electrical gradients. Now that weknow that each neuron has a membranepotential we want to observe how a neu-ron receives and transmits signals.

2.2.2.2 The neuron is activated bychanges in the membranepotential

Above we have learned that sodium andpotassium can di�use through the mem-brane - sodium slowly, potassium faster.

20 D. Kriesel – A Brief Introduction to Neural Networks (ZETA2-EN)

Page 9: Chapter 2 Biological neural networks - WordPress.com · Chapter 2 Biological neural networks ... brainstem or the (truncus cerebri)re-spectively is phylogenetically much older. Roughly

dkriesel.com 2.2 The neuron

They move through channels within themembrane, the sodium and potassiumchannels. In addition to these per-manently open channels responsible fordi�usion and balanced by the sodium-potassium pump, there also exist channelsthat are not always open but which onlyresponse "if required". Since the openingof these channels changes the concentra-tion of ions within and outside of the mem-brane, it also changes the membrane po-tential.

These controllable channels are opened assoon as the accumulated received stimulusexceeds a certain threshold. For example,stimuli can be received from other neuronsor have other causes. There exist, for ex-ample, specialized forms of neurons, thesensory cells, for which a light incidencecould be such a stimulus. If the incom-ing amount of light exceeds the threshold,controllable channels are opened.

The said threshold (the threshold poten-tial) lies at about ≠55 mV. As soon as thereceived stimuli reach this value, the neu-ron is activated and an electrical signal,an action potential, is initiated. Thenthis signal is transmitted to the cells con-nected to the observed neuron, i.e. thecells "listen" to the neuron. Now we wantto take a closer look at the di�erent stagesof the action potential (Fig. 2.4 on the nextpage):

Resting state: Only the permanentlyopen sodium and potassium channelsare permeable. The membranepotential is at ≠70 mV and activelykept there by the neuron.

Stimulus up to the threshold: A stimu-lus opens channels so that sodiumcan pour in. The intracellular chargebecomes more positive. As soon asthe membrane potential exceeds thethreshold of ≠55 mV, the action po-tential is initiated by the opening ofmany sodium channels.

Depolarization: Sodium is pouring in. Re-member: Sodium wants to pour intothe cell because there is a lower in-tracellular than extracellular concen-tration of sodium. Additionally, thecell is dominated by a negative en-vironment which attracts the posi-tive sodium ions. This massive in-flux of sodium drastically increasesthe membrane potential - up to ap-prox. +30 mV - which is the electricalpulse, i.e., the action potential.

Repolarization: Now the sodium channelsare closed and the potassium channelsare opened. The positively chargedions want to leave the positive inte-rior of the cell. Additionally, the intra-cellular concentration is much higherthan the extracellular one, which in-creases the e�ux of ions even more.The interior of the cell is once againmore negatively charged than the ex-terior.

Hyperpolarization: Sodium as well aspotassium channels are closed again.At first the membrane potential isslightly more negative than the rest-ing potential. This is due to thefact that the potassium channels closemore slowly. As a result, (positively

D. Kriesel – A Brief Introduction to Neural Networks (ZETA2-EN) 21

Page 10: Chapter 2 Biological neural networks - WordPress.com · Chapter 2 Biological neural networks ... brainstem or the (truncus cerebri)re-spectively is phylogenetically much older. Roughly

Chapter 2 Biological neural networks dkriesel.com

Figure 2.4: Initiation of action potential over time.

22 D. Kriesel – A Brief Introduction to Neural Networks (ZETA2-EN)

Page 11: Chapter 2 Biological neural networks - WordPress.com · Chapter 2 Biological neural networks ... brainstem or the (truncus cerebri)re-spectively is phylogenetically much older. Roughly

dkriesel.com 2.2 The neuron

charged) potassium e�uses because ofits lower extracellular concentration.After a refractory period of 1 ≠ 2ms the resting state is re-establishedso that the neuron can react to newlyapplied stimuli with an action poten-tial. In simple terms, the refractoryperiod is a mandatory break a neu-ron has to take in order to regenerate.The shorter this break is, the moreoften a neuron can fire per time.

Then the resulting pulse is transmitted bythe axon.

2.2.2.3 In the axon a pulse isconducted in a saltatory way

We have already learned that the axonis used to transmit the action potentialacross long distances (remember: You willfind an illustration of a neuron includingan axon in Fig. 2.3 on page 17). The axonis a long, slender extension of the soma.In vertebrates it is normally coated by amyelin sheath that consists of Schwanncells (in the PNS) or oligodendrocytes(in the CNS) 1, which insulate the axonvery well from electrical activity. At a dis-tance of 0.1≠2mm there are gaps betweenthese cells, the so-called nodes of Ran-vier. The said gaps appear where one in-sulate cell ends and the next one begins.It is obvious that at such a node the axonis less insulated.1 Schwann cells as well as oligodendrocytes are vari-

eties of the glial cells. There are about 50 timesmore glial cells than neurons: They surround theneurons (glia = glue), insulate them from eachother, provide energy, etc.

Now you may assume that these less in-sulated nodes are a disadvantage of theaxon - however, they are not. At thenodes, mass can be transferred betweenthe intracellular and extracellular area, atransfer that is impossible at those partsof the axon which are situated betweentwo nodes (internodes) and therefore in-sulated by the myelin sheath. This masstransfer permits the generation of signalssimilar to the generation of the action po-tential within the soma. The action po-tential is transferred as follows: It doesnot continuously travel along the axon butjumps from node to node. Thus, a seriesof depolarization travels along the nodes ofRanvier. One action potential initiates thenext one, and mostly even several nodesare active at the same time here. Thepulse "jumping" from node to node is re-sponsible for the name of this pulse con-ductor: saltatory conductor.

Obviously, the pulse will move faster if itsjumps are larger. Axons with large in-ternodes (2 mm) achieve a signal disper-sion of approx. 180 meters per second.However, the internodes cannot grow in-definitely, since the action potential to betransferred would fade too much until itreaches the next node. So the nodes havea task, too: to constantly amplify the sig-nal. The cells receiving the action poten-tial are attached to the end of the axon –often connected by dendrites and synapses.As already indicated above, the action po-tentials are not only generated by informa-tion received by the dendrites from otherneurons.

D. Kriesel – A Brief Introduction to Neural Networks (ZETA2-EN) 23

Page 12: Chapter 2 Biological neural networks - WordPress.com · Chapter 2 Biological neural networks ... brainstem or the (truncus cerebri)re-spectively is phylogenetically much older. Roughly

Chapter 2 Biological neural networks dkriesel.com

2.3 Receptor cells aremodified neurons

Action potentials can also be generated bysensory information an organism receivesfrom its environment through its sensorycells. Specialized receptor cells are ableto perceive specific stimulus energies suchas light, temperature and sound or the ex-istence of certain molecules (like, for exam-ple, the sense of smell). This is workingbecause of the fact that these sensory cellsare actually modified neurons. They donot receive electrical signals via dendritesbut the existence of the stimulus beingspecific for the receptor cell ensures thatthe ion channels open and an action po-tential is developed. This process of trans-forming stimulus energy into changes inthe membrane potential is called sensorytransduction. Usually, the stimulus en-ergy itself is too weak to directly causenerve signals. Therefore, the signals areamplified either during transduction or bymeans of the stimulus-conducting ap-paratus. The resulting action potentialcan be processed by other neurons and isthen transmitted into the thalamus, whichis, as we have already learned, a gatewayto the cerebral cortex and therefore can re-ject sensory impressions according to cur-rent relevance and thus prevent an abun-dance of information to be managed.

2.3.1 There are di�erent receptorcells for various types ofperceptions

Primary receptors transmit their pulsesdirectly to the nervous system. A goodexample for this is the sense of pain.Here, the stimulus intensity is propor-tional to the amplitude of the action po-tential. Technically, this is an amplitudemodulation.

Secondary receptors, however, continu-ously transmit pulses. These pulses con-trol the amount of the related neurotrans-mitter, which is responsible for transfer-ring the stimulus. The stimulus in turncontrols the frequency of the action poten-tial of the receiving neuron. This processis a frequency modulation, an encoding ofthe stimulus, which allows to better per-ceive the increase and decrease of a stimu-lus.

There can be individual receptor cells orcells forming complex sensory organs (e.g.eyes or ears). They can receive stimuliwithin the body (by means of the intero-ceptors) as well as stimuli outside of thebody (by means of the exteroceptors).

After having outlined how information isreceived from the environment, it will beinteresting to look at how the informationis processed.

24 D. Kriesel – A Brief Introduction to Neural Networks (ZETA2-EN)

Page 13: Chapter 2 Biological neural networks - WordPress.com · Chapter 2 Biological neural networks ... brainstem or the (truncus cerebri)re-spectively is phylogenetically much older. Roughly

dkriesel.com 2.3 Receptor cells

2.3.2 Information is processed onevery level of the nervoussystem

There is no reason to believe that all re-ceived information is transmitted to thebrain and processed there, and that thebrain ensures that it is "output" in theform of motor pulses (the only thing anorganism can actually do within its envi-ronment is to move). The information pro-cessing is entirely decentralized. In orderto illustrate this principle, we want to takea look at some examples, which leads usagain from the abstract to the fundamen-tal in our hierarchy of information process-ing.

Û It is certain that information is pro-cessed in the cerebrum, which is themost developed natural informationprocessing structure.

Û The midbrain and the thalamus,which serves – as we have alreadylearned – as a gateway to the cere-bral cortex, are situated much lowerin the hierarchy. The filtering of in-formation with respect to the currentrelevance executed by the midbrainis a very important method of infor-mation processing, too. But even thethalamus does not receive any prepro-cessed stimuli from the outside. Nowlet us continue with the lowest level,the sensory cells.

Û On the lowest level, i.e. at the recep-tor cells, the information is not onlyreceived and transferred but directlyprocessed. One of the main aspects of

this subject is to prevent the transmis-sion of "continuous stimuli" to the cen-tral nervous system because of sen-sory adaptation: Due to continu-ous stimulation many receptor cellsautomatically become insensitive tostimuli. Thus, receptor cells are nota direct mapping of specific stimu-lus energy onto action potentials butdepend on the past. Other sensorschange their sensitivity according tothe situation: There are taste recep-tors which respond more or less to thesame stimulus according to the nutri-tional condition of the organism.

Û Even before a stimulus reaches thereceptor cells, information processingcan already be executed by a preced-ing signal carrying apparatus, for ex-ample in the form of amplification:The external and the internal earhave a specific shape to amplify thesound, which also allows – in asso-ciation with the sensory cells of thesense of hearing – the sensory stim-ulus only to increase logarithmicallywith the intensity of the heard sig-nal. On closer examination, this isnecessary, since the sound pressure ofthe signals for which the ear is con-structed can vary over a wide expo-nential range. Here, a logarithmicmeasurement is an advantage. Firstly,an overload is prevented and secondly,the fact that the intensity measure-ment of intensive signals will be lessprecise, doesn’t matter as well. If a jetfighter is starting next to you, small

D. Kriesel – A Brief Introduction to Neural Networks (ZETA2-EN) 25

Page 14: Chapter 2 Biological neural networks - WordPress.com · Chapter 2 Biological neural networks ... brainstem or the (truncus cerebri)re-spectively is phylogenetically much older. Roughly

Chapter 2 Biological neural networks dkriesel.com

changes in the noise level can be ig-nored.

Just to get a feeling for sensory organsand information processing in the organ-ism, we will briefly describe "usual" lightsensing organs, i.e. organs often found innature. For the third light sensing organdescribed below, the single lens eye, wewill discuss the information processing inthe eye.

2.3.3 An outline of common lightsensing organs

For many organisms it turned out to be ex-tremely useful to be able to perceive elec-tromagnetic radiation in certain regions ofthe spectrum. Consequently, sensory or-gans have been developed which can de-tect such electromagnetic radiation andthe wavelength range of the radiation per-ceivable by the human eye is called visiblerange or simply light. The di�erent wave-lengths of this electromagnetic radiationare perceived by the human eye as di�er-ent colors. The visible range of the elec-tromagnetic radiation is di�erent for eachorganism. Some organisms cannot see thecolors (=wavelength ranges) we can see,others can even perceive additional wave-length ranges (e.g. in the UV range). Be-fore we begin with the human being – inorder to get a broader knowledge of thesense of sight– we briefly want to look attwo organs of sight which, from an evolu-tionary point of view, exist much longerthan the human.

2.3.3.1 Compound eyes and pinholeeyes only provide high temporalor spatial resolution

Let us first take a look at the so-calledcompound eye (Fig. 2.5 on the nextpage), which is, for example, common ininsects and crustaceans. The compound Compound eye:

high temp.,lowspatialresolution

eye consists of a great number of small,individual eyes. If we look at the com-pound eye from the outside, the individ-ual eyes are clearly visible and arrangedin a hexagonal pattern. Each individualeye has its own nerve fiber which is con-nected to the insect brain. Since the indi-vidual eyes can be distinguished, it is ob-vious that the number of pixels, i.e. thespatial resolution, of compound eyes mustbe very low and the image is blurred. Butcompound eyes have advantages, too, espe-cially for fast-flying insects. Certain com-pound eyes process more than 300 imagesper second (to the human eye, however,movies with 25 images per second appearas a fluent motion).

Pinhole eyes are, for example, found inoctopus species and work – as you canguess – similar to a pinhole camera. A pinhole

camera:high spat.,lowtemporalresolution

pinhole eye has a very small opening forlight entry, which projects a sharp imageonto the sensory cells behind. Thus, thespatial resolution is much higher than inthe compound eye. But due to the verysmall opening for light entry the resultingimage is less bright.

26 D. Kriesel – A Brief Introduction to Neural Networks (ZETA2-EN)

Page 15: Chapter 2 Biological neural networks - WordPress.com · Chapter 2 Biological neural networks ... brainstem or the (truncus cerebri)re-spectively is phylogenetically much older. Roughly

dkriesel.com 2.3 Receptor cells

Figure 2.5: Compound eye of a robber fly

2.3.3.2 Single lens eyes combine theadvantages of the other twoeye types, but they are morecomplex

The light sensing organ common in verte-brates is the single lense eye. The result-ing image is a sharp, high-resolution imageof the environment at high or variable lightintensity. On the other hand it is morecomplex. Similar to the pinhole eye thelight enters through an opening (pupil)and is projected onto a layer of sensorycells in the eye. (retina). But in contrastSingle

lense eye:high temp.

and spat.resolution

to the pinhole eye, the size of the pupil canbe adapted to the lighting conditions (bymeans of the iris muscle, which expandsor contracts the pupil). These di�erencesin pupil dilation require to actively focusthe image. Therefore, the single lens eyecontains an additional adjustable lens.

2.3.3.3 The retina does not onlyreceive information but is alsoresponsible for informationprocessing

The light signals falling on the eye arereceived by the retina and directly pre-processed by several layers of information-processing cells. We want to briefly dis-cuss the di�erent steps of this informa-tion processing and in doing so, we followthe way of the information carried by thelight:

Photoreceptors receive the light signalund cause action potentials (thereare di�erent receptors for di�erentcolor components and light intensi-ties). These receptors are the reallight-receiving part of the retina andthey are sensitive to such an extentthat only one single photon fallingon the retina can cause an action po-tential. Then several photoreceptorstransmit their signals to one single

bipolar cell. This means that here the in-formation has already been summa-rized. Finally, the now transformedlight signal travels from several bipo-lar cells 2 into

ganglion cells. Various bipolar cells cantransmit their information to one gan-glion cell. The higher the numberof photoreceptors that a�ect the gan-glion cell, the larger the field of per-ception, the receptive field, whichcovers the ganglions – and the less

2 There are di�erent kinds of bipolar cells, as well,but to discuss all of them would go too far.

D. Kriesel – A Brief Introduction to Neural Networks (ZETA2-EN) 27

Page 16: Chapter 2 Biological neural networks - WordPress.com · Chapter 2 Biological neural networks ... brainstem or the (truncus cerebri)re-spectively is phylogenetically much older. Roughly

Chapter 2 Biological neural networks dkriesel.com

sharp is the image in the area of thisganglion cell. So the information isalready reduced directly in the retinaand the overall image is, for exam-ple, blurred in the peripheral fieldof vision. So far, we have learnedabout the information processing inthe retina only as a top-down struc-ture. Now we want to take a look atthe

horizontal and amacrine cells. Thesecells are not connected from thefront backwards but laterally. Theyallow the light signals to influencethemselves laterally directly duringthe information processing in theretina – a much more powerfulmethod of information processingthan compressing and blurring.When the horizontal cells are excitedby a photoreceptor, they are able toexcite other nearby photoreceptorsand at the same time inhibit moredistant bipolar cells and receptors.This ensures the clear perception ofoutlines and bright points. Amacrinecells can further intensify certainstimuli by distributing informationfrom bipolar cells to several ganglioncells or by inhibiting ganglions.

These first steps of transmitting visual in-formation to the brain show that informa-tion is processed from the first moment theinformation is received and, on the otherhand, is processed in parallel within mil-lions of information-processing cells. Thesystem’s power and resistance to errorsis based upon this massive division ofwork.

2.4 The amount of neurons inliving organisms atdi�erent stages ofdevelopment

An overview of di�erent organisms andtheir neural capacity (in large part from[RD05]):

302 neurons are required by the nervoussystem of a nematode worm, whichserves as a popular model organismin biology. Nematodes live in the soiland feed on bacteria.

104 neurons make an ant (To simplifymatters we neglect the fact that someant species also can have more or lesse�cient nervous systems). Due to theuse of di�erent attractants and odors,ants are able to engage in complexsocial behavior and form huge stateswith millions of individuals. If you re-gard such an ant state as an individ-ual, it has a cognitive capacity similarto a chimpanzee or even a human.

With 105 neurons the nervous system ofa fly can be constructed. A fly canevade an object in real-time in three-dimensional space, it can land uponthe ceiling upside down, has a consid-erable sensory system because of com-pound eyes, vibrissae, nerves at theend of its legs and much more. Thus,a fly has considerable di�erential andintegral calculus in high dimensionsimplemented "in hardware". We allknow that a fly is not easy to catch.Of course, the bodily functions are

28 D. Kriesel – A Brief Introduction to Neural Networks (ZETA2-EN)

Page 17: Chapter 2 Biological neural networks - WordPress.com · Chapter 2 Biological neural networks ... brainstem or the (truncus cerebri)re-spectively is phylogenetically much older. Roughly

dkriesel.com 2.4 The amount of neurons in living organisms

also controlled by neurons, but theseshould be ignored here.

With 0.8 · 106 neurons we have enoughcerebral matter to create a honeybee.Honeybees build colonies and haveamazing capabilities in the field ofaerial reconnaissance and navigation.

4 · 106 neurons result in a mouse, andhere the world of vertebrates alreadybegins.

1.5 · 107 neurons are su�cient for a rat,an animal which is denounced as be-ing extremely intelligent and are of-ten used to participate in a varietyof intelligence tests representative forthe animal world. Rats have an ex-traordinary sense of smell and orien-tation, and they also show social be-havior. The brain of a frog can bepositioned within the same dimension.The frog has a complex build withmany functions, it can swim and hasevolved complex behavior. A frogcan continuously target the said flyby means of his eyes while jumpingin three-dimensional space and andcatch it with its tongue with consid-erable probability.

5 · 107 neurons make a bat. The bat cannavigate in total darkness through aroom, exact up to several centime-ters, by only using their sense of hear-ing. It uses acoustic signals to localizeself-camouflaging insects (e.g. somemoths have a certain wing structurethat reflects less sound waves and theecho will be small) and also eats itsprey while flying.

1.6 · 108 neurons are required by thebrain of a dog, companion of man forages. Now take a look at another pop-ular companion of man:

3 · 108 neurons can be found in a cat,which is about twice as much as ina dog. We know that cats are veryelegant, patient carnivores that canshow a variety of behaviors. By theway, an octopus can be positionedwithin the same magnitude. Onlyvery few people know that, for exam-ple, in labyrinth orientation the octo-pus is vastly superior to the rat.

For 6 · 109 neurons you already get achimpanzee, one of the animals beingvery similar to the human.

1011 neurons make a human. Usually,the human has considerable cognitivecapabilities, is able to speak, to ab-stract, to remember and to use toolsas well as the knowledge of other hu-mans to develop advanced technolo-gies and manifold social structures.

With 2 · 1011 neurons there are nervoussystems having more neurons thanthe human nervous system. Here weshould mention elephants and certainwhale species.

Our state-of-the-art computers are notable to keep up with the aforementionedprocessing power of a fly. Recent researchresults suggest that the processes in ner-vous systems might be vastly more pow-erful than people thought until not longago: Michaeva et al. describe a separate,

D. Kriesel – A Brief Introduction to Neural Networks (ZETA2-EN) 29

Page 18: Chapter 2 Biological neural networks - WordPress.com · Chapter 2 Biological neural networks ... brainstem or the (truncus cerebri)re-spectively is phylogenetically much older. Roughly

Chapter 2 Biological neural networks dkriesel.com

synapse-integrated information way of in-formation processing [MBW+10]. Poster-ity will show if they are right.

2.5 Transition to technicalneurons: neural networksare a caricature of biology

How do we change from biological neuralnetworks to the technical ones? Throughradical simplification. I want to brieflysummarize the conclusions relevant for thetechnical part:

We have learned that the biological neu-rons are linked to each other in a weightedway and when stimulated they electricallytransmit their signal via the axon. Fromthe axon they are not directly transferredto the succeeding neurons, but they firsthave to cross the synaptic cleft where thesignal is changed again by variable chem-ical processes. In the receiving neuronthe various inputs that have been post-processed in the synaptic cleft are summa-rized or accumulated to one single pulse.Depending on how the neuron is stimu-lated by the cumulated input, the neuronitself emits a pulse or not – thus, the out-put is non-linear and not proportional tothe cumulated input. Our brief summarycorresponds exactly with the few elementsof biological neural networks we want totake over into the technical approxima-tion:

Vectorial input: The input of technicalneurons consists of many components,

therefore it is a vector. In nature aneuron receives pulses of 103 to 104

other neurons on average.

Scalar output: The output of a neuron isa scalar, which means that the neu-ron only consists of one component.Several scalar outputs in turn formthe vectorial input of another neuron.This particularly means that some-where in the neuron the various inputcomponents have to be summarized insuch a way that only one componentremains.

Synapses change input: In technical neu-ral networks the inputs are prepro-cessed, too. They are multiplied bya number (the weight) – they areweighted. The set of such weights rep-resents the information storage of aneural network – in both biologicaloriginal and technical adaptation.

Accumulating the inputs: In biology, theinputs are summarized to a pulse ac-cording to the chemical change, i.e.,they are accumulated – on the techni-cal side this is often realized by theweighted sum, which we will get toknow later on. This means that afteraccumulation we continue with onlyone value, a scalar, instead of a vec-tor.

Non-linear characteristic: The input ofour technical neurons is also not pro-portional to the output.

Adjustable weights: The weights weight-ing the inputs are variable, similar to

30 D. Kriesel – A Brief Introduction to Neural Networks (ZETA2-EN)

Page 19: Chapter 2 Biological neural networks - WordPress.com · Chapter 2 Biological neural networks ... brainstem or the (truncus cerebri)re-spectively is phylogenetically much older. Roughly

dkriesel.com 2.5 Technical neurons as caricature of biology

the chemical processes at the synap-tic cleft. This adds a great dynamicto the network because a large part ofthe "knowledge" of a neural network issaved in the weights and in the formand power of the chemical processesin a synaptic cleft.

So our current, only casually formulatedand very simple neuron model receives avectorial input

x̨,

with components xi. These are multipliedby the appropriate weights wi and accumu-lated: ÿ

i

wixi.

The aforementioned term is calledweighted sum. Then the nonlinearmapping f defines the scalar output y:

y = f

Aÿ

i

wixi

B

.

After this transition we now want to spec-ify more precisely our neuron model andadd some odds and ends. Afterwards wewill take a look at how the weights can beadjusted.

Exercises

Exercise 4. It is estimated that a hu-man brain consists of approx. 1011 nervecells, each of which has about 103 to 104

synapses. For this exercise we assume 103

synapses per neuron. Let us further as-sume that a single synapse could save 4

bits of information. Naïvely calculated:How much storage capacity does the brainhave? Note: The information which neu-ron is connected to which other neuron isalso important.

D. Kriesel – A Brief Introduction to Neural Networks (ZETA2-EN) 31