Fundamentals of Neural Networks has been written for students and for . Don Fausett for introducing me to neural networks, and for his patience, en-. Author: Laurene V. Fausett Fundamentals of Neural Networks: Architectures, Algorithms and Applications · Read more · Principles of artificial neural networks. Fundamentals of Neural Networks by Laurene Fausett - Ebook download as PDF File .pdf), Text File .txt) or read book online.
|Language:||English, Spanish, Japanese|
|Genre:||Academic & Education|
|Distribution:||Free* [*Register to download]|
Fundamentals of neural networks. NEUROCOMPUTINC ELSEVIER Neurocomputing 10 () Book reviews Material to be included in this section can. Main text: Fundamentals of Neural. Networks: Architectures, Algorithms, and. Applications, Laurene Fausett, Prentice-. Hall, • Supplementary Material. systematic study of the artificial neural network. Four years later The interest in neural networks comes from the networks' ability to mimic Chapter 2 − Fundamentals of NN Fausett, L., Fundamentals of Neural Networks, Prentice- Hall.
Although our interest lies almost exclusively in the computational capabilities of neural networks. A generic biological neuron is illustrated in Figure 1. In fact. The action of the chemical transmitter modifies the incoming signal typically. The signals are electric im- pulses that are transmitted across a synaptic gap by means of a chemical process.
The many dendrites receive signals from other neurons. The ions most directly involved are potassium. A detailed consideration of these ideas for specific nets. Several key features of the processing elements of artificial neural net- works are suggested by the properties of biological neurons.. For some researchers. There is a close analogy between the structure of a biological neuron i.
The transmission of the signal from a particular neuron is accomplished by an action potential resulting from differential concentrations of ions on either side of the neuron's axon sheath the brain's "white matter".
A biological neuron has three types of components that are of particular interest in understanding an artificial neuron: This corresponds to looking at discrete time steps and summing all activity signals received or signals sent at a particular point in time. It is often supposed that a cell either fires or doesn't at any instant of time.
A synapse's strength may be modified by experience. Yet another important characteristic that artificial neural networks share with biological neural systems is fault tolerance. An example of this is our ability to recognize a person in a picture we have not seen before or to recognize a person after a long period of time. Under appropriate circumstances sufficient input. Short-term memory corresponds to the signals sent by the neurons. The processing element receives many signals.
Memory is distributed: Long-term memory resides in the neurons' synapses or weights. The output from a particular neuron may go to many other neurons the axon branches. Humans are born with as many as billion neurons.
Most of these are in the brain. Signals may be modified by a weight at the receiving synapse. Information processing is local although other means of transmission. In spite of our continuous loss of neurons. The processing element sums the weighted inputs. Other features of artificial neural networks that are suggested by biological neu- rons are: Neurotransmitters for synapses may be excitatory or inhibitory.
Biological neural systems are fault tolerant in two respects. Even in the case of wire-based telephone transmission. Section 6. Separating the action of a backpropagation net into smaller pieces to make it more local and therefore.
A brief sampling of some of the areas in which neural networks are currently being applied suggests the breadth of their appli- cability.
One of the first commercial applications was and still is to suppress noise on a telephone line. Even for uses of artificial neural networks that are not intended primarily to model biological neural systems. The switching involved in conventional echo suppression is very disruptive with path delays of this length.
The examples range from commercial successes to areas of active re- search that show promise for the future.
The adaptive noise cancellation idea is quite simple. One example is the use of a planar array of neurons. The study of neural networks is an extremely interdisciplinary field. The topological nature of these maps has com- putational advantages.
At the end of a long- distance line. See Widrow and Stearns. In a similar manner. The need for adaptive echo can- celers has become more pressing with the development of transcontinental sat- ellite links for long-distance telephone circuitsx'l'he two-way round-trip time delay for the radio transmission is on the order of half a second.
To make the problem more challenging. As an example of the application of neural networks to control problems. The second module is the controller. The emulator has several hidden units and is trained using backpropagation which is the subject of Chapter 6. After the emulator is trained. One specific area in which many neural network applications have been developed is the automatic recognition of handwritten characters digits or letters.
The neural net is able to learn how to steer the truck in order for the trailer to reach the dock. At each time step. The large. Information is available describing the position of the cab of the truck. The truck moves a fixed distance at each time step. The error is then determined and the weights on the controller are adjusted. The training process for the controller is similar to the recurrent backpropagation described in Chapter 7.
As with a driver. This process continues until either the trailer reaches the dock or the rig jackknifes. The first called the emulator learns to compute the new position of the truck. The neural net solution to this problem uses two modules. This module learns the "feel" of how a trailer truck responds to various steering signals.
General-purpose multilayer neural nets. The net performs surprisingly well. This net has several layers. It has been called the "Instant Physician" [Hecht- Nielsen. Even when an application is based on a standard training algorithm.
It is a good example. A traditional approach to the problem would typically involve constructing a set of rules for the standard pronunciation of various groups of letters.
The idea behind this application is to train an autoassociative memory neural network the "Brain-State-in-a-Box. In cases where there are ambiguities in the training data. An alternative approach to the problem of recognizing handwritten char- acters is the "neocognitron" described in Chapter 7. One of the most widely known examples of a neural network approach to.
In novel situations. When a par- ticular set of symptoms occurs frequently in the training set. After training. This back- propagation net has several hidden layers.
Lippmann summarizes the characteristics of many of these nets. The second stage of learning corresponds to the net recognizing the boundaries between words. In contrast to the need to construct rules and look-up tables for the exceptions.
The written input includes both the letter that is currently being spoken and three letters before and after it to provide a context. The result is a babbling sound. Additional symbols are used to indicate the end of a word or punc- tuation. NETtalk's only requirement is a set of examples of the written input.
One net that is of particular interest.. As the net groups similar inputs. It is interesting that there are several fairly distinct stages to the response of the net as training progresses. Because the correspondence between phonemes and written letters is very regular in Finnish for which the net was developed.
The net is trained using the 1. A number of useful systems now have a limited vocabulary or grammar or require retraining for different speakers. He calls his net a "phonetic type- writer. Several types of neural networks have been used for speech recognition. See Kohonen for a more extensive description. The net learns quite quickly to distinguish vowels from consonants.
After as few as to passes through the training data. After the speech input signals are mapped to the phoneme regions which has been done without telling the net what a phoneme is. The input to the net is based on short segments a few milliseconds long of the speech waveform. We mention only one of many examples here. Detailed discussions of these ideas for a number of specific nets are presented in the remaining chapters.
When disagreement did occur. A second neural net was trained to evaluate the risk of default on a loan. The building blocks of our examination here are the network architectures and the methods of setting the weights training. Using an independent measure of the quality of the mortgages certified. The basic idea behind the neural network approach to mortgage risk assessment is to use past experience to train the net to provide more consistent and reliable evaluation of mortgage applications.
Using data from several experienced mortgage evaluators. A total of 4. The purpose in each of these is to determine whether the applicant should be given a loan. The target output from the net is an "accept" or "reject.
The training input includes information on the applicant's years of employment. Although it may be thought that the rules which form the basis for mortgage underwriting are well understood. Let us now consider some of the fundamental features of how neural networks operate. Although delinquency can result from many causes that are not reflected in the information available on a loan application.
The decisions in the second kind of underwriting are more difficult. In addition. In both kinds of underwriting. In effect. The net shown in Figure 1. If any neuron in a layer for instance. In the typical single- layer net shown in Figure 1. Single-Layer Net A single-layer net has one layer of connection weights.
By contrast. The arrangement of neurons into layers and the connection patterns within and between layers is called the net architecture.
Key factors in determining the behavior of a neuron are its activation function and the pattern of weighted connections over which it sends and receives signals. To be more specific. Within each layer. Many neural nets have an input layer in which the activation of each unit is equal to an external input signal.
The single-layer and multilayer nets illustrated in Figures 1. For pattern classification. In determining the number of layers.
This view is motivated by the fact that the weights in a net contain extremely important information. The net illustrated in Figure 1. Neural nets are often classified as single layer or multilayer. The fully interconnected competitive net in Figure 1. S A multilayer neural net. The problems that require multilayer nets may still represent a classification or association of patterns.
Multilayer net A multilayer net is a net with one or more layers or levels of nodes the so- called hidden units between the input units and the output units. Note that for a single- layer net. Several examples of these nets are discussed in Chapters 4 and 5.
Multilayer nets can solve more complicated problems than can single-layer nets. These two examples illustrate the fact that the same type of net can be used for different problems. The characteristics of the problems for which a single-layer net is satisfactory are considered in Chapters 2 and 3.
Competitive layer A competitive layer forms a part of a large number of neural networks. For pattern association. An example of the architecture for a competitive. Pattern classification and pattern association may be considered special forms of the more general problem of mapping input vectors or patterns to the specified output vectors or patterns.
In the next chapter. Pattern association is another special form of a mapping problem. For convenience. Supervised training In perhaps the most typical neural net setting. For more difficult classification problems. The operation of a winner-take-all competition. Some of the simplest and historically earliest neural nets are designed to perform pattern classification.
I if it does not belong. A neural net that is trained to associate a set of input vectors with a corresponding. This process is known as supervised training. We summarize here the basic characteristics of supervised and unsupervised training and the types of problems for which each. The weights are then adjusted according to a learning algorithm.
These nets are trained using a supervised algorithm. The characteristics of a classifi- cation problem that determines whether a single-layer net is adequate are con- sidered in Chapter 2 also.
There is some ambiguity in the labeling of training methods as supervised or unsupervised. In this type of neural net. Many of the tasks that neural nets can be trained to perform fall into the areas of mapping. The single-layer nets in Chapter 2 pattern classification nets and Chapter 3 pattern association nets use supervised training the Hebb rule or the delta rule.
Associative memory neural nets.. Unsupervised learning is also used for other tasks. The net modifies the weights so that the most similar input vectors are assigned to the same output or cluster unit. Unsupervised training Self-organizing neural nets group similar input vectors together without the use of training data to specify what a typical member of each group looks like or to which group each vector belongs.
Multilayer neural nets can be trained to perform a nonlinear mapping from an n-dimensional space of input vectors n-tuples to an m-dimensional output space-i. Other forms of supervised learning are used for some of the nets in Chapter 4 learning vector quantization and counterpropagation and Chap- ter 7.
A sequence of input vectors is provided. Fixed weights are also used in contrast-enhancing nets see Section 4. Each learning algorithm will be described in detail. Self- organizing nets are described in Chapters 4 Kohonen self-organizing maps and Chapter 5 adaptive resonance theory.
Fixed-weight nets Still other types of neural nets can solve constrained optimization problems.. Such nets may work well fot problems that can cause difficulty for traditional tech- niques. The Boltzmann machine without learning and the continuous Hopfield net Chapter 7 can be used for constrained optimization problems.
If the desired output vector is the same as the input vector. The neural net will produce an exemplar representative vector for each cluster formed. Examples are included in Chapter 7. Backpropagation the generalized delta rule is used to train the multilayer nets in Chapter 6. When these nets are designed. The logistic function and the hyperbolic tangent functions are the most common. The binary step function is also known as the threshold function or Heaviside function.
Single-layer nets often use a step function to convert the net input. They are especially advantageous for use in neural nets trained by backpropagation. In most cases. The use of a threshold in this regard is discussed in Section 2. In order to achieve the advantages of multilayer nets. For the input units. The logistic function. This function is illustrated in Figure 1. As is shown in Section Figure 1. I X Figure 1.
The most com- mon range is from. Introduction Chap. To emphasize the range of the function. It is illustrated in Figure 1. A more extensive discussion of the choice of activation functions and different forms of sigmoid functions is given in Section 6. Figure Bipolar sigmoid. We have 1 -. A step activation function sets the activation of a neuron to 1 when- ever its net input is greater than the specified threshold value Oj. This is the jth column of the weight matrix.
Y j Activations of units Xi. Wij Weight on connection from unit Xi to unit Y j: Some authors use the opposite convention. A bias acts like a weight on a connection from a unit with a constant activation of 1 see Figure 1. W Weight matrix: For input units Xi.. The bias is treated exactly like any other weight. Learning rate: The learning rate is used to conrtol the amount of weight adjust- ment at each step of training.. The idea of a threshold such that if the net input to a neuron is greater than the threshold then the unit fires is one feature of a McCulloch-Pitts neuron that is used in many artificial neurons today.
Hebb learning Donald Hebb. These researchers recognized that combining many simple neurons into neural systems was the source of in- creased computational power. This time delay allows the net to model some physiological processes. The flow of information through the net assumes a unit time step for a signal to travel from one neuron to the next. They have. The his- tory of neural networks shows the interplay among biological experimentation.
The neurons can be arranged into a net to produce any output that can be represented as a combination of logic functions. His premise was that if two neurons were active simultaneously" then the strength of the connection between them should be increased. This section presents a very brief summary of the history of neural networks.
Refinements were subsequently made to this rather general statement to allow computer simulations [Rochester. The idea is closely related to the correlation matrix learning developed by Kohonen and Anderson among others. Results of a primarily biological nature are not included.
The weights on a McCulloch-Pitts neuron are set so that the neuron performs a particular simple logic function. Frank Rosenblatt The Widrow-Hoff learning rule for a single-layer network is a precursor of the backpropagation rule for multilayer nets.
This results in the smallest mean squared error. The early successes with perceptrons led to enthusiastic claims. Rosenblatt's work describes many types of perceptrons. The First Golden Age of Neural Nehvorks Although today neural networks are often viewed as an alternative to or com- plement of traditional computing.
The similarity of models developed in psychology by Rosenblatt to those developed in electrical engineering by Widrow and Hoff is evidence of the interdisciplinary nature of neural networks. Perceptrons Together with several other researchers [Block. Like the neurons developed by McCulloch and Pitts and by Hebb. The delta rule adjusts the weights to reduce the difference between the net input to the output unit and the desired output.
The perceptron rule adjusts the connection weights to a unit when- ever the response of the unit is incorrect. Perceptron learning can be proved to con- verge to the correct weights if there are weights that will solve the problem at hand i. The perceptron learning rule uses an iterative weight adjustment that is more powerful than the Hebb rule. The difference in learning rules. The most typical perceptron consisted of an input layer the retina connected by paths with fixed weights to associator neurons.
Johnson and Brown and Anderson and Rosenfeld discuss the interaction between von Neumann find early neural network researchers such as Warren McCulloch. The response indicates a classification of the input pattern.
His more recent work [Kohonen. Among the areas of application for these nets are medical diagnosis and learning multiplication tables. His work. Klimasauskas lists publica- tions by Grossberg from to He developed these ideas into his "Brain-State-in-a-Box" [Anderson.
Grossberg Stephen Grossberg. These nets have been applied to speech recognition for Finnish and Japanese words [Kohonen. Anderson James Anderson. The introductions to each are especially useful. Kohonen The early work of Teuvo Kohonen Anderson and Rosenfeld and Anderson.
Many of the cur- rent leaders in the field began to publish their work during the s. Parker's work came to the attention of the Parallel Distributed Processing Group led by psychologists David Rumelhart. Hopfield has developed a number of neural networks based on fixed weights and adaptive activations [Hopfield. One ex- ample of such a net. This method was also discovered independently by David Parker and by LeCun before it became widely known.
Adaptive resonance theory nets for binary input patterns ART! A method for propagating information about errors at the output units back to the hidden units had been discovered in the previous decade [Werbos.
Together with David Tank. Neocognitron Kunihiko Fukushima and his colleagues at NHK Laboratories in Tokyo have developed a series of specialized neural nets for character recognition.
This deficiency was corrected in the neocognitron [Fukushima. An earlier self-organizing network. These nets can serve as associative memory nets and can be used to solve con- straint satisfaction problems such as the "Traveling Salesman Problem.
It displays several important features found in many neural net- works.
San Diego. He is also cofounder of Synaptics. The activation of a McCulloch-Pitts neuron is binary.. Carver Mead. The founder of HNC. The requirements for McCulloch-Pitts neurons may be summarized as follows: Hardware implementation Another reason for renewed interest in neural networks in addition to solving the problem of how to train a multilayer net is improved computational capa- bilities.. DARPA is a valuable summary of the state of the art in artificial neural networks especially with regard to successful applications.
Optical neural nets [Farhat. Cooper is chairman of Nestor. Nobel laureate Leon Cooper. Hecht-Nielsen is "an industrialist.
McCulloch-Pitts neurons are connected by directed. That is. To quote from the preface to his book. These nets incorporate such classical ideas as simulated annealing and Bayesian decision theory. The threshold for unit Y is It takes one time step for the signals to pass from the X units to Y.. The simple example of a McCulloch-Pitts neuron shown in Figure 1. The use of discrete time steps enables a network of McCulloch-Pitts neurons to.
Each connection path is either excitatory. It takes one time step for a signal to pass over one connection link. Each neuron has a fixed threshold such that if the net input to the neuron is greater than the threshold. The connection from Xl to Y is excitatory. The threshold is set so that inhibition is absolute.
All excitatory connections into a particular neuron have the same weights. The McCulloch-Pitts Neuron 27 These excitatory connections have the same positive weight because they are going into the same unit. A connection path is excitatory if the weight on the path is positive. For convenience.. The activation function for unit Y is f y. Using these simple neurons as building blocks. Since analysis. Although all excitatory weights coming into any particular unit must be the same.
The binary form of the functions for AND. Each of these functions acts on two input values.. Simple networks of McCulloch-Pitts neurons. The activation of a neuron Xi at time t is determined by the activations.
The activation of unit Xi at time t is denoted Xi t. Logic functions will be used as simple examples for a number of neural nets. Y will fire if it receives k or more excitatory inputs and no inhibitory inputs.
In Section 1. OR The OR function gives the response "true" if either of the input values is "true". The threshold on unit f is 2. The threshold on unit fis 2. The response is "true" if the first input value. In other words. XI X2. The threshold for unit Y is 2. XOR can be expressed as Figure 1. Xio is "true" and the second input value. Neurons 2 1 Figure 1. Hot and cold Example 1. ORZ2' Units Z]. The model is designed to give only the first perception of heat or cold that is received by the perceptor units.
The example is an elaboration of one originally presented by McCulloch and Pitts . The use of discrete time steps enables the network of McCulloch- Pitts neurons shown in Figure 1. In the figure. The activation of unit Y2 at time t is Y2 t. To see that the net shown in Figure Now consider the neurons illustrated in Figure 1. The switching involved in conventional echo suppression is very disruptive with path delays of this length.
Even in the case of wire-based telephone transmission, the repeater amplifiers introduce echoes in the signal. The adaptive noise cancellation idea is quite simple. At the end of a long- distance line, the incoming signal is applied to both the telephone system com- ponent called the hybrid and the adaptive filter the AOALINE type of neural net. See Widrow and Stearns, , for a more detailed discussion. However, a driver with experience accomplishes the feat with remarkable ease.
Information is available describing the position of the cab of the truck, the position of the rear of the trailer, the fixed position of the loading dock, and the angles that the truck and the trailer make with the loading dock.
The neural net is able to learn how to steer the truck in order for the trailer to reach the dock, starting with the truck and trailer in any initial configuration that allows enough clearance for a solution to be possible. To make the problem more challenging, the truck is allowed only to back up.
The neural net solution to this problem uses two modules. The first called the emulator learns to compute the new position of the truck, given its current position and the steering angle. The truck moves a fixed distance at each time step.
This module learns the "feel" of how a trailer truck responds to various steering signals, in much the same way as a driver learns the behavior of such a rig.
The emulator has several hidden units and is trained using backpropagation which is the subject of Chapter 6. The second module is the controller. After the emulator is trained, the con- troller learns to give the correct series of steering signals to the truck so that the trailer arrives at the dock with its back parallel to the dock.
At each time step, the controller gives a steering signal and the emulator determines the new position of the truck and trailer. This process continues until either the trailer reaches the dock or the rig jackknifes. The error is then determined and the weights on the controller are adjusted. As with a driver, performance improves with practice, and the neural con- troller learns to provide a series of steering signals that direct the truck and trailer to the dock, regardless of the starting position as long as a solution is possible.
Initially, the truck may be facing toward the dock, may be facing away from the dock, or may be at any angle in between. Similarly, the angle between the truck and the trailer may have an initial value short of that in a jack-knife situation. The training process for the controller is similar to the recurrent backpropagation described in Chapter 7. Many interesting problems fall into the general area of pattern recognition. One specific area in which many neural network applications have been developed is the automatic recognition of handwritten characters digits or letters.
The large Sec. It is a good example, however, of the type of infor- mation processing that humans can perform relatively easily. General-purpose multilayer neural nets, such as the backpropagation net a multilayer net trained by backpropagation described in Chapter 6, have been used for recognizing handwritten zip codes [Le Cun et aI. Even when an application is based on a standard training algorithm, it is quite common to cus- tomize the architecture to improve the performance of the application.
This back- propagation net has several hidden layers, but the pattern of connections from one layer to the next is quite localized. An alternative approach to the problem of recognizing handwritten char- acters is the "neocognitron" described in Chapter 7.
This net has several layers, each with a highly structured pattern of connections from the previous layer and to the subsequent layer. However, its training is a layer-by-layer process, spe- cialized for just such an application. It has been called the "Instant Physician" [Hecht- Nielsen, ]. The idea behind this application is to train an autoassociative memory neural network the "Brain-State-in-a-Box," described in Section 3.
After training, the net can be presented with input consisting of a set of symptoms; it will then find the fun stored pattern that represents the "best" diagnosis and treatment. The net performs surprisingly well, given its simple structure. No notes for slide. United States Edition Pie 1. United States Edition Pie 2. Book details Author: Laurene V. Fausett Pages: Pearson Language: English ISBN Description this book Please continue to the next pagenone https: If you want to download this book, click link in the last page 5.
United States Edition Pie Click this link: You just clipped your first slide! Clipping is a handy way to collect important slides you want to go back to later. Now customize the name of a clipboard to store your clips. Visibility Others can see my Clipboard.