Deep Learning - Exercise 06

Learning goal:
Experiment with different network topologies and transfer functions for a MLP

Experiment with MLP network topologies

Goto this excellent implementation of a MLP which runs in your browser!

Now step through the sequence of the following experiments:


I. Experiments with circle dataset

I.1 tanh

  • Network topology: 2-2-1
    Using a MLP with inputs X1,X2 only and only one hidden layer with 2 neurons (network topology: 2-2-1) with tanh as transfer function: can it learn to classify the circle dataset (default dataset) correctly (near to 100%)?
  • Network topology: 2-3-1
    Now add a third neuron in the hidden layer and see whether it can learn to classify the circle dataset.
  • Network topology: 2-2-2-1
    Instead of adding a third neuron in the hidden layer, repeat the experiment with two hidden layers two neurons each.
  • Network topology: 2-2-2-2-1
    Also try three hidden layers with 2 neurons each. What is special for this case? (keyword: oscillations)

I.2 linear transfer function

Network topology: 2-X-Y-Z-1
Try to find a MLP with a network topology of your choice that uses only the linear transfer function. Could you find one?

I.3 rectified linear unit

  • Network topology: 2-2-1
    Instead of using tanh or the linear transfer function, switch now to "ReLU" and try to learn the circle dataset classification task with a 2-2-1 network. Does it work?
  • Network topology: 2-3-1
    Seems we need one neuron more here as well as in the case of tanh, or? So add another neuron and repeat the experiment. Compare the form of the "borders" that the MLP classifier learns when using ReLU vs. using tanh as transfer function.



II. Experiments with Gaussian dataset

First think about whether a MLP with no hidden layers (i.e., a Perceptron) that uses linear transfer functions can learn to classify this dataset.

Then try it out!

Does the classifiation rate improve when adding neurons or hidden layers?


III. Experiments with eXclusive OR (XOR) dataset

III.1 linear transfer function

Try to find a MLP with a topology of your choice that can correctly classify this dataset.

III.2 tanh

Network topology: 2-N-1

Try out tanh as transfer function and this topology with N=1,2,3,4,5,6 and observe how the classification error decreases in each new experiment as you add another neuron to the hidden layer.

III.3 ReLU

Network topology: 2-N-1

Try out ReLU as transfer function and this topology with N=1,2,3,4,5,6 and observe how the classificaition error decreases in each new experiment as you add another neuron to the hidden layer.

Note: test topology 2-3-1 with different new initialized weights! Does Backpropagation always find the same solution?


IV. Experiments with spiral dataset

IV.1 linear transfer function

Try to find any MLP with a topology of your choice than can classify this dataset.

IV.2 tanh

First try out one hidden layer and increase the number of neurons in this layer. Then add more hidden layers with some neurons in it and observe what happens.

IV.3 ReLU

Conduct the same experiments with ReLU as transfer function.

Can you reach a solution similar to the one depecited below using just X1 and X2 as input? (in my case it was a 2-5-4-1 MLP)

Also try out a 2-8-8-8-8-8-8-1 MLP with ReLU and all features as input. Here is a good solution I could observe for the spiral dataset: