Assignment 3.b: Experiment with Neural Network

Background:

In this assignment, you will experiment with neural network for solving different types of practical problems. You will study how to use neural network for learning Boolean functions, real-valued functions, and classification tasks. First, you will design a 2-layer neural network. In the hidden layer, there will be m number of nodes. In the output layer, use only 1 node. Write the algorithm in your preferred language. For learning the neural network, you will use the multilayer version of Gradient Descent learning algorithm which is called Back-propagation algorithm. However, rather than using the normal gradient descent version, you will use the stochastic gradient descent algorithm. The pseudo-code of this algorithm will be like:

While  not converged to minimum

{

                For each sample s, do the following

Step 1: Find the outputs for each network nodes. First find outputs for hidden layer nodes. Then using the outputs of hidden layer nodes, find outputs for the output layer node.

Step 2: Find the weight update vectors. First find the update vectors for the output layer node. Then find update vectors for the hidden layer nodes. Note that, in this step, find the update vectors and save them. Do not apply the updates in this step.

Step 3: Update the weights. Update all weights of output layer nodes and hidden layer  nodes with the update vector found in Step 2 above.

                end

}

Problems to solve:

In this assignment you will study and solve the following three types of problems:

1.       Learning Boolean functions using neural network:

You will learn the Boolean functions using neural network. In this problem, you will use sigmoid activation function for hidden layer nodes and linear/sigmoid activation function for output layer node. Use the data sets provided to train your neural network.

2.       Learning real values function using neural network:

You will learn the Boolean functions using neural network. In this problem, you will use sigmoid activation function for hidden layer nodes and linear activation function for output layer node. Use the data set provided to train your neural network.

3.       Learning classification task using neural network:

You will learn the Boolean functions using neural network. In this problem, you will use sigmoid activation function for hidden layer nodes and linear/sigmoid activation function for output layer node. Use the data set provided to train your neural network.

When to stop the Learning?

You can stop your learning after sufficient number of iterations. The number of iterations that will be sufficient is not fixed and it will depend on your learning parameter eta and problem complexity, number of nodes in hidden layer etc. Hence, you have to experiment with different value of iterations (such as 100, 1000, 10000). You can also stop your learning when you will find that the value of error function begins to increase rather than decrease. At each step of the algorithm, it is expected that, value of the error function will decrease by some amount. Whenever you find that, value of error function has been increased from the previous iteration, and then you can stop learning.

General Instructions:

1.       Experiment with varying hidden layer nodes, m (start using m=3, then test with m=5, 10, 20, 50 nodes).

2.       Experiment with learning rate, eta (start using eta=0.1, then test with eta=0.3, 0.5, 0.9).

3.       Experiment with iterations, (use 100, 1000, 10000) iterations.

4.       For each of the data sets, report the following:

a.       Report the value of error for each data set, m, eta, iterations.

b.       Best value of m, eta, iterations for each data set which gives you lowest error.