Information Theory Assignment

The concept of entropy can be applied to any language by assuming that the alphabetical characters are generated randomly by a data source. To better understand this concept, you need to do the following for both English and Arabic languages:

Part 1:

1. Copy form the internet an article of your choice, the article should be at least 1000 words, paste the article in a text file.

2. Convert all capital letters (for the English) to small letters and remove all punctuation marks from the article. Just leave the spaces.

3. Write a Matlab code that generates the probability distribution function (PDF) of your article.

4. Write a Matlab code that computes the entropy of the text.

5. Compare the results of both languages with each other.

Part 2:

During your studies of the Information Theory, you have seen that the entropy of a binary source follows the horse-shoe function given that we know the probability (p, 1 - p) of each event accurately. In this part you are required to:

1. Generate the horse-shoe function using Matlab assuming that p is known perfectly.

2. Generate the horse-shoe function using Matlab by generating N samples of a binary random variable. The function should be generated using four different values of N (10, 100. 1,000, 10,000).

3. Compare the results for the four cases (N = [10, 100. 1,000, 10,000]) with the ideal one by plotting them on the same figure.

4. Compute and draw the KL distance between the ideal case (p) and the other four cases where p is obtained using simulation. What is your conclusion? Show your solution in details.