Interval Coding

1. For the ensemble X with alphabet A = { 1, 2, 3} and probabilities p = ( 1=4, 1=3, 5=12 )

(a) Evaluate the cumulative distribution function F(a) = P(x ≤ a) for a ∈ A and give the values of F(1), F(2), and F(3)

(b) Evaluate the binary intervals for [F(0), F(1))₂, [F(1), F(2))₂, and [F(2), F(3))₂ (where F(0) = 0). Use the repeating notation (e.g., 0:101) where necessary.

(c) Construct the Shannon-Fano-Elias code for X, code the input sequence x = 3 3 1 1 2, and decode the sequence 011001110.

(d) Evaluate the expected length L(C,X) for the code just constructed.

(e) Is the code you just constructed an optimal code in terms of expected length? Explain why or why not?

Arithmetic Coding

1. Consider the Dirichlet multinomial model with m = (3, 3, 3) for the alphabet A = {a, b, c}. Evaluate the following probabilities under this model:

(a) P(x = b)
(b) P(x = a|baa)
(c) P(x = b|ac)
(d) P(x = c|s) where s is a sequence consisting of 100 as and 200 bs and no cs.

2. Using an arithmetic coder with the above Dirichlet multinomial model:

(a) Code the sequence cba.
(b) Decode the rst three symbols of the input sequence 11111 : : : of all 1s. Be sure to show the intervals and probabilities at each step.

3. Without doing a new coding, describe how the arithmetic code for cba would be dierent from the previous one if a Dirichlet multinomial model with m = (1, 100, 1) was used. Explain what sort of input sequences would be given the longest arithmetic codes under this new model?

Lempel-Ziv Coding

1. Use the LZ78 algorithm to code the sequence aaabababaaaa. Show the steps you used to build the code.

2. What kind of sequences from the alphabet A = {a, b}:
(a) Result in the longest possible LZ77 codes, assuming a window size of W = 2?
Give an example.
(b) Result in the longest possible LZ78 codes? Give an example and show the resulting tree.

Noisy-Channel Coding

1. Consider a channel with inputs X = fa, b, cg, outputs Y = fa, cg, and transition matrix

(a) Assuming a uniform distribution over input symbols, what is the mutual information I(X, Y ) between the input and output of the channel? What is the average probability of error?

(b) Assuming a distribution p = (0:5, 0, 0:5) over input symbols, what is the mutual information and average probability of error?

(c) Using the previous answer, give a non-zero lower bound for the capacity of this channel?

(d) Design a simple code for this channel that achieves a maximal probability of error less than 0:05.

2. Let Q be a channel with four inputs and outputs, each expressed as a 2-bit symbol.

That is, X = Y = { 00, 01, 10, 11}. Each time this channel is used, exactly one of the two input bits is flipped and the other are transmitted unchanged. Both bits are equally likely to be the one that is
ipped.

(a) Write out the transition matrix for this channel

(b) Evaluate the capacity of this channel.

(c) Design a block code with zero error for this channel with the highest possible rate.