ECE 428: Information Theory

Spring 2004

3/30/04 | Introduction/history; basic information measures - entropy, relative entropy, mutual information. | [slides] |

4/1/04 | Convexity, Jensen's inequality and consequences, convexity properties of entropy and mutual information. | [Notes] |

4/6/04 | The Data Processing Inequality, Fano's Inequality, the Asymptotic Equipartition Property. | [Notes] |

4/8/04 | Typical sets, Shannon's Source Coding theorem and converse, Entropy rate of a stochastic process. | [Notes] |

4/13/04 | Entropy rate of a stochastic process, Shannon-McMillan-Breiman Theorem, Markov Chains. | [Notes] |

4/15/04 | Lossless source codes, uniquely decodable codes, instantaneous codes, Kraft inequality, optimal codes. | |

4/20/04 | Kraft inequality for uniquely decodable codes, Shannon codes, Huffman Codes. | |

4/22/04 | Optimality of Huffman Codes, universal source codes, Lempel-Ziv Compression | [Notes] |

4/27/04 | Introduction to channel coding, discrete-time/discrete alphabet channels, discrete memoryless channels, information capacity. | |

4/29/04 | Calculating capacity- symmetric channels, convex optimization; Joint typicality | |

5/4/04 | MID-TERM EXAM. | |

5/7/04 | Joint typicality; block codes, random coding argument. | |

5/11/04 | Proof of the direct part of channel coding theorem for DMC; expurgated codes; implementation issues. | |

5/13/04 | Strong Coding theorems, error exponents and reliability functions; Converse to the coding theorem. | [Notes] |

5/18/04 | Converse to the coding theorem for bit error probability, the joint source channel coding theorem, feedback capacity | |

5/20/04 | Feedback capacity, continuous-valued random variables and differential entropy. | |

5/25/04 | More on Differential entropy, Gaussian RV's, capacity of the discrete-time additive white Gaussian noise channel. | |

5/27/04 | Coding theorem and converse for additive white Gausian noise channel, High SNR and low SNR regimes, parallel Gaussian channels and water-filling. | |

6/1/04 | Continuous-time (waveform) channel models, sampling and equivalent discrete-time channels, colored Gaussian noise channels. | |

6/3/04 | Rate-Distortion theory, quantization, Lloyd-Max algorithm, vector quantizers, rate-distortion function, Gaussian sources |