ECE 428: Information Theory
Spring 2004


Lecture Topics/Handouts



3/30/04 Introduction/history; basic information measures - entropy, relative entropy, mutual information. [slides]
4/1/04 Convexity, Jensen's inequality and consequences, convexity properties of entropy and mutual information. [Notes]
4/6/04 The Data Processing Inequality, Fano's Inequality, the Asymptotic Equipartition Property. [Notes]
4/8/04 Typical sets, Shannon's Source Coding theorem and converse, Entropy rate of a stochastic process. [Notes]
4/13/04 Entropy rate of a stochastic process, Shannon-McMillan-Breiman Theorem, Markov Chains. [Notes]
4/15/04 Lossless source codes, uniquely decodable codes, instantaneous codes, Kraft inequality, optimal codes.
4/20/04 Kraft inequality for uniquely decodable codes, Shannon codes, Huffman Codes.
4/22/04 Optimality of Huffman Codes, universal source codes, Lempel-Ziv Compression [Notes]
4/27/04 Introduction to channel coding, discrete-time/discrete alphabet channels, discrete memoryless channels, information capacity.
4/29/04 Calculating capacity- symmetric channels, convex optimization; Joint typicality
5/4/04 MID-TERM EXAM.
5/7/04 Joint typicality; block codes, random coding argument.
5/11/04 Proof of the direct part of channel coding theorem for DMC; expurgated codes; implementation issues.
5/13/04 Strong Coding theorems, error exponents and reliability functions; Converse to the coding theorem. [Notes]
5/18/04 Converse to the coding theorem for bit error probability, the joint source channel coding theorem, feedback capacity
5/20/04 Feedback capacity, continuous-valued random variables and differential entropy.
5/25/04 More on Differential entropy, Gaussian RV's, capacity of the discrete-time additive white Gaussian noise channel.
5/27/04 Coding theorem and converse for additive white Gausian noise channel, High SNR and low SNR regimes, parallel Gaussian channels and water-filling.
6/1/04 Continuous-time (waveform) channel models, sampling and equivalent discrete-time channels, colored Gaussian noise channels.
6/3/04 Rate-Distortion theory, quantization, Lloyd-Max algorithm, vector quantizers, rate-distortion function, Gaussian sources