3/30/04 |
Introduction/history; basic information measures - entropy,
relative entropy, mutual information. | [slides] |
4/1/04 | Convexity, Jensen's inequality and consequences,
convexity properties of entropy and mutual information.
| [Notes] |
4/6/04 | The Data Processing Inequality, Fano's
Inequality, the Asymptotic Equipartition Property. |
[Notes] |
4/8/04 | Typical sets, Shannon's Source Coding theorem and
converse, Entropy rate of a stochastic process. |
[Notes] |
4/13/04 | Entropy rate of a stochastic process,
Shannon-McMillan-Breiman Theorem,
Markov Chains. |
[Notes] |
4/15/04 | Lossless source codes, uniquely decodable
codes, instantaneous codes, Kraft inequality, optimal codes. |
|
4/20/04 | Kraft inequality for uniquely decodable
codes, Shannon codes, Huffman Codes. |
|
4/22/04 | Optimality of Huffman Codes, universal
source codes, Lempel-Ziv Compression |
[Notes] |
4/27/04 | Introduction to channel coding,
discrete-time/discrete alphabet channels,
discrete memoryless channels, information capacity. |
|
4/29/04 | Calculating capacity- symmetric channels,
convex optimization; Joint typicality |
|
5/4/04 | MID-TERM EXAM. |
|
5/7/04 | Joint typicality; block codes, random coding argument. |
|
5/11/04 | Proof of the direct part of channel coding
theorem for DMC; expurgated codes; implementation issues. |
|
5/13/04 | Strong Coding theorems, error exponents and
reliability functions; Converse to the coding theorem. |
[Notes] |
5/18/04 | Converse to the coding theorem for bit
error probability, the joint source channel coding
theorem, feedback capacity |
|
5/20/04 | Feedback capacity, continuous-valued
random variables and differential entropy. |
|
5/25/04 | More on Differential entropy, Gaussian
RV's, capacity of the
discrete-time additive white Gaussian noise channel. |
|
5/27/04 | Coding theorem and converse for additive
white Gausian noise channel, High SNR and low SNR regimes,
parallel Gaussian channels and water-filling.
|
|
6/1/04 | Continuous-time (waveform) channel models,
sampling and equivalent discrete-time channels, colored
Gaussian noise channels. |
|
6/3/04 | Rate-Distortion theory, quantization,
Lloyd-Max algorithm, vector quantizers,
rate-distortion function, Gaussian sources |
|