Signal Processing and Communications Laboratory

Department of Engineering

3F1 - Signals and Systems

Michaelmas Term 2013

Jossy Sayir

Information Theory Slides

The slides I used for the lecture are available for download:

  • Lecture_1 22 November 2013: Course organisation, historical background, Shannon's entropy
  • Lecture_2 27 November 2013: Prefix-free codes, converse source coding theorem, Shannon-Fano coding, Huffman coding
  • Lecture_3 29 November 2013: Coding for memoryless sources and for sources with memory, Shannon's twin experiment.
  • Lecture_4 4 December 2013: Channel capacity, coding theorem, continuous information theory and the Gaussian channel.

Note that some proofs are not included in the notes as these were done on the black board. The proofs of all statements in this part of the course are easy and should be approachable as an exercise for undergraduate students. Please do not hesitate to contact me if you are having any difficulty with a proof. In some cases I have added the proofs to the slides on this page after I found that some students were struggling with them, so please be sure to check the online slides as these may differ in places from the ones I handed out during the lecture.

Course Handouts

The handouts are available for download.

Example paper

The example paper and the crib are available for download. Please use the crib reasonably: attempt solving the problems on your own before you look at the crib, as you will not benefit as much from the examples otherwise. For the same reason, do not pass on the crib to next year's students as you would be doing them a disservice.