Analytic Information Theory
From Compression to Learning

Explores problems of information and learning theory, using tools from analytic combinatorics to analyze precise behavior of source codes.

Michael Drmota (Author), Wojciech Szpankowski (Author)

9781108474443, Cambridge University Press

Hardback, published 7 September 2023

400 pages
28 x 19 x 2.6 cm, 0.793 kg

'Drmota and Szpankowski, leading experts in the mathematical analysis of discrete structures, present here a compelling treatment unifying modern and classical results in information theory and analytic combinatorics. This book is certain to be a standard reference for years to come.' Robert Sedgewick, Princeton University

Through information theory, problems of communication and compression can be precisely modeled, formulated, and analyzed, and this information can be transformed by means of algorithms. Also, learning can be viewed as compression with side information. Aimed at students and researchers, this book addresses data compression and redundancy within existing methods and central topics in theoretical data compression, demonstrating how to use tools from analytic combinatorics to discover and analyze precise behavior of source codes. It shows that to present better learnable or extractable information in its shortest description, one must understand what the information is, and then algorithmically extract it in its most compact form via an efficient compression algorithm. Part I covers fixed-to-variable codes such as Shannon and Huffman codes, variable-to-fixed codes such as Tunstall and Khodak codes, and variable-to-variable Khodak codes for known sources. Part II discusses universal source coding for memoryless, Markov, and renewal sources.

Part I. Known Sources: 1. Preliminaries
2. Shannon and Huffman FV codes
3. Tunstall and Khodak VF codes
4. Divide-and-conquer VF codes
5. Khodak VV codes
6. Non-prefix one-to-one codes
7. Advanced data structures: tree compression
8. Graph and structure compression
Part II. Universal Codes: 9. Minimax redundancy and regret
10. Redundancy of universal memoryless sources
11. Markov types and redundancy for Markov sources
12. Non-Markovian sources: redundancy of renewal processes
A. Probability
B. Generating functions
C. Complex asymptotics
D. Mellin transform and Tauberian theorems
E. Exponential sums and uniform distribution mod 1
F. Diophantine approximation
References
Index.

Subject Areas: Mathematical theory of computation [UYA]