- **Source Coding** - Assume a [[Set|set]] of possible messages $\{1,..,M\}$ and we want to represent each possible message by a string of $n$ symbols from our code alphabet $\mathcal{X}$ (e.g. $0$ and $1$). The rate of an $(M,n)$ code is $R = \frac{\log(M)}{n},$where the numerator is the number of symbols from the source alphabet needed to encode the message. The rate therefore is the average number of bits (in the original message) that each symbol of the code is responsible for representing. - Lossless data compression possible, if $R \geq H(X)$ - **Channel Coding - Information Rate** - Proportion of data stream that is non-redundant$R = \frac{\textrm{length of information message}}{\textrm{length \, of \, codeword}}$ - Transmission rate over a [[Discrete Memoryless Channel|channel]] in $[\textrm{information bits per channel use}]$ or bits per symbol. - Average [[Shannon Entropy|entropy]] per symbol - The channel encoder maps one of $2^{nR}$ possible codewords uniquely to a codeword of length $n$.