Characterizing Messages

Main Menu Previous Topic Next Topic

**Channel Capacity**

Note: in order to run the simulation referred to in this slide, go here, to the Java Applet version. You will be directed to download the latest version of the Java plug-in.

Channel capacity is a measure of how quickly information moves across a
channel. Information is related to how many possible input messages we have
to choose from. In the case of the simple binary channel, we have a choice of
two symbols in every interval. This means that we can choose 2^{n}
different messages of length n. Shannon calls this a capacity of one bit per time period.

Notice that transmitting two bits in this channel corresponds to a choice of four different messages in two time periods, three bits is a choice of eight messages in three time periods, and so on. Because the number of choices increases exponentially, we might come to the conclusion that capacity should depend on the log of the number of message choices. In fact, Shannon arrives at the following expression for channel capacity:

,

where N is the number of possible messages of length T. In our example,
N(T) is simply 2^{T}, so the term inside the limit is always 1. If the base of
the logarithm is taken to be two, the resulting capacity is said to be measured in *bits*. Thus the binary
channel has a capacity of 1 bit per interval, as our intuition dictates.

You might wonder if the limit is necessary in the above expression. This is because different symbols may take different times to transmit. Consider the Morse Code. A dot might be represented by a 1 followed by a 0, while a dash may be three 1's followed by a 0. The agents to the left will transmit a simple message using this system. Press "Go" repeatedly to make this happen.

Notice that transmitting a dash takes twice as long as transmitting a dot. In this case, N(T) / T is not constant in T, so we cannot discern the capacity from short sequences. There is only one possible sequence of length 2 and none of length 1. For any meaning, we must consider how many messages are possible as the sequences grow long. The above limit can be calculated using the method of differences to be 0.34 bits per period

Previous Slide Next Slide