preview

Information Theory : The Backbone Of Modern Communication

Better Essays

Research Question How can information theory be utilized to improve efficiency of communication and data storage in modern computing networks?
Literature Review Information theory: the backbone of modern communication. Before Claude Shannon published his groundbreaking paper A Mathematical Theory of Communication, information as a quantity was poorly understood. A Mathematical Theory of Communciation put forth the foundation of information theory, the mathematical study of information and its properties (Cruise, 2014e). Through information theory, scientists and mathematicians have created quantative measures of structural complexity, compressability, communication complexity, and information content, providing a scholarly way …show more content…

Information Theory Entropy. In order to discuss information theory in relation to computer science, one must first understand the tenants of the theory. The primary keystone of information theory sits in Shannon 's definition of the amount of information a source generates, measured by Shannon 's entropy. Shannon’s entropy also defines the limit of compression of a message from a particular source, as data cannot be compressed beyond its information content without loss. As stated above, the entropy of an information source is H(x) = , or the negative sum across the entire message space of the probability of the ith symbol 's occurrence times the base 2 log of the probability of the ith symbol 's occurrence. Consider an information source that is randomly generating lowercase letters from the English alphabet. Since each symbol has an equal probability of (1/26), H(x) can simplify to , and since 26 and 1/26 cancel, the entropy of the source, or the amount of information it provides per symbol, evaluates to log2 1/26, or ~4.7 bits per symbol. Now imagine our source only transmits letters a-e, with probability increasing across the list, from 1/7 to ⅓. The entropy of the source evaluates to {-(1/7 log sub 2 1/7 + ⅙ log sub 2 ⅙ +

Get Access