Skip to content

Information Theory

Jump to bottom
Miguel Amezola edited this page Sep 21, 2016 · 15 revisions

Symbols are signals. Communications as selection.

Information can be measured and compared using a measurement called entropy. The bit is a measure of surprise?

The message space is the set of all possible messages. Polybius square was a 5 by 5 grid that could represent 25 distinct messages. Sushruta Samhita: Given six different spices, how many possible different tastes can you make? Given n yes or no questions, there are 2^n possible answer sequences. Example: Lord George Murray's Shutter Telegraph.

Signal vs Noise

Discrete source.

Capacity is the rate of sending information.

References

  1. C. E. Shannon: A mathematical theory of communication. Bell System Technical Journal, vol. 27, pp. 379–423 and 623–656, July and October 1948

  2. "What Is FASTA Format?" FASTA Format. Zhang Lab, University of Michigan, n.d. Web. 24 Sept. 2016.

  3. Rost, B.; J.Liu; R.Nair; K.O. Wrzeszczynski; Y. Ofran (2003). "Automatic prediction of protein function". Cellular and Molecular Life Sciences. 60 (12): 2637–2650. doi:10.1007/s00018-003-3114-8. PMID 14685688.

  4. Alpaydin, Ethem. "Introduction to Machine Learning." MIT Press, 04 Dec. 2009. Web. 24 Sept. 2016.

  5. Nielsen, Michael. "Neural Networks and Deep Learning." Neural Networks and Deep Learning. N.p., June 2016. Web. 24 Sept. 2016.

  6. "Implementing a Neural Network from Scratch in Python – An Introduction." WildML. N.p., 10 Jan. 2016. Web. 24 Sept. 2016.

  7. "Recurrent Neural Networks Tutorial, Part 1 – Introduction to RNNs." WildML. N.p., 08 July 2016. Web. 24 Sept. 2016.

  8. Singh, Jagjit. Great Ideas in Information Theory, Language and Cybernetics. New York: Dover Publications, 1966. Print.

  9. "Structural Bioinformatics Guide: Protein Structure, Protein Sequence Analysis and Alignment, Homology Modeling." Protein Structure and Structural Bioinformatics. N.p., n.d. Web. 25 Sept. 2016.

  10. Kurzynski, M.; Chelminiak, P. Stochastic Dynamics of Proteins and the Action of Biological Molecular Machines. Entropy 2014, 16, 1969-1982.

  11. Strait, B J, and T G Dewey. “The Shannon Information Entropy of Protein Sequences.” Biophysical Journal 71.1 (1996): 148–155. Print.

  12. Heunen, Christiaan Johan Marie., Mehrnoosh Sadrzadeh, and Edward Grefenstette. Quantum Physics and Linguistics: A Compositional, Diagrammatic Discourse. Oxford, United Kingdom: Oxford UP, 2013. Print.

Clone this wiki locally