Another research direction explores how modern AI can inspire new extensions of classical information theory.
Traditional information theory primarily focuses on the acquisition, transmission, and storage of data, emphasizing statistical structure rather than semantic meaning. However, recent advances in artificial intelligence—particularly embedding models and large language models—provide new ways to represent and manipulate semantic information.
Embedding methods operationalize the linguistic insight so that the meaning of a word can be characterized by the contexts in which it appears. These developments suggest a natural path toward extending classical information theory beyond purely statistical representations.
Our research investigates the development of distributional semantic information theory, in which semantic meaning is represented through contextual probability distributions and learned embeddings.
Within this framework:
-
semantic structures emerge from high-order statistical dependencies,
-
embeddings provide numerical representations of meaning, and
-
learning systems model complex semantic information sources.
Large language models can therefore be understood as components of an extended information-theoretic framework: they can be viewed as extremely high-order conditional models of sequences that, for the first time, provide a concrete and scalable realization of the semantic source models that classical information theory lacked. This connection suggests that the tools and questions of information theory are far from exhausted — a new chapter may be opening.