Follow up on World Knowledge: Compression and Representation
It sounds like magic and hardly believable how potent Large Language Models (LLMs) are with their impressive and skillful expression of knowledge. The goal of this article is to demystify why LLMs are so capable.
There are different schools of what intelligence and conscience are, what reasoning means and how people think. Lots of papers have been written on this subject since the dawn of human civilization. We will focus here on a particular and, I think, quite plausible paradigm that LLMs and Deep Neural Networks (DNN) are based on.
When LLMs and DNNs are learning the world and navigating through the world knowledge in order to produce results, they do it through facts acquisition and drawing analogies.
Reasoning through analogy
When humans learn and reason they don’t rely on formal logic during processing new information. It is mostly memorization of new facts and associating them with prior relevant knowledge that we are dealing with. We don’t do exhaustive if-then-else analysis in our daily routine. Most of the time we bet on our intuition.
What is intuition after all? It is an extensive accumulation of prior experience with implicit iterative refinement of what matters and what does not. Intuition heavily relies on analogy with similar cases. It helps us infer reliable conclusions with confidence even in situations that we have never been in. All that without too much thinking, often instantaneously. We don’t subject every inference to first-order-logic rigor, far from that.
Brain and DNA in humans
Humans have DNA in their cells that has codified how the human body will evolve, the brain develop, and all kinds of predispositions and traits. All those are subject to environmental and social impact throughout human evolution.
The Brain on its part embodies our way of thinking, reasoning, adapting and communicating through complex structure of neurons and synapses. That is what we call intelligence.
When a child is born and a new life begins both the DNA and embryonic brain have imprinted or “front-loaded”, if I may, what the Creator or evolution (or both) have destined for a human to bootstrap from. Accordingly this “front-loadedness” helps human explore the wild world with confidence and self-enrich throughout one’s life long journey.
Memory section of the brain stores all the worthy experiences acquired throughout a human’s life. The human brain’s memory function is extremely powerful and efficient. Profound experiences and “useful” knowledge stay for a long time, whereas anything transitory and not really helpful in the long run gets discarded from the costly memory.
Similar concepts, different hardware
Let’s see now how DNN and LLM are organized and function on a conceptual level. The combination of LLM+DNN (embeddings and parameters + architecture) can be loosely considered as the counterparts of DNA+Brain. The former is imbedded in an electronic hardware, whereas the latter in a biological body. Surprise-surprise both systems share electrical means for communication between circuits/cells. (Sort of, in the human body it is actually electro-chemical communication). A combination of DNN(with parameters)+embeddings can be considered as a brain with DNA in a biological body: embeddings vs DNA, DNN vs Brain.
Point is, an intelligent system – biological or electronic – is front-loaded with potency to be intelligent and evolve thanks to DNA+Brain and embeddings+DNN mechanisms in respective cases.
Human and artificial intelligence: similar concepts, different hardware.
Keep reading. The next post will further elaborate about this analogy and continue building plausible intuition why LLMs are so effective in producing surprising and convincing outputs that rival and sometimes exceed human intelligence.
Leave a Reply