Triton 47 › AI & computation › The ChatGPT year
LLM – Large Language Models : What are they really?
Index note training = thinking, inference = lookup; path counting (256^8 ≈ 1.8e19); memory recall via stimulant
LLM learns the information in structural form in its network.
When LLM is being trained, its as if it is thinking,
but when we use LLM while inference, we are solely using it as a lookup on that trained (thought) state, and it provides the matching output (sequence) given the input.
Know the abstraction involved, you can associate an idea or item with another item that usually is number and computation (in world), and figure out back the idea or item in its original space by mapping it back. Numerics learn the structure of information and processing there of the abstracted ideas in its own space, which is another form of computation (in numbers), which is controlled by learned weights which (may) behave the same way as reasoning does (or can be made maybe).
Humans have grown the reasoning and knowledge base, and information that genes create on the mind, but conscious experi of us which can be (or is) all abundant like electromagnetic fields and gravity and matter in this universe.
A feed forward network with 1-3-1 neurons can encode 3 active paths, while 3-3-1 can encode 9 active paths, while 3-3-3 neurons can encode 27 active paths. A feed forward network with 8 layers each 256 neurons can encode 1.8446744e+19 paths, i.e. that many different possible combinations or variants of paths (a series of active neurons), hence those many concepts at maximum. While brain has billions of neurons, although a fraction of it are learned, and those learned are triggered by the triggering path, which activate the volume of cells representing that knowledge. e.g. try seeing the information to recall the memory, however, certain paths may get blocked over time which are rendered inaccessible and hence experience of mind may change. To activate those pathways, similar stimulant which was around at the time of learning can help trigger the pathway, hence recalling the memories. How much we learn and experience, and miss it in our brains!, and in this universe, a specific set of stimulant activations in the brain is associated with a unique conscious experience, at least for neuron structures inhabiting ionic flow which is charged mass flow at speed in a complexity, which is capable of sensations of senses, e.g. vision, sound, etc.
The more you think on something, the more resources in the brain are provided to that thing and more neuron use would be for that thing. Brain has limitations of size and number and its structural complexity. The mathematical construct of space of all experiences is ginormous.
Original: https://triton7777.wordpress.com/2023/12/26/llm-large-language-models-what-are-they-really/