Maps of Meaning
“How do I reply to this?” - is not only a question that haunts your mind when you’re talking to your crush, but one that also haunts the mind of the AI researcher. It may also be said that finding the answer to this question is harder for the AI researcher than you. But the AI researcher has the power of data with her. Imagine being able to find the statistically most likely successful response to that particular text in history, because let’s be honest, you and them are not the first.
And so, the researcher starts. She first makes a model that gives you the statistical best fit for the next phrase, tries out some models that give her the next statistically most probable word but only gets as far as impersonating autocorrect. She then learns about deep learning and builds complex models that learn from his messages, but ends up in loops of regurgitating messages. We follow her thoughts as she finds her way out of the friendzone, and uncover the structure of Large Language Models like ChatGPT in the process.
But that is far from a trivial task, and the researcher thinks about it in a very particular way. The next section lays out some prerequisite terminology for the uninitiated.
A few words ..
A Vector, in the context of machine learning, is a numerical encoding of some entity like words in a fixed-dimensional vector space of word encodings. These numerical encodings or embeddings attempt to quantify the relevant properties of the original entity. In the case of a word, these properties could be its abstract meaning, context, and connotation. This makes it a high-info feature representation of the word. Vectors are useful because they make abstract things computable. Thus, we could measure similarity through dot products, interpolate data points (like king - man + woman ≈ queen), and in the context of word embeddings for LLMs, enable the model to reason geometrically – turning meaning itself into a shape in space.
A Vector Space, although mentioned earlier, is a set of vectors complete with the ability to follow key arithmetic rules(or axioms) that define a vector space. These rules ensure that any arithmetic operations you perform on the vectors do not break the math and are consistent, with logical rules like the sum of two vectors from a particular vector space also being inside that vector space. Thus, these ground rules essentially enable you to reliably use operations like addition and scalar multiplication and all the more complex theorems that rely on them. For word embeddings, their vector space could be imagined to be an abstract n-dimensional geometrical space where the words love, affection, warmth and adoration are close together, but far from the word engineering for example.

An Embedding, is a now-ubiquitous technique in the researcher's arsenal, to convert words into mathematical objects. Consider making a huge spreadsheet, with each column representing a quality (such as gender, location, usefulness in writing this article, or something much harder to put in words). Now, you give each number its own row, figure out a rating system to score that word on each metric, and voilá! You have an information-rich numerical encoding of this word. In the context of Large Language Models (and this blog), these encodings are called dense embeddings, and represented by vectors.

Some smart people figured out about a decade back how to go from Word-2-Vec(tor), something you can try out yourself at
Using this technique, you can embed these words in a vector space, an n-dimensional coordinate system (n>>3) where n is the number of columns in our spreadsheet, i.e. the number of qualities we are judging our vector on, i.e. the number of axes our vector space has. This geometrical space has two important properties:

Similar words, like "love", "affection", and "warmth" are placed close together, and far away from dissimilar words like "engineering"
You can do arithmetic on these words, for example, you can do King - man + woman, then search for the closest vector to the result. Can you guess it? Yes, It's "Queen". And if you're wondering why it's approximately Queen and not exactly, I'd wager the Queens were the most impactful rock band of their era.

One last definition: A linear transformation is any mathematical operation that reshapes our vector space while preserving its fundamental geometry (ie: above two properties) . It essentially changes the axes through which the vector is represented, either creating new metrics altogether (such as introducing a metric for “how much aura something has” which wasn’t there before, while preserving the total number of dimensions too) or making them more “intense” (For example if you dial up the intensity of military relations axis, India’s vector might be closer to Russia’s vector than America’s. But if you boost up the cultural influence axis, said situation may flip)
.
Thus, we can use linear transformations to refine the meaning of a word by changing the lens through which information is viewed. You don’t know it yet, but this is profound. Any embedding of a word is by definition static, unchanging with time or use. But if we can transform the axes it’s written in, we might be able to make for ourselves a dynamic understanding of words, which changes with context
Attention Loading ...
The researcher set to work improving her model.
The first thing she noticed was that her model had no way to separate homonyms. For example, the word ‘bank’ brings to mind the establishment that deals with money, but it might as well be used to describe the ground beside a river, or to base someone’s confidence in something i.e. ‘... bank on Lamine Yamal to win the tournament.’ All these individual meanings are tangled up in the static embedding.
Decades of hard work, research and public discourse has led the community (both in their private and professional endeavours to fix such problems) to a unanimous answer: Attention is All He Needs !
Attention is All You Need - NeurIPS 2017
Here's the basic idea: During the training process, the model learns three transformation matrices for each attention layer in the architecture, to project the embedding of the word, which initially is essentially the average of all possible meanings into a Q (Query), K (Key) and V (Value) vector each. Each of these vectors have a particular job,
Query: This asks “What word in the sentence are relevant to me?”. In ‘The robot moved the box.’, the relevant words for the ‘robot’ would be ‘moved’ and ‘box’ with the initial ‘the’ also being a less useful but relevant word.
Key: This asks “What am I?”. In the prior sentence, ‘robot’ would be the agent, ‘moved’ would be an action and ‘box’ would be the object being acted upon.
Value: This represents the feedback on the actual utility of the Q-K answer based on the expertise of the layer - like a consultant for the researcher that derives insights from raw data.
Comparing the Query and Key values, we compute a similarity score which tells us that ‘robot’ is strongly correlated with ‘moved’ and ‘box’ in the sentence. These scores are then converted into percentages or attention weights that add up to 100%. In our sentence, ‘moved’ might get a score of 60% and ‘box’ a score of 30% with others getting very low scores. These weights are then used to create a weighted blend of all the corresponding value vectors, giving us a more context aware representation of the word ‘robot’ with a 60% contribution from ‘moved’ and 30% contribution from ‘box’. This tries to tell the model that the robot is defined by the action of movement it performed on the box object.
Mathematically, this is achieved through the Scaled Dot Product Attention Function:

QKT computes the similarity score by taking the dot product of the Query vector with the transpose of the Key vector, the mathematical equivalent of comparing the query and key vectors. This score is then scaled by the square root of the dimension of the K vector d_K, which is then put through the softmax function which converts the scaled similarity scores to the percentages summing up to 100%.
The scaling isn’t an intuitive conclusion from the core idea, but it is necessary to account for the nature of the softmax function where numerically high scores would reduce even similar but slightly lower scores to near zero, owing to the exponential transform. The motivation for using the softmax function rather than linear scaling for example, is simply to allow the model to highlight and focus on the most important relationships.
Thus attention’s job is to warp and twist the space of words with the context of the sentence to pull similar ideas and words together and highlight inter-word relationships to enrich the meaning of a single word. This functionally resolves the problem of not ‘knowing’ which bank is being referred to in “Arthur Morgan robs bank of 20,000 dollars in daring daylight robbery.” – although it is technically possible to have money hidden near a river bank as well, but highly unlikely.
Analyzing ...
Having tackled the problem of enriching our embeddings with the context of a sentence to bring out the right meanings, the researcher now needs a way to actually analyze and understand the information. For this, she reverts back to her deep learning knowledge and achieves it through something known as a Multi-Layer Perceptron (MLP) or Feed-Forward Network (FFN), big complicated words for essentially a big complicated function.
For the uninitiated, Feed-Forward Networks are a category of Neural Networks with only unidirectional flow of data, essentially acting as a more intuitive visual representation of a deeply nested non-linear mathematical function.
In essence, the more the layers, the deeper the analysis can be, and the more neurons there are per layer, the richer or more varied the analysis can be. However, we cannot actually keep increasing the number of layers and adding more neurons in reality, as the function tends to learn the data itself to boost its performance on the data, rather than trying to capture the relationships in the data.
In the researcher’s application, any one of these functions attempts to understand something about the sentence, ranging from a more robust understanding of the Subject-Verb-Object relationship of a sentence, to understanding sarcasm or irony and outputting feedback based on its analysis. During training, this function essentially carves up our high dimensional thought space into regions that each correspond to a particular concept. The function then analyzes the new sentence with respect to these regions and produces a feedback based on this understanding. It might think “This vector is ambiguous regarding the ethics of political misinformation, so I will add a component that pushes it towards a clearer representation.” or “This vector lacks historical context regarding the engineering of the modern banana, which it probably should have in this context, so I will add a component to introduce that feature.” and enrich the data.
‘A Unit of Thought’
With her text having gone through one pass of an attention transform, and one pass of the MLP feedback, she thus has on her hands a single unit of thought (or a decoder block). One unit of thought thus transforms the space to better capture the context of the text, and the MLP produces its feedback on the text. Now, from an architecture perspective, we also have an Add & Norm layer, which just adds the feedback generated by the MLP to the vector, with the normalization part of it just ensuring convenience from a technical perspective.
Thus, one unit of thought enriches the understanding. But say for example that the missing historical context of the engineering of the banana that the MLP tried to introduce into the understanding is only imperfect as of now, what then? The solution is to simply think again upon this new richer understanding. Thus, basic LLMs employed a series of decoder blocks sequentially connected to each other to keep enriching the understanding with better and better context and analysis.
To complete this transformer, we just need to complete the task of predicting the next word with our deep understanding. To achieve this, we simply apply the transpose of the linear transformation we used to convert the generate the embedding for the original word, going back to the space of all words, and using the softmax function again to get probabilities for the next probable word.
Replying
And thus assuming that her model is pretty good at predicting the next word, the researcher now simply loops through the model again and again using the latest word to predict the next word in order to generate a coherent reply, knowing when to stop through a silence token which signals a stop in the text.
A token just refers to often occurring parts of words, something that the researcher realized makes her model more robust to her generation’s crazy speed of inventing new vocabulary. She couldn’t possibly have encoded skibidi, yeet and rizz beforehand! Tokenizing them into ["sk", "ib", "idi"], ["ye", "et"] and ["r", "izz"] (or something close to that) helps her better deal with the messiness of human language.
She trained her model not only on messages, but on most of the vast wealth of human language data. Her model was now pretty good at continuing conversations in the most natural of mannerisms. Make no mistake, she had just achieved a breakthrough, which just kept getting better and better the bigger it became and the more data it received. A particular strength of its was being able to continue every conversation, whether it was about love, about poetry or even about law and politics. The researcher did however notice a few problems, one was a problem even observed in humans that are such ‘jack of all trades’ – the model was good enough but sometimes it just made mistakes, and it hadn’t had much focused training on each subject, meaning slow replies!
But knowing how the technology worked, she simply changed the architecture of the model, splitting the FFNs into ‘expert’ – one that focused on grammar, one that focused on love and another that maybe focused simply on 18th century poetry, training each one of them separately with a simple router to ‘route’ the query to the necessary expert. This mixture of experts architecture allowed much faster and much more accurate replies by only engaging the relevant experts to answer a particular query.
Another problem altogether was one of hallucination, especially in long chat sessions. Not unlike any other person chatting late into the night, the model simply dreamed into existence things that never happened and referenced them in conversations leading to awkward situations to say the least. But being an Astrid S fan, the researcher figured out a solution to this problem deriving inspiration from one of her most popular songs – Think Before I Talk. She discovered that a Chain of Thought, or simply telling the model to think things through and check previous conversations and plan out the response in itself reduced the rate of hallucination drastically.
Aftermath
Especially with all the improvements, the researcher had achieved something massive, she’d just taught a machine to act human, more than overachieving on her earlier goal apart from the casual ‘you do kinda sound like a robot imo’. Recognizing the potential in this, she had also revealed the technology to the public and set off an arms race of people trying to achieve godlike intelligence, a model that is so much superior to human intelligence that it makes no mistakes, can do any task imaginable flawlessly and can even make novel discoveries. They call this AGI or Artificial General Intelligence. Every company and country on earth today is pumping huge amounts of money into achieving god, caring neither about the sanctity of the arts, nor the environment or even just human beings, but simply chasing the potentially massive private economic incentives. Some might even go as far as calling this blind investment into an increasingly saturating technology a bubble, but there is one clear winner – Jensen Huang’s leather jacket.
As for the AI researcher, her crush today is happily married with two kids. It’s taken about a decade and a half for her to get to this point after all. But no need to worry, her AI boyfriend is much more loving, attentive and sycophantic than any human could ever be. They live a beautiful life together, talking to each other about their days and daily problems, cooking and baking together, solving problems together, venting about serious issues to each other and sharing their deepest secrets. All that remains in making this a true relationship acceptable to the Catholic Church is the human form!