Why We Need a Translation in the First Place

When we say AI "understands language," we are telling a small white lie. A language model does not see letters, words or sentences the way we do. In its world, everything is a number. Just as a calculator works only with digits, the one and only thing flowing through a modern AI system is vast strings of numbers. So there is a basic problem to solve: how do we translate text written by a human into numbers a machine can process?

This translation may sound mechanical, but the real difficulty is not in the mechanics. Mapping a word to some arbitrary number is easy; you give every word in the dictionary an ID and you are done. But that throws meaning straight in the bin. If you assign 4172 to "king" and 9038 to "queen," nothing in those two numbers carries the faintest trace of how closely related the two concepts are. The real challenge is to do the translation so that meaning travels along with the text.

In this article we will build the three cornerstones of that translation, one at a time: first tokenization, which splits text into pieces; then embeddings, which load meaning onto each piece; and finally the vector space in which those meanings live. The goal is to make this quiet mechanism at the heart of modern AI understandable without requiring any background in mathematics.

Tokenization: Cutting Text Into Bite-Size Pieces

The first step of the translation is breaking text into manageable pieces. These pieces are called tokens. A token is often not a whole word; sometimes it is part of a word, sometimes a suffix, sometimes just a few letters. The system feeding the model splits incoming text into these tokens and assigns each one its index in the model's vocabulary. Perhaps the simplest analogy is food: you cannot swallow a sentence whole, you first chew it into bites. Tokenization is the step where the model cuts text into digestible bites.

So why use these odd sub-pieces instead of whole words? Because fitting every word in the world into a single dictionary is impossible. New words, typos, names and technical terms can be invented endlessly. Instead, models use a fixed-size vocabulary made of frequently occurring pieces. A long, rare word like "unbelievable" might be split into familiar parts such as "un," "believ" and "able." This way the model can reconstruct even a word it has never seen out of pieces it already knows, much as you can sound out an unfamiliar word by breaking it into syllables.

This splitting looks innocent, but its consequences run deep. How many tokens a text breaks into directly determines cost, speed and how much context the model can hold at once, because models measure text in tokens. And token boundaries do not fall with equal efficiency in every language. Many tokenizers built for English shred words in agglutinative languages like Turkish at points that have nothing to do with meaning, which means burning more tokens to say the same thing. Contrary to what most people assume, tokenization is not a technical footnote; it is a foundational design decision that quietly shapes product quality.

Embedding: Loading Meaning Onto a Token

Token IDs are identifiers, not coordinates of meaning. A model maps each token ID to a learned input vector, then changes its representation as the token is processed in context. A separate sentence or document embedding model can turn a passage into one vector for search. These two uses of “embedding” are related but not interchangeable. Google's embedding guide distinguishes simple static vectors from richer representations.

The vector values are learned during training rather than assigned as human-readable labels. Similar contexts can produce useful geometric relationships, but an individual coordinate usually cannot be read as “royal” or “alive.” A word can also change meaning with context: “bank” in a financial document is different from “bank” beside a river. Static word vectors give one representation per word, while contextual models can represent the occurrence differently.

Think of it with a simple analogy. Suppose we score every word along a few axes: how "alive" it is, how "royal" it is, how "masculine versus feminine" it is. "King" scores high on royal and high on masculine; "queen" scores high on royal and high on feminine; "banana" scores low on all three. In real embeddings these axes number in the hundreds and none of them carry labels as tidy as ours; but the idea is the same. Each word becomes a list of coordinates describing the features that make up its meaning.

Vector Space: A Map of Meaning

Since we have turned every word into a list of numbers, that is, into coordinates, we can now picture every word as a point, just like a city pinned on a map by latitude and longitude. With two numbers we can build a two-dimensional map, with three a three-dimensional space. Because embeddings contain hundreds of numbers, the place where words live is a space of hundreds of dimensions. We cannot visualize it, but the logic is exactly the same: each word is a point with its own position in this vast space. This space is called the vector space.

The beauty of this map is that the positions on it are not random. When embeddings are learned well, words close in meaning land close together. "King," "queen," "throne" and "palace" gather in one neighborhood, while "banana," "apple" and "strawberry" cluster in a completely different one. Meaning itself has turned into physical proximity. Just as cities near each other on a geographic map often share a similar climate, points near each other on this map of meaning carry similar meanings.

Classic static word-embedding demonstrations show that some relationships can appear as directions in a vector space. The “king − man + woman ≈ queen” example is an illustration from particular embeddings, not a general rule for every model or language. Such analogies can also reflect stereotypes in training data. Test the relationships relevant to the task instead of assuming the geometry always behaves this neatly.

Semantic Similarity: The Angle Between Two Meanings

Now that words have become points on a map, we can ask a very powerful question: how similar are two words? To measure this, we look at how much the two points "face the same direction." The most common method is cosine similarity. The name sounds intimidating, but the idea is simple: picture each vector as an arrow reaching from the center of the space out to its point. If two arrows point the same way, the meanings they represent are similar. The smaller the angle between the arrows, the greater the similarity.

Cosine similarity turns exactly this angle into a number. If two arrows point in precisely the same direction, the similarity is close to 1; if they are completely unrelated, standing at a right angle, it is near 0; if they point in opposite directions, it approaches -1. An important subtlety: cosine similarity considers only the direction of the arrows, not their length. So even if a concept has a "longer" vector because it appears very often, the similarity verdict is not affected; only the direction of the meaning speaks.

For a suitable embedding model, “automobile” and “car” may receive similar vectors despite different spelling. The score depends on the model, language, and context. In a real search product, pairwise vector similarity produces candidates; keyword matching, metadata filters, and a reranker may still be needed to place the most useful result first. See our semantic search guide for the retrieval workflow.

Sentences and Documents: From a Single Word to a Whole Text

Search systems commonly use a dedicated embedding model to represent queries and document passages. Long documents are usually divided into smaller passages before indexing so that relevant sections can be found and cited. Query and document vectors must come from compatible model settings; an arbitrary word vector and an arbitrary document vector cannot be compared meaningfully. The Sentence Transformers documentation shows separate query and document encoding methods for retrieval.

This compression is a subtler job than for words. The meaning of a sentence is not merely the sum of its words; their order, which word modifies which, and the surrounding context all come into play. "The dog bit the man" and "The man bit the dog" contain the same words but say entirely different things. So models that produce sentence embeddings account for the relationships between words and generate a single holistic representation. The result is again a vector, but now it carries the meaning of a sentence or a paragraph.

This ability opens the door to one of AI's most useful applications: semantic search. You turn a user's question into a vector, then, among the vectors of the thousands of documents you hold, you retrieve the ones facing the most similar direction, that is, those with the highest cosine similarity to the question. The user gets not the documents that happen to contain the exact words of the question, but the documents closest to its meaning. Documents that say the same thing in different words, which keyword matching misses entirely, are caught this way too.

What Does This Do in a Real Domain Like Law?

These concepts can seem abstract, but their value shows up when they touch ground in a concrete domain. Turkish law is exactly such a domain. What a lawyer is after is usually not a document that happens to contain a particular word, but rulings that deal with a particular legal situation. When a user searches for "termination of a lease for just cause," they want rulings on that same legal issue to surface even if those rulings never use that exact phrase. Semantic similarity makes this possible because it reaches past the words to grasp the essence of the matter.

For a legal corpus, a practical pipeline indexes versioned passages with document IDs, dates, and access permissions. A query retrieves candidate passages, then a reranker or reviewer checks relevance before an answer cites the source. Semantic similarity can find differently phrased material, but it can also retrieve a passage that is topically close and legally inapplicable. Dates, jurisdiction, authority, and exact terminology must remain visible.

Turkish morphology and specialist vocabulary make evaluation especially important. Test inflected forms, abbreviations, citations, and terms whose legal meaning differs from everyday use. Compare semantic search with a keyword baseline on a labeled set of real questions; measure whether the correct passage appears in the top results and whether irrelevant but similar passages are rejected. Tokenization and embeddings matter, but they are only parts of a reliable search system.

Wrapping Up: The Intuition Beneath the Numbers

Tokenization turns text into token IDs. A language model maps those IDs to input vectors and updates its internal representations as it processes context. A search system may use a separate model to map whole queries and passages into comparable vectors. These are related techniques with different jobs.

Vector geometry gives a useful way to rank candidate passages: a query and a passage with similar representations may be related. The score is a retrieval signal, not proof that the passage is correct or applicable. A reliable system keeps source, date and access metadata and checks the answer against the cited text.

When evaluating a semantic search system, test real questions against a labeled corpus and compare it with keyword search. That tells you whether the chosen embedding model helps users find the right material in this domain, including Turkish inflections and specialist terms. Our RAG and semantic search solutions page explains how we run that kind of embedding test on real queries.

Sources and Further Reading