Transformer

In recent years, Transformers have revolutionized the field of deep learning, especially in Natural Language Processing (NLP). From powering ChatGPT, Google Translate, to even helping in fashion recommendation systems, Transformers are at the core of today’s intelligent systems.

Transformer

• A transformer model is a neural network architecture designed for processing sequential data, introduced in the paper “Attention is All You Need” by Vaswani et al. in 2017.
• Unlike RNNs, which process data sequentially (step by step), Transformers process entire sequences in parallel.
• Transformers transformed the field of deep learning by introducing a more efficient self-attention mechanism, effectively replacing traditional recurrent neural networks (RNNs).

Major Models based on Transformers

• BERT (Bidirectional Encoder Representations from Transformers) – Google’s encoder-decoder model for language understanding (e.g., search engines).
• GPT (Generative Pre-trained Transformer) – OpenAI’s decoder-only model powering ChatGPT and other generative AI tools.
• Vision Transformers (ViTs) – Outperform CNNs in some image-related tasks.
• Multimodal Models – Used in text-to-image diffusion models (e.g., DALL·E, Stable Diffusion) and vision-language models (VLMs).

How Transformer Model Works

• Transformers use query, key, and value vectors (like a database system) to calculate attention weights, determining how much each part of the input sequence relates to others.
• The query represents what a token searches for, the key holds its identity, and the value provides its content—weights prioritize relevant connections while ignoring irrelevant ones.
• This self-attention mechanism allows models to dynamically understand context by treating their learned vocabulary like a “database” of language patterns.

Let’s break down the Transformer architecture in simple terms:

01. Tokenization and Input Embeddings

• Tokenization
  • Humans read text as characters (letters, numbers, symbols), but AI models break language into tokens (the smallest meaningful units).
  • Each token is assigned a unique ID number, allowing the model to efficiently process text by referencing these IDs instead of raw words.
  • Tokenization simplifies text for AI processing.
• Input Embeddings
  • Before analyzing relationships between tokens, the model converts each token ID into a numerical vector (called an embedding).
  • These embeddings can be:
    • Learned during training (the model adjusts them to better represent word meanings) or
    • Taken from pre-trained models (like Word2Vec or GloVe) for a head start.
  • These initial embeddings provide a context-free representation of each token, which the transformer later refines using attention mechanisms.
  • Embeddings turn words into numbers, enabling mathematical analysis of language.

02. Positional Encoding

• Positional encoding is a method to incorporate information about the order of words in a sequence, which is crucial for understanding the meaning of the text but is not inherently present in the transformer’s self-attention mechanism.
• Transformers process sequences in parallel, so they need a way to distinguish between tokens based on their position.
• Positional encoding adds unique vectors to each token’s embedding, reflecting its position within the sequence.
• This allows the model to differentiate between “the cat sat” and “sat the cat” and understand the sentence structure.

03. Generating Query, Key, and Value Vectors

• After positional encoding, each token embedding is transformed into three distinct vectors through learned linear projections:

1. 1. Query (Q) = Embedding × W_Q (dim =d_k )
   2. Key (K) = Embedding × W_K (dim =d_k )
   3. Value (V) = Embedding × W_V (dim =d_v)

• These weight matrices (W_Q, W_K, W_V) are trained during self-supervised pretraining, enabling the model to dynamically project inputs into spaces optimized for attention-based relationships.
• The separate dimensions (d_k for Q/K, d_v for V) allow flexible representation learning.

04. Attention Mechanism (Self-Attention)

• The transformer’s attention mechanism calculates attention weights by comparing each token’s query vector with the key vectors of every other token in the sequence.
• Once attention weights are computed, each token is associated with a vector indicating how much influence each other token in the sequence should have on it.
• Now multiply the value vector of each token by its  attention weight 
• Then, summing these weighted values to capture the overall contextual influence of all other tokens on the target.
• This contextually enriched representation is then added to the target token’s original embedding, creating a final representation that incorporates both its inherent meaning and its relationships with the surrounding text.

05. Multi-Head Attention

• Transformers use multi-head attention to capture diverse relationships between tokens.
• Each token embedding is split into h equal parts.
• Each part goes through parallel Q, K, V matrices, forming attention heads.
• These heads process information independently, learning different semantic patterns.
• Outputs from all heads are concatenated before entering the feedforward layer.

06. Feedforward Neural Network

• Linear Transformation 1

  • The attention output for each token is passed through a fully connected (dense) layer:  Output₁=ReLU(XW₁+b₁) 

  • Here, X is the input from the attention layer.
  • W₁​ and b₁​ are learned weights and biases.
  • ReLU activation introduces non-linearity.

• Linear Transformation 2

  • The output from the first layer is passed through another dense layer:

    Output₂=Output₁W₂+b₂​

  • This brings the dimensionality back to match the input shape.

07.  Residual Connections & Layer Normalization

• Residual connections add the original (position-encoded) token embedding back to the attention-updated vector to preserve the token’s original meaning.
• This helps balance new contextual information with the initial semantic content.
• The combined vector is then passed through a feedforward layer and layer normalization to maintain consistent size and training stability across layers.

         Final Output = LayerNorm ( X + Output₂)

08. Generating outputs

• Now the model is capable of giving its final outputs based on learned contextual information
• In case of autoregressive LLMs, the final layer uses softmax to predict next-token probabilities from the vocabulary
• Selects output token via sampling (controlled by hyperparameters like temperature/top-k)

Encoder and Decoder

The original Transformer has two parts:

Encoder

• Takes the input sequence.
• Processes it using self-attention and feedforward layers.

Decoder

• Takes the encoder output + previous outputs to generate new sequences (e.g., in translation or text generation).

For classification tasks, we often just use the encoder (like in BERT).

For text generation, we utilize both the encoder and decoder (similar to GPT or T5).

Key Advantages Over Older Models

• Better at Long-Range Dependencies
  • Older models like RNNs (Recurrent Neural Networks) process data sequentially, making it hard to understand connections between distant elements (e.g., words at the start and end of a long sentence).
  • LSTMs (Long Short-Term Memory networks) improved RNNs but still struggled with very long sequences.
  • Transformers analyze entire sequences at once, making them far better at capturing long-range patterns.
• Parallel Processing for Speed & Efficiency
  • Unlike RNNs (which work step-by-step), transformers process data in parallel, making them much faster, especially when using GPUs.
  • This efficiency allows training on massive datasets, leading to more powerful AI models.
• Superior to CNNs for Some Tasks
  • CNNs (Convolutional Neural Networks) process images in small sections, missing broader patterns.
  • Transformers use attention to analyze entire images or texts at once, improving performance in tasks like object detection and language understanding.

Real-World Applications

• NLP (chatbots, text generation, summarization)
• Machine Translation (Google Translate)
• Fashion Recommendation (e.g., “Show me vintage-style dresses under $100”)
• Search Engines
• Computer Vision (image segmentation, object detection,Image Captioning)
• Generative AI (text-to-speech, image generation).

Python Implementation for Sentiment Analysis(Using Hugging face Transformer Library)

# Import Necessary library
from transformers import pipeline

# Load a pre-trained model
classifier = pipeline("sentiment-analysis")

# Run the classifier
result = classifier(input('What is your review about our delivered product:'))[0]

# Get label and score separately
label = result['label']
score = result['score']

# Print them
print("Label:", label)
print("Score:", "{:.2f}%".format(score * 100))

if label == 'POSITIVE' and score > 0.9:
    print("The sentiment is strongly positive."),
else:
    print("The sentiment is strongly negative.") 

Output:
What is your review about our delivered product:
I ordered dress for size M but it is tight seems smaller size

Label: NEGATIVE
Score: 96.59%
The sentiment is strongly negative.

References

1. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention Is All You Need. arXiv:1706.03762

2. IBM. (n.d.). Transformer Model Explained. IBM Think. Retrieved from https://www.ibm.com/think/topics/transformer-model