Supervised Fine Tuning: Delete the Generic Model

Q: What's the difference between supervised fine tuning and reinforcement learning from human feedback (RLHF)?

**Supervised fine tuning** uses labeled input-output pairs and optimizes for prediction accuracy. RLHF uses human preferences as rewards and optimizes for alignment with human values. SFT happens first in the training pipeline. RLHF happens after SFT to further refine behavior. For most enterprise tasks, SFT alone is sufficient. RLHF adds complexity and requires significantly more computational resources. Only pursue RLHF if you need the model to navigate subjective trade-offs or ethical boundaries.

Supervised Fine Tuning: Delete the Generic Model

Supervised fine tuning is the process of taking a pretrained language model and teaching it your specific task through labeled examples. No fluff. No theoretical nonsense. You feed it domain-specific data, adjust the weights, and deploy a model that actually understands your business context.

Most companies waste months arguing about whether to fine tune or rely on prompt engineering. Wrong question. The real question is: can your current model hit your accuracy target? If not, supervised fine tuning is your delete button for generic responses.

This article destroys the academic hand-waving and shows you how to build production SFT pipelines that scale. We'll cover dataset preparation, training infrastructure, evaluation metrics, and deployment patterns that actually work in enterprise environments.

▹What Supervised Fine Tuning Actually Does
▹When to Fine Tune vs When to Prompt
▹Dataset Construction That Doesn't Waste GPU Time
▹Training Infrastructure for Real Engineers
▹Evaluation Metrics Beyond Perplexity
▹Deployment Patterns That Scale
▹Cost Analysis: Cloud vs On-Premise
▹Common Failure Modes and How to Avoid Them
▹FAQ

What Supervised Fine Tuning Actually Does

Supervised fine tuning modifies a pretrained model's parameters using task-specific labeled data. You start with something like GPT, LLaMA, or Mistral. These models understand language but don't understand your legal documents, medical records, or code repositories.

The training process is straightforward:

▹Collect input-output pairs for your task
▹Compute loss between model predictions and ground truth
▹Backpropagate gradients through the model
▹Update weights to minimize error

The mathematics is identical to standard supervised learning. The difference is scale. You're updating billions of parameters across multiple GPUs or TPUs.

Here's what changes during fine tuning:

▹Token probability distributions shift toward domain-specific vocabulary
▹Attention patterns adapt to your data structure
▹Output formatting aligns with your expected responses

The model doesn't learn new capabilities. It specializes existing ones. If the base model can't reason about code, fine tuning won't magically add that ability. You're optimizing, not inventing.

For enterprises dealing with database optimization tools, fine tuning can adapt language models to generate SQL queries specific to your schema and performance requirements.

When to Fine Tune vs When to Prompt

The decision matrix is simple:

Use prompting when:

▹Your task fits in the context window (< 4,000 tokens typically)
▹You need flexibility to change behavior rapidly
▹Your domain knowledge can be expressed in natural language
▹You're prototyping or testing hypotheses

Use supervised fine tuning when:

▹You need consistent output formatting that prompt engineering can't achieve
▹Your task requires domain knowledge too large for context windows
▹You're serving millions of requests and latency matters
▹You need to compress expert knowledge into model weights

Let's be precise. Prompt engineering has a token cost. Every instruction, every example, every piece of context burns tokens. At scale, those tokens compound. Supervised fine tuning moves that cost from inference time to training time.

Example: A customer support classifier. Prompt engineering approach requires 500 tokens of instructions per request. Fine tuning approach requires 0 instruction tokens. At 10 million requests per month, you're saving 5 billion tokens. That's real money.

The AI agent integration patterns we deploy at ByteForth rely heavily on fine tuned models because agents need deterministic behavior, not creative interpretation.

Dataset Construction That Doesn't Waste GPU Time

Your dataset quality determines everything. Garbage in, garbage out. No amount of GPU cycles fixes bad data.

Minimum viable dataset:

▹1,000 high-quality examples minimum
▹10,000+ for complex tasks
▹Balanced across all output categories
▹Representative of production distribution

Do not oversample edge cases. Do not create synthetic data until you've exhausted real data. Models learn distribution patterns. If your training distribution doesn't match production, your model fails in production.

Data Format

Use JSONL. One example per line. Simple structure:

{"input": "Your input text here", "output": "Expected completion"}
{"input": "Another example", "output": "Another completion"}

For instruction following:

{"instruction": "Summarize this document", "input": "Long document text...", "output": "Concise summary"}

Quality Over Quantity

One engineer manually reviewing 1,000 examples beats 100,000 scraped examples with noise. Manual review catches:

▹Inconsistent formatting
▹Contradictory labels
▹Ambiguous inputs
▹Edge cases that break assumptions

We've seen companies waste $50,000 in GPU time training on datasets that should have been filtered in 2 hours of human review.

Train-Test Split

80/20 split. Hold out 20% for validation. Never train on validation data. This is basic but teams forget under deadline pressure.

Use stratified sampling if you have imbalanced categories. Otherwise you'll train on one distribution and validate on another.

Training Infrastructure for Real Engineers

You need GPUs. Specifically:

▹Single GPU: Fine tune models up to 7B parameters using LoRA or QLoRA
▹Multi-GPU: Fine tune models 13B-70B parameters with distributed training
▹TPU pods: Fine tune models > 70B parameters (rarely necessary)

The Hardware Reality

For a 7B parameter model with full fine tuning:

▹Memory requirement: ~28 GB (4 bytes per parameter)
▹Training time on single A100: ~6-12 hours for 10,000 examples
▹Cost on AWS: ~$30-50 per training run

For parameter-efficient approaches like LoRA:

▹Memory requirement: ~12 GB
▹Training time: ~3-6 hours
▹Cost: ~$15-25 per training run

Training Frameworks

Use Hugging Face Transformers with the Trainer API. It handles distributed training, gradient accumulation, and checkpointing.

Basic training script:

from transformers import AutoModelForCausalLM, AutoTokenizer, Trainer, TrainingArguments
import torch

model = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-v0.1")
tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-v0.1")

training_args = TrainingArguments(
    output_dir="./results",
    num_train_epochs=3,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    learning_rate=2e-5,
    warmup_steps=100,
    logging_steps=10,
    save_steps=500,
    fp16=True,
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
)

trainer.train()

LoRA vs Full Fine Tuning

LoRA (Low-Rank Adaptation) trains small adapter matrices instead of all parameters. Trade-offs:

LoRA advantages:

▹90% less memory
▹3x faster training
▹Easy to swap adapters for different tasks

Full fine tuning advantages:

▹Slightly higher final accuracy
▹Better for dramatic distribution shifts
▹No adapter overhead at inference

For most enterprise tasks, LoRA wins. The accuracy difference is < 2% but the infrastructure savings are massive.

Companies implementing on premise ERP software often prefer full fine tuning when they control hardware and want maximum model performance without cloud dependencies.

Evaluation Metrics Beyond Perplexity

Perplexity is useless for production deployment. It measures how surprised the model is by the test data. You care about task performance, not statistical surprise.

Task-Specific Metrics

Classification tasks:

▹Accuracy, Precision, Recall, F1
▹Confusion matrix analysis
▹Per-class performance breakdown

Generation tasks:

▹BLEU/ROUGE for semantic similarity
▹Exact match percentage
▹Human evaluation on sample outputs

Structured output tasks:

▹Format validity rate
▹Schema compliance
▹Parsing success rate

The Human Evaluation Requirement

You must manually review 100-200 outputs before deployment. Automated metrics miss:

▹Subtle tone problems
▹Factual inaccuracies
▹Formatting edge cases
▹Cultural insensitivity

Set up a simple evaluation interface. Show model input and output. Rate on a 1-5 scale. Takes 2 hours. Prevents production disasters.

A/B Testing in Production

The only metric that matters is production performance. Deploy the fine tuned model to 10% of traffic. Measure:

▹Task completion rate
▹User satisfaction scores
▹Downstream conversion metrics
▹Error rates and escalations

Compare against your baseline (prompt-engineered model or human operators). If the fine tuned model doesn't beat baseline by > 10%, don't deploy it.

Deployment Patterns That Scale

Inference infrastructure determines whether your fine tuning investment pays off.

Deployment Options

Option 1: Cloud API (AWS Bedrock, Azure OpenAI)

▹Pros: Zero infrastructure management
▹Cons: Vendor lock-in, ongoing API costs, latency variability
▹Cost: $0.002-0.02 per 1,000 tokens

Option 2: Self-Hosted on GPU Instances

▹Pros: Full control, lower marginal cost at scale
▹Cons: DevOps overhead, upfront GPU costs
▹Cost: $1-3 per hour for GPU instances

Option 3: Serverless GPU (Modal, Replicate)

▹Pros: Pay-per-request, auto-scaling
▹Cons: Cold start latency, limited customization
▹Cost: $0.0001-0.001 per inference

Serving Infrastructure

Use vLLM for maximum throughput. It implements:

▹PagedAttention for efficient memory management
▹Continuous batching for higher utilization
▹Optimized CUDA kernels for speed

Deployment example with FastAPI:

from vllm import LLM, SamplingParams
from fastapi import FastAPI

app = FastAPI()
llm = LLM(model="./fine_tuned_model", tensor_parallel_size=4)

@app.post("/generate")
async def generate(prompt: str):
    sampling_params = SamplingParams(temperature=0.7, max_tokens=256)
    outputs = llm.generate([prompt], sampling_params)
    return {"response": outputs[0].outputs[0].text}

Caching Strategy

Most requests are repetitive. Implement semantic caching:

▹Embed incoming prompts using a smaller embedding model
▹Check vector database for similar cached responses
▹Return cached response if similarity > 0.95
▹Otherwise, generate and cache

This reduces inference cost by 40-70% in production environments.

For systems requiring AI inventory management, caching is critical because SKU queries and forecasting requests follow predictable patterns.

Cost Analysis: Cloud vs On-Premise

Let's calculate real numbers for a 7B parameter model serving 1 million requests per month.

Cloud API Approach

▹Average tokens per request: 200
▹Cost per 1,000 tokens: $0.01
▹Monthly cost: 1M requests * 200 tokens / 1,000 * $0.01 = $2,000/month
▹Annual cost: $24,000

Self-Hosted GPU Approach

▹Hardware: 1x A100 GPU instance
▹Cost: $2.50/hour * 730 hours = $1,825/month
▹Annual cost: $21,900
▹Breakeven: Immediate at this volume

On-Premise Hardware

▹Upfront cost: 1x NVIDIA A100 = $12,000
▹Power consumption: ~400W * $0.10/kWh * 730 hours = $30/month
▹Annual cost: $12,000 + ($30 * 12) = $12,360 first year, $360 ongoing
▹Breakeven: 6 months

At enterprise scale (10M+ requests/month), on-premise hardware destroys cloud economics. The multi tenant architecture patterns we deploy allow hardware amortization across multiple clients.

Common Failure Modes and How to Avoid Them

Overfitting

Symptom: Perfect training accuracy, terrible validation accuracy.

Cause: Too many training epochs, too small dataset, model capacity too high.

Fix: Stop training when validation loss stops improving. Use early stopping callbacks. Add more data.

Catastrophic Forgetting

Symptom: Model performs well on fine tuning task but forgets pretrained capabilities.

Cause: Learning rate too high, too many training steps.

Fix: Use lower learning rates (2e-5 to 5e-5). Train for fewer epochs. Consider LoRA instead of full fine tuning.

Dataset Bias

Symptom: Model performs poorly on production data despite good validation metrics.

Cause: Training data doesn't match production distribution.

Fix: Continuously sample production requests. Retrain monthly with updated data. Monitor performance drift.

Memory Errors

Symptom: CUDA out of memory during training.

Fix: Reduce batch size. Enable gradient checkpointing. Use gradient accumulation. Switch to LoRA.

training_args = TrainingArguments(
    per_device_train_batch_size=1,  # Reduce batch size
    gradient_accumulation_steps=16,  # Accumulate gradients
    gradient_checkpointing=True,     # Save memory
    fp16=True,                       # Use half precision
)

Inference Latency

Symptom: Model takes > 2 seconds per request.

Cause: Large model, no optimization, sequential processing.

Fix: Use vLLM. Enable continuous batching. Implement request queuing. Consider model quantization.

Enterprises running AWS managed services often encounter latency issues when scaling fine tuned models without proper serving infrastructure.

FAQ

What's the difference between supervised fine tuning and reinforcement learning from human feedback (RLHF)?+

Supervised fine tuning uses labeled input-output pairs and optimizes for prediction accuracy. RLHF uses human preferences as rewards and optimizes for alignment with human values. SFT happens first in the training pipeline. RLHF happens after SFT to further refine behavior. For most enterprise tasks, SFT alone is sufficient. RLHF adds complexity and requires significantly more computational resources. Only pursue RLHF if you need the model to navigate subjective trade-offs or ethical boundaries.

How many GPUs do I actually need to fine tune a 70B parameter model?+

For full fine tuning of a 70B model, you need minimum 8x A100 80GB GPUs using DeepSpeed ZeRO-3 optimization. With LoRA, you can fine tune on 4x A100 40GB GPUs. With QLoRA (quantized LoRA), you can technically fine tune on 2x A100 40GB GPUs but training becomes extremely slow. The practical answer: if you're asking this question, start with a 7B or 13B model. 70B models are overkill for 95% of enterprise tasks. The performance gain rarely justifies the 10x infrastructure cost.

Can I fine tune proprietary models like GPT-4 or Claude?+

No. Proprietary models from OpenAI, Anthropic, and similar vendors do not allow direct fine tuning of their largest models. They offer limited fine tuning APIs for smaller variants (like GPT-3.5) but you're constrained to their infrastructure and pricing. If you need full control over supervised fine tuning, use open source models like LLaMA, Mistral, or Falcon. You own the weights. You own the infrastructure. You own the data. Zero vendor lock-in. This is why companies building AI infrastructure default to open source models for production systems.

Supervised fine tuning is the delete button for generic AI. Build your dataset. Train your model. Deploy to production. Measure ROI. Everything else is noise.