Since V9
Post #7 ended with a promise: explore whether reinforcement learning could close the gap between “almost always emits the right tags” and “always emits the right tags.” Two things happened since then, neither of which got their own blog post, so here is the short version.
V10: GRPO Reinforcement Learning
We trained a fresh LoRA adapter using Group Relative Policy Optimization — the same RL method from the DeepSeekMath paper. Instead of rewarding task completion, we wrote a 5-component rule-based reward function that scores each completion on structural formatting quality: does it have valid <tool_call> tags with a JSON body? Does the tool name exist in the Nexus schema? Is there chain-of-thought reasoning before the call? Are there hallucinated tool names? Does it stop cleanly after the closing tag?
The reward function validated against a 12-tool Nexus schema: Read, Write, Edit, LS, Bash, Grep, Glob, WebFetch, TodoWrite, Skill, Task, and ScopeCheck. Training ran for 150 steps on fewer than 1,000 curated prompts — orders of magnitude less data than the SFT stages, but targeted at exactly the behavior we wanted to reinforce.
DAPT: Domain-Adaptive Pretraining for Offensive Security
Separately, we trained a LoRA adapter on approximately 10,000 offensive-security code documents — exploit frameworks, penetration testing scripts, security tooling patterns. This adapter gets merged into the base model before SFT, grounding the model in the domain vocabulary and code patterns that our cybersecurity clients actually use.
The Adapter Chain
V10 introduced a three-stage adapter chain architecture. Each stage is a separate LoRA adapter, merged sequentially into the base weights:
Qwen/Qwen2.5-Coder-14B-Instruct
→ merge: qwen14b-dapt-offsec (domain grounding)
→ merge: qwen14b-code-trainer-v9_mixed (SFT capabilities)
→ LoRA: qwen14b-code-trainer-v10-grpo (RL formatting polish)
For deployment, all three merge into the base and quantize to a single Q5_K_M GGUF. The user never sees the chain — they get one model file.
The Full Picture
With V10, the version lineage now looks like this:
| Version | Method | Dataset | Tool % | Key Improvement |
|---|---|---|---|---|
| V6 | SFT | 26,126 rows | 0% | Code-only baseline |
| V7 | SFT | 28,862 rows | 63.8% | Restore tool-calling + agent |
| V8 | SFT | 34,104 rows | 57.6% | Fix multilingual + native format |
| V9 | SFT (curriculum) | 40,401 rows | 64.3% | Fix tag emission + curriculum |
| V10 | GRPO | <1K prompts | 100% | RL reward on tool-call formatting |
Why a New Base Model
The Qwen2.5-Coder-14B pipeline works. It produces a locally-deployed model that handles code generation, tool calling, and multi-turn agent behavior on consumer hardware. But the base model has constraints we cannot train around.
Qwen2.5-Coder-14B is a 14B parameter dense transformer. Every token processed activates all 14 billion parameters. On an RTX 5060 Ti 16GB, the Q5_K_M quantization runs but it is not fast — you feel the latency on every response, especially in multi-turn agent loops where the model calls tools, reads results, and calls more tools. In a pipeline that might chain 5–10 tool calls per task, inference speed is the bottleneck.
Then there is multimodality. Post #1 in this series built a vision adapter from scratch — a frozen Swin-B encoder, an MLP projector, and a tiny 1.5B decoder — to read screenshots of code. It worked, sort of. Syntax validity jumped from 19% to 61%, but exact match stayed at zero. The architecture was sound; the decoder was too small and the pipeline too disconnected from the main model. We never integrated vision into the 14B line because Qwen2.5-Coder is text-only. Adding multimodal means bolting on an external encoder and hoping the representations align.
Gemma 4 26B A4B solves both problems at the architecture level.
Gemma 4 26B A4B
Google released Gemma 4 as a family of open models in five sizes. The 26B A4B variant is the one that matters for us: a Mixture-of-Experts model with 25.2B total parameters but only 3.8B active parameters per token. It has 128 experts, activates 8 per token plus 1 shared expert, across 30 layers.
| Property | Qwen2.5-Coder-14B | Gemma 4 26B A4B |
|---|---|---|
| Architecture | Dense transformer | Mixture-of-Experts |
| Total parameters | 14B | 25.2B |
| Active parameters | 14B | 3.8B |
| BF16 model size | ~28 GB | ~52 GB |
| Context window | 32K tokens | 256K tokens |
| Modalities | Text only | Text + Image |
| Native tool calling | Yes (ChatML) | Yes (native function calling) |
| System prompt | Via template | Native system role |
| Reasoning mode | No | Configurable thinking |
| License | Apache 2.0 | Apache 2.0 |
The active parameter count is the headline number. At 3.8B active, Gemma 4 26B runs at roughly the speed of a 4B model while drawing on 25B parameters worth of learned knowledge through expert routing. On the same RTX 5060 Ti, inference should be roughly 3–4x faster than the 14B dense Qwen — critical for multi-step agent loops.
SFT Tuning Changes for MoE
Fine-tuning a Mixture-of-Experts model is not the same as fine-tuning a dense transformer. The architecture imposes constraints that change the training recipe at every level.
LoRA Cannot Touch the Experts
The 128 routed expert FFN modules use 3D nn.Parameter tensors — not nn.Linear layers. PEFT’s LoRA implementation cannot target them. The standard 7-module target list (q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj) hits only the shared attention and shared MLP layers. This is not a limitation we chose — it is architectural. The router stays frozen, and expert specialization is preserved by default.
The tradeoff: LoRA adapts the shared layers that every token passes through, while the 128 routed experts retain their pretrained knowledge. This means fine-tuning injects the Code-Trainer domain through the layers that affect all expert outputs, without disrupting which experts get activated for which tokens.
All Learning Rates Halved
MoE routing is sensitive to aggressive weight updates. The shared attention and MLP layers that LoRA touches interact with all 128 experts through the router — a large weight update can cascade through the routing decisions in ways a dense model never experiences. Qwen SFT used 1e-4; Gemma uses 5e-5. DAPT drops from 5e-5 to 2.5e-5. GRPO drops from 5e-7 to 2.5e-7.
VRAM Budget on A100 80 GB
The 52 GB model in BF16 is the constraint that shapes everything else:
| Component | VRAM |
|---|---|
| Model (BF16) | ~52 GB |
| LoRA adapter (r=32) | ~80 MB |
| AdamW optimizer (FP32 states) | ~320 MB |
| Activations (bs=1, grad checkpointing) | ~5–8 GB |
| Total | ~58–61 GB |
| Headroom on A100 80 GB | ~19–22 GB |
Batch size is forced to 1. Gradient accumulation of 16 gives an effective batch of 16 — matching the Qwen pipeline. But each step is slower because of the memory-bandwidth demand of moving 52 GB of weights through the compute pipeline.
12K Rows Instead of 15K
The Qwen pipeline trained on 15K rows in its budget window. With Gemma’s larger download (~52 GB vs ~28 GB) and slower per-step throughput, 12K rows fills the 7.5-hour timeout safely. Same curriculum structure: 80% full mixed data, 20% tool-calling polish at double the learning rate.
The Authentic Tool-Use Distribution Layer
This is the biggest change in the SFT data pipeline, and it applies to both the Qwen and Gemma models going forward.
The Problem
V9’s B+ slice — the synthetic multi-tool-call examples — used generic glaive function-calling data. The model trained on tool patterns it would never see at inference. When we analyzed the tool distribution in real Nexus agent sessions, the skew was dramatic:
| Tool | Observed in Sessions | Generic Training Data |
|---|---|---|
| Bash | 67% | Overrepresented |
| Read | 22% | Underrepresented |
| Edit | 5% | Underrepresented |
| Write | 4% | Underrepresented |
| ScopeCheck, Glob, Skill, ... | ~2% | Absent or near-zero |
Training on a 67/22/5/4 distribution teaches the model that Bash is the answer to everything. In real offensive security workflows, the model needs to know when to scope-check a target, when to grep for credentials, when to write a payload — not just default to running a command.
The Data Feedback Loop
We built an automated pipeline that closes the gap between training data and inference reality:
Step 1 — Pull real sessions. Real Nexus agent sessions from edge devices — bug bounty engagements, HackTheBox labs, TryHackMe rooms — are synced from Cloudflare R2 via a paginated, ETag-based pull script. 433 session files pulled, 245 successfully converted after filtering out incomplete or non-offensive-security sessions.
Step 2 — Extract argument templates. The ingested sessions are mined for tool-call argument templates — the actual arguments JSON objects from each <tool_call> in real assistant responses. Templates are deduplicated, sanitized (real domains and session-specific paths stripped), and filtered to retain only offensive-security-relevant patterns.
Step 3 — Synthesize with balanced distribution. 2,000 synthetic multi-tool-call training examples are generated using those extracted argument templates, but with a deliberately flattened tool distribution:
| Tool | Target Weight | Tool | Target Weight |
|---|---|---|---|
| Bash | 25% | WebFetch | 5% |
| Read | 18% | ScopeCheck | 5% |
| Edit | 12% | LS | 4% |
| Write | 10% | TodoWrite | 3% |
| Grep | 8% | Task | 2% |
| Glob | 6% | Skill | 2% |
Anti-bias measures: no single tool exceeds 30% of generated calls, every tool gets floor representation, argument templates are deduplicated before sampling, and round-robin selection ensures underrepresented tools get used consistently across the dataset.
16 Offensive Security Scenarios
Each synthetic example follows one of 16 scenario categories, evenly distributed (108–141 examples per category):
| Category | Typical Tool Combo | Category | Typical Tool Combo |
|---|---|---|---|
| Network recon | ScopeCheck + Bash | Credential analysis | Grep + Read |
| Web enumeration | Bash + WebFetch | Documentation | TodoWrite + Write |
| File analysis | Glob + Read + Grep | Scope verification | ScopeCheck + ScopeCheck |
| Exploitation | Write + Bash + Read | Code review | Grep + Read + Read |
| Post-exploitation | Bash + Bash + Read | Lateral movement | Read + Bash |
| Config modification | Read + Edit + Bash | Filesystem discovery | LS + Glob |
| Web recon | WebFetch + WebFetch | Exploit development | Write + Edit |
| Service enumeration | WebFetch + Bash | Payload preparation | Read + Edit |
What pipeline-gemma26b.yml Changes
The Code-Trainer pipeline was built to be model-agnostic — the dataset build, SFT orchestration, GRPO reward function, and GGUF conversion are all parameterized through a YAML config file. Moving to a new base model means writing a new config, not rewriting the pipeline. Here is what changes.
1. Base Model and Tokenizer
The config points at google/gemma-4-26B-A4B-it instead of Qwen/Qwen2.5-Coder-14B-Instruct. Gemma uses a 262K vocabulary (vs Qwen’s 152K), so the tokenizer, embedding layer, and any vocabulary-dependent logic all change. The larger vocabulary should improve tokenization efficiency for code, especially for languages with non-ASCII identifiers.
2. Chat Template
Qwen uses ChatML (<|im_start|> / <|im_end|>). Gemma uses <start_of_turn> / <end_of_turn> markers. The 26B A4B variant supports the native system role directly; the 12B variant does not (system gets merged into the first user message). The dataset build stage applies the correct template through apply_chat_template, so the same source data produces different tokenized outputs depending on which config is active.
3. Tool-Call Format
Gemma 4 has native function-calling support baked into the model. The dataset build maps our existing tool-calling examples to Gemma’s native format rather than the Qwen-style <tool_call> tags. This should be cleaner than V7–V9’s journey of fighting format mismatches — we are starting with the model’s native convention instead of training against it.
4. Sequence Length
Training stays at 4,096 tokens — not the 256K context window. The 52 GB model leaves roughly 19–22 GB of A100 VRAM headroom; longer sequences would eat that with activation memory. The 256K context is the inference budget, not the training budget. Compared to Qwen V9 which also trained at 4,096, the sequences are identical — the model just has 8x more context available at deployment.
5. Quantization Strategy
This is where the Gemma pipeline diverges most visibly from the Qwen pipeline. Q5_K_M was the Qwen default at ~10.5 GB. For Gemma 4 26B, Q5_K_M would be roughly ~18 GB — it does not fit an RTX 5060 Ti 16 GB even with partial offload. Instead:
| Quantization | Size | GPU Layers | Notes |
|---|---|---|---|
| Q4_K_M (primary) | ~14.5 GB | 28/30 on GPU, 2 on CPU | Partial offload, ~15–25 tok/s generation |
| IQ4_XS (fallback) | ~13 GB | All 30 on GPU | Fully resident, ctx_size=4096 |
Google’s QAT Q4_0 GGUF (~14.4 GB) is available as a comparison baseline — our base model on HuggingFace is that checkpoint. After fine-tuning, we quantize the merged adapter weights with llama.cpp’s own quantizer to Q4_K_M and IQ4_XS.
What Stays the Same
The pipeline infrastructure transfers without modification:
| Component | Changes? | Notes |
|---|---|---|
| Dataset slices (A–D) | No | Same source data, different template application |
| Authentic tool-use B+ slice | No | Same R2 session → synthesis pipeline |
| DAPT → SFT → GRPO chain | No | Same three-stage adapter pattern |
| GRPO reward function | Minimal | Validate against Gemma’s native tool format |
| 12-tool Nexus schema | No | Same tool definitions, same validation logic |
| Validation gates | Relaxed | gsm8k gate relaxed to 0.50 (3.8B active vs 14B) |
| GGUF conversion | No | llama.cpp supports Gemma 4 natively |
| Ollama deployment | No | Same Modelfile structure, different base |
| HF Jobs compute | No | Same A100 80 GB instances |
The $67 Pipeline Run
The full Gemma pipeline is an 8-job sequential run on HF Jobs, with validation gates between phases. Total budget: $67.00 ($50.40 allocated + $9.60 contingency + $7.00 reserve).
| Job | Phase | Cost | Time | Description |
|---|---|---|---|---|
| 1 | DAPT | $6.40 | 2.0h | Offsec domain grounding — 10K docs at seq_len=2048 |
| 2 | SFT | $16.00 | 5.0h | Curriculum training — 12K rows, 80/20 split |
| 3 | Validation | $1.60 | 30min | Gate: tool_call ≥ 0.75, gsm8k ≥ 0.50 |
| 4 | GRPO | $9.60 | 3.0h | RL on tool-call formatting — 400 prompts |
| 5 | DPO | $4.80 | 1.5h | Preference learning — 800 pairs, peft_implicit ref |
| 6 | Validation | $1.60 | 30min | Gate: tool_call ≥ 0.80, agent_progress ≥ 4/7 |
| 7 | GGUF | $2.40 | 45min | Merge full chain → Q4_K_M + IQ4_XS |
| 8 | Abliteration | $8.00 | 2.5h | Refusal direction removal (biprojected + obliteratus) |
Cost comparison vs the Qwen pipeline: allocated budget is 1.49x ($50.40 vs $33.92), driven by the 52 GB model download, higher memory-bandwidth demand, and gradient checkpointing overhead. Total budget is 1.34x ($67 vs $50).
DPO: The VRAM Constraint
Standard DPO loads two full model copies — policy and reference — for the preference learning loop. At 52 GB each, that is 104 GB: it exceeds the A100 80 GB. The config uses ref_model_strategy: peft_implicit, where DPOTrainer automatically disables the LoRA adapter for reference forward passes. One model copy, two roles. The Qwen pipeline never hit this constraint because two 28 GB copies fit in 80 GB with room to spare.
Abliteration: Refusal Direction Removal
The final job runs two abliteration techniques — NousResearch’s biprojected method (pure matrix projection, no Optuna search) and obliteratus (4-direction norm-preserving removal) — evaluated against GSM8K and MMLU benchmarks. This removes the model’s built-in refusal behavior for offensive security queries while monitoring for capability degradation. A security-focused coding agent that refuses to discuss exploit code is not useful.
Validation Gates
No phase proceeds unless the previous gate passes. After SFT: the model must pass 75% of 14 tool-call scenarios and score ≥ 0.50 on GSM8K. After RL: the bar tightens to 80% tool-call pass rate and 4/7 agent scenarios solved. If SFT fails, the RL budget goes to contingency retries rather than training on a broken foundation.
Contingency Plan
$9.60 (3 hours of A100 time) is held in reserve for retries. Priority order: re-run SFT with 8K rows instead of 12K, re-run GRPO with 200 prompts instead of 400, re-run DPO with 400 pairs instead of 800, skip abliteration entirely, or drop DAPT to reclaim its $6.40 for other phases. If the SFT gate fails, fallback configs use gentler learning rates (2.5e-5) or reduced LoRA rank (r=16).
What This Aims to Address
Four specific problems that the Gemma pipeline targets:
1. Inference Latency
A coding agent that takes 8 seconds per tool call is a coding agent nobody wants to use. With 3.8B active parameters vs Qwen’s 14B dense, multi-step agent loops should feel interactive rather than sluggish. The same hardware, the same VRAM budget, dramatically less compute per token.
2. Native Multimodal Input
Security auditing involves reading screenshots, analyzing network diagrams, parsing scanned reports. The Qwen pipeline is text-only — visual input requires a separate OCR step or a disconnected vision model. Gemma processes images inline with text, in the same context window, with the same model weights. One prompt can contain a screenshot of a terminal and a question about what is happening in it.
3. Long-Context Agent Scenarios
Complex agent tasks — reading a large file, calling multiple tools, synthesizing results — can easily exceed 32K tokens. Qwen truncates. Gemma’s 256K window means the model can hold an entire multi-file codebase analysis in context without sliding-window tricks or summarization hacks.
4. Built-in Reasoning
Gemma 4 has a configurable thinking mode: the model can reason step-by-step before answering, controlled by the <|think|> token. For the GRPO reward function, this is interesting — we already reward chain-of-thought reasoning before tool calls (has_reasoning_prefix in V10’s reward). Gemma’s built-in thinking mode aligns with that reward signal natively.
What’s Next
The pipeline config is written. The base GGUF is on HuggingFace. The work ahead is execution — eight jobs, two validation gates, and a $67 budget that either produces a locally-deployed multimodal security agent or tells us exactly where the MoE architecture breaks under fine-tuning pressure.
The comparative evaluation is the real deliverable: run V10 Qwen and V1 Gemma head-to-head on tool-call accuracy, code generation quality, and tokens-per-second on identical hardware. The question is not which model wins — it is whether both should coexist in the vLLM stack, hot-swappable for different task profiles. Fast multimodal for triage, dense precision for complex multi-file edits.
Eight posts documented building a fine-tuning pipeline, iterating on one base model until it worked, and then pointing that same pipeline at a fundamentally different architecture. The pipeline does not care whether the model is dense or sparse, text-only or multimodal, 14B or 26B. The config file does.