> ## Documentation Index > Fetch the complete documentation index at: https://docs.monolex.ai/llms.txt > Use this file to discover all available pages before exploring further. # Data Reduction > How MonoTerm achieves 99.95% IPC data reduction through hash-based diffing ## The Problem Traditional terminals send the **entire screen** every time anything changes. ``` You type one character: "a" What changes? ┌─────────────────────────────────────────────────────────────┐ │ Line 1: ################################ (unchanged) │ │ Line 2: ################################ (unchanged) │ │ Line 3: ################################ (unchanged) │ │ ... │ │ Line 39: ################################ (unchanged) │ │ Line 40: $ command_here| (ONE CHAR!) │ └─────────────────────────────────────────────────────────────┘ Traditional terminal sends: ALL 40 lines = ~50,000 bytes Actual change: 1 character = ~25 bytes Wasted: 99.95% of data! ``` *** ## MonoTerm Solution: Hash-Based Diffing Each line gets a unique fingerprint (hash). Only lines with changed hashes are transmitted. ### How It Works Each line in the terminal buffer is assigned a 64-bit hash computed from its content: ``` Line Content Hash ─────────── ──── "Hello World" -> 0xA3F2B71C "$ ls -la" -> 0x8C4D91E5 "file.txt 1024" -> 0x2B7A03F9 ``` When comparing frames, MonoTerm only compares hashes: | Old Hash | New Hash | Result | | ---------- | ---------- | ----------------- | | 0xA3F2B71C | 0xA3F2B71C | SAME (skip!) | | 0x8C4D91E5 | 0x8C4D91E5 | SAME (skip!) | | 0x2B7A03F9 | 0x5E1C82A4 | DIFFERENT (send!) | **Only lines where hash changed get transmitted.** *** ## Hash Algorithm: FNV-1a MonoTerm uses **FNV-1a** (Fowler-Noll-Vo), a fast non-cryptographic hash optimized for short strings like terminal lines. ``` ┌───────────────────────────────────────────────────────────────────────┐ │ LINE FINGERPRINTING │ ├───────────────────────────────────────────────────────────────────────┤ │ │ │ Each line becomes a unique 64-bit fingerprint: │ │ │ │ "$ ls -la" + [green] + [bold] │ │ │ │ │ ▼ │ │ ┌─────────────────────────────────────────────────────────────────┐ │ │ │ FNV-1a Hash │ │ │ │ ───────────────────────────────── │ │ │ │ • Combines: characters + colors + attributes │ │ │ │ • Fast: ~5 GB/s on modern CPUs │ │ │ │ • Low collision probability (1 in 18 quintillion) │ │ │ └─────────────────────────────────────────────────────────────────┘ │ │ │ │ │ ▼ │ │ OUTPUT: 0x8C4D91E52B7A03F9 (64-bit fingerprint) │ │ │ │ Same content → Same fingerprint (always!) │ │ Different content → Different fingerprint │ │ │ └───────────────────────────────────────────────────────────────────────┘ ``` ### Why FNV-1a? | Algorithm | Verdict | Reason | | --------- | -------- | ----------------------------------------------- | | FNV-1a | **Used** | Extremely fast, simple, great for short strings | | SHA-256 | Not used | Too slow, cryptographic overkill | | xxHash | Not used | Fast but more complex than needed | | CRC32 | Not used | Too many collisions | **Key Benefits:** * Same line = Same fingerprint (deterministic) * Fingerprint comparison: **Single CPU instruction** (O(1)) * No character-by-character comparison needed! *** ## Line Fingerprint System MonoTerm maintains a fingerprint for each line, enabling instant change detection. ``` ┌───────────────────────────────────────────────────────────────────────┐ │ HOW LINE FINGERPRINTS WORK │ ├───────────────────────────────────────────────────────────────────────┤ │ │ │ Each line has a unique fingerprint based on its content: │ │ │ │ ┌─────────────────────────────────────────────────────────────────┐ │ │ │ Line: "$ ls -la" (green, bold) │ │ │ │ │ │ │ │ │ ▼ │ │ │ │ Fingerprint: 0x8C4D91E5 │ │ │ └─────────────────────────────────────────────────────────────────┘ │ │ │ │ What's included in fingerprint: │ │ │ │ ✓ Characters ("$ ls -la") │ │ ✓ Foreground color (green) │ │ ✓ Background color (default) │ │ ✓ Text style (bold, underline, etc.) │ │ │ │ Comparison speed: │ │ │ │ Traditional: Compare 120 characters = 120 operations │ │ MonoTerm: Compare 1 fingerprint = 1 operation │ │ │ │ Speedup: 120x faster per line! │ │ │ └───────────────────────────────────────────────────────────────────────┘ ``` *** ## DiffHint Modes MonoTerm has four modes based on what changed: **Scenario:** Only cursor moved, no content changed * Data sent: Cursor position only (\~10 bytes) * Reduction: **99.98%** * Action: No buffer update, no render **Scenario:** 1-3 lines changed (normal typing) - **Most Common** * Data sent: Only dirty rows + cursor (\~25-100 bytes) * Reduction: **99.95%** * Action: Inject only changed lines, render only those rows ``` [skip] [skip] [skip] ... [SEND] [skip] [skip] ^ | Only changed line ``` **Scenario:** Major screen change (clear screen, scrolling, resize) * Data sent: Bottom 2V lines (\~100,000 bytes for 40-row viewport) * Reduction: 0% (but optimal for big changes) * Triggers: * Buffer size changed (resize) * History grew significantly (fast scrolling) * Why: Fingerprint comparison would miss changes **Scenario:** Invalid frame (incomplete, corrupted) * Data sent: Nothing * Action: Wait for next valid frame * Used for: Glitch protection (ybase oscillation) *** ## Smart Comparison Range (2V) MonoTerm only compares the **bottom 2V lines** (2× viewport height) for change detection, not the entire buffer. ### Why 2V Range? ``` ┌───────────────────────────────────────────────────────────────────────┐ │ THE 2V COMPARISON WINDOW │ ├───────────────────────────────────────────────────────────────────────┤ │ │ │ Terminal buffer can have 100,000+ lines of history │ │ Comparing ALL lines would be too slow │ │ │ │ Solution: Only compare bottom 2V (2 × viewport height) │ │ │ │ ┌─────────────────────────────────────────────────────┐ │ │ │ Old history (not compared) │ │ │ │ ...thousands of lines... │ │ │ ├─────────────────────────────────────────────────────┤ ← 2V start │ │ │ ╔═══════════════════════════════════════════════╗ │ │ │ │ ║ Recent lines (compared for changes) ║ │ ← V lines │ │ │ ╠═══════════════════════════════════════════════╣ │ │ │ │ ║ Viewport (what you see) ║ │ ← V lines │ │ │ ╚═══════════════════════════════════════════════╝ │ │ │ └─────────────────────────────────────────────────────┘ │ │ │ │ For V=40 rows: Compare only bottom 80 lines │ │ Instead of: 100,000 lines │ │ │ │ Result: Fast comparison even with huge history! │ │ │ └───────────────────────────────────────────────────────────────────────┘ ``` ### When Full Mode Triggers | Condition | Why Full Mode? | | ---------------------- | ----------------------------------- | | Buffer size changed | Resize invalidates old fingerprints | | History grew > V lines | Old 2V window no longer aligns | | First frame | No previous fingerprints to compare | ### The Safety Rule: Why Exactly 2V? ``` ╔═══════════════════════════════════════════════════════════════════════════════╗ ║ THE 2V SAFETY RULE ║ ╠═══════════════════════════════════════════════════════════════════════════════╣ ║ ║ ║ MonoTerm uses TWO related settings: ║ ║ ║ ║ ┌─────────────────────────────────────────────────────────────────────────┐ ║ ║ │ │ ║ ║ │ COMPARISON WINDOW = 2V (bottom 80 lines for 40-row viewport) │ ║ ║ │ ─────────────────────────────────────────────────────────── │ ║ ║ │ This is HOW MANY lines we check for changes │ ║ ║ │ │ ║ ║ │ FULL MODE TRIGGER = V (history changes more than 40 lines) │ ║ ║ │ ─────────────────────────────────────────────────────────── │ ║ ║ │ This is WHEN we give up and refresh everything │ ║ ║ │ │ ║ ║ └─────────────────────────────────────────────────────────────────────────┘ ║ ║ ║ ║ THE SAFETY RULE: ║ ║ ┌─────────────────────────────────────────────────────────────────────────┐ ║ ║ │ │ ║ ║ │ Full Mode Trigger < Comparison Window │ ║ ║ │ (V) < (2V) │ ║ ║ │ │ ║ ║ │ This ensures we ALWAYS catch all changes! │ ║ ║ │ │ ║ ║ └─────────────────────────────────────────────────────────────────────────┘ ║ ║ ║ ╠═══════════════════════════════════════════════════════════════════════════════╣ ║ ║ ║ WHY THIS MATTERS (Visual Proof): ║ ║ ───────────────────────────────── ║ ║ ║ ║ Imagine history grows by 30 lines (less than V=40): ║ ║ ║ ║ BEFORE AFTER ║ ║ ┌────────────────────────┐ ┌────────────────────────┐ ║ ║ │ old history │ │ old history │ ║ ║ │ (not compared) │ │ + 30 new lines │ ║ ║ │ │ │ (not compared) │ ║ ║ ├────────────────────────┤ ├────────────────────────┤ ║ ║ │ ╔══════════════════╗ │ │ ╔══════════════════╗ │ ║ ║ │ ║ 2V WINDOW ║ │ ───► │ ║ 2V WINDOW ║ │ ║ ║ │ ║ (80 lines) ║ │ │ ║ (80 lines) ║ │ ║ ║ │ ║ ┌────────────┐ ║ │ │ ║ ┌────────────┐ ║ │ ║ ║ │ ║ │ VIEWPORT │ ║ │ │ ║ │ VIEWPORT │ ║ │ ║ ║ │ ║ │ (40 lines) │ ║ │ │ ║ │ (40 lines) │ ║ │ ║ ║ │ ║ └────────────┘ ║ │ │ ║ └────────────┘ ║ │ ║ ║ │ ╚══════════════════╝ │ │ ╚══════════════════╝ │ ║ ║ └────────────────────────┘ └────────────────────────┘ ║ ║ ║ ║ ✓ 30 < 40, so we stay in Partial mode ║ ║ ✓ 2V window (80 lines) covers the 30-line shift ║ ║ ✓ All changes are detected! ║ ║ ║ ╠═══════════════════════════════════════════════════════════════════════════════╣ ║ ║ ║ What if history grows by 50 lines (more than V=40)? ║ ║ ║ ║ BEFORE AFTER ║ ║ ┌────────────────────────┐ ┌────────────────────────┐ ║ ║ │ old history │ │ old history │ ║ ║ │ │ │ + 50 new lines │ ║ ║ │ │ │ │ ║ ║ ├────────────────────────┤ ├────────────────────────┤ ║ ║ │ ╔══════════════════╗ │ │ ╔══════════════════╗│ ║ ║ │ ║ 2V WINDOW ║ │ ───► │ ║ 2V WINDOW ║│ ║ ║ │ ║ OLD POSITION ║ │ │ ║ NEW POSITION ║│ ← SHIFTED! ║ ║ │ ╚══════════════════╝ │ │ ╚══════════════════╝│ ║ ║ └────────────────────────┘ └────────────────────────┘ ║ ║ ║ ║ ✗ 50 > 40, windows don't overlap properly ║ ║ ✗ Fingerprint comparison might miss changes ║ ║ → SOLUTION: Switch to Full mode (refresh everything) ║ ║ ║ ╚═══════════════════════════════════════════════════════════════════════════════╝ ``` The 2V window ensures that even when content shifts, we can still detect all changes. When shifts exceed our detection range, we automatically switch to Full mode for safety. *** ## Change Detection Flow ``` ┌───────────────────────────────────────────────────────────────────────┐ │ HOW MONOTERM DETECTS CHANGES │ ├───────────────────────────────────────────────────────────────────────┤ │ │ │ STEP 1: Compare fingerprints in 2V range │ │ ───────────────────────────────────────── │ │ │ │ Old New │ │ ┌─────────────┐ ┌─────────────┐ │ │ │ Line 1: A │ ═══ │ Line 1: A │ → SAME │ │ │ Line 2: B │ ═══ │ Line 2: B │ → SAME │ │ │ Line 3: C │ ≠≠≠ │ Line 3: X │ → CHANGED! │ │ │ Line 4: D │ ═══ │ Line 4: D │ → SAME │ │ └─────────────┘ └─────────────┘ │ │ │ │ STEP 2: Decide mode │ │ ─────────────────── │ │ │ │ ┌─────────────────────────────────────────────────────┐ │ │ │ Buffer size changed? ──────────────→ FULL │ │ │ │ History grew > V? ──────────────→ FULL │ │ │ │ No changes? ──────────────→ NONE │ │ │ │ Some changes? ──────────────→ PARTIAL │ │ │ └─────────────────────────────────────────────────────┘ │ │ │ │ STEP 3: Send only what's needed │ │ ─────────────────────────────── │ │ │ │ PARTIAL: Send only changed lines (Line 3 in example) │ │ FULL: Send all lines in 2V range │ │ NONE: Send nothing (cursor update only) │ │ │ └───────────────────────────────────────────────────────────────────────┘ ``` **Speed:** * 80 fingerprint comparisons (for 40-row viewport) * Each comparison: single CPU instruction * Total: microseconds per frame *** ## Performance Comparison ### Hash Comparison vs Character-by-Character ``` Line 1: "Hello World" Line 2: "Hello World" Compare: H==H, e==e, l==l, l==l, o==o, ' '==' ', ... Operations: O(n) where n = line length For 120-char line: 120 comparisons For 40 lines: 4,800 comparisons per frame ``` ``` Line 1 hash: 0xA3F2B71C Line 2 hash: 0xA3F2B71C Compare: 0xA3F2B71C == 0xA3F2B71C (single CPU instruction) Operations: O(1) - just one comparison! For 40 lines: 40 comparisons per frame ``` **Speedup: 4,800 / 40 = 120x faster comparison!** *** ## Real-World Examples You type: `ls -la` | Frame | Content | Mode | Data Sent | | ----- | -------- | ------- | ------------- | | 1 | `l` | Partial | \~1,254 bytes | | 2 | `ls` | Partial | \~1,254 bytes | | 3 | `ls ` | Partial | \~1,254 bytes | | 4 | `ls -` | Partial | \~1,254 bytes | | 5 | `ls -l` | Partial | \~1,254 bytes | | 6 | `ls -la` | Partial | \~1,254 bytes | * **Total sent:** 6 x 1,254 = 7,524 bytes * **Traditional:** 6 x 50,000 = 300,000 bytes * **Reduction:** 97.5% You run: `ls -la` in a directory with 10 files * Output: 10 lines of file listing * Dirty rows: 10 (new lines) + 1 (prompt moved) = 11 rows * Mode: **Partial** (11 \< 20 threshold) * Data sent: 11 x 1,254 = \~13,794 bytes * Traditional: 50,000 bytes * **Reduction:** 72.4% You run: `clear` * All 40 rows change (cleared + new prompt) * Dirty rows: 40 (100%) * Mode: **Full** (40 > 20 threshold) * Data sent: 50,000 bytes This is correct! Full mode is more efficient when everything changed. *** ## True Partial Mode MonoTerm sends only the changed lines, not just renders them partially. ### Before vs After ``` ┌────────────────────────────────────────────────────────────┐ │ Rust Backend: Partial mode detected │ │ BUT sends ALL 1040 lines anyway │ │ Data: ~50KB │ │ │ │ Frontend: Creates new buffer (1080 lines) │ │ Copies ALL lines │ │ Refreshes only changed rows │ │ │ │ Problem: 50KB transferred for 2 changed lines! │ └────────────────────────────────────────────────────────────┘ ``` ``` ┌────────────────────────────────────────────────────────────┐ │ Rust Backend: Partial mode detected │ │ Sends ONLY 2 changed lines │ │ Data: ~0.5KB │ │ │ │ Frontend: Reuses existing buffer │ │ Updates only 2 lines │ │ Refreshes only changed rows │ │ │ │ Result: 99% less data, no memory allocation! │ └────────────────────────────────────────────────────────────┘ ``` ### Results | Metric | Stage 1 | Stage 2 | | ----------------- | -------- | ---------- | | IPC data | 50KB | 0.5KB | | Reduction | - | **99.95%** | | Buffer allocation | Recreate | Reused | | GC pressure | High | Eliminated | *** ## Frontend: Buffer Reuse ``` ┌───────────────────────────────────────────────────────────────────────┐ │ BUFFER REUSE DECISION LOGIC │ ├───────────────────────────────────────────────────────────────────────┤ │ │ │ INPUT: diffHint, linesObj, neededLength, update │ │ │ │ CHECKS: │ │ ──────── │ │ isPartialMode = diffHint has 'Partial' key │ │ isNoneMode = diffHint === 'None' │ │ bufferSizeMatches = linesObj._length === neededLength │ │ │ │ DECISION: │ │ ────────── │ │ (isPartialMode || isNoneMode) && bufferSizeMatches? │ │ │ │ │ YES ─┴──► PARTIAL/NONE MODE: Reuse existing buffer │ │ │ │ │ └──► if (update.lines.length > 0): │ │ injectLines(linesObj._array, update.lines) │ │ │ │ NO ─────► FULL MODE: Recreate entire buffer │ │ │ │ │ ├──► newArray = new Array(neededLength) │ │ │ │ │ ├──► for each index: newArray[i] = new BufferLine(cols) │ │ │ │ │ ├──► linesObj._array = newArray │ │ │ │ │ └──► injectLines(newArray, update.lines) │ │ │ └───────────────────────────────────────────────────────────────────────┘ ``` *** ## ACK Gate: Flow Control The ACK mechanism prevents buffer overflow by ensuring the frontend processes each update before receiving the next. ``` Backend Frontend │ │ │ ── GridUpdate (epoch=5) ────→ │ │ [waiting_ack = true] │ │ │ │ ←──────── ACK (epoch=5) ────── │ │ [waiting_ack = false] │ │ │ │ ── GridUpdate (epoch=6) ────→ │ │ ... │ ``` *** ## Performance Summary ### By Scenario | Scenario | DiffHint | IPC Data | Buffer | Render | | ------------ | -------- | -------- | --------- | ------ | | Typing | Partial | \~0.05KB | Reused | 2.5% | | `ls` | Full | \~50KB | Recreated | 100% | | Cursor blink | None | \~0.1KB | Reused | 0% | | vim scroll | Full | \~50KB | Recreated | 100% | ### Overall Reduction | Activity | Reduction | | ------------------- | ----------------------------- | | Normal typing | **99.95%** (50KB -> 25 bytes) | | Command output | 70-90% | | Screen clear | 0% (optimal for that case) | | **Overall average** | **90%+** | *** ## Summary * **FNV-1a** hash for line fingerprinting * **2V range** comparison (bottom 80 lines) * **O(1)** fingerprint comparison * **DiffHint** modes: None, Partial, Full, Skip * Smart Full mode trigger (buffer/history changes) * **99.95%** data reduction (50KB → 25 bytes) * Faster IPC (less data to transfer) * Less CPU usage (less data to process) * Better for remote connections * Lower memory/GC pressure