Data Reduction - Monolex Docs

The Problem

Traditional terminals send the entire screen every time anything changes.

You type one character: "a"

What changes?
┌─────────────────────────────────────────────────────────────┐
│  Line 1:  ################################  (unchanged)     │
│  Line 2:  ################################  (unchanged)     │
│  Line 3:  ################################  (unchanged)     │
│  ...                                                        │
│  Line 39: ################################  (unchanged)     │
│  Line 40: $ command_here|                   (ONE CHAR!)     │
└─────────────────────────────────────────────────────────────┘

Traditional terminal sends: ALL 40 lines = ~50,000 bytes
Actual change: 1 character = ~25 bytes

Wasted: 99.95% of data!

MonoTerm Solution: Hash-Based Diffing

Each line gets a unique fingerprint (hash). Only lines with changed hashes are transmitted.

How It Works

Each line in the terminal buffer is assigned a 64-bit hash computed from its content:

Line Content                    Hash
───────────                     ────
"Hello World"             ->    0xA3F2B71C
"$ ls -la"                ->    0x8C4D91E5
"file.txt  1024"          ->    0x2B7A03F9

When comparing frames, MonoTerm only compares hashes:

Old Hash	New Hash	Result
0xA3F2B71C	0xA3F2B71C	SAME (skip!)
0x8C4D91E5	0x8C4D91E5	SAME (skip!)
0x2B7A03F9	0x5E1C82A4	DIFFERENT (send!)

Only lines where hash changed get transmitted.

Hash Algorithm: FNV-1a

MonoTerm uses FNV-1a (Fowler-Noll-Vo), a fast non-cryptographic hash optimized for short strings like terminal lines.

┌───────────────────────────────────────────────────────────────────────┐
│  LINE FINGERPRINTING                                                  │
├───────────────────────────────────────────────────────────────────────┤
│                                                                       │
│  Each line becomes a unique 64-bit fingerprint:                       │
│                                                                       │
│  "$ ls -la" + [green] + [bold]                                        │
│           │                                                           │
│           ▼                                                           │
│  ┌─────────────────────────────────────────────────────────────────┐  │
│  │                    FNV-1a Hash                                  │  │
│  │   ─────────────────────────────────                             │  │
│  │   • Combines: characters + colors + attributes                  │  │
│  │   • Fast: ~5 GB/s on modern CPUs                                │  │
│  │   • Low collision probability (1 in 18 quintillion)             │  │
│  └─────────────────────────────────────────────────────────────────┘  │
│           │                                                           │
│           ▼                                                           │
│  OUTPUT: 0x8C4D91E52B7A03F9  (64-bit fingerprint)                     │
│                                                                       │
│  Same content → Same fingerprint (always!)                            │
│  Different content → Different fingerprint                            │
│                                                                       │
└───────────────────────────────────────────────────────────────────────┘

Why FNV-1a?

Algorithm	Verdict	Reason
FNV-1a	Used	Extremely fast, simple, great for short strings
SHA-256	Not used	Too slow, cryptographic overkill
xxHash	Not used	Fast but more complex than needed
CRC32	Not used	Too many collisions

Key Benefits:

Same line = Same fingerprint (deterministic)
Fingerprint comparison: Single CPU instruction (O(1))
No character-by-character comparison needed!

Line Fingerprint System

MonoTerm maintains a fingerprint for each line, enabling instant change detection.

┌───────────────────────────────────────────────────────────────────────┐
│  HOW LINE FINGERPRINTS WORK                                           │
├───────────────────────────────────────────────────────────────────────┤
│                                                                       │
│  Each line has a unique fingerprint based on its content:             │
│                                                                       │
│  ┌─────────────────────────────────────────────────────────────────┐  │
│  │  Line: "$ ls -la"  (green, bold)                                │  │
│  │                         │                                       │  │
│  │                         ▼                                       │  │
│  │  Fingerprint: 0x8C4D91E5                                        │  │
│  └─────────────────────────────────────────────────────────────────┘  │
│                                                                       │
│  What's included in fingerprint:                                      │
│                                                                       │
│     ✓ Characters ("$ ls -la")                                         │
│     ✓ Foreground color (green)                                        │
│     ✓ Background color (default)                                      │
│     ✓ Text style (bold, underline, etc.)                              │
│                                                                       │
│  Comparison speed:                                                    │
│                                                                       │
│     Traditional: Compare 120 characters = 120 operations              │
│     MonoTerm:    Compare 1 fingerprint = 1 operation                  │
│                                                                       │
│     Speedup: 120x faster per line!                                    │
│                                                                       │
└───────────────────────────────────────────────────────────────────────┘

DiffHint Modes

MonoTerm has four modes based on what changed:

None
Partial
Full
Skip

Scenario: Only cursor moved, no content changed

Data sent: Cursor position only (~10 bytes)
Reduction: 99.98%
Action: No buffer update, no render

Scenario: 1-3 lines changed (normal typing) - Most Common

Data sent: Only dirty rows + cursor (~25-100 bytes)
Reduction: 99.95%
Action: Inject only changed lines, render only those rows

[skip] [skip] [skip] ... [SEND] [skip] [skip]
                          ^
                          |
                    Only changed line

Smart Comparison Range (2V)

MonoTerm only compares the bottom 2V lines (2× viewport height) for change detection, not the entire buffer.

Why 2V Range?

┌───────────────────────────────────────────────────────────────────────┐
│  THE 2V COMPARISON WINDOW                                             │
├───────────────────────────────────────────────────────────────────────┤
│                                                                       │
│  Terminal buffer can have 100,000+ lines of history                   │
│  Comparing ALL lines would be too slow                                │
│                                                                       │
│  Solution: Only compare bottom 2V (2 × viewport height)               │
│                                                                       │
│  ┌─────────────────────────────────────────────────────┐              │
│  │  Old history (not compared)                         │              │
│  │  ...thousands of lines...                           │              │
│  ├─────────────────────────────────────────────────────┤ ← 2V start   │
│  │  ╔═══════════════════════════════════════════════╗  │              │
│  │  ║  Recent lines (compared for changes)          ║  │ ← V lines    │
│  │  ╠═══════════════════════════════════════════════╣  │              │
│  │  ║  Viewport (what you see)                      ║  │ ← V lines    │
│  │  ╚═══════════════════════════════════════════════╝  │              │
│  └─────────────────────────────────────────────────────┘              │
│                                                                       │
│  For V=40 rows: Compare only bottom 80 lines                          │
│  Instead of: 100,000 lines                                            │
│                                                                       │
│  Result: Fast comparison even with huge history!                      │
│                                                                       │
└───────────────────────────────────────────────────────────────────────┘

When Full Mode Triggers

Condition	Why Full Mode?
Buffer size changed	Resize invalidates old fingerprints
History grew > V lines	Old 2V window no longer aligns
First frame	No previous fingerprints to compare

The Safety Rule: Why Exactly 2V?

╔═══════════════════════════════════════════════════════════════════════════════╗
║  THE 2V SAFETY RULE                                                           ║
╠═══════════════════════════════════════════════════════════════════════════════╣
║                                                                               ║
║  MonoTerm uses TWO related settings:                                          ║
║                                                                               ║
║  ┌─────────────────────────────────────────────────────────────────────────┐  ║
║  │                                                                         │  ║
║  │  COMPARISON WINDOW = 2V (bottom 80 lines for 40-row viewport)           │  ║
║  │  ───────────────────────────────────────────────────────────            │  ║
║  │  This is HOW MANY lines we check for changes                            │  ║
║  │                                                                         │  ║
║  │  FULL MODE TRIGGER = V (history changes more than 40 lines)             │  ║
║  │  ───────────────────────────────────────────────────────────            │  ║
║  │  This is WHEN we give up and refresh everything                         │  ║
║  │                                                                         │  ║
║  └─────────────────────────────────────────────────────────────────────────┘  ║
║                                                                               ║
║  THE SAFETY RULE:                                                             ║
║  ┌─────────────────────────────────────────────────────────────────────────┐  ║
║  │                                                                         │  ║
║  │         Full Mode Trigger  <  Comparison Window                         │  ║
║  │                  (V)       <        (2V)                                │  ║
║  │                                                                         │  ║
║  │  This ensures we ALWAYS catch all changes!                              │  ║
║  │                                                                         │  ║
║  └─────────────────────────────────────────────────────────────────────────┘  ║
║                                                                               ║
╠═══════════════════════════════════════════════════════════════════════════════╣
║                                                                               ║
║  WHY THIS MATTERS (Visual Proof):                                             ║
║  ─────────────────────────────────                                            ║
║                                                                               ║
║  Imagine history grows by 30 lines (less than V=40):                          ║
║                                                                               ║
║  BEFORE                              AFTER                                    ║
║  ┌────────────────────────┐         ┌────────────────────────┐                ║
║  │ old history            │         │ old history            │                ║
║  │ (not compared)         │         │ + 30 new lines         │                ║
║  │                        │         │ (not compared)         │                ║
║  ├────────────────────────┤         ├────────────────────────┤                ║
║  │ ╔══════════════════╗   │         │ ╔══════════════════╗   │                ║
║  │ ║  2V WINDOW       ║   │  ───►   │ ║  2V WINDOW       ║   │                ║
║  │ ║  (80 lines)      ║   │         │ ║  (80 lines)      ║   │                ║
║  │ ║  ┌────────────┐  ║   │         │ ║  ┌────────────┐  ║   │                ║
║  │ ║  │ VIEWPORT   │  ║   │         │ ║  │ VIEWPORT   │  ║   │                ║
║  │ ║  │ (40 lines) │  ║   │         │ ║  │ (40 lines) │  ║   │                ║
║  │ ║  └────────────┘  ║   │         │ ║  └────────────┘  ║   │                ║
║  │ ╚══════════════════╝   │         │ ╚══════════════════╝   │                ║
║  └────────────────────────┘         └────────────────────────┘                ║
║                                                                               ║
║  ✓ 30 < 40, so we stay in Partial mode                                        ║
║  ✓ 2V window (80 lines) covers the 30-line shift                              ║
║  ✓ All changes are detected!                                                  ║
║                                                                               ║
╠═══════════════════════════════════════════════════════════════════════════════╣
║                                                                               ║
║  What if history grows by 50 lines (more than V=40)?                          ║
║                                                                               ║
║  BEFORE                              AFTER                                    ║
║  ┌────────────────────────┐         ┌────────────────────────┐                ║
║  │ old history            │         │ old history            │                ║
║  │                        │         │ + 50 new lines         │                ║
║  │                        │         │                        │                ║
║  ├────────────────────────┤         ├────────────────────────┤                ║
║  │ ╔══════════════════╗   │         │    ╔══════════════════╗│                ║
║  │ ║  2V WINDOW       ║   │  ───►   │    ║  2V WINDOW       ║│                ║
║  │ ║  OLD POSITION    ║   │         │    ║  NEW POSITION    ║│  ← SHIFTED!    ║
║  │ ╚══════════════════╝   │         │    ╚══════════════════╝│                ║
║  └────────────────────────┘         └────────────────────────┘                ║
║                                                                               ║
║  ✗ 50 > 40, windows don't overlap properly                                    ║
║  ✗ Fingerprint comparison might miss changes                                  ║
║  → SOLUTION: Switch to Full mode (refresh everything)                         ║
║                                                                               ║
╚═══════════════════════════════════════════════════════════════════════════════╝

The 2V window ensures that even when content shifts, we can still detect all changes. When shifts exceed our detection range, we automatically switch to Full mode for safety.

Change Detection Flow

┌───────────────────────────────────────────────────────────────────────┐
│  HOW MONOTERM DETECTS CHANGES                                         │
├───────────────────────────────────────────────────────────────────────┤
│                                                                       │
│  STEP 1: Compare fingerprints in 2V range                             │
│  ─────────────────────────────────────────                            │
│                                                                       │
│  Old                    New                                           │
│  ┌─────────────┐       ┌─────────────┐                                │
│  │ Line 1: A   │  ═══  │ Line 1: A   │  → SAME                        │
│  │ Line 2: B   │  ═══  │ Line 2: B   │  → SAME                        │
│  │ Line 3: C   │  ≠≠≠  │ Line 3: X   │  → CHANGED!                    │
│  │ Line 4: D   │  ═══  │ Line 4: D   │  → SAME                        │
│  └─────────────┘       └─────────────┘                                │
│                                                                       │
│  STEP 2: Decide mode                                                  │
│  ───────────────────                                                  │
│                                                                       │
│       ┌─────────────────────────────────────────────────────┐         │
│       │  Buffer size changed?  ──────────────→  FULL        │         │
│       │  History grew > V?     ──────────────→  FULL        │         │
│       │  No changes?           ──────────────→  NONE        │         │
│       │  Some changes?         ──────────────→  PARTIAL     │         │
│       └─────────────────────────────────────────────────────┘         │
│                                                                       │
│  STEP 3: Send only what's needed                                      │
│  ───────────────────────────────                                      │
│                                                                       │
│       PARTIAL: Send only changed lines (Line 3 in example)            │
│       FULL: Send all lines in 2V range                                │
│       NONE: Send nothing (cursor update only)                         │
│                                                                       │
└───────────────────────────────────────────────────────────────────────┘

Speed:

80 fingerprint comparisons (for 40-row viewport)
Each comparison: single CPU instruction
Total: microseconds per frame

Performance Comparison

Hash Comparison vs Character-by-Character

Traditional
MonoTerm

Line 1: "Hello World"
Line 2: "Hello World"

Compare: H==H, e==e, l==l, l==l, o==o, ' '==' ', ...

Operations: O(n) where n = line length
For 120-char line: 120 comparisons
For 40 lines: 4,800 comparisons per frame

Line 1 hash: 0xA3F2B71C
Line 2 hash: 0xA3F2B71C

Compare: 0xA3F2B71C == 0xA3F2B71C (single CPU instruction)

Operations: O(1) - just one comparison!
For 40 lines: 40 comparisons per frame

Speedup: 4,800 / 40 = 120x faster comparison!

Real-World Examples

Scenario 1: Normal Typing

You type: ls -la

Frame	Content	Mode	Data Sent
1	`l`	Partial	~1,254 bytes
2	`ls`	Partial	~1,254 bytes
3	`ls`	Partial	~1,254 bytes
4	`ls -`	Partial	~1,254 bytes
5	`ls -l`	Partial	~1,254 bytes
6	`ls -la`	Partial	~1,254 bytes

Total sent: 6 x 1,254 = 7,524 bytes
Traditional: 6 x 50,000 = 300,000 bytes
Reduction: 97.5%

Scenario 2: Command Output

You run: ls -la in a directory with 10 files

Output: 10 lines of file listing
Dirty rows: 10 (new lines) + 1 (prompt moved) = 11 rows
Mode: Partial (11 < 20 threshold)
Data sent: 11 x 1,254 = ~13,794 bytes
Traditional: 50,000 bytes
Reduction: 72.4%

Scenario 3: Clear Screen

You run: clear

All 40 rows change (cleared + new prompt)
Dirty rows: 40 (100%)
Mode: Full (40 > 20 threshold)
Data sent: 50,000 bytes

This is correct! Full mode is more efficient when everything changed.

True Partial Mode

MonoTerm sends only the changed lines, not just renders them partially.

Before vs After

Before (Render Only Partial)
After (True Partial)

┌────────────────────────────────────────────────────────────┐
│  Rust Backend:   Partial mode detected                     │
│                  BUT sends ALL 1040 lines anyway           │
│                  Data: ~50KB                               │
│                                                            │
│  Frontend:       Creates new buffer (1080 lines)           │
│                  Copies ALL lines                          │
│                  Refreshes only changed rows               │
│                                                            │
│  Problem: 50KB transferred for 2 changed lines!            │
└────────────────────────────────────────────────────────────┘

┌────────────────────────────────────────────────────────────┐
│  Rust Backend:   Partial mode detected                     │
│                  Sends ONLY 2 changed lines                │
│                  Data: ~0.5KB                              │
│                                                            │
│  Frontend:       Reuses existing buffer                    │
│                  Updates only 2 lines                      │
│                  Refreshes only changed rows               │
│                                                            │
│  Result: 99% less data, no memory allocation!              │
└────────────────────────────────────────────────────────────┘

Results

Metric	Stage 1	Stage 2
IPC data	50KB	0.5KB
Reduction	-	99.95%
Buffer allocation	Recreate	Reused
GC pressure	High	Eliminated

Frontend: Buffer Reuse

┌───────────────────────────────────────────────────────────────────────┐
│  BUFFER REUSE DECISION LOGIC                                          │
├───────────────────────────────────────────────────────────────────────┤
│                                                                       │
│  INPUT: diffHint, linesObj, neededLength, update                      │
│                                                                       │
│  CHECKS:                                                              │
│  ────────                                                             │
│  isPartialMode = diffHint has 'Partial' key                           │
│  isNoneMode    = diffHint === 'None'                                  │
│  bufferSizeMatches = linesObj._length === neededLength                │
│                                                                       │
│  DECISION:                                                            │
│  ──────────                                                           │
│  (isPartialMode || isNoneMode) && bufferSizeMatches?                  │
│       │                                                               │
│  YES ─┴──► PARTIAL/NONE MODE: Reuse existing buffer                   │
│            │                                                          │
│            └──► if (update.lines.length > 0):                         │
│                      injectLines(linesObj._array, update.lines)       │
│                                                                       │
│  NO ─────► FULL MODE: Recreate entire buffer                          │
│            │                                                          │
│            ├──► newArray = new Array(neededLength)                    │
│            │                                                          │
│            ├──► for each index: newArray[i] = new BufferLine(cols)    │
│            │                                                          │
│            ├──► linesObj._array = newArray                            │
│            │                                                          │
│            └──► injectLines(newArray, update.lines)                   │
│                                                                       │
└───────────────────────────────────────────────────────────────────────┘

ACK Gate: Flow Control

The ACK mechanism prevents buffer overflow by ensuring the frontend processes each update before receiving the next.

Backend                          Frontend
   │                                │
   │ ── GridUpdate (epoch=5) ────→  │
   │    [waiting_ack = true]        │
   │                                │
   │ ←──────── ACK (epoch=5) ────── │
   │    [waiting_ack = false]       │
   │                                │
   │ ── GridUpdate (epoch=6) ────→  │
   │    ...                         │

Performance Summary

By Scenario

Scenario	DiffHint	IPC Data	Buffer	Render
Typing	Partial	~0.05KB	Reused	2.5%
`ls`	Full	~50KB	Recreated	100%
Cursor blink	None	~0.1KB	Reused	0%
vim scroll	Full	~50KB	Recreated	100%

Overall Reduction

Activity	Reduction
Normal typing	99.95% (50KB -> 25 bytes)
Command output	70-90%
Screen clear	0% (optimal for that case)
Overall average	90%+

Summary

Technology

FNV-1a hash for line fingerprinting
2V range comparison (bottom 80 lines)
O(1) fingerprint comparison
DiffHint modes: None, Partial, Full, Skip
Smart Full mode trigger (buffer/history changes)

Benefits

99.95% data reduction (50KB → 25 bytes)
Faster IPC (less data to transfer)
Less CPU usage (less data to process)
Better for remote connections
Lower memory/GC pressure

Documentation Index

​The Problem

​MonoTerm Solution: Hash-Based Diffing

​How It Works

​Hash Algorithm: FNV-1a

​Why FNV-1a?

​Line Fingerprint System

​DiffHint Modes

​Smart Comparison Range (2V)

​Why 2V Range?

​When Full Mode Triggers

​The Safety Rule: Why Exactly 2V?

​Change Detection Flow

​Performance Comparison

​Hash Comparison vs Character-by-Character

​Real-World Examples

​True Partial Mode

​Before vs After

​Results

​Frontend: Buffer Reuse

​ACK Gate: Flow Control

​Performance Summary

​By Scenario

​Overall Reduction

​Summary

Technology

Benefits

The Problem

MonoTerm Solution: Hash-Based Diffing

How It Works

Hash Algorithm: FNV-1a

Why FNV-1a?

Line Fingerprint System

DiffHint Modes

Smart Comparison Range (2V)

Why 2V Range?

When Full Mode Triggers

The Safety Rule: Why Exactly 2V?

Change Detection Flow

Performance Comparison

Hash Comparison vs Character-by-Character

Real-World Examples

True Partial Mode

Before vs After

Results

Frontend: Buffer Reuse

ACK Gate: Flow Control

Performance Summary

By Scenario

Overall Reduction

Summary