Model</th>	Implementation</th>	Speed</th>	Design</th>	Review Signal</th></tr></thead>
Codex</strong></td>	B</td>	A+</td>	C</td>	D</td></tr>
Claude Code</strong></td>	A-</td>	D</td>	B</td>	B</td></tr>
Gemini CLI</strong></td>	C</td>	D</td>	B</td>	B+</td></tr> </tbody></table> Gemini CLI is currently free and excellent as a second opinion. I haven’t tried Open Code or DeepSeek yet, but hear they excel at implementation over design.</p> For the cost-conscious:</p> Claude Code (Opus)</strong> for design ONLY.</li> Codex</strong> for implementation.</li> Gemini CLI</strong> for review and second opinions.</li> DeepSeek</strong> as a budget-friendly third opinion.</li> </ul> The Magic Words</h2> I’ve found that no amount of CLAUDE.md / AGENTS.md / skills beats manually prompting them at specific points.</p> You will obviously get better results the more attention you pay to what they've done, giving them more specific instruction. However, even broad ones like these can work wonders:</p> For design and review:</strong></p> "I suspect this design is unsafe, unscalable, or won't integrate well, but I can’t articulate why. Please find the holes in this design, especially where a better solution is obvious."</p> </blockquote> For test reviews:</strong></p> "Carefully review these tests. Are they verifying that the code is CORRECT or just that it's successfully broken? Are they testing robust invariants of the system, or just that THIS specific marginal case works?"</p> </blockquote> For implementation reviews:</strong></p> "Review this implementation for bugs. If you find one, add a test that proves it. Does this introduce redundant systems? Should it integrate with existing authorities? Look for tech debt and hacks. Determine if the test coverage is robust and sufficient."</p> </blockquote> CLAUDE.md / AGENTS.md</h2> See my CLAUDE.md</a>. The sections below "Output" are safe to copy. The "Contributing" section should be tailored to your project.</p> Summary</h2> LLMs can suggest a 100x velocity increase, but the reality is closer to 10x. Parallelizing work streams (compiler, runtime, VM) can multiply this, potentially reaching 40x.</p> However, LLMs struggle with merge conflicts and may ignore test failures. They also tend to change unrelated code, like comments.</p> Workaround 1:</em> Instruct them to remove non-essential comment changes or irrelevant edits before committing.</li> Workaround 2:</em> Make sure you have CI setup so that they can't pretend the system already didn't work and that they didn't break it (they will).</li> </ul> Biggest Mistakes</h2> 1. Assuming LLMs could re-architect</h3> I initially allowed LLMs to build a "pile of crap," assuming they could fix it later. They are great at building such piles but struggle to refine them into a functioning language. I could have saved 66% of my headaches by enforcing design guardrails early.</p> 2. Moving too fast</h3> Blitzing through features usually resulted in more work and headaches. Even with LLMs, rushing is costly. Delivering at 100x is tempting, but the "headache factor" of LLM mistakes is significant.</p> I had great success when I followed my learnings closely for this FSM / Thunks / Streams PR</a>. The FSM implementation went especially well, which is where I slowed down the most.</li> </ul> </li> I had much pain and suffering when I ignored my learnings, and prematurely merged in a "hotfix"</a>. It ended up being a "Hot Bug Party" instead.</li> </ul> </li> </ul> 3. Not understanding LLM limitations early</h3> I should have generated a robust test suite from the start. LLMs break architectural invariants easily. They can generate testing frameworks quickly; use them to create "fortress files" that protect your design goals.</p> Why Software Jobs Aren't Imminent Targets</h2> Engineering was never about typing speed. It’s about ensuring code works as intended. While LLMs could generate my project in days - it wouldn't actually</em> work, and the bottleneck is review, ensuring it's implemented correctly and architecturally soundly, that it's sufficiently tested to have any guarantees it actually</em> works, rather than it just happens to pass a slim set of tests, which may in fact be testing nothing.</p> Most engineers spend only a third of their time writing code or less. Even if LLMs automate that, cheaper code will likely increase demand for engineering. The market will lag technological possibility by 5–10 years.</p> Disruption will be asymmetric. Startups, new grads, and managers may see more impact than the general engineering population (or vice versa).</p> Conclusion</h2> In six months of part-time tinkering, I’ve built a competitive runtime and language. This would have been impossible without LLMs acting as a 50x force multiplier.</p> I can "code" while walking the dog, at the gym, during lunch, instead of doom scrolling before bed, etc.</p> While LLMs can be frustratingly inconsistent, they are the most amazing tools I’ve used. I’ve never been more excited about the future of engineering.</p> You can see my full Development Process</a> for how I built a complex language that competes with Rust and Go on concurrent performance and nearly</em> works.</p> pprof integration 2026-05-08T00:00:00+00:00 `clear profile</code> emits its runtime profile data in pprof's gzipped protobuf format alongside the existing text dumps, so you can use the standard pprof</code> tool for flamegraphs, call graphs, and source views.</p>` Install</h2> Tool</th> When you need it</th> Install</th></tr></thead> pprof</code></td> Always (the viewer)</td> go install github.com/google/pprof@latest</code></td></tr> perf_to_profile</code></td> If you want CPU flamegraphs from perf.data</code></td> go install github.com/google/perf_data_converter/src/cmd/perf_to_profile@latest</code></td></tr> graphviz</code></td> If you use pprof -svg</code> / pprof -dot</code></td> apt install graphviz</code> / brew install graphviz</code></td></tr> </tbody></table> The web UI (pprof -http=:8080</code>) and pprof -top</code> text view do not need graphviz.</p> Use</h2> clear</span> profile foo.cht </span># -> writes foo.profile/heap.pb.gz, lock.pb.gz, mvcc.pb.gz </span># (and cpu.pb.gz if perf_to_profile is on PATH) </span> </span>pprof -http</span>=:8080 ./foo foo.profile/heap.pb.gz </span>pprof -top -alloc_space</span> foo.profile/heap.pb.gz </span>pprof -top -inuse_space</span> foo.profile/heap.pb.gz </span>pprof -top -delay</span> foo.profile/lock.pb.gz </span>pprof -base</span> before/heap.pb.gz after/heap.pb.gz </span># regression diff </span></code></pre> What's in each file</h2> heap.pb.gz</code></h3> Sample columns: alloc_objects</code> / alloc_space</code> / inuse_objects</code> / inuse_space</code>. Each call site is one sample with its hex address as a label (pprof -tags heap.pb.gz</code>).</p> lock.pb.gz</code></h3> Sample columns: contentions</code> / delay</code> / hold</code> / acquisitions</code>. Read+write contention sums into contentions</code>; delay</code> is the total wait (read+write); hold</code> is total exclusive hold time.</p> mvcc.pb.gz</code></h3> Sample columns: reads</code> / commits</code> / retries</code> / cow_bytes</code>. cow_bytes = struct_size * (commits + retries)</code> — the byte volume moved by copy-on-write commits, the most direct cost signal for @shared:versioned</code> cells.</p> channels.pb.gz</code></h3> Sample columns: pushes</code> / pops</code> / push_blocked</code> / pop_blocked</code> / max_depth</code>. One sample per registered channel; the synthetic function name is channel#<id></code> and the channel's capacity travels as a label (pprof -tags channels.pb.gz</code>).</p> cpu.pb.gz</code></h3> Standard CPU profile from perf.data</code>, converted by perf_to_profile</code>. Sample columns are whatever Go's pprof shows (samples</code> / cpu</code> ns).</p> CLEAR source mapping</h2> The transpiler emits // CLR:N</code> markers in transpiled.zig</code>. Our converter walks those back to the user's .cht</code> line and stamps it onto each pprof Location, so pprof -list <fn></code> shows CLEAR source lines (not Zig).</p> Sampling</h2> Stack traces for alloc-profile</strong> are captured on every alloc by default. For workloads where the per-alloc unwind cost matters, --sample=N</code> records every Nth event and scales the captured values by N so doctor / pprof see estimated totals:</p> clear</span> profile foo.cht</span> --sample</span>=100 </span></code></pre> Header records the chosen sample_n</code> so consumers can rescale or flag the approximation.</p> Stack traces for lock and mvcc profiles</h2> --sync-callstacks</code> (off by default) turns on per-record stack capture in lock-profile and mvcc-profile, so pprof's tree/flame views and per-caller attribution work for these profiles too:</p> clear</span> profile foo.cht</span> --sync-callstacks </span># auto-bumps --sample to 100 </span>clear</span> profile foo.cht</span> --sync-callstacks --sample</span>=1 </span># full capture, full cost </span></code></pre> Off by default because the FP walk costs ~100-500ns per record. Uncontended mutex acquire is ~10-20ns and MVCC commit fast paths are ~20-50ns, so the trace can dominate the operation it's measuring. When the flag is set without an explicit --sample</code>, we auto-default to --sample=100</code> to keep the cost manageable.</p> With --sync-callstacks</code> on, each (lock, caller-trace) pair becomes its own row in lock-profile, and same for (cell, caller-trace) in mvcc-profile. Doctor aggregates rows back to one-per-lock for its existing diagnoses; pprof tree views show the per-caller breakdown.</p> Doctor flags built on this data</h2> clear doctor</code> consumes the same profile dirs and grew three new flags that draw on the multi-frame trace data:</p> clear</span> doctor foo.profile/</span> --cumulative </span># rank functions by cum bytes </span>clear</span> doctor foo.profile/</span> --focus</span>=intToString </span># filter to traces that touch this function </span>clear</span> doctor foo.profile/</span> --ignore</span>=intToString </span># drop traces that touch this function </span>clear</span> doctor foo.profile/</span> --peek</span>=processRequest </span># callers + callees of one function </span>clear</span> doctor foo.profile/</span> --by</span>=allocs </span># sort heap by allocation count, not bytes </span>clear</span> doctor foo.profile/</span> --by</span>=inuse_bytes </span># sort by allocs - frees (live bytes) </span>clear</span> doctor old.profile/</span> --diff</span> new.profile/ </span># perf-regression diff between two runs </span></code></pre> --cumulative</code> aggregates bytes/allocs across every frame in each trace, so a function high in the call stack accrues its callees' costs. --focus=REGEX</code> keeps only sites whose trace touches a function matching the pattern. --diff</code> reports per-function deltas in allocation, lock contention, and MVCC retries, with directional arrows and "newly contended" / "retries eliminated" annotations.</p> Notes</h2> We do not emit a Mapping message for the binary, so pprof</code> prints "Main binary filename not available" and skips its own symbolization. Function names still appear because we resolve via addr2line</code> at conversion time.</li> alloc-profile captures stacks unconditionally (multi-frame v2, comma-separated leaf-first in alloc.txt</code>).</li> lock-profile (v3) and mvcc-profile (v2) capture stacks only when --sync-callstacks</code> is on. The 12th column of locks.txt</code> and the 8th column of mvcc.txt</code> carry -</code> (off) or comma-separated leaf- first addrs (on). Tab-separated to allow commas in the trace field.</li> </ul> STRICT EXTREME: The Path to High-Integrity CLEAR (@strict:extreme) 2026-05-01T00:00:00+00:00 STRICT</code> and STRICT EXTREME</code> mode are not yet available. DEFAULT</code> mode is the focus for the v0.1 release.</p> </div> Overview</h2> The "CLEAR Story" is a transition from Productivity</strong> (Ruby-like ease) to Performance</strong> (C-like speed) to Integrity</strong> (Ada/SPARK-level safety). While standard CLEAR provides memory and concurrency safety, STRICT EXTREME</strong> is a compilation mode designed for mission-critical systems (HFT, Aerospace, Medical) where runtime failures (Panics) are unacceptable.</p> Unlike other high-integrity languages, CLEAR does not require a different syntax for this level of safety. Instead, it uses Progressive Formal Verification</strong> to prove properties about the code that the Annotator</code> and OwnershipGraph</code> are already tracking silently.</p> The Integrity Spectrum</h2> Mode</th> Safety Guarantee</th> Developer Experience</th></tr></thead> Standard</strong></td> Memory & Concurrency Safety</td> "It just works" (Ruby-like)</td></tr> STRICT</strong></td> Performance & Effect Safety</td> No hidden HEAP/BLOCKING</td></tr> STRICT EXTREME</strong></td> Liveness & Panic Safety</strong></td> Formally Proven (Ada-like)</td></tr> </tbody></table> Core Pillars of STRICT EXTREME</h2> 1. The "No-Panic" Guarantee (Formal Proof)</h3> In standard mode, CLEAR prevents memory corruption but may !!</code> (Panic) on an out-of-bounds index or an integer overflow. In STRICT EXTREME</strong>, the compiler uses a SMT solver (like Z3) to prove that no runtime panic is possible</strong>.</p> Constraint Propagation</strong>: The compiler tracks value ranges. If it cannot prove idx < list.length()</code>, it rejects the build.</li> Proof Assistance</strong>: The clear fix</code> tool suggests @pre</code> (pre-conditions) or @post</code> (post-conditions) to satisfy the prover.</li> </ul> 2. Guaranteed Bounded Execution (Stack & Time)</h3> Ada-level safety requires knowing that a program will not run out of memory or hang.</p> Stack Depth Proof</strong>: The compiler walks the call graph and proves the maximum possible stack usage. Recursive functions without a proven base case are rejected.</li> TIGHT Loop Validation</strong>: Loops must be proven to terminate. "Infinite" loops are only allowed in dedicated "Supervisor" fibers.</li> </ul> 3. Total Effect Isolation</h3> While STRICT</code> mode requires annotations for HEAP and BLOCKING, STRICT EXTREME</strong> requires a Total Effect Signature</strong>.</p> Deterministic Paths</strong>: Functions can be marked @pure</code> (no side effects) or @deterministic</code> (identical inputs always yield identical timing/output).</li> Alloc-Free Zones</strong>: Proves that a "Hot Path" performs zero allocations (not even arena allocations) to guarantee zero jitter.</li> </ul> 4. Liveness & Deadlock Immunity</h3> Leveraging the OwnershipGraph</code>, the compiler proves that no circular dependency of @locked</code> or @shared</code> objects exists.</p> Lock-Ordering</strong>: The compiler enforces a global "Acquisition Hierarchy." If you try to acquire Lock B</code> while holding Lock A</code>, but elsewhere Lock A</code> is acquired after Lock B</code>, the build fails.</li> </ul> Implementation Path: "Progressive Disclosure"</h2> The beauty of the CLEAR architecture is that STRICT EXTREME</strong> does not hinder the early-stage developer:</p> Draft Phase</strong>: The developer writes code in Standard mode. The compiler silently tracks ownership and effects.</li> Optimization Phase</strong>: The developer turns on STRICT</code> mode to remove bottlenecks (allocations/blocking).</li> Certification Phase</strong>: The developer turns on STRICT EXTREME</code>. The compiler takes the existing metadata and "Hardens" it using formal verification.</li> </ol> Architecture & Logic</h2> Because CLEAR's memory model is a Directed Acyclic Graph (DAG)</strong> (enforced by Affine movement and Arena scopes), it is mathematically "Decidable." Proving properties about CLEAR code is significantly simpler than proving properties about C++ or Rust because aliasing is strictly controlled and visible</strong> to the Annotator</code>.</p> Goal for v0.3 Roadmap</h3> Integration of a formal solver into the Annotator</code> to provide "Panic-Free" proofs for core data-processing pipelines.</p> Deadlock in CLEAR 2026-04-22T00:00:00+00:00 Matrix row:</strong> Deadlock — C: F, Rust: F, Go: F, Pony: A+, BEAM: A+, CLEAR: A-</p> The asterisk: CLEAR uses locks for read-heavy workloads where MVCC unpredictability is a non-starter. Locks can deadlock in standard mode. CLEAR provides runtime detection + recovery, and STRICT EXTREME</code> (roadmap) will add compile-time lock-ordering enforcement.</p> Why the AB/BA Pattern Is Not the Main CLEAR Footgun</h2> The classic two-thread deadlock (thread A holds lock 1, waits for lock 2; thread B holds lock 2, waits for lock 1) requires two actors holding their respective lock simultaneously. In CLEAR's cooperative fiber scheduler, fibers only yield at explicit yield points (NEXT</code>, I/O). There is no yield between acquiring the first lock and acquiring the second inside a WITH EXCLUSIVE</code> block, so the AB/BA cycle cannot form on a single scheduler: the first fiber grabs both locks before the second fiber ever runs.</p> The AB/BA pattern does become dangerous with @parallel</code> (multi-scheduler, multiple OS threads), because the fibers then run on different OS threads simultaneously. But in the common single-scheduler case, it is not the footgun to worry about.</p> The Real Footgun: Re-Entrant Locking</h2> CLEAR's @locked</code> uses a non-recursive pthread_mutex_t</code>. If a fiber tries to acquire a lock it already holds, pthread_mutex_lock</code> blocks the OS thread — and since the fiber holding the lock IS that OS thread, it can never release it. Deadlock.</p> STRUCT State </span>{ </span>value: </span>Int64 </span>} </span> </span>FN</span> main() </span>RETURNS Void </span>-> </span> s = </span>State</span>{ </span>value: </span>0 </span>} </span>@locked</span>; </span> </span> t = </span>BG </span>{ </span> </span>WITH EXCLUSIVE</span> s </span>AS</span> outer { </span> outer.value = </span>1</span>; </span> </span># DEADLOCK: trying to re-acquire the same mutex this fiber holds. </span> </span># pthread_mutex_lock blocks the OS thread. The lock is never </span> </span># released because the releaser IS the blocked fiber. </span> </span>WITH EXCLUSIVE</span> s </span>AS</span> inner { </span> inner.value = </span>2</span>; </span> } </span> } </span> }; </span> </span> </span>NEXT</span> t; </span># hangs here </span>END </span></code></pre> This compiles without error and hangs at runtime.</p> Fix:</strong> restructure so the same lock is never acquired twice on the same call path. Pass the already-locked inner value to helper functions instead of locking again:</p> FN</span> update_inner(</span>inner: </span>State</span>) </span>RETURNS Void </span>-> </span> </span># Takes the already-unlocked value; no lock needed here. </span> inner.value = </span>2</span>; </span>END </span> </span>FN</span> main() </span>RETURNS Void </span>-> </span> s = </span>State</span>{ </span>value: </span>0 </span>} </span>@locked</span>; </span> </span> t = </span>BG </span>{ </span> </span>WITH EXCLUSIVE</span> s </span>AS</span> inner { </span> inner.value = </span>1</span>; </span> update_inner(inner); </span># pass the unlocked value, not the lock </span> } </span> }; </span> </span> </span>NEXT</span> t; </span>END </span></code></pre> The NEXT-Inside-WITH Footgun</h2> A subtler variant: fiber A holds a lock and calls NEXT</code> on a promise produced by fiber B — but fiber B also needs that lock to complete.</p> STRUCT State </span>{ </span>value: </span>Int64 </span>} </span> </span>FN</span> main() </span>RETURNS Void </span>-> </span> s = </span>State</span>{ </span>value: </span>0 </span>} </span>@locked</span>; </span> </span> producer = </span>BG </span>{ </span> </span># producer needs 's' to do its work </span> </span>WITH EXCLUSIVE</span> s </span>AS</span> inner { </span> inner.value = </span>42</span>; </span> } </span> }; </span> </span> consumer = </span>BG </span>{ </span> </span>WITH EXCLUSIVE</span> s </span>AS</span> outer { </span> </span># consumer holds 's', then waits for producer </span> </span>NEXT</span> producer; </span># DEADLOCK: producer can't acquire 's' — consumer holds it </span> print(outer.value.toString()); </span> } </span> }; </span> </span> </span>NEXT</span> consumer; </span># hangs here </span>END </span></code></pre> The fiber holding the lock yields via NEXT</code>, but the lock is NOT released when a fiber yields — WITH EXCLUSIVE</code> holds the lock for the duration of the block, not just until the next yield point. The producer is stuck waiting for the lock that the consumer holds while waiting for the producer.</p> Fix:</strong> resolve the promise before entering the lock, or restructure so the two operations don't depend on each other:</p> FN</span> main() </span>RETURNS Void </span>-> </span> s = </span>State</span>{ </span>value: </span>0 </span>} </span>@locked</span>; </span> </span> producer = </span>BG </span>{ </span> </span>WITH EXCLUSIVE</span> s </span>AS</span> inner { </span> inner.value = </span>42</span>; </span> } </span> }; </span> </span> </span>NEXT</span> producer; </span># wait for producer BEFORE acquiring the lock </span> </span> </span>WITH</span> s </span>AS</span> inner { </span> print(inner.value.toString()); </span># 42 </span> } </span>END </span></code></pre> STRICT EXTREME: Compile-Time Lock Ordering (Roadmap)</h2> In STRICT EXTREME</code> mode (planned, not yet implemented), the compiler will enforce a global acquisition hierarchy via the OwnershipGraph</code>. Re-entrant acquisition and AB/BA cycles will be detected at build time:</p> error: re-entrant lock acquisition </span> fiber acquires 's' at line 8 (WITH EXCLUSIVE) </span> then acquires 's' again at line 11 (WITH EXCLUSIVE) </span> fix: pass the already-locked pointer to helper functions instead </span></code></pre> See docs/strict-extreme.md</a> for the full STRICT EXTREME roadmap.</p> Fix: Single Lock Scope (Always Safe)</h2> Restructure so each operation needs only one lock at a time. No re-entrancy, no NEXT-inside-WITH:</p> FN</span> main() </span>RETURNS Void </span>-> </span> a = </span>Account</span>{ </span>balance: </span>100 </span>} </span>@locked</span>; </span> b = </span>Account</span>{ </span>balance: </span>200 </span>} </span>@locked</span>; </span> </span> t1 = </span>BG </span>{ </span> amount = </span>0</span>; </span> </span>WITH EXCLUSIVE</span> a </span>AS</span> ra { </span> amount = ra.balance / </span>2</span>; </span> ra.balance = ra.balance - amount; </span> } </span> </span># a is unlocked here. No re-entrancy possible. </span> </span>WITH EXCLUSIVE</span> b </span>AS</span> rb { </span> rb.balance = rb.balance + amount; </span> } </span> }; </span> </span> </span>NEXT</span> t1; </span>END </span></code></pre> The tradeoff: releasing between the two operations creates a TOCTOU window (see examples/footguns/05_toctou</a>).</p> Why C, Go, and Rust Get F</h2> All three languages expose the same non-recursive mutex with no re-entrancy detection and no built-in deadlock timeout:</p> C</strong> (pthread_mutex_lock</code>): hangs forever on re-entrant lock or AB/BA cycle. No detection without Helgrind or TSan.</li> Go</strong> (sync.Mutex</code>): hangs forever for partial deadlocks. Detects global deadlock (all goroutines blocked) with a panic. Race detector does not catch deadlocks.</li> Rust</strong> (std::sync::Mutex</code>): hangs forever. Attempting to lock from the same thread panics on some platforms (documented as "may panic or deadlock"). parking_lot</code> adds optional cycle detection.</li> </ul> Summary</h2> Pattern</th> Current CLEAR behavior</th> Planned fix</th></tr></thead> Re-entrant lock (same fiber re-acquires)</td> Hangs — non-recursive mutex</td> STRICT EXTREME: compile error</td></tr> NEXT inside WITH (holding lock while awaiting)</td> Hangs — lock not released on yield</td> STRICT EXTREME: compile error</td></tr> AB/BA across @parallel fibers</td> Hangs — same as C/Go/Rust</td> STRICT EXTREME: compile error</td></tr> AB/BA on single scheduler</td> Does not occur — no yield between lock acquisitions</td> N/A</td></tr> </tbody></table> Why CLEAR 2026-04-20T00:00:00+00:00 CLEAR is designed to be:</p> Correct</li> Safe</li> Understandable</li> Scalable</li> BLAZING</em> fast</li> </ol> But Why?</h2> Rust to have memory safety with no overhead - primarily to use for Servo to replace C for Browser Engines and for Kernel development. There are a number of tradeoffs Rust was not willing to make for it to be either 1) easy / understandable, or 2) a better Go.</p> Yet people are desperate to use Rust as a better Go because it has so much potential to be that.</p> Pony eliminates nearly all concurrency hazards by design. Rust merely ensures memory safety - which is perhaps the most common and critical concurrency hazard. But despite its safety, Pony is not widely used because the learning curve is too steep.</p> Rust & Pony were both not willing to make tradeoffs to prioritize ease-of-use or understandability.</p> Go made those trade-offs. But Go is a thin veneer over C with the most sophisticated runtime in the world bolted on. To achieve best in class speeds Go - like C - relies on loading a footgun and exposing yourself to a number of hazards and developer discipline and/or choosing the right libraries to get it right. It provides valuable best-in-class tooling to help mitigate some</em> of these problems.</p> CLEAR exists because it thinks Rust is not truly safe enough inherently, Pony is not easy enough, and Go is currently the most practical trade-off but is too dangerous and cannot fix its inherent problems.</p> Further, although Pony literally forces you into the actor pattern - at least at this stage, it is not inherently distributive. The actor model does not require serialization, distributed fault tolerance, supervision, etc. Pony uses the ideas to achieve safety on a single machine, but just because you wrote the code that allows</em> for distribution easily, does not mean it actually can be distributed effectively by default.</p> Pony did not want to make any sacrifices on safety. This led to a language that is impractical and/or not competitive for many of workloads (with a high cognitive burden).</p> CLEAR aims</strong> to take the lessons of Pony, Rust, Go, and BEAM and combine them. The goal is to make trade-offs to achieve understandability, and to sacrifice some</em> safety if it means giving you tools necessary to realistically accomplish common workloads.</p> Rubric</h2> Feature</th> C</th> Rust/Tokio</th> Go</th> Pony</th> BEAM</th> CLEAR</th></tr></thead> Cognitive Load</strong></td> F </td> D+</td> B</td> D-</td> C+</td> A-</strong></td></tr> Memory Safety</strong></td> F</td> A+</td> B </td> A+</td> A+</td> A+</strong></td></tr> Raw Speed</strong></td> A+</td> A</td> B+</td> A-</td> C</td> A-</strong></td></tr> Throughput</strong></td> A</td> A</td> A+</td> A</td> A</td> A</strong></td></tr> Memory Usage</strong></td> A+</td> A+</td> B-</td> B+</td> C-</td> A-</strong></td></tr> Predictability</strong></td> A+</td> A+</td> B-</td> B+</td> A-</td> A-</strong></td></tr> Backpressure</strong></td> F</td> A</td> A+</td> C</td> B</td> A</strong></td></tr> Starvation</strong></td> F</td> B</td> A</td> B</td> A+</td> A-</strong></td></tr> Fault Tolerance</strong></td> F</td> F</td> F</td> F</td> A+</td> ?</strong></td></tr> Deadlock</strong></td> F</td> C</td> C+</td> A+</td> A+</td> A- </strong></td></tr> Memory Ordering</strong></td> F</td> C</td> B</td> A+</td> A+</td> A-</strong></td></tr> Logical TOCTOU</strong></td> F</td> F</td> F</td> A+</td> A-</td> B+</strong></td></tr> Causal Ordering</strong></td> F</td> F</td> F</td> B+</td> C</td> B </strong></td></tr> Stream Ordering</strong></td> F</td> F</td> F</td> F</td> F</td> ?</strong></td></tr> </tbody></table> Non-CLEAR Ratings: Context & Justification</h3> All ratings are inherently subjective.</strong></p> The C Cognitive Load Controversy:</strong> C scoring an F</strong> may be controversial, but "Cognitive Load" here captures the true difficulty of writing correct</em> concurrent systems. C’s F is the price it pays for A+ ratings elsewhere; you could write C with a B in cognitive load (e.g., using a single global lock), but throughput would plummet to an F. Similarly, you could bolt on a Garbage Collector to improve safety, but speed and predictability would drop to a C.</li> The “Best Practices Fallacy”:</strong> Given perfect discipline and SOTA libraries, one can achieve better ratings in C, Rust, or Go. However, "doing it right" is not a scalable systems architecture. These ratings reflect the language’s inherent, default guarantees.</li> The Rust/Tokio Note:</strong> Rust is graded slightly unfairly as its evaluation includes Tokio. While not part of the core language, it is the de facto</em> production standard and its semantics must be evaluated.</li> The Go Scheduler:</strong> Go experts may feel there isn't enough separation in Throughput/Backpressure. CLEAR praises Go’s runtime and scheduler as one of the most sophisticated pieces of software ever written; these ratings may not do its operational excellence enough justice.</li> Go Memory Safety:</strong> Go is memory-safe in the conventional sense, but it does not provide strong concurrency-safety or invariant-safety by default. Its practical model often relies on discipline around shared mutable state.</li> The Pony Asterisk:</strong> Pony’s flawless safety comes at a hit to expressiveness. By forbidding shared mutable state, certain highly efficient data structures (like lock-free graphs) are nearly impossible to write idiomatically.</li> </ul> CLEAR Ratings: The Architecture</h2> Cognitive Load:</strong> CLEAR is designed so that Profile Guided Optimization (PGO)</strong> and automated tooling solve the heavy lifting. You write intuitive, sequential code, and the profiler suggests (and injects) the necessary optimization directives based on actual workloads.</li> Memory Safety:</strong> Like Rust, CLEAR utilizes Affine Ownership</strong> to guarantee memory safety.</li> Logical TOCTOU:</strong> Values behind Arcs/Locks cannot escape lexical scope. The compiler can generate Loom tests</strong> in a deterministic VM to catch dependencies and break them. Use-after-free and race conditions are included in Memory Safety. Time-related bugs are captured in Causal & Stream Ordering categories.</li> Deadlock:</strong> Technical deadlock is not part of CLEAR’s intended concurrency model, and by v0.3 the runtime aims to detect and prevent unbounded lock-wait cycles from silently persisting. Locks are intentionally more cumbersome to use because they are rarely the best-performing or safest solution. When developers opt into lock-based coordination anyway, they are also opting into explicit responsibility for handling lock-related failures correctly. A language like Pony or BEAM which does not have locks deserves a better ranking.</li> Starvation & Backpressure:</strong> Like BEAM, CLEAR will</em> prevent CPU starvation via its cooperative scheduler with automatically injected yielding. It separately tracks per-task memory consumption to kill runaway tasks and enforce backpressure.</li> Memory Consumption:</strong> CLEAR uses MVCC</strong> as a default synchronization technique. This adds memory overhead to eliminate common classes of bugs (deadlocks, contention).</li> Causal Ordering:</strong> CLEAR separates time as a tense</strong> in the type system, and offers A+</strong> causal ordering with @split</code> streams and solves the concurrent write problem with MVCC and transaction failure handling. But it allows you to "load the foot-gun" (unsafe escape hatches) for specific workloads where predictability is paramount.</li> Stream Ordering:</strong> CLEAR has not yet determined if it will offer any guarantees to protect against stream ordering related bugs. Frameworks like Flink and Kafka exist to solve this problem. It may not make sense to try to solve it in the runtime/language.</li> Fault Tolerance:</strong> It is still too early in the design stage to determine a realistic score for CLEAR's fault tolerance story—particularly regarding issues that arise from shared mutable memory and the fact that idempotence is not a first-class function color.</li> </ul> Fault Tolerance</h3> CLEAR hopes to have a reasonable ceiling of a B</strong> (similar to Causal Ordering). It will likely give you tools to shoot yourself in the foot to prioritize understandability and broad workload support—for example, you could mark tasks as killable which aren't, or tasks as retrying which aren't idempotent. But CLEAR will have the systems in place to easily support an A+</strong> architecture when paired with modern infrastructure.</p> The fact that everything else scores an F</strong> besides BEAM is evidence that solving these problems purely at the language level may no longer be the best choice.</p> The approach CLEAR is currently most interested in is targeting the five most common catastrophic faults:</p> CPU Starvation</strong> (Time)</li> Local Heap Exhaustion</strong> (Space)</li> Lexical Orphanhood</strong> (The spawning task died; the work is no longer relevant)</li> Data-Flow Severance</strong> (Producing a stream that no one is listening to)</li> Deadlocks</strong> (Unbounded lock-wait cycles)</li> </ol> CLEAR aims to either safely assassinate the offending fibers automatically (Deadlocks/Orphans) or detect the threshold and force the developer to handle the outcome (OOM limits).</p> Whatever CLEAR's final solution ends up being, BEAM folks are likely to look at it and say, "That's basically an F." But CLEAR aims to provide pragmatic value. Like BEAM folks, CLEAR is of the view that killing entire processes with important state just because you have no built-in fault tolerance is insane. Unlike BEAM, CLEAR is unlikely to eliminate the possibility of needing</em> to do that occasionally</em>, but it aims to reduce that need to practically zero.</p> How Does CLEAR think about Cognitive Burden</h2> In CLEAR's perspective, a program becomes difficult to understand because of global complexity.</p> A function is impossible to understand if the entire codebase must be understood to understand the function and line of code you are looking at.</p> This is why CLEAR aims to completely eradicate all forms of global complexity. If you can understand any function fully by just looking at the function, it is simple.</p> Rust is not considered difficult because of Affine Ownership. It is considered difficult because of "Fighting the Borrow Checker" which mainly boils down to Rust's obsession that it must achieve zero-cost abstractions in ALL</em> cases - which includes shared-mutable borrows. This is almost entirely the source of confusing lifetime annotations in Rust. If you eliminate this demand, you eliminate most of the "Fighting the Borrow Checker" in Rust complaints. CLEAR has done this. This is because this is Rust's main source of global complexity.</p> Pony is not considered difficult because it merely has capabilities or because it forces you into the actor pattern. It is difficult because capabilities do are not intuitive. You aren't describing what you want</em> to do, you are describing what can be done</em>. This is less intuitive.</p> Go claims to be easy</em> - which implies simplicity. But Go allows and often forces you to load footguns to achieve decent performance, lending itself to global complexity and hard-to-understand programs.</p> Go may claim to be easy, but if you need to write 20 lines of imperative arcana in the precise ceremonial order or rely on a set of libraries to do common things efficiently - it is not truly easy.</p> In essence, CLEAR thinks SQL proved that it is relatively easy to describe what</em> you want to do (your intent), and it is certainly easy to recognize it after the Profiler points you in the right direction for efficiency and safety (one-line optimizations clearly separate from code).</p> It is much harder to think like the computer and tell it HOW</em> to do that, or list what CAN</em> be done and hope that matches what you want to do.</p> How Does CLEAR think about Performance and Speed</h2> Raw Speed</strong> being a metric should raise some eyebrows. Performance is not a monolith. It is a balance of competing trade-offs.</p> From CLEAR's perspective - Correct, Safe, and Understandable are by far the most important pillars of the design that are being balanced.</p> When CLEAR thinks about speed, it mainly thinks about single machine</strong> concurrent throughput in compute heavy workloads. CLEAR is looking toward the future of compute rather than the present.</p> In the future, core count is expected to continue growing faster than memory. Cache per core has not meaningfully grown much in decades, and CLEAR assumes</em> this will not grow as meaningfully as core count in the future.</p> CLEAR thinks the main Raw Speed</strong> metrics are: 1) compiling via Zig to get LLVM best in class optimizations to single-core performance, 2) maximizing for cache locality</strong> as memory read stalls can dominate compute performance benchmarks.</p> In general, CLEAR aims to overcome</em> added memory, co-operative yielding, and any safety checks by gaining a level of cache locality that is nearly impossible to reach in any other language.</p> How?</p> The core pillar of CLEAR's design is to maximize local reasoning and minimize global complexity. This is what makes CLEAR simple</strong>, even if it may not be as easy</strong> as Go. It also should</strong> allow CLEAR to make compiler optimizations WRT escape analysis that no other language (that we are aware of) can make to maximize cache locality.</p> PROPAGANDA 2026-04-05T00:00:00+00:00 Cheating is all you need.</em></p> Software should be performant, robust, AND resilient.</li> It should also be effortless to write and understand.</li> It should be able to run anywhere, optimized for distributed parallelism and concurrency.</li> </ul> They told you "Pick one." They lied.</p> You can have it all, if you're willing to CHEAT.</p> Commands like SQL. Pipelines like Bash. Easy like Ruby. Speed like C.</strong></p> Being a genius like antirez</em> isn't scalable. It's not something everyone can be.</p> Everyone else can CHEAT.</p> WHAT IS CHEAT?</h2> CLEAR is a memory safe language like Rust, with better ergonomics than Swift.</p> It runs on a Go-like Runtime (the CHEAT runtime) that makes concurrent code as easy, safe, and fast as possible.</p> CORE CLEAR PHILOSOPHY</h2> Code should work.</li> Code that is easy to understand, write, AND test is more likely to work.</li> Dependencies should be Strictly-Correct, as that IS</em> their business logic.</li> Applications should FIRST</em> be Business-Logic correct, only Strictly-Correct if need be.</li> The language and compiler should never get in the way of Business-Logic correctness.</li> Making things Strictly-Correct, once Business-Logic correct, should be easy.</li> Making things BLAZING</em> fast, once Strictly-Correct, should be easy.</li> </ul> manifesto EXTRA 2026-04-03T00:00:00+00:00 Handling Array Access</h3> x = getMyVector(); </span>y = x[10]; -- compiler error! </span>y = x[10] OR ELSE 0; -- OKAY! </span> </span> </span>x: Int[10] = getMyVector(); </span>y = x[9]; -- OK </span>z = x[10]; -- COMPILER ERROR: Out of Bounds </span>zz = x[10] OR ELSE 0; -- OK - with a strong WARNING* (compiler error in STRICT mode) </span> </span>FN myVectorGetterDoer(v) -> </span> -- preferred to use a GUARD clause </span> z = GUARD v.size >= 56 OR ELSE CAST(v AS Int(56)); -- TODO: Is this too hard for the compiler to know? </span> </span> x = v[10]; -- OKAY, proven >10 </span> y = v[23]; -- OKAY, proven >23 </span> z = v[55]; -- OKAY, proven >55 </span> RETURN x + y - z + 5; </span>END </span> </span> </span>FN myVectorGetterDoer(v) -> </span> x = v[10]; </span> y = v[23]; </span> z = v[55]; </span> RETURN x + y - z + 5; </span>CATCH </span> RETURN 0; -- OK: caught here, returning a value </span>END </span> </span>FN myVectorGetterAdder(v) -> </span> x = v[10] OR RETURN 5; -- OK: IFF you return a value, not the error. </span> x = v[10] OR RETURN v[0]; -- COMPILER ERROR: OBVIOUSLY! </span> RETURN x + 5; </span>END </span> </span>-- here v is a DYNAMIC Vector. </span>FN myVectorGetterWhichTakesADynamicArrayAndResizes(v) -> </span> -- ... do a bunch of stuff </span> </span> x: Int[10] = v; -- COMPILER ERROR! </span> xx: Int[10] = TRUNCATE(v); -- OK: acknolwedge you could be losing data. </span> </span> y = xx[9]; -- OK: it will be 0 by default, if v was < 10 items. </span> </span> -- ... do a bunch of stuff </span> RETURN x + 5; </span>END </span> </span>-- here v is a FIXED Vector of size 1. </span>FN myVectorGetterWhichTakesADynamicArrayAndResizes(v) -> </span> -- ... do a bunch of stuff </span> </span> x: Int[10] = v; -- OK: perhaps surprisingly, you're allowed to upsize, no harm no foul </span> y = x[9]; -- OK: it will be 0 by default, if v was < 10 items. </span> </span> -- ... do a bunch of stuff </span> RETURN x + 5; </span>END </span></code></pre> Handling Division by Zero</h3> FN myDivider(x, y) -> </span> y = GAURD y != 0 OR ELSE 1; -- OKAY, if y is MUTABLE </span> RETURN x / y; </span>end </span> </span>-- this is fine in a function </span> </span>FN typicalFunc(myObj) -> </span> health = GUARD myObj.health != 0 OR ELSE 1; </span> -- Do a bunch of stuff </span> RETURN myObj.strength / myObj.health; </span>end </span> </span>-- Say you had a complex formula with lots of division </span>-- You might not want to have </span> </span>FN typicalFunc(myObj) -> </span> health = GUARD myObj.health != 0 OR ELSE 1; </span> -- 10 more guards, needing to use local variables below </span> -- Do a bunch of stuff </span> -- RETURN <your equation with a bunch of division> </span>end </span> </span>-- Instead you can do: </span> </span>FN typicalFunc(myObj) -> </span> -- RETURN <your equation with a bunch of division> </span>CATCH </span> RETURN -1; </span>end </span> </span>-- HOWEVER - there is no such option for code outside of a function. </span>-- You can do: </span> </span>FN main() </span> -- Good design, no code execution outside of main </span> -- In here, you do do all kind of division </span>CATCH </span> -- Handle however you want, except with code that raises an uncaught error </span>END </span> </span>-- If you just have a script like: </span> </span>x = readSomeFileGetSomeNumber(); </span>y = 10; </span>callSomeFunc(y / x); </span> </span>-- You'll get a DivisionByZero error, so you must do: </span> </span> </span>x = GUARD readSomeFileGetSomeNumber() != 0 OR ELSE 1; </span>y = 10 </span>callSomeFunc(y / x); </span></code></pre> ASYNC/AWAIT/COLLECT</h3> listOfUsers </span> s> FILTER %.isActive </span> s> COLLECT -- Converts Stream<User> to List<User> </span> s> map(%(x) -> x.size > 5); -- Operates on the List<User> object </span> </span>-- Did not prefix with AWAIT -> assumes intentend ASYNC -> immediately starts next step </span> </span>file_paths </span> s> concurrentRead </span> s> COLLECT -- Wait for ALL files to be fetched/materialized. </span> s> aggregate; -- Start aggregation only on the complete set. </span> </span> </span>-- Did not prefix with AWAIT -> assumes intentend ASYNC -> immediately starts next step </span> </span>file_paths </span> s> concurrentRead </span> s> COLLECT -- Wait for ALL files to be fetched/materialized. </span> s> aggregate </span> s> COLLECT; -- waits until THIS finishes to resume (which may happen before the other two finish) </span> </span>-- The above is nearly equivalent to: </span> </span>AWAIT file_paths </span> s> concurrentRead </span> s> COLLECT -- Wait for ALL files to be fetched/materialized. </span> s> aggregate; </span> </span>-- This is preferred, but not enforced. </span>-- Ending in COLLECT will convert from Stream<T> to List<T> -> even if it's not assigned to anything. </span></code></pre> MUTABILITY vs IMMUTABILITY</h3> user = %{ name: "Alice", active: false } </span>user.active = true; -- Error! </span>user = %{ name: "Alice", active: true} -- Error!! </span> </span>user = %{ name: "Alice", active: false } </span>upDatedUser = user MUTATE { active: true } -- No error! </span> </span>x = 5; </span>x += 10; -- ERROR! </span> </span>MUTABLE x = 5; </span>x += 10; </span></code></pre> Example compiler errors:</p> Error: Cannot set field 'active' on 'user' to true. </span> : This is a READ-ONLY VIEW of memory owned by 'DatabaseQueryResult'. </span> : To modify, use: </span> : </span> : newUser = user MUTATE { active: true }; </span> : </span> : Or, if you want to modify it later, you might want it to be MUTABLE, use: </span> : </span> : MUTABLE newUser = DEEP_COPY(user_data); </span> : newUser.active = true; </span> : </span> : You can set fields on MUTABLE data. </span> </span> </span>Error: Cannot assign to 'x': it is immutable. </span> : x is IMMUTABLE. </span> : </span> : You can create a new binding: </span> : </span> : newX = x + 5; </span> : </span> : Or, if you want to modify it later, declare it mutable: </span> : </span> : MUTABLE newX = x; </span> : newX = 5; </span></code></pre> THE CONFUSING TYPES</h2> Errors</h3> You don't need to check for errors by default or specify them in your return types.</li> The Compiler and the SMOOTH operator takes care of this for you.</li> </ul> FN myCarelessFn %(myList) -> </span> myList </span> s> fetchData -- this could return an error! </span> s> parseData -- this could return an error! </span> s> renderPage; -- this could return an error! </span> </span>-- It's fine to not catch any of them here! </span>-- You're allowed to pass on the problem to your end user! </span>END </span> </span>FN mySomewhatCarefullFn %(myList) -> </span> myList </span> s> fetchData -- this could return an error! </span> s> parseData -- this could return an error! </span> s> renderPage; -- this could return an error! </span> </span>CATCH -- anything </span> -- No matter what I want my users to get the default page. </span> RETURN makeDefaultPage(); </span>END </span> </span>FN myMoreCarefullFn %(myList) -> </span> myList </span> s> fetchData -- doesn't ever throw an error! </span> s> parseData -- doesn't ever throw an error! </span> s> renderPage; -- doesn't ever throw an error! </span> </span>CATCH -- anything </span> -- This is still allowed, since no matter what you can OOM </span> RETURN makeDefaultPage(); </span>END </span> </span>FN myQuiteCarefullFn %(myList) -> </span> myList </span> s> fetchData OR SKIP -- SKIP any error </span> s> OTHERWISE(fetchFromBackup) -- OTHERWISE only happens if an error happend, and this may raise error, it's fine! </span> s> parseData -- this could return an error! </span> s> renderPage; -- this could return an error! </span> </span>CATCH -- anything EXCEPT fetchData error -> It was already handled inline </span> RETURN makeDefaultPage(); </span>END </span> </span>FN myReallyCarefullFn %(myList) -> </span> myList </span> s> fetchData OR SKIP -- SKIP any error </span> s> OTHERWISE(fetchFromBackup) OR RETURN -- Don't proceed any further, bubble this right up to the user </span> s> parseData -- this could return an error! </span> s> renderPage; -- this could return an error! </span> </span>CATCH -- anything </span> -- EXCEPT fetchData error -> It was already handled inline </span> -- EXCEPT fetchFromBackup -> It was already handled inline </span> RETURN makeDefaultPage(); </span>END </span> </span>FN myVeryCarefullFn %(myList) -> </span> myList </span> s> fetchData OR SKIP -- SKIP any error </span> s> OTHERWISE(fetchFromBackup) OR RETURN -- Don't proceed any further, bubble this right up to the user </span> s> parseData OR EXIT -- Don't proceed any further, but try to pass this to a catch block below </span> s> renderPage; -- this could return an error! </span> </span>CATCH -- anything </span> -- EXCEPT fetchData error -> It was already handled inline </span> -- EXCEPT fetchFromBackup -> It was already handled inline </span> -- DOES CATCH parseData (and renderPage) errors </span> RETURN makeDefaultPage(); </span>END </span> </span>FN myExtremelyCarefullFn %(myList) -> </span> myList </span> s> fetchData OR SKIP -- SKIP any error </span> s> OTHERWISE(fetchFromBackup) OR RETURN -- Don't proceed any further, bubble this right up to the user </span> s> parseData OR GOTO_RECOVER -- Don't proceed any further, JUMP to the first RECOVER down the chain </span> s> renderPage -- this could return an error! </span> s> RECOVER(makeDefaultPage()); </span> </span>CATCH -- anything </span> -- EXCEPT fetchData error -> It was already handled inline </span> -- EXCEPT fetchFromBackup -> It was already handled inline </span> -- DOES CATCH parseData (and renderPage) errors </span> RETURN makeDefaultPage(); </span>END </span> </span> </span>FN myMostCarefulFn %(myList) -> </span> myList </span> s> fetchData OR SKIP -- SKIP any error </span> s> OTHERWISE(fetchFromBackup) OR RETURN -- Don't proceed any further, bubble this right up to the user </span> s> parseData OR GOTO_RECOVER -- Don't proceed any further, JUMP to the first RECOVER down the chain </span> s> renderPage OR EXIT "RenderPage failed" </span> s> RECOVER(makeDefaultPage()) OR EXIT "RenderBackupPageFailed"; </span> </span>CATCH -- anything </span> -- Here, there might be two RenderPage errors </span> -- Down here, we might want to do two different things based on the same Error (with the context string from above) </span> -- In both cases, we have acccess to the Error object `%e` </span> -- With %e.message and %e.snapshot (whatever went into the pipe) set. </span> -- </span> -- WARNING: To operate on %e.snapshot, you must cast it from Any to Whatever it is. </span> -- This is dangerous, as it could itself raise an error. </span> -- If this something like this is not acceptable, CHEAT is not for you. </span> -- </span> -- You CAN log it without issue (with a default limit to how big the string is) </span> -- Though, that could contain PII, so unless you control the parent, and can ensure it's safe for logging </span> -- Do so at your own risk </span> </span> RETURN makeDefaultPage(); </span>END </span> </span>-- Testing Error handling is easy! </span>TEST "MyFn handles errors gracefully" -> </span> result = ERROR("Boom") s> myFn; </span> ASSERT result == defaultPage; -- Success! </span>END </span></code></pre> Strings Vs Bytes</h3> String by default.</p> s = "😊"; -- String (Immutable) </span>MUTABLE s = "😊"; -- String (Mutable, enforces UTF-8 on write) </span>s: String[10] = "😊"; -- String (Immutable, fixed size UTF-8 buffer, very uncommon use case -> mainly for cache locality, not beginner need) </span>MUTABLE s: String[10] = "😊"; -- String (Mutable, fixed size UTF-8 buffer, more common use case, but not beginner) </span> </span>b: UInt8[] = %[0, 10, 255]; -- Buffer (Immutable -> common / default use case for non-beginners) </span>MUTABLE b: UInt8[] = %[0, 10]; -- Buffer (Mutable, Dynamic -> common use case) </span>b: UInt8[10] = ...; -- Buffer (Immutable, Fixed Size -> you might want for cache locality) </span>MUTABLE b: UInt8[10] = ...; -- Buffer (Mutable, Fixed Size -> common use case) </span></code></pre> Lists vs Streams</h3> users = db.all s> filter; -- LIST, default </span>users = db.all s> filter s> COLLECT; -- LIST, default, superfluous </span>users = AWAIT db.all s> filter; -- LIST, default, superfluous </span> </span>userCursor: Stream = db.all s> filter; -- STREAM, not-default due to EXPLICIT type </span>userCursor = db.all s> filter s> ITER; -- STREAM, not-default due to EXPLICIT ITER </span>userCursor = ASYNC db.all s> filter; -- STREAM, not-default due to EXPLICIT ASYNC </span></code></pre> Streams are handles, not data</h3> A stream is an immutable</em> pointer to a mutable</em> partition data (the stream / flow where x is COLLECT</code>ing).</li> However, you can have an mutable</em> pointer.</li> </ul> Therefore:</p> MUTABLE source: Stream = primary_db.users; -- OKAY </span> </span>TRY </span> source.peek(); -- Check connection </span>CATCH </span> -- I need to reassign the variable to the backup stream! </span> source = backup_db.users; </span>END </span> </span>source s> map... </span> </span>-- ... </span> </span>MUTABLE s: Stream = db.all s> ... ; -- OKAY </span>MUTABLE s: Stream(10) = db.all s> ... ; -- Compiler error, `... returns a single Stream, not a Vector of Streams`. </span></code></pre> </h2> See I'm too dumb to open a file in Zig</a> - for common problems of why being a genius doens't scale - even for something as simple as reading a text file...</p> I wanted to write code that scales linearly to 128+ cores, but feels like Ruby. That didn't exist, so I made it. 2026-04-03T00:00:00+00:00 The Goal</h2> I wanted the scaling of a 128-core shared-nothing engine, but the ergonomics of Ruby. I wanted to write high-level, readable logic and have a "Magic Button" to make it fast.</p> In my latest benchmarks, this approach is already 6x faster than Go</strong> and 1.4x faster than Rust</strong> on a 128-core KV-store workload. But the speed isn't the story—the scaling</strong> is.</p> The Problem: The "Lock Tax"</h2> I wanted to write shared-nothing code easily and correctly. Writing this kind of architecture in C or C++ is notoriously difficult. You are constantly fighting race conditions, manual memory management, and the silent performance killer: Cache-Line Bouncing</strong>.</p> In traditional concurrent languages like Go or Rust, you eventually hit a "Contention Ceiling." You add more cores, but the overhead of keeping caches in sync (the "Lock Tax") means you stop seeing speedups. At 64 or 128 cores, these languages often hit negative scaling—adding more hardware actually makes your app slower.</p> Why not Actors?</h3> Languages like Erlang or Elixir "solve" this with the Actor Pattern. They isolate state by forbidding shared memory entirely. It’s a beautiful model, and it works perfectly for telecom or niche IO cases where you have millions of slow processes sending small messages.</p> But for high-performance systems, the Actor model just swaps one bottleneck for another: The Copying Tax</strong>. If you can't share memory, you have to copy it. When you're trying to perform 10 million operations per second on a KV store, the cost of copying keys and values between actors becomes far more expensive than the locks you were trying to avoid.</p> Actors are "Shared-Nothing" conceptually, but they are "Copy-Everything" in practice. I wanted True Shared-Nothing</strong>: Zero locks, zero contention, and zero-copy performance.</p> Why a library wasn't enough</h2> I didn't want to build a new language. I knew the "Ecosystem Desert" is where most new languages go to die. I tried to achieve my goals by building a library for existing languages, but I hit a wall every time:</p> Go:</strong> You can't escape the Garbage Collector, and the scheduler is too opaque to guarantee the thread-pinning required for true shared-nothing performance.</li> Rust:</strong> The type system is brilliant but rigid. To change a concurrency model, you have to refactor every function signature in the call chain. I couldn't find a way to make optimization a "one-line directive" without fighting the borrow checker at every turn.</li> Zig:</strong> I love Zig's performance, but even with its comptime</code> magic, the code never felt like Ruby. It’s a "Manual Transmission" language; I wanted a "Smart Transmission" that chose the gears for me.</li> </ul> I wanted code to work like SQL</strong>. In SQL, you write a query that describes what</em> you want, and then you add an INDEX</code> to optimize how</em> it's retrieved. You don't rewrite the whole query in C just to make it scale. I wanted that "Optimization via Directive" for general-purpose logic.</p> The Breakthrough: The Zig Bridge</h2> I finally realized that I didn't need to build a "Walled Garden." By transpiling CLEAR to Zig</strong>, I got native access to the entire C standard library and the LLVM toolchain for free.</p> It didn't matter if people would use my language on Day 1, because CLEAR has access to everything</strong> that already exists in C and Zig. I didn't have to build a new world; I just had to build a better way to talk to the one we already have.</p> The Result: Shared-Nothing Capabilities</h2> CLEAR achieves its performance through Capabilities</strong>. Instead of bake-in a concurrency model into a Type, you apply it to a Binding</strong>:</p> -- Write it like Ruby: </span>MUTABLE map: HashMap<String> = {}; </span> </span>-- Optimize it with one line: </span>MUTABLE map: HashMap<String>@sharded(128) = {}; </span></code></pre> By adding @sharded(128)</code> and @pinned</code>, you tell the CLEAR compiler to generate a shared-nothing, lock-free architecture that scales linearly. The business logic remains identical. The compiler handles the infrastructure.</p> Conclusion</h2> CLEAR is currently in v0.1. It’s not ready for your production database yet, but it’s ready to prove that the "Iron Triangle" of Performance, Scaling, and Ergonomics can</em> be broken.</p> CLEAR has not yet proved that the triangle is</em> broken—a v0.1 release is just the beginning. But I believe these results are definitive proof that with further development, the triangle will</em> be broken.</p> I’m not here to ask you to switch your stack. I’m here to show you a solution to the "Many-Core" problem that is only going to get harder as our hardware keeps growing.</p> Problem solved.</strong></p> Profiling 2026-04-02T00:00:00+00:00 Quick Start</h2> ./clear</span> profile myapp.cht </span>./clear</span> doctor myapp.profile/ </span></code></pre> clear profile</code> builds with allocation tracking enabled (zero overhead in normal builds), runs the program, and collects:</p> Heap profile</strong>: allocation counts and bytes per call site</li> CPU profile</strong>: perf record</code> sampling (when available)</li> Syscalls</strong>: strace -c</code> breakdown</li> Hardware counters</strong>: perf stat</code> (cycles, cache misses, branch misses)</li> </ul> clear doctor</code> reads the profile data and prints actionable optimization advice with CLEAR source line numbers.</p> Example: Optimizing Benchmark 17 (KV Store)</h2> Step 1: Profile</h3> ./clear</span> profile benchmarks/concurrent/09_kvstore/bench.cht </span>./clear</span> doctor benchmarks/concurrent/09_kvstore/bench.profile/ </span></code></pre> Step 2: Read the CPU profile</h3> === CPU Profile === </span> 37.49% hash_map.HashMapUnmanaged.getIndex -- hashmap probing </span> 12.21% pthread_rwlock_unlock -- RwLock release </span> 6.37% memcpy -- value copying </span> 5.29% hash.wyhash.Wyhash.hash -- key hashing </span> 4.88% ShardedStringMap.put -- shard selection + lock </span> 2.84% pthread_rwlock_rdlock -- RwLock acquire </span></code></pre> Observation</strong>: 15% of CPU is in pthread_rwlock_unlock</code> + pthread_rwlock_rdlock</code>. That's lock overhead, not useful work.</p> Step 3: Check hardware counters</h3> === Hardware Counters === </span> 5,390,314,521 instructions # 1.66 insn per cycle </span> 10,001,037 cache-misses # 52.87% of all cache refs </span> 60,878,759 L1-dcache-load-misses # 4.88% of all L1-dcache accesses </span></code></pre> Observation</strong>: 53% LLC cache miss rate. The hashmap's random-access probing misses the cache on every other access. This is inherent to hash tables with large working sets - not fixable by resharding.</p> Step 4: Form a hypothesis</h3> Two bottlenecks:</p> Lock contention (15% CPU)</strong>: The benchmark uses @writeLocked</code> (RwLock). RwLock has high per-operation overhead even for single-writer access. For write-heavy workloads, @locked</code> (Mutex) should be cheaper.</li> Cache misses (53% LLC)</strong>: Inherent to random hash probing across 1M keys. Not fixable without changing the data structure.</li> </ol> Step 5: Test the change</h3> Switch from @shared:sharded(128):writeLocked</code> to @shared:sharded(128):locked</code>:</p> Config</th> SET</th> GET</th> Zipf</th> Mixed</th> Total</th></tr></thead> @writeLocked</code> (before)</td> 110ms</td> 17ms</td> 18ms</td> 77ms</td> 246ms</td></tr> @locked</code> (after)</td> 58ms</strong></td> 20ms</td> 24ms</td> 26ms</strong></td> 198ms</strong></td></tr> </tbody></table> SET: 47% faster</strong> (Mutex has lower overhead for exclusive access)</li> Mixed: 66% faster</strong> (RwLock writer starvation was killing mixed workloads)</li> Total: 20% faster</strong></li> GET/Zipf: slightly slower (Mutex blocks concurrent readers that RwLock allows)</li> </ul> Step 6: Compare against Rust and Go</h3> </th> Rust (DashMap)</th> Go (sync.RWMutex)</th> CLEAR @locked</th></tr></thead> SET</td> 75ms</td> 1217ms</td> 58ms</strong></td></tr> GET</td> 14ms</td> 529ms</td> 20ms</td></tr> Zipf</td> 13ms</td> 17ms</td> 24ms</td></tr> Mixed</td> 19ms</td> 26ms</td> 26ms</td></tr> Total</strong></td> 237ms</strong></td> 1806ms</strong></td> 198ms</strong></td></tr> </tbody></table> CLEAR beats Rust (DashMap) by 17% total. SET is 22% faster than DashMap.</p> What the profiler told us</h3> The profiler didn't just confirm "it's slow" - it pointed to the specific mechanism:</p> CPU profile showed 15% in rwlock functions</li> Hardware counters confirmed cache misses are inherent (not fixable)</li> This directed the fix to the lock type, not the shard count</li> </ol> We also tested reducing shards from 128 to 32 - this made things worse</strong> (more contention per shard), confirming that the shard count wasn't the problem.</p> How It Works</h2> Heap Profiling</h3> Built into the runtime allocator VTable. Every allocation in the runtime passes @returnAddress()</code> (the caller's return address). The profiler records this address, allocation count, and bytes per site in a fixed-size hash table (1024 sites, no heap allocations inside the profiler).</p> Key runtime helpers (charAtCodepoint</code>, intToString</code>, substr</code>, concat</code>, join</code>) also call profileAlloc()</code> with their own @returnAddress()</code>, giving function-level attribution.</p> Controlled by a comptime flag (CLEAR_PROFILE</code>). When not set, all profiling code is eliminated at compile time - zero overhead in production.</p> CPU Profiling</h3> Uses Linux perf record</code> for sampling-based CPU profiling. Requires perf_event_paranoid <= 2</code>:</p> sudo</span> sysctl kernel.perf_event_paranoid=2 </span># temporary, resets on reboot </span></code></pre> Source Mapping</h3> clear doctor</code> maps addresses back to CLEAR source lines using:</p> addr2line</code> to resolve addresses to Zig file:line</li> // CLR:N</code> comments in the transpiled Zig to map back to CLEAR line numbers</li> </ol> Profile builds use -fno-strip</code> to retain debug symbols.</p> Environment Variables</h2> Variable</th> Set by</th> Purpose</th></tr></thead> CLEAR_ALLOC_PROFILE</code></td> clear profile</code></td> Output path for allocation data</td></tr> CLEAR_THREADS</code></td> User</td> Number of scheduler threads</td></tr> </tbody></table> Architecture 2026-03-31T00:00:00+00:00 1. Arena-Based Memory & Isolation</strong></p> CLEAR leverages affine types and Arena-based memory to give you near High-Frequency Trading standards of allocation with zero thought.</li> Due to optimizing broadly for a fiber-runtime, there are cases when the stack could be better leveraged, but CLEAR chooses not to for large stack objects (to keep fiber stacks small, until most fibers are transformed seamlessly into Finite State Machines by v0.4).</li> </ul> 2. Deterministic Shared-Memory and a Declarative Concurrency Model</strong></p> Parallelism is achieved via BG/DO/CONCURRENT</code>, which creates isolated execution contexts. Rather than telling the code how</em> to achieve fast speeds, you simply tell write the code that expresses your intent - the what</em>.</li> CLEAR provides one-line capability optimizations. Changing your app from an Arc/RwLock strategy suffering from synchronization bottle-necks to a shared-nothing architecture which can compete with Dragonfly DB can be a one-line change.</li> </ul> </li> Spawning a process creates a lightweight, isolated fiber / memory arena (or Finite State Machine in the near future).</li> </ul> 3. Implicit "Railway" Error Handling</strong></p> CLEAR treats errors as data, but handles them via control flow.</li> The SMOOTH</code> operator \|></code> (aka the PIPE</code> or \|> </code> in Elixir, etc) acts as a guard. It automatically bubbles errors down the chain, to be handled elsewhere, or allows them to be handled inline elegantly.</li> </ul> </li> This ensures code reads top-to-bottom & is left-sided (the "Happy Path") -- making it always clear what's desired vs what's the fallback.</li> The Result:</em> No if err != nil boilerplate. No Pyramid of Doom. No checkOk clutter. No if .nil?</code> everywhere.</li> </ul> 4. A Type system that just works</em></strong></p> CLEAR is easy to write for beginners. The Type system is implicit, staying out-of-the-way by powerful Type inference.</li> The compiler takes care of the confusing parts of the type system for you.</li> The Result:</em> If you can write code in JavaScript, Lua, Python, or Ruby - you can write blazing fast CLEAR code.</li> </ul> 5. Fortress Architecture</strong></p> All PUBLIC</code> functions must return a single type with a suitable default - or explicitly error</li> All PUBLIC</code> structs must have a suitable default - or explicitly an error A PUBLIC</code> function obviously cannot take a non-PUBLIC</code> struct as an input.</li> </ul> </li> The Result:</em> Guarantees that only YOUR</em> code can cause run-time errors (or none if in STRICT</code> mode). You can auto-gen tests for PUBLIC functions for all permutations of POSSIBLE unexpected inputs. This allows you to spot problems easily and arrive at robust, working code quickly.</li> </ul> </li> </ul> </li> </ul> CLEAR for Scripters 2026-03-31T00:00:00+00:00 If you already know Logic (JavaScript/Python), the only thing stopping you from writing System-Level code is Memory.</p> In Ruby/Python/JavaScript, Memory is "Magic."</li> In C, it's an incomprehensible arcana.</li> In CLEAR, Memory is "Physics."</li> </ul> Here are the 9 Rules of Physics in CLEAR.</p> 1. Physics of change: MUTABLE</code> vs IMMUTABLE</code> (Default)</h3> In Ruby or Python, you can change the value of any variable by default (they behave as MUTABLE</code>).</p> name = </span>"</span>Bob</span>" </span>name = </span>"</span>Alice</span>" </span></code></pre> In CLEAR, binding a name for the first time creates an immutable</strong> variable. Reassigning it is a compiler error.</p> name = </span>"</span>Bob</span>"</span>; </span>name = </span>"</span>Alice</span>"</span>; </span># COMPILER ERROR! `name` is immutable. </span></code></pre> x = value</code> = READ</strong> Only (immutable, no keyword needed).</li> MUTABLE x = value</code> = READ/WRITE</strong>.</li> </ul> The "Gotcha":</strong> If you want to modify a variable, it must be MUTABLE</code>.</p> MUTABLE </span>name = </span>"</span>Bob</span>"</span>; </span>name = </span>"</span>Alice</span>"</span>; </span># OK </span></code></pre> 2. Physics of SHAPE: TYPES</h3> In CLEAR, variables have types to ensure safety and speed.</p> MUTABLE </span>name = </span>"</span>Bob</span>"</span>; </span>name = </span>1</span>; </span># COMPILER ERROR: `name` is a String, cannot assign a Float64. </span></code></pre> 3. Physics of Size: FIXED vs. DYNAMIC</h3> In CLEAR, you choose the physics you need.</p> FIXED</strong> Size: Optimized for speed and cache locality.</li> DYNAMIC</strong> Size: Grows indefinitely, but requires the Warehouse (HEAP).</li> </ul> 4. The Two Worlds: STACK vs. HEAP</h3> The STACK (Default)</h4> Think of this as your Backpack</strong>.</p> Pros: Instant access (L1 Cache). Blazing</em> fast.</li> Cons: Itty-bitty space. Fixed size.</li> Behavior: When you finish a task (Function returns), you dump your backpack into the incinerator. Everything inside is gone. POOF</em>.</li> </ul> The HEAP</h4> Think of this as a Warehouse</strong>.</p> Pros: (nearly) Unlimited space. Can grow/shrink.</li> Cons: You have to drive there to get stuff (Slower).</li> </ul> How to Choose</h4> In CLEAR, the compiler and runtime handle the physics of where data lives for you in 99% of cases. You don't need to use a sigil to choose between stack and heap.</p> x = [1, 2]</code> → CLEAR decides the most efficient location.</li> list = [1, 2, 3]</code> → This list is optimized for performance and safety.</li> </ul> If you need to explicitly force an object onto the heap (e.g., for recursive structures or large buffers), you can use the indirect</code> capability (similar to Box</code> in Rust).</p> # Recursive structures use 'indirect' to avoid infinite size on stack </span>STRUCT Node </span>{ </span> </span>value: </span>Int64</span>, </span> </span>left: </span>?Node </span>@indirect</span>, </span> </span>left: </span>?Node </span>@indirect </span>} </span></code></pre> The "Gotcha": If you try to .push!</code> or .pop!</code> to a fixed-size array, the compiler will yell at you. It’s not being mean; it’s telling you that physics forbids it.</p> x = [</span>1</span>, </span>2</span>, </span>3</span>]; </span># ... do something, now I need to add to `x`, what do I do? </span>x.append!(</span>4</span>); </span># COMPILER ERROR! `x` is immutable. </span></code></pre> 5. Physics of Capability: Capabilities vs Types</h3> In CLEAR, we separate Types</strong> from Capabilities</strong>.</p> Types</strong> describe what</em> the data is (e.g., User</code>, Account</code>).</li> Capabilities</strong> describe how</em> you access it (e.g., shared</code>, multiowned</code>, alwaysMutable</code>).</li> </ul> The Rule:</strong> Functions take Types</strong>, not Capabilities</strong>.</p> The CLEAR Model</h3> Ownership:</strong> Rc = multiowned</code>, Arc = shared</code></li> Synchronization:</strong> RwLock = shared:writeLocked</code>, Mutex = shared:locked</code> coming soon</strong> -> Mvcc = shared:versioned</code>, :actor</code> uses Object Actor Pattern combined with compiler aware SHARDING</li> </ul> </li> Interior Mutability:</strong> Cell, RefCell -> combined = alwaysMutable</code> Automatically acts like Cell for data under 16 bytes</li> alwaysMutable</code> must be unwrapped before individually passing into a function as an argument, like any other capability</li> </ul> </li> Existence:</strong> Option, Result => not a capability -> a tense: T?</code> = Optional T</code></li> Unwrapped like in Rust and Zig with .?</code></li> </ul> </li> Future:</strong>: Something that will arrive later ~T</code> = Future T</code></li> ~T[]</code> = Future T</code>s (like a Stream).</li> </ul> </li> </ul> affUser = </span>User</span>.new(); </span># creates `affine User` (default) </span>a = affUser; </span># OKAY, affine MOVE, affUser is dead </span>b = affUser; </span># Compiler error, affUser is dead </span> </span>sharedU = </span>SHARE</span>(</span>User</span>.new()); </span># turns `affine User` into `shared User` (Arc) </span>c = sharedU; </span># OKAY, sharedU is not dead </span></code></pre> Why it's superior: Zero Blast Radius Refactoring</h4> In Rust, capabilities like Arc</code>, Rc</code>, and Mutex</code> infect function signatures. Changing from Rc<User></code> to Arc<User></code> forces a massive refactor because every function signature and call site must change.</p> In CLEAR, if you need thread-safety, you change one line</strong> at the definition site:</p> # Change multiowned (Rc) to shared (Arc) </span>sharedU = </span>SHARE</span>(</span>User</span>.new()); </span></code></pre> Your functions (which just take User</code>) never knew about the capability, so they don't need to change.</p> Synchronization Strategies</h4> For multi-threaded shared</code> objects, you choose the strategy:</p> shared:read</code>: (MVCC) Optimized for massive read scaling.</li> shared:writeLocked</code>: (RwLock<Arc<T>></code>) Multiple readers OR one writer.</li> shared:locked</code>: (Mutex<Arc<T>></code>) One thread at a time.</li> </ul> Interior Mutability</h4> For complex data, use alwaysMutable</code> (RefCell</code>). CLEAR handles the lock for you:</p> # 99% Case: Compiler handles temporary lock </span>user.login_count += </span>1</span>; </span> </span># 1% Case: Scoped mutation </span>WITH</span> user.config { </span> _.theme = </span>"</span>Light</span>"</span>; </span> _.retries = </span>5</span>; </span>} </span></code></pre> 6. Physics of Sight (SCOPES)</h3> Functions can ONLY see what is explicitly passed into them:</p> x = </span>10</span>; </span>FN</span> add() -> </span> </span>RETURN</span> x + </span>5</span>; </span># COMPILER ERROR: I don't know what 'x' is. </span>END </span></code></pre> Use USE</code> for upvalues:</p> x = </span>10</span>; </span>FN</span> add(n) </span>USE </span>(x) -> </span> </span>RETURN</span> n + x; </span># OK </span>END </span></code></pre> 7. Physics of Time: The ARENA (Lifetimes)</h3> Variable lifetimes follow a simple birth/death cycle: they live as long as the Function they were born in.</p> The ARENA Rule:</h4> When a function starts, it opens a clean ARENA (A bank of memory). Any variable you create lives in this ARENA. When the function ends, the entire ARENA is wiped. POOF</em>.</p> 9. Cheating Death: The TAKES</code> Keyword</h3> In 99% of cases, when you pass a variable to a function, you are just letting that function Borrow</strong> it. If a function needs to store that object in a long-lived structure (like a Tree or Global List), it must explicitly TAKE</strong> responsibility for it.</p> # This function promises to adopt the 'child' </span>FN</span> addChild!(</span>MUTABLE </span>parent: </span>Node</span>, </span>TAKES </span>child: </span>Node</span>) -> </span> parent.list.push!(</span>GIVE</span> child); </span>END </span> </span>node = </span>Node</span>{</span>val: </span>1</span>}; </span>addChild!(root, </span>GIVE</span> node); </span># I surrender ownership. </span>node.print(); </span># COMPILER ERROR: Variable 'node' is dead. </span></code></pre> 10. Simplified Lifetimes: WITH RESTRICT</code></h3> Rust's borrow checker is hard because its side effects are non-local and implicit. In CLEAR, borrows that "poison" (restrict) a mutable variable are explicitly scoped using WITH RESTRICT</code>.</p> MUT</span> node = buildTree(); </span>WITH RESTRICT</span> node.child { </span> </span># Inside this block, node.child is immutable (restricted). </span> gc = node.grandChild(); </span> </span> node.child.name = </span>"</span>OK</span>"</span>; </span># COMPILER ERROR: node.child is RESTRICTed. </span>} </span># Outside the block, node.child is mutable again. </span></code></pre> Path-Based Scoping:</strong> CLEAR allows you to restrict only the specific part of a data structure you are using (e.g., node.child</code>), leaving the rest of the object mutable. This minimizes "poison" and makes complex architectures easier to reason about.</p> This ensures that "poisoning" is always visible and local.</p> OPINIONS 2026-03-31T00:00:00+00:00 Boiler Plate is bad.</li> Understandable code is king. The easier it is to UNDERSTAND</em> code, the more likely it is to arrive at a correct state.</li> Understanding what a function is supposed</em> to do should not be bogged down by it's error handling logic. Like test code, error handling code should be separate as much as possible.</li> Read it as an addendum IFF you care.</li> </ul> </li> Implicit behavior / magic can</em> make things harder</em> to understand.</li> </ul> </li> You should be able to test anything, and it should be EASY. There should be ZERO test code in production code. Java @visibleForTesting is nice, but it shouldn't be necessary at all.</li> </ul> </li> The easier it is to write tests, the more likely you are to actually arrive at working code.</li> </ul> </li> Handling Errors should "just work". You should not always have to checkOk</code>. We can assume</em> okay, and look to the bottom to see what happens when not okay, if we ever care.</li> </ul> </li> The easier it is to handle errors, the more likely you are to handle them correctly!</li> </ul> </li> If your code compiles in STRICT</code> mode, it will not produce a Run-time Error unless the system explicitly encounters a resource boundary, or the program executes an operation whose inputs are logically impossible to resolve correctly. TODO: Investigate Type Coercion Failure claims. Currently not implemented, but on the roadmap.</li> </ul> </li> TODO: Investigate claims of all errors being handled somewhere</em>.</li> </ul> </li> Compiler errors should tell you exactly</em> what's wrong, how to fix it, and be easy to understand.</li> Types should be your friend, helping you write working code faster</em>, not an enemy that is constantly slowing you down.</li> 90% of your time should be spent writing the code you need, and 10% debugging, handling errors, fighting compilers - not the other way around.</li> Writing efficient code should be the default, and the default should be easy. Writing ineffiecent code should be obviously</em> wrong.</li> </ul> </li> Anything that can</em> be 1-line should</em> be! Readability and understandability beat cleverness -- unless direly critical to performance.</li> The constructs and syntax of a language should lend itself to one-liners, as they are often easier to UNDERSTAND</em>.</li> </ul> </li> Code should be as declaractive as possible. Every function should look like a clean chain of exactly what it should</em> be doing. It should not look like a nested mess of how</em> it's handling errors and undesirable states. Though it MUST</em> have the capability to do that.</li> </ul> </li> </ul> </li> </ul> </li> You should not need to worry about memory management OR a garbage collector. Unless you're a rocket scientist, the right compiler can figure it out better than you can.</li> A good SQL engine beats all but the absolute most elite programmers.</li> A good compiler can also take an easy language and beat all but the most elite programmers (see LuaJIT).</li> CLEAR does</em> ask you think about WHERE</em> an object lives. Cache locality is literally 100x faster. If you wrote something cache-locality optimized in Ruby, it would crush a pointer cache miss in C.</li> Therefore, CLEAR is designed around making it as easy as possible to ensure you DON'T cache miss unless you absolutely must. This means you do need to think about if you want something on the STACK (default, fast) or the HEAP (slow, but sometimes required).</li> </ul> </li> </ul> </li> </ul> </li> You should not need to worry about a Global-Interpreter-Lock (GIL). Code should</em> be able to run in parallel or concurrently EFFICIENTLY</em> by default.</li> </ul> </li> Code should be as left-sided as possible. How many hours have YOU</em> spent tracking down paren-syntax errors, figuring out which condition you're in, etc???</li> Time spent figuring out WHERE</em> you even are logically, is time wasted, that could be spent getting things done.</li> </ul> </li> Someone who doesn't know CLEAR</em> should be able to look at CLEAR</em> code and intuit what it does.</li> Publicly-Exported APIs SHOULD</em> have style-enforcements. At a minimum, if you're making a library, anyone should be able to understand the API.</li> All PUBLIC</code> functions MUST</em> either explicitly</em> RAISE</code> an error OR handle all errors. All PUBLIC</code> structs MUST</em> either have a suitable default or !!</code> suffix. All PUBLIC</code> Union Types MUST</em> be projectable for ease of use.</li> </ul> </li> </ul> </li> </ul> </li> Internal-code can be Chill</em>-Correct. No interpretting errors because you have an unnused variable.</li> No forbidding your code to run for a test because you didn't follow a covention, etc.</li> Your code can fail because of your</code> problems, but not becuase of your dependencies problems. STRICT</code> mode compilation eliminates Chill</em>-Correct and can guarantee you won't encounter virtually all preventable run-time errors.</li> </ul> </li> </ul> </li> </ol> WHO IS CLEAR NOT* FOR 2026-03-31T00:00:00+00:00 CLEAR is opinionated. The specific optimizations that make it fast and safe for 99% of Business Logic make it extremely hostile to 1% of Architectural patterns.</p> You are building a Pointer-Heavy Engine (like a Graph Database).</li> </ol> CLEAR prevents Memory Leaks by forbidding reference cycles in shared</code> objects (Shared A -> B, B -> A).</li> If your architecture relies on a "Soup of Mutable Objects" where everything references everything else, CLEAR will fight you.</li> The Alternative:</em> Architect your data using IDs and centralized lookups (like a relational database), or use Rust/C++ for manual pointer management.</li> </ul> You need to model Inherently Unsafe / Cyclic Relationships</li> </ol> If your architecture relies on Rust-style "Weak Pointers" to manage reference cycles (A -> B -> A) OR if you need recursive fine-grained locking, and you are strictly managing memory and deadlock risks manually.</li> </ul> </li> CLEAR guarantees safety by forbidding these patterns entirely! They're rare!</li> The Alternative:</em> If you absolutely need a doubly-linked list or a cyclic graph with individual node locking, that is "Engine Code," not "Business Logic." Write that specific component in Zig (where you can manage the unsafe pointers yourself) and import it into CLEAR as a safe handle.</li> </ul> Modules & Build System 2026-03-31T00:00:00+00:00 CLEAR uses a simple, explicit module system based on the REQUIRE</code> keyword. It separates code into Files</strong>, Packages</strong>, and Native Modules</strong>.</p> REQUIRE — Importing Code</h2> Importing a CLEAR module makes its PUB</code> and package-private symbols available under a namespace.</p> Local File Imports</h3> Use a relative path string to import files in your own project:</p> REQUIRE </span>"</span>math_utils.cht</span>"</span>; </span> </span>FN</span> main() </span>RETURNS Void </span>-> </span> res = math_utils.add(</span>10</span>, </span>20</span>); </span>END </span></code></pre> Package Imports</h3> Use the pkg:</code> prefix to import registered libraries:</p> REQUIRE </span>"</span>pkg:geometry</span>"</span>; </span> </span>FN</span> main() </span>RETURNS Void </span>-> </span> p = geometry.Point{ </span>x: </span>1</span>, </span>y: </span>2 </span>}; </span>END </span></code></pre> Aliasing (AS)</h3> You can rename any import to avoid collisions:</p> REQUIRE </span>"</span>math_utils.cht</span>" </span>AS</span> m; </span>REQUIRE </span>"</span>pkg:geometry</span>" </span>AS</span> geo; </span> </span>FN</span> main() </span>RETURNS Void </span>-> </span> sq = m.square(geo.distance(p1, p2)); </span>END </span></code></pre> Visibility</h2> CLEAR enforces three levels of visibility. Access control is checked at compile-time by the semantic annotator.</p> Keyword</th> Level</th> Visibility</th></tr></thead> PRIVATE</code></td> File Private</td> Only accessible within the .cht</code> file where it is defined.</td></tr> (None)</em></td> Package Private</td> Accessible to any file in the same directory</strong>.</td></tr> PUB</code></td> Public</td> Accessible to any file that REQUIRE</code>s this module or package.</td></tr> </tbody></table> Example</h3> # internal_helper.cht </span> </span>PRIVATE FN</span> secret_calc() </span>RETURNS Int64 </span>-> ... </span>END </span> </span>FN</span> package_helper() </span>RETURNS Int64 </span>-> </span> </span>RETURN</span> secret_calc(); </span># OK: same file </span>END </span> </span>PUB FN</span> public_api() </span>RETURNS Int64 </span>-> </span> </span>RETURN</span> package_helper(); </span># OK: same package </span>END </span></code></pre> # main.cht </span>REQUIRE </span>"</span>internal_helper.cht</span>"</span>; </span> </span>FN</span> main() </span>RETURNS Void </span>-> </span> internal_helper.public_api(); </span># OK: PUB </span> internal_helper.package_helper(); </span># OK: same directory </span> </span># internal_helper.secret_calc(); -- ERROR: PRIVATE </span>END </span></code></pre> Future: STRICT Mode</h3> In future versions (v0.2+), PUB</code> declarations at a package boundary will require explicit effect declarations</strong> (e.g., EFFECTS :alloc</code>) when compiled in STRICT</code> mode. This ensures that library side-effects are always documented and verified.</p> Build System</h2> CLEAR features a built-in dependency manager that handles recursive REQUIRE</code> calls, prevents circular dependencies, and manages the transpilation graph.</p> Package Registration</h3> Packages are mapped to file paths at compile-time using the --pkg</code> flag:</p> ruby</span> src/transpiler.rb</span> --pkg</span> geometry=./libs/geometry/lib.cht main.cht > main.zig </span></code></pre> Module Transpilation</h3> To compile a library as a standalone Zig module (without the CLEAR runtime footer), use the --module</code> flag. This is useful when building CLEAR code to be consumed by native Zig applications.</p> ruby</span> src/transpiler.rb</span> --module</span> my_lib.cht > my_lib.zig </span></code></pre> How it Works</h3> Local Imports</strong>: The transpiler inlines the generated Zig code of the local file into a const struct</code> namespace in the caller's Zig file.</li> Package Imports</strong>: The transpiler emits a Zig @import("namespace")</code>. The actual wiring of these modules is handled by the Zig build system using the --dep</code> and -M</code> flags.</li> Deduplication</strong>: Each file is parsed and annotated exactly once, even if required by multiple files.</li> </ol> Project Structure (Recommended)</h2> my_project/ </span>├── main.cht # Entry point </span>├── logic.cht # Local module </span>└── lib/ </span> └── math/ </span> ├── lib.cht # Package entry point (exports PUB symbols) </span> └── private.cht # Internal logic (default visibility) </span></code></pre> CLEAR Language Walkthrough 2026-03-31T00:00:00+00:00 This guide showcases CLEAR: a memory-safe language that combines the ergonomics of scripting with the safety of affine types and the performance of hand-optimized C.</p> 1. Immutability & Mutability</h2> Bindings are immutable by default. Reassignment requires the MUTABLE</code> keyword.</p> x = </span>5</span>; </span># OKAY: Immutable binding (default) </span>name = </span>"</span>Alice</span>"</span>; </span># OKAY: Immutable string </span>pi = </span>3.14159</span>; </span># OKAY: Immutable float </span> </span>x = </span>6</span>; </span># COMPILER ERROR: x is immutable </span> </span>MUTABLE</span> counter = </span>0</span>; </span># OKAY: Explicit mutability </span>counter = </span>1</span>; </span># OKAY: can reassign mutable binding </span></code></pre> 2. Primitive Types</h2> CLEAR provides a comprehensive set of primitives for precise control over memory.</p> Type</th> Description</th> Example</th></tr></thead> Int8</code> .. Int64</code></td> Signed integers (8, 16, 32, 64-bit)</td> 42_i64</code>, -1_i8</code></td></tr> UInt8</code> .. UInt64</code></td> Unsigned integers (8, 16, 32, 64-bit)</td> 42_u64</code>, -1_u8</code></td></tr> Float32</code>, Float64</code></td> IEEE-754 floating point</td> 3.14159_f64</code>, 1.0_f32</code></td></tr> Bool</code></td> Boolean logic</td> TRUE</code>, FALSE</code></td></tr> Byte</code></td> Raw 8-bit data</td> 0x41_b</code></td></tr> String</code></td> UTF-8 encoded text (affine)</td> "hello"</code></td></tr> Void</code></td> Absence of a value</td> RETURN;</code></td></tr> </tbody></table> 3. Types vs Capabilities</h2> CLEAR distinguishes between what data is</em> (Type) and how it is accessed</em> (Capability). In Rust, capabilities like Arc</code>, Rc</code>, and Mutex</code> are part of the type, leading to "function coloring" and high refactoring costs. In CLEAR, functions take Types</strong>, not Capabilities</strong>.</p> Capability Annotations</h3> Capabilities are applied at the declaration site</strong> with @</code> suffixes:</p> Capability</th> Rust Equivalent</th> CLEAR Syntax</th></tr></thead> Single-threaded shared ownership</td> Rc<T></code></td> value @multiowned</code></td></tr> Multi-threaded shared ownership</td> Arc<T></code></td> value @shared</code></td></tr> Mutex (exclusive lock)</td> Arc<Mutex<T>></code></td> value @shared:locked</code></td></tr> RwLock (read-write lock)</td> Arc<RwLock<T>></code></td> value @shared:writeLocked</code></td></tr> Heap pointer</td> Box<T></code></td> value @indirect</code></td></tr> Thread-local pointer</td> (no direct equivalent)</em></td> value @local</code></td></tr> </tbody></table> Zero Blast Radius Refactoring</h3> In Rust, changing Rc<User></code> to Arc<User></code> means rewriting every function signature in the call chain. In CLEAR, functions take the plain type:</p> STRUCT User </span>{ </span>name: </span>String } </span> </span>FN</span> process(</span>u: </span>User</span>) </span>RETURNS Void </span>-> </span> print(u.name); </span> </span>RETURN</span>; </span>END </span> </span>FN</span> main() </span>RETURNS Void </span>-> </span> </span># At the call site, capabilities are unwrapped: </span> shared_u = </span>User</span>{ </span>name: </span>"</span>Alice</span>" </span>} </span>@shared</span>; </span> </span>WITH</span> shared_u </span>AS</span> val { process(val); } </span> </span>RETURN</span>; </span>END </span></code></pre> If you change @shared</code> to @multiowned</code>, the process</code> function remains untouched. The refactor is a one-line change at the declaration.</p> WITH Blocks for Capability Unwrapping</h3> For multi-statement access, use WITH</code> blocks:</p> MUTABLE </span>counter: </span>Int64@shared</span>:locked </span>= </span>0</span>; </span> </span>WITH EXCLUSIVE</span> counter </span>AS</span> c { </span> c += </span>1</span>; </span> print(c.toString()); </span>} </span></code></pre> 4. Affine Ownership: GIVE & TAKES</h2> CLEAR uses affine types</strong> by default. Every value has exactly one owner. When you assign a value, ownership is moved</strong>, not copied.</p> FN</span> process(</span>TAKES </span>s: </span>String) </span>RETURNS Void </span>-> </span> print(s); </span> </span># s is destroyed here (end of scope) </span>END </span> </span>FN</span> main() </span>RETURNS Void </span>-> </span> msg = </span>"</span>Hello</span>"</span>; </span> </span> process(msg); </span># OKAY: Implicit transfer (by FN signature) </span> </span> print(</span>GIVE</span> msg); </span># COMPILER ERROR: USE AFTER MOVE: You can't GIVE `msg`. `process(msg)` TOOK it away. </span> print(msg); </span># COMPILER ERROR: USE AFTER MOVE: You can't use `msg`. `process(msg)` TOOK it away. </span> </span> </span>RETURN</span>; </span>END </span></code></pre> For more details, see docs/sharing-capabilities.md</a>.</p> Parameter Ownership</h3> Function parameters are implicit borrows</strong> by default. Only TAKES</code> transfers ownership.</p> Param Style</th> Ownership</th> Can Move Into Container?</th> Can Mutate?</th></tr></thead> v: Value</code></td> Borrow</td> No</td> No</td></tr> MUTABLE v: Value</code></td> Borrow</td> No</td> Yes</td></tr> TAKES v: Value</code></td> Owned</td> Yes</td> No</td></tr> TAKES MUTABLE v: Value</code></td> Owned</td> Yes</td> Yes</td></tr> </tbody></table> UNION Value </span>{ </span>Nil</span>, </span>Num: </span>Float64</span>, </span>Lambda </span>{ </span>body: </span>Value @indirect </span>} } </span> </span># Borrow: can read, cannot store or move </span>FN </span>inspect(</span>v: </span>Value</span>) </span>RETURNS </span>String -> </span> </span>MATCH</span> v </span>START </span> </span>Value</span>.Num </span>AS</span> n -> </span>RETURN</span> n.toString();, </span> </span>DEFAULT </span>-> </span>RETURN </span>"</span>other</span>"</span>; </span> </span>END </span>END </span> </span># Owned: can store into collections </span>FN</span> store!(</span>TAKES </span>v: </span>Value</span>, </span>MUTABLE </span>map: </span>HashMap</span><</span>Value</span>>) </span>RETURNS Void </span>-> </span> map[</span>"</span>item</span>"</span>] = v; </span># OKAY: v is owned via TAKES </span> </span>RETURN</span>; </span>END </span> </span>FN</span> bad!(</span>v: </span>Value</span>, </span>MUTABLE </span>map: </span>HashMap</span><</span>Value</span>>) </span>RETURNS Void </span>-> </span> map[</span>"</span>item</span>"</span>] = v; </span># COMPILER ERROR: Cannot move borrowed value 'v' </span> </span>RETURN</span>; </span>END </span></code></pre> Zero implicit copies.</strong> Copy types (primitives, strings, enums) can be freely used after assignment. Non-Copy types (unions with @indirect</code> or []T</code> variants, structs with heap data) follow move semantics. There are never implicit deep copies.</p> 5. Sharded Shared-Nothing Architecture</h2> The @sharded</code> capability partitions data across threads, enabling massive parallelism without lock contention by automatically pinning threads to specific data shards.</p> # A sharded map distributes keys across independent thread-local heaps </span>MUTABLE </span>registry: </span>HashMap</span><</span>Int64</span>></span>@sharded</span>(</span>8</span>) = {}; </span> </span># CLEAR automatically pins this fiber to the correct shard </span>BG </span>{ </span> registry[</span>"</span>key</span>"</span>] = </span>42</span>; </span>} </span> </span># Sharding is also available for Pools and Lists </span>MUTABLE </span>users: </span>User</span>[</span>100</span>]</span>@pool</span>:sharded</span>(</span>4</span>) = []; </span>MUTABLE </span>logs: </span>String[]</span>@list</span>:sharded</span>(</span>2</span>) = []; </span> </span># Note: Sharding provides peak throughput but carries a risk of </span># data skew if keys/items are not uniformly distributed. </span></code></pre> Shared-nothing architectures present problems if your workloads can be heavily skewed.</p> </div> 6. Function Signatures</h2> Functions support explicit types, failable returns (!T</code>), and optional types (?T</code>).</p> STRUCT Point </span>{ </span>x: </span>Float64</span>, </span>y: </span>Float64 </span>} </span> </span>FN</span> sum(</span>p: </span>Point</span>) </span>RETURNS Float64 </span>-> </span> </span>RETURN </span>p.x + p.y; </span>END </span> </span>FN</span> main() </span>RETURNS Void </span>-> </span> p = </span>Point</span>{ </span>x: </span>3.0</span>, </span>y: </span>4.0 </span>}; </span> d = sum(p); </span> </span>RETURN</span>; </span>END </span></code></pre> Failable and optional returns:</p> FN</span> findUser(</span>id: </span>Int64</span>) </span>RETURNS </span>!?User -> </span> </span>IF</span> id < </span>0 </span>-> </span>RAISE </span>"</span>Invalid ID</span>"</span>; </span> </span>RETURN</span> result; </span>END </span></code></pre> Recursion</h3> Recursive functions must be explicitly annotated with @reentrant</code>:</p> FN</span> fib(</span>n: </span>Int64</span>) </span>RETURNS Int64 @reentrant </span>-> </span> </span>IF</span> n <= </span>1 </span>-> </span>RETURN</span> n; </span> </span>RETURN</span> fib(n - </span>1</span>) + fib(n - </span>2</span>); </span>END </span></code></pre> 7. Basic Control Flow</h2> CLEAR provides standard control flow constructs with support for one-line shorthands.</p> # 1. IF / ELSE_IF / ELSE </span>x = </span>75</span>; </span>IF</span> x > </span>100 </span>THEN </span> print(</span>"</span>Large</span>"</span>); </span>ELSE_IF</span> x > </span>50 </span>THEN </span> print(</span>"</span>Medium</span>"</span>); </span>ELSE </span> print(</span>"</span>Small</span>"</span>); </span>END </span> </span># 2. WHILE loops </span>MUTABLE</span> i = </span>0</span>; </span>WHILE</span> i < </span>10 </span>DO </span> print(i.toString()); </span> i += </span>1</span>; </span>END </span> </span># 3. FOR loops (Range iteration) </span>FOR</span> j </span>IN </span>(</span>1 </span>..= </span>5</span>) -> print(j.toString()); </span># OKAY: Inclusive range </span>FOR</span> j </span>IN </span>(</span>1 </span>..< </span>5</span>) -> print(j.toString()); </span># OKAY: Exclusive range </span></code></pre> 8. Enums, Unions, and Pattern Matching</h2> Enums</h3> Simple enumerations for discrete states:</p> ENUM Direction </span>{ </span>North</span>, </span>South</span>, </span>East</span>, </span>West </span>} </span> </span>FN</span> describe(</span>d: </span>Direction</span>) </span>RETURNS </span>String -> </span> </span>MATCH</span> d </span>START </span> </span>Direction</span>.North -> </span>RETURN </span>"</span>up</span>"</span>;, </span> </span>Direction</span>.South -> </span>RETURN </span>"</span>down</span>"</span>;, </span> </span>Direction</span>.East -> </span>RETURN </span>"</span>right</span>"</span>;, </span> </span>Direction</span>.West -> </span>RETURN </span>"</span>left</span>"</span>; </span> </span>END </span>END </span></code></pre> Tagged Unions (Sum Types)</h3> Unions carry a payload per variant. Unit variants (no payload) are also supported:</p> UNION Result </span>{ </span>Ok: </span>Float64</span>, </span>Err: </span>String, </span>Empty </span>} </span> </span>FN</span> main() </span>RETURNS Void </span>-> </span> </span># Payload variant </span> r = </span>Result</span>{ </span>Ok: </span>42.0 </span>}; </span> </span> </span># Unit variant (no payload) </span> e = </span>Result</span>.Empty; </span> </span> </span>MATCH</span> r </span>START </span> </span>Result</span>.Ok -> print(</span>"</span>success</span>"</span>);, </span> </span>Result</span>.Err -> print(</span>"</span>error</span>"</span>);, </span> </span>Result</span>.Empty -> print(</span>"</span>nothing</span>"</span>); </span> </span>END </span> </span>RETURN</span>; </span>END </span></code></pre> MATCH AS and Ownership</h3> MATCH ... AS</code> payload extraction follows the same ownership rules as parameters. The binding is a borrow</strong> of the source by default, or a move</strong> if the source is owned.</p> Source Ownership</th> MATCH AS Produces</th> Can Store in Struct?</th></tr></thead> Borrowed (parameter, map.get</code>)</td> Borrow</td> No</td></tr> Owned (local, TAKES param)</td> Move (consumes source)</td> Yes</td></tr> </tbody></table> UNION Value </span>{ </span>Nil</span>, </span>Num: </span>Float64</span>, </span>List: </span>Value</span>[] } </span> </span># Borrowed source: MATCH AS produces a borrow </span>FN</span> sum(</span>v: </span>Value</span>) </span>RETURNS Float64 </span>-> </span> </span>MATCH</span> v </span>START </span> </span>Value</span>.List </span>AS</span> items -> </span># items is &[]Value (borrow) </span> </span>RETURN</span> items.length(); </span># OKAY: reading a borrow </span> </span>DEFAULT </span>-> </span>RETURN </span>0.0</span>; </span> </span>END </span>END </span> </span># Owned source: MATCH AS moves the payload </span>FN</span> take!(</span>TAKES </span>v: </span>Value</span>) </span>RETURNS Value</span>[] -> </span> </span>MATCH</span> v </span>START </span> </span>Value</span>.List </span>AS</span> items -> </span># items is []Value (owned, v consumed) </span> </span>RETURN</span> items; </span># OKAY: returning owned data </span> </span>DEFAULT </span>-> </span>RETURN List</span>[]; </span> </span>END </span>END </span></code></pre> Storing a borrowed MATCH AS</code> binding into a struct or union variant is a compile error:</p> FN</span> bad(</span>items: </span>Value</span>[]) </span>RETURNS Value </span>-> </span> </span># items is a borrowed parameter </span> </span>RETURN Value</span>.Lambda{ </span>params:</span> items }; </span># COMPILER ERROR: Cannot store borrowed 'items' </span>END </span></code></pre> Generics</h3> Structs and unions support type parameters:</p> STRUCT Pair</span><</span>T</span>> { </span>first: </span>T</span>, </span>second: </span>T </span>} </span> </span>p = </span>Pair</span><</span>Int64</span>>{ </span>first: </span>1</span>, </span>second: </span>2 </span>}; </span></code></pre> 9. Higher-Order Functions & Error Handling</h2> CLEAR supports powerful functional pipelines via the Smooth operator \|></code>.</p> # 1. Pipelines: Filter, Aggregate, Transform </span>alive = entities \|> </span>WHERE</span> _.health > </span>0</span>; </span>total = scores \|> </span>SUM</span> _.value; </span>names = users \|> </span>SELECT</span> _.name; </span> </span># 2. Side effects & Function Piping </span>entities \|> </span>EACH </span>{ _.x += _.vx; }; </span>result = data \|> process \|> validate \|> format; </span> </span># 3. Error Handling: Inline OR / OR RAISE </span>val = parseInt(</span>"</span>abc</span>"</span>) </span>OR </span>0</span>; </span># OKAY: Fallback value </span>content = readFile(</span>"</span>config.json</span>"</span>) </span>OR RAISE</span>; </span># OKAY: Explicit propagation </span> </span># 4. Function-level CATCH </span>FN</span> main() </span>RETURNS Void </span>-> </span> result = loadConfig(</span>"</span>config.json</span>"</span>) </span>OR RAISE</span>; </span> print(</span>"</span>Config: ${result}</span>"</span>); </span> </span>RETURN</span>; </span>CATCH</span> e </span> print(</span>"</span>Failed to load: ${e}</span>"</span>); </span> </span>RETURN</span>; </span>END </span></code></pre> See docs/pipelines.md#operators</a> for a full list of higher-order function operators.</p> Pipelines are automatically fused for efficiency.</p> 10. Time as Tense (~T)</h2> Tense represents a value that will exist in the future. CLEAR eliminates the complexity of Future/Promise/Observable</code> with a single unified tense.</p> ~User</code> is read as "Future User"</strong>.</li> ~User[]</code> is read as "Future Users"</strong>.</li> A STREAM</strong> of users is simply one way to produce "Future Users".</li> </ul> # 1. Promise (~T): A single future value </span>p: </span>~</span>String </span>= </span>BG </span>{ sleep(</span>100</span>); </span>RETURN </span>"</span>Data</span>"</span>; }; </span>val = </span>NEXT </span>p; </span># OKAY: Blocks until ready </span> </span># 2. Open Stream (~?T[]): Asynchronous generator </span># NEXT on ~?T[] returns ?T, with NIL signaling exhaustion. </span>gen: </span>~?Int64[] = </span>BG STREAM </span>{ </span> </span>YIELD </span>10</span>; </span> </span>YIELD </span>20</span>; </span>}; </span>val = </span>NEXT</span> gen; </span># OKAY: Returns ?Int64 (NIL when exhausted) </span> </span># 3. Infinite Stream (~T[INF]): Lazy rendezvous generator </span>counter: </span>~</span>Int64</span>[</span>INF</span>] = </span>BG STREAM </span>{ </span> </span>MUTABLE</span> i = </span>0</span>; </span> </span>WHILE TRUE DO </span>{ </span>YIELD</span> i; i += </span>1</span>; } </span>}; </span>v1 = </span>NEXT</span> counter; </span># OKAY: Returns Int64 (never NIL) </span></code></pre> Stream Capabilities: @shared</code> vs @split</code></h3> Streams can also carry sharing capabilities:</p> @shared</code> means the stream handle may be shared across threads, but it is not ordered replay. Multiple threads calling NEXT</code> are sharing access to one producer.</li> @split</code> means ordered fanout / pub-sub semantics. Each cloned handle sees the same items in the same order, and memoized items are released once every live split owner has consumed them.</li> </ul> source: </span>~?Int64[]@split = </span>BG STREAM </span>{ </span> </span>YIELD </span>10</span>; </span> </span>YIELD </span>20</span>; </span>}; </span> </span>a: </span>~?Int64[]@split = </span>CLONE</span> source; </span>b: </span>~?Int64[]@split = </span>CLONE</span> source; </span> </span>ASSERT NEXT</span> a == </span>10</span>, </span>"</span>first subscriber sees first value</span>"</span>; </span>ASSERT NEXT</span> b == </span>10</span>, </span>"</span>second subscriber sees the same first value</span>"</span>; </span>ASSERT NEXT</span> a == </span>20</span>, </span>"</span>order is preserved</span>"</span>; </span>ASSERT NEXT</span> b == </span>20</span>, </span>"</span>order is preserved for all subscribers</span>"</span>; </span></code></pre> CLONE</code> is mandatory for @split</code> streams. Plain assignment moves the handle.</p> This is the model CLEAR intends for pub/sub style workloads. For a concrete benchmark in this area, see benchmarks/16_pubsub/bench.cht</a>. That benchmark still uses the older sharing path today, but it is the right benchmark to watch as @split</code> becomes the native pub/sub primitive.</p> 11. Collections: Array, List, and Pool</h2> All collections are automatically monomorphized</strong> -- the compiler generates zero-overhead, type-specific native code for every unique T</code>.</p> Sigil</th> Collection</th> Purpose</th></tr></thead> T[N]</code></td> Array</code></td> Fixed-size, stack-allocated (if small)</td></tr> T[]@list</code></td> List</code></td> Dynamic-size, heap-backed</td></tr> T[N]@pool</code></td> Pool</code></td> Fixed-capacity, generational handles</td></tr> T[]@set</code></td> Set</code></td> Dynamic-size, heap-backed, distinct items</td></tr> T[]@ring</code></td> Ring</code></td> Circular Ring Buffer (mainly for Streams - v0.2)</td></tr> </tbody></table> STRUCT User </span>{ </span>name: </span>String } </span> </span># 1. Fixed Array </span>vals = [</span>10</span>, </span>20</span>, </span>30</span>]; </span> </span># 2. Dynamic List </span>MUTABLE </span>items: </span>Int64</span>[]</span>@list </span>= []; </span>items.append(</span>42</span>); </span> </span># 3. Generational Pool </span># Pools provide peak cache locality. Switching from List to @pool:soa </span># (Structure of Arrays) is a one-line refactor for massive speed. </span>MUTABLE </span>users: </span>User</span>[</span>100</span>]</span>@pool </span>= []; </span> </span>id = users.insert(</span>User</span>{ </span>name: </span>"</span>Alice</span>" </span>}); </span># Returns stable handle </span>user = users.get(id) </span>OR RAISE</span>; </span># Returns ?T (checks stale handles) </span></code></pre> Element-Level Capabilities</h3> Capabilities normally apply to the collection</strong> (T[]@shared</code> = one shared list). For rare cases where each element</strong> needs its own capability, place the capability before the array suffix:</p> # Collection-level (common): one Arc wrapping the whole list </span>MUTABLE </span>shared_list: </span>User</span>[]</span>@list</span>:shared </span>= []; </span> </span># Element-level (rare): each element is individually Arc'd </span>MUTABLE </span>arc_users: </span>User@shared</span>[]</span>@list </span>= []; </span> </span># Element-level with pool: each slot holds an Arc'd user </span>MUTABLE </span>arc_pool: </span>User@shared</span>[</span>100</span>]</span>@pool </span>= []; </span></code></pre> Element-level capabilities are primarily useful in struct fields where individual elements need independent lifetime management (e.g., a graph where each node is shared across multiple edges).</p> 12. Strings and Buffers</h2> Strings in CLEAR are Copy (like Rust's &str</code> — a pointer + length). Assignment copies the slice header.</p> x = </span>"</span>hello</span>"</span>; </span>y = x; </span># x is moved (strings are owned, non-Copy) </span># z = x; # would be use-after-move error </span>z = </span>COPY</span> y; </span># OK: explicit deep-copy </span> </span># String concatenation </span>full = </span>"</span>foo</span>" </span>+ </span>"</span>bar</span>"</span>; </span># "foobar" (single allocation, no intermediate) </span></code></pre> Like arrays, Strings have capabilities:</p> String@raw</code> - raw byte buffer; no UTF-8 encoding assumption, O(1) byte indexing, zero-copy slicing. Use for binary data, network buffers, parsers operating on bytes.</li> String@symbol</code> - compile-time interned identifier. Literals use Ruby-style :ok</code> syntax. The compiler embeds the string in read-only static memory and deduplicates identical literals, so equality is O(1) pointer comparison and the value needs no cleanup (program-lifetime). Use for status tags, option names, event names, and map keys where the set of values is known at compile time. Note: interned HashMaps carry their own per-map symbol table for runtime string keys - this avoids a globally locked intern table, which would create a contention hotspot incompatible with CLEAR's scalability goals.</li> String@ring</code> (v0.2) - circular buffer for streaming.</li> </ul> 13. Graphs, Cycles, and @indirect</h2> For recursive or cyclic data structures, use @indirect</code> (heap-allocated pointer, like Rust's Box<T></code>). Combined with @reentrant</code> for recursive traversal:</p> # Recursive tree node using @indirect for child pointers </span>STRUCT Node </span>{ </span> </span>value: </span>Int64</span>, </span> </span>left: </span>?Node@indirect, </span># Optional heap-allocated child </span> </span>right: </span>?Node@indirect </span>} </span> </span># Recursive traversal must be @reentrant </span>FN</span> sumTree(</span>n: </span>Node</span>) </span>RETURNS Int64 @reentrant </span>-> </span> </span>MUTABLE</span> total = n.value; </span> </span>IF</span> n.left -> total += sumTree(n?.left </span>OR </span>0</span>); </span> </span>IF</span> n.right -> total += sumTree(n?.right </span>OR </span>0</span>); </span> </span>RETURN</span> total; </span>END </span></code></pre> @indirect</code> gives the node a stable heap address, enabling graph structures. @multiowned</code> (Rc) enables shared ownership for DAGs. For cyclic graphs, use @link</code> -- CLEAR's weak reference.</p> ?.</code> is the safe navigate operator to peek into optional types. It combines with OR</code> to handle missing data like an error.</p> Weak References with @link</h3> @link</code> creates a weak reference that does not keep the target alive. It works with both @multiowned</code> (WeakRc) and @shared</code> (WeakArc).</p> LINK expr</code> -- downgrade a strong reference to a weak reference</li> RESOLVE expr</code> -- upgrade a weak reference back to an optional strong reference (?T</code>)</li> </ul> # Parent-child with back-pointer cycle </span>STRUCT Parent </span>{ </span> </span>name: </span>String, </span> </span>child: </span>?Child@multiowned@indirect </span>} </span> </span>STRUCT Child </span>{ </span> </span>name: </span>String, </span> </span>parent: </span>?Parent@link </span># weak back-pointer, breaks the cycle </span>} </span> </span>FN</span> main() </span>RETURNS Void </span>-> </span> p = </span>Parent</span>{ </span>name: </span>"</span>Alice</span>"</span>, </span>child: </span>NIL </span>} </span>@multiowned</span>; </span> </span> </span># LINK downgrades the strong Rc to a WeakRc </span> weak_p = </span>LINK </span>p; </span> </span> </span># RESOLVE returns ?Parent@multiowned; bind first, then use ?. to safely access fields </span> resolved = </span>RESOLVE</span> weak_p; </span> name = resolved?.name </span>OR </span>"</span>dropped</span>"</span>; </span> </span>ASSERT </span>name == </span>"</span>Alice</span>"</span>, </span>"</span>resolved</span>"</span>; </span> </span> </span>RETURN</span>; </span>END </span></code></pre> Key rules:</p> LINK</code> only works on @multiowned</code> or @shared</code> values (compile-time error otherwise)</li> RESOLVE</code> only works on @link</code> values (compile-time error otherwise)</li> RESOLVE</code> returns ?T</code> -- bind the result first, then use resolved?.field OR fallback</code> to handle the dropped case</li> Cleanup is automatic: weak references are released when they go out of scope</li> No runtime cost when the strong reference is still alive; RESOLVE is a simple count check</li> </ul> 14. Concurrency: BG & DO</h2> CLEAR makes background tasks and fork-join parallelism trivial.</p> # BG: Background execution </span>p: </span>~</span>Int64 </span>= </span>BG </span>{ </span>RETURN</span> slowComputation(); }; </span> </span># DO: Fork-Join parallel execution </span>DO </span>{ </span> step1(), </span> step2() </span>} </span> </span># CONCURRENT: Parallel pipelines </span>results = items \|> </span>CONCURRENT</span>(</span>workers: </span>8</span>) </span>SELECT</span> transform(_); </span></code></pre> BG Stack Modifiers</h3> Modifier</th> Stack Size</th> Use</th></tr></thead> @micro</code></td> 4 KB</td> Lightweight tasks</td></tr> @standard</code></td> 16 KB</td> Default</td></tr> @large</code></td> 64 KB</td> File I/O, deep recursion</td></tr> @xl</code></td> 256 KB</td> Very deep call stacks</td></tr> @pinned</code></td> (any)</td> Pin to local scheduler</td></tr> @arena</code></td> (any)</td> Thread-local arena; implies @pinned</td></tr> </tbody></table> 15. Modules & Imports</h2> CLEAR uses a simple namespace-based module system via REQUIRE</code>.</p> REQUIRE </span>"</span>math_utils.cht</span>" </span>AS</span> m; </span># Local file alias </span>REQUIRE </span>"</span>pkg:geometry</span>"</span>; </span># Package import </span> </span>FN</span> main() </span>RETURNS Void </span>-> </span> p = geometry.Point{ </span>x: </span>1</span>, </span>y: </span>2 </span>}; </span> print(m.square(</span>10</span>).toString()); </span> </span>RETURN</span>; </span>END </span></code></pre> See docs/modules.md</a> for visibility (PUB</code>/PRIVATE</code>) and build details.</p> 16. FFI: Native Integration</h2> CLEAR integrates directly with Zig and C libraries via EXTERN</code> declarations. All EXTERN FN calls automatically trampoline to the OS stack (g0) for fiber safety.</p> Importing Functions and Types</h3> # Import a struct type from a Zig module </span>EXTERN STRUCT Vec2 </span>{ </span>x: </span>Float64</span>, </span>y: </span>Float64 </span>} </span>FROM </span>"</span>math_native</span>"</span>; </span> </span># Import a free function </span>EXTERN FN</span> computeDistance(</span>v: </span>Vec2</span>) </span>RETURNS Float64 FROM </span>"</span>math_native</span>"</span>; </span> </span># Import with allocator injection (EFFECTS) </span>EXTERN FN</span> parseJson(</span>data: </span>String) </span>RETURNS </span>!</span>JsonDoc </span> </span>EFFECTS </span>:alloc:heap </span>FROM </span>"</span>json_native</span>"</span>; </span></code></pre> Local Struct Definitions (no FROM)</h3> For passing default options or defining Zig-compatible layouts:</p> # Local struct (no external module) </span>EXTERN STRUCT ParseOptions </span>{}; </span>EXTERN STRUCT JsonRecord </span>{ </span>id: </span>Int64</span>, </span>data: </span>Int64</span>[] }; </span> </span># Comptime type parameters for generic FFI </span>EXTERN FN</span> parseFromSliceLeaky<</span>T</span>>(</span>comptime: </span>T</span>, </span>content: </span>String, </span>options: </span>ParseOptions</span>) </span> </span>RETURNS </span>!</span>T EFFECTS </span>:alloc:heap </span>FROM </span>"</span>std.json</span>"</span>; </span></code></pre> Method Calls on EXTERN Structs</h3> EXTERN STRUCT Dir </span>{} </span>FROM </span>"</span>std.fs</span>"</span>; </span>EXTERN FN</span> cwd() </span>RETURNS Dir FROM </span>"</span>std.fs</span>"</span>; </span>EXTERN FN Dir</span>.makePath(</span>self: </span>Dir</span>, </span>path: </span>String) </span>RETURNS Void FROM </span>"</span>std.fs</span>"</span>; </span> </span># Chained method call (both trampolined to g0) </span>cwd().makePath(</span>"</span>data</span>"</span>); </span></code></pre> EXTERN STRUCT CLOSE (RAII)</h3> EXTERN STRUCT Buffer </span>{ </span>data: </span>String } </span> </span>CLOSE </span>"</span>deinit</span>" </span>FROM </span>"</span>native_resource</span>"</span>; </span> </span># Buffer.deinit() is auto-called when buf goes out of scope </span>buf = createBuffer(</span>"</span>hello</span>"</span>); </span># defer buf.deinit() emitted automatically </span></code></pre> See json_api</a> for an example.</li> See docs/agents/ffi.md</a> for the complete FFI guide.</li> </ul> 17. Memory Model</h2> Arena Allocation</h3> Every function has its own memory arena. Variables live as long as the function they were born in.</p> Stack</strong> (~0ns): Primitives and small structs (up to 128 slots)</li> Frame Arena</strong> (~2ns): Large structs, temporary buffers, string concat</li> Heap</strong> (~60ns): Dynamic collections, cross-fiber data, Rc/Arc</li> </ul> Escaping the Arena</h3> RETURN</code>: Values stay in the caller's frame arena (no heap dupe for strings).</li> GIVE</code>: Ownership is transferred to the callee.</li> @shared</code> / @multiowned</code>: Reference-counted objects that live as long as they are referenced.</li> </ul> Loop Arena Rewind</h3> Inside loops, the frame arena is rewound each iteration to prevent unbounded growth. Mutable variables that accumulate across iterations (like string concatenation) are automatically detected and excluded from the rewind.</p> CLEAR additionally inserts cooperative yielding into loops to prevent head-of-line blocking, etc.</p> This destroys SIMD optimizations. CLEAR can detect when a SIMD optimization is possible and warn you to use TIGHT</code> loops:</p> TIGHT FOR</span> i </span>IN </span>(</span>1</span>..<</span>1000</span>) -> ... </span></code></pre> CLEAR will also warn you when you're using a TIGHT</code> loop and you should not.</p> Misusing TIGHT</code> loops can destroy your p99 latency.</p> </div> 18. The Reality of Concurrency</h2> In an ideal world, every workload would distribute perfectly across available cores and memory. In reality, concurrency is difficult because of the "messy middle":</p> Skew:</strong> Data is rarely uniform. Sharded collections can develop "hot shards" that bottleneck the system.</li> Outliers:</strong> The 0.1% of workers that take 100x longer than typical (due to cache misses or deep recursion) destroy p99 response times.</li> Non-Cooperation:</strong> Problems like head-of-line blocking and thundering herds can stall an entire thread pool.</li> Variadic Depth:</strong> Real-world recursion and loops often exceed the fixed stack sizes of traditional fibers.</li> </ol> CLEAR handles these issues through its Control Plane</strong> -- an active runtime observer that uses live telemetry to manage execution.</p> Fiber Overflow:</strong> The runtime detects imminent stack overflows and auto-upsizes future tasks to @large</code> or @xl</code> stacks.</li> Workload Mismatch (v0.2):</strong> Telemetry-driven alerting when the runtime detects contented or skewed workloads. ./clear profile</code> can automatically suggest fixes.</li> Shared-Nothing Safety:</strong> By enforcing sharding and affine moves at the language level, the Control Plane can optimize memory layout without fear of data races.</li> Workload Migration (v0.4+):</strong> Telemetry-driven migration allows the runtime to detect contention and skew in real-time and react to it, IF</em> you have the protection enabled (added memory).</li> </ul> See docs/control-plane.md</a> for more on how CLEAR manages the reality of high-performance systems.</p> 19. Function Lifetimes</h2> Functions cannot return borrowed data freely, but can with a lifetime. This is simplified from Rust.</p> STRUCT Node </span>{ </span>left: </span>?Node, </span>right: </span>?Node } </span> </span>FN</span> grandChild(</span>n: </span>Node</span>) </span>RETURNS</span> n.left:?Node -> </span> </span>RETURN</span> n.left?.right; </span>END </span></code></pre> Lifetimes are scoped with <param>.<path>:Type</code>. In the example above, n.left</code> is the path and ?Node</code> is the return type.</p> This is a less restrictive lifetime than r:Bar</code>.</p> Standard IMMUTABLE</code> returned borrows just work</em>:</p> bar = root.identity(); </span>print(bar.index); </span></code></pre> MUTABLE</code> returned borrows require a scope - because the object you're borrowing from is restricted while you hold the borrow to prevent use-after-free:</p> WITH RESTRICT</span> node </span>AS</span> n { </span> </span>MUTABLE</span> gc = n.grandChild(); </span> gc.value = </span>0</span>; </span> node.left = </span>Nil</span>; </span># COMPIILER ERROR: `node` is RESTRICTED. It's immutable in this scope. </span> n.left = </span>Nil</span>; </span># COMPIILER ERROR: `n` is RESTRICTED. It's immutable in this scope. </span>} </span></code></pre> 20. STRUCT lifetimes</h2> It's sometimes useful to define a STRUCT with a borrowed field (iterators are a prime example):</p> STRUCT SliceIter </span>{ </span> </span>source: </span>BORROWED Int64</span>[], </span># Allows `source` to be a BORROW, must be initialized in a scope. </span> </span>pos: </span>Int64</span>, </span> </span>len: </span>Int64 </span>} </span> </span>FN</span> hasNext(</span>iter: </span>SliceIter</span>) </span>RETURNS Bool </span>-> </span> </span>RETURN</span> iter.pos < iter.len; </span>END </span> </span>FN</span> iterExample() </span>RETURNS Void </span>-> </span> </span>WITH BORROWED</span> data </span>AS</span> ref { </span># create a borrowed version </span> </span>MUTABLE</span> iter = </span>SliceIter</span>{ </span>source:</span> ref, </span>pos: </span>0</span>, </span>len:</span> n }; </span> </span>WHILE</span> hasNext(iter) </span>DO </span> total += currentVal(iter); </span> iter = advance(iter); </span> </span>END </span> </span>RETURN</span> iter; </span># COMPILER ERROR: You can't return `iter`. It's BORROWED. You don't own it. </span> } </span>END </span></code></pre> Quick Reference: Capabilities</h2> Annotation</th> Purpose</th></tr></thead> @multiowned</code></td> Single-threaded shared ownership (Rc)</td></tr> @shared</code></td> Multi-threaded shared ownership (Arc)</td></tr> @shared:locked</code></td> Arc + Mutex</td></tr> @shared:writeLocked</code></td> Arc + RwLock</td></tr> @locked</code></td> Mutex (single-scheduler)</td></tr> @writeLocked</code></td> RwLock (single-scheduler)</td></tr> @local</code></td> Thread-local heap pointer</td></tr> @indirect</code></td> Explicit heap allocation (Box)</td></tr> @link</code></td> To create cyclic graphs (WeakRef/Ref)</td></tr> @sharded(N)</code></td> Shared-nothing partitioned across N shards</td></tr> </tbody></table> Quick Reference: Sigils</h2> Sigil</th> Meaning</th> Example</th></tr></thead> @</code></td> Capability / pipeline binding</td> value @shared</code>, \|> process AS @p</code></td></tr> !</code></td> Mutation suffix</td> FN increment!(...)</code></td></tr> \|></code></td> Smooth operator (pipeline)</td> items \|> WHERE _ > 5</code></td></tr> _</code></td> Pipeline element placeholder</td> \|> SELECT _.name</code></td></tr> !T</code></td> Error union type</td> RETURNS !Float64</code></td></tr> ?T</code></td> Optional type</td> RETURNS ?User</code></td></tr> ~T</code></td> Promise / stream type</td> p: ~Int64 = BG { 42; }</code></td></tr> </tbody></table> Integer Overflow Operators</h2> CLEAR uses fixed-width integers (Int64, Int32, etc.) that can overflow. Three tiers of arithmetic operators control what happens on overflow:</p> Default: +</code>, -</code>, </code></h3> Panics on overflow in debug</strong> builds, wraps silently in release</strong> builds. This matches Rust's semantics - catches accidental overflow during development, zero overhead in production.</p> a: </span>Int64 </span>= 9223372036854775807_i64; </span>b = a + 1_i64; </span># debug: PANIC! release: wraps to -9223372036854775808 </span></code></pre> Wrapping: %+</code>, %-</code>, %</code></h3> Always wraps on overflow in all</strong> build modes. Use for hash functions, random number generators, checksums, and any code that intentionally overflows.</p> # LCG random number generator (intentional overflow) </span>MUTABLE </span>state: </span>Int64 </span>= seed; </span>state = state </span>%</span> 6364136223846793005_i64 %+ 1442695040888963407_i64; </span> </span># Max + 1 wraps to min </span>max: Int64 = 9223372036854775807_i64; </span>min = max %+ 1_i64; # always -9223372036854775808, never panics </span></code></pre> Checked: !+</code>, !-</code>, !</code></h3> Always panics on overflow in all</strong> build modes, including release. Use for financial calculations, safety-critical code, and anywhere overflow indicates a logic error.</p> # Financial calculation: overflow means a bug, even in production </span>balance = balance !+ deposit; </span># panics if result exceeds Int64 range </span>total = quantity !* price; </span># panics on overflow, even in release </span></code></pre> Summary</h3> Operator</th> Debug build</th> Release build</th> Use case</th></tr></thead> +</code>, -</code>, </code></td> panic</td> wrap</td> General arithmetic</td></tr> %+</code>, %-</code>, %</code></td> wrap</td> wrap</td> Hash, RNG, checksum</td></tr> !+</code>, !-</code>, !</code></td> panic</td> panic</td> Financial, safety</td></tr> </tbody></table> Float arithmetic (Float64</code>) is unaffected - IEEE 754 handles overflow via infinity/NaN.</p> Concurrency 2026-03-26T00:00:00+00:00 CLEAR uses cooperative fibers for concurrency.</p> Fibers are lightweight threads managed by the CLEAR runtime — each has its own stack (4-256KB) and runs until it yields. No OS threads are created per fiber; the scheduler multiplexes fibers onto a thread pool.</p> CLEAR will support limited Finite State Machines in v0.2, and by v0.4 - full support for Finite State Machines is planned.</p> </div> BG — Background Fibers</h2> Spawn a fiber that runs concurrently and returns a Promise:</p> p = </span>BG </span>{ expensive_computation(data); }; </span># ... do other work ... </span>result = </span>NEXT </span>p; </span># block until the fiber finishes </span></code></pre> The last expression in the body becomes the promise's value. NEXT</code> consumes the promise and returns the value.</p> By v0.4 Finite State Machines will be the default, and you will opt-in to stack-based fibers for re-entrant tasks or tasks that use FFI.</p> </div> Modifiers</h3> Modifiers go inside the braces, before a -></code>:</p> BG </span>{ </span>@large</span>:pinned </span>-> heavy_work(); } </span></code></pre> Modifier</th> Effect</th></tr></thead> @micro</code></td> 4 KB fiber stack</td></tr> @standard</code></td> 16 KB fiber stack (default)</td></tr> @large</code></td> 64 KB fiber stack</td></tr> @xl</code></td> 256 KB fiber stack</td></tr> @service</code></td> OS thread</strong> (not a fiber). Full OS stack. For heavy non-cooperative compute.</td></tr> @pinned</code></td> Pin to local scheduler (no work stealing)</td></tr> @parallel</code></td> Distribute to least-loaded scheduler</td></tr> @arena</code></td> Thread-local arena allocation; only works with @pinned tasks</td></tr> </tbody></table> Combine with :</code> — @large:@arena</code> gives a large stack with arena allocation.</p> By v0.2 @stateMachine</code> and @stack</code> will be options. The compiler will warn you when you should use a state machine, and block you from using it when it's not an option.</p> @service — OS Thread Spawning</h3> @service</code> spawns a dedicated OS thread</strong> instead of a green fiber. Use it for heavy-compute tasks that won't cooperatively yield - the OS handles preemption.</p> p = </span>BG </span>{ </span>@service </span>-> </span> trainModel(dataset); </span># runs on its own OS thread </span>}; </span># fiber continues concurrently </span>result = </span>NEXT </span>p; </span># blocks fiber until OS thread finishes </span></code></pre> The OS thread gets its own Runtime with a 64 KB frame arena. No scheduler is involved - the thread runs independently until completion, then signals the Promise.</p> When to use @service vs fibers:</strong></p> Fibers (@standard</code>/@large</code>): I/O-bound, short compute bursts, cooperative yielding</li> @service</code>: CPU-bound, long-running, no yields, true OS-level parallelism</li> </ul> Captures</h3> BG blocks capture outer variables by value</strong> (moved, not borrowed):</p> x = </span>42.0</span>; </span>p = </span>BG </span>{ x + </span>1.0</span>; }; </span># x is moved into the fiber </span># x is no longer usable here (affine ownership) </span></code></pre> THEN Chains</h3> Chain sequential steps inside a single fiber:</p> result = </span>BG </span>{ </span> fetch(</span>"</span>https://api.example.com/data</span>"</span>) </span>AS</span> response </span> </span>THEN</span> parse(response) </span>AS</span> parsed </span> </span>THEN</span> transform(parsed); </span>}; </span></code></pre> Each AS name</code> binds the result for subsequent steps. The last step's value becomes the promise result.</p> This is equivalent to:</p> result = </span>BG </span>{ </span> response = </span>NEXT</span> fetch(</span>"</span>https://api.example.com/data</span>"</span>); </span> parsed = </span>NEXT</span> parse(response); </span> transform(parsed); </span>}; </span></code></pre> THEN</code> is mostly useful for short-chained tasks, especially in a complex pipeline:</p> result = </span>BG </span>{ </span> fetch(</span>"</span>https://api.example.com/data1</span>"</span>) </span>AS</span> r </span>THEN</span> parse(r), </span> fetch(</span>"</span>https://api.example.com/data2</span>"</span>) </span>AS</span> r </span>THEN</span> parse(r) </span>}; </span># result = ~T[] -> this is only valid when all T are the same. </span></code></pre> Error handling:</strong> use OR</code> before the AS</code> binding. If a step returns !T</code>, handle it inline:</p> # ILLUSTRATIVE </span>result = </span>BG </span>{ </span> fetch(url) </span>OR RAISE </span># propagate error to caller </span> </span>AS</span> response </span>THEN</span> parse(response) </span>OR</span> default_value </span> </span>AS</span> parsed </span>THEN</span> transform(parsed); </span>}; </span></code></pre> OR RAISE</code> - propagate the error (caller sees it via NEXT</code>)</li> OR value</code> - replace error with a fallback, chain continues</li> </ul> DO — Fork-Join</h2> Execute multiple branches concurrently, wait for all to complete:</p> # ILLUSTRATIVE </span>DO </span>{ </span> update_database(record), </span> send_notification(user), </span> log_event(event) </span>} </span># All three are done here. </span></code></pre> Branches are separated by commas. Each runs in its own fiber. The DO block waits for all branches before continuing. Returns Void</code>.</p> Branch Modifiers</h3> Each branch can have its own modifiers:</p> # ILLUSTRATIVE </span>DO </span>{ </span> </span>@large </span>-> heavy_computation(), </span> </span>@pinned </span>-> cache_local_work() </span>} </span></code></pre> NEXT — Promise Resolution</h2> Block until a promise resolves:</p> p: </span>~</span>Float64 </span>= </span>BG </span>{ </span>42.0</span>; }; </span>result: </span>Float64 </span>= </span>NEXT </span>p; </span></code></pre> Promise Type</th> NEXT returns</th> Behavior</th></tr></thead> ~T</code></td> T</code></td> Consumes the promise (one-shot)</td></tr> ~T @shared</code></td> T</code></td> Returns cached result (safe for multiple NEXT)</td></tr> ~T[?]</code></td> ?T</code></td> Returns next value or nil (open stream)</td></tr> ~T[INF]</code></td> T</code></td> Returns next value, never nil (infinite stream)</td></tr> </tbody></table> BG STREAM — Generators</h2> Spawn a fiber that yields values over time:</p> # Open stream (finite) </span>s: </span>~</span>Float64</span>[?] = </span>BG STREAM </span>{ </span> </span>YIELD </span>1.0</span>; </span> </span>YIELD </span>4.0</span>; </span> </span>YIELD </span>9.0</span>; </span>}; </span>v1 = </span>NEXT</span> s; </span># 1.0 </span>v2 = </span>NEXT</span> s; </span># 4.0 </span>v3 = </span>NEXT</span> s; </span># 9.0 </span>v4 = </span>NEXT</span> s; </span># NIL (exhausted) </span> </span># Infinite stream </span>counter: </span>~</span>Float64</span>[</span>INF</span>] = </span>BG STREAM </span>{ </span> </span>MUTABLE</span> i = </span>0.0</span>; </span> </span>WHILE TRUE DO </span> </span>YIELD</span> i; </span> i = i + </span>1.0</span>; </span> </span>END </span>}; </span>v1 = </span>NEXT</span> counter; </span># 0.0 </span>v2 = </span>NEXT</span> counter; </span># 1.0 (blocks until generator yields) </span></code></pre> CONCURRENT — Parallel Pipelines</h2> Apply pipeline operators in parallel with a persistent worker pool:</p> # ILLUSTRATIVE </span>results = items \|> </span>CONCURRENT</span>(</span>workers: </span>8</span>) </span>SELECT</span> transform(_); </span>filtered = items \|> </span>CONCURRENT</span>(</span>workers: </span>4</span>) </span>WHERE</span> predicate(_); </span>items \|> </span>CONCURRENT</span>(</span>workers: </span>2</span>) </span>EACH </span>{ _.value = </span>0.0</span>; }; </span></code></pre> Options</h3> Option</th> Type</th> Default</th> Effect</th></tr></thead> workers</code></td> Number</td> 8</td> Number of persistent worker fibers</td></tr> batch</code></td> Number</td> 1</td> Items each worker pulls per fetch from the shared index (larger = less index contention)</td></tr> capacity</code></td> Number</td> auto (worker count rounded up to a power of 2, clamped 4-64)</td> Bounded channel ring-buffer size; only valid with a streaming / SHARD</code> / range source</td></tr> parallel</code></td> Bool</td> FALSE</td> TRUE = distribute workers across schedulers (multi-core)</td></tr> size</code></td> Identifier</td> STANDARD</td> Stack size: MICRO, STANDARD, LARGE, XL</td></tr> </tbody></table> Supported Operators</h3> CONCURRENT SELECT</strong> — parallel map. Returns transformed array (order preserved).</li> CONCURRENT WHERE</strong> — parallel filter. Returns matching elements (order preserved).</li> CONCURRENT EACH</strong> — parallel side effects. Returns Void.</li> </ul> Error Handling</h3> # Skip failed items </span>results = items </span> \|> </span>CONCURRENT</span>(</span>workers: </span>4</span>) </span>SELECT</span> risky_fn(_) </span>OR PRUNE</span>; </span> </span># Propagate first error </span>results = items </span> \|> </span>CONCURRENT</span>(</span>workers: </span>4</span>) </span>SELECT</span> risky_fn(_) </span>OR RAISE</span>; </span></code></pre> How It Works</h3> CONCURRENT spawns N persistent worker fibers that pull items from a shared atomic index. Zero per-item allocation — workers reuse their stack and context across all items. This is fundamentally different from spawning one fiber per item (which is what BG does).</p> results = items </span> \|> </span>SELECT BG </span>{ risky_fn(_) </span>OR RAISE </span>}; </span># this is valid, but it would spawn N tasks, all at once. It is NOT recommended. </span></code></pre> Default (workers on local scheduler): </span> CONCURRENT(workers: 8) SELECT transform(_) </span> → 8 fibers on one thread, cooperative scheduling </span> </span>Multi-core (workers distributed): </span> CONCURRENT(workers: 8, parallel: TRUE) SELECT transform(_) </span> → 8 fibers spread across all schedulers </span></code></pre> Performance</h3> Pattern</th> Per-item cost</th> Use when</th></tr></thead> CONCURRENT(workers: N)</code></td> ~0 (atomic fetchAdd only)</td> Bulk processing, batch transforms</td></tr> Individual BG { }</code></td> ~60μs (GPA alloc)</td> Dynamic spawning, I/O-bound tasks</td></tr> </tbody></table> CONCURRENT is 30x faster than individual BG spawns for batch workloads.</li> For I/O-bound tasks (network requests, file reads), the 60μs BG spawn cost is negligible compared to I/O latency.</li> The big benefit is that workers: N</code> handles backpressure to avoid spawning more tasks than you can handle.</li> </ul> Multi-Threading</h2> CLEAR defaults to single-threaded scheduling. Set CLEAR_THREADS</code> to enable multi-core:</p> CLEAR_THREADS</span>=</span>0 </span>./my_program </span># auto-detect CPU count </span>CLEAR_THREADS</span>=</span>4 </span>./my_program </span># 4 scheduler threads </span>CLEAR_THREADS</span>=</span>1 </span>./my_program </span># single-threaded (default) </span></code></pre> Each scheduler thread runs its own event loop with work stealing. Idle schedulers steal tasks from busy ones. @pinned</code> tasks are exempt from stealing.</p> Safety Rules</h2> The compiler enforces these at compile time:</p> Rule</th> Reason</th></tr></thead> @parallel</code> + @local</code> = error</td> @local has no synchronization</td></tr> @parallel</code> + @multiowned</code> = error</td> Rc is non-atomic; use @shared (Arc)</td></tr> @arena</code> + @parallel</code> = error</td> Arena memory is thread-local</td></tr> Non-@pinned BG capturing from @pinned scope = error</td> Thread-local memory would escape</td></tr> YIELD</code> outside BG STREAM = error</td> YIELD only valid in generators</td></tr> NEXT</code> on non-promise = error</td> Can only await promises and streams</td></tr> </tbody></table> When to Use What</h2> Goal</th> Construct</th></tr></thead> Process a batch of items</td> \|> CONCURRENT(workers: N) SELECT/WHERE/EACH</code></td></tr> Fire off a background task</td> BG { work(); }</code></td></tr> Run independent tasks concurrently</td> DO { task1(), task2(), task3() }</code></td></tr> Generate values lazily</td> BG STREAM { YIELD ...; }</code></td></tr> Chain async steps</td> BG { step1() AS r THEN step2(r) }</code></td></tr> Handle requests (server)</td> BG { @pinned:@arena -> handle(conn); }</code></td></tr> </tbody></table> Known Limitations (v0.1)</h2> Dynamic spawn overhead</strong>: spawning many individual BG</code> tasks is competitive with Go at moderate thread counts (roughly parity at 8 threads on the 100K-spawn benchmark) but slower at high core counts — about 1.6x Go at small scale, up to ~3.5x at 32 threads. The remaining gap is idle schedulers spinning on the work-stealing deque instead of parking, not per-spawn allocation. Workaround: use CONCURRENT(workers: N)</code> for bulk workloads. Scheduler parking (the fix) is tracked for post-v0.1.</p> No general bounded channel</strong>: CLEAR has Promises (one-shot) and Streams (unbounded). CONCURRENT</code> streaming uses an internal bounded ring-buffer channel for backpressure (tunable via the capacity:</code> option), but there is no general user-facing bounded producer/consumer channel primitive yet.</p> Pipelines and Higher-Order Functions 2026-03-26T00:00:00+00:00 CLEAR's pipeline system lets you transform, filter, aggregate, and iterate collections using the smooth operator (\|></code>).</p> Every pipeline operator works on arrays, @list</code>, @pool</code>, sharded collections, @pool:soa</code>, streams ~T[]</code>, etc - the same syntax regardless of the underlying storage.</p> _</code> is the placeholder value for the value being iterated over.</p> scores </span> \|> </span>WHERE</span> _ <= </span>50 </span> \|> </span>SUM</span> _; </span> </span>users </span> \|> </span>SELECT</span> _.name </span> \|> </span>DISTINCT</span> _; </span> </span>entities </span> \|> </span>EACH </span>{ _.health = _.health - </span>1.0</span>; }; </span></code></pre> This document describes both the collection pipeline model and the stream/future pipeline surface. The full operator set is supported for finite streams (~T[]</code>, ~T[N]</code>); infinite streams (~T[INF]</code>) require LIMIT</code> to bound them and then support all non-materialization operators.</p> The Smooth Operator (\|></code>)</h2> \|></code> pipes a value into a function or operator. It's CLEAR's equivalent of \|></code> (Elixir) or .</code> method chaining (Ruby), but it also works with collection operators.</p> # Pipe to a function: x \|> f → f(x) </span>result = data </span> \|> process </span> \|> validate </span> \|> format; </span> </span># Pipe to an operator: list \|> WHERE predicate </span>alive = entities </span> \|> </span>WHERE</span> _.health > </span>0</span>; </span></code></pre> Pipelines chain left to right. Each stage passes its result to the next.</p> Named Pipeline Bindings (AS $v</code>)</h2> The AS $v</code> syntax binds the current pipeline element to a named reference that persists across subsequent stages.</p> bill = users </span>AS $u </span> \|> </span>UNNEST $u</span>.orders </span> \|> </span>SUM</span> _.price </span>$u</span>.discount; </span></code></pre> Why bindings exist</h3> Without a binding, _</code> always refers to the current</em> element - the item being iterated at the innermost level. After UNNEST</code>, _</code> becomes each inner element (an Order), and the outer element (the User) is no longer reachable.</p> # Without AS $u: $u is not available inside the fold. </span># `_` after UNNEST is the Order, not the User. </span>bill = users </span> \|> </span>UNNEST</span> _.orders </span> \|> </span>SUM</span> _.price; </span># can access order.price, but NOT user.discount </span></code></pre> AS $u</code> captures the outer element before the UNNEST</code> replaces _</code>, keeping it accessible:</p> bill = users </span>AS $u </span> \|> </span>UNNEST $u</span>.orders </span># $u = the User; _ = each Order </span> \|> </span>SUM</span> _.price * </span>$u</span>.discount; </span># cross-reference: order.price * user.discount </span></code></pre> The compiler fuses this into a single nested loop with no intermediate allocations.</p> Scope of a binding</h3> A binding created with AS $v</code> is visible from the point of declaration to the end of the pipeline expression. It cannot be used after the pipeline terminates:</p> bill = users </span>AS $u </span> \|> </span>UNNEST $u</span>.orders </span># $u is in scope </span> \|> </span>WHERE</span> _.qty > </span>1 </span># $u still in scope </span> \|> </span>SUM</span> _.price * </span>$u</span>.discount; </span># $u still in scope </span> </span># $u is not accessible here (out of scope after the pipeline ends) </span></code></pre> Multiple pipelines in the same function can each use their own $u</code> - bindings are scoped to the pipeline expression, not the function.</p> Naming the inner element (AS $o)</h3> When you UNNEST and need a name for the inner element (instead of _</code>), use a second binding after the UNNEST expression:</p> bill = users </span>AS $u </span> \|> </span>UNNEST $u</span>.orders </span>AS $o </span># $u = User, $o = Order </span> \|> </span>SUM $o</span>.price * </span>$u</span>.discount; </span></code></pre> AS $o</code> after the UNNEST expression binds the inner element. Both $u</code> and $o</code> are available in the fold.</p> Supported combinations</h3> UNNEST binding chains</strong> - fold operators that work after AS $u \|> UNNEST</code>:</p> Fold</th> Example</th></tr></thead> SUM</td> \|> SUM _.price * $u.discount</code></td></tr> COUNT</td> \|> COUNT TRUE</code></td></tr> AVERAGE</td> \|> AVERAGE _.price</code></td></tr> MIN</td> \|> MIN _.price</code></td></tr> MAX</td> \|> MAX _.price</code></td></tr> ANY</td> \|> ANY _.price > 50.0</code></td></tr> ALL</td> \|> ALL $u.discount > 0.0</code></td></tr> FIND</td> \|> FIND _.price > 10.0</code> (returns ?ElemType</code>)</td></tr> </tbody></table> Intermediate WHERE</code> stages filter the inner elements before the fold:</p> total = users </span>AS $u </span> \|> </span>UNNEST $u</span>.orders </span> \|> </span>WHERE</span> _.qty > </span>1 </span># filter inner elements </span> \|> </span>SUM</span> _.price * </span>$u</span>.discount; </span></code></pre> SELECT</code> is not supported in UNNEST binding chains (the inner projection would lose the $u</code> context). Use field access in the fold expression instead.</p> CONCURRENT binding</strong> - $u</code> is also accessible inside CONCURRENT</code> stages that operate directly on the bound list (without an intermediate UNNEST):</p> Operator</th> Example</th></tr></thead> CONCURRENT SELECT</td> AS $u \|> CONCURRENT SELECT $u.val * 2.0</code></td></tr> CONCURRENT SUM</td> AS $u \|> CONCURRENT SUM $u.score</code></td></tr> CONCURRENT COUNT</td> AS $u \|> CONCURRENT COUNT $u.active</code></td></tr> CONCURRENT MIN</td> AS $u \|> CONCURRENT MIN $u.score</code></td></tr> CONCURRENT MAX</td> AS $u \|> CONCURRENT MAX $u.score</code></td></tr> CONCURRENT AVERAGE</td> AS $u \|> CONCURRENT AVERAGE $u.score</code></td></tr> CONCURRENT WHERE</td> AS $u \|> CONCURRENT WHERE $u.score > 50.0</code></td></tr> </tbody></table> In all concurrent cases, $u</code> resolves to the item being processed by the current worker.</p> The _</code> Variable</h2> Inside pipeline expressions, _</code> refers to the current element. For struct elements, access fields with _.fieldname</code>:</p> # _ is the element itself (for scalar collections) </span>nums: </span>Float64</span>[] = [</span>1.0</span>, </span>3.0</span>, </span>7.0</span>, </span>9.0</span>]; </span> </span>big = nums </span> \|> </span>WHERE</span> _ > </span>5.0</span>; </span> </span>ASSERT</span> length(big) == </span>2</span>, </span>"</span>WHERE filters by element value</span>"</span>; </span> </span># _.field for struct collections </span>users = [</span>User</span>{</span>name: </span>"</span>alice</span>"</span>}, </span>User</span>{</span>name: </span>"</span>bob</span>"</span>}]; </span> </span>names = users </span> \|> </span>SELECT</span> _.name; </span> </span>scores = [</span>Score</span>{</span>value: </span>10.0</span>}, </span>Score</span>{</span>value: </span>20.0</span>}]; </span> </span>total = scores </span> \|> </span>SUM</span> _.value; </span> </span>ASSERT</span> total == </span>30.0</span>, </span>"</span>SUM aggregates field values</span>"</span>; </span></code></pre> In EACH blocks, _</code> is mutable — you can assign to fields:</p> pool </span> \|> </span>EACH </span>{ _.health = _.health - damage; }; </span></code></pre> Operators</h2> Transform</h3> Operator</th> Syntax</th> Returns</th> Description</th></tr></thead> SELECT</strong></td> list \|> SELECT expr</code></td> ExprType[]</code></td> Project each element through an expression</td></tr> WHERE</strong></td> list \|> WHERE pred</code></td> ElemType[]</code></td> Keep elements matching a boolean predicate</td></tr> ORDER_BY</strong></td> list \|> ORDER_BY key</code></td> ElemType[]</code></td> Sort by key expression</td></tr> LIMIT</strong></td> list \|> LIMIT n</code></td> ElemType[]</code></td> First N elements</td></tr> SKIP</strong></td> list \|> SKIP n</code></td> ElemType[]</code></td> Drop first N elements, return rest</td></tr> DISTINCT</strong></td> list \|> DISTINCT key</code></td> ElemType[]</code></td> Unique by key (first occurrence wins)</td></tr> UNNEST</strong></td> list \|> UNNEST expr</code></td> InnerType[]</code></td> Flatten nested arrays (flatmap)</td></tr> INDEX</strong></td> list \|> INDEX key</code></td> HashMap<ElemType[]></code></td> Group into a hashmap by key</td></tr> </tbody></table> SELECT accepts any expression, including struct literals. This lets you project into a different struct type in one step:</p> STRUCT Raw </span>{ </span>id: </span>Int64</span>, </span>score: </span>Float64 </span>} </span>STRUCT Summary </span>{ </span>key: </span>Int64</span>, </span>normalized: </span>Float64 </span>} </span> </span>raws: </span>Raw</span>[] = [</span>Raw</span>{ </span>id: </span>1</span>, </span>score: </span>100.0 </span>}, </span>Raw</span>{ </span>id: </span>2</span>, </span>score: </span>200.0 </span>}]; </span> </span>summaries = raws \|> </span>SELECT Summary</span>{ </span>key:</span> _.id, </span>normalized:</span> _.score / </span>100.0 </span>}; </span> </span>ASSERT</span> summaries[</span>0</span>].key == </span>1</span>, </span>"</span>id preserved</span>"</span>; </span>ASSERT</span> summaries[</span>1</span>].normalized == </span>2.0</span>, </span>"</span>score normalized</span>"</span>; </span></code></pre> The result type is inferred from the expression - Summary[]</code> above, not Raw[]</code>.</p> Pipeline fusion with SELECT T{}:</strong> SELECT composes with WHERE and aggregates in a single fused loop - no intermediate list allocation:</p> # WHERE before SELECT: filter on the raw element (efficient) </span>high = raws </span> \|> </span>WHERE</span> _.score > </span>75.0 </span> \|> </span>SELECT Summary</span>{ </span>key:</span> _.id, </span>normalized:</span> _.score / </span>100.0 </span>}; </span> </span># SELECT before aggregate: project then sum/min/max a field </span>total = raws </span> \|> </span>SELECT Summary</span>{ </span>key:</span> _.id, </span>normalized:</span> _.score / </span>100.0 </span>} </span> \|> </span>SUM</span> _.normalized; </span> </span># SELECT before WHERE: build struct first, then filter on a struct field (less efficient) </span>norm_high = raws </span> \|> </span>SELECT Summary</span>{ </span>key:</span> _.id, </span>normalized:</span> _.score / </span>100.0 </span>} </span> \|> </span>WHERE</span> _.normalized > </span>0.75</span>; </span></code></pre> When SELECT precedes a fold or WHERE, the compiler binds the projected value to a temp variable per iteration. This avoids the Zig restriction on struct literals in arithmetic/boolean expression positions, so the generated code is always valid without changing semantics.</p> Aggregate</h3> Operator</th> Syntax</th> Returns</th> Empty list</th></tr></thead> SUM</strong></td> list \|> SUM expr</code></td> Int64 / UInt64 / Float32 / Float64</td> 0</td></tr> AVERAGE</strong></td> list \|> AVERAGE expr</code></td> Float64</code></td> 0</td></tr> MIN</strong></td> list \|> MIN expr</code></td> matches expr type</td> panics</td></tr> MAX</strong></td> list \|> MAX expr</code></td> matches expr type</td> panics</td></tr> REDUCE</strong></td> list \|> REDUCE(init) expr</code></td> type of init</td> init</td></tr> </tbody></table> The return type is driven by the expression type. SUM widens small integers to Int64</code>/UInt64</code>; floats stay at their original width (Float32</code> stays Float32</code>). AVERAGE always returns Float64</code>. MIN and MAX preserve the exact expression type. REDUCE is the general fold - acc</code> is the mutable accumulator, _</code> is the current element:</p> nums: </span>Float64</span>[] = [</span>2.0</span>, </span>3.0</span>, </span>4.0</span>]; </span> </span>product = nums </span> \|> </span>REDUCE</span>(</span>1.0</span>) acc * _; </span> </span>ASSERT</span> product == </span>24.0</span>, </span>"</span>REDUCE multiplies 234</span>"</span>; </span></code></pre> Query</h3> Operator</th> Syntax</th> Returns</th> Description</th></tr></thead> COUNT</strong></td> list \|> COUNT pred</code></td> Int64</code></td> Number of matches</td></tr> ANY</strong></td> list \|> ANY pred</code></td> Bool</code></td> True if any match (short-circuits)</td></tr> ALL</strong></td> list \|> ALL pred</code></td> Bool</code></td> True if all match (short-circuits)</td></tr> FIND</strong></td> list \|> FIND pred</code></td> ?ElemType</code></td> First match or null</td></tr> </tbody></table> Side Effects</h3> Operator</th> Syntax</th> Returns</th> Description</th></tr></thead> EACH</strong></td> list \|> EACH { body }</code></td> Void</code></td> Iterate with mutable _</code>; side-effect only</td></tr> TAP</strong></td> list \|> TAP { body }</code></td> ElemType[]</code></td> Observe each element (read-only _</code>), pass collection through</td></tr> </tbody></table> EACH is the only operator where _</code> is mutable. Use it for in-place updates:</p> entities </span> \|> </span>EACH </span>{ _.x = _.x + _.vx; _.y = _.y + _.vy; }; </span></code></pre> TAP (Debugging / Observation)</h3> TAP runs a body for each element but passes the collection through unchanged. Unlike EACH, _</code> is read-only and TAP returns the original collection:</p> result = scores </span> \|> </span>WHERE</span> _.points > </span>100 </span> \|> </span>TAP </span>{ print(</span>"</span>score: ${_.points.toString()}</span>"</span>); } </span> \|> </span>SUM</span> _.points; </span></code></pre> WINDOW (Sliding Window)</h3> WINDOW produces a sliding window of size N over the collection. _</code> inside the body is the sub-slice (a Float64[]</code> or ElemType[]</code> of length N). The result is an array of the body expression evaluated per window.</p> data: </span>Float64</span>[] = [</span>1.0</span>, </span>2.0</span>, </span>3.0</span>, </span>4.0</span>, </span>5.0</span>]; </span> </span># Each window is a 3-element slice; body projects to window length </span>lengths = data \|> </span>WINDOW</span>(</span>3</span>) _.length(); </span>ASSERT</span> lengths.length() == </span>3</span>, </span>"</span>5 elements, window 3 -> 3 windows</span>"</span>; </span> </span># Access individual elements in the window </span>firsts = data \|> </span>WINDOW</span>(</span>2</span>) _[</span>0</span>]; </span>ASSERT</span> firsts[</span>0</span>] == </span>1.0</span>, </span>"</span>first element of first window</span>"</span>; </span></code></pre> Number of result windows = max(0, len - size + 1)</code>. A window larger than the list produces an empty result.</p> WINDOW (Batch / Tumbling Window)</h3> WINDOW(size: N)</code>, WINDOW(time: 'Xms')</code>, or WINDOW(size: N, time: 'Xms')</code> produces non-overlapping batches. _</code> inside the body is a T[]</code> batch. The result is a heap-allocated list of the body expression evaluated per batch.</p> Flush conditions (checked after every item):</p> size: N</code> - flush when the batch accumulates N items</li> time: 'Xms'</code> - flush when elapsed time since the first item in the batch >= timeout; time units: ms</code>, s</code>, min</code>, h</code></li> Both specified - flush on whichever fires first (first-of-either)</li> </ul> A partial batch at the end is always included. Works on all source types: arrays, ~T[]</code>, ~T[N]</code>, and ~T[INF]</code>.</p> FN</span> sumBatch(</span>batch: </span>Int64</span>[]) </span>RETURNS Int64 </span>-> </span> </span>MUTABLE </span>s: </span>Int64 </span>= </span>0</span>; </span> batch \|> </span>EACH </span>{ s = s + _; }; </span> </span>RETURN</span> s; </span>END </span> </span># Array source, size-only: [1..7] -> [1,2,3], [4,5,6], [7] </span>arr: </span>Int64</span>[] = [</span>1</span>, </span>2</span>, </span>3</span>, </span>4</span>, </span>5</span>, </span>6</span>, </span>7</span>]; </span>sums = arr \|> </span>WINDOW</span>(</span>size: </span>3</span>) sumBatch(_); </span>ASSERT</span> sums.length() == </span>3</span>; </span># 3 batches </span>ASSERT</span> sums[</span>2</span>] == </span>7</span>; </span># partial final batch </span> </span># Open stream, size + time (first-of-either) </span>gen: </span>~?Int64[] = </span>BG STREAM </span>{ </span>MUTABLE </span>i: </span>Int64 </span>= </span>1</span>; </span>WHILE</span> i <= </span>10 </span>DO YIELD</span> i; i = i + </span>1</span>; </span>END </span>}; </span>sums2 = gen \|> </span>WINDOW</span>(</span>size: </span>4</span>, </span>time: </span>"</span>500ms</span>"</span>) sumBatch(_); </span>ASSERT</span> sums2.length() == </span>3</span>; </span># [1-4], [5-8], [9-10] </span> </span># Time-only: entire array arrives fast, all items in one final batch </span>arr2: </span>Int64</span>[] = [</span>100</span>, </span>200</span>, </span>300</span>]; </span>lens = arr2 \|> </span>WINDOW</span>(</span>time: </span>"</span>1s</span>"</span>) _.length(); </span>ASSERT</span> lens[</span>0</span>] == </span>3</span>; </span># one batch of 3 </span></code></pre> Note:</strong> The sliding-window form WINDOW(N)</code> (positional integer, no named params) is the existing collection-only operator above. The named-param form WINDOW(size:, time:)</code> is the new batching operator and works on both collections and streams.</p> </blockquote> JOIN (Left Outer Join)</h3> JOIN performs a left outer join between two collections using a two-parameter lambda predicate. Every element of the left collection appears in the result exactly once; the right field is NIL</code> when no right element matches.</p> STRUCT User </span>{ </span>id: </span>Int64</span>, </span>name: </span>String } </span>STRUCT Order </span>{ </span>userId: </span>Int64</span>, </span>amount: </span>Float64 </span>} </span> </span>results = users \|> </span>JOIN</span>(orders) </span>%(</span>u, o</span>) </span>-> u.id == o.userId; </span># results: JoinResult_User_Order[] </span># results[i].left -- the User </span># results[i].right -- ?Order (NIL if no match) </span></code></pre> The result element type is an anonymous struct { left: L, right: ?R }</code>. The lambda must take exactly two parameters (left element, right element) and return Bool.</p> Stream and Future Compatibility</h2> Pipelines over futures/streams are currently supported in a narrower subset than pipelines over collections.</p> Stream kinds</h3> Type</th> Meaning</th> NEXT</code> result</th> Pipeline support</th></tr></thead> ~T</code></td> Single future value</td> T</code></td> Not a pipeline source</td></tr> ~?T[]</code></td> Open stream</td> ?T</code></td> Not a pipeline source yet</td></tr> ~T[]</code></td> Finite dynamic stream</td> ?T</code></td> Full non-concurrent operator set (see table)</td></tr> ~T[N]</code></td> Finite bounded stream</td> ?T</code></td> Full non-concurrent operator set plus CONCURRENT EACH/SELECT/WHERE</code></td></tr> ~T[INF]</code></td> Infinite stream</td> T</code></td> Fusible stages + all terminals via LIMIT</code></td></tr> </tbody></table> Operator support matrix</h3> The columns are the three pipeline-capable stream types. "Stage" operators filter or transform in a fused while loop without materializing; "terminal" operators consume the stream and produce a scalar or collection result.</p> Stages (fusible, zero intermediate allocations)</h4> Operator</th> ~T[]</code></th> ~T[N]</code></th> ~T[INF]</code></th></tr></thead> WHERE</code></td> yes</td> yes</td> yes (LIMIT must appear in chain)</td></tr> SELECT</code></td> yes</td> yes</td> yes (LIMIT must appear in chain)</td></tr> SKIP</code></td> yes</td> yes</td> yes (LIMIT must appear in chain)</td></tr> TAKE_WHILE</code></td> yes</td> yes</td> yes (LIMIT must appear in chain)</td></tr> LIMIT</code></td> yes</td> yes</td> yes - converts stream to T[]</code></td></tr> TAP</code></td> yes</td> yes</td> not yet</td></tr> </tbody></table> LIMIT</code> can appear anywhere in the chain relative to other fusible stages. The compiler fuses the entire chain into a single while loop - counter \|> WHERE _ > 0 \|> LIMIT 5 \|> EACH</code> and counter \|> LIMIT 5 \|> WHERE _ > 0 \|> EACH</code> both work; they differ only in whether LIMIT counts pre- or post-filter items.</p> Fold terminals (produce a scalar or optional value)</h4> Operator</th> ~T[]</code></th> ~T[N]</code></th> ~T[INF]</code></th></tr></thead> EACH</code></td> yes</td> yes</td> via LIMIT</td></tr> SUM</code></td> yes</td> yes</td> via LIMIT</td></tr> COUNT</code></td> yes</td> yes</td> via LIMIT</td></tr> AVERAGE</code></td> yes</td> yes</td> via LIMIT</td></tr> MIN</code></td> yes</td> yes</td> via LIMIT</td></tr> MAX</code></td> yes</td> yes</td> via LIMIT</td></tr> ANY</code></td> yes</td> yes</td> via LIMIT</td></tr> ALL</code></td> yes</td> yes</td> via LIMIT</td></tr> FIND</code></td> yes</td> yes</td> via LIMIT</td></tr> REDUCE</code></td> yes</td> yes</td> via LIMIT</td></tr> </tbody></table> "via LIMIT" means LIMIT must appear earlier in the same pipeline chain. LIMIT converts ~T[INF]</code> to T[]</code> (a regular list); the fold terminal then operates on that list. Example: counter \|> WHERE _ > 0 \|> LIMIT 5 \|> SUM _</code>.</p> Materialization terminals (produce a new collection)</h4> These operators consume the stream and produce a new heap-allocated collection. DISTINCT</code> and INDEX</code> are fully supported for all stream types; others require materializing first with .toList()</code>.</p> Operator</th> ~T[]</code></th> ~T[N]</code></th> ~T[INF]</code></th></tr></thead> DISTINCT</code></td> yes - returns T[]@set</code></td> yes - returns T[]@set</code></td> via LIMIT - returns T[]@set</code></td></tr> INDEX</code></td> yes</td> yes</td> via LIMIT</td></tr> ORDER_BY</code></td> not yet</td> not yet</td> not yet</td></tr> UNNEST</code></td> not yet</td> not yet</td> not yet</td></tr> WINDOW(N)</code> (sliding)</td> not yet</td> not yet</td> not yet</td></tr> WINDOW(size:, time:)</code> (batch)</td> yes</td> yes</td> yes</td></tr> JOIN</code></td> not yet</td> not yet</td> not yet</td></tr> </tbody></table> "via LIMIT" means LIMIT</code> must appear earlier in the chain (same rule as fold terminals). DISTINCT</code> returns T[]@set</code> (a Set), supporting .count()</code> and .contains?()</code>.</p> If you need ORDER_BY</code>, UNNEST</code>, sliding WINDOW(N)</code>, or JOIN</code> on a stream, materialize first. The batching WINDOW(size:, time:)</code> operator works directly on streams without materialization.</p> s: </span>~</span>Int64</span>[] = </span>0 </span>..< </span>10</span>; </span>vals = s.toList(); </span>total = vals </span> \|> </span>WHERE</span> _ > </span>3 </span> \|> </span>SUM</span> _; </span></code></pre> Concurrent operators</h4> Operator</th> ~T[]</code></th> ~T[N]</code></th> ~T[INF]</code></th></tr></thead> CONCURRENT EACH</code></td> not yet</td> yes</td> not yet</td></tr> CONCURRENT SELECT</code></td> not yet</td> yes</td> not yet</td></tr> CONCURRENT WHERE</code></td> not yet</td> yes</td> not yet</td></tr> </tbody></table> ~T[N]</code> concurrent pipelines are native: they consume promise slots directly without materializing through .toList()</code>, and lower through MIR-visible builtin helpers. Example:</p> nums: </span>~</span>Float64</span>[</span>4</span>] = [</span>BG </span>{ </span>1.0</span>; }, </span>BG </span>{ </span>2.0</span>; }, </span>BG </span>{ </span>3.0</span>; }, </span>BG </span>{ </span>4.0</span>; }]; </span> </span>doubled = nums </span> \|> </span>CONCURRENT</span>(</span>workers: </span>2</span>) </span>SELECT</span> _ * </span>2.0</span>; </span></code></pre> Direct range expressions still use the non-concurrent path unless first bound as ~T[N]</code>.</p> Open streams</h3> Open streams (~?T[]</code>) are still NEXT</code>-driven only:</p> gen: </span>~?Int64[] = </span>BG STREAM </span>{ </span> </span>YIELD </span>1</span>; </span> </span>YIELD </span>2</span>; </span>}; </span> </span>v1 = </span>NEXT</span> gen; </span>v2 = </span>NEXT</span> gen; </span>v3 = </span>NEXT</span> gen; </span># NIL </span></code></pre> SKIP and LIMIT (Pagination)</h3> SKIP and LIMIT are complementary: SKIP drops the first N elements, LIMIT takes the first N.</p> # Pagination: page 3, 10 items per page </span>page = items </span> \|> </span>SKIP </span>20 </span> \|> </span>LIMIT </span>10</span>; </span> </span># Skip header row, process the rest </span>data = rows </span> \|> </span>SKIP </span>1 </span> \|> </span>SELECT</span> parseRow; </span></code></pre> Chaining</h2> Operators compose naturally:</p> # Filter, sort, take top 3 </span>leaderboard = scores </span> \|> </span>WHERE</span> _.points > </span>100 </span> \|> </span>ORDER_BY</span> _.points </span> \|> </span>LIMIT </span>3</span>; </span> </span># Count active users with high scores </span>n = users </span> \|> </span>WHERE</span> _.active == </span>TRUE </span> \|> </span>COUNT</span> _.score > </span>1000</span>; </span></code></pre> Collection Compatibility</h2> Every operator works on every collection type:</p> # Array </span>nums: </span>Float64</span>[] = [</span>1</span>, </span>2</span>, </span>3</span>]; </span>total = nums </span> \|> </span>SUM</span> _; </span> </span># List </span>MUTABLE</span> data = </span>List</span>[]; </span>avg = data </span> \|> </span>AVERAGE</span> _.value; </span> </span># Pool </span>MUTABLE </span>pool: </span>Entity</span>[</span>1000</span>]</span>@pool </span>= []; </span>alive = pool </span> \|> </span>WHERE</span> _.health > </span>0</span>; </span> </span># Pool with SOA (field-slice iteration — cache-optimal) </span>MUTABLE </span>soa_pool: </span>Entity</span>[</span>1000</span>]</span>@pool</span>:soa </span>= []; </span>total_hp = soa_pool </span> \|> </span>SUM</span> _.health; </span># iterates only the health array </span> </span># List with SOA </span>MUTABLE </span>soa_list: </span>Entity</span>[]</span>@list</span>:soa </span>= []; </span>avg = soa_list </span> \|> </span>AVERAGE</span> _.health; </span># contiguous f64 slice </span> </span># Sharded (parallel EACH via DO blocks) </span>MUTABLE </span>sharded: </span>Entity</span>[</span>10000</span>]</span>@pool</span>:sharded</span>(</span>4</span>) = []; </span>sharded </span> \|> </span>EACH </span>{ _.processed = </span>TRUE</span>; }; </span></code></pre> Loop Fusion</h2> The compiler automatically fuses chains of WHERE and SELECT stages ending in a fold (SUM, REDUCE, AVERAGE, MIN, MAX, COUNT, ANY, ALL, FIND) into a single loop with zero intermediate allocations.</p> # Written as 3 stages: </span>result = data </span> \|> </span>WHERE</span> _ > </span>500.0 </span> \|> </span>SELECT</span> _ * _ </span> \|> </span>SUM</span> _; </span> </span># Compiled as a single loop (no intermediate arrays): </span># for (data) \|it\| { if (it > 500) { sum += it * it; } } </span></code></pre> This eliminates the allocation and iteration overhead of intermediate lists. Stages that require materialization (ORDER_BY, DISTINCT, INDEX) break the fusion chain - operations before them are fused separately.</p> SOA Optimization</h2> When a @pool:soa</code> is used in a pipeline, the compiler rewrites field accesses to iterate directly over contiguous field arrays instead of striding over whole structs. This happens automatically for all operators — no syntax change needed.</p> STRUCT Entity </span>{ </span>x: </span>Float64</span>, </span>y: </span>Float64</span>, </span>vx: </span>Float64</span>, </span>vy: </span>Float64</span>, </span>health: </span>Float64 </span>} </span>MUTABLE </span>pool: </span>Entity</span>[</span>10000</span>]</span>@pool</span>:soa </span>= []; </span> </span># SUM _.health iterates only the health array (contiguous f64[]). </span># Without :soa, it would load all 5 fields per element. </span>total = pool </span> \|> </span>SUM</span> _.health; </span></code></pre> For WHERE and FIND, the predicate uses field-slice access (fast), and the struct is reassembled only for matching elements.</p> The compiler warns when SOA would help:</p> NOTE: Pipeline accesses 1 of 5 fields (health). Consider @soa </span> for better cache performance on 'Entity'. </span></code></pre> Concurrency</h2> The CONCURRENT</code> modifier currently parallelizes collection pipelines for SELECT</code>, WHERE</code>, and EACH</code>:</p> MUTABLE </span>data: </span>Score</span>[</span>10000</span>]</span>@pool</span>:sharded</span>(</span>4</span>) = []; </span> </span># Parallel WHERE: one fiber per shard </span>results = data </span> \|> </span>CONCURRENT WHERE</span> dbFetch(_.id).val > threshold; </span> </span># Process in parallel </span>results = data </span> \|> </span>CONCURRENT</span>(</span>parallel: </span>TRUE</span>) </span>SELECT</span> dbFetch(_.id).name; </span></code></pre> Options:</p> workers: N</code> (fiber worker size)</li> pin: TRUE</code> (pin to cores)</li> size: MICRO\|STANDARD\|LARGE\|XL</code> (stack size).</li> parallel</code>: TRUE` (run on multiple cores)</li> </ul> CONCURRENT</code> compatibility</h3> Source kind</th> CONCURRENT SELECT</code></th> CONCURRENT WHERE</code></th> CONCURRENT EACH</code></th></tr></thead> Arrays / @list</code> / @pool</code> / @pool:soa</code></td> Yes</td> Yes</td> Yes</td></tr> @sharded(...)</code> collections</td> Yes</td> Yes</td> Yes</td></tr> Finite streams ~T[]</code></td> Not yet</td> Not yet</td> Not yet</td></tr> Bounded streams ~T[N]</code></td> Yes</td> Yes</td> Yes</td></tr> Open streams ~?T[]</code></td> Not yet</td> Not yet</td> Not yet</td></tr> Infinite streams ~T[INF]</code></td> Not yet</td> Not yet</td> Not yet</td></tr> </tbody></table> So today:</p> CONCURRENT</code> is for collection sources.</li> finite streams can participate in non-concurrent fused pipelines for the supported operators above.</li> stream CONCURRENT</code> will come later once the MIR-safe native lowering is in place.</li> </ul> CLEAR Collections: Array, List, Pool 2026-03-25T00:00:00+00:00 CLEAR has three collection types, each designed for a different access pattern. The choice is a capability annotation — the element type and functions that operate on it stay the same regardless of which collection you use.</p> Quick Reference</h2> Collection</th> Zig Type</th> Access</th> Growth</th> Safety</th> Use when</th></tr></thead> T[N]</code></td> [N]T</code></td> Index O(1)</td> Fixed</td> Bounds-checked</td> Size known at compile time</td></tr> T[]@list</code></td> ArrayListUnmanaged(T)</code></td> Index O(1), append O(1)</td> Dynamic</td> Bounds-checked</td> Size unknown, sequential access</td></tr> T[N]@pool</code></td> Pool(T)</code></td> Handle O(1)</td> Fixed</td> Generational handles</strong></td> Frequent insert/remove, stable references</td></tr> </tbody></table> Amortized O(1) — occasional reallocation when capacity is exceeded.</p> Arrays — T[N]</code></h2> Fixed-size, stack-allocated. The size is part of the type.</p> MUTABLE </span>scores: </span>Int64</span>[</span>5</span>] = [</span>10</span>, </span>20</span>, </span>30</span>, </span>40</span>, </span>50</span>]; </span>x = scores[</span>2</span>]; </span># 30 </span>scores[</span>0</span>] = </span>99</span>; </span># mutation via index </span>ASSERT</span> x == </span>30</span>, </span>"</span>index access</span>"</span>; </span>ASSERT</span> scores[</span>0</span>] == </span>99</span>, </span>"</span>mutation via index</span>"</span>; </span></code></pre> When to use</strong>: Size is known at compile time. No insertions or removals. The fastest option — no heap allocation, no bounds-growth overhead.</p> Limitations</strong>: Cannot append, remove, or resize. Index out of bounds is a runtime panic.</p> Lists — @list</code></h2> Dynamic-size, heap-allocated. Backed by std.ArrayListUnmanaged(T)</code>.</p> MUTABLE</span> users = </span>List</span>[]; </span>users.append(</span>"</span>Alice</span>"</span>); </span>users.append(</span>"</span>Bob</span>"</span>); </span>n = users.length(); </span># 2 </span>name = users[</span>0</span>]; </span># "Alice" </span></code></pre> When to use</strong>: Size is unknown or grows over time. Elements accessed by index. The general-purpose dynamic array — equivalent to Vec<T></code> in Rust or []T</code> in Go.</p> Limitations</strong>: Removing from the middle is O(N) (shift elements). Handles/pointers to elements are invalidated on reallocation. Not suitable for frequent insert/remove of interior elements.</p> Variant — sharded list</strong>: T[]@list:sharded(N)</code> splits the list into N shards for parallel pipeline operations (\|> EACH</code>, \|> SUM</code>). Each shard is an independent list. Round-robin distribution on append.</p> Lazy inference</strong>: List[]</code> creates an untyped list. The element type is inferred from the first append</code>:</p> MUTABLE</span> items = </span>List</span>[]; </span>items.append(</span>42.0</span>); </span># type inferred as Float64[]@list </span></code></pre> Pools — @pool</code></h2> Handle-based, pre-allocated with fixed capacity. Backed by Pool(T)</code> with generational handles</strong> for ABA safety. The capacity is specified at declaration time and all slots are allocated up front — no dynamic resizing, no reallocation. Insert and remove are O(1) via an internal free stack.</p> STRUCT Enemy </span>{ </span>hp: </span>Int64</span>, </span>name: </span>String } </span> </span>MUTABLE </span>enemies: </span>Enemy</span>[</span>1000</span>]</span>@pool </span>= []; </span>id1: </span>Id</span><</span>Enemy</span>> = enemies.insert(</span>Enemy</span>{ </span>hp: </span>100</span>, </span>name: </span>"</span>Goblin</span>" </span>}); </span>id2: </span>Id</span><</span>Enemy</span>> = enemies.insert(</span>Enemy</span>{ </span>hp: </span>200</span>, </span>name: </span>"</span>Dragon</span>" </span>}); </span> </span># Access via handle — returns ?T (optional). Must use OR to unwrap: </span>goblin = enemies.get(id1) </span>OR Enemy</span>{ </span>hp: </span>0</span>, </span>name: </span>"</span>none</span>" </span>}; </span> </span># Or via [] syntax (equivalent to .get): </span>dragon = enemies[id2] </span>OR Enemy</span>{ </span>hp: </span>0</span>, </span>name: </span>"</span>none</span>" </span>}; </span> </span># Remove: slot is freed, generation increments </span>enemies.remove(id1); </span> </span># Stale handle returns null — OR provides a safe fallback: </span>gone = enemies.get(id1) </span>OR Enemy</span>{ </span>hp: </span>0</span>, </span>name: </span>"</span>removed</span>" </span>}; </span></code></pre> Important: every pool access returns ?T</code> (optional).</strong> Like @writeLocked</code>, using a pool is not a one-line optimization — it changes how you access data. Every pool.get(id)</code> or pool[id]</code> requires OR default</code> to handle the case where the handle is stale or the slot was removed. This is by design: the pool guarantees you never read invalid data, but you must handle the "not found" case explicitly.</p> How Generational Handles Work</h3> Each pool slot stores a generation counter alongside the data:</p> Slot: </span>{ </span>generation:</span> u32, </span>alive:</span> bool, </span>value: </span>T </span>} </span>Handle </span>(</span>Id</span><</span>T</span>>): u64 = [</span>generation:</span> upper </span>32</span> bits][</span>index:</span> lower </span>32</span> bits] </span></code></pre> When you insert</code>, the handle encodes the current generation and slot index. When you remove</code>, the slot's generation increments. When you get</code> with a stale handle, the generation in the handle doesn't match the slot's generation — the pool returns null instead of the wrong data.</p> This prevents the classic use-after-free</strong> / type confusion</strong> bug:</p> 1. Insert User at slot 0, generation 0 → handle = 0x00000000_00000000 </span>2. Remove User → slot 0 generation becomes 1 </span>3. Insert Projectile at slot 0, generation 1 → handle = 0x00000001_00000000 </span>4. Try to read old User handle (gen 0) → GENERATION MISMATCH → null </span></code></pre> No garbage collector. No reference counting. Just a 4-byte integer comparison per access.</p> When to use pools</h3> Game entities</strong> (ECS): enemies, projectiles, particles — frequently spawned and destroyed</li> Connection pools</strong>: TCP clients, database handles — allocated and returned</li> Object caches</strong>: LRU-style caches where entries are evicted and reused</li> Any workload with frequent insert/remove</strong> where index-based access would leave holes</li> </ul> Pools vs Lists</h3> Operation</th> List T[]@list</code></th> Pool T[N]@pool</code></th></tr></thead> Append/Insert</td> O(1) amortized</td> O(1) via free stack</td></tr> Access</td> list[i]</code> → T</code> (direct)</td> pool[id]</code> → ?T</code> (requires OR</code>)</td></tr> Remove from middle</td> O(N) shift</td> O(1) mark-dead</td></tr> Stable references</td> No (realloc invalidates)</td> Yes (handles survive realloc)</td></tr> Capacity</td> Dynamic (grows on demand)</td> Fixed (pre-allocated at declaration)</td></tr> Memory after remove</td> Compacted</td> Holes (reused on next insert via free stack)</td></tr> Safety after remove</td> Index may now point to different element</td> Handle returns null (generational)</td></tr> Ergonomic cost</td> Direct access — no unwrapping</td> Every access needs OR</code> fallback</td></tr> </tbody></table> Decision Tree</h2> Is the size fixed at compile time? </span>├── Yes → T[N] (array) </span>└── No </span> ├── Access pattern? </span> │ ├── Sequential / index-based → T[]@list (list) </span> │ └── Handle-based / frequent insert+remove → T[N]@pool (pool) </span> └── Need stable references across insert/remove? </span> ├── Yes → T[N]@pool (generational handles survive mutations) </span> └── No → T[]@list (simpler, more cache-friendly for iteration) </span></code></pre> Note: pools require a compile-time capacity N</code>. All slots are pre-allocated up front, giving O(1) insert/remove via an internal free stack with no runtime reallocation. Choose a capacity that covers your expected maximum — the pool will panic if you exceed it.</p> Hash Maps — HashMap<V></code> and HashMap<K, V></code></h2> Key-value maps. String-keyed by default, numeric-keyed with explicit HashMap<K, V></code>.</p> MUTABLE </span>scores: </span>HashMap</span><</span>Int64</span>> = {}; </span># String → Int64 </span>scores[</span>"</span>alice</span>"</span>] = 100_i64; </span>scores[</span>"</span>bob</span>"</span>] = 200_i64; </span>val = scores[</span>"</span>alice</span>"</span>]; </span># 100 </span>scores.delete(</span>"</span>bob</span>"</span>); </span>scores.contains?(</span>"</span>alice</span>"</span>); </span># TRUE </span>scores.count(); </span># 1 </span></code></pre> When to use</strong>: Lookup by key. The general-purpose associative container.</p> SOA Layout — Cache-Optimal Iteration</h2> By default, collections store elements as Array of Structures</strong> (AOS): each element is a contiguous block of all its fields. When a pipeline accesses only a subset of fields, the CPU loads entire cache lines but uses only a fraction — wasting memory bandwidth.</p> The :soa</code> capability switches to Structure of Arrays</strong> layout: each field gets its own contiguous array. A pipeline that touches one field iterates a dense, cache-friendly slice.</p> STRUCT Entity </span>{ </span>x: </span>Float64</span>, </span>y: </span>Float64</span>, </span>vx: </span>Float64</span>, </span>vy: </span>Float64</span>, </span>health: </span>Float64</span>, </span>mana: </span>Float64</span>, </span>name: </span>String, </span>level: </span>Float64 </span>} </span> </span># AOS (default): each entity is 8 fields × 8 bytes = 64 bytes per cache line. </span># SUM _.health loads all 8 fields but uses only 1 — 87.5% wasted bandwidth. </span>MUTABLE </span>pool: </span>Entity</span>[</span>10000</span>]</span>@pool </span>= []; </span> </span># SOA: health values are stored in a contiguous f64 array. </span># SUM _.health touches only that array — zero waste. </span>MUTABLE </span>pool: </span>Entity</span>[</span>10000</span>]</span>@pool</span>:soa </span>= []; </span></code></pre> The compiler detects when SOA would help and suggests it:</p> NOTE: Pipeline accesses 1 of 8 fields (health). Consider @soa </span> for better cache performance on 'Entity'. </span></code></pre> When :soa</code> helps most:</strong></p> Pipelines that touch < 50% of fields (SUM, MIN, MAX, AVERAGE on one field)</li> Large structs (8+ fields)</li> High-volume iteration (thousands of elements)</li> </ul> When :soa</code> doesn't help:</strong></p> Small structs (< 4 fields) — cache lines already fit the whole struct</li> EACH that reads/writes most fields — no bandwidth savings</li> Random-access by handle (pool.get(id)</code>) — must reassemble the struct</li> </ul> How it works under the hood:</strong></p> All pipeline operators (SUM, MIN, MAX, AVERAGE, COUNT, ANY, ALL, WHERE, FIND, SELECT) automatically use field-slice iteration on :soa</code> pools. The compiler rewrites _.health</code> to data.items(.health)[i]</code> — a direct index into the contiguous field array. No materialization, no struct reassembly for the common case.</p> For operators that produce struct output (WHERE, FIND), structs are reassembled only for matching elements — the predicate still uses field-slice access, so most iterations touch only the predicate field.</p> :soa</code> works on both @pool</code> and @list</code>:</p> MUTABLE </span>pool: </span>Entity</span>[</span>10000</span>]</span>@pool</span>:soa </span>= []; </span># SOA pool (generational handles) </span>MUTABLE </span>items: </span>Entity</span>[]</span>@list</span>:soa </span>= []; </span># SOA list (dense, indexed) </span></code></pre> Same API as their non-SOA counterparts. The difference is invisible except in pipeline performance.</p> FFI restriction:</strong> @soa</code> collections cannot be passed to EXTERN FN</code> — SOA memory layout is incompatible with the C ABI. Materialize to a regular collection first if you need FFI.</p> Sharding — True Shared-Nothing Parallel Collections</h2> Lists, pools, and hash maps all support sharding for parallel access:</p> MUTABLE </span>data: </span>Float64</span>[]</span>@list</span>:sharded</span>(</span>4</span>) = []; </span>MUTABLE </span>entities: </span>Enemy</span>[</span>10000</span>]</span>@pool</span>:sharded</span>(</span>4</span>) = []; </span>MUTABLE </span>counts: </span>HashMap</span><</span>Int64</span>></span>@sharded</span>(</span>4</span>) = {}; </span></code></pre> Sharding splits the collection into N independent partitions. Each shard is a complete, independent data structure — no shared state, no locks, no atomic operations between shards.</p> How it works:</strong></p> Lists</strong>: Round-robin distribution on append</code>. length()</code> sums across shards.</li> Pools</strong>: Round-robin distribution on insert</code>. Handle encodes shard index in upper bits.</li> Hash maps</strong>: Key-based routing via hash(key) % N</code>. Same key always maps to same shard.</li> </ul> Pipeline operations</strong> (\|> EACH</code>, \|> SUM</code>, \|> WHERE</code>, etc.) process each shard in parallel via DO blocks — one fiber per shard, no contention between them.</p> The DragonflyDB Model — Each Thread Owns Its Partition</h3> @sharded(N)</code> uses the same architecture as DragonflyDB: each scheduler thread owns a subset of shards, and fibers pinned to that scheduler access those shards with zero locks, zero atomics, and zero synchronization.</p> Shard ownership</strong>: Shard i</code> is owned by scheduler i % S</code> (where S = number of schedulers). With 8 shards and 2 schedulers, each scheduler owns 4 shards.</p> Hot path</strong> (shard owned by the current scheduler): Direct access. No locks, no atomics. Cooperative scheduling ensures only one fiber runs at a time per scheduler, so there's no data race.</p> Cold path</strong> (shard owned by another scheduler): The operation is transparently routed to the owning scheduler via SPSC ring buffers. The calling fiber yields until the owning scheduler executes the operation inline and signals completion via an atomic done flag. Correct, but slower — design your workload so each fiber processes ONLY its own partition.</p> SHARD</code> Pipeline — True Shared-Nothing Workloads</h3> The SHARD</code> pipeline operator partitions work so each fiber owns its shard exclusively:</p> MUTABLE </span>map: </span>HashMap</span><String></span>@sharded</span>(</span>8</span>) = {}; </span>n = </span>1000000</span>; </span> </span># SHARD routes each key to the scheduler that owns its shard. </span># CONCURRENT EACH runs one fiber per shard — zero locks, zero routing. </span>(</span>0</span>..<n) </span> \|> </span>SHARD</span>(</span>"</span>key:${toString(_)}</span>"</span>, map) </span> \|> </span>CONCURRENT EACH </span>{ map[_] = </span>"</span>value</span>"</span>; }; </span></code></pre> How it works</strong>:</p> SHARD hashes each key, appends it to the owning shard's queue</li> One fiber per shard is pinned to the owning scheduler</li> Each fiber processes only its own queue — no cross-shard access, no locks</li> The result is true DragonflyDB-style shared-nothing: N partitions, N fibers, zero synchronization</li> </ol> When to use @sharded(N)</code> vs @sharded(N):locked</code></strong>:</p> @sharded(N)</code> + CONCURRENT EACH</code>: each fiber owns its partition. Maximum throughput.</li> @sharded(N):locked</code>: any fiber can access any key. RwLock per shard. Use when the workload can't be partitioned (e.g., random key distribution across all fibers).</li> </ul> One-Line Optimization</h3> @sharded(N)</code> is a one-line declaration change. Functions don't need to know — a function that takes HashMap<String></code> seamlessly accepts HashMap<String>@sharded(8)</code>:</p> # One-line change: add @sharded(8) to the declaration </span>MUTABLE </span>map: </span>HashMap</span><String></span>@sharded</span>(</span>8</span>) = {}; </span> </span># Functions: no @sharded needed in parameter types </span>FN</span> doWork!(</span>MUTABLE </span>map: </span>HashMap</span><String>, </span>key: </span>String) </span>RETURNS </span>!</span>Void </span>-> </span> map[key] = </span>"</span>value</span>"</span>; </span>END </span> </span># BG blocks: auto-pinned when they capture a @sharded map </span>BG </span>{ doWork!(map, </span>"</span>hello</span>"</span>); } </span># [Note] BG block auto-pinned — captures shared/locked resource. </span></code></pre> The compiler handles everything:</p> Unified API</strong>: All HashMap variants (plain, @sharded</code>, @sharded:locked</code>) have the same .put()</code>/.get()</code> interface. Functions use Zig anytype</code> generics to accept any variant.</li> Auto-pinning</strong>: BG blocks that capture a @sharded</code> map are automatically pinned to a scheduler. No @pinned</code> annotation needed.</li> Fiber distribution</strong>: Pinned BG fibers are distributed round-robin across schedulers via spawnPinned()</code>, so each scheduler gets work.</li> </ul> Sharding Variants</h3> Variant</th> Model</th> Locking</th> Use when</th></tr></thead> @sharded(N)</code></td> Shared-nothing</td> None (fiber-pinned)</td> Partitioned workloads (CONCURRENT EACH)</td></tr> @sharded(N):locked</code></td> RwLock per shard</td> Per-shard RwLock</td> Cross-shard access from any fiber</td></tr> @sharded(N):writeLocked</code></td> RwLock per shard</td> Per-shard RwLock</td> Same as :locked (alias)</td></tr> </tbody></table> When You Need Shared Mutable Maps</h3> For rare cases where multiple schedulers must read/write the same keys concurrently (e.g., a global session registry), use @shared:writeLocked</code>:</p> MUTABLE </span>sessions: </span>HashMap</span><</span>Session</span>> </span>@shared</span>:writeLocked </span>= {}; </span># Arc<RwLock<HashMap>>: readers are parallel, writers are serialized </span></code></pre> This is CLEAR's escape hatch — it works like Java's ConcurrentHashMap</code> but with explicit intent. The capability annotation makes the cost visible at the declaration site, and the compiler's capability audit will tell you if you're paying for it unnecessarily.</p> Hashing — Secure by Default</h2> CLEAR uses seeded Wyhash</strong> as the default hash function for all hash maps. The seed is randomized per process, which prevents Hash DoS attacks (where an attacker crafts keys that all collide, turning O(1) lookups into O(n) traversals).</p> This follows CLEAR's "Correct > Scalable > Fast" ordering: the default is safe against adversarial input, with near-zero overhead compared to unsalted hashes.</p> Future: @fastTrusted</code> Capability</h3> For performance-critical internal infrastructure where keys are not attacker-controlled (e.g., internal service meshes, compiler symbol tables, game engine registries), CLEAR plans to offer an opt-in unsalted hash:</p> # Planned syntax (not yet implemented): </span>MUTABLE </span>symbols: </span>HashMap</span><</span>SymbolInfo</span>></span>@fastTrusted </span>= {}; </span>MUTABLE </span>fast_sharded: </span>HashMap</span><</span>Int64</span>></span>@sharded</span>(</span>8</span>)</span>:fastTrusted </span>= {}; </span></code></pre> @fastTrusted</code> would switch to unsalted FNV-1a or raw Wyhash — faster, but vulnerable to Hash DoS if keys come from untrusted input. The name is deliberate: it forces the developer to declare "I trust this data source."</p> Why not yet</strong>: The default seeded Wyhash is fast enough for v1 (within ~5% of unsalted FNV). The capability will be added when real-world profiling shows the hash function is the bottleneck, not before. Premature escape hatches encourage premature optimization.</p> Shared Data in CLEAR 2026-03-25T00:00:00+00:00 CLEAR separates:</p> what data is</strong> (Types) from</li> how it's accessed</strong> (Gates & Boundaries) from</li> the strategy it uses</strong> (Capabilities) from</li> when it's available</strong> (Tense: ~T</code>)</li> </ol> Types, Capabilities, and Execution Boundaries</h2> Rust is not inherently complicated, nor is its type system much more complex than C++'s. The actual friction comes from Rust using a 'God-like' type system forced to handle too many things at once.</p> CLEAR breaks these into 4 separate components:</p> Types:</strong> The actual data / memory (User)</li> Capabilities:</strong> What you’re allowed to do with that data (User @shared:locked</code>)</li> Execution Boundaries:</strong> When you're do things concurrently or in parallel (BG/DO/CONCURRENT</code>)</li> Synchronization Gates:</strong> Where you explicitly mutate state concurrently (WITH sharedUser AS user { … }</code>)</li> </ul> In CLEAR, concurrency is a property of the execution boundary, not the data type. CLEAR requires explicit WITH</code> blocks for synchronization. While slightly less ergonomic than Rust for a simple Mutex, this design choice yields key structural dividends:</p> Zero-Friction Pass-Through:</strong> ~85% of your codebase handles shared data without a single synchronization annotation, making code substantially more maintainable / refactorable to optimize for different synchronization strategies.</li> Polymorphic Synchronization:</strong> A single function can gracefully handle any of LOCKED</code>, SNAPSHOTTED</code>, ACTOR</code>, BUFFERED</code> etc data using a single block of code.</li> Pinpoint Cost Visibility:</strong> You know exactly where and when your code blocks, yields, retries and the associated failure methods. Latency costs are isolated to the 15% of functions that actually mutate state, rather than hidden in a type signature 10 levels up the stack.</li> </ul> You do not need to worry about a developer accidentally “sneaking” a slow actor synchronization into a hot loop. The costs are enforced ONLY at the synchronization boundaries (WITH</code> blocks). The compiler will simply reject incompatible synchronization strategies inside hot loops.</p> FN</span> transact( </span> </span>a: </span>Account@shared</span>, </span> </span>b: </span>Account@shared</span>, </span> </span>amount: </span>Float64 </span>) </span> </span>RETURNS </span>!</span>Bool </span> </span>REQUIRES</span> a, </span>b: </span>LOCKED </span>\| </span>SNAPSHOTTED </span>\| </span>BUFFERED </span>-> </span> </span>IF</span> amount <= </span>0 </span>-> </span>RAISE</span>; </span> </span> </span>WITH </span> </span>POLYMORPHIC</span> a </span>AS</span> acctA, </span> </span>POLYMORPHIC</span> b </span>AS</span> acctB { </span> </span>IF</span> acctA.balance <= amount -> </span>RAISE</span>; </span> acctA.balance -= amount; </span> acctB.balance += amount; </span> } </span> </span>ON LockTimeout </span>... </span> </span>ON Conflict </span>... </span> </span>ON QueueFull </span>... </span> </span>END </span>END </span></code></pre> There is a user-defined default policy for handling failure methods. Not every function needs to specify failures:</p> FN</span> transact(</span>a: </span>Account@shared</span>, </span>b: </span>Account@shared</span>) </span>RETURNS </span>!</span>Bool </span>-> </span> </span>WITH POLYMORPHIC</span> a </span>AS</span> acctA, </span>POLYMORPHIC</span> b </span>AS</span> acctB { ... } </span>END </span></code></pre> This function will take any shared object: Mutex (@locked</code>), RwLock (@writeLocked</code>), MVCC (@versioned</code>), AtomicPtr (@atomic</code>), RingBuffer (@buffered</code>), Actor (@actor</code>). It will handle failure methods by the user-defined synchronization policy (or the system default):</p> SYNC POLICY </span> </span>ON MvccConflict RETRY</span>(</span>3</span>) </span>THEN RAISE </span> </span>ON AtomicConflict RETRY</span>(</span>3</span>) </span>THEN RAISE </span> </span>ON LockTimeout RETRY</span>(</span>3</span>) </span>THEN RAISE </span> </span>ON QueueFull BLOCK </span> </span>ON ActorTimeout RAISE </span>END </span></code></pre> Third-party code / libraries must compile in STRICT</code> mode, which requires them to explicitly handle all failure methods. There is no chance your default sync policy creates spooky-action-at-distance in third party failure handling.</p> FAQ:</p> But what about lock acquisition? CLEAR automatically sorts locks. It raises errors on deadlock that MUST be handled if possible (regardless of compilation mode), rather than deadlocking.</li> </ul> </li> But what about the Default Policy causing Spooky-Action-at-a-Distance internally? Compile in STRICT</code> mode where this is not allowed, and then you know all synchronization failure methods are handled locally.</li> Default synchronization failure policies are meant to be a speed up for prototyping, not something to use in production.</li> </ul> </li> But what about distributed failures? @actor</code> in CLEAR does not imply a distributed actor.</li> There is @shared:actor</code> (single-machine) and @distributed:actor</code> (n-machines).</li> @distributed</code> is not interchangeable with @shared</code>.</li> FN foo(x: T@shared)</code> does not accept a @distributed:actor</code>.</li> FN foo(x: T@distributed)</code> does not accept a @shared:actor</code> (it may not exist on the machine!)</li> </ul> </li> </ul> @actor</code> and @buffered</code> are in scope for v0.3. They are not currently available. @distributed</code> is in scope for v0.4.</p> </div> Analogy</h3> Think of DO/BG/CONCURRENT</code> blocks as Execution Boundaries, and ownership capabilities (@shared</code>) as your Keys to pass the Boundary. A type without proper ownership cannot pass through an Execution Boundary (that is unsafe).</p> If you want to do something in parallel (on multiple cores at once), the data must be @shared</code>. If you want to do something concurrently (on a single core at once), the data can be @shared</code> OR @local</code>.</p> Similarly, a type with a synchronization capability (@locked</code>, @versioned</code>, etc) requires crossing a gate for safe access.</p> The WITH</code> block acts as this gate to safe access as the BG/DO blocks act as the gate to safe boundary crossing. WITH</code> blocks require a type of permission to safely access synchronized data - this gives the reader insight into the cost.</p> Ownership (the keys)</th> Execution Boundaries</th> Synchronization</th> Access Gates</th> Access Permission (cost model)</th></tr></thead> @shared (any cores)</td> BG {}</td> @locked, @writeLocked</td> WITH {}</td> EXCLUSIVE</td></tr> @local (pinned to a core)</td> DO {}</td> @versioned, @atomic</td> </td> SNAPSHOT</td></tr> </td> CONCURRENT {}</td> @actor, @buffered</td> </td> EVENTUAL</td></tr> </tbody></table> @multiOwned</code> is an ownership capability, but it does not grant access through Execution Boundaries. It allows an object to have multiple aliases (multiple owners - i.e. for a graph), but it does NOT grant permission to cross Execution Boundaries.</p> Complexity Reduced? Or just moved around?</h3> It is a fair critique from Rust or Go that this is nothing new, and arguably more complex. Rust has a single system to keep track of. CLEAR has three. If anything, that’s more complex, not less.</p> CLEAR thinks it is clearer to break the problem down into steps. For one, developers not familiar with concurrent code know that anything @shared</code> is something concurrent. In Rust, it is hard to distinguish between Arc<RwLock<T>></code>, DashMap<T></code>, and Vec<T></code>. Which one is something concurrent, that could have logical race bugs?</p> Secondly, while this might appear to be unergonomic, the concurrent benchmarks in CLEAR are only ~70% of the code as Rust (much more expressive than non-Rust people probably assume), and ~30% of the code as Go.</p> Lines of code on its own is a terrible benchmark for complexity. But CLEAR did not attempt to cherry-pick this metric. This is on top of the benefit of having polymorphic synchronization for almost no cost, and the ability to benchmark different strategies almost free.</p> Thirdly and most crucially, polymorphic synchronization allows your code to be maintainable in a way that is very difficult to achieve in Rust or Go. Arc<RwLock<T>></code> is viral in Rust. T@shared</code> is not in CLEAR.</p> Step 1) update your with blocks to use POLYMORPHIC access permission.</li> Step 2) change your synchronization strategy from @locked to @versioned.</li> Step 3) run clear profile</code>.</li> Step 4) profit.</li> </ul> CLEAR levels the playing field for non-experts</h3> In Rust or Go, you need to fully understand the implications of your code before you even write and test it. Unless you are an expert (and even if you are), this is difficult to get right. If you get it wrong, it is typically painful to rearchitect your app to do it right. It is also possible after your code continues to grow, you may need to switch again (possibly back to where you started), which can be equally painful.</p> CLEAR was designed assuming that you cannot get everything right from the get-go, and to make it as easy as possible to experiment and optimize to get it right. It was designed assuming you want to get there, even if you would have no chance in Rust or Go, because you need to understand things at a deeper level.</p> CLEAR was designed to act like a high-level language like Ruby, but give you systems level speed.</p> Capabilities in Depth</h2> When multiple fibers need to access the same data, CLEAR must answer two questions:</p> Lifetime</strong>: How does the data stay alive when multiple fibers reference it?</li> Thread safety</strong>: Is the reference count safe across schedulers (OS threads)?</li> </ol> Quick Reference</h2> Capability</th> Zig Type</th> Refcount</th> Thread-safe?</th> @parallel</code></th> Use when</th></tr></thead> @local</code></td> T</code></td> None</td> No (single-scheduler)</td> Error</strong></td> Fibers on one scheduler share mutable state</td></tr> @multiowned</code></td> Rc(T)</code></td> Non-atomic</td> No (single-scheduler)</td> Error</strong></td> Multiple owners, single scheduler, read-mostly</td></tr> @shared</code></td> Arc(T)</code></td> Atomic</td> Yes</td> Allowed</td> Cross-scheduler sharing (rare)</td></tr> </tbody></table> @local — Zero-Cost Shared Mutable Reference</h2> MUTABLE</span> c = </span>Counter</span>{ </span>value: </span>0 </span>} </span>@local</span>; </span> </span>BG </span>{ c.value += </span>1</span>; } </span># direct field access, no WITH block </span>BG </span>{ print(c.value); } </span># direct read </span></code></pre> What it does</strong>: Heap-allocates the value and returns a bare T</code> pointer. No Mutex, no RwLock, no reference counting. Multiple fibers share the pointer by value copy.</p> Why it's safe</strong>: The compiler auto-pins all BG/DO blocks that capture @local</code> variables to the local scheduler. Cooperative scheduling on a single OS thread means no two fibers ever execute simultaneously — no data races are possible.</p> When to use it</strong>: This is the default choice</strong> for shared mutable state within a function scope. It's the fastest option — zero synchronization overhead.</p> Compile-time enforcement</strong>:</p> BG </span>{ </span>@parallel </span>-> c.value = </span>1</span>; } </span># ERROR: @local variable cannot be used in @parallel block </span></code></pre> @multiowned — Non-Atomic Reference Counting (Rc)</h2> node = </span>TreeNode</span>{ </span>left: </span>NIL</span>, </span>right: </span>NIL </span>} </span>@multiowned</span>; </span> </span># Multiple owners via WITH block: </span>WITH</span> node </span>AS</span> val { print(val.left); } </span></code></pre> What it does</strong>: Wraps the value in Rc(T)</code> — a non-atomic reference-counted pointer. Each WITH</code> unwrap increments the refcount; scope exit decrements it. The value is freed when the last reference is released.</p> What it does</strong>: Wraps the value in Rc(T)</code> — a non-atomic reference-counted pointer. Each WITH</code> unwrap increments the refcount; scope exit decrements it. The value is freed when the last reference is released.</p> Why it exists</strong>: For graph structures</strong> and shared ownership</strong> patterns where multiple variables need to keep the same value alive. Unlike @local</code> (which has one owner and shared pointers), @multiowned</code> has multiple owners with automatic lifetime management.</p> Read-only access</strong>: WITH node AS val</code> provides read-only</strong> access to the inner value. @multiowned</code> does not support mutation — it's shared ownership, not shared mutation. This is analogous to Rust's Rc<T></code> (not Rc<RefCell<T>></code>).</p> For shared mutation</strong>, use:</p> @alwaysMutable</code> — interior mutability (RefCell</code>), mutate through const bindings</li> @local</code> — zero-cost, single-scheduler, direct field writes</li> @locked</code> — mutex-protected, cross-scheduler safe</li> </ul> Why it's NOT thread-safe</strong>: Rc</code> uses a plain integer for its refcount — no atomic CAS, no memory barriers. If two threads increment/decrement simultaneously, the count corrupts (use-after-free or double-free).</p> Compile-time enforcement</strong>:</p> BG </span>{ </span>@parallel </span>-> </span>WITH</span> node </span>AS</span> val { ... } } </span># ERROR: @multiowned (Rc) variable cannot be used in @parallel block — </span># Rc uses a non-atomic reference count. Use @shared (Arc) for cross-scheduler sharing. </span></code></pre> When to use it</strong>: Graphs, trees, and shared ownership patterns where all fibers run on the same scheduler and data is read-only. If you need mutation, use @local</code>. If you need cross-scheduler sharing, use @shared</code>.</p> @shared — Atomic Reference Counting (Arc)</h2> Technically @shared</code> means permission to cross an execution boundary. Atomics and atomic pointers (@shared:atomic</code>) are NOT wrapped in Arc.</p> </div> config = </span>AppConfig</span>{ </span>port: </span>8080 </span>} </span>@shared</span>; </span> </span>BG </span>{ </span>@parallel </span>-> </span>WITH</span> config </span>AS</span> c { print(c.port); } } </span># OK </span>BG </span>{ </span>@parallel </span>-> </span>WITH</span> config </span>AS</span> c { print(c.port); } } </span># OK </span></code></pre> What it does</strong>: Wraps the value in Arc(T)</code> — an atomic reference-counted pointer. The refcount uses hardware atomic instructions (lock-prefixed CAS on x86), so it's safe to increment/decrement from any thread.</p> Why it's expensive</strong>: Every clone/drop of an Arc</code> bounces the cache line containing the refcount between CPU cores. On a 16-core machine, this "cache-line bouncing" can cost ~100ns per operation (vs ~1ns for a non-atomic increment).</p> When to use it</strong>: Only when you genuinely need</strong> cross-scheduler sharing — data accessed by fibers on different OS threads. This is rare in practice: most shared state is within a single function scope (use @local</code>) or within a single scheduler (use @multiowned</code>).</p> Combining with Sync Capabilities</h2> Sharing capabilities can be combined with sync capabilities for mutable cross-thread access:</p> # Arc + Mutex: one-line mutations auto-lock </span>MUTABLE</span> counter = </span>Counter</span>{ </span>value: </span>0 </span>} </span>@shared</span>:locked</span>; </span>BG </span>{ </span>@parallel </span>-> counter.value += </span>1</span>; } </span># auto mutex </span> </span># Arc + RwLock: cross-scheduler read-heavy access </span>MUTABLE</span> config = </span>Config</span>{ </span>port: </span>8080 </span>} </span>@shared</span>:writeLocked</span>; </span>BG </span>{ </span>@parallel </span>-> </span>WITH</span> config </span>AS</span> c { print(c.port); } } </span># read lock </span>BG </span>{ </span>@parallel </span>-> config.port = </span>9090</span>; } </span># auto write lock </span></code></pre> @local</code> does not combine with @locked</code> or @writeLocked</code> — it's already a bare pointer with no wrapper. Mutation is direct.</p> Decision Tree</h2> Do multiple fibers need to access this data? </span>├── No → default (no capability, single owner) </span>└── Yes </span> ├── Does it need mutable access? </span> │ ├── Same scheduler → @local (zero cost, direct field writes) </span> │ └── Cross-scheduler → @locked or @writeLocked (mutex/rwlock) </span> └── Do multiple fibers need to OWN it (keep it alive)? </span> ├── Read-only, same scheduler → @multiowned (Rc) </span> ├── Read-only, cross-scheduler → @shared (Arc) </span> └── Mutable + shared ownership → @local (same scheduler) </span> or @shared:locked (cross-scheduler) </span> </span>Stable heap address needed (graph edges, self-referential)? </span>└── @indirect (combinable with any of the above) </span></code></pre> Note on primitives</strong>: Capabilities cannot be applied to primitive types (Int64</code>, Float64</code>, Bool</code>, Byte</code>, Float32</code>). Wrap in a STRUCT</code> first — this makes the intent explicit and gives you named fields.</p> Auto-Pinning</h2> The compiler automatically pins BG/DO blocks to the local scheduler when they capture @local</code>, @multiowned</code>, @shared</code>, @locked</code>, or @writeLocked</code> variables. This is a cache-line bouncing optimization</strong> for thread-safe types and a safety requirement</strong> for non-thread-safe types.</p> To override auto-pinning for thread-safe types:</p> BG </span>{ </span>@parallel </span>-> ... } </span># distribute across schedulers </span></code></pre> For non-thread-safe types (@local</code>, @multiowned</code>), @parallel</code> is a compile error.</p> THE ANATOMY OF A GOOD FUNCTION 2026-03-01T00:00:00+00:00 The most desired behavior from any function is that it works</strong>.</p> A function should exist to serve the user, not the compiler. It should (in order of preference):</p> Return (and do) what the user wants</li> Return a suitable default (if possible/applicable)</li> Return an explicit set of errors</li> </ol> It should return exactly ONE</em> type. A user should never need to post-process a function's output to do what they want.</p> ALL PUBLIC FUNCTIONS MUST BE GOOD</h2> PUBLIC FN</span> getFullyFormedUser(</span>id: </span>Number</span>) -> </span>RETURNS User OR User</span>::DEFAULT </span> -- </span>1</span>. </span>Input </span>Hygiene: </span>Handle</span> invalid inputs uniformly at the top. </span> </span>GUARD</span> id > </span>0 </span>OR RETURN DEFAULT</span>; </span> </span> -- </span>2</span>. </span>The Happy </span>Path: </span>A</span> clear, easy-to-follow stream of data using </span>'</span>s></span>' </span> </span>RETURN</span> id </span> s> fetchUserFromCache </span> s> </span>OTHERWISE</span>(fetchFromDb!!) </span>OR EXIT </span>-- </span>If</span> cache misses, try </span>DB </span> s> hydrateFromOtherDb; </span> </span>-- </span>3</span>. </span>The Catch</span>-</span>All: </span>-- </span>All</span> error logic is tucked away at the bottom. </span>CATCH </span> </span>RETURN DEFAULT</span>; -- </span>Return</span> suitable default on error </span>END </span></code></pre> INTERNAL FUNCTIONS CAN BE BAD</h2> -- </span>This</span> can </span>return</span> an error </span>`</span>!!</span>` </span>-- </span>Internal</span> code can be </span>"</span>Chill-Correct</span>" </span>until</span> you</span>'</span>re ready to make it public. </span>FN reciprocalId(id) -> </span> RETURN functionThatCanError!!(id); </span>END </span></code></pre> Comparisons</h2> C:</strong> High Danger. Manual memory management and error checking.</li> Go:</strong> Visual Noise. Repetitive if err != nil</code> boilerplate kills flow.</li> Rust:</strong> High Cognitive Load. Mandatory unwrapping of every Enum/Result case.</li> CLEAR:</strong> Represents Flow</strong>. It projects the "View" of the data you want, handling the Union/Error logic implicitly based on your definitions.</li> </ul> -- </span>CLEAR</span> projects the view of the data you want </span>-- </span>1</span>. </span>Handles</span> the </span>Error </span>(defaults to empty/nil </span>if</span> it fails) </span>-- </span>2</span>. </span>Handles</span> the </span>Union </span>(projects </span>View</span>) </span>PRINT</span>(getUser().email); </span></code></pre> THE FINAL PATH</h2> CLEAR is designed as a language that is native to Property-Based Testing</strong>.</p> Because EVERY</em> public function follows a strict contract:</p> Inputs and outputs have suitable defaults (or explicit explosions).</li> We can programmatically generate permutations of every possible input for testing.</li> CLEAR enforces a Strict Contract at the Public boundary (RETURNS Type OR Default). This solves the Oracle Problem</em> for automated testing.</li> </ol> The easier you make code to understand and to test, the easier you make it to get working</strong> code.</p> And working</strong> code is all that really matters.</p> VERDICT</h2> C/Elixir represent Chaos</strong>.</li> Rust/Go represent Bureaucracy</strong>.</li> CLEAR represents Flow</strong>.</li> </ul> THE GIVE PROBLEM 2026-03-01T00:00:00+00:00 In CLEAR, we separate Types</strong> from Capabilities</strong>.</p> Type:</strong> What the data is (e.g., User</code>).</li> Capability:</strong> How the data is accessed (e.g., affine</code>, shared</code>).</li> </ul> By default, objects in CLEAR are affine</strong> (meaning they can only be used once, and passing them to a function "borrows" them). When a function needs to take ownership of an object (e.g., to store it in a long-lived list), it must explicitly TAKES</code> it.</p> The Friction: GIVE</code> vs. COPY</code></h2> If you have an affine object and you need to pass it to a function that TAKES</code> it, you have two choices:</p> GIVE it:</strong> Transfer ownership. The original variable becomes "dead" and can no longer be used.</li> COPY it:</strong> Create a duplicate. The original variable remains "alive" and can still be used.</li> </ol> STRUCT User </span>{ </span>name: </span>String } </span> </span>FN</span> addToCache(</span>TAKES </span>u: </span>User</span>) </span>RETURNS Void </span>-> </span> -- </span>This</span> function now owns </span>'</span>u</span>' </span> </span>RETURN</span>; </span>END </span> </span>u = </span>User</span>{ </span>name: </span>"</span>Alice</span>" </span>}; </span>addToCache(</span>GIVE</span> u); -- </span>Explicit Move </span>print(u.name); -- </span>COMPILER </span>ERROR:</span> u is dead. </span> </span>u2 = </span>User</span>{ </span>name: </span>"</span>Bob</span>" </span>}; </span>addToCache(</span>COPY</span> u2); -- </span>Explicit Copy </span>print(u2.name); -- </span>OKAY:</span> u2 is still alive. </span></code></pre> Why this is Better than Rust</h3> In Rust, "moving" or "copying" is often implicit based on the type (e.g., i32</code> is Copy</code>, String</code> is not). This makes it difficult to reason about whether a variable is still alive after a function call without knowing the exact implementation details of the type.</p> In CLEAR, the developer's intent</strong> is always explicit:</p> If you see GIVE</code>, you know the variable is gone.</li> If you see COPY</code>, you know you're paying a performance cost to keep it.</li> If you see neither, it's a Borrow</strong> (safe and fast).</li> </ul> Summary</h2> Keyword</th> Meaning</th> Result</th></tr></thead> GIVE</code></td> Move ownership</td> Original is dead, zero-cost transfer</td></tr> COPY</code></td> Duplicate data</td> Original is alive, high-cost duplicate</td></tr> (None)</td> Borrow</td> Original is alive, zero-cost reference</td></tr> </tbody></table> CLEAR optimizes for local reasoning and explicit intent, making ownership semantics intuitive for everyone.</strong></p> REENTRANCY ATE MY BABY 2025-12-19T00:00:00+00:00 WITH blocks are non-reentrant</h2> In CLEAR, when you open a WITH</code> block on an object to acquire a capability (like mutation or sharing), no other WITH</code> block for that same object can start until the first one finishes.</p> This applies to other threads, but crucially, it also applies to your own recursive functions or callbacks.</li> This restriction gives you Rust-level determinism for mutable access to shared objects, with significantly less cognitive overhead.</li> </ul> The Mental Model: Capabilities are Edges</h3> In CLEAR, we separate Types</strong> from Capabilities</strong>:</p> Types</strong> (e.g., Player</code>) are passed into functions.</li> Capabilities</strong> (e.g., shared:locked</code>, alwaysMutable</code>) are managed at the "edges" using WITH</code> blocks.</li> </ul> The Rule:</strong> Functions take the Type</strong>, not the Capability</strong>. You must unwrap the capability using WITH</code> before calling the function.</p> -- </span>CLEAR</span>'</span>s Solution: Unwrapping at the call site </span>WITH sharedPlayer AS player { </span> -- </span>'</span>player</span>'</span> is now the raw Type (User), unwrapped from its </span>'</span>shared</span>'</span> capability </span> damage!(player); -- OKAY </span> heal!(player); -- OKAY </span> </span> -- heal!(sharedPlayer); -- COMPILER ERROR! </span> -- Functions take User, not shared:locked User. </span>} </span></code></pre> This prevents the "Infinite Mirror" pattern where a function modifying an object accidentally tries to re-acquire a lock it already holds.</p> Why this matters</h3> Re-entrancy bugs (where state changes under your feet while you're in the middle of an operation) are the root of most concurrent complexity.</p> The Problem (Sheer Chaos):</h4> In most languages (Java, C, Go, Swift), you cannot trust from one line of code to the next. While you are reading a list, another thread can pop()</code> from it, causing your next line to crash with an IndexOutOfBounds</code> error.</p> The CLEAR Solution: Scoped Sovereignty</h4> By banning re-entrancy via WITH</code> blocks, CLEAR guarantees that while you are inside that scope:</p> You are the sole owner</strong> of that object's state.</li> No one else</strong> (not other threads, and not your own callbacks) can touch it.</li> </ol> This turns a global problem</strong> (unpredictable concurrency bugs) into a local problem</strong> (simple logic).</p> Comparison with Other Models</h3> Actor Model (Erlang/Go Channels):</strong> Safe, but treats your RAM like a slow network cable. It often leads to "God Actors" that serialize your entire program.</li> Rust (Borrow Checker):</strong> Perfect safety, but high cognitive cost. Rust forces you to infect every function signature with lifetimes and capability types (like Arc<RwLock<T>></code>).</li> CLEAR:</strong> Gives you Rust's safety and performance, but keeps your functions "pure" by handling capabilities at the call site.</li> </ul> Simplified Refactoring</h3> In Rust, if you change an object from Rc</code> (single-threaded) to Arc</code> (multi-threaded), you must update every function signature that takes that object.</p> In CLEAR, you only change the capability at the definition site. Because your functions only take the raw Type (e.g., User</code>), the rest of your codebase remains untouched. The "Refactoring Blast Radius" is zero.</p> A final note on local reasoning</h3> CLEAR optimizes for the ROI on brain power</strong>.</p> Rust</strong> optimizes for the toaster (minimum RAM/CPU at any cognitive cost).</li> CLEAR</strong> optimizes for the developer (maximum safety and speed with minimum cognitive load).</li> </ul> By enforcing explicit scopes for capabilities, CLEAR ensures that "poisoning" (restricting access to data) is always visible and local. You don't need to be a systems programming expert to write blazing-fast, thread-safe code. You just need to follow the WITH</code> block.</p> SHARED MUTABLE DEATH 2025-12-18T00:00:00+00:00 CLEAR - The Unfair Advantage</h2> STRUCT Account </span>{ </span> </span>balance: </span>Float64</span>, </span>} </span> </span>STRUCT Cache </span>{ </span> -- </span>Capabilities</span> are properties of the storage, not the type </span> </span>items: shared:</span>locked </span>HashMap</span><String, </span>shared:</span>locked </span>Account</span>>, </span>} </span> </span>FN</span> insert(</span>cache: </span>Cache</span>, </span>id: </span>String, </span>account: </span>Account</span>) -> </span> -- </span>Acquire </span>shared:</span>locked capability </span> sharedAccount = </span>SHARE:</span>LOCKED</span>(account); </span> </span> </span>WITH</span> cache.items { </span> -- </span>Inside</span> here, the </span>HashMap</span> is unwrapped and locked </span> _.insert(id, </span>GIVE</span> sharedAccount); </span> } </span>END </span> </span>FN</span> transact(</span>cache: </span>Cache</span>) -> </span> -- </span>CLEAR</span> automatically handles lock ordering </span>in </span>WITH</span> blocks to prevent deadlocks </span> </span>WITH</span> cache.items.get(</span>"</span>a</span>"</span>) </span>AS</span> a, cache.items.get(</span>"</span>b</span>"</span>) </span>AS</span> b { </span> a.balance += </span>10</span>; </span> b.balance -= </span>10</span>; </span> } </span>END </span></code></pre> What can go wrong? Not deadlock, not memory corruption.</li> CLEAR makes illegal states impossible at the concurrency level while</strong> being easy and ergonomic.</li> </ul> CLEAR distinguishes between Types</strong> and Capabilities</strong>:</p> Account</code> is a Type.</li> shared:locked</code> is a Capability.</li> </ul> Functions take Account</code>, not shared:locked Account</code>. This means your business logic remains pure and decoupled from your synchronization strategy.</p> Refactoring Blast Radius: Zero</h3> If you need to change from a Mutex</code> (shared:locked</code>) to an RwLock</code> (shared:writeLocked</code>), you only change the definition in the Struct. Your functions don't change because they never knew about the capability in the first place.</p> Rust - This is Safe?</h2> // Rust infection: capabilities (Arc, RwLock) are part of the type system </span>struct </span>Cache </span>{ </span> </span>sharedItems</span>: Arc<RwLock<HashMap<String, Arc<RwLock<Account>>>>>, </span>} </span> </span>// Function signatures are infected by the capabilities </span>fn </span>transfer</span>(</span>from</span>: &Arc<RwLock<Account>>, </span>to</span>: &Arc<RwLock<Account>>, </span>amount</span>: </span>i32</span>) { </span> </span>// Manual lock ordering required to avoid deadlock </span> </span>let </span>(first, second, swap) = </span>if </span>Arc::as_ptr(from) < Arc::as_ptr(to) { </span> (from, to, false) </span> } </span>else </span>{ </span> (to, from, true) </span> }; </span> </span>// ... </span>} </span></code></pre> The Safety Gap</h3> Rust guarantees you will not have a data race (memory corruption).</li> It does not guarantee you will avoid a deadlock (program halt).</li> CLEAR guarantees both by managing lock acquisition order and separating capabilities from types.</li> </ul> Swift - This is Ergonomic?</h2> Swift's Actors introduce Reentrancy</strong>. If you await</code> inside an actor, the state can change underneath you, leading to logic races. CLEAR avoids this by using explicit WITH</code> scopes that maintain invariants.</p> Go - This is Simple?</h2> In Go, Mutexes are just structs. If you accidentally pass an Account</code> by value, you copy the Mutex, and you're now protecting nothing. CLEAR treats capabilities as compiler-enforced metadata, making this error impossible.</p> C - YOLO</h2> C is both memory-unsafe and liveness-unsafe. CLEAR is the simplest language that is both memory-safe and liveness-safe.</p> CLEAR rests its case. It is both:</p> The simplest</em> language for concurrency.</li> The only liveness-safe</em> language that prevents deadlocks by design.</li> </ol> THE DLL PROBLEM 2025-12-03T00:00:00+00:00 In most languages (C, C++, Rust), you can easily create a Doubly Linked List (DLL).</p> typedef struct</span> Node { </span> </span>int</span> val; </span> </span>struct</span> Node* prev; </span> </span>struct</span> Node* next; </span>} </span>Node</span>; </span></code></pre> In a DLL, every node has a pointer to its neighbor, and every neighbor has a pointer back.</p> This is a Cycle</strong>.</p> Cycles are the enemy of deterministic memory management.</p> If you use Reference Counting</strong> (Swift/Python), A will never die because B holds a reference to it, and B will never die because A holds a reference to it. They leak memory forever.</li> If you use Arena-Based Memory</strong> (CLEAR), child nodes die when the function returns. If you try to point back to a parent, you're pointing to a corpse.</li> </ol> The Solution: IDs and Centralized Lookups</h2> In CLEAR, we forbid shared mutable cycles.</p> This seems like a huge limitation, until you realize that the vast majority of programs do not need DLLs.</strong></p> You need a DLL for: A text editor's buffer, a kernel's task scheduler, or a high-performance LRU cache.</li> You do NOT need a DLL for: A web server, a database client, a CLI tool, or a business application.</li> </ul> In CLEAR, you'd have to use a list to manage the pointers, like so.</p> STRUCT List </span>{ </span> </span>nodes: </span>Node</span>[] </span>} </span> </span>STRUCT Node </span>{ </span> </span>val: </span>Int64</span>, </span> </span>prev_idx: </span>Int64</span>, </span> </span>next_idx: </span>Int64 </span>} </span></code></pre> If this IS</em> your product/business, CLEAR will probably hinder you rather than help you.</li> CLEAR is designed to make optimization for cache locality as EASY as possible (without having to manage literally all memory like C).</li> By using IDs/Indices instead of raw pointers, CLEAR guarantees: Memory Safety:</strong> No dangling pointers.</li> Concurrency Safety:</strong> No race conditions on cyclic structures.</li> Refactoring Ease:</strong> You can move the entire list in memory without breaking pointers.</li> </ol> </li> </ul> CLEAR optimizes for the 99% case, where safety and organization are more important than raw pointer soup.</strong></p> Atomic Power: Shared Memory Without the Garbage Collector 2025-12-01T00:00:00+00:00 In CLEAR, 99% of your data lives in Arenas</strong> (notebooks) that are incinerated when a function returns. This is fast and safe, but it has one flaw: Data cannot survive the death of its creator.</strong></p> Sometimes, you need data to live longer than the function that created it.</li> Sometimes, you need data to be shared across multiple threads running in parallel.</li> </ul> Enter the 'shared' Capability and shared:atomic</code>.</strong></p> 1. What is the 'shared' Capability?</h2> In CLEAR, we separate Types</strong> from Capabilities</strong>.</p> Type:</strong> What the object is</em> (e.g., User</code>).</li> Capability:</strong> How the object is accessed</em> (e.g., shared</code>).</li> </ul> The shared</code> capability (Arc in Rust) allows an object to live in the Global Heap</strong>, survive outside its creator's arena, and be shared safely across threads.</p> -- </span>Acquisition: </span>Acquire </span>'</span>shared</span>'</span> capability </span>sharedU = </span>SHARE</span>(</span>User</span>.new()); </span> </span>-- </span>Usage: </span>Pass</span> the </span>'</span>raw</span>' </span>Type</span> to functions </span>process(sharedU); </span></code></pre> 2. Atomics: Primitives in the Ether</h2> For simple numeric types like counters, CLEAR uses the shared:atomic</code></strong> capability. An Atomic primitive lives in the Global Heap and can be safely mutated by multiple threads at the same time.</p> counter = </span>SHARE:</span>atomic(</span>0</span>); -- </span>Create</span> an atomic integer </span>counter.increment!(); -- </span>Thread</span>-safe mutation </span></code></pre> 3. How CLEAR Handles Them: "The Magic Count"</h2> CLEAR does not use a Tracing Garbage Collector (like Java/Go) that pauses your program. Instead, CLEAR uses Deterministic Reference Counting</strong>.</p> Every shared</code> object has a tiny backpack holding a number (the Reference Count). This number represents how many people are holding a handle to it.</p> Creation:</strong> Count = 1.</p> </li> Sharing:</strong> When you pass a shared</code> object to a thread (SPAWN</code>), Count increments.</p> </li> Drop:</strong> When a thread finishes, its handle goes out of scope and Count decrements.</p> </li> Destruction:</strong> The microsecond Count hits 0, the memory is freed instantly.</p> </li> The Result:</strong> Zero "Stop the World" pauses. Memory is freed precisely when it's no longer needed.</p> </li> </ul> 4. The Law: No Shared Cycles</h2> Reference counting has one fatal weakness: Cycles.</strong> If A holds B, and B holds A, their RefCounts will never hit 0. They will leak memory forever.</p> CLEAR refuses to stop your program to scan for cycles. Therefore, CLEAR enforces The Law</strong>:</p> A shared</code> object CANNOT hold a reference to another shared</code> object.</strong></p> The Topology</h3> This forces your memory to be a DAG</strong> (Directed Acyclic Graph). Data ownership always flows "Down" or "Across," never "Back Up."</p> By forbidding cycles, CLEAR guarantees:</p> No Memory Leaks:</strong> It is mathematically impossible to leak a shared</code> object.</li> No Deadlocks:</strong> You avoid the circular dependencies that often lead to circular locking.</li> No GC Pauses:</strong> There's no need to scan the heap.</li> </ol> Summary</h2> Feature</th> Arena (Default)</th> Shared / Atomic</th></tr></thead> Location</strong></td> Local Arena</td> Global Heap</td></tr> Lifetime</strong></td> Function Scope</td> Reference Counted</td></tr> Speed</strong></td> Blazing Fast (O(1))</td> Fast (System Malloc)</td></tr> Concurrency</strong></td> Thread-Private (Safe)</td> Thread-Safe</td></tr> Usage</strong></td> 99% of variables</td> Shared counters, Queues, Configs</td></tr> Cost</strong></td> Zero GC</td> Zero GC (Deterministic)</td></tr> </tbody></table> THE MATCH DILEMMA 2025-11-26T00:00:00+00:00 In standard Tagged-Union / Enum languages (Rust, Swift, Kotlin), the compiler forces you to MATCH</code> every single case of a Union type.</p> match</span> user { </span> User::SignedIn(u) => println!(</span>"</span>{}</span>"</span>, u.email), </span> User::Guest => println!(</span>"</span>Welcome guest</span>"</span>), </span> User::Banned => println!(</span>"</span>You are banned</span>"</span>), </span>} </span></code></pre> The Friction:</em> If you only care about the email of signed-in users, you still have to write code to handle the Guest and Banned cases. If you add a User::Moderator</code> later, your code breaks everywhere until you handle that new case.</p> This is Bureaucracy</strong>. It prioritizes the compiler's needs over the developer's intent.</p> The Solution: CLEAR (Fortress Architecture)</h2> CLEAR is a Fortress Language</strong>. It handles the "Safety Gap" at the public boundary, allowing your internal logic to flow smoothly.</p> In CLEAR, any Public Union must</em> be able to behave as a Product (a struct with fields) by projecting a View</strong>.</p> -- </span>CLEAR</span> allows this! </span>-- </span>If</span> the user is </span>NOT SignedIn</span>, .email returns a suitable default (empty string) </span>-- or handles the </span>Nil </span>case</span> implicitly </span>if</span> defined </span>in</span> the </span>Fortress</span> boundary. </span>PRINT</span>(getUser().email); </span></code></pre> Selective Matching</h3> In CLEAR, you match only what you care about.</p> CASE</span> getUser() </span>OF </span> </span>SignedInUser AS</span> u => doSignedInUserThing(u); </span> -- </span>No</span> need to match </span>Guest </span>or </span>Banned </span>if</span> you don</span>'</span>t care! </span> -- They flow through to the default or are ignored. </span>END </span></code></pre> The Benefit:</em> Refactoring Blast Radius: Zero.</strong> Adding a new case to a Union doesn't break existing code that doesn't care about that new case.</p> Why this is Safe</h3> CLEAR projects a View</strong> of the data. If you access a field that only exists in one variant of a Union:</p> If the variant matches:</strong> You get the data.</li> If it doesn't:</strong> You get a suitable default (defined at the boundary) or an explosion (!!</code>) if you've marked it as mandatory.</li> </ol> This preserves understandability</strong> and velocity</strong> without sacrificing safety.</p> Summary</h2> Language</th> Approach</th> Result</th></tr></thead> Rust</strong></td> Exhaustive Match</td> Bureaucracy & Fragility</td></tr> Go</strong></td> Type Assertion</td> Error-Prone & Verbose</td></tr> CLEAR</strong></td> Projected View</td> Flow & Flexibility</strong></td></tr> </tbody></table> CLEAR optimizes for the developer's intent, not the compiler's pedantry.</strong></p> CLEAR EXAMPLES 2025-11-22T00:00:00+00:00 Example Grammar</h2> FN</span> processUsers(</span>users: </span>User</span>[]) -> </span> -- </span>Pipelines</span> with </span>SMOOTH</span> operator </span>'</span>s></span>' </span> result = users </span>AS @u </span> s> </span>UNNEST</span> _.orders </span> s> </span>SELECT</span> _.price * </span>@u</span>.discount </span> s> </span>REDUCE</span>(</span>0</span>, (acc, x) -> acc + x ); </span> </span> -- </span>Default</span> to </span>"</span>high</span>"</span>; override </span>if</span> result is low </span> </span>MUTABLE</span> info = { </span>"</span>sum</span>"</span>: [</span>10</span>, </span>9</span>], </span>"</span>status</span>"</span>: </span>"</span>high</span>" </span>}; </span> </span>IF</span> result <= </span>1000 </span>THEN </span> info = { </span>"</span>sum</span>"</span>: [</span>1</span>, </span>0</span>], </span>"</span>status</span>"</span>: </span>"</span>low</span>" </span>}; </span> </span>END </span> </span> </span>RETURN</span> info[</span>"</span>sum</span>"</span>][</span>0</span>]; </span>END </span> </span>-- </span>Lambda</span> with % sigil </span>myLambda = </span>%(</span>x: Number</span>) </span>-> </span> doIt(); </span> </span>RETURN TRUE</span>; </span>END </span></code></pre> SIGILS</h3> Sigil</th> Meaning</th> Example</th></tr></thead> %(...)</code></td> Lambda parameters</td> %(x, y) -> x + y</code></td></tr> @</code></td> Pipeline binding / Directives</td> users AS @u</code></td></tr> !!</code></td> Explicit panic</td> list.get(i)!!</code></td></tr> _</code></td> Pipeline placeholder</td> s> SELECT _.name</code></td></tr> s></code></td> SMOOTH operator</td> data s> process</code></td></tr> </tbody></table> STRUCTS</h3> STRUCT Point </span>{ </span> </span>name: </span>String, </span> </span>x: </span>Number DEFAULT </span>10</span>, </span> </span>y: </span>Number DEFAULT </span>20 </span>} </span> </span>-- </span>Recursive</span> structure with </span>'</span>indirect</span>' </span>STRUCT Node </span>{ </span> </span>value: </span>Int64</span>, </span> </span>left:</span> indirect </span>Node</span>, </span> </span>right:</span> indirect </span>Node </span>} </span> </span>FN Point</span>::distance(</span>self: </span>Point</span>, </span>other: </span>Point</span>) -> </span>Number </span> -- ... </span>END </span> </span>myPoint = </span>Point</span>{ </span>x: </span>10</span>, </span>y: </span>20 </span>}; </span>oPoint = </span>Point</span>{ </span>x: </span>10</span>, </span>y: </span>10 </span>}; </span> </span>-- </span>Method</span> call syntax (</span>UFCS</span>) </span>d = myPoint.distance(oPoint); </span></code></pre> CONTROL FLOW</h3> -- </span>WHILE Loop </span>WHILE</span> x > </span>0 </span>DO </span> ... </span> </span>IF </span>... </span>THEN BREAK </span>END </span>END </span> </span>-- </span>IF</span>/</span>ELSE </span>IF</span> x == </span>1 </span>THEN </span> do1(); </span>ELSE_IF</span> x == </span>2 </span>THEN </span> do2(); </span>ELSE </span> doElse(); </span>END </span> </span>-- </span>WITH Capability Block </span>WITH</span> sharedAccount </span>AS</span> acc { </span> acc.balance += </span>100</span>; </span>} </span></code></pre> ERROR HANDLING</h3> FN</span> myFunc(a, b, c) -> </span> -- </span>OR RAISE</span> bubbles up the error </span> val = fetchData(a, b, c) </span>OR RAISE </span> s> parseHeader </span>OR EXIT </span>"</span>Invalid Header</span>" </span> s> parseBody </span>OR EXIT </span>"</span>Invalid Body</span>" </span> s> fetchUser </span> s> </span>RECOVER</span>(</span>DefaultUser</span>()) </span> </span>CATCH ParseError WITH</span>(</span>"</span>Invalid Header</span>"</span>) </span> logInvalidHeader(%e.snapshot); </span> </span>RETURN</span> defaultPage(); </span>DEFAULT </span> logUnknownError(%e); </span> </span>RAISE </span>%e; </span>END </span></code></pre>

CLEAR — Retrospective

Can AI Un-Slop Itself?

2026-05-16T00:00:00+00:00

Everyone knows that LLMs can, at least, sometimes create slop.

The interesting question isn’t whether they can create slop. It’s: can they un-slop themselves?

Problem</h2>
I dreamed of a programming language for 10 years. After Gemini 3.1-pro, I figured LLMs were good enough that I should at least finally see what this AI "vibe-coding" craze was all about.
I set on a 6-month journey to build a programming language.

Within 2-months, I had a custom runtime built in Zig "competitive" with Go & Tokio.</li>
Within 3-months, I had an Affine Ownership-based "memory safe" language like Rust - but (in my opinion) much more intuitive.</li> </ul>
The problem is: this only barely worked, and no one wants a barely working programming language!
What didn’t work</h2>
WRT the language:</h3>

Architecturally, the LLMs built it without a MIR-pass.</li>
Without boring you too much, it’s virtually impossible to guarantee affine ownership / memory safety without a MIR-pass.</li>
So I had a memory safe language that WASN’T memory safe...</li>
Nobody wants a Rust that regularly leaks memory and still has UAF and double free bugs...</li> </ul>
WRT the runtime:</h3>

More of the same</li>
There was no TSan testing in place</li>
There was no memory ordering testing in place</li>
The LLMs built a fiber runtime - complete with hammer tests... and never bothered to test if it’s actually thread safe...</li>
Spoiler: it wasn't!</li> </ul>
What happened next?</h3>
I told the LLMs to fix their slop!
In about ~35 minutes, they screamed:

“Done, you have a complete MIR pass in place! Everything is memory safe.” </blockquote>
They proceeded to feed me this same line of crap for ~1000 commits over 2 months, and me constantly asking:

“If the MIR pass is done, then how come X is still there and Y is still segfaulting?” </blockquote>
LLMs:

“Oh, yes, there’s just that one small part left. Okay, now it’s done!” </blockquote>
What happened next?</h2>
The definition of insanity is trying the same thing and expecting different results!
I gave up on LLMs' ability to tell me the truth about the state of a non-trivial codebase.
I got a wild idea...

If I can’t trust LLMs...</li>
Can I trust them to build me tools & systems to have more trust in them???</li> </ul>
I had LLMs build me a set of tools that don’t exist in Ruby (it's turtles all the way down).

I built the compiler in Ruby, because, initially this started off as a project I was building by hand.</li>
The initial scope was merely to play around with Syntax and see if I could develop something I liked.</li>
Never did I ever dream that I would actually get something this far.</li>
Everyone knows that a good compiler eventually self-hosts, and the goal was - like Crystal - my language would be quite close to Ruby (so LLMs should be able to migrate it easily).</li> </ol>
Anyway, Ruby has decent tooling around “code health” like SimpleCov / Flay / Flog / Reek / Debride, etc.
The problem is, if you’re building a compiler with LLMs these aren’t the core metrics you need.

LLMs repeat themselves constantly... but then forget to do something the right way in one place... and then also don’t test it.</li>
Especially in a dynamically-typed language, LLMs will randomly decide to use strings or symbols or nil to represent something, and then make thousands of defensive checks throughout your codebase... rather than fixing the problem at the source.</li>
LLMs will regularly “refactor” your code and remove 99% of usage, but still leave 1% and then move on - leaving you with competing systems... and then later use the old one that doesn’t do everything you need, and get that usage back up to 10% (plus bugs).</li> </ul>
The list goes on.
What did they build?</h2>
I’m used to developing... like myself. LLMs are different. It’s sort of like pair-programming with a genius who is occasionally extremely dumb and sometimes downright adversarial.
How do you learn to work with this crazy partner?
I rarely need to think to implement code, and I type quite fast... So LLMs are not that much faster than me at implementation. They are faster at design, but definitely not better.
Regardless, the enormous benefit they have is that they scale, whereas I do not.
If I have to review all of their code line-by-line, I’d rather just write it myself.

I don't have enough time to review the Rust code base line-by-line.</li>
If I want a safer and more intuitive Rust, no one is going to build it for me.</li>
I either get LLMs to do it, or it never gets done. Period.</li> </ul>
The goal wasn’t to try LLMs and find a reason not to use them... The goal was to find a way TO use them effectively.
If I have to review EVERYTHING LLMs write line-by-line, the bottle neck is me. The entire goal of LLM development is to get the bottleneck NOT to be me - i.e. to scale.
You can’t scale yourself... but maybe you can scale with LLMs.
So, rather than say:

I’ll just insert myself as a bottleneck and review everything line-by-line with caution… </blockquote>
I thought:

What if I put a system in place that makes it very hard for them to poke holes in? </blockquote>
What does the system look like?</h2>

Save almost everything to docs/agents (compressed)

This gives LLMs a review signal to see if what was implemented is 1) fully implemented, and 2) matches the agreement</li> </ul> </li>
Competing sets of tests. An individual LLM that touches more than 1 set of tests when not specifically asked immediately warrants manual review.

At a minimum, I need to know that they didn’t put an exception in for their new feature to be allowed to leak memory or segfault… Because they regularly try to do that rather than write working code.</li> </ul> </li>
Mutation tests

This helps you find all of their do-nothing tests that are giving you an illusion of safety, rather than providing any real value</li>
Typically, I trust myself to be able to write tests, so I don’t mutant test my own codebase.</li>
Where I work - they don’t trust me. We have mutant testing. More than once, I’ve found more than once that I’ve written a do nothing test, so, hey, maybe we should all be mutant testing anyway!</li> </ul> </li>
Tooling & Reporting

I’ll cover this next</li> </ul> </li>
Review

I’ve found the typical review process looks like:</li>
LLM screams done</li>
I ask a different LLM to check the spec and see if it’s done</li>
The LLM points out 2-3 major things NOT implemented, 2-3 things extremely poorly implemented, and 10+ things that are almost certainly bugs</li>
Feed that back to the implementation LLM</li>
Repeat for 3-5 rounds</li>
Only here do I look at anything they’ve done.</li>
This is typically ~20k lines of code per week...</li>
I can't look it over with a jade-handled spyglass inquisitively.</li>
I need to be able to pinpoint the things that NEED to be reviewed so I can review them.</li> </ul> </li> </ul>
Tooling & Reporting</h2>
Tooling & Reporting in general is a sweet spot for LLMs. They are very good at getting things nearly working.
The problem is: most things need to actually work.
In some cases, you are far better off with nothing than something that has very bad failure methods.
If you’re using a report to give out a billion dollars, you probably want a report that actually works.
But if you want a report that just gives you a probability of how much AI slop there is in a commit, and how dangerous that AI slop might be... something is far better than nothing.
The buzz phrase going around is:

LLMs increase implementation throughput far faster than they increase verification throughput. </blockquote>
What does that mean?
Invest in verification throughput!
I will continue to invest in layers of defense to protect myself from their common pitfalls and accept that 1) almost no software is perfect, even compilers, and 2) the system I have in place with LLMs is doing at least 10x better than I could do on my own.
Conclusion</h2>
Interestingly, I'm building a language that syntically makes the vast majority of bugs simply impossible to represent, and - when possible - the compiler autogenerates all the failure methods that might unfold, so you can see exactly where a bug might be.
I wanted to do this before working with LLMs. Now I want it more than ever before!
The question is: can LLMs ever get this language to work?
Time will tell... but - what I can tell you is - there’s not a shot in hell I’d ever get this done myself.
It at least seems possible with LLMs.

How to Vibe Code something that *actually* works

2026-05-15T00:00:00+00:00

LLMs are outstanding at writing code that sort of works, and finding some bugs.
The problem is: production systems need to actually work, and bugs need to be fixed correctly, otherwise it's turtles-all-the-way-down with bugs.
If you rely on LLMs to blindly fix bugs in a system that is not extremely well tested, they’ll likely “fix” one bug, and silently introduce 3 others.
Some things you can count on:</h2>

If you ask an LLM to build you something non trivial, at best, it will be sort of architecturally correct.</li>
Everything fails from here. It is very hard to arrive at something that actually works when you have the fundamentally wrong design.</li>
LLMs simply do not have the contextual capability yet to analyze medium+ sized codebases and figure out:

Is this actually broken?</li>
If so, how do I fix it?</li>
And what is the realistic path to actually fixing it without just re-inventing another system with all of the same problems (or worse)!</li> </ul> </li> </ol>
What DOES NOT work:</h2>

Test coverage alone is almost meaningless in a vibe code project.

LLMs regularly test that code successfully does not work:

This feature is intended not to work at this stage (even when you told them specifically TO fix it at that stage)</li>
Testing obviously wrong behavior: 1 + 1 = 3</code> (successfully)</li>
Tests that test nothing - these hit lines, but don’t actually test that anything works - LLMs are famous for this type of test when you blindly tell them to increase coverage</li> </ul> </li> </ul> </li>
Branch coverage alone fails for similar reasons. A line covered means nothing, but critically a branch taken does not imply all possible paths to that branch were taken</li> All possible paths in a non-trivial program is typically an undecidable halting problem: it is essentially infinite This is exactly why writing non-trivial working code is hard!</li> </ul> </li> </ul> </li> Cyclomatic complexity scores Some functions are inherently complex.</li> Breaking them up for the sake of satisfying cyclomatic complexity often makes your code worse, not better.</li> </ul> </li> </ol> Goodhart's Law:</h3> "When a measure becomes a target, it ceases to be a good measure." </blockquote> LLMs are VERY good at gaming metrics.</li> If you want code that actually works, you don’t arrive at it by gaming metrics. You arrive at it by using your brain, something LLMs don’t have.</li> </ul> </li> LLMs are a tool that can HELP you make sound decisions, and can sometimes make sound decisions.</li> But just like you cannot trust yourself, you can also not trust LLMs.</li> </ul> What DOES help:</h2> Mutation Tests</h3> A good test is load bearing. When production code changes, it should break it.</li> LLMs write tests that test nothing.</li> Mutation tests help you find them and turn them into load bearing tests that actually have value. LLMs can often fix this, and the diff should be small and easy to review.</li> </ul> </li> </ul> Coupling and Cohesion</h3> This measures how many other classes/modules rely on a piece of code (Afferent) and how many things that code relies on (Efferent).</li> LLMs default to writing spaghetti code. This helps you find it. Fixing it correctly will require using your brain.</li> The diff will typically not be small, and require a lot of review - though you can use LLM convergence to help.</li> </ul> </li> </ul> Code Duplication (DRYness)</h3> You want your production code to be DRY, your test code to repeat itself.</li> LLMs are famous for repeating themselves incorrectly. LLMs can often fix this.</li> Though telling them to do it blindly can cause bugs rather than reduce them.</li> Use your brain!</li> </ul> </li> </ul> The key is: these help. But they does not guarantee you’ll get working code, or even close. It helps you find the most common LLM problems.</li> It does not help fix them correctly, and relying on LLMs to figure out how to do that on their own is a fool's errand.</li> YOU, the human, are the driver.</li> LLMs are the car helping you arrive where you want to go faster than you could without them. If you ask the car to drive you somewhere, it might drive you off a cliff.</li> </ul> </li> </ul> The Key to Success - Signal Diversity</h2> LLMs are sort of good at everything. You can make them far better if you build a self-correcting fortress to: 1) reject their failures, and 2) accept their wins. How do you do that?</h3> A system that has worked for me seems related to how hedge funds like Bridgewater think about investments: Ray Dalio advises that, rather than focusing on a specific number of individual stocks, true diversification requires 15 or more good, uncorrelated return streams to significantly lower risk. </blockquote> If you vibe code, and you want code that works, do it in a way that you have several streams independently verifying that your code is implemented correctly. You do not need to solve the halting problem to vibe code with LLMs and get reasonable results. All you need is diversity: Your design docs Have LLMs spec out a design for you. Compress it to something actionable for them to look back on.</li> They can use this as a first layer of defense to check if what was done actually works.</li> </ul> </li> Your actual code LLMs are decent at finding some bugs.</li> They can look directly at code and get a signal if it actually works.</li> </ul> </li> Your unit tests Advise LLMs to independently write tests without looking at implementation, only function signatures and the compressed design doc.</li> They can look at the test code independent of the actual code to see if it’s a 1) test nothing test, 2) a test this successfully fails test, or 3) what you want - a test that actually tests it works correctly according to your spec / design.</li> </ul> </li> Your integration tests Advise LLMs to independently write feature/integration tests without looking at implementation, and to avoid unnecessary mocks at all costs.</li> LLMs can easily write do-nothing integration tests that mock literally everything and test nothing.</li> Just as with unit tests, this is yet another layer for them to independently look at as a signal for if the code is adequately tested</li> </ul> </li> Fuzz Tests Integration tests mainly test happy paths.</li> Without fuzzing / formal verification, it is difficult to explore all the necessary paths to ensure your code actually works.</li> Integration tests are a minimal first line of defense.</li> The reason fuzz testing is great to combine with integration testing, implemented separately - is diversity.</li> You will find several bugs here - often times the first bugs will be that the LLM wrote the fuzz testing incorrectly. But almost always they will find the gaps in your unit tests and integration tests.</li> </ul> </li> Mutant Tests Fuzz tests are valuable on their own, but mutant testing is how you know they are load bearing.</li> When combined with Fuzz testing, written independently, looking only at your docs, you are unlikely to arrive at a situation where 5 layers of code and testing all test that your code either 1) does nothing or 2) fails successfully.</li> Though LLMs can do it!</li> </ul> </li> Tooling You need reports that show all the common pitfalls LLMs have: Code Duplication - especially PARTIAL: where they checked some invariant in slightly different ways - more times than not a bug when working with LLMs - have them review each case.</li> Branch coverage - especially DECISION based: where you have untested branches specifically on key decision points - the pillars of your system / product.</li> Afferent and Efferent Reporting: to give you insights into where LLMs inevitably turned your code into spaghetti.</li> </ul> </li> </ul> </li> Review!!! Even with 6 layers of protection and tooling & reporting on code health specifically for LLM pitfalls, you need to review their work after they’ve screamed done and written all the tests.</li> Have 2-3 LLMs review on 1) architecture, 2) bugs, 3) gaps, 4) tech debt. Have them converge on which of the feedbacks are actually worth doing, and which are hallucinations or moving in the wrong direction. Save this plan to docs/agents/*.md design doc.</li> </ul> </li> Have one LLM implement the plan until the 2-3 others converge on the plan being done correctly.</li> At a BARE minimum, get the LLMs to converge on how YOU can be convinced the code actually works. Check that it works! Even at this stage, you should not be surprised if it is completely broken…</li> At a REASONABLE minimum, review the final integration testing to make sure it’s not mocking everything important.</li> </ul> </li> </ol> What is Convergence?</h3> If one LLM tells you something, there's some unknown probability that it's wrong.</li> If 2-3 tell you the same thing, it certainly does not mean it's right, but it does mean it's more likely to be right.</li> If they all agree and something and it makes sense to you, that's probably the best you're going to do on your own.</li> Never take DESIGN from LLMs blindly - i.e. just one. This is letting the car pick where to drive.</li> </ul> </li> If you personally don't know where to go, have the LLMs first agree, and then use your brain to make sure it makes sense.</li> Then, and only then, should you proceed.</li> </ul> Here is my Review Process</a>, complete with the magic words I use at different stages of development that have led to far better outcomes.
What I learned (the hard way) Building a Memory Safe Programming Language with LLMs over the last 6 months 2026-05-11T00:00:00+00:00 Background</h2> For years, I dreamed up a language like Rust, but substantially safer and more intuitive. I thought outsourcing this to LLMs was reasonable. The goal was primarily to understand their capabilities and limitations. My initial expectations were low. Verifying compiler correctness is notoriously difficult. While harder problems exist, compilers are challenging. Controlling the language limits the problem space, yet for the goals I had - the problem space would be inherently difficult to verify correctness. Even though LLMs are particularly weak here, they have completely blown away my expectations. I have been both humbled and endlessly frustrated by their powers and limitations. The language is CLEAR</a>, and these are my learnings from this six-month adventure. Where LLMs excel</h2> Generating ideas and signals</li> Filtering and reducing noise</li> Identifying problems</li> Applying optimizations (with strict guardrails)</li> </ul> Where LLMs are decent, but not as good as I expected</h2> Refactoring—especially re-architecting</li> </ul> Where LLMs do not excel</h2> Getting things correct</li> Obeying design principles</li> Resolving merge conflicts</li> </ul> Finding a hole in a ship is easier than building one that never leaks. Extracting high value from LLMs requires strict discipline. They make it easy to cut corners, even unintentionally. When you lack ideas, LLMs can flood you with them. Usually, you can weed out the bad ones. When flooded with data, LLMs sort it reasonably well, though they will make mistakes. You can solve many problems by having LLMs generate scripts to produce and sift through data. I’m planning to release a tool that allows LLMs to loop until they fully type a Ruby codebase. They did this for a ~40k line transpiler in a day after tooling was built.</li> The tooling took another 1-2 days to develop.</li> You can see it in nil-kill</a>; I’ll release this as a standalone Gem shortly.</li> </ul> </li> I could never find the signal in that data without spending significant time on tooling. Without LLMs, I wouldn’t have spent the time to build that tooling by hand.</li> </ul> </li> LLMs will miss things, but they can find a shocking amount of signal in data that is easy to generate but impossible for me to derive value from manually.</li> </ul> LLMs struggle when absolute correctness is required, as in language development. They are good at getting close quickly, but bad at being actually correct independently. The goal of any tool is to maximize value without introducing negatives. A hammer is great for hitting nails but terrible for knitting sweaters. Where to use LLMs aggressively</h2> LLMs can prototype things unbelievably fast. In many cases, a "mostly correct" prototype is valuable. However, think twice before prototyping even with an LLM: If speed AND correctness are required, there may be significant performance implications.</li> A prototype’s simplicity often comes from its incompleteness or incorrectness.</li> </ul> Second opinions generally: Two heads are better than one. LLM commit reviews may be noisy, but the signal is often worth it. Second opinions specifically for reasons to reject something: Signal on a critical bug is worth the noise. I have had success having LLMs hunt for bugs. If 1 in 10 is a signal, that’s a win; I’ve seen rates closer to 1 in 3. It’s easier for LLMs to find bugs in code than to write correct code initially. Second opinions on design: Several designs I was excited about were shattered by an LLM pointing out a flaw I hadn’t considered. I’m not a language expert, and even after six months, I still feel like a novice. Experts might have better first attempts. Where to use LLMs with caution</h2> Starting points: An LLM-designed system is often mediocre. Workaround: Treat the LLM’s output as an initial idea. Feed it to other LLMs, noting that while you want to move in that direction, the specific design is flawed. This turns a weakness into a strength by using them as second opinions. Results improve if you start with a strong personal opinion and iterate between multiple models.</li> </ul> Testing their own code: LLMs often verify that broken code "works" correctly. Even in reviews, they may miss bugs and praise flawed commits. Workaround 1: Don’t ask an LLM to test its own code. Instead, ask a different model for the ideal strategy to test similar code.</li> Workaround 2: Use multiple LLMs to review code, specifically asking them to find bugs or poor implementations. This iterative process often reveals hidden issues.</li> </ul> Where to use LLMs as a last resort</h2> Following directions: LLMs won't always run tests, even when instructed. If you need a task performed reliably, use tooling. Integrate CI/CD (like GitHub Actions) immediately; LLMs can set this up quickly, but they cannot be trusted to run tests manually every time. Verifying correctness: LLMs are disappointing here. They can generate complex SVGs but fail to count the "r's" in "strawberry." Workaround 1: Use LLMs for search. Ask for the best ways to verify correctness.</li> Workaround 2: Iterate between models to refine verification methods.</li> Workaround 3: Build tooling that provides a high signal-to-noise ratio regarding code quality. Code coverage is a start, but focus on building project-specific tooling for deeper insights.</li> </ul> What LLMs I use and how</h2> Model</th> Implementation</th> Speed</th> Design</th> Review Signal</th></tr></thead> Codex</td> B</td> A+</td> C</td> D</td></tr> Claude Code</td> A-</td> D</td> B</td> B</td></tr> Gemini CLI</td> C</td> D</td> B</td> B+</td></tr> </tbody></table> Gemini CLI is currently free and excellent as a second opinion. I haven’t tried Open Code or DeepSeek yet, but hear they excel at implementation over design. For the cost-conscious: Claude Code (Opus) for design ONLY.</li> Codex for implementation.</li> Gemini CLI for review and second opinions.</li> DeepSeek as a budget-friendly third opinion.</li> </ul> The Magic Words</h2> I’ve found that no amount of CLAUDE.md / AGENTS.md / skills beats manually prompting them at specific points. You will obviously get better results the more attention you pay to what they've done, giving them more specific instruction. However, even broad ones like these can work wonders: For design and review: "I suspect this design is unsafe, unscalable, or won't integrate well, but I can’t articulate why. Please find the holes in this design, especially where a better solution is obvious." </blockquote> For test reviews: "Carefully review these tests. Are they verifying that the code is CORRECT or just that it's successfully broken? Are they testing robust invariants of the system, or just that THIS specific marginal case works?" </blockquote> For implementation reviews: "Review this implementation for bugs. If you find one, add a test that proves it. Does this introduce redundant systems? Should it integrate with existing authorities? Look for tech debt and hacks. Determine if the test coverage is robust and sufficient." </blockquote> CLAUDE.md / AGENTS.md</h2> See my CLAUDE.md</a>. The sections below "Output" are safe to copy. The "Contributing" section should be tailored to your project. Summary</h2> LLMs can suggest a 100x velocity increase, but the reality is closer to 10x. Parallelizing work streams (compiler, runtime, VM) can multiply this, potentially reaching 40x. However, LLMs struggle with merge conflicts and may ignore test failures. They also tend to change unrelated code, like comments. Workaround 1: Instruct them to remove non-essential comment changes or irrelevant edits before committing.</li> Workaround 2: Make sure you have CI setup so that they can't pretend the system already didn't work and that they didn't break it (they will).</li> </ul> Biggest Mistakes</h2> 1. Assuming LLMs could re-architect</h3> I initially allowed LLMs to build a "pile of crap," assuming they could fix it later. They are great at building such piles but struggle to refine them into a functioning language. I could have saved 66% of my headaches by enforcing design guardrails early. 2. Moving too fast</h3> Blitzing through features usually resulted in more work and headaches. Even with LLMs, rushing is costly. Delivering at 100x is tempting, but the "headache factor" of LLM mistakes is significant. I had great success when I followed my learnings closely for this FSM / Thunks / Streams PR</a>. The FSM implementation went especially well, which is where I slowed down the most.</li> </ul> </li> I had much pain and suffering when I ignored my learnings, and prematurely merged in a "hotfix"</a>. It ended up being a "Hot Bug Party" instead.</li> </ul> </li> </ul> 3. Not understanding LLM limitations early</h3> I should have generated a robust test suite from the start. LLMs break architectural invariants easily. They can generate testing frameworks quickly; use them to create "fortress files" that protect your design goals. Why Software Jobs Aren't Imminent Targets</h2> Engineering was never about typing speed. It’s about ensuring code works as intended. While LLMs could generate my project in days - it wouldn't actually work, and the bottleneck is review, ensuring it's implemented correctly and architecturally soundly, that it's sufficiently tested to have any guarantees it actually works, rather than it just happens to pass a slim set of tests, which may in fact be testing nothing. Most engineers spend only a third of their time writing code or less. Even if LLMs automate that, cheaper code will likely increase demand for engineering. The market will lag technological possibility by 5–10 years. Disruption will be asymmetric. Startups, new grads, and managers may see more impact than the general engineering population (or vice versa). Conclusion</h2> In six months of part-time tinkering, I’ve built a competitive runtime and language. This would have been impossible without LLMs acting as a 50x force multiplier. I can "code" while walking the dog, at the gym, during lunch, instead of doom scrolling before bed, etc. While LLMs can be frustratingly inconsistent, they are the most amazing tools I’ve used. I’ve never been more excited about the future of engineering. You can see my full Development Process</a> for how I built a complex language that competes with Rust and Go on concurrent performance and nearly works. pprof integration 2026-05-08T00:00:00+00:00 clear profile</code> emits its runtime profile data in pprof's gzipped protobuf format alongside the existing text dumps, so you can use the standard pprof</code> tool for flamegraphs, call graphs, and source views. Install</h2> Tool</th> When you need it</th> Install</th></tr></thead> pprof</code></td> Always (the viewer)</td> go install github.com/google/pprof@latest</code></td></tr> perf_to_profile</code></td> If you want CPU flamegraphs from perf.data</code></td> go install github.com/google/perf_data_converter/src/cmd/perf_to_profile@latest</code></td></tr> graphviz</code></td> If you use pprof -svg</code> / pprof -dot</code></td> apt install graphviz</code> / brew install graphviz</code></td></tr> </tbody></table> The web UI (pprof -http=:8080</code>) and pprof -top</code> text view do not need graphviz. Use</h2> clear profile foo.cht # -> writes foo.profile/heap.pb.gz, lock.pb.gz, mvcc.pb.gz # (and cpu.pb.gz if perf_to_profile is on PATH) pprof -http=:8080 ./foo foo.profile/heap.pb.gz pprof -top -alloc_space foo.profile/heap.pb.gz pprof -top -inuse_space foo.profile/heap.pb.gz pprof -top -delay foo.profile/lock.pb.gz pprof -base before/heap.pb.gz after/heap.pb.gz # regression diff </code></pre> What's in each file</h2> heap.pb.gz</code></h3> Sample columns: alloc_objects</code> / alloc_space</code> / inuse_objects</code> / inuse_space</code>. Each call site is one sample with its hex address as a label (pprof -tags heap.pb.gz</code>). lock.pb.gz</code></h3> Sample columns: contentions</code> / delay</code> / hold</code> / acquisitions</code>. Read+write contention sums into contentions</code>; delay</code> is the total wait (read+write); hold</code> is total exclusive hold time. mvcc.pb.gz</code></h3> Sample columns: reads</code> / commits</code> / retries</code> / cow_bytes</code>. cow_bytes = struct_size * (commits + retries)</code> — the byte volume moved by copy-on-write commits, the most direct cost signal for @shared:versioned</code> cells. channels.pb.gz</code></h3> Sample columns: pushes</code> / pops</code> / push_blocked</code> / pop_blocked</code> / max_depth</code>. One sample per registered channel; the synthetic function name is channel#<id></code> and the channel's capacity travels as a label (pprof -tags channels.pb.gz</code>). cpu.pb.gz</code></h3> Standard CPU profile from perf.data</code>, converted by perf_to_profile</code>. Sample columns are whatever Go's pprof shows (samples</code> / cpu</code> ns). CLEAR source mapping</h2> The transpiler emits // CLR:N</code> markers in transpiled.zig</code>. Our converter walks those back to the user's .cht</code> line and stamps it onto each pprof Location, so pprof -list <fn></code> shows CLEAR source lines (not Zig). Sampling</h2> Stack traces for alloc-profile are captured on every alloc by default. For workloads where the per-alloc unwind cost matters, --sample=N</code> records every Nth event and scales the captured values by N so doctor / pprof see estimated totals: clear profile foo.cht --sample=100 </code></pre> Header records the chosen sample_n</code> so consumers can rescale or flag the approximation. Stack traces for lock and mvcc profiles</h2> --sync-callstacks</code> (off by default) turns on per-record stack capture in lock-profile and mvcc-profile, so pprof's tree/flame views and per-caller attribution work for these profiles too: clear profile foo.cht --sync-callstacks # auto-bumps --sample to 100 clear profile foo.cht --sync-callstacks --sample=1 # full capture, full cost </code></pre> Off by default because the FP walk costs ~100-500ns per record. Uncontended mutex acquire is ~10-20ns and MVCC commit fast paths are ~20-50ns, so the trace can dominate the operation it's measuring. When the flag is set without an explicit --sample</code>, we auto-default to --sample=100</code> to keep the cost manageable. With --sync-callstacks</code> on, each (lock, caller-trace) pair becomes its own row in lock-profile, and same for (cell, caller-trace) in mvcc-profile. Doctor aggregates rows back to one-per-lock for its existing diagnoses; pprof tree views show the per-caller breakdown. Doctor flags built on this data</h2> clear doctor</code> consumes the same profile dirs and grew three new flags that draw on the multi-frame trace data: clear doctor foo.profile/ --cumulative # rank functions by cum bytes clear doctor foo.profile/ --focus=intToString # filter to traces that touch this function clear doctor foo.profile/ --ignore=intToString # drop traces that touch this function clear doctor foo.profile/ --peek=processRequest # callers + callees of one function clear doctor foo.profile/ --by=allocs # sort heap by allocation count, not bytes clear doctor foo.profile/ --by=inuse_bytes # sort by allocs - frees (live bytes) clear doctor old.profile/ --diff new.profile/ # perf-regression diff between two runs </code></pre> --cumulative</code> aggregates bytes/allocs across every frame in each trace, so a function high in the call stack accrues its callees' costs. --focus=REGEX</code> keeps only sites whose trace touches a function matching the pattern. --diff</code> reports per-function deltas in allocation, lock contention, and MVCC retries, with directional arrows and "newly contended" / "retries eliminated" annotations. Notes</h2> We do not emit a Mapping message for the binary, so pprof</code> prints "Main binary filename not available" and skips its own symbolization. Function names still appear because we resolve via addr2line</code> at conversion time.</li> alloc-profile captures stacks unconditionally (multi-frame v2, comma-separated leaf-first in alloc.txt</code>).</li> lock-profile (v3) and mvcc-profile (v2) capture stacks only when --sync-callstacks</code> is on. The 12th column of locks.txt</code> and the 8th column of mvcc.txt</code> carry -</code> (off) or comma-separated leaf- first addrs (on). Tab-separated to allow commas in the trace field.</li> </ul> STRICT EXTREME: The Path to High-Integrity CLEAR (@strict:extreme) 2026-05-01T00:00:00+00:00 STRICT</code> and STRICT EXTREME</code> mode are not yet available. DEFAULT</code> mode is the focus for the v0.1 release. </div> Overview</h2> The "CLEAR Story" is a transition from Productivity (Ruby-like ease) to Performance (C-like speed) to Integrity (Ada/SPARK-level safety). While standard CLEAR provides memory and concurrency safety, STRICT EXTREME is a compilation mode designed for mission-critical systems (HFT, Aerospace, Medical) where runtime failures (Panics) are unacceptable. Unlike other high-integrity languages, CLEAR does not require a different syntax for this level of safety. Instead, it uses Progressive Formal Verification to prove properties about the code that the Annotator</code> and OwnershipGraph</code> are already tracking silently. The Integrity Spectrum</h2> Mode</th> Safety Guarantee</th> Developer Experience</th></tr></thead> Standard</td> Memory & Concurrency Safety</td> "It just works" (Ruby-like)</td></tr> STRICT</td> Performance & Effect Safety</td> No hidden HEAP/BLOCKING</td></tr> STRICT EXTREME</td> Liveness & Panic Safety</td> Formally Proven (Ada-like)</td></tr> </tbody></table> Core Pillars of STRICT EXTREME</h2> 1. The "No-Panic" Guarantee (Formal Proof)</h3> In standard mode, CLEAR prevents memory corruption but may !!</code> (Panic) on an out-of-bounds index or an integer overflow. In STRICT EXTREME, the compiler uses a SMT solver (like Z3) to prove that no runtime panic is possible. Constraint Propagation: The compiler tracks value ranges. If it cannot prove idx < list.length()</code>, it rejects the build.</li> Proof Assistance: The clear fix</code> tool suggests @pre</code> (pre-conditions) or @post</code> (post-conditions) to satisfy the prover.</li> </ul> 2. Guaranteed Bounded Execution (Stack & Time)</h3> Ada-level safety requires knowing that a program will not run out of memory or hang. Stack Depth Proof: The compiler walks the call graph and proves the maximum possible stack usage. Recursive functions without a proven base case are rejected.</li> TIGHT Loop Validation: Loops must be proven to terminate. "Infinite" loops are only allowed in dedicated "Supervisor" fibers.</li> </ul> 3. Total Effect Isolation</h3> While STRICT</code> mode requires annotations for HEAP and BLOCKING, STRICT EXTREME requires a Total Effect Signature. Deterministic Paths: Functions can be marked @pure</code> (no side effects) or @deterministic</code> (identical inputs always yield identical timing/output).</li> Alloc-Free Zones: Proves that a "Hot Path" performs zero allocations (not even arena allocations) to guarantee zero jitter.</li> </ul> 4. Liveness & Deadlock Immunity</h3> Leveraging the OwnershipGraph</code>, the compiler proves that no circular dependency of @locked</code> or @shared</code> objects exists. Lock-Ordering: The compiler enforces a global "Acquisition Hierarchy." If you try to acquire Lock B</code> while holding Lock A</code>, but elsewhere Lock A</code> is acquired after Lock B</code>, the build fails.</li> </ul> Implementation Path: "Progressive Disclosure"</h2> The beauty of the CLEAR architecture is that STRICT EXTREME does not hinder the early-stage developer: Draft Phase: The developer writes code in Standard mode. The compiler silently tracks ownership and effects.</li> Optimization Phase: The developer turns on STRICT</code> mode to remove bottlenecks (allocations/blocking).</li> Certification Phase: The developer turns on STRICT EXTREME</code>. The compiler takes the existing metadata and "Hardens" it using formal verification.</li> </ol> Architecture & Logic</h2> Because CLEAR's memory model is a Directed Acyclic Graph (DAG) (enforced by Affine movement and Arena scopes), it is mathematically "Decidable." Proving properties about CLEAR code is significantly simpler than proving properties about C++ or Rust because aliasing is strictly controlled and visible to the Annotator</code>. Goal for v0.3 Roadmap</h3> Integration of a formal solver into the Annotator</code> to provide "Panic-Free" proofs for core data-processing pipelines. Deadlock in CLEAR 2026-04-22T00:00:00+00:00 Matrix row: Deadlock — C: F, Rust: F, Go: F, Pony: A+, BEAM: A+, CLEAR: A-* The asterisk: CLEAR uses locks for read-heavy workloads where MVCC unpredictability is a non-starter. Locks can deadlock in standard mode. CLEAR provides runtime detection + recovery, and STRICT EXTREME</code> (roadmap) will add compile-time lock-ordering enforcement. Why the AB/BA Pattern Is Not the Main CLEAR Footgun</h2> The classic two-thread deadlock (thread A holds lock 1, waits for lock 2; thread B holds lock 2, waits for lock 1) requires two actors holding their respective lock simultaneously. In CLEAR's cooperative fiber scheduler, fibers only yield at explicit yield points (NEXT</code>, I/O). There is no yield between acquiring the first lock and acquiring the second inside a WITH EXCLUSIVE</code> block, so the AB/BA cycle cannot form on a single scheduler: the first fiber grabs both locks before the second fiber ever runs. The AB/BA pattern does become dangerous with @parallel</code> (multi-scheduler, multiple OS threads), because the fibers then run on different OS threads simultaneously. But in the common single-scheduler case, it is not the footgun to worry about. The Real Footgun: Re-Entrant Locking</h2> CLEAR's @locked</code> uses a non-recursive pthread_mutex_t</code>. If a fiber tries to acquire a lock it already holds, pthread_mutex_lock</code> blocks the OS thread — and since the fiber holding the lock IS that OS thread, it can never release it. Deadlock. STRUCT State { value: Int64 } FN main() RETURNS Void -> s = State{ value: 0 } @locked; t = BG { WITH EXCLUSIVE s AS outer { outer.value = 1; # DEADLOCK: trying to re-acquire the same mutex this fiber holds. # pthread_mutex_lock blocks the OS thread. The lock is never # released because the releaser IS the blocked fiber. WITH EXCLUSIVE s AS inner { inner.value = 2; } } }; NEXT t; # hangs here END </code></pre> This compiles without error and hangs at runtime. Fix: restructure so the same lock is never acquired twice on the same call path. Pass the already-locked inner value to helper functions instead of locking again: FN update_inner(inner: State) RETURNS Void -> # Takes the already-unlocked value; no lock needed here. inner.value = 2; END FN main() RETURNS Void -> s = State{ value: 0 } @locked; t = BG { WITH EXCLUSIVE s AS inner { inner.value = 1; update_inner(inner); # pass the unlocked value, not the lock } }; NEXT t; END </code></pre> The NEXT-Inside-WITH Footgun</h2> A subtler variant: fiber A holds a lock and calls NEXT</code> on a promise produced by fiber B — but fiber B also needs that lock to complete. STRUCT State { value: Int64 } FN main() RETURNS Void -> s = State{ value: 0 } @locked; producer = BG { # producer needs 's' to do its work WITH EXCLUSIVE s AS inner { inner.value = 42; } }; consumer = BG { WITH EXCLUSIVE s AS outer { # consumer holds 's', then waits for producer NEXT producer; # DEADLOCK: producer can't acquire 's' — consumer holds it print(outer.value.toString()); } }; NEXT consumer; # hangs here END </code></pre> The fiber holding the lock yields via NEXT</code>, but the lock is NOT released when a fiber yields — WITH EXCLUSIVE</code> holds the lock for the duration of the block, not just until the next yield point. The producer is stuck waiting for the lock that the consumer holds while waiting for the producer. Fix: resolve the promise before entering the lock, or restructure so the two operations don't depend on each other: FN main() RETURNS Void -> s = State{ value: 0 } @locked; producer = BG { WITH EXCLUSIVE s AS inner { inner.value = 42; } }; NEXT producer; # wait for producer BEFORE acquiring the lock WITH s AS inner { print(inner.value.toString()); # 42 } END </code></pre> STRICT EXTREME: Compile-Time Lock Ordering (Roadmap)</h2> In STRICT EXTREME</code> mode (planned, not yet implemented), the compiler will enforce a global acquisition hierarchy via the OwnershipGraph</code>. Re-entrant acquisition and AB/BA cycles will be detected at build time: error: re-entrant lock acquisition fiber acquires 's' at line 8 (WITH EXCLUSIVE) then acquires 's' again at line 11 (WITH EXCLUSIVE) fix: pass the already-locked pointer to helper functions instead </code></pre> See docs/strict-extreme.md</a> for the full STRICT EXTREME roadmap. Fix: Single Lock Scope (Always Safe)</h2> Restructure so each operation needs only one lock at a time. No re-entrancy, no NEXT-inside-WITH: FN main() RETURNS Void -> a = Account{ balance: 100 } @locked; b = Account{ balance: 200 } @locked; t1 = BG { amount = 0; WITH EXCLUSIVE a AS ra { amount = ra.balance / 2; ra.balance = ra.balance - amount; } # a is unlocked here. No re-entrancy possible. WITH EXCLUSIVE b AS rb { rb.balance = rb.balance + amount; } }; NEXT t1; END </code></pre> The tradeoff: releasing between the two operations creates a TOCTOU window (see examples/footguns/05_toctou</a>). Why C, Go, and Rust Get F</h2> All three languages expose the same non-recursive mutex with no re-entrancy detection and no built-in deadlock timeout: C (pthread_mutex_lock</code>): hangs forever on re-entrant lock or AB/BA cycle. No detection without Helgrind or TSan.</li> Go (sync.Mutex</code>): hangs forever for partial deadlocks. Detects global deadlock (all goroutines blocked) with a panic. Race detector does not catch deadlocks.</li> Rust (std::sync::Mutex</code>): hangs forever. Attempting to lock from the same thread panics on some platforms (documented as "may panic or deadlock"). parking_lot</code> adds optional cycle detection.</li> </ul> Summary</h2> Pattern</th> Current CLEAR behavior</th> Planned fix</th></tr></thead> Re-entrant lock (same fiber re-acquires)</td> Hangs — non-recursive mutex</td> STRICT EXTREME: compile error</td></tr> NEXT inside WITH (holding lock while awaiting)</td> Hangs — lock not released on yield</td> STRICT EXTREME: compile error</td></tr> AB/BA across @parallel fibers</td> Hangs — same as C/Go/Rust</td> STRICT EXTREME: compile error</td></tr> AB/BA on single scheduler</td> Does not occur — no yield between lock acquisitions</td> N/A</td></tr> </tbody></table> Why CLEAR 2026-04-20T00:00:00+00:00 CLEAR is designed to be: Correct</li> Safe</li> Understandable</li> Scalable</li> BLAZING fast</li> </ol> But Why?</h2> Rust to have memory safety with no overhead - primarily to use for Servo to replace C for Browser Engines and for Kernel development. There are a number of tradeoffs Rust was not willing to make for it to be either 1) easy / understandable, or 2) a better Go. Yet people are desperate to use Rust as a better Go because it has so much potential to be that. Pony eliminates nearly all concurrency hazards by design. Rust merely ensures memory safety - which is perhaps the most common and critical concurrency hazard. But despite its safety, Pony is not widely used because the learning curve is too steep. Rust & Pony were both not willing to make tradeoffs to prioritize ease-of-use or understandability. Go made those trade-offs. But Go is a thin veneer over C with the most sophisticated runtime in the world bolted on. To achieve best in class speeds Go - like C - relies on loading a footgun and exposing yourself to a number of hazards and developer discipline and/or choosing the right libraries to get it right. It provides valuable best-in-class tooling to help mitigate some of these problems. CLEAR exists because it thinks Rust is not truly safe enough inherently, Pony is not easy enough, and Go is currently the most practical trade-off but is too dangerous and cannot fix its inherent problems. Further, although Pony literally forces you into the actor pattern - at least at this stage, it is not inherently distributive. The actor model does not require serialization, distributed fault tolerance, supervision, etc. Pony uses the ideas to achieve safety on a single machine, but just because you wrote the code that allows for distribution easily, does not mean it actually can be distributed effectively by default. Pony did not want to make any sacrifices on safety. This led to a language that is impractical and/or not competitive for many of workloads (with a high cognitive burden). CLEAR aims to take the lessons of Pony, Rust, Go, and BEAM and combine them. The goal is to make trade-offs to achieve understandability, and to sacrifice some safety if it means giving you tools necessary to realistically accomplish common workloads. Rubric</h2> Feature</th> C</th> Rust/Tokio</th> Go</th> Pony</th> BEAM</th> CLEAR</th></tr></thead> Cognitive Load</td> F *</td> D+</td> B</td> D-</td> C+</td> A-</td></tr> Memory Safety</td> F</td> A+</td> B *</td> A+</td> A+</td> A+</td></tr> Raw Speed</td> A+</td> A</td> B+</td> A-</td> C</td> A-</td></tr> Throughput</td> A</td> A</td> A+</td> A</td> A</td> A</td></tr> Memory Usage</td> A+</td> A+</td> B-</td> B+</td> C-</td> A-</td></tr> Predictability</td> A+</td> A+</td> B-</td> B+</td> A-</td> A-</td></tr> Backpressure</td> F</td> A</td> A+</td> C</td> B</td> A</td></tr> Starvation</td> F</td> B</td> A</td> B</td> A+</td> A-</td></tr> Fault Tolerance</td> F</td> F</td> F</td> F</td> A+</td> ?</td></tr> Deadlock</td> F</td> C</td> C+</td> A+</td> A+</td> A- *</td></tr> Memory Ordering</td> F</td> C</td> B</td> A+</td> A+</td> A-</td></tr> Logical TOCTOU</td> F</td> F</td> F</td> A+</td> A-</td> B+</td></tr> Causal Ordering</td> F</td> F</td> F</td> B+</td> C</td> B *</td></tr> Stream Ordering</td> F</td> F</td> F</td> F</td> F</td> ?</td></tr> </tbody></table> Non-CLEAR Ratings: Context & Justification</h3> All ratings are inherently subjective. The C Cognitive Load Controversy: C scoring an F may be controversial, but "Cognitive Load" here captures the true difficulty of writing correct concurrent systems. C’s F is the price it pays for A+ ratings elsewhere; you could write C with a B in cognitive load (e.g., using a single global lock), but throughput would plummet to an F. Similarly, you could bolt on a Garbage Collector to improve safety, but speed and predictability would drop to a C.</li> The “Best Practices Fallacy”: Given perfect discipline and SOTA libraries, one can achieve better ratings in C, Rust, or Go. However, "doing it right" is not a scalable systems architecture. These ratings reflect the language’s inherent, default guarantees.</li> The Rust/Tokio Note: Rust is graded slightly unfairly as its evaluation includes Tokio. While not part of the core language, it is the de facto production standard and its semantics must be evaluated.</li> The Go Scheduler: Go experts may feel there isn't enough separation in Throughput/Backpressure. CLEAR praises Go’s runtime and scheduler as one of the most sophisticated pieces of software ever written; these ratings may not do its operational excellence enough justice.</li> Go Memory Safety: Go is memory-safe in the conventional sense, but it does not provide strong concurrency-safety or invariant-safety by default. Its practical model often relies on discipline around shared mutable state.</li> The Pony Asterisk: Pony’s flawless safety comes at a hit to expressiveness. By forbidding shared mutable state, certain highly efficient data structures (like lock-free graphs) are nearly impossible to write idiomatically.</li> </ul> CLEAR Ratings: The Architecture</h2> Cognitive Load: CLEAR is designed so that Profile Guided Optimization (PGO) and automated tooling solve the heavy lifting. You write intuitive, sequential code, and the profiler suggests (and injects) the necessary optimization directives based on actual workloads.</li> Memory Safety: Like Rust, CLEAR utilizes Affine Ownership to guarantee memory safety.</li> Logical TOCTOU: Values behind Arcs/Locks cannot escape lexical scope. The compiler can generate Loom tests in a deterministic VM to catch dependencies and break them. Use-after-free and race conditions are included in Memory Safety. Time-related bugs are captured in Causal & Stream Ordering categories.</li> Deadlock: Technical deadlock is not part of CLEAR’s intended concurrency model, and by v0.3 the runtime aims to detect and prevent unbounded lock-wait cycles from silently persisting. Locks are intentionally more cumbersome to use because they are rarely the best-performing or safest solution. When developers opt into lock-based coordination anyway, they are also opting into explicit responsibility for handling lock-related failures correctly. A language like Pony or BEAM which does not have locks deserves a better ranking.</li> Starvation & Backpressure: Like BEAM, CLEAR will prevent CPU starvation via its cooperative scheduler with automatically injected yielding. It separately tracks per-task memory consumption to kill runaway tasks and enforce backpressure.</li> Memory Consumption: CLEAR uses MVCC as a default synchronization technique. This adds memory overhead to eliminate common classes of bugs (deadlocks, contention).</li> Causal Ordering: CLEAR separates time as a tense in the type system, and offers A+ causal ordering with @split</code> streams and solves the concurrent write problem with MVCC and transaction failure handling. But it allows you to "load the foot-gun" (unsafe escape hatches) for specific workloads where predictability is paramount.</li> Stream Ordering: CLEAR has not yet determined if it will offer any guarantees to protect against stream ordering related bugs. Frameworks like Flink and Kafka exist to solve this problem. It may not make sense to try to solve it in the runtime/language.</li> Fault Tolerance: It is still too early in the design stage to determine a realistic score for CLEAR's fault tolerance story—particularly regarding issues that arise from shared mutable memory and the fact that idempotence is not a first-class function color.</li> </ul> Fault Tolerance</h3> CLEAR hopes to have a reasonable ceiling of a B (similar to Causal Ordering). It will likely give you tools to shoot yourself in the foot to prioritize understandability and broad workload support—for example, you could mark tasks as killable which aren't, or tasks as retrying which aren't idempotent. But CLEAR will have the systems in place to easily support an A+ architecture when paired with modern infrastructure. The fact that everything else scores an F besides BEAM is evidence that solving these problems purely at the language level may no longer be the best choice. The approach CLEAR is currently most interested in is targeting the five most common catastrophic faults: CPU Starvation (Time)</li> Local Heap Exhaustion (Space)</li> Lexical Orphanhood (The spawning task died; the work is no longer relevant)</li> Data-Flow Severance (Producing a stream that no one is listening to)</li> Deadlocks (Unbounded lock-wait cycles)</li> </ol> CLEAR aims to either safely assassinate the offending fibers automatically (Deadlocks/Orphans) or detect the threshold and force the developer to handle the outcome (OOM limits). Whatever CLEAR's final solution ends up being, BEAM folks are likely to look at it and say, "That's basically an F." But CLEAR aims to provide pragmatic value. Like BEAM folks, CLEAR is of the view that killing entire processes with important state just because you have no built-in fault tolerance is insane. Unlike BEAM, CLEAR is unlikely to eliminate the possibility of needing to do that occasionally, but it aims to reduce that need to practically zero. How Does CLEAR think about Cognitive Burden</h2> In CLEAR's perspective, a program becomes difficult to understand because of global complexity. A function is impossible to understand if the entire codebase must be understood to understand the function and line of code you are looking at. This is why CLEAR aims to completely eradicate all forms of global complexity. If you can understand any function fully by just looking at the function, it is simple. Rust is not considered difficult because of Affine Ownership. It is considered difficult because of "Fighting the Borrow Checker" which mainly boils down to Rust's obsession that it must achieve zero-cost abstractions in ALL cases - which includes shared-mutable borrows. This is almost entirely the source of confusing lifetime annotations in Rust. If you eliminate this demand, you eliminate most of the "Fighting the Borrow Checker" in Rust complaints. CLEAR has done this. This is because this is Rust's main source of global complexity. Pony is not considered difficult because it merely has capabilities or because it forces you into the actor pattern. It is difficult because capabilities do are not intuitive. You aren't describing what you want to do, you are describing what can be done. This is less intuitive. Go claims to be easy - which implies simplicity. But Go allows and often forces you to load footguns to achieve decent performance, lending itself to global complexity and hard-to-understand programs. Go may claim to be easy, but if you need to write 20 lines of imperative arcana in the precise ceremonial order or rely on a set of libraries to do common things efficiently - it is not truly easy. In essence, CLEAR thinks SQL proved that it is relatively easy to describe what you want to do (your intent), and it is certainly easy to recognize it after the Profiler points you in the right direction for efficiency and safety (one-line optimizations clearly separate from code). It is much harder to think like the computer and tell it HOW to do that, or list what CAN be done and hope that matches what you want to do. How Does CLEAR think about Performance and Speed</h2> Raw Speed being a metric should raise some eyebrows. Performance is not a monolith. It is a balance of competing trade-offs. From CLEAR's perspective - Correct, Safe, and Understandable are by far the most important pillars of the design that are being balanced. When CLEAR thinks about speed, it mainly thinks about single machine concurrent throughput in compute heavy workloads. CLEAR is looking toward the future of compute rather than the present. In the future, core count is expected to continue growing faster than memory. Cache per core has not meaningfully grown much in decades, and CLEAR assumes this will not grow as meaningfully as core count in the future. CLEAR thinks the main Raw Speed metrics are: 1) compiling via Zig to get LLVM best in class optimizations to single-core performance, 2) maximizing for cache locality as memory read stalls can dominate compute performance benchmarks. In general, CLEAR aims to overcome added memory, co-operative yielding, and any safety checks by gaining a level of cache locality that is nearly impossible to reach in any other language. How? The core pillar of CLEAR's design is to maximize local reasoning and minimize global complexity. This is what makes CLEAR simple, even if it may not be as easy as Go. It also should allow CLEAR to make compiler optimizations WRT escape analysis that no other language (that we are aware of) can make to maximize cache locality. PROPAGANDA 2026-04-05T00:00:00+00:00 Cheating is all you need. Software should be performant, robust, AND resilient.</li> It should also be effortless to write and understand.</li> It should be able to run anywhere, optimized for distributed parallelism and concurrency.</li> </ul> They told you "Pick one." They lied. You can have it all, if you're willing to CHEAT. Commands like SQL. Pipelines like Bash. Easy like Ruby. Speed like C. Being a genius like antirez isn't scalable. It's not something everyone can be. Everyone else can CHEAT. WHAT IS CHEAT?</h2> CLEAR is a memory safe language like Rust, with better ergonomics than Swift. It runs on a Go-like Runtime (the CHEAT runtime) that makes concurrent code as easy, safe, and fast as possible. CORE CLEAR PHILOSOPHY</h2> Code should work.</li> Code that is easy to understand, write, AND test is more likely to work.</li> Dependencies should be Strictly-Correct, as that IS their business logic.</li> Applications should FIRST be Business-Logic correct, only Strictly-Correct if need be.</li> The language and compiler should never get in the way of Business-Logic correctness.</li> Making things Strictly-Correct, once Business-Logic correct, should be easy.</li> Making things BLAZING fast, once Strictly-Correct, should be easy.</li> </ul> manifesto EXTRA 2026-04-03T00:00:00+00:00 Handling Array Access</h3> x = getMyVector(); y = x[10]; -- compiler error! y = x[10] OR ELSE 0; -- OKAY! x: Int[10] = getMyVector(); y = x[9]; -- OK z = x[10]; -- COMPILER ERROR: Out of Bounds zz = x[10] OR ELSE 0; -- OK - with a strong *WARNING* (compiler error in STRICT mode) FN myVectorGetterDoer(v) -> -- preferred to use a GUARD clause z = GUARD v.size >= 56 OR ELSE CAST(v AS Int(56)); -- TODO: Is this too hard for the compiler to know? x = v[10]; -- OKAY, proven >10 y = v[23]; -- OKAY, proven >23 z = v[55]; -- OKAY, proven >55 RETURN x + y - z + 5; END FN myVectorGetterDoer(v) -> x = v[10]; y = v[23]; z = v[55]; RETURN x + y - z + 5; CATCH RETURN 0; -- OK: caught here, returning a value END FN myVectorGetterAdder(v) -> x = v[10] OR RETURN 5; -- OK: IFF you return a value, not the error. x = v[10] OR RETURN v[0]; -- COMPILER ERROR: OBVIOUSLY! RETURN x + 5; END -- here v is a DYNAMIC Vector. FN myVectorGetterWhichTakesADynamicArrayAndResizes(v) -> -- ... do a bunch of stuff x: Int[10] = v; -- COMPILER ERROR! xx: Int[10] = TRUNCATE(v); -- OK: acknolwedge you *could* be losing data. y = xx[9]; -- OK: it will be 0 by default, if v was < 10 items. -- ... do a bunch of stuff RETURN x + 5; END -- here v is a FIXED Vector of size 1. FN myVectorGetterWhichTakesADynamicArrayAndResizes(v) -> -- ... do a bunch of stuff x: Int[10] = v; -- OK: perhaps surprisingly, you're allowed to upsize, no harm no foul y = x[9]; -- OK: it will be 0 by default, if v was < 10 items. -- ... do a bunch of stuff RETURN x + 5; END </code></pre> Handling Division by Zero</h3> FN myDivider(x, y) -> y = GAURD y != 0 OR ELSE 1; -- OKAY, if y is MUTABLE RETURN x / y; end -- this is fine in a function FN typicalFunc(myObj) -> health = GUARD myObj.health != 0 OR ELSE 1; -- Do a bunch of stuff RETURN myObj.strength / myObj.health; end -- Say you had a complex formula with lots of division -- You might not want to have FN typicalFunc(myObj) -> health = GUARD myObj.health != 0 OR ELSE 1; -- 10 more guards, needing to use local variables below -- Do a bunch of stuff -- RETURN <your equation with a bunch of division> end -- Instead you can do: FN typicalFunc(myObj) -> -- RETURN <your equation with a bunch of division> CATCH RETURN -1; end -- HOWEVER - there is no such option for code outside of a function. -- You can do: FN main() -- Good design, no code execution outside of main -- In here, you do do all kind of division CATCH -- Handle however you want, except with code that raises an uncaught error END -- If you just have a script like: x = readSomeFileGetSomeNumber(); y = 10; callSomeFunc(y / x); -- You'll get a DivisionByZero error, so you must do: x = GUARD readSomeFileGetSomeNumber() != 0 OR ELSE 1; y = 10 callSomeFunc(y / x); </code></pre> ASYNC/AWAIT/COLLECT</h3> listOfUsers s> FILTER %.isActive s> COLLECT -- Converts Stream<User> to List<User> s> map(%(x) -> x.size > 5); -- Operates on the List<User> object -- Did not prefix with AWAIT -> *assumes* intentend ASYNC -> immediately starts next step file_paths s> concurrentRead s> COLLECT -- Wait for ALL files to be fetched/materialized. s> aggregate; -- Start aggregation only on the complete set. -- Did not prefix with AWAIT -> *assumes* intentend ASYNC -> immediately starts next step file_paths s> concurrentRead s> COLLECT -- Wait for ALL files to be fetched/materialized. s> aggregate s> COLLECT; -- waits until THIS finishes to resume (which may happen before the other two finish) -- The above is *nearly* equivalent to: AWAIT file_paths s> concurrentRead s> COLLECT -- Wait for ALL files to be fetched/materialized. s> aggregate; -- This is preferred, but not enforced. -- Ending in COLLECT will convert from Stream<T> to List<T> -> even if it's not assigned to anything. </code></pre> MUTABILITY vs IMMUTABILITY</h3> user = %{ name: "Alice", active: false } user.active = true; -- Error! user = %{ name: "Alice", active: true} -- Error!! user = %{ name: "Alice", active: false } upDatedUser = user MUTATE { active: true } -- No error! x = 5; x += 10; -- ERROR! MUTABLE x = 5; x += 10; </code></pre> Example compiler errors: Error: Cannot set field 'active' on 'user' to true. : This is a READ-ONLY VIEW of memory owned by 'DatabaseQueryResult'. : To modify, use: : : newUser = user MUTATE { active: true }; : : Or, if you want to modify it later, you might want it to be MUTABLE, use: : : MUTABLE newUser = DEEP_COPY(user_data); : newUser.active = true; : : You can set fields on MUTABLE data. Error: Cannot assign to 'x': it is immutable. : x is IMMUTABLE. : : You can create a new binding: : : newX = x + 5; : : Or, if you want to modify it later, declare it mutable: : : MUTABLE newX = x; : newX = 5; </code></pre> THE CONFUSING TYPES</h2> Errors</h3> You don't need to check for errors by default or specify them in your return types.</li> The Compiler and the SMOOTH operator takes care of this for you.</li> </ul> FN myCarelessFn %(myList) -> myList s> fetchData -- this could return an error! s> parseData -- this could return an error! s> renderPage; -- this could return an error! -- It's fine to not catch any of them here! -- You're allowed to pass on the problem to your end user! END FN mySomewhatCarefullFn %(myList) -> myList s> fetchData -- this could return an error! s> parseData -- this could return an error! s> renderPage; -- this could return an error! CATCH -- anything -- No matter what I want my users to get the default page. RETURN makeDefaultPage(); END FN myMoreCarefullFn %(myList) -> myList s> fetchData -- doesn't ever throw an error! s> parseData -- doesn't ever throw an error! s> renderPage; -- doesn't ever throw an error! CATCH -- anything -- This is still allowed, since no matter what you can OOM RETURN makeDefaultPage(); END FN myQuiteCarefullFn %(myList) -> myList s> fetchData OR SKIP -- SKIP any error s> OTHERWISE(fetchFromBackup) -- OTHERWISE only happens if an error happend, and this may raise error, it's fine! s> parseData -- this could return an error! s> renderPage; -- this could return an error! CATCH -- anything EXCEPT fetchData error -> It was already handled inline RETURN makeDefaultPage(); END FN myReallyCarefullFn %(myList) -> myList s> fetchData OR SKIP -- SKIP any error s> OTHERWISE(fetchFromBackup) OR RETURN -- Don't proceed any further, bubble this right up to the user s> parseData -- this could return an error! s> renderPage; -- this could return an error! CATCH -- anything -- EXCEPT fetchData error -> It was already handled inline -- EXCEPT fetchFromBackup -> It was already handled inline RETURN makeDefaultPage(); END FN myVeryCarefullFn %(myList) -> myList s> fetchData OR SKIP -- SKIP any error s> OTHERWISE(fetchFromBackup) OR RETURN -- Don't proceed any further, bubble this right up to the user s> parseData OR EXIT -- Don't proceed any further, but try to pass this to a catch block below s> renderPage; -- this could return an error! CATCH -- anything -- EXCEPT fetchData error -> It was already handled inline -- EXCEPT fetchFromBackup -> It was already handled inline -- DOES CATCH parseData (and renderPage) errors RETURN makeDefaultPage(); END FN myExtremelyCarefullFn %(myList) -> myList s> fetchData OR SKIP -- SKIP any error s> OTHERWISE(fetchFromBackup) OR RETURN -- Don't proceed any further, bubble this right up to the user s> parseData OR GOTO_RECOVER -- Don't proceed any further, JUMP to the first RECOVER down the chain s> renderPage -- this could return an error! s> RECOVER(makeDefaultPage()); CATCH -- anything -- EXCEPT fetchData error -> It was already handled inline -- EXCEPT fetchFromBackup -> It was already handled inline -- DOES CATCH parseData (and renderPage) errors RETURN makeDefaultPage(); END FN myMostCarefulFn %(myList) -> myList s> fetchData OR SKIP -- SKIP any error s> OTHERWISE(fetchFromBackup) OR RETURN -- Don't proceed any further, bubble this right up to the user s> parseData OR GOTO_RECOVER -- Don't proceed any further, JUMP to the first RECOVER down the chain s> renderPage OR EXIT "RenderPage failed" s> RECOVER(makeDefaultPage()) OR EXIT "RenderBackupPageFailed"; CATCH -- anything -- Here, there might be two RenderPage errors -- Down here, we might want to do two different things based on the same Error (with the context string from above) -- In both cases, we have acccess to the Error object `%e` -- With %e.message and %e.snapshot (whatever went into the pipe) set. -- -- WARNING: To operate on %e.snapshot, you must cast it from Any to Whatever it is. -- This is dangerous, as it could itself raise an error. -- If this something like this is not acceptable, CHEAT is not for you. -- -- You CAN log it without issue (with a default limit to how big the string is) -- Though, that could contain PII, so unless you control the parent, and can ensure it's safe for logging -- Do so at your own risk RETURN makeDefaultPage(); END -- Testing Error handling is easy! TEST "MyFn handles errors gracefully" -> result = ERROR("Boom") s> myFn; ASSERT result == defaultPage; -- Success! END </code></pre> Strings Vs Bytes</h3> String by default. s = "😊"; -- String (Immutable) MUTABLE s = "😊"; -- String (Mutable, enforces UTF-8 on write) s: String[10] = "😊"; -- String (Immutable, fixed size UTF-8 buffer, very uncommon use case -> mainly for cache locality, not beginner need) MUTABLE s: String[10] = "😊"; -- String (Mutable, fixed size UTF-8 buffer, more common use case, but not beginner) b: UInt8[*] = %[0, 10, 255]; -- Buffer (Immutable -> common / default use case for non-beginners) MUTABLE b: UInt8[] = %[0, 10]; -- Buffer (Mutable, Dynamic -> common use case) b: UInt8[10] = ...; -- Buffer (Immutable, Fixed Size -> you might want for cache locality) MUTABLE b: UInt8[10] = ...; -- Buffer (Mutable, Fixed Size -> common use case) </code></pre> Lists vs Streams</h3> users = db.all s> filter; -- LIST, default users = db.all s> filter s> COLLECT; -- LIST, default, superfluous users = AWAIT db.all s> filter; -- LIST, default, superfluous userCursor: Stream = db.all s> filter; -- STREAM, not-default due to EXPLICIT type userCursor = db.all s> filter s> ITER; -- STREAM, not-default due to EXPLICIT ITER userCursor = ASYNC db.all s> filter; -- STREAM, not-default due to EXPLICIT ASYNC </code></pre> Streams are handles, not data</h3> A stream is an immutable pointer to a mutable partition data (the stream / flow where x is COLLECT</code>ing).</li> However, you can have an mutable pointer.</li> </ul> Therefore: MUTABLE source: Stream = primary_db.users; -- OKAY TRY source.peek(); -- Check connection CATCH -- I need to reassign the variable to the backup stream! source = backup_db.users; END source s> map... -- ... MUTABLE s: Stream = db.all s> ... ; -- OKAY MUTABLE s: Stream(10) = db.all s> ... ; -- Compiler error, `... returns a single Stream, not a Vector of Streams`. </code></pre> </h2> See I'm too dumb to open a file in Zig</a> - for common problems of why being a genius doens't scale - even for something as simple as reading a text file... I wanted to write code that scales linearly to 128+ cores, but feels like Ruby. That didn't exist, so I made it. 2026-04-03T00:00:00+00:00 The Goal</h2> I wanted the scaling of a 128-core shared-nothing engine, but the ergonomics of Ruby. I wanted to write high-level, readable logic and have a "Magic Button" to make it fast. In my latest benchmarks, this approach is already 6x faster than Go and 1.4x faster than Rust on a 128-core KV-store workload. But the speed isn't the story—the scaling is. The Problem: The "Lock Tax"</h2> I wanted to write shared-nothing code easily and correctly. Writing this kind of architecture in C or C++ is notoriously difficult. You are constantly fighting race conditions, manual memory management, and the silent performance killer: Cache-Line Bouncing. In traditional concurrent languages like Go or Rust, you eventually hit a "Contention Ceiling." You add more cores, but the overhead of keeping caches in sync (the "Lock Tax") means you stop seeing speedups. At 64 or 128 cores, these languages often hit negative scaling—adding more hardware actually makes your app slower. Why not Actors?</h3> Languages like Erlang or Elixir "solve" this with the Actor Pattern. They isolate state by forbidding shared memory entirely. It’s a beautiful model, and it works perfectly for telecom or niche IO cases where you have millions of slow processes sending small messages. But for high-performance systems, the Actor model just swaps one bottleneck for another: The Copying Tax. If you can't share memory, you have to copy it. When you're trying to perform 10 million operations per second on a KV store, the cost of copying keys and values between actors becomes far more expensive than the locks you were trying to avoid. Actors are "Shared-Nothing" conceptually, but they are "Copy-Everything" in practice. I wanted True Shared-Nothing: Zero locks, zero contention, and zero-copy performance. Why a library wasn't enough</h2> I didn't want to build a new language. I knew the "Ecosystem Desert" is where most new languages go to die. I tried to achieve my goals by building a library for existing languages, but I hit a wall every time: Go: You can't escape the Garbage Collector, and the scheduler is too opaque to guarantee the thread-pinning required for true shared-nothing performance.</li> Rust: The type system is brilliant but rigid. To change a concurrency model, you have to refactor every function signature in the call chain. I couldn't find a way to make optimization a "one-line directive" without fighting the borrow checker at every turn.</li> Zig: I love Zig's performance, but even with its comptime</code> magic, the code never felt like Ruby. It’s a "Manual Transmission" language; I wanted a "Smart Transmission" that chose the gears for me.</li> </ul> I wanted code to work like SQL. In SQL, you write a query that describes what you want, and then you add an INDEX</code> to optimize how it's retrieved. You don't rewrite the whole query in C just to make it scale. I wanted that "Optimization via Directive" for general-purpose logic. The Breakthrough: The Zig Bridge</h2> I finally realized that I didn't need to build a "Walled Garden." By transpiling CLEAR to Zig, I got native access to the entire C standard library and the LLVM toolchain for free. It didn't matter if people would use my language on Day 1, because CLEAR has access to everything that already exists in C and Zig. I didn't have to build a new world; I just had to build a better way to talk to the one we already have. The Result: Shared-Nothing Capabilities</h2> CLEAR achieves its performance through Capabilities. Instead of bake-in a concurrency model into a Type, you apply it to a Binding: -- Write it like Ruby: MUTABLE map: HashMap<String> = {}; -- Optimize it with one line: MUTABLE map: HashMap<String>@sharded(128) = {}; </code></pre> By adding @sharded(128)</code> and @pinned</code>, you tell the CLEAR compiler to generate a shared-nothing, lock-free architecture that scales linearly. The business logic remains identical. The compiler handles the infrastructure. Conclusion</h2> CLEAR is currently in v0.1. It’s not ready for your production database yet, but it’s ready to prove that the "Iron Triangle" of Performance, Scaling, and Ergonomics can be broken. CLEAR has not yet proved that the triangle is broken—a v0.1 release is just the beginning. But I believe these results are definitive proof that with further development, the triangle will be broken. I’m not here to ask you to switch your stack. I’m here to show you a solution to the "Many-Core" problem that is only going to get harder as our hardware keeps growing. Problem solved. Profiling 2026-04-02T00:00:00+00:00 Quick Start</h2> ./clear profile myapp.cht ./clear doctor myapp.profile/ </code></pre> clear profile</code> builds with allocation tracking enabled (zero overhead in normal builds), runs the program, and collects: Heap profile: allocation counts and bytes per call site</li> CPU profile: perf record</code> sampling (when available)</li> Syscalls: strace -c</code> breakdown</li> Hardware counters: perf stat</code> (cycles, cache misses, branch misses)</li> </ul> clear doctor</code> reads the profile data and prints actionable optimization advice with CLEAR source line numbers. Example: Optimizing Benchmark 17 (KV Store)</h2> Step 1: Profile</h3> ./clear profile benchmarks/concurrent/09_kvstore/bench.cht ./clear doctor benchmarks/concurrent/09_kvstore/bench.profile/ </code></pre> Step 2: Read the CPU profile</h3> === CPU Profile === 37.49% hash_map.HashMapUnmanaged.getIndex -- hashmap probing 12.21% pthread_rwlock_unlock -- RwLock release 6.37% memcpy -- value copying 5.29% hash.wyhash.Wyhash.hash -- key hashing 4.88% ShardedStringMap.put -- shard selection + lock 2.84% pthread_rwlock_rdlock -- RwLock acquire </code></pre> Observation: 15% of CPU is in pthread_rwlock_unlock</code> + pthread_rwlock_rdlock</code>. That's lock overhead, not useful work. Step 3: Check hardware counters</h3> === Hardware Counters === 5,390,314,521 instructions # 1.66 insn per cycle 10,001,037 cache-misses # 52.87% of all cache refs 60,878,759 L1-dcache-load-misses # 4.88% of all L1-dcache accesses </code></pre> Observation: 53% LLC cache miss rate. The hashmap's random-access probing misses the cache on every other access. This is inherent to hash tables with large working sets - not fixable by resharding. Step 4: Form a hypothesis</h3> Two bottlenecks: Lock contention (15% CPU): The benchmark uses @writeLocked</code> (RwLock). RwLock has high per-operation overhead even for single-writer access. For write-heavy workloads, @locked</code> (Mutex) should be cheaper.</li> Cache misses (53% LLC): Inherent to random hash probing across 1M keys. Not fixable without changing the data structure.</li> </ol> Step 5: Test the change</h3> Switch from @shared:sharded(128):writeLocked</code> to @shared:sharded(128):locked</code>: Config</th> SET</th> GET</th> Zipf</th> Mixed</th> Total</th></tr></thead> @writeLocked</code> (before)</td> 110ms</td> 17ms</td> 18ms</td> 77ms</td> 246ms</td></tr> @locked</code> (after)</td> 58ms</td> 20ms</td> 24ms</td> 26ms</td> 198ms</td></tr> </tbody></table> SET: 47% faster (Mutex has lower overhead for exclusive access)</li> Mixed: 66% faster (RwLock writer starvation was killing mixed workloads)</li> Total: 20% faster</li> GET/Zipf: slightly slower (Mutex blocks concurrent readers that RwLock allows)</li> </ul> Step 6: Compare against Rust and Go</h3> </th> Rust (DashMap)</th> Go (sync.RWMutex)</th> CLEAR @locked</th></tr></thead> SET</td> 75ms</td> 1217ms</td> 58ms</td></tr> GET</td> 14ms</td> 529ms</td> 20ms</td></tr> Zipf</td> 13ms</td> 17ms</td> 24ms</td></tr> Mixed</td> 19ms</td> 26ms</td> 26ms</td></tr> Total</td> 237ms</td> 1806ms</td> 198ms</td></tr> </tbody></table> CLEAR beats Rust (DashMap) by 17% total. SET is 22% faster than DashMap. What the profiler told us</h3> The profiler didn't just confirm "it's slow" - it pointed to the specific mechanism: CPU profile showed 15% in rwlock functions</li> Hardware counters confirmed cache misses are inherent (not fixable)</li> This directed the fix to the lock type, not the shard count</li> </ol> We also tested reducing shards from 128 to 32 - this made things worse (more contention per shard), confirming that the shard count wasn't the problem. How It Works</h2> Heap Profiling</h3> Built into the runtime allocator VTable. Every allocation in the runtime passes @returnAddress()</code> (the caller's return address). The profiler records this address, allocation count, and bytes per site in a fixed-size hash table (1024 sites, no heap allocations inside the profiler). Key runtime helpers (charAtCodepoint</code>, intToString</code>, substr</code>, concat</code>, join</code>) also call profileAlloc()</code> with their own @returnAddress()</code>, giving function-level attribution. Controlled by a comptime flag (CLEAR_PROFILE</code>). When not set, all profiling code is eliminated at compile time - zero overhead in production. CPU Profiling</h3> Uses Linux perf record</code> for sampling-based CPU profiling. Requires perf_event_paranoid <= 2</code>: sudo sysctl kernel.perf_event_paranoid=2 # temporary, resets on reboot </code></pre> Source Mapping</h3> clear doctor</code> maps addresses back to CLEAR source lines using: addr2line</code> to resolve addresses to Zig file:line</li> // CLR:N</code> comments in the transpiled Zig to map back to CLEAR line numbers</li> </ol> Profile builds use -fno-strip</code> to retain debug symbols. Environment Variables</h2> Variable</th> Set by</th> Purpose</th></tr></thead> CLEAR_ALLOC_PROFILE</code></td> clear profile</code></td> Output path for allocation data</td></tr> CLEAR_THREADS</code></td> User</td> Number of scheduler threads</td></tr> </tbody></table> Architecture 2026-03-31T00:00:00+00:00 1. Arena-Based Memory & Isolation CLEAR leverages affine types and Arena-based memory to give you near High-Frequency Trading standards of allocation with zero thought.</li> Due to optimizing broadly for a fiber-runtime, there are cases when the stack could be better leveraged, but CLEAR chooses not to for large stack objects (to keep fiber stacks small, until most fibers are transformed seamlessly into Finite State Machines by v0.4).</li> </ul> 2. Deterministic Shared-Memory and a Declarative Concurrency Model Parallelism is achieved via BG/DO/CONCURRENT</code>, which creates isolated execution contexts. Rather than telling the code how to achieve fast speeds, you simply tell write the code that expresses your intent - the what.</li> CLEAR provides one-line capability optimizations. Changing your app from an Arc/RwLock strategy suffering from synchronization bottle-necks to a shared-nothing architecture which can compete with Dragonfly DB can be a one-line change.</li> </ul> </li> Spawning a process creates a lightweight, isolated fiber / memory arena (or Finite State Machine in the near future).</li> </ul> 3. Implicit "Railway" Error Handling CLEAR treats errors as data, but handles them via control flow.</li> The SMOOTH</code> operator |></code> (aka the PIPE</code> or |> </code> in Elixir, etc) acts as a guard. It automatically bubbles errors down the chain, to be handled elsewhere, or allows them to be handled inline elegantly.</li> </ul> </li> This ensures code reads top-to-bottom & is left-sided (the "Happy Path") -- making it always clear what's desired vs what's the fallback.</li> The Result: No if err != nil boilerplate. No Pyramid of Doom. No checkOk clutter. No if .nil?</code> everywhere.</li> </ul> 4. A Type system that just works CLEAR is easy to write for beginners. The Type system is implicit, staying out-of-the-way by powerful Type inference.</li> The compiler takes care of the confusing parts of the type system for you.</li> The Result: If you can write code in JavaScript, Lua, Python, or Ruby - you can write blazing fast CLEAR code.</li> </ul> 5. Fortress Architecture All PUBLIC</code> functions must return a single type with a suitable default - or explicitly error</li> All PUBLIC</code> structs must have a suitable default - or explicitly an error A PUBLIC</code> function obviously cannot take a non-PUBLIC</code> struct as an input.</li> </ul> </li> The Result: Guarantees that only YOUR code can cause run-time errors (or none if in STRICT</code> mode). You can auto-gen tests for PUBLIC functions for all permutations of POSSIBLE unexpected inputs. This allows you to spot problems easily and arrive at robust, working code quickly.</li> </ul> </li> </ul> </li> </ul> CLEAR for Scripters 2026-03-31T00:00:00+00:00 If you already know Logic (JavaScript/Python), the only thing stopping you from writing System-Level code is Memory. In Ruby/Python/JavaScript, Memory is "Magic."</li> In C, it's an incomprehensible arcana.</li> In CLEAR, Memory is "Physics."</li> </ul> Here are the 9 Rules of Physics in CLEAR. 1. Physics of change: MUTABLE</code> vs IMMUTABLE</code> (Default)</h3> In Ruby or Python, you can change the value of any variable by default (they behave as MUTABLE</code>). name = "Bob" name = "Alice" </code></pre> In CLEAR, binding a name for the first time creates an immutable variable. Reassigning it is a compiler error. name = "Bob"; name = "Alice"; # COMPILER ERROR! `name` is immutable. </code></pre> x = value</code> = READ Only (immutable, no keyword needed).</li> MUTABLE x = value</code> = READ/WRITE.</li> </ul> The "Gotcha": If you want to modify a variable, it must be MUTABLE</code>. MUTABLE name = "Bob"; name = "Alice"; # OK </code></pre> 2. Physics of SHAPE: TYPES</h3> In CLEAR, variables have types to ensure safety and speed. MUTABLE name = "Bob"; name = 1; # COMPILER ERROR: `name` is a String, cannot assign a Float64. </code></pre> 3. Physics of Size: FIXED vs. DYNAMIC</h3> In CLEAR, you choose the physics you need. FIXED Size: Optimized for speed and cache locality.</li> DYNAMIC Size: Grows indefinitely, but requires the Warehouse (HEAP).</li> </ul> 4. The Two Worlds: STACK vs. HEAP</h3> The STACK (Default)</h4> Think of this as your Backpack. Pros: Instant access (L1 Cache). Blazing fast.</li> Cons: Itty-bitty space. Fixed size.</li> Behavior: When you finish a task (Function returns), you dump your backpack into the incinerator. Everything inside is gone. POOF.</li> </ul> The HEAP</h4> Think of this as a Warehouse. Pros: (nearly) Unlimited space. Can grow/shrink.</li> Cons: You have to drive there to get stuff (Slower).</li> </ul> How to Choose</h4> In CLEAR, the compiler and runtime handle the physics of where data lives for you in 99% of cases. You don't need to use a sigil to choose between stack and heap. x = [1, 2]</code> → CLEAR decides the most efficient location.</li> list = [1, 2, 3]</code> → This list is optimized for performance and safety.</li> </ul> If you need to explicitly force an object onto the heap (e.g., for recursive structures or large buffers), you can use the indirect</code> capability (similar to Box</code> in Rust). # Recursive structures use 'indirect' to avoid infinite size on stack STRUCT Node { value: Int64, left: ?Node @indirect, left: ?Node @indirect } </code></pre> The "Gotcha": If you try to .push!</code> or .pop!</code> to a fixed-size array, the compiler will yell at you. It’s not being mean; it’s telling you that physics forbids it. x = [1, 2, 3]; # ... do something, now I need to add to `x`, what do I do? x.append!(4); # COMPILER ERROR! `x` is immutable. </code></pre> 5. Physics of Capability: Capabilities vs Types</h3> In CLEAR, we separate Types from Capabilities. Types describe what the data is (e.g., User</code>, Account</code>).</li> Capabilities describe how you access it (e.g., shared</code>, multiowned</code>, alwaysMutable</code>).</li> </ul> The Rule: Functions take Types, not Capabilities. The CLEAR Model</h3> Ownership: Rc = multiowned</code>, Arc = shared</code></li> Synchronization: RwLock = shared:writeLocked</code>, Mutex = shared:locked</code> coming soon -> Mvcc = shared:versioned</code>, :actor</code> uses Object Actor Pattern combined with compiler aware SHARDING</li> </ul> </li> Interior Mutability: Cell, RefCell -> combined = alwaysMutable</code> Automatically acts like Cell for data under 16 bytes</li> alwaysMutable</code> must be unwrapped before individually passing into a function as an argument, like any other capability</li> </ul> </li> Existence: Option, Result => not a capability -> a tense: T?</code> = Optional T</code></li> Unwrapped like in Rust and Zig with .?</code></li> </ul> </li> Future:: Something that will arrive later ~T</code> = Future T</code></li> ~T[]</code> = Future T</code>s (like a Stream).</li> </ul> </li> </ul> affUser = User.new(); # creates `affine User` (default) a = affUser; # OKAY, affine MOVE, affUser is dead b = affUser; # Compiler error, affUser is dead sharedU = SHARE(User.new()); # turns `affine User` into `shared User` (Arc) c = sharedU; # OKAY, sharedU is not dead </code></pre> Why it's superior: Zero Blast Radius Refactoring</h4> In Rust, capabilities like Arc</code>, Rc</code>, and Mutex</code> infect function signatures. Changing from Rc<User></code> to Arc<User></code> forces a massive refactor because every function signature and call site must change. In CLEAR, if you need thread-safety, you change one line at the definition site: # Change multiowned (Rc) to shared (Arc) sharedU = SHARE(User.new()); </code></pre> Your functions (which just take User</code>) never knew about the capability, so they don't need to change. Synchronization Strategies</h4> For multi-threaded shared</code> objects, you choose the strategy: shared:read</code>: (MVCC) Optimized for massive read scaling.</li> shared:writeLocked</code>: (RwLock<Arc<T>></code>) Multiple readers OR one writer.</li> shared:locked</code>: (Mutex<Arc<T>></code>) One thread at a time.</li> </ul> Interior Mutability</h4> For complex data, use alwaysMutable</code> (RefCell</code>). CLEAR handles the lock for you: # 99% Case: Compiler handles temporary lock user.login_count += 1; # 1% Case: Scoped mutation WITH user.config { _.theme = "Light"; _.retries = 5; } </code></pre> 6. Physics of Sight (SCOPES)</h3> Functions can ONLY see what is explicitly passed into them: x = 10; FN add() -> RETURN x + 5; # COMPILER ERROR: I don't know what 'x' is. END </code></pre> Use USE</code> for upvalues: x = 10; FN add(n) USE (x) -> RETURN n + x; # OK END </code></pre> 7. Physics of Time: The ARENA (Lifetimes)</h3> Variable lifetimes follow a simple birth/death cycle: they live as long as the Function they were born in. The ARENA Rule:</h4> When a function starts, it opens a clean ARENA (A bank of memory). Any variable you create lives in this ARENA. When the function ends, the entire ARENA is wiped. POOF. 9. Cheating Death: The TAKES</code> Keyword</h3> In 99% of cases, when you pass a variable to a function, you are just letting that function Borrow it. If a function needs to store that object in a long-lived structure (like a Tree or Global List), it must explicitly TAKE responsibility for it. # This function promises to adopt the 'child' FN addChild!(MUTABLE parent: Node, TAKES child: Node) -> parent.list.push!(GIVE child); END node = Node{val: 1}; addChild!(root, GIVE node); # I surrender ownership. node.print(); # COMPILER ERROR: Variable 'node' is dead. </code></pre> 10. Simplified Lifetimes: WITH RESTRICT</code></h3> Rust's borrow checker is hard because its side effects are non-local and implicit. In CLEAR, borrows that "poison" (restrict) a mutable variable are explicitly scoped using WITH RESTRICT</code>. MUT node = buildTree(); WITH RESTRICT node.child { # Inside this block, node.child is immutable (restricted). gc = node.grandChild(); node.child.name = "OK"; # COMPILER ERROR: node.child is RESTRICTed. } # Outside the block, node.child is mutable again. </code></pre> Path-Based Scoping: CLEAR allows you to restrict only the specific part of a data structure you are using (e.g., node.child</code>), leaving the rest of the object mutable. This minimizes "poison" and makes complex architectures easier to reason about. This ensures that "poisoning" is always visible and local. OPINIONS 2026-03-31T00:00:00+00:00 Boiler Plate is bad.</li> Understandable code is king. The easier it is to UNDERSTAND code, the more likely it is to arrive at a correct state.</li> Understanding what a function is supposed to do should not be bogged down by it's error handling logic. Like test code, error handling code should be separate as much as possible.</li> Read it as an addendum IFF you care.</li> </ul> </li> Implicit behavior / magic can make things harder to understand.</li> </ul> </li> You should be able to test anything, and it should be EASY. There should be ZERO test code in production code. Java @visibleForTesting is nice, but it shouldn't be necessary at all.</li> </ul> </li> The easier it is to write tests, the more likely you are to actually arrive at working code.</li> </ul> </li> Handling Errors should "just work". You should not always have to checkOk</code>. We can assume okay, and look to the bottom to see what happens when not okay, if we ever care.</li> </ul> </li> The easier it is to handle errors, the more likely you are to handle them correctly!</li> </ul> </li> If your code compiles in STRICT</code> mode, it will not produce a Run-time Error unless the system explicitly encounters a resource boundary, or the program executes an operation whose inputs are logically impossible to resolve correctly. TODO: Investigate Type Coercion Failure claims. Currently not implemented, but on the roadmap.</li> </ul> </li> TODO: Investigate claims of all errors being handled somewhere.</li> </ul> </li> Compiler errors should tell you exactly what's wrong, how to fix it, and be easy to understand.</li> Types should be your friend, helping you write working code faster, not an enemy that is constantly slowing you down.</li> 90% of your time should be spent writing the code you need, and 10% debugging, handling errors, fighting compilers - not the other way around.</li> Writing efficient code should be the default, and the default should be easy. Writing ineffiecent code should be obviously wrong.</li> </ul> </li> Anything that can be 1-line should be! Readability and understandability beat cleverness -- unless direly critical to performance.</li> The constructs and syntax of a language should lend itself to one-liners, as they are often easier to UNDERSTAND.</li> </ul> </li> Code should be as declaractive as possible. Every function should look like a clean chain of exactly what it should be doing. It should not look like a nested mess of how it's handling errors and undesirable states. Though it MUST have the capability to do that.</li> </ul> </li> </ul> </li> </ul> </li> You should not need to worry about memory management OR a garbage collector. Unless you're a rocket scientist, the right compiler can figure it out better than you can.</li> A good SQL engine beats all but the absolute most elite programmers.</li> A good compiler can also take an easy language and beat all but the most elite programmers (see LuaJIT).</li> CLEAR does ask you think about WHERE an object lives. Cache locality is literally 100x faster. If you wrote something cache-locality optimized in Ruby, it would crush a pointer cache miss in C.</li> Therefore, CLEAR is designed around making it as easy as possible to ensure you DON'T cache miss unless you absolutely must. This means you do need to think about if you want something on the STACK (default, fast) or the HEAP (slow, but sometimes required).</li> </ul> </li> </ul> </li> </ul> </li> You should not need to worry about a Global-Interpreter-Lock (GIL). Code should be able to run in parallel or concurrently EFFICIENTLY by default.</li> </ul> </li> Code should be as left-sided as possible. How many hours have YOU spent tracking down paren-syntax errors, figuring out which condition you're in, etc???</li> Time spent figuring out WHERE you even are logically, is time wasted, that could be spent getting things done.</li> </ul> </li> Someone who doesn't know CLEAR should be able to look at CLEAR code and intuit what it does.</li> Publicly-Exported APIs SHOULD have style-enforcements. At a minimum, if you're making a library, anyone should be able to understand the API.</li> All PUBLIC</code> functions MUST either explicitly RAISE</code> an error OR handle all errors. All PUBLIC</code> structs MUST either have a suitable default or !!</code> suffix. All PUBLIC</code> Union Types MUST be projectable for ease of use.</li> </ul> </li> </ul> </li> </ul> </li> Internal-code can be Chill-Correct. No interpretting errors because you have an unnused variable.</li> No forbidding your code to run for a test because you didn't follow a covention, etc.</li> Your code can fail because of your</code> problems, but not becuase of your dependencies problems. STRICT</code> mode compilation eliminates Chill-Correct and can guarantee you won't encounter virtually all preventable run-time errors.</li> </ul> </li> </ul> </li> </ol> WHO IS CLEAR *NOT* FOR 2026-03-31T00:00:00+00:00 CLEAR is opinionated. The specific optimizations that make it fast and safe for 99% of Business Logic make it extremely hostile to 1% of Architectural patterns. You are building a Pointer-Heavy Engine (like a Graph Database).</li> </ol> CLEAR prevents Memory Leaks by forbidding reference cycles in shared</code> objects (Shared A -> B, B -> A).</li> If your architecture relies on a "Soup of Mutable Objects" where everything references everything else, CLEAR will fight you.</li> The Alternative: Architect your data using IDs and centralized lookups (like a relational database), or use Rust/C++ for manual pointer management.</li> </ul> You need to model Inherently Unsafe / Cyclic Relationships</li> </ol> If your architecture relies on Rust-style "Weak Pointers" to manage reference cycles (A -> B -> A) OR if you need recursive fine-grained locking, and you are strictly managing memory and deadlock risks manually.</li> </ul> </li> CLEAR guarantees safety by forbidding these patterns entirely! They're rare!</li> The Alternative: If you absolutely need a doubly-linked list or a cyclic graph with individual node locking, that is "Engine Code," not "Business Logic." Write that specific component in Zig (where you can manage the unsafe pointers yourself) and import it into CLEAR as a safe handle.</li> </ul> Modules & Build System 2026-03-31T00:00:00+00:00 CLEAR uses a simple, explicit module system based on the REQUIRE</code> keyword. It separates code into Files, Packages, and Native Modules. REQUIRE — Importing Code</h2> Importing a CLEAR module makes its PUB</code> and package-private symbols available under a namespace. Local File Imports</h3> Use a relative path string to import files in your own project: REQUIRE "math_utils.cht"; FN main() RETURNS Void -> res = math_utils.add(10, 20); END </code></pre> Package Imports</h3> Use the pkg:</code> prefix to import registered libraries: REQUIRE "pkg:geometry"; FN main() RETURNS Void -> p = geometry.Point{ x: 1, y: 2 }; END </code></pre> Aliasing (AS)</h3> You can rename any import to avoid collisions: REQUIRE "math_utils.cht" AS m; REQUIRE "pkg:geometry" AS geo; FN main() RETURNS Void -> sq = m.square(geo.distance(p1, p2)); END </code></pre> Visibility</h2> CLEAR enforces three levels of visibility. Access control is checked at compile-time by the semantic annotator. Keyword</th> Level</th> Visibility</th></tr></thead> PRIVATE</code></td> File Private</td> Only accessible within the .cht</code> file where it is defined.</td></tr> (None)</td> Package Private</td> Accessible to any file in the same directory.</td></tr> PUB</code></td> Public</td> Accessible to any file that REQUIRE</code>s this module or package.</td></tr> </tbody></table> Example</h3> # internal_helper.cht PRIVATE FN secret_calc() RETURNS Int64 -> ... END FN package_helper() RETURNS Int64 -> RETURN secret_calc(); # OK: same file END PUB FN public_api() RETURNS Int64 -> RETURN package_helper(); # OK: same package END </code></pre> # main.cht REQUIRE "internal_helper.cht"; FN main() RETURNS Void -> internal_helper.public_api(); # OK: PUB internal_helper.package_helper(); # OK: same directory # internal_helper.secret_calc(); -- ERROR: PRIVATE END </code></pre> Future: STRICT Mode</h3> In future versions (v0.2+), PUB</code> declarations at a package boundary will require explicit effect declarations (e.g., EFFECTS :alloc</code>) when compiled in STRICT</code> mode. This ensures that library side-effects are always documented and verified. Build System</h2> CLEAR features a built-in dependency manager that handles recursive REQUIRE</code> calls, prevents circular dependencies, and manages the transpilation graph. Package Registration</h3> Packages are mapped to file paths at compile-time using the --pkg</code> flag: ruby src/transpiler.rb --pkg geometry=./libs/geometry/lib.cht main.cht > main.zig </code></pre> Module Transpilation</h3> To compile a library as a standalone Zig module (without the CLEAR runtime footer), use the --module</code> flag. This is useful when building CLEAR code to be consumed by native Zig applications. ruby src/transpiler.rb --module my_lib.cht > my_lib.zig </code></pre> How it Works</h3> Local Imports: The transpiler inlines the generated Zig code of the local file into a const struct</code> namespace in the caller's Zig file.</li> Package Imports: The transpiler emits a Zig @import("namespace")</code>. The actual wiring of these modules is handled by the Zig build system using the --dep</code> and -M</code> flags.</li> Deduplication: Each file is parsed and annotated exactly once, even if required by multiple files.</li> </ol> Project Structure (Recommended)</h2> my_project/ ├── main.cht # Entry point ├── logic.cht # Local module └── lib/ └── math/ ├── lib.cht # Package entry point (exports PUB symbols) └── private.cht # Internal logic (default visibility) </code></pre> CLEAR Language Walkthrough 2026-03-31T00:00:00+00:00 This guide showcases CLEAR: a memory-safe language that combines the ergonomics of scripting with the safety of affine types and the performance of hand-optimized C. 1. Immutability & Mutability</h2> Bindings are immutable by default. Reassignment requires the MUTABLE</code> keyword. x = 5; # OKAY: Immutable binding (default) name = "Alice"; # OKAY: Immutable string pi = 3.14159; # OKAY: Immutable float x = 6; # COMPILER ERROR: x is immutable MUTABLE counter = 0; # OKAY: Explicit mutability counter = 1; # OKAY: can reassign mutable binding </code></pre> 2. Primitive Types</h2> CLEAR provides a comprehensive set of primitives for precise control over memory. Type</th> Description</th> Example</th></tr></thead> Int8</code> .. Int64</code></td> Signed integers (8, 16, 32, 64-bit)</td> 42_i64</code>, -1_i8</code></td></tr> UInt8</code> .. UInt64</code></td> Unsigned integers (8, 16, 32, 64-bit)</td> 42_u64</code>, -1_u8</code></td></tr> Float32</code>, Float64</code></td> IEEE-754 floating point</td> 3.14159_f64</code>, 1.0_f32</code></td></tr> Bool</code></td> Boolean logic</td> TRUE</code>, FALSE</code></td></tr> Byte</code></td> Raw 8-bit data</td> 0x41_b</code></td></tr> String</code></td> UTF-8 encoded text (affine)</td> "hello"</code></td></tr> Void</code></td> Absence of a value</td> RETURN;</code></td></tr> </tbody></table> 3. Types vs Capabilities</h2> CLEAR distinguishes between what data is (Type) and how it is accessed (Capability). In Rust, capabilities like Arc</code>, Rc</code>, and Mutex</code> are part of the type, leading to "function coloring" and high refactoring costs. In CLEAR, functions take Types, not Capabilities. Capability Annotations</h3> Capabilities are applied at the declaration site with @</code> suffixes: Capability</th> Rust Equivalent</th> CLEAR Syntax</th></tr></thead> Single-threaded shared ownership</td> Rc<T></code></td> value @multiowned</code></td></tr> Multi-threaded shared ownership</td> Arc<T></code></td> value @shared</code></td></tr> Mutex (exclusive lock)</td> Arc<Mutex<T>></code></td> value @shared:locked</code></td></tr> RwLock (read-write lock)</td> Arc<RwLock<T>></code></td> value @shared:writeLocked</code></td></tr> Heap pointer</td> Box<T></code></td> value @indirect</code></td></tr> Thread-local pointer</td> (no direct equivalent)</td> value @local</code></td></tr> </tbody></table> Zero Blast Radius Refactoring</h3> In Rust, changing Rc<User></code> to Arc<User></code> means rewriting every function signature in the call chain. In CLEAR, functions take the plain type: STRUCT User { name: String } FN process(u: User) RETURNS Void -> print(u.name); RETURN; END FN main() RETURNS Void -> # At the call site, capabilities are unwrapped: shared_u = User{ name: "Alice" } @shared; WITH shared_u AS val { process(val); } RETURN; END </code></pre> If you change @shared</code> to @multiowned</code>, the process</code> function remains untouched. The refactor is a one-line change at the declaration. WITH Blocks for Capability Unwrapping</h3> For multi-statement access, use WITH</code> blocks: MUTABLE counter: Int64@shared:locked = 0; WITH EXCLUSIVE counter AS c { c += 1; print(c.toString()); } </code></pre> 4. Affine Ownership: GIVE & TAKES</h2> CLEAR uses affine types by default. Every value has exactly one owner. When you assign a value, ownership is moved, not copied. FN process(TAKES s: String) RETURNS Void -> print(s); # s is destroyed here (end of scope) END FN main() RETURNS Void -> msg = "Hello"; process(msg); # OKAY: Implicit transfer (by FN signature) print(GIVE msg); # COMPILER ERROR: USE AFTER MOVE: You can't GIVE `msg`. `process(msg)` TOOK it away. print(msg); # COMPILER ERROR: USE AFTER MOVE: You can't use `msg`. `process(msg)` TOOK it away. RETURN; END </code></pre> For more details, see docs/sharing-capabilities.md</a>. Parameter Ownership</h3> Function parameters are implicit borrows by default. Only TAKES</code> transfers ownership. Param Style</th> Ownership</th> Can Move Into Container?</th> Can Mutate?</th></tr></thead> v: Value</code></td> Borrow</td> No</td> No</td></tr> MUTABLE v: Value</code></td> Borrow</td> No</td> Yes</td></tr> TAKES v: Value</code></td> Owned</td> Yes</td> No</td></tr> TAKES MUTABLE v: Value</code></td> Owned</td> Yes</td> Yes</td></tr> </tbody></table> UNION Value { Nil, Num: Float64, Lambda { body: Value @indirect } } # Borrow: can read, cannot store or move FN inspect(v: Value) RETURNS String -> MATCH v START Value.Num AS n -> RETURN n.toString();, DEFAULT -> RETURN "other"; END END # Owned: can store into collections FN store!(TAKES v: Value, MUTABLE map: HashMap<Value>) RETURNS Void -> map["item"] = v; # OKAY: v is owned via TAKES RETURN; END FN bad!(v: Value, MUTABLE map: HashMap<Value>) RETURNS Void -> map["item"] = v; # COMPILER ERROR: Cannot move borrowed value 'v' RETURN; END </code></pre> Zero implicit copies. Copy types (primitives, strings, enums) can be freely used after assignment. Non-Copy types (unions with @indirect</code> or []T</code> variants, structs with heap data) follow move semantics. There are never implicit deep copies. 5. Sharded Shared-Nothing Architecture</h2> The @sharded</code> capability partitions data across threads, enabling massive parallelism without lock contention by automatically pinning threads to specific data shards. # A sharded map distributes keys across independent thread-local heaps MUTABLE registry: HashMap<Int64>@sharded(8) = {}; # CLEAR automatically pins this fiber to the correct shard BG { registry["key"] = 42; } # Sharding is also available for Pools and Lists MUTABLE users: User[100]@pool:sharded(4) = []; MUTABLE logs: String[]@list:sharded(2) = []; # Note: Sharding provides peak throughput but carries a risk of # data skew if keys/items are not uniformly distributed. </code></pre> Shared-nothing architectures present problems if your workloads can be heavily skewed. </div> 6. Function Signatures</h2> Functions support explicit types, failable returns (!T</code>), and optional types (?T</code>). STRUCT Point { x: Float64, y: Float64 } FN sum(p: Point) RETURNS Float64 -> RETURN p.x + p.y; END FN main() RETURNS Void -> p = Point{ x: 3.0, y: 4.0 }; d = sum(p); RETURN; END </code></pre> Failable and optional returns: FN findUser(id: Int64) RETURNS !?User -> IF id < 0 -> RAISE "Invalid ID"; RETURN result; END </code></pre> Recursion</h3> Recursive functions must be explicitly annotated with @reentrant</code>: FN fib(n: Int64) RETURNS Int64 @reentrant -> IF n <= 1 -> RETURN n; RETURN fib(n - 1) + fib(n - 2); END </code></pre> 7. Basic Control Flow</h2> CLEAR provides standard control flow constructs with support for one-line shorthands. # 1. IF / ELSE_IF / ELSE x = 75; IF x > 100 THEN print("Large"); ELSE_IF x > 50 THEN print("Medium"); ELSE print("Small"); END # 2. WHILE loops MUTABLE i = 0; WHILE i < 10 DO print(i.toString()); i += 1; END # 3. FOR loops (Range iteration) FOR j IN (1 ..= 5) -> print(j.toString()); # OKAY: Inclusive range FOR j IN (1 ..< 5) -> print(j.toString()); # OKAY: Exclusive range </code></pre> 8. Enums, Unions, and Pattern Matching</h2> Enums</h3> Simple enumerations for discrete states: ENUM Direction { North, South, East, West } FN describe(d: Direction) RETURNS String -> MATCH d START Direction.North -> RETURN "up";, Direction.South -> RETURN "down";, Direction.East -> RETURN "right";, Direction.West -> RETURN "left"; END END </code></pre> Tagged Unions (Sum Types)</h3> Unions carry a payload per variant. Unit variants (no payload) are also supported: UNION Result { Ok: Float64, Err: String, Empty } FN main() RETURNS Void -> # Payload variant r = Result{ Ok: 42.0 }; # Unit variant (no payload) e = Result.Empty; MATCH r START Result.Ok -> print("success");, Result.Err -> print("error");, Result.Empty -> print("nothing"); END RETURN; END </code></pre> MATCH AS and Ownership</h3> MATCH ... AS</code> payload extraction follows the same ownership rules as parameters. The binding is a borrow of the source by default, or a move if the source is owned. Source Ownership</th> MATCH AS Produces</th> Can Store in Struct?</th></tr></thead> Borrowed (parameter, map.get</code>)</td> Borrow</td> No</td></tr> Owned (local, TAKES param)</td> Move (consumes source)</td> Yes</td></tr> </tbody></table> UNION Value { Nil, Num: Float64, List: Value[] } # Borrowed source: MATCH AS produces a borrow FN sum(v: Value) RETURNS Float64 -> MATCH v START Value.List AS items -> # items is &[]Value (borrow) RETURN items.length(); # OKAY: reading a borrow DEFAULT -> RETURN 0.0; END END # Owned source: MATCH AS moves the payload FN take!(TAKES v: Value) RETURNS Value[] -> MATCH v START Value.List AS items -> # items is []Value (owned, v consumed) RETURN items; # OKAY: returning owned data DEFAULT -> RETURN List[]; END END </code></pre> Storing a borrowed MATCH AS</code> binding into a struct or union variant is a compile error: FN bad(items: Value[]) RETURNS Value -> # items is a borrowed parameter RETURN Value.Lambda{ params: items }; # COMPILER ERROR: Cannot store borrowed 'items' END </code></pre> Generics</h3> Structs and unions support type parameters: STRUCT Pair<T> { first: T, second: T } p = Pair<Int64>{ first: 1, second: 2 }; </code></pre> 9. Higher-Order Functions & Error Handling</h2> CLEAR supports powerful functional pipelines via the Smooth operator |></code>. # 1. Pipelines: Filter, Aggregate, Transform alive = entities |> WHERE _.health > 0; total = scores |> SUM _.value; names = users |> SELECT _.name; # 2. Side effects & Function Piping entities |> EACH { _.x += _.vx; }; result = data |> process |> validate |> format; # 3. Error Handling: Inline OR / OR RAISE val = parseInt("abc") OR 0; # OKAY: Fallback value content = readFile("config.json") OR RAISE; # OKAY: Explicit propagation # 4. Function-level CATCH FN main() RETURNS Void -> result = loadConfig("config.json") OR RAISE; print("Config: ${result}"); RETURN; CATCH e print("Failed to load: ${e}"); RETURN; END </code></pre> See docs/pipelines.md#operators</a> for a full list of higher-order function operators. Pipelines are automatically fused for efficiency. 10. Time as Tense (~T)</h2> Tense represents a value that will exist in the future. CLEAR eliminates the complexity of Future/Promise/Observable</code> with a single unified tense. ~User</code> is read as "Future User".</li> ~User[]</code> is read as "Future Users".</li> A STREAM of users is simply one way to produce "Future Users".</li> </ul> # 1. Promise (~T): A single future value p: ~String = BG { sleep(100); RETURN "Data"; }; val = NEXT p; # OKAY: Blocks until ready # 2. Open Stream (~?T[]): Asynchronous generator # NEXT on ~?T[] returns ?T, with NIL signaling exhaustion. gen: ~?Int64[] = BG STREAM { YIELD 10; YIELD 20; }; val = NEXT gen; # OKAY: Returns ?Int64 (NIL when exhausted) # 3. Infinite Stream (~T[INF]): Lazy rendezvous generator counter: ~Int64[INF] = BG STREAM { MUTABLE i = 0; WHILE TRUE DO { YIELD i; i += 1; } }; v1 = NEXT counter; # OKAY: Returns Int64 (never NIL) </code></pre> Stream Capabilities: @shared</code> vs @split</code></h3> Streams can also carry sharing capabilities: @shared</code> means the stream handle may be shared across threads, but it is not ordered replay. Multiple threads calling NEXT</code> are sharing access to one producer.</li> @split</code> means ordered fanout / pub-sub semantics. Each cloned handle sees the same items in the same order, and memoized items are released once every live split owner has consumed them.</li> </ul> source: ~?Int64[]@split = BG STREAM { YIELD 10; YIELD 20; }; a: ~?Int64[]@split = CLONE source; b: ~?Int64[]@split = CLONE source; ASSERT NEXT a == 10, "first subscriber sees first value"; ASSERT NEXT b == 10, "second subscriber sees the same first value"; ASSERT NEXT a == 20, "order is preserved"; ASSERT NEXT b == 20, "order is preserved for all subscribers"; </code></pre> CLONE</code> is mandatory for @split</code> streams. Plain assignment moves the handle. This is the model CLEAR intends for pub/sub style workloads. For a concrete benchmark in this area, see benchmarks/16_pubsub/bench.cht</a>. That benchmark still uses the older sharing path today, but it is the right benchmark to watch as @split</code> becomes the native pub/sub primitive. 11. Collections: Array, List, and Pool</h2> All collections are automatically monomorphized -- the compiler generates zero-overhead, type-specific native code for every unique T</code>. Sigil</th> Collection</th> Purpose</th></tr></thead> T[N]</code></td> Array</code></td> Fixed-size, stack-allocated (if small)</td></tr> T[]@list</code></td> List</code></td> Dynamic-size, heap-backed</td></tr> T[N]@pool</code></td> Pool</code></td> Fixed-capacity, generational handles</td></tr> T[]@set</code></td> Set</code></td> Dynamic-size, heap-backed, distinct items</td></tr> T[]@ring</code></td> Ring</code></td> Circular Ring Buffer (mainly for Streams - v0.2)</td></tr> </tbody></table> STRUCT User { name: String } # 1. Fixed Array vals = [10, 20, 30]; # 2. Dynamic List MUTABLE items: Int64[]@list = []; items.append(42); # 3. Generational Pool # Pools provide peak cache locality. Switching from List to @pool:soa # (Structure of Arrays) is a one-line refactor for massive speed. MUTABLE users: User[100]@pool = []; id = users.insert(User{ name: "Alice" }); # Returns stable handle user = users.get(id) OR RAISE; # Returns ?T (checks stale handles) </code></pre> Element-Level Capabilities</h3> Capabilities normally apply to the collection (T[]@shared</code> = one shared list). For rare cases where each element needs its own capability, place the capability before the array suffix: # Collection-level (common): one Arc wrapping the whole list MUTABLE shared_list: User[]@list:shared = []; # Element-level (rare): each element is individually Arc'd MUTABLE arc_users: User@shared[]@list = []; # Element-level with pool: each slot holds an Arc'd user MUTABLE arc_pool: User@shared[100]@pool = []; </code></pre> Element-level capabilities are primarily useful in struct fields where individual elements need independent lifetime management (e.g., a graph where each node is shared across multiple edges). 12. Strings and Buffers</h2> Strings in CLEAR are Copy (like Rust's &str</code> — a pointer + length). Assignment copies the slice header. x = "hello"; y = x; # x is moved (strings are owned, non-Copy) # z = x; # would be use-after-move error z = COPY y; # OK: explicit deep-copy # String concatenation full = "foo" + "bar"; # "foobar" (single allocation, no intermediate) </code></pre> Like arrays, Strings have capabilities: String@raw</code> - raw byte buffer; no UTF-8 encoding assumption, O(1) byte indexing, zero-copy slicing. Use for binary data, network buffers, parsers operating on bytes.</li> String@symbol</code> - compile-time interned identifier. Literals use Ruby-style :ok</code> syntax. The compiler embeds the string in read-only static memory and deduplicates identical literals, so equality is O(1) pointer comparison and the value needs no cleanup (program-lifetime). Use for status tags, option names, event names, and map keys where the set of values is known at compile time. Note: interned HashMaps carry their own per-map symbol table for runtime string keys - this avoids a globally locked intern table, which would create a contention hotspot incompatible with CLEAR's scalability goals.</li> String@ring</code> (v0.2) - circular buffer for streaming.</li> </ul> 13. Graphs, Cycles, and @indirect</h2> For recursive or cyclic data structures, use @indirect</code> (heap-allocated pointer, like Rust's Box<T></code>). Combined with @reentrant</code> for recursive traversal: # Recursive tree node using @indirect for child pointers STRUCT Node { value: Int64, left: ?Node@indirect, # Optional heap-allocated child right: ?Node@indirect } # Recursive traversal must be @reentrant FN sumTree(n: Node) RETURNS Int64 @reentrant -> MUTABLE total = n.value; IF n.left -> total += sumTree(n?.left OR 0); IF n.right -> total += sumTree(n?.right OR 0); RETURN total; END </code></pre> @indirect</code> gives the node a stable heap address, enabling graph structures. @multiowned</code> (Rc) enables shared ownership for DAGs. For cyclic graphs, use @link</code> -- CLEAR's weak reference. ?.</code> is the safe navigate operator to peek into optional types. It combines with OR</code> to handle missing data like an error. Weak References with @link</h3> @link</code> creates a weak reference that does not keep the target alive. It works with both @multiowned</code> (WeakRc) and @shared</code> (WeakArc). LINK expr</code> -- downgrade a strong reference to a weak reference</li> RESOLVE expr</code> -- upgrade a weak reference back to an optional strong reference (?T</code>)</li> </ul> # Parent-child with back-pointer cycle STRUCT Parent { name: String, child: ?Child@multiowned@indirect } STRUCT Child { name: String, parent: ?Parent@link # weak back-pointer, breaks the cycle } FN main() RETURNS Void -> p = Parent{ name: "Alice", child: NIL } @multiowned; # LINK downgrades the strong Rc to a WeakRc weak_p = LINK p; # RESOLVE returns ?Parent@multiowned; bind first, then use ?. to safely access fields resolved = RESOLVE weak_p; name = resolved?.name OR "dropped"; ASSERT name == "Alice", "resolved"; RETURN; END </code></pre> Key rules: LINK</code> only works on @multiowned</code> or @shared</code> values (compile-time error otherwise)</li> RESOLVE</code> only works on @link</code> values (compile-time error otherwise)</li> RESOLVE</code> returns ?T</code> -- bind the result first, then use resolved?.field OR fallback</code> to handle the dropped case</li> Cleanup is automatic: weak references are released when they go out of scope</li> No runtime cost when the strong reference is still alive; RESOLVE is a simple count check</li> </ul> 14. Concurrency: BG & DO</h2> CLEAR makes background tasks and fork-join parallelism trivial. # BG: Background execution p: ~Int64 = BG { RETURN slowComputation(); }; # DO: Fork-Join parallel execution DO { step1(), step2() } # CONCURRENT: Parallel pipelines results = items |> CONCURRENT(workers: 8) SELECT transform(_); </code></pre> BG Stack Modifiers</h3> Modifier</th> Stack Size</th> Use</th></tr></thead> @micro</code></td> 4 KB</td> Lightweight tasks</td></tr> @standard</code></td> 16 KB</td> Default</td></tr> @large</code></td> 64 KB</td> File I/O, deep recursion</td></tr> @xl</code></td> 256 KB</td> Very deep call stacks</td></tr> @pinned</code></td> (any)</td> Pin to local scheduler</td></tr> @arena</code></td> (any)</td> Thread-local arena; implies @pinned</td></tr> </tbody></table> 15. Modules & Imports</h2> CLEAR uses a simple namespace-based module system via REQUIRE</code>. REQUIRE "math_utils.cht" AS m; # Local file alias REQUIRE "pkg:geometry"; # Package import FN main() RETURNS Void -> p = geometry.Point{ x: 1, y: 2 }; print(m.square(10).toString()); RETURN; END </code></pre> See docs/modules.md</a> for visibility (PUB</code>/PRIVATE</code>) and build details. 16. FFI: Native Integration</h2> CLEAR integrates directly with Zig and C libraries via EXTERN</code> declarations. All EXTERN FN calls automatically trampoline to the OS stack (g0) for fiber safety. Importing Functions and Types</h3> # Import a struct type from a Zig module EXTERN STRUCT Vec2 { x: Float64, y: Float64 } FROM "math_native"; # Import a free function EXTERN FN computeDistance(v: Vec2) RETURNS Float64 FROM "math_native"; # Import with allocator injection (EFFECTS) EXTERN FN parseJson(data: String) RETURNS !JsonDoc EFFECTS :alloc:heap FROM "json_native"; </code></pre> Local Struct Definitions (no FROM)</h3> For passing default options or defining Zig-compatible layouts: # Local struct (no external module) EXTERN STRUCT ParseOptions {}; EXTERN STRUCT JsonRecord { id: Int64, data: Int64[] }; # Comptime type parameters for generic FFI EXTERN FN parseFromSliceLeaky<T>(comptime: T, content: String, options: ParseOptions) RETURNS !T EFFECTS :alloc:heap FROM "std.json"; </code></pre> Method Calls on EXTERN Structs</h3> EXTERN STRUCT Dir {} FROM "std.fs"; EXTERN FN cwd() RETURNS Dir FROM "std.fs"; EXTERN FN Dir.makePath(self: Dir, path: String) RETURNS Void FROM "std.fs"; # Chained method call (both trampolined to g0) cwd().makePath("data"); </code></pre> EXTERN STRUCT CLOSE (RAII)</h3> EXTERN STRUCT Buffer { data: String } CLOSE "deinit" FROM "native_resource"; # Buffer.deinit() is auto-called when buf goes out of scope buf = createBuffer("hello"); # defer buf.deinit() emitted automatically </code></pre> See json_api</a> for an example.</li> See docs/agents/ffi.md</a> for the complete FFI guide.</li> </ul> 17. Memory Model</h2> Arena Allocation</h3> Every function has its own memory arena. Variables live as long as the function they were born in. Stack (~0ns): Primitives and small structs (up to 128 slots)</li> Frame Arena (~2ns): Large structs, temporary buffers, string concat</li> Heap (~60ns): Dynamic collections, cross-fiber data, Rc/Arc</li> </ul> Escaping the Arena</h3> RETURN</code>: Values stay in the caller's frame arena (no heap dupe for strings).</li> GIVE</code>: Ownership is transferred to the callee.</li> @shared</code> / @multiowned</code>: Reference-counted objects that live as long as they are referenced.</li> </ul> Loop Arena Rewind</h3> Inside loops, the frame arena is rewound each iteration to prevent unbounded growth. Mutable variables that accumulate across iterations (like string concatenation) are automatically detected and excluded from the rewind. CLEAR additionally inserts cooperative yielding into loops to prevent head-of-line blocking, etc. This destroys SIMD optimizations. CLEAR can detect when a SIMD optimization is possible and warn you to use TIGHT</code> loops: TIGHT FOR i IN (1..<1000) -> ... </code></pre> CLEAR will also warn you when you're using a TIGHT</code> loop and you should not. Misusing TIGHT</code> loops can destroy your p99 latency. </div> 18. The Reality of Concurrency</h2> In an ideal world, every workload would distribute perfectly across available cores and memory. In reality, concurrency is difficult because of the "messy middle": Skew: Data is rarely uniform. Sharded collections can develop "hot shards" that bottleneck the system.</li> Outliers: The 0.1% of workers that take 100x longer than typical (due to cache misses or deep recursion) destroy p99 response times.</li> Non-Cooperation: Problems like head-of-line blocking and thundering herds can stall an entire thread pool.</li> Variadic Depth: Real-world recursion and loops often exceed the fixed stack sizes of traditional fibers.</li> </ol> CLEAR handles these issues through its Control Plane -- an active runtime observer that uses live telemetry to manage execution. Fiber Overflow: The runtime detects imminent stack overflows and auto-upsizes future tasks to @large</code> or @xl</code> stacks.</li> Workload Mismatch (v0.2): Telemetry-driven alerting when the runtime detects contented or skewed workloads. ./clear profile</code> can automatically suggest fixes.</li> Shared-Nothing Safety: By enforcing sharding and affine moves at the language level, the Control Plane can optimize memory layout without fear of data races.</li> Workload Migration (v0.4+): Telemetry-driven migration allows the runtime to detect contention and skew in real-time and react to it, IF you have the protection enabled (added memory).</li> </ul> See docs/control-plane.md</a> for more on how CLEAR manages the reality of high-performance systems. 19. Function Lifetimes</h2> Functions cannot return borrowed data freely, but can with a lifetime. This is simplified from Rust. STRUCT Node { left: ?Node, right: ?Node } FN grandChild(n: Node) RETURNS n.left:?Node -> RETURN n.left?.right; END </code></pre> Lifetimes are scoped with <param>.<path>:Type</code>. In the example above, n.left</code> is the path and ?Node</code> is the return type. This is a less restrictive lifetime than r:Bar</code>. Standard IMMUTABLE</code> returned borrows just work: bar = root.identity(); print(bar.index); </code></pre> MUTABLE</code> returned borrows require a scope - because the object you're borrowing from is restricted while you hold the borrow to prevent use-after-free: WITH RESTRICT node AS n { MUTABLE gc = n.grandChild(); gc.value = 0; node.left = Nil; # COMPIILER ERROR: `node` is RESTRICTED. It's immutable in this scope. n.left = Nil; # COMPIILER ERROR: `n` is RESTRICTED. It's immutable in this scope. } </code></pre> 20. STRUCT lifetimes</h2> It's sometimes useful to define a STRUCT with a borrowed field (iterators are a prime example): STRUCT SliceIter { source: BORROWED Int64[], # Allows `source` to be a BORROW, must be initialized in a scope. pos: Int64, len: Int64 } FN hasNext(iter: SliceIter) RETURNS Bool -> RETURN iter.pos < iter.len; END FN iterExample() RETURNS Void -> WITH BORROWED data AS ref { # create a borrowed version MUTABLE iter = SliceIter{ source: ref, pos: 0, len: n }; WHILE hasNext(iter) DO total += currentVal(iter); iter = advance(iter); END RETURN iter; # COMPILER ERROR: You can't return `iter`. It's BORROWED. You don't own it. } END </code></pre> Quick Reference: Capabilities</h2> Annotation</th> Purpose</th></tr></thead> @multiowned</code></td> Single-threaded shared ownership (Rc)</td></tr> @shared</code></td> Multi-threaded shared ownership (Arc)</td></tr> @shared:locked</code></td> Arc + Mutex</td></tr> @shared:writeLocked</code></td> Arc + RwLock</td></tr> @locked</code></td> Mutex (single-scheduler)</td></tr> @writeLocked</code></td> RwLock (single-scheduler)</td></tr> @local</code></td> Thread-local heap pointer</td></tr> @indirect</code></td> Explicit heap allocation (Box)</td></tr> @link</code></td> To create cyclic graphs (WeakRef/Ref)</td></tr> @sharded(N)</code></td> Shared-nothing partitioned across N shards</td></tr> </tbody></table> Quick Reference: Sigils</h2> Sigil</th> Meaning</th> Example</th></tr></thead> @</code></td> Capability / pipeline binding</td> value @shared</code>, |> process AS @p</code></td></tr> !</code></td> Mutation suffix</td> FN increment!(...)</code></td></tr> |></code></td> Smooth operator (pipeline)</td> items |> WHERE _ > 5</code></td></tr> _</code></td> Pipeline element placeholder</td> |> SELECT _.name</code></td></tr> !T</code></td> Error union type</td> RETURNS !Float64</code></td></tr> ?T</code></td> Optional type</td> RETURNS ?User</code></td></tr> ~T</code></td> Promise / stream type</td> p: ~Int64 = BG { 42; }</code></td></tr> </tbody></table> Integer Overflow Operators</h2> CLEAR uses fixed-width integers (Int64, Int32, etc.) that can overflow. Three tiers of arithmetic operators control what happens on overflow: Default: +</code>, -</code>, *</code></h3> Panics on overflow in debug builds, wraps silently in release builds. This matches Rust's semantics - catches accidental overflow during development, zero overhead in production. a: Int64 = 9223372036854775807_i64; b = a + 1_i64; # debug: PANIC! release: wraps to -9223372036854775808 </code></pre> Wrapping: %+</code>, %-</code>, %*</code></h3> Always wraps on overflow in all build modes. Use for hash functions, random number generators, checksums, and any code that intentionally overflows. # LCG random number generator (intentional overflow) MUTABLE state: Int64 = seed; state = state %* 6364136223846793005_i64 %+ 1442695040888963407_i64; # Max + 1 wraps to min max: Int64 = 9223372036854775807_i64; min = max %+ 1_i64; # always -9223372036854775808, never panics </code></pre> Checked: !+</code>, !-</code>, !*</code></h3> Always panics on overflow in all build modes, including release. Use for financial calculations, safety-critical code, and anywhere overflow indicates a logic error. # Financial calculation: overflow means a bug, even in production balance = balance !+ deposit; # panics if result exceeds Int64 range total = quantity !* price; # panics on overflow, even in release </code></pre> Summary</h3> Operator</th> Debug build</th> Release build</th> Use case</th></tr></thead> +</code>, -</code>, *</code></td> panic</td> wrap</td> General arithmetic</td></tr> %+</code>, %-</code>, %*</code></td> wrap</td> wrap</td> Hash, RNG, checksum</td></tr> !+</code>, !-</code>, !*</code></td> panic</td> panic</td> Financial, safety</td></tr> </tbody></table> Float arithmetic (Float64</code>) is unaffected - IEEE 754 handles overflow via infinity/NaN. Concurrency 2026-03-26T00:00:00+00:00 CLEAR uses cooperative fibers for concurrency. Fibers are lightweight threads managed by the CLEAR runtime — each has its own stack (4-256KB) and runs until it yields. No OS threads are created per fiber; the scheduler multiplexes fibers onto a thread pool. CLEAR will support limited Finite State Machines in v0.2, and by v0.4 - full support for Finite State Machines is planned. </div> BG — Background Fibers</h2> Spawn a fiber that runs concurrently and returns a Promise: p = BG { expensive_computation(data); }; # ... do other work ... result = NEXT p; # block until the fiber finishes </code></pre> The last expression in the body becomes the promise's value. NEXT</code> consumes the promise and returns the value. By v0.4 Finite State Machines will be the default, and you will opt-in to stack-based fibers for re-entrant tasks or tasks that use FFI. </div> Modifiers</h3> Modifiers go inside the braces, before a -></code>: BG { @large:pinned -> heavy_work(); } </code></pre> Modifier</th> Effect</th></tr></thead> @micro</code></td> 4 KB fiber stack</td></tr> @standard</code></td> 16 KB fiber stack (default)</td></tr> @large</code></td> 64 KB fiber stack</td></tr> @xl</code></td> 256 KB fiber stack</td></tr> @service</code></td> OS thread (not a fiber). Full OS stack. For heavy non-cooperative compute.</td></tr> @pinned</code></td> Pin to local scheduler (no work stealing)</td></tr> @parallel</code></td> Distribute to least-loaded scheduler</td></tr> @arena</code></td> Thread-local arena allocation; only works with @pinned tasks</td></tr> </tbody></table> Combine with :</code> — @large:@arena</code> gives a large stack with arena allocation. By v0.2 @stateMachine</code> and @stack</code> will be options. The compiler will warn you when you should use a state machine, and block you from using it when it's not an option. @service — OS Thread Spawning</h3> @service</code> spawns a dedicated OS thread instead of a green fiber. Use it for heavy-compute tasks that won't cooperatively yield - the OS handles preemption. p = BG { @service -> trainModel(dataset); # runs on its own OS thread }; # fiber continues concurrently result = NEXT p; # blocks fiber until OS thread finishes </code></pre> The OS thread gets its own Runtime with a 64 KB frame arena. No scheduler is involved - the thread runs independently until completion, then signals the Promise. When to use @service vs fibers: Fibers (@standard</code>/@large</code>): I/O-bound, short compute bursts, cooperative yielding</li> @service</code>: CPU-bound, long-running, no yields, true OS-level parallelism</li> </ul> Captures</h3> BG blocks capture outer variables by value (moved, not borrowed): x = 42.0; p = BG { x + 1.0; }; # x is moved into the fiber # x is no longer usable here (affine ownership) </code></pre> THEN Chains</h3> Chain sequential steps inside a single fiber: result = BG { fetch("https://api.example.com/data") AS response THEN parse(response) AS parsed THEN transform(parsed); }; </code></pre> Each AS name</code> binds the result for subsequent steps. The last step's value becomes the promise result. This is equivalent to: result = BG { response = NEXT fetch("https://api.example.com/data"); parsed = NEXT parse(response); transform(parsed); }; </code></pre> THEN</code> is mostly useful for short-chained tasks, especially in a complex pipeline: result = BG { fetch("https://api.example.com/data1") AS r THEN parse(r), fetch("https://api.example.com/data2") AS r THEN parse(r) }; # result = ~T[] -> this is only valid when all T are the same. </code></pre> Error handling: use OR</code> before the AS</code> binding. If a step returns !T</code>, handle it inline: # ILLUSTRATIVE result = BG { fetch(url) OR RAISE # propagate error to caller AS response THEN parse(response) OR default_value AS parsed THEN transform(parsed); }; </code></pre> OR RAISE</code> - propagate the error (caller sees it via NEXT</code>)</li> OR value</code> - replace error with a fallback, chain continues</li> </ul> DO — Fork-Join</h2> Execute multiple branches concurrently, wait for all to complete: # ILLUSTRATIVE DO { update_database(record), send_notification(user), log_event(event) } # All three are done here. </code></pre> Branches are separated by commas. Each runs in its own fiber. The DO block waits for all branches before continuing. Returns Void</code>. Branch Modifiers</h3> Each branch can have its own modifiers: # ILLUSTRATIVE DO { @large -> heavy_computation(), @pinned -> cache_local_work() } </code></pre> NEXT — Promise Resolution</h2> Block until a promise resolves: p: ~Float64 = BG { 42.0; }; result: Float64 = NEXT p; </code></pre> Promise Type</th> NEXT returns</th> Behavior</th></tr></thead> ~T</code></td> T</code></td> Consumes the promise (one-shot)</td></tr> ~T @shared</code></td> T</code></td> Returns cached result (safe for multiple NEXT)</td></tr> ~T[?]</code></td> ?T</code></td> Returns next value or nil (open stream)</td></tr> ~T[INF]</code></td> T</code></td> Returns next value, never nil (infinite stream)</td></tr> </tbody></table> BG STREAM — Generators</h2> Spawn a fiber that yields values over time: # Open stream (finite) s: ~Float64[?] = BG STREAM { YIELD 1.0; YIELD 4.0; YIELD 9.0; }; v1 = NEXT s; # 1.0 v2 = NEXT s; # 4.0 v3 = NEXT s; # 9.0 v4 = NEXT s; # NIL (exhausted) # Infinite stream counter: ~Float64[INF] = BG STREAM { MUTABLE i = 0.0; WHILE TRUE DO YIELD i; i = i + 1.0; END }; v1 = NEXT counter; # 0.0 v2 = NEXT counter; # 1.0 (blocks until generator yields) </code></pre> CONCURRENT — Parallel Pipelines</h2> Apply pipeline operators in parallel with a persistent worker pool: # ILLUSTRATIVE results = items |> CONCURRENT(workers: 8) SELECT transform(_); filtered = items |> CONCURRENT(workers: 4) WHERE predicate(_); items |> CONCURRENT(workers: 2) EACH { _.value = 0.0; }; </code></pre> Options</h3> Option</th> Type</th> Default</th> Effect</th></tr></thead> workers</code></td> Number</td> 8</td> Number of persistent worker fibers</td></tr> batch</code></td> Number</td> 1</td> Items each worker pulls per fetch from the shared index (larger = less index contention)</td></tr> capacity</code></td> Number</td> auto (worker count rounded up to a power of 2, clamped 4-64)</td> Bounded channel ring-buffer size; only valid with a streaming / SHARD</code> / range source</td></tr> parallel</code></td> Bool</td> FALSE</td> TRUE = distribute workers across schedulers (multi-core)</td></tr> size</code></td> Identifier</td> STANDARD</td> Stack size: MICRO, STANDARD, LARGE, XL</td></tr> </tbody></table> Supported Operators</h3> CONCURRENT SELECT — parallel map. Returns transformed array (order preserved).</li> CONCURRENT WHERE — parallel filter. Returns matching elements (order preserved).</li> CONCURRENT EACH — parallel side effects. Returns Void.</li> </ul> Error Handling</h3> # Skip failed items results = items |> CONCURRENT(workers: 4) SELECT risky_fn(_) OR PRUNE; # Propagate first error results = items |> CONCURRENT(workers: 4) SELECT risky_fn(_) OR RAISE; </code></pre> How It Works</h3> CONCURRENT spawns N persistent worker fibers that pull items from a shared atomic index. Zero per-item allocation — workers reuse their stack and context across all items. This is fundamentally different from spawning one fiber per item (which is what BG does). results = items |> SELECT BG { risky_fn(_) OR RAISE }; # this is valid, but it would spawn N tasks, all at once. It is NOT recommended. </code></pre> Default (workers on local scheduler): CONCURRENT(workers: 8) SELECT transform(_) → 8 fibers on one thread, cooperative scheduling Multi-core (workers distributed): CONCURRENT(workers: 8, parallel: TRUE) SELECT transform(_) → 8 fibers spread across all schedulers </code></pre> Performance</h3> Pattern</th> Per-item cost</th> Use when</th></tr></thead> CONCURRENT(workers: N)</code></td> ~0 (atomic fetchAdd only)</td> Bulk processing, batch transforms</td></tr> Individual BG { }</code></td> ~60μs (GPA alloc)</td> Dynamic spawning, I/O-bound tasks</td></tr> </tbody></table> CONCURRENT is 30x faster than individual BG spawns for batch workloads.</li> For I/O-bound tasks (network requests, file reads), the 60μs BG spawn cost is negligible compared to I/O latency.</li> The big benefit is that workers: N</code> handles backpressure to avoid spawning more tasks than you can handle.</li> </ul> Multi-Threading</h2> CLEAR defaults to single-threaded scheduling. Set CLEAR_THREADS</code> to enable multi-core: CLEAR_THREADS=0 ./my_program # auto-detect CPU count CLEAR_THREADS=4 ./my_program # 4 scheduler threads CLEAR_THREADS=1 ./my_program # single-threaded (default) </code></pre> Each scheduler thread runs its own event loop with work stealing. Idle schedulers steal tasks from busy ones. @pinned</code> tasks are exempt from stealing. Safety Rules</h2> The compiler enforces these at compile time: Rule</th> Reason</th></tr></thead> @parallel</code> + @local</code> = error</td> @local has no synchronization</td></tr> @parallel</code> + @multiowned</code> = error</td> Rc is non-atomic; use @shared (Arc)</td></tr> @arena</code> + @parallel</code> = error</td> Arena memory is thread-local</td></tr> Non-@pinned BG capturing from @pinned scope = error</td> Thread-local memory would escape</td></tr> YIELD</code> outside BG STREAM = error</td> YIELD only valid in generators</td></tr> NEXT</code> on non-promise = error</td> Can only await promises and streams</td></tr> </tbody></table> When to Use What</h2> Goal</th> Construct</th></tr></thead> Process a batch of items</td> |> CONCURRENT(workers: N) SELECT/WHERE/EACH</code></td></tr> Fire off a background task</td> BG { work(); }</code></td></tr> Run independent tasks concurrently</td> DO { task1(), task2(), task3() }</code></td></tr> Generate values lazily</td> BG STREAM { YIELD ...; }</code></td></tr> Chain async steps</td> BG { step1() AS r THEN step2(r) }</code></td></tr> Handle requests (server)</td> BG { @pinned:@arena -> handle(conn); }</code></td></tr> </tbody></table> Known Limitations (v0.1)</h2> Dynamic spawn overhead: spawning many individual BG</code> tasks is competitive with Go at moderate thread counts (roughly parity at 8 threads on the 100K-spawn benchmark) but slower at high core counts — about 1.6x Go at small scale, up to ~3.5x at 32 threads. The remaining gap is idle schedulers spinning on the work-stealing deque instead of parking, not per-spawn allocation. Workaround: use CONCURRENT(workers: N)</code> for bulk workloads. Scheduler parking (the fix) is tracked for post-v0.1. No general bounded channel: CLEAR has Promises (one-shot) and Streams (unbounded). CONCURRENT</code> streaming uses an internal bounded ring-buffer channel for backpressure (tunable via the capacity:</code> option), but there is no general user-facing bounded producer/consumer channel primitive yet. Pipelines and Higher-Order Functions 2026-03-26T00:00:00+00:00 CLEAR's pipeline system lets you transform, filter, aggregate, and iterate collections using the smooth operator (|></code>). Every pipeline operator works on arrays, @list</code>, @pool</code>, sharded collections, @pool:soa</code>, streams ~T[]</code>, etc - the same syntax regardless of the underlying storage. _</code> is the placeholder value for the value being iterated over. scores |> WHERE _ <= 50 |> SUM _; users |> SELECT _.name |> DISTINCT _; entities |> EACH { _.health = _.health - 1.0; }; </code></pre> This document describes both the collection pipeline model and the stream/future pipeline surface. The full operator set is supported for finite streams (~T[]</code>, ~T[N]</code>); infinite streams (~T[INF]</code>) require LIMIT</code> to bound them and then support all non-materialization operators. The Smooth Operator (|></code>)</h2> |></code> pipes a value into a function or operator. It's CLEAR's equivalent of |></code> (Elixir) or .</code> method chaining (Ruby), but it also works with collection operators. # Pipe to a function: x |> f → f(x) result = data |> process |> validate |> format; # Pipe to an operator: list |> WHERE predicate alive = entities |> WHERE _.health > 0; </code></pre> Pipelines chain left to right. Each stage passes its result to the next. Named Pipeline Bindings (AS $v</code>)</h2> The AS $v</code> syntax binds the current pipeline element to a named reference that persists across subsequent stages. bill = users AS $u |> UNNEST $u.orders |> SUM _.price * $u.discount; </code></pre> Why bindings exist</h3> Without a binding, _</code> always refers to the current element - the item being iterated at the innermost level. After UNNEST</code>, _</code> becomes each inner element (an Order), and the outer element (the User) is no longer reachable. # Without AS $u: $u is not available inside the fold. # `_` after UNNEST is the Order, not the User. bill = users |> UNNEST _.orders |> SUM _.price; # can access order.price, but NOT user.discount </code></pre> AS $u</code> captures the outer element before the UNNEST</code> replaces _</code>, keeping it accessible: bill = users AS $u |> UNNEST $u.orders # $u = the User; _ = each Order |> SUM _.price * $u.discount; # cross-reference: order.price * user.discount </code></pre> The compiler fuses this into a single nested loop with no intermediate allocations. Scope of a binding</h3> A binding created with AS $v</code> is visible from the point of declaration to the end of the pipeline expression. It cannot be used after the pipeline terminates: bill = users AS $u |> UNNEST $u.orders # $u is in scope |> WHERE _.qty > 1 # $u still in scope |> SUM _.price * $u.discount; # $u still in scope # $u is not accessible here (out of scope after the pipeline ends) </code></pre> Multiple pipelines in the same function can each use their own $u</code> - bindings are scoped to the pipeline expression, not the function. Naming the inner element (AS $o)</h3> When you UNNEST and need a name for the inner element (instead of _</code>), use a second binding after the UNNEST expression: bill = users AS $u |> UNNEST $u.orders AS $o # $u = User, $o = Order |> SUM $o.price * $u.discount; </code></pre> AS $o</code> after the UNNEST expression binds the inner element. Both $u</code> and $o</code> are available in the fold. Supported combinations</h3> UNNEST binding chains - fold operators that work after AS $u |> UNNEST</code>: Fold</th> Example</th></tr></thead> SUM</td> |> SUM _.price * $u.discount</code></td></tr> COUNT</td> |> COUNT TRUE</code></td></tr> AVERAGE</td> |> AVERAGE _.price</code></td></tr> MIN</td> |> MIN _.price</code></td></tr> MAX</td> |> MAX _.price</code></td></tr> ANY</td> |> ANY _.price > 50.0</code></td></tr> ALL</td> |> ALL $u.discount > 0.0</code></td></tr> FIND</td> |> FIND _.price > 10.0</code> (returns ?ElemType</code>)</td></tr> </tbody></table> Intermediate WHERE</code> stages filter the inner elements before the fold: total = users AS $u |> UNNEST $u.orders |> WHERE _.qty > 1 # filter inner elements |> SUM _.price * $u.discount; </code></pre> SELECT</code> is not supported in UNNEST binding chains (the inner projection would lose the $u</code> context). Use field access in the fold expression instead. CONCURRENT binding - $u</code> is also accessible inside CONCURRENT</code> stages that operate directly on the bound list (without an intermediate UNNEST): Operator</th> Example</th></tr></thead> CONCURRENT SELECT</td> AS $u |> CONCURRENT SELECT $u.val * 2.0</code></td></tr> CONCURRENT SUM</td> AS $u |> CONCURRENT SUM $u.score</code></td></tr> CONCURRENT COUNT</td> AS $u |> CONCURRENT COUNT $u.active</code></td></tr> CONCURRENT MIN</td> AS $u |> CONCURRENT MIN $u.score</code></td></tr> CONCURRENT MAX</td> AS $u |> CONCURRENT MAX $u.score</code></td></tr> CONCURRENT AVERAGE</td> AS $u |> CONCURRENT AVERAGE $u.score</code></td></tr> CONCURRENT WHERE</td> AS $u |> CONCURRENT WHERE $u.score > 50.0</code></td></tr> </tbody></table> In all concurrent cases, $u</code> resolves to the item being processed by the current worker. The _</code> Variable</h2> Inside pipeline expressions, _</code> refers to the current element. For struct elements, access fields with _.fieldname</code>: # _ is the element itself (for scalar collections) nums: Float64[] = [1.0, 3.0, 7.0, 9.0]; big = nums |> WHERE _ > 5.0; ASSERT length(big) == 2, "WHERE filters by element value"; # _.field for struct collections users = [User{name: "alice"}, User{name: "bob"}]; names = users |> SELECT _.name; scores = [Score{value: 10.0}, Score{value: 20.0}]; total = scores |> SUM _.value; ASSERT total == 30.0, "SUM aggregates field values"; </code></pre> In EACH blocks, _</code> is mutable — you can assign to fields: pool |> EACH { _.health = _.health - damage; }; </code></pre> Operators</h2> Transform</h3> Operator</th> Syntax</th> Returns</th> Description</th></tr></thead> SELECT</td> list |> SELECT expr</code></td> ExprType[]</code></td> Project each element through an expression</td></tr> WHERE</td> list |> WHERE pred</code></td> ElemType[]</code></td> Keep elements matching a boolean predicate</td></tr> ORDER_BY</td> list |> ORDER_BY key</code></td> ElemType[]</code></td> Sort by key expression</td></tr> LIMIT</td> list |> LIMIT n</code></td> ElemType[]</code></td> First N elements</td></tr> SKIP</td> list |> SKIP n</code></td> ElemType[]</code></td> Drop first N elements, return rest</td></tr> DISTINCT</td> list |> DISTINCT key</code></td> ElemType[]</code></td> Unique by key (first occurrence wins)</td></tr> UNNEST</td> list |> UNNEST expr</code></td> InnerType[]</code></td> Flatten nested arrays (flatmap)</td></tr> INDEX</td> list |> INDEX key</code></td> HashMap<ElemType[]></code></td> Group into a hashmap by key</td></tr> </tbody></table> SELECT accepts any expression, including struct literals. This lets you project into a different struct type in one step: STRUCT Raw { id: Int64, score: Float64 } STRUCT Summary { key: Int64, normalized: Float64 } raws: Raw[] = [Raw{ id: 1, score: 100.0 }, Raw{ id: 2, score: 200.0 }]; summaries = raws |> SELECT Summary{ key: _.id, normalized: _.score / 100.0 }; ASSERT summaries[0].key == 1, "id preserved"; ASSERT summaries[1].normalized == 2.0, "score normalized"; </code></pre> The result type is inferred from the expression - Summary[]</code> above, not Raw[]</code>. Pipeline fusion with SELECT T{}: SELECT composes with WHERE and aggregates in a single fused loop - no intermediate list allocation: # WHERE before SELECT: filter on the raw element (efficient) high = raws |> WHERE _.score > 75.0 |> SELECT Summary{ key: _.id, normalized: _.score / 100.0 }; # SELECT before aggregate: project then sum/min/max a field total = raws |> SELECT Summary{ key: _.id, normalized: _.score / 100.0 } |> SUM _.normalized; # SELECT before WHERE: build struct first, then filter on a struct field (less efficient) norm_high = raws |> SELECT Summary{ key: _.id, normalized: _.score / 100.0 } |> WHERE _.normalized > 0.75; </code></pre> When SELECT precedes a fold or WHERE, the compiler binds the projected value to a temp variable per iteration. This avoids the Zig restriction on struct literals in arithmetic/boolean expression positions, so the generated code is always valid without changing semantics. Aggregate</h3> Operator</th> Syntax</th> Returns</th> Empty list</th></tr></thead> SUM</td> list |> SUM expr</code></td> Int64 / UInt64 / Float32 / Float64</td> 0</td></tr> AVERAGE</td> list |> AVERAGE expr</code></td> Float64</code></td> 0</td></tr> MIN</td> list |> MIN expr</code></td> matches expr type</td> panics</td></tr> MAX</td> list |> MAX expr</code></td> matches expr type</td> panics</td></tr> REDUCE</td> list |> REDUCE(init) expr</code></td> type of init</td> init</td></tr> </tbody></table> The return type is driven by the expression type. SUM widens small integers to Int64</code>/UInt64</code>; floats stay at their original width (Float32</code> stays Float32</code>). AVERAGE always returns Float64</code>. MIN and MAX preserve the exact expression type. REDUCE is the general fold - acc</code> is the mutable accumulator, _</code> is the current element: nums: Float64[] = [2.0, 3.0, 4.0]; product = nums |> REDUCE(1.0) acc * _; ASSERT product == 24.0, "REDUCE multiplies 2*3*4"; </code></pre> Query</h3> Operator</th> Syntax</th> Returns</th> Description</th></tr></thead> COUNT</td> list |> COUNT pred</code></td> Int64</code></td> Number of matches</td></tr> ANY</td> list |> ANY pred</code></td> Bool</code></td> True if any match (short-circuits)</td></tr> ALL</td> list |> ALL pred</code></td> Bool</code></td> True if all match (short-circuits)</td></tr> FIND</td> list |> FIND pred</code></td> ?ElemType</code></td> First match or null</td></tr> </tbody></table> Side Effects</h3> Operator</th> Syntax</th> Returns</th> Description</th></tr></thead> EACH</td> list |> EACH { body }</code></td> Void</code></td> Iterate with mutable _</code>; side-effect only</td></tr> TAP</td> list |> TAP { body }</code></td> ElemType[]</code></td> Observe each element (read-only _</code>), pass collection through</td></tr> </tbody></table> EACH is the only operator where _</code> is mutable. Use it for in-place updates: entities |> EACH { _.x = _.x + _.vx; _.y = _.y + _.vy; }; </code></pre> TAP (Debugging / Observation)</h3> TAP runs a body for each element but passes the collection through unchanged. Unlike EACH, _</code> is read-only and TAP returns the original collection: result = scores |> WHERE _.points > 100 |> TAP { print("score: ${_.points.toString()}"); } |> SUM _.points; </code></pre> WINDOW (Sliding Window)</h3> WINDOW produces a sliding window of size N over the collection. _</code> inside the body is the sub-slice (a Float64[]</code> or ElemType[]</code> of length N). The result is an array of the body expression evaluated per window. data: Float64[] = [1.0, 2.0, 3.0, 4.0, 5.0]; # Each window is a 3-element slice; body projects to window length lengths = data |> WINDOW(3) _.length(); ASSERT lengths.length() == 3, "5 elements, window 3 -> 3 windows"; # Access individual elements in the window firsts = data |> WINDOW(2) _[0]; ASSERT firsts[0] == 1.0, "first element of first window"; </code></pre> Number of result windows = max(0, len - size + 1)</code>. A window larger than the list produces an empty result. WINDOW (Batch / Tumbling Window)</h3> WINDOW(size: N)</code>, WINDOW(time: 'Xms')</code>, or WINDOW(size: N, time: 'Xms')</code> produces non-overlapping batches. _</code> inside the body is a T[]</code> batch. The result is a heap-allocated list of the body expression evaluated per batch. Flush conditions (checked after every item): size: N</code> - flush when the batch accumulates N items</li> time: 'Xms'</code> - flush when elapsed time since the first item in the batch >= timeout; time units: ms</code>, s</code>, min</code>, h</code></li> Both specified - flush on whichever fires first (first-of-either)</li> </ul> A partial batch at the end is always included. Works on all source types: arrays, ~T[]</code>, ~T[N]</code>, and ~T[INF]</code>. FN sumBatch(batch: Int64[]) RETURNS Int64 -> MUTABLE s: Int64 = 0; batch |> EACH { s = s + _; }; RETURN s; END # Array source, size-only: [1..7] -> [1,2,3], [4,5,6], [7] arr: Int64[] = [1, 2, 3, 4, 5, 6, 7]; sums = arr |> WINDOW(size: 3) sumBatch(_); ASSERT sums.length() == 3; # 3 batches ASSERT sums[2] == 7; # partial final batch # Open stream, size + time (first-of-either) gen: ~?Int64[] = BG STREAM { MUTABLE i: Int64 = 1; WHILE i <= 10 DO YIELD i; i = i + 1; END }; sums2 = gen |> WINDOW(size: 4, time: "500ms") sumBatch(_); ASSERT sums2.length() == 3; # [1-4], [5-8], [9-10] # Time-only: entire array arrives fast, all items in one final batch arr2: Int64[] = [100, 200, 300]; lens = arr2 |> WINDOW(time: "1s") _.length(); ASSERT lens[0] == 3; # one batch of 3 </code></pre> Note: The sliding-window form WINDOW(N)</code> (positional integer, no named params) is the existing collection-only operator above. The named-param form WINDOW(size:, time:)</code> is the new batching operator and works on both collections and streams. </blockquote> JOIN (Left Outer Join)</h3> JOIN performs a left outer join between two collections using a two-parameter lambda predicate. Every element of the left collection appears in the result exactly once; the right field is NIL</code> when no right element matches. STRUCT User { id: Int64, name: String } STRUCT Order { userId: Int64, amount: Float64 } results = users |> JOIN(orders) %(u, o) -> u.id == o.userId; # results: JoinResult_User_Order[] # results[i].left -- the User # results[i].right -- ?Order (NIL if no match) </code></pre> The result element type is an anonymous struct { left: L, right: ?R }</code>. The lambda must take exactly two parameters (left element, right element) and return Bool. Stream and Future Compatibility</h2> Pipelines over futures/streams are currently supported in a narrower subset than pipelines over collections. Stream kinds</h3> Type</th> Meaning</th> NEXT</code> result</th> Pipeline support</th></tr></thead> ~T</code></td> Single future value</td> T</code></td> Not a pipeline source</td></tr> ~?T[]</code></td> Open stream</td> ?T</code></td> Not a pipeline source yet</td></tr> ~T[]</code></td> Finite dynamic stream</td> ?T</code></td> Full non-concurrent operator set (see table)</td></tr> ~T[N]</code></td> Finite bounded stream</td> ?T</code></td> Full non-concurrent operator set plus CONCURRENT EACH/SELECT/WHERE</code></td></tr> ~T[INF]</code></td> Infinite stream</td> T</code></td> Fusible stages + all terminals via LIMIT</code></td></tr> </tbody></table> Operator support matrix</h3> The columns are the three pipeline-capable stream types. "Stage" operators filter or transform in a fused while loop without materializing; "terminal" operators consume the stream and produce a scalar or collection result. Stages (fusible, zero intermediate allocations)</h4> Operator</th> ~T[]</code></th> ~T[N]</code></th> ~T[INF]</code></th></tr></thead> WHERE</code></td> yes</td> yes</td> yes (LIMIT must appear in chain)</td></tr> SELECT</code></td> yes</td> yes</td> yes (LIMIT must appear in chain)</td></tr> SKIP</code></td> yes</td> yes</td> yes (LIMIT must appear in chain)</td></tr> TAKE_WHILE</code></td> yes</td> yes</td> yes (LIMIT must appear in chain)</td></tr> LIMIT</code></td> yes</td> yes</td> yes - converts stream to T[]</code></td></tr> TAP</code></td> yes</td> yes</td> not yet</td></tr> </tbody></table> LIMIT</code> can appear anywhere in the chain relative to other fusible stages. The compiler fuses the entire chain into a single while loop - counter |> WHERE _ > 0 |> LIMIT 5 |> EACH</code> and counter |> LIMIT 5 |> WHERE _ > 0 |> EACH</code> both work; they differ only in whether LIMIT counts pre- or post-filter items. Fold terminals (produce a scalar or optional value)</h4> Operator</th> ~T[]</code></th> ~T[N]</code></th> ~T[INF]</code></th></tr></thead> EACH</code></td> yes</td> yes</td> via LIMIT</td></tr> SUM</code></td> yes</td> yes</td> via LIMIT</td></tr> COUNT</code></td> yes</td> yes</td> via LIMIT</td></tr> AVERAGE</code></td> yes</td> yes</td> via LIMIT</td></tr> MIN</code></td> yes</td> yes</td> via LIMIT</td></tr> MAX</code></td> yes</td> yes</td> via LIMIT</td></tr> ANY</code></td> yes</td> yes</td> via LIMIT</td></tr> ALL</code></td> yes</td> yes</td> via LIMIT</td></tr> FIND</code></td> yes</td> yes</td> via LIMIT</td></tr> REDUCE</code></td> yes</td> yes</td> via LIMIT</td></tr> </tbody></table> "via LIMIT" means LIMIT must appear earlier in the same pipeline chain. LIMIT converts ~T[INF]</code> to T[]</code> (a regular list); the fold terminal then operates on that list. Example: counter |> WHERE _ > 0 |> LIMIT 5 |> SUM _</code>. Materialization terminals (produce a new collection)</h4> These operators consume the stream and produce a new heap-allocated collection. DISTINCT</code> and INDEX</code> are fully supported for all stream types; others require materializing first with .toList()</code>. Operator</th> ~T[]</code></th> ~T[N]</code></th> ~T[INF]</code></th></tr></thead> DISTINCT</code></td> yes - returns T[]@set</code></td> yes - returns T[]@set</code></td> via LIMIT - returns T[]@set</code></td></tr> INDEX</code></td> yes</td> yes</td> via LIMIT</td></tr> ORDER_BY</code></td> not yet</td> not yet</td> not yet</td></tr> UNNEST</code></td> not yet</td> not yet</td> not yet</td></tr> WINDOW(N)</code> (sliding)</td> not yet</td> not yet</td> not yet</td></tr> WINDOW(size:, time:)</code> (batch)</td> yes</td> yes</td> yes</td></tr> JOIN</code></td> not yet</td> not yet</td> not yet</td></tr> </tbody></table> "via LIMIT" means LIMIT</code> must appear earlier in the chain (same rule as fold terminals). DISTINCT</code> returns T[]@set</code> (a Set), supporting .count()</code> and .contains?()</code>. If you need ORDER_BY</code>, UNNEST</code>, sliding WINDOW(N)</code>, or JOIN</code> on a stream, materialize first. The batching WINDOW(size:, time:)</code> operator works directly on streams without materialization. s: ~Int64[] = 0 ..< 10; vals = s.toList(); total = vals |> WHERE _ > 3 |> SUM _; </code></pre> Concurrent operators</h4> Operator</th> ~T[]</code></th> ~T[N]</code></th> ~T[INF]</code></th></tr></thead> CONCURRENT EACH</code></td> not yet</td> yes</td> not yet</td></tr> CONCURRENT SELECT</code></td> not yet</td> yes</td> not yet</td></tr> CONCURRENT WHERE</code></td> not yet</td> yes</td> not yet</td></tr> </tbody></table> ~T[N]</code> concurrent pipelines are native: they consume promise slots directly without materializing through .toList()</code>, and lower through MIR-visible builtin helpers. Example: nums: ~Float64[4] = [BG { 1.0; }, BG { 2.0; }, BG { 3.0; }, BG { 4.0; }]; doubled = nums |> CONCURRENT(workers: 2) SELECT _ * 2.0; </code></pre> Direct range expressions still use the non-concurrent path unless first bound as ~T[N]</code>. Open streams</h3> Open streams (~?T[]</code>) are still NEXT</code>-driven only: gen: ~?Int64[] = BG STREAM { YIELD 1; YIELD 2; }; v1 = NEXT gen; v2 = NEXT gen; v3 = NEXT gen; # NIL </code></pre> SKIP and LIMIT (Pagination)</h3> SKIP and LIMIT are complementary: SKIP drops the first N elements, LIMIT takes the first N. # Pagination: page 3, 10 items per page page = items |> SKIP 20 |> LIMIT 10; # Skip header row, process the rest data = rows |> SKIP 1 |> SELECT parseRow; </code></pre> Chaining</h2> Operators compose naturally: # Filter, sort, take top 3 leaderboard = scores |> WHERE _.points > 100 |> ORDER_BY _.points |> LIMIT 3; # Count active users with high scores n = users |> WHERE _.active == TRUE |> COUNT _.score > 1000; </code></pre> Collection Compatibility</h2> Every operator works on every collection type: # Array nums: Float64[] = [1, 2, 3]; total = nums |> SUM _; # List MUTABLE data = List[]; avg = data |> AVERAGE _.value; # Pool MUTABLE pool: Entity[1000]@pool = []; alive = pool |> WHERE _.health > 0; # Pool with SOA (field-slice iteration — cache-optimal) MUTABLE soa_pool: Entity[1000]@pool:soa = []; total_hp = soa_pool |> SUM _.health; # iterates only the health array # List with SOA MUTABLE soa_list: Entity[]@list:soa = []; avg = soa_list |> AVERAGE _.health; # contiguous f64 slice # Sharded (parallel EACH via DO blocks) MUTABLE sharded: Entity[10000]@pool:sharded(4) = []; sharded |> EACH { _.processed = TRUE; }; </code></pre> Loop Fusion</h2> The compiler automatically fuses chains of WHERE and SELECT stages ending in a fold (SUM, REDUCE, AVERAGE, MIN, MAX, COUNT, ANY, ALL, FIND) into a single loop with zero intermediate allocations. # Written as 3 stages: result = data |> WHERE _ > 500.0 |> SELECT _ * _ |> SUM _; # Compiled as a single loop (no intermediate arrays): # for (data) |it| { if (it > 500) { sum += it * it; } } </code></pre> This eliminates the allocation and iteration overhead of intermediate lists. Stages that require materialization (ORDER_BY, DISTINCT, INDEX) break the fusion chain - operations before them are fused separately. SOA Optimization</h2> When a @pool:soa</code> is used in a pipeline, the compiler rewrites field accesses to iterate directly over contiguous field arrays instead of striding over whole structs. This happens automatically for all operators — no syntax change needed. STRUCT Entity { x: Float64, y: Float64, vx: Float64, vy: Float64, health: Float64 } MUTABLE pool: Entity[10000]@pool:soa = []; # SUM _.health iterates only the health array (contiguous f64[]). # Without :soa, it would load all 5 fields per element. total = pool |> SUM _.health; </code></pre> For WHERE and FIND, the predicate uses field-slice access (fast), and the struct is reassembled only for matching elements. The compiler warns when SOA would help: NOTE: Pipeline accesses 1 of 5 fields (health). Consider @soa for better cache performance on 'Entity'. </code></pre> Concurrency</h2> The CONCURRENT</code> modifier currently parallelizes collection pipelines for SELECT</code>, WHERE</code>, and EACH</code>: MUTABLE data: Score[10000]@pool:sharded(4) = []; # Parallel WHERE: one fiber per shard results = data |> CONCURRENT WHERE dbFetch(_.id).val > threshold; # Process in parallel results = data |> CONCURRENT(parallel: TRUE) SELECT dbFetch(_.id).name; </code></pre> Options: workers: N</code> (fiber worker size)</li> pin: TRUE</code> (pin to cores)</li> size: MICRO|STANDARD|LARGE|XL</code> (stack size).</li> parallel</code>: TRUE` (run on multiple cores)</li> </ul> CONCURRENT</code> compatibility</h3> Source kind</th> CONCURRENT SELECT</code></th> CONCURRENT WHERE</code></th> CONCURRENT EACH</code></th></tr></thead> Arrays / @list</code> / @pool</code> / @pool:soa</code></td> Yes</td> Yes</td> Yes</td></tr> @sharded(...)</code> collections</td> Yes</td> Yes</td> Yes</td></tr> Finite streams ~T[]</code></td> Not yet</td> Not yet</td> Not yet</td></tr> Bounded streams ~T[N]</code></td> Yes</td> Yes</td> Yes</td></tr> Open streams ~?T[]</code></td> Not yet</td> Not yet</td> Not yet</td></tr> Infinite streams ~T[INF]</code></td> Not yet</td> Not yet</td> Not yet</td></tr> </tbody></table> So today: CONCURRENT</code> is for collection sources.</li> finite streams can participate in non-concurrent fused pipelines for the supported operators above.</li> stream CONCURRENT</code> will come later once the MIR-safe native lowering is in place.</li> </ul> CLEAR Collections: Array, List, Pool 2026-03-25T00:00:00+00:00 CLEAR has three collection types, each designed for a different access pattern. The choice is a capability annotation — the element type and functions that operate on it stay the same regardless of which collection you use. Quick Reference</h2> Collection</th> Zig Type</th> Access</th> Growth</th> Safety</th> Use when</th></tr></thead> T[N]</code></td> [N]T</code></td> Index O(1)</td> Fixed</td> Bounds-checked</td> Size known at compile time</td></tr> T[]@list</code></td> ArrayListUnmanaged(T)</code></td> Index O(1), append O(1)*</td> Dynamic</td> Bounds-checked</td> Size unknown, sequential access</td></tr> T[N]@pool</code></td> Pool(T)</code></td> Handle O(1)</td> Fixed</td> Generational handles</td> Frequent insert/remove, stable references</td></tr> </tbody></table> * Amortized O(1) — occasional reallocation when capacity is exceeded. Arrays — T[N]</code></h2> Fixed-size, stack-allocated. The size is part of the type. MUTABLE scores: Int64[5] = [10, 20, 30, 40, 50]; x = scores[2]; # 30 scores[0] = 99; # mutation via index ASSERT x == 30, "index access"; ASSERT scores[0] == 99, "mutation via index"; </code></pre> When to use: Size is known at compile time. No insertions or removals. The fastest option — no heap allocation, no bounds-growth overhead. Limitations: Cannot append, remove, or resize. Index out of bounds is a runtime panic. Lists — @list</code></h2> Dynamic-size, heap-allocated. Backed by std.ArrayListUnmanaged(T)</code>. MUTABLE users = List[]; users.append("Alice"); users.append("Bob"); n = users.length(); # 2 name = users[0]; # "Alice" </code></pre> When to use: Size is unknown or grows over time. Elements accessed by index. The general-purpose dynamic array — equivalent to Vec<T></code> in Rust or []T</code> in Go. Limitations: Removing from the middle is O(N) (shift elements). Handles/pointers to elements are invalidated on reallocation. Not suitable for frequent insert/remove of interior elements. Variant — sharded list: T[]@list:sharded(N)</code> splits the list into N shards for parallel pipeline operations (|> EACH</code>, |> SUM</code>). Each shard is an independent list. Round-robin distribution on append. Lazy inference: List[]</code> creates an untyped list. The element type is inferred from the first append</code>: MUTABLE items = List[]; items.append(42.0); # type inferred as Float64[]@list </code></pre> Pools — @pool</code></h2> Handle-based, pre-allocated with fixed capacity. Backed by Pool(T)</code> with generational handles for ABA safety. The capacity is specified at declaration time and all slots are allocated up front — no dynamic resizing, no reallocation. Insert and remove are O(1) via an internal free stack. STRUCT Enemy { hp: Int64, name: String } MUTABLE enemies: Enemy[1000]@pool = []; id1: Id<Enemy> = enemies.insert(Enemy{ hp: 100, name: "Goblin" }); id2: Id<Enemy> = enemies.insert(Enemy{ hp: 200, name: "Dragon" }); # Access via handle — returns ?T (optional). Must use OR to unwrap: goblin = enemies.get(id1) OR Enemy{ hp: 0, name: "none" }; # Or via [] syntax (equivalent to .get): dragon = enemies[id2] OR Enemy{ hp: 0, name: "none" }; # Remove: slot is freed, generation increments enemies.remove(id1); # Stale handle returns null — OR provides a safe fallback: gone = enemies.get(id1) OR Enemy{ hp: 0, name: "removed" }; </code></pre> Important: every pool access returns ?T</code> (optional). Like @writeLocked</code>, using a pool is not a one-line optimization — it changes how you access data. Every pool.get(id)</code> or pool[id]</code> requires OR default</code> to handle the case where the handle is stale or the slot was removed. This is by design: the pool guarantees you never read invalid data, but you must handle the "not found" case explicitly. How Generational Handles Work</h3> Each pool slot stores a generation counter alongside the data: Slot: { generation: u32, alive: bool, value: T } Handle (Id<T>): u64 = [generation: upper 32 bits][index: lower 32 bits] </code></pre> When you insert</code>, the handle encodes the current generation and slot index. When you remove</code>, the slot's generation increments. When you get</code> with a stale handle, the generation in the handle doesn't match the slot's generation — the pool returns null instead of the wrong data. This prevents the classic use-after-free / type confusion bug: 1. Insert User at slot 0, generation 0 → handle = 0x00000000_00000000 2. Remove User → slot 0 generation becomes 1 3. Insert Projectile at slot 0, generation 1 → handle = 0x00000001_00000000 4. Try to read old User handle (gen 0) → GENERATION MISMATCH → null </code></pre> No garbage collector. No reference counting. Just a 4-byte integer comparison per access. When to use pools</h3> Game entities (ECS): enemies, projectiles, particles — frequently spawned and destroyed</li> Connection pools: TCP clients, database handles — allocated and returned</li> Object caches: LRU-style caches where entries are evicted and reused</li> Any workload with frequent insert/remove where index-based access would leave holes</li> </ul> Pools vs Lists</h3> Operation</th> List T[]@list</code></th> Pool T[N]@pool</code></th></tr></thead> Append/Insert</td> O(1) amortized</td> O(1) via free stack</td></tr> Access</td> list[i]</code> → T</code> (direct)</td> pool[id]</code> → ?T</code> (requires OR</code>)</td></tr> Remove from middle</td> O(N) shift</td> O(1) mark-dead</td></tr> Stable references</td> No (realloc invalidates)</td> Yes (handles survive realloc)</td></tr> Capacity</td> Dynamic (grows on demand)</td> Fixed (pre-allocated at declaration)</td></tr> Memory after remove</td> Compacted</td> Holes (reused on next insert via free stack)</td></tr> Safety after remove</td> Index may now point to different element</td> Handle returns null (generational)</td></tr> Ergonomic cost</td> Direct access — no unwrapping</td> Every access needs OR</code> fallback</td></tr> </tbody></table> Decision Tree</h2> Is the size fixed at compile time? ├── Yes → T[N] (array) └── No ├── Access pattern? │ ├── Sequential / index-based → T[]@list (list) │ └── Handle-based / frequent insert+remove → T[N]@pool (pool) └── Need stable references across insert/remove? ├── Yes → T[N]@pool (generational handles survive mutations) └── No → T[]@list (simpler, more cache-friendly for iteration) </code></pre> Note: pools require a compile-time capacity N</code>. All slots are pre-allocated up front, giving O(1) insert/remove via an internal free stack with no runtime reallocation. Choose a capacity that covers your expected maximum — the pool will panic if you exceed it. Hash Maps — HashMap<V></code> and HashMap<K, V></code></h2> Key-value maps. String-keyed by default, numeric-keyed with explicit HashMap<K, V></code>. MUTABLE scores: HashMap<Int64> = {}; # String → Int64 scores["alice"] = 100_i64; scores["bob"] = 200_i64; val = scores["alice"]; # 100 scores.delete("bob"); scores.contains?("alice"); # TRUE scores.count(); # 1 </code></pre> When to use: Lookup by key. The general-purpose associative container. SOA Layout — Cache-Optimal Iteration</h2> By default, collections store elements as Array of Structures (AOS): each element is a contiguous block of all its fields. When a pipeline accesses only a subset of fields, the CPU loads entire cache lines but uses only a fraction — wasting memory bandwidth. The :soa</code> capability switches to Structure of Arrays layout: each field gets its own contiguous array. A pipeline that touches one field iterates a dense, cache-friendly slice. STRUCT Entity { x: Float64, y: Float64, vx: Float64, vy: Float64, health: Float64, mana: Float64, name: String, level: Float64 } # AOS (default): each entity is 8 fields × 8 bytes = 64 bytes per cache line. # SUM _.health loads all 8 fields but uses only 1 — 87.5% wasted bandwidth. MUTABLE pool: Entity[10000]@pool = []; # SOA: health values are stored in a contiguous f64 array. # SUM _.health touches only that array — zero waste. MUTABLE pool: Entity[10000]@pool:soa = []; </code></pre> The compiler detects when SOA would help and suggests it: NOTE: Pipeline accesses 1 of 8 fields (health). Consider @soa for better cache performance on 'Entity'. </code></pre> When :soa</code> helps most: Pipelines that touch < 50% of fields (SUM, MIN, MAX, AVERAGE on one field)</li> Large structs (8+ fields)</li> High-volume iteration (thousands of elements)</li> </ul> When :soa</code> doesn't help: Small structs (< 4 fields) — cache lines already fit the whole struct</li> EACH that reads/writes most fields — no bandwidth savings</li> Random-access by handle (pool.get(id)</code>) — must reassemble the struct</li> </ul> How it works under the hood: All pipeline operators (SUM, MIN, MAX, AVERAGE, COUNT, ANY, ALL, WHERE, FIND, SELECT) automatically use field-slice iteration on :soa</code> pools. The compiler rewrites _.health</code> to data.items(.health)[i]</code> — a direct index into the contiguous field array. No materialization, no struct reassembly for the common case. For operators that produce struct output (WHERE, FIND), structs are reassembled only for matching elements — the predicate still uses field-slice access, so most iterations touch only the predicate field. :soa</code> works on both @pool</code> and @list</code>: MUTABLE pool: Entity[10000]@pool:soa = []; # SOA pool (generational handles) MUTABLE items: Entity[]@list:soa = []; # SOA list (dense, indexed) </code></pre> Same API as their non-SOA counterparts. The difference is invisible except in pipeline performance. FFI restriction: @soa</code> collections cannot be passed to EXTERN FN</code> — SOA memory layout is incompatible with the C ABI. Materialize to a regular collection first if you need FFI. Sharding — True Shared-Nothing Parallel Collections</h2> Lists, pools, and hash maps all support sharding for parallel access: MUTABLE data: Float64[]@list:sharded(4) = []; MUTABLE entities: Enemy[10000]@pool:sharded(4) = []; MUTABLE counts: HashMap<Int64>@sharded(4) = {}; </code></pre> Sharding splits the collection into N independent partitions. Each shard is a complete, independent data structure — no shared state, no locks, no atomic operations between shards. How it works: Lists: Round-robin distribution on append</code>. length()</code> sums across shards.</li> Pools: Round-robin distribution on insert</code>. Handle encodes shard index in upper bits.</li> Hash maps: Key-based routing via hash(key) % N</code>. Same key always maps to same shard.</li> </ul> Pipeline operations (|> EACH</code>, |> SUM</code>, |> WHERE</code>, etc.) process each shard in parallel via DO blocks — one fiber per shard, no contention between them. The DragonflyDB Model — Each Thread Owns Its Partition</h3> @sharded(N)</code> uses the same architecture as DragonflyDB: each scheduler thread owns a subset of shards, and fibers pinned to that scheduler access those shards with zero locks, zero atomics, and zero synchronization. Shard ownership: Shard i</code> is owned by scheduler i % S</code> (where S = number of schedulers). With 8 shards and 2 schedulers, each scheduler owns 4 shards. Hot path (shard owned by the current scheduler): Direct access. No locks, no atomics. Cooperative scheduling ensures only one fiber runs at a time per scheduler, so there's no data race. Cold path (shard owned by another scheduler): The operation is transparently routed to the owning scheduler via SPSC ring buffers. The calling fiber yields until the owning scheduler executes the operation inline and signals completion via an atomic done flag. Correct, but slower — design your workload so each fiber processes ONLY its own partition. SHARD</code> Pipeline — True Shared-Nothing Workloads</h3> The SHARD</code> pipeline operator partitions work so each fiber owns its shard exclusively: MUTABLE map: HashMap<String>@sharded(8) = {}; n = 1000000; # SHARD routes each key to the scheduler that owns its shard. # CONCURRENT EACH runs one fiber per shard — zero locks, zero routing. (0..<n) |> SHARD("key:${toString(_)}", map) |> CONCURRENT EACH { map[_] = "value"; }; </code></pre> How it works: SHARD hashes each key, appends it to the owning shard's queue</li> One fiber per shard is pinned to the owning scheduler</li> Each fiber processes only its own queue — no cross-shard access, no locks</li> The result is true DragonflyDB-style shared-nothing: N partitions, N fibers, zero synchronization</li> </ol> When to use @sharded(N)</code> vs @sharded(N):locked</code>: @sharded(N)</code> + CONCURRENT EACH</code>: each fiber owns its partition. Maximum throughput.</li> @sharded(N):locked</code>: any fiber can access any key. RwLock per shard. Use when the workload can't be partitioned (e.g., random key distribution across all fibers).</li> </ul> One-Line Optimization</h3> @sharded(N)</code> is a one-line declaration change. Functions don't need to know — a function that takes HashMap<String></code> seamlessly accepts HashMap<String>@sharded(8)</code>: # One-line change: add @sharded(8) to the declaration MUTABLE map: HashMap<String>@sharded(8) = {}; # Functions: no @sharded needed in parameter types FN doWork!(MUTABLE map: HashMap<String>, key: String) RETURNS !Void -> map[key] = "value"; END # BG blocks: auto-pinned when they capture a @sharded map BG { doWork!(map, "hello"); } # [Note] BG block auto-pinned — captures shared/locked resource. </code></pre> The compiler handles everything: Unified API: All HashMap variants (plain, @sharded</code>, @sharded:locked</code>) have the same .put()</code>/.get()</code> interface. Functions use Zig anytype</code> generics to accept any variant.</li> Auto-pinning: BG blocks that capture a @sharded</code> map are automatically pinned to a scheduler. No @pinned</code> annotation needed.</li> Fiber distribution: Pinned BG fibers are distributed round-robin across schedulers via spawnPinned()</code>, so each scheduler gets work.</li> </ul> Sharding Variants</h3> Variant</th> Model</th> Locking</th> Use when</th></tr></thead> @sharded(N)</code></td> Shared-nothing</td> None (fiber-pinned)</td> Partitioned workloads (CONCURRENT EACH)</td></tr> @sharded(N):locked</code></td> RwLock per shard</td> Per-shard RwLock</td> Cross-shard access from any fiber</td></tr> @sharded(N):writeLocked</code></td> RwLock per shard</td> Per-shard RwLock</td> Same as :locked (alias)</td></tr> </tbody></table> When You Need Shared Mutable Maps</h3> For rare cases where multiple schedulers must read/write the same keys concurrently (e.g., a global session registry), use @shared:writeLocked</code>: MUTABLE sessions: HashMap<Session> @shared:writeLocked = {}; # Arc<RwLock<HashMap>>: readers are parallel, writers are serialized </code></pre> This is CLEAR's escape hatch — it works like Java's ConcurrentHashMap</code> but with explicit intent. The capability annotation makes the cost visible at the declaration site, and the compiler's capability audit will tell you if you're paying for it unnecessarily. Hashing — Secure by Default</h2> CLEAR uses seeded Wyhash as the default hash function for all hash maps. The seed is randomized per process, which prevents Hash DoS attacks (where an attacker crafts keys that all collide, turning O(1) lookups into O(n) traversals). This follows CLEAR's "Correct > Scalable > Fast" ordering: the default is safe against adversarial input, with near-zero overhead compared to unsalted hashes. Future: @fastTrusted</code> Capability</h3> For performance-critical internal infrastructure where keys are not attacker-controlled (e.g., internal service meshes, compiler symbol tables, game engine registries), CLEAR plans to offer an opt-in unsalted hash: # Planned syntax (not yet implemented): MUTABLE symbols: HashMap<SymbolInfo>@fastTrusted = {}; MUTABLE fast_sharded: HashMap<Int64>@sharded(8):fastTrusted = {}; </code></pre> @fastTrusted</code> would switch to unsalted FNV-1a or raw Wyhash — faster, but vulnerable to Hash DoS if keys come from untrusted input. The name is deliberate: it forces the developer to declare "I trust this data source." Why not yet: The default seeded Wyhash is fast enough for v1 (within ~5% of unsalted FNV). The capability will be added when real-world profiling shows the hash function is the bottleneck, not before. Premature escape hatches encourage premature optimization. Shared Data in CLEAR 2026-03-25T00:00:00+00:00 CLEAR separates: what data is (Types) from</li> how it's accessed (Gates & Boundaries) from</li> the strategy it uses (Capabilities) from</li> when it's available (Tense: ~T</code>)</li> </ol> Types, Capabilities, and Execution Boundaries</h2> Rust is not inherently complicated, nor is its type system much more complex than C++'s. The actual friction comes from Rust using a 'God-like' type system forced to handle too many things at once. CLEAR breaks these into 4 separate components: Types: The actual data / memory (User)</li> Capabilities: What you’re allowed to do with that data (User @shared:locked</code>)</li> Execution Boundaries: When you're do things concurrently or in parallel (BG/DO/CONCURRENT</code>)</li> Synchronization Gates: Where you explicitly mutate state concurrently (WITH sharedUser AS user { … }</code>)</li> </ul> In CLEAR, concurrency is a property of the execution boundary, not the data type. CLEAR requires explicit WITH</code> blocks for synchronization. While slightly less ergonomic than Rust for a simple Mutex, this design choice yields key structural dividends: Zero-Friction Pass-Through: ~85% of your codebase handles shared data without a single synchronization annotation, making code substantially more maintainable / refactorable to optimize for different synchronization strategies.</li> Polymorphic Synchronization: A single function can gracefully handle any of LOCKED</code>, SNAPSHOTTED</code>, ACTOR</code>, BUFFERED</code> etc data using a single block of code.</li> Pinpoint Cost Visibility: You know exactly where and when your code blocks, yields, retries and the associated failure methods. Latency costs are isolated to the 15% of functions that actually mutate state, rather than hidden in a type signature 10 levels up the stack.</li> </ul> You do not need to worry about a developer accidentally “sneaking” a slow actor synchronization into a hot loop. The costs are enforced ONLY at the synchronization boundaries (WITH</code> blocks). The compiler will simply reject incompatible synchronization strategies inside hot loops. FN transact( a: Account@shared, b: Account@shared, amount: Float64 ) RETURNS !Bool REQUIRES a, b: LOCKED | SNAPSHOTTED | BUFFERED -> IF amount <= 0 -> RAISE; WITH POLYMORPHIC a AS acctA, POLYMORPHIC b AS acctB { IF acctA.balance <= amount -> RAISE; acctA.balance -= amount; acctB.balance += amount; } ON LockTimeout ... ON Conflict ... ON QueueFull ... END END </code></pre> There is a user-defined default policy for handling failure methods. Not every function needs to specify failures: FN transact(a: Account@shared, b: Account@shared) RETURNS !Bool -> WITH POLYMORPHIC a AS acctA, POLYMORPHIC b AS acctB { ... } END </code></pre> This function will take any shared object: Mutex (@locked</code>), RwLock (@writeLocked</code>), MVCC (@versioned</code>), AtomicPtr (@atomic</code>), RingBuffer (@buffered</code>), Actor (@actor</code>). It will handle failure methods by the user-defined synchronization policy (or the system default): SYNC POLICY ON MvccConflict RETRY(3) THEN RAISE ON AtomicConflict RETRY(3) THEN RAISE ON LockTimeout RETRY(3) THEN RAISE ON QueueFull BLOCK ON ActorTimeout RAISE END </code></pre> Third-party code / libraries must compile in STRICT</code> mode, which requires them to explicitly handle all failure methods. There is no chance your default sync policy creates spooky-action-at-distance in third party failure handling. FAQ: But what about lock acquisition? CLEAR automatically sorts locks. It raises errors on deadlock that MUST be handled if possible (regardless of compilation mode), rather than deadlocking.</li> </ul> </li> But what about the Default Policy causing Spooky-Action-at-a-Distance internally? Compile in STRICT</code> mode where this is not allowed, and then you know all synchronization failure methods are handled locally.</li> Default synchronization failure policies are meant to be a speed up for prototyping, not something to use in production.</li> </ul> </li> But what about distributed failures? @actor</code> in CLEAR does not imply a distributed actor.</li> There is @shared:actor</code> (single-machine) and @distributed:actor</code> (n-machines).</li> @distributed</code> is not interchangeable with @shared</code>.</li> FN foo(x: T@shared)</code> does not accept a @distributed:actor</code>.</li> FN foo(x: T@distributed)</code> does not accept a @shared:actor</code> (it may not exist on the machine!)</li> </ul> </li> </ul> @actor</code> and @buffered</code> are in scope for v0.3. They are not currently available. @distributed</code> is in scope for v0.4. </div> Analogy</h3> Think of DO/BG/CONCURRENT</code> blocks as Execution Boundaries, and ownership capabilities (@shared</code>) as your Keys to pass the Boundary. A type without proper ownership cannot pass through an Execution Boundary (that is unsafe). If you want to do something in parallel (on multiple cores at once), the data must be @shared</code>. If you want to do something concurrently (on a single core at once), the data can be @shared</code> OR @local</code>. Similarly, a type with a synchronization capability (@locked</code>, @versioned</code>, etc) requires crossing a gate for safe access. The WITH</code> block acts as this gate to safe access as the BG/DO blocks act as the gate to safe boundary crossing. WITH</code> blocks require a type of permission to safely access synchronized data - this gives the reader insight into the cost. Ownership (the keys)</th> Execution Boundaries</th> Synchronization</th> Access Gates</th> Access Permission (cost model)</th></tr></thead> @shared (any cores)</td> BG {}</td> @locked, @writeLocked</td> WITH {}</td> EXCLUSIVE</td></tr> @local (pinned to a core)</td> DO {}</td> @versioned, @atomic</td> </td> SNAPSHOT</td></tr> </td> CONCURRENT {}</td> @actor, @buffered</td> </td> EVENTUAL</td></tr> </tbody></table> @multiOwned</code> is an ownership capability, but it does not grant access through Execution Boundaries. It allows an object to have multiple aliases (multiple owners - i.e. for a graph), but it does NOT grant permission to cross Execution Boundaries. Complexity Reduced? Or just moved around?</h3> It is a fair critique from Rust or Go that this is nothing new, and arguably more complex. Rust has a single system to keep track of. CLEAR has three. If anything, that’s more complex, not less. CLEAR thinks it is clearer to break the problem down into steps. For one, developers not familiar with concurrent code know that anything @shared</code> is something concurrent. In Rust, it is hard to distinguish between Arc<RwLock<T>></code>, DashMap<T></code>, and Vec<T></code>. Which one is something concurrent, that could have logical race bugs? Secondly, while this might appear to be unergonomic, the concurrent benchmarks in CLEAR are only ~70% of the code as Rust (much more expressive than non-Rust people probably assume), and ~30% of the code as Go. Lines of code on its own is a terrible benchmark for complexity. But CLEAR did not attempt to cherry-pick this metric. This is on top of the benefit of having polymorphic synchronization for almost no cost, and the ability to benchmark different strategies almost free. Thirdly and most crucially, polymorphic synchronization allows your code to be maintainable in a way that is very difficult to achieve in Rust or Go. Arc<RwLock<T>></code> is viral in Rust. T@shared</code> is not in CLEAR. Step 1) update your with blocks to use POLYMORPHIC access permission.</li> Step 2) change your synchronization strategy from @locked to @versioned.</li> Step 3) run clear profile</code>.</li> Step 4) profit.</li> </ul> CLEAR levels the playing field for non-experts</h3> In Rust or Go, you need to fully understand the implications of your code before you even write and test it. Unless you are an expert (and even if you are), this is difficult to get right. If you get it wrong, it is typically painful to rearchitect your app to do it right. It is also possible after your code continues to grow, you may need to switch again (possibly back to where you started), which can be equally painful. CLEAR was designed assuming that you cannot get everything right from the get-go, and to make it as easy as possible to experiment and optimize to get it right. It was designed assuming you want to get there, even if you would have no chance in Rust or Go, because you need to understand things at a deeper level. CLEAR was designed to act like a high-level language like Ruby, but give you systems level speed. Capabilities in Depth</h2> When multiple fibers need to access the same data, CLEAR must answer two questions: Lifetime: How does the data stay alive when multiple fibers reference it?</li> Thread safety: Is the reference count safe across schedulers (OS threads)?</li> </ol> Quick Reference</h2> Capability</th> Zig Type</th> Refcount</th> Thread-safe?</th> @parallel</code></th> Use when</th></tr></thead> @local</code></td> *T</code></td> None</td> No (single-scheduler)</td> Error</td> Fibers on one scheduler share mutable state</td></tr> @multiowned</code></td> Rc(T)</code></td> Non-atomic</td> No (single-scheduler)</td> Error</td> Multiple owners, single scheduler, read-mostly</td></tr> @shared</code></td> Arc(T)</code></td> Atomic</td> Yes</td> Allowed</td> Cross-scheduler sharing (rare)</td></tr> </tbody></table> @local — Zero-Cost Shared Mutable Reference</h2> MUTABLE c = Counter{ value: 0 } @local; BG { c.value += 1; } # direct field access, no WITH block BG { print(c.value); } # direct read </code></pre> What it does: Heap-allocates the value and returns a bare *T</code> pointer. No Mutex, no RwLock, no reference counting. Multiple fibers share the pointer by value copy. Why it's safe: The compiler auto-pins all BG/DO blocks that capture @local</code> variables to the local scheduler. Cooperative scheduling on a single OS thread means no two fibers ever execute simultaneously — no data races are possible. When to use it: This is the default choice for shared mutable state within a function scope. It's the fastest option — zero synchronization overhead. Compile-time enforcement: BG { @parallel -> c.value = 1; } # ERROR: @local variable cannot be used in @parallel block </code></pre> @multiowned — Non-Atomic Reference Counting (Rc)</h2> node = TreeNode{ left: NIL, right: NIL } @multiowned; # Multiple owners via WITH block: WITH node AS val { print(val.left); } </code></pre> What it does: Wraps the value in Rc(T)</code> — a non-atomic reference-counted pointer. Each WITH</code> unwrap increments the refcount; scope exit decrements it. The value is freed when the last reference is released. What it does: Wraps the value in Rc(T)</code> — a non-atomic reference-counted pointer. Each WITH</code> unwrap increments the refcount; scope exit decrements it. The value is freed when the last reference is released. Why it exists: For graph structures and shared ownership patterns where multiple variables need to keep the same value alive. Unlike @local</code> (which has one owner and shared pointers), @multiowned</code> has multiple owners with automatic lifetime management. Read-only access: WITH node AS val</code> provides read-only access to the inner value. @multiowned</code> does not support mutation — it's shared ownership, not shared mutation. This is analogous to Rust's Rc<T></code> (not Rc<RefCell<T>></code>). For shared mutation, use: @alwaysMutable</code> — interior mutability (RefCell</code>), mutate through const bindings</li> @local</code> — zero-cost, single-scheduler, direct field writes</li> @locked</code> — mutex-protected, cross-scheduler safe</li> </ul> Why it's NOT thread-safe: Rc</code> uses a plain integer for its refcount — no atomic CAS, no memory barriers. If two threads increment/decrement simultaneously, the count corrupts (use-after-free or double-free). Compile-time enforcement: BG { @parallel -> WITH node AS val { ... } } # ERROR: @multiowned (Rc) variable cannot be used in @parallel block — # Rc uses a non-atomic reference count. Use @shared (Arc) for cross-scheduler sharing. </code></pre> When to use it: Graphs, trees, and shared ownership patterns where all fibers run on the same scheduler and data is read-only. If you need mutation, use @local</code>. If you need cross-scheduler sharing, use @shared</code>. @shared — Atomic Reference Counting (Arc)</h2> Technically @shared</code> means permission to cross an execution boundary. Atomics and atomic pointers (@shared:atomic</code>) are NOT wrapped in Arc. </div> config = AppConfig{ port: 8080 } @shared; BG { @parallel -> WITH config AS c { print(c.port); } } # OK BG { @parallel -> WITH config AS c { print(c.port); } } # OK </code></pre> What it does: Wraps the value in Arc(T)</code> — an atomic reference-counted pointer. The refcount uses hardware atomic instructions (lock-prefixed CAS on x86), so it's safe to increment/decrement from any thread. Why it's expensive: Every clone/drop of an Arc</code> bounces the cache line containing the refcount between CPU cores. On a 16-core machine, this "cache-line bouncing" can cost ~100ns per operation (vs ~1ns for a non-atomic increment). When to use it: Only when you genuinely need cross-scheduler sharing — data accessed by fibers on different OS threads. This is rare in practice: most shared state is within a single function scope (use @local</code>) or within a single scheduler (use @multiowned</code>). Combining with Sync Capabilities</h2> Sharing capabilities can be combined with sync capabilities for mutable cross-thread access: # Arc + Mutex: one-line mutations auto-lock MUTABLE counter = Counter{ value: 0 } @shared:locked; BG { @parallel -> counter.value += 1; } # auto mutex # Arc + RwLock: cross-scheduler read-heavy access MUTABLE config = Config{ port: 8080 } @shared:writeLocked; BG { @parallel -> WITH config AS c { print(c.port); } } # read lock BG { @parallel -> config.port = 9090; } # auto write lock </code></pre> @local</code> does not combine with @locked</code> or @writeLocked</code> — it's already a bare pointer with no wrapper. Mutation is direct. Decision Tree</h2> Do multiple fibers need to access this data? ├── No → default (no capability, single owner) └── Yes ├── Does it need mutable access? │ ├── Same scheduler → @local (zero cost, direct field writes) │ └── Cross-scheduler → @locked or @writeLocked (mutex/rwlock) └── Do multiple fibers need to OWN it (keep it alive)? ├── Read-only, same scheduler → @multiowned (Rc) ├── Read-only, cross-scheduler → @shared (Arc) └── Mutable + shared ownership → @local (same scheduler) or @shared:locked (cross-scheduler) Stable heap address needed (graph edges, self-referential)? └── @indirect (combinable with any of the above) </code></pre> Note on primitives: Capabilities cannot be applied to primitive types (Int64</code>, Float64</code>, Bool</code>, Byte</code>, Float32</code>). Wrap in a STRUCT</code> first — this makes the intent explicit and gives you named fields. Auto-Pinning</h2> The compiler automatically pins BG/DO blocks to the local scheduler when they capture @local</code>, @multiowned</code>, @shared</code>, @locked</code>, or @writeLocked</code> variables. This is a cache-line bouncing optimization for thread-safe types and a safety requirement for non-thread-safe types. To override auto-pinning for thread-safe types: BG { @parallel -> ... } # distribute across schedulers </code></pre> For non-thread-safe types (@local</code>, @multiowned</code>), @parallel</code> is a compile error. THE ANATOMY OF A GOOD FUNCTION 2026-03-01T00:00:00+00:00 The most desired behavior from any function is that it works. A function should exist to serve the user, not the compiler. It should (in order of preference): Return (and do) what the user wants</li> Return a suitable default (if possible/applicable)</li> Return an explicit set of errors</li> </ol> It should return exactly ONE type. A user should never need to post-process a function's output to do what they want. ALL PUBLIC FUNCTIONS MUST BE GOOD</h2> PUBLIC FN getFullyFormedUser(id: Number) -> RETURNS User OR User::DEFAULT -- 1. Input Hygiene: Handle invalid inputs *uniformly* at the top. GUARD id > 0 OR RETURN DEFAULT; -- 2. The Happy Path: A clear, easy-to-follow stream of data using 's>' RETURN id s> fetchUserFromCache s> OTHERWISE(fetchFromDb!!) OR EXIT -- If cache misses, try DB s> hydrateFromOtherDb; -- 3. The Catch-All: -- All error logic is tucked away at the bottom. CATCH RETURN DEFAULT; -- Return suitable default on error END </code></pre> INTERNAL FUNCTIONS CAN BE BAD</h2> -- This can return an error `!!` -- Internal code can be "Chill-Correct" until you're ready to make it public. FN reciprocalId(id) -> RETURN functionThatCanError!!(id); END </code></pre> Comparisons</h2> C: High Danger. Manual memory management and error checking.</li> Go: Visual Noise. Repetitive if err != nil</code> boilerplate kills flow.</li> Rust: High Cognitive Load. Mandatory unwrapping of every Enum/Result case.</li> CLEAR: Represents Flow. It projects the "View" of the data you want, handling the Union/Error logic implicitly based on your definitions.</li> </ul> -- CLEAR projects the view of the data you want -- 1. Handles the Error (defaults to empty/nil if it fails) -- 2. Handles the Union (projects View) PRINT(getUser().email); </code></pre> THE FINAL PATH</h2> CLEAR is designed as a language that is native to Property-Based Testing. Because EVERY public function follows a strict contract: Inputs and outputs have suitable defaults (or explicit explosions).</li> We can programmatically generate permutations of every possible input for testing.</li> CLEAR enforces a Strict Contract at the Public boundary (RETURNS Type OR Default). This solves the Oracle Problem for automated testing.</li> </ol> The easier you make code to understand and to test, the easier you make it to get working code. And working code is all that really matters. VERDICT</h2> C/Elixir represent Chaos.</li> Rust/Go represent Bureaucracy.</li> CLEAR represents Flow.</li> </ul> THE GIVE PROBLEM 2026-03-01T00:00:00+00:00 In CLEAR, we separate Types from Capabilities. Type: What the data is (e.g., User</code>).</li> Capability: How the data is accessed (e.g., affine</code>, shared</code>).</li> </ul> By default, objects in CLEAR are affine (meaning they can only be used once, and passing them to a function "borrows" them). When a function needs to take ownership of an object (e.g., to store it in a long-lived list), it must explicitly TAKES</code> it. The Friction: GIVE</code> vs. COPY</code></h2> If you have an affine object and you need to pass it to a function that TAKES</code> it, you have two choices: GIVE it: Transfer ownership. The original variable becomes "dead" and can no longer be used.</li> COPY it: Create a duplicate. The original variable remains "alive" and can still be used.</li> </ol> STRUCT User { name: String } FN addToCache(TAKES u: User) RETURNS Void -> -- This function now owns 'u' RETURN; END u = User{ name: "Alice" }; addToCache(GIVE u); -- Explicit Move print(u.name); -- COMPILER ERROR: u is dead. u2 = User{ name: "Bob" }; addToCache(COPY u2); -- Explicit Copy print(u2.name); -- OKAY: u2 is still alive. </code></pre> Why this is Better than Rust</h3> In Rust, "moving" or "copying" is often implicit based on the type (e.g., i32</code> is Copy</code>, String</code> is not). This makes it difficult to reason about whether a variable is still alive after a function call without knowing the exact implementation details of the type. In CLEAR, the developer's intent is always explicit: If you see GIVE</code>, you know the variable is gone.</li> If you see COPY</code>, you know you're paying a performance cost to keep it.</li> If you see neither, it's a Borrow (safe and fast).</li> </ul> Summary</h2> Keyword</th> Meaning</th> Result</th></tr></thead> GIVE</code></td> Move ownership</td> Original is dead, zero-cost transfer</td></tr> COPY</code></td> Duplicate data</td> Original is alive, high-cost duplicate</td></tr> (None)</td> Borrow</td> Original is alive, zero-cost reference</td></tr> </tbody></table> CLEAR optimizes for local reasoning and explicit intent, making ownership semantics intuitive for everyone. REENTRANCY ATE MY BABY 2025-12-19T00:00:00+00:00 WITH blocks are non-reentrant</h2> In CLEAR, when you open a WITH</code> block on an object to acquire a capability (like mutation or sharing), no other WITH</code> block for that same object can start until the first one finishes. This applies to other threads, but crucially, it also applies to your own recursive functions or callbacks.</li> This restriction gives you Rust-level determinism for mutable access to shared objects, with significantly less cognitive overhead.</li> </ul> The Mental Model: Capabilities are Edges</h3> In CLEAR, we separate Types from Capabilities: Types (e.g., Player</code>) are passed into functions.</li> Capabilities (e.g., shared:locked</code>, alwaysMutable</code>) are managed at the "edges" using WITH</code> blocks.</li> </ul> The Rule: Functions take the Type, not the Capability. You must unwrap the capability using WITH</code> before calling the function. -- CLEAR's Solution: Unwrapping at the call site WITH sharedPlayer AS player { -- 'player' is now the raw Type (User), unwrapped from its 'shared' capability damage!(player); -- OKAY heal!(player); -- OKAY -- heal!(sharedPlayer); -- COMPILER ERROR! -- Functions take User, not shared:locked User. } </code></pre> This prevents the "Infinite Mirror" pattern where a function modifying an object accidentally tries to re-acquire a lock it already holds. Why this matters</h3> Re-entrancy bugs (where state changes under your feet while you're in the middle of an operation) are the root of most concurrent complexity. The Problem (Sheer Chaos):</h4> In most languages (Java, C, Go, Swift), you cannot trust from one line of code to the next. While you are reading a list, another thread can pop()</code> from it, causing your next line to crash with an IndexOutOfBounds</code> error. The CLEAR Solution: Scoped Sovereignty</h4> By banning re-entrancy via WITH</code> blocks, CLEAR guarantees that while you are inside that scope: You are the sole owner of that object's state.</li> No one else (not other threads, and not your own callbacks) can touch it.</li> </ol> This turns a global problem (unpredictable concurrency bugs) into a local problem (simple logic). Comparison with Other Models</h3> Actor Model (Erlang/Go Channels): Safe, but treats your RAM like a slow network cable. It often leads to "God Actors" that serialize your entire program.</li> Rust (Borrow Checker): Perfect safety, but high cognitive cost. Rust forces you to infect every function signature with lifetimes and capability types (like Arc<RwLock<T>></code>).</li> CLEAR: Gives you Rust's safety and performance, but keeps your functions "pure" by handling capabilities at the call site.</li> </ul> Simplified Refactoring</h3> In Rust, if you change an object from Rc</code> (single-threaded) to Arc</code> (multi-threaded), you must update every function signature that takes that object. In CLEAR, you only change the capability at the definition site. Because your functions only take the raw Type (e.g., User</code>), the rest of your codebase remains untouched. The "Refactoring Blast Radius" is zero. A final note on local reasoning</h3> CLEAR optimizes for the ROI on brain power. Rust optimizes for the toaster (minimum RAM/CPU at any cognitive cost).</li> CLEAR optimizes for the developer (maximum safety and speed with minimum cognitive load).</li> </ul> By enforcing explicit scopes for capabilities, CLEAR ensures that "poisoning" (restricting access to data) is always visible and local. You don't need to be a systems programming expert to write blazing-fast, thread-safe code. You just need to follow the WITH</code> block. SHARED MUTABLE DEATH 2025-12-18T00:00:00+00:00 CLEAR - The Unfair Advantage</h2> STRUCT Account { balance: Float64, } STRUCT Cache { -- Capabilities are properties of the storage, not the type items: shared:locked HashMap<String, shared:locked Account>, } FN insert(cache: Cache, id: String, account: Account) -> -- Acquire shared:locked capability sharedAccount = SHARE:LOCKED(account); WITH cache.items { -- Inside here, the HashMap is unwrapped and locked _.insert(id, GIVE sharedAccount); } END FN transact(cache: Cache) -> -- CLEAR automatically handles lock ordering in WITH blocks to prevent deadlocks WITH cache.items.get("a") AS a, cache.items.get("b") AS b { a.balance += 10; b.balance -= 10; } END </code></pre> What can go wrong? Not deadlock, not memory corruption.</li> CLEAR makes illegal states impossible at the concurrency level while being easy and ergonomic.</li> </ul> CLEAR distinguishes between Types and Capabilities: Account</code> is a Type.</li> shared:locked</code> is a Capability.</li> </ul> Functions take Account</code>, not shared:locked Account</code>. This means your business logic remains pure and decoupled from your synchronization strategy. Refactoring Blast Radius: Zero</h3> If you need to change from a Mutex</code> (shared:locked</code>) to an RwLock</code> (shared:writeLocked</code>), you only change the definition in the Struct. Your functions don't change because they never knew about the capability in the first place. Rust - This is Safe?</h2> // Rust infection: capabilities (Arc, RwLock) are part of the type system struct Cache { sharedItems: Arc<RwLock<HashMap<String, Arc<RwLock<Account>>>>>, } // Function signatures are infected by the capabilities fn transfer(from: &Arc<RwLock<Account>>, to: &Arc<RwLock<Account>>, amount: i32) { // Manual lock ordering required to avoid deadlock let (first, second, swap) = if Arc::as_ptr(from) < Arc::as_ptr(to) { (from, to, false) } else { (to, from, true) }; // ... } </code></pre> The Safety Gap</h3> Rust guarantees you will not have a data race (memory corruption).</li> It does not guarantee you will avoid a deadlock (program halt).</li> CLEAR guarantees both by managing lock acquisition order and separating capabilities from types.</li> </ul> Swift - This is Ergonomic?</h2> Swift's Actors introduce Reentrancy. If you await</code> inside an actor, the state can change underneath you, leading to logic races. CLEAR avoids this by using explicit WITH</code> scopes that maintain invariants. Go - This is Simple?</h2> In Go, Mutexes are just structs. If you accidentally pass an Account</code> by value, you copy the Mutex, and you're now protecting nothing. CLEAR treats capabilities as compiler-enforced metadata, making this error impossible. C - YOLO</h2> C is both memory-unsafe and liveness-unsafe. CLEAR is the simplest language that is both memory-safe and liveness-safe. CLEAR rests its case. It is both: The simplest language for concurrency.</li> The only liveness-safe language that prevents deadlocks by design.</li> </ol> THE DLL PROBLEM 2025-12-03T00:00:00+00:00 In most languages (C, C++, Rust), you can easily create a Doubly Linked List (DLL). typedef struct Node { int val; struct Node* prev; struct Node* next; } Node; </code></pre> In a DLL, every node has a pointer to its neighbor, and every neighbor has a pointer back. This is a Cycle. Cycles are the enemy of deterministic memory management. If you use Reference Counting (Swift/Python), A will never die because B holds a reference to it, and B will never die because A holds a reference to it. They leak memory forever.</li> If you use Arena-Based Memory (CLEAR), child nodes die when the function returns. If you try to point back to a parent, you're pointing to a corpse.</li> </ol> The Solution: IDs and Centralized Lookups</h2> In CLEAR, we forbid shared mutable cycles. This seems like a huge limitation, until you realize that the vast majority of programs do not need DLLs. You need a DLL for: A text editor's buffer, a kernel's task scheduler, or a high-performance LRU cache.</li> You do NOT need a DLL for: A web server, a database client, a CLI tool, or a business application.</li> </ul> In CLEAR, you'd have to use a list to manage the pointers, like so. STRUCT List { nodes: Node[] } STRUCT Node { val: Int64, prev_idx: Int64, next_idx: Int64 } </code></pre> If this IS your product/business, CLEAR will probably hinder you rather than help you.</li> CLEAR is designed to make optimization for cache locality as EASY as possible (without having to manage literally all memory like C).</li> By using IDs/Indices instead of raw pointers, CLEAR guarantees: Memory Safety: No dangling pointers.</li> Concurrency Safety: No race conditions on cyclic structures.</li> Refactoring Ease: You can move the entire list in memory without breaking pointers.</li> </ol> </li> </ul> CLEAR optimizes for the 99% case, where safety and organization are more important than raw pointer soup. Atomic Power: Shared Memory Without the Garbage Collector 2025-12-01T00:00:00+00:00 In CLEAR, 99% of your data lives in Arenas (notebooks) that are incinerated when a function returns. This is fast and safe, but it has one flaw: Data cannot survive the death of its creator. Sometimes, you need data to live longer than the function that created it.</li> Sometimes, you need data to be shared across multiple threads running in parallel.</li> </ul> Enter the 'shared' Capability and shared:atomic</code>. 1. What is the 'shared' Capability?</h2> In CLEAR, we separate Types from Capabilities. Type: What the object is (e.g., User</code>).</li> Capability: How the object is accessed (e.g., shared</code>).</li> </ul> The shared</code> capability (Arc in Rust) allows an object to live in the Global Heap, survive outside its creator's arena, and be shared safely across threads. -- Acquisition: Acquire 'shared' capability sharedU = SHARE(User.new()); -- Usage: Pass the 'raw' Type to functions process(sharedU); </code></pre> 2. Atomics: Primitives in the Ether</h2> For simple numeric types like counters, CLEAR uses the shared:atomic</code> capability. An Atomic primitive lives in the Global Heap and can be safely mutated by multiple threads at the same time. counter = SHARE:atomic(0); -- Create an atomic integer counter.increment!(); -- Thread-safe mutation </code></pre> 3. How CLEAR Handles Them: "The Magic Count"</h2> CLEAR does not use a Tracing Garbage Collector (like Java/Go) that pauses your program. Instead, CLEAR uses Deterministic Reference Counting. Every shared</code> object has a tiny backpack holding a number (the Reference Count). This number represents how many people are holding a handle to it. Creation: Count = 1. </li> Sharing: When you pass a shared</code> object to a thread (SPAWN</code>), Count increments. </li> Drop: When a thread finishes, its handle goes out of scope and Count decrements. </li> Destruction: The microsecond Count hits 0, the memory is freed instantly. </li> The Result: Zero "Stop the World" pauses. Memory is freed precisely when it's no longer needed. </li> </ul> 4. The Law: No Shared Cycles</h2> Reference counting has one fatal weakness: Cycles. If A holds B, and B holds A, their RefCounts will never hit 0. They will leak memory forever. CLEAR refuses to stop your program to scan for cycles. Therefore, CLEAR enforces The Law: A shared</code> object CANNOT hold a reference to another shared</code> object. The Topology</h3> This forces your memory to be a DAG (Directed Acyclic Graph). Data ownership always flows "Down" or "Across," never "Back Up." By forbidding cycles, CLEAR guarantees: No Memory Leaks: It is mathematically impossible to leak a shared</code> object.</li> No Deadlocks: You avoid the circular dependencies that often lead to circular locking.</li> No GC Pauses: There's no need to scan the heap.</li> </ol> Summary</h2> Feature</th> Arena (Default)</th> Shared / Atomic</th></tr></thead> Location</td> Local Arena</td> Global Heap</td></tr> Lifetime</td> Function Scope</td> Reference Counted</td></tr> Speed</td> Blazing Fast (O(1))</td> Fast (System Malloc)</td></tr> Concurrency</td> Thread-Private (Safe)</td> Thread-Safe</td></tr> Usage</td> 99% of variables</td> Shared counters, Queues, Configs</td></tr> Cost</td> Zero GC</td> Zero GC (Deterministic)</td></tr> </tbody></table> THE MATCH DILEMMA 2025-11-26T00:00:00+00:00 In standard Tagged-Union / Enum languages (Rust, Swift, Kotlin), the compiler forces you to MATCH</code> every single case of a Union type. match user { User::SignedIn(u) => println!("{}", u.email), User::Guest => println!("Welcome guest"), User::Banned => println!("You are banned"), } </code></pre> The Friction: If you only care about the email of signed-in users, you still have to write code to handle the Guest and Banned cases. If you add a User::Moderator</code> later, your code breaks everywhere until you handle that new case. This is Bureaucracy. It prioritizes the compiler's needs over the developer's intent. The Solution: CLEAR (Fortress Architecture)</h2> CLEAR is a Fortress Language. It handles the "Safety Gap" at the public boundary, allowing your internal logic to flow smoothly. In CLEAR, any Public Union must be able to behave as a Product (a struct with fields) by projecting a View. -- CLEAR allows this! -- If the user is NOT SignedIn, .email returns a suitable default (empty string) -- or handles the Nil case implicitly if defined in the Fortress boundary. PRINT(getUser().email); </code></pre> Selective Matching</h3> In CLEAR, you match only what you care about. CASE getUser() OF SignedInUser AS u => doSignedInUserThing(u); -- No need to match Guest or Banned if you don't care! -- They flow through to the default or are ignored. END </code></pre> The Benefit: Refactoring Blast Radius: Zero. Adding a new case to a Union doesn't break existing code that doesn't care about that new case. Why this is Safe</h3> CLEAR projects a View of the data. If you access a field that only exists in one variant of a Union: If the variant matches: You get the data.</li> If it doesn't: You get a suitable default (defined at the boundary) or an explosion (!!</code>) if you've marked it as mandatory.</li> </ol> This preserves understandability and velocity without sacrificing safety. Summary</h2> Language</th> Approach</th> Result</th></tr></thead> Rust</td> Exhaustive Match</td> Bureaucracy & Fragility</td></tr> Go</td> Type Assertion</td> Error-Prone & Verbose</td></tr> CLEAR</td> Projected View</td> Flow & Flexibility</td></tr> </tbody></table> CLEAR optimizes for the developer's intent, not the compiler's pedantry. CLEAR EXAMPLES 2025-11-22T00:00:00+00:00 Example Grammar</h2> FN processUsers(users: User[]) -> -- Pipelines with SMOOTH operator 's>' result = users AS @u s> UNNEST _.orders s> SELECT _.price * @u.discount s> REDUCE(0, (acc, x) -> acc + x ); -- Default to "high"; override if result is low MUTABLE info = { "sum": [10, 9], "status": "high" }; IF result <= 1000 THEN info = { "sum": [1, 0], "status": "low" }; END RETURN info["sum"][0]; END -- Lambda with % sigil myLambda = %(x: Number) -> doIt(); RETURN TRUE; END </code></pre> SIGILS</h3> Sigil</th> Meaning</th> Example</th></tr></thead> %(...)</code></td> Lambda parameters</td> %(x, y) -> x + y</code></td></tr> @</code></td> Pipeline binding / Directives</td> users AS @u</code></td></tr> !!</code></td> Explicit panic</td> list.get(i)!!</code></td></tr> _</code></td> Pipeline placeholder</td> s> SELECT _.name</code></td></tr> s></code></td> SMOOTH operator</td> data s> process</code></td></tr> </tbody></table> STRUCTS</h3> STRUCT Point { name: String, x: Number DEFAULT 10, y: Number DEFAULT 20 } -- Recursive structure with 'indirect' STRUCT Node { value: Int64, left: indirect Node, right: indirect Node } FN Point::distance(self: Point, other: Point) -> Number -- ... END myPoint = Point{ x: 10, y: 20 }; oPoint = Point{ x: 10, y: 10 }; -- Method call syntax (UFCS) d = myPoint.distance(oPoint); </code></pre> CONTROL FLOW</h3> -- WHILE Loop WHILE x > 0 DO ... IF ... THEN BREAK END END -- IF/ELSE IF x == 1 THEN do1(); ELSE_IF x == 2 THEN do2(); ELSE doElse(); END -- WITH Capability Block WITH sharedAccount AS acc { acc.balance += 100; } </code></pre> ERROR HANDLING</h3> FN myFunc(a, b, c) -> -- OR RAISE bubbles up the error val = fetchData(a, b, c) OR RAISE s> parseHeader OR EXIT "Invalid Header" s> parseBody OR EXIT "Invalid Body" s> fetchUser s> RECOVER(DefaultUser()) CATCH ParseError WITH("Invalid Header") logInvalidHeader(%e.snapshot); RETURN defaultPage(); DEFAULT logUnknownError(%e); RAISE %e; END </code></pre>

CLEAR — Retrospective

Can AI Un-Slop Itself?

How to Vibe Code something that *actually* works

What DOES help:</h2>

The Key to Success - Signal Diversity</h2> LLMs are sort of</em> good at everything.</p> You can make them far better if you build a self-correcting fortress to: 1) reject their failures, and 2) accept their wins.</p>

What I learned (the hard way) Building a Memory Safe Programming Language with LLMs over the last 6 months

Where LLMs excel</h2> Generating ideas and signals</li> Filtering and reducing noise</li> Identifying problems</li>

Where LLMs are decent, but not as good as I expected</h2> Refactoring—especially re-architecting</li> </ul> Where LLMs do not excel</h2> Getting things correct</li> Obeying design principles</li>

Where LLMs do not excel</h2> Getting things correct</li> Obeying design principles</li>

CLAUDE.md / AGENTS.md</h2> See my CLAUDE.md</a>. The sections below "Output" are safe to copy. The "Contributing" section should be tailored to your project.</p>

Biggest Mistakes</h2>

2. Moving too fast</h3> Blitzing through features usually resulted in more work and headaches. Even with LLMs, rushing is costly. Delivering at 100x is tempting, but the "headache factor" of LLM mistakes is significant.</p>

3. Not understanding LLM limitations early</h3> I should have generated a robust test suite from the start. LLMs break architectural invariants easily. They can generate testing frameworks quickly; use them to create "fortress files" that protect your design goals.</p>

pprof integration

What's in each file</h2>

STRICT EXTREME: The Path to High-Integrity CLEAR (@strict:extreme)

Core Pillars of STRICT EXTREME</h2>

Deadlock in CLEAR

Fix: Single Lock Scope (Always Safe)</h2> Restructure so each operation needs only one lock at a time. No re-entrancy, no NEXT-inside-WITH:</p>

Why CLEAR

PROPAGANDA

manifesto EXTRA

THE CONFUSING TYPES</h2>

</h2> See I'm too dumb to open a file in Zig</a> - for common problems of why being a genius doens't scale - even for something as simple as reading a text file...</p>

I wanted to write code that scales linearly to 128+ cores, but feels like Ruby. That didn't exist, so I made it.

Profiling

Architecture

CLEAR for Scripters

4. The Two Worlds: STACK vs. HEAP</h3>

7. Physics of Time: The ARENA (Lifetimes)</h3> Variable lifetimes follow a simple birth/death cycle: they live as long as the Function they were born in.</p>

The ARENA Rule:</h4> When a function starts, it opens a clean ARENA (A bank of memory). Any variable you create lives in this ARENA. When the function ends, the entire ARENA is wiped. POOF</em>.</p>

OPINIONS

WHO IS CLEAR *NOT* FOR

Modules & Build System

Build System</h2> CLEAR features a built-in dependency manager that handles recursive REQUIRE</code> calls, prevents circular dependencies, and manages the transpilation graph.</p>

How to Vibe Code something that actually works

The Key to Success - Signal Diversity</h2>
LLMs are sort of</em> good at everything.</p>
You can make them far better if you build a self-correcting fortress to: 1) reject their failures, and 2) accept their wins.</p>

CLAUDE.md / AGENTS.md</h2>
See my CLAUDE.md</a>. The sections below "Output" are safe to copy. The "Contributing" section should be tailored to your project.</p>

3. Not understanding LLM limitations early</h3>
I should have generated a robust test suite from the start. LLMs break architectural invariants easily. They can generate testing frameworks quickly; use them to create "fortress files" that protect your design goals.</p>

</h2>
See I'm too dumb to open a file in Zig</a> - for common problems of why being a genius doens't scale - even for something as simple as reading a text file...</p>

7. Physics of Time: The ARENA (Lifetimes)</h3>
Variable lifetimes follow a simple birth/death cycle: they live as long as the Function they were born in.</p>

The ARENA Rule:</h4>
When a function starts, it opens a clean ARENA (A bank of memory). Any variable you create lives in this ARENA. When the function ends, the entire ARENA is wiped. POOF</em>.</p>

WHO IS CLEAR NOT FOR

Build System</h2>
CLEAR features a built-in dependency manager that handles recursive `REQUIRE</code> calls, prevents circular dependencies, and manages the transpilation graph.</p>`