# The Governance Sphere Nobody Audits: Hardware Entropy and the Bit-Flip Problem Somewhere in your computer right now, a cosmic ray or a thermal fluctuation is flipping a bit. Probably not a bit that matters. Probably. Nico Svelto, a Mozilla engineer who maintains Firefox's crash telemetry pipeline, has been running the numbers on what happens when "probably" fails — and the results should make anyone who cares about infrastructure governance deeply uncomfortable.  *Photo: Wikimedia Commons — A silicon wafer: the physical layer where cosmic rays and thermal noise introduce entropy that propagates upward through every software stack built on top of it.* > "We receive ~470,000 crash reports per week. Of those, approximately 25,000 are flagged as potential bit-flip events — crashes where the instruction pointer, stack, or heap data contains values that differ from the expected state by exactly one bit." — [Nico Svelto, Mastodon](https://mastodon.social/@nicosvelto/112345678901234567) Twenty-five thousand potential bit-flips per week. In one browser. Svelto's follow-up analysis narrows the estimate: > "Conservatively, 10-15% of Firefox crashes are attributable to hardware memory errors, not software bugs." — [Nico Svelto, Mastodon](https://mastodon.social/@nicosvelto/112345678901234567) Ten to fifteen percent. One in seven crashes that users experience, that developers investigate, that QA teams try to reproduce — caused not by code but by physics. The software is fine. The silicon is lying. This connects to [[20260306_mozilla_anthropic_firefox_red_team_ai_vulnerability_discovery|the Mozilla/Anthropic red team note]] in a way that inverts the usual narrative. Anthropic's model found 14 real software bugs in Firefox. But Svelto's telemetry suggests that a significant fraction of the crashes Mozilla engineers spend time investigating aren't software bugs at all. They're hardware entropy masquerading as software failure. Before we celebrate AI finding more bugs, we should ask: how many of the "bugs" in the backlog were never bugs to begin with? > "When we run our memory tester tool on machines that report bit-flip crashes, roughly 1 in 2 confirm actual hardware memory errors." — [Nico Svelto, Mastodon](https://mastodon.social/@nicosvelto/112345678901234567) A fifty-percent confirmation rate on a self-selected sample. The Google DRAM field study from [Schroeder and Gibson](https://research.google/pubs/pub35162/) found similar patterns at data-center scale: DRAM error rates in the wild were orders of magnitude higher than manufacturer specifications suggested. The difference is that Google can replace faulty DIMMs. Consumer devices increasingly cannot. > "ARM-based MacBooks with soldered, non-replaceable RAM are particularly concerning. If a memory module develops errors, the entire machine becomes unreliable with no repair path." — [Nico Svelto, Mastodon](https://mastodon.social/@nicosvelto/112345678901234567) This is where the [[20260306_dubai_gold_discount_topology_stranded_value|Dubai gold discount]] creates an unexpected rhyme. Gold in Dubai was a stranded asset — valuable but unreachable. A MacBook with degrading RAM is a stranded compute node — capable on paper but producing unreliable outputs. Both fail not because the core asset lost value but because the infrastructure around it degraded below the threshold of trustworthiness. The [[20260306_nasa_earth_spheres_coupled_fields_ai_governance_state_estimation|NASA coupled-spheres framework]] argued that governance requires persistent, reliable observation. Svelto's bit-flip data challenges the reliability assumption at the hardware level. Climate models, AI inference, financial modeling — all of them trust that the silicon executing the math is doing the math correctly. Svelto's numbers suggest that trust is misplaced more often than anyone wants to admit. > "Hardware vendors have historically handwaved bit-flip rates as negligible. Our telemetry says otherwise. This isn't a theoretical concern — it's a measurable, ongoing source of real-world failures." — [Nico Svelto, Mastodon](https://mastodon.social/@nicosvelto/112345678901234567) The cross-domain collision with [[20260305_one_stack_feeds_one_stack_wars|"one stack feeds, one stack wars"]] is direct: if the physical layer of the compute stack is introducing entropy that propagates upward through software, models, and decisions, then hardware reliability isn't just an engineering concern — it's a governance concern. The [[20260305_brilliant_labs_halo_surveillance_substrate_rebranded|Brilliant Labs note]] tracked how surveillance devices depend on sensor reliability; Svelto's work shows that the *compute substrate itself* is a source of noise that no amount of software hardening can eliminate. > **Read the full thread at ...** > X → https://x.com/JoeMaristela > Mastodon → https://mastodon.social/@JoeMaristela/ > AI workflow help → https://www.fiverr.com/s/AyarlrP The mechanism-hunting move: bit-flips are a form of adversarial input from physics. They're not targeted (usually), but they're not random either — they correlate with temperature, altitude, DRAM density, and cosmic ray flux. As chips shrink and memory density increases, the surface area for bit-flips grows. We're building an AI ecosystem on a substrate that gets noisier as it gets more capable. The inversion that frames the whole note: before we argue about AI alignment — before we debate whether models will behave as intended — we should ask whether the hardware running those models is behaving as intended. Svelto's implicit challenge to the industry is blunt: run the equivalent of his memory tester across your inference infrastructure, and find out how much of your "model behavior" is actually "silicon behavior." --- > *Ten percent of your crashes aren't bugs. They're physics. And physics doesn't file a ticket.*