The Havok String Bug That Wasn't

Four symptoms, one pairs(_G) error, and the forum report I almost filed.

The report that didn't ship

Late on a Wednesday night, I was about two clicks away from posting a detailed bug report to the ESOUI forums. The draft was titled "GetAddOnRootDirectoryPath — Lua string method failure on API return value." It had a reproduction, a hypothesis, control-group comparisons, and two competing explanations grounded in things I'd read on the ESOUI wiki. It looked, for what it's worth, like a pretty good bug report.

The bug it described did not exist.

This is the story of how I spent roughly twenty hours chasing a Havok Script engine quirk that wasn't there, how the real cause turned out to be a single line in a completely different file, and how that one line produced four separate symptoms that each looked like an independent problem. It's also a case study in the difference between reproducing a bug and understanding it — a distinction I thought I already understood before this week.

If you write ESO addons and you've ever iterated pairs(_G), you should probably read to the end. If you don't write ESO addons, the general debugging lessons are transferable and the detective story is, I hope, entertaining.

What I was building

The project is called AddOn Conflict Inspector (ACI for short, internal name ZZZ_AddOnInspector). It's a diagnostic tool for regular ESO players — not developers — who want to know why their game feels sluggish, which of their libraries are actually being used, whether two addons are fighting over the same SavedVariables, and so on. Think of it as a "health checkup" slash command.

Structurally it's five Lua files: Core (lifecycle and utilities), Hooks (the runtime instrumentation that intercepts event registrations and ZO_SavedVars calls), Inventory (static metadata collection via GetAddOnManager()), Analysis (clustering, dependency graphs, health scoring), and Commands (the /aci slash command suite — eleven subcommands at last count). The intended flow is simple: on EVENT_ADD_ON_LOADED, install hooks and register commands; on EVENT_PLAYER_ACTIVATED, collect a snapshot of the loaded environment and print an initial report.

I'd finished Phase 0 (proof-of-concept, three hypotheses validated) and Phase 1 (the full five-module implementation) and was getting ready to call it done. I had about 75 addons in my test environment, 227 tracked event registrations, and a working health score that classified my environment as "yellow — normal range for 1–2 months after a patch." Everything looked fine.

Everything was not fine.

Part 1: the first symptom

The first thing that tipped me off was a small, almost cosmetic issue. ACI has an "embedded sub-addon" concept. Some addons ship multiple logical components as subfolders — HarvestMap, for instance, deploys HarvestMap/ at the top level and then separately ships HarvestMap/Modules/HarvestMapDLC/, HarvestMap/Modules/HarvestMapNF/, and so on. Each of these shows up in the addon manager as a distinct entry, but they're not really independent addons and shouldn't be counted as out-of-date standalone packages when they're just riding along with a parent.

The detection is trivial: check whether the addon's root directory path contains another / after /AddOns/. In pseudocode:

local pos = rootPath:find("/AddOns/", 1, true)
if pos then
    local after = rootPath:sub(pos + 8)
    if after:find("/", 1, true) then
        -- embedded
    end
end

I wrote this inside CollectMetadata, which runs on PLAYER_ACTIVATED, and the output said: zero embedded sub-addons. In an environment where HarvestMap alone contributes ten, this was obviously wrong.

So I tried a second approach. Then a third. Then a fourth. Each variant computed isEmbedded in a slightly different way. Each produced embeddedCount = 0. Four independent string-manipulation strategies, all failing on data I could clearly see in the addon manager.

Then I added a /aci dump command that serialized the collected metadata to SavedVariables for offline inspection, and — importantly — recomputed the embedded flag during serialization, using the same logic, on the same rootPath values. That one said embeddedCount = 10. Correct.

So I had two calls to essentially the same code, operating on what should have been the same data, producing different answers. Inside CollectMetadata during PLAYER_ACTIVATED: all four attempts failed. From a slash command after everything had loaded: worked perfectly.

This is where a reasonable person starts forming hypotheses.

Part 2: the Havok hypothesis

My first hypothesis was that the strings I was getting from GetAddOnRootDirectoryPath(i) weren't actually normal Lua strings — or at least not entirely normal Lua strings. The reasoning went like this: if I call the API and immediately try rootPath:find("/") on the return value, it fails. If I take the same return value, stuff it into a table field, and later call :find on the table field, it works. Therefore, there's something special about the "direct return" version that gets sanitized by the table round-trip.

This wasn't unreasonable. ESO runs on Havok Script, a Lua 5.1 variant with ZOS's own modifications on top. And there's a small but real history of ESO addon authors hitting odd string-method edge cases.

Tech Box 1: ESO's Havok Script engine

ESO uses Havok Script, a commercial Lua 5.1 implementation from Havok (now part of Microsoft). For addon authors, it looks and behaves like standard Lua 5.1 about 99% of the time — the remaining 1% is where things get interesting.

The ESOUI wiki has a page on the embedded security system that describes how ZOS locks certain methods ("protected" for combat-unsafe calls, "private" for functions addons can't call at all). There are also sporadic forum threads going back to 2014 about string.find crashing the client on certain non-ASCII inputs, string.gsub behaving strangely with specific patterns, and so on. Most of these are long-since fixed, and the ones that aren't typically involve UTF-8 sequences rather than pure ASCII.

None of the historical string-method quirks I could find matched my situation exactly. But they were close enough to be suggestive. The pattern of "API return value behaves differently than a stored copy" felt like something a scripting engine with custom memory management might plausibly do.

That's the first warning sign I should have heeded, and didn't: "plausible" is not "proven."

By the end of Tuesday night I had collected five pieces of evidence that the Havok string bug was real:

1. Specific and reproducible. The failure happened on every run, on every embedded sub-addon, across four different string-manipulation strategies.
2. Had a clean workaround. Copying the value through a table field made the methods work.
3. Plausibly documented. ESOUI wiki acknowledges Havok Script's security system and method protection.
4. Historical precedent. Multiple forum threads over the years describing string method oddities.
5. Coherent mechanism. A custom string representation that gets normalized during Lua table insertion is at least internally consistent as a hypothesis. Five pieces of evidence, none of which contradicted the hypothesis. I wrote up a detailed bug report draft with a reproduction, a proposed root cause in two flavors (an internal string type vs. a Lua string interner quirk), and the workaround I was already using in production. I was going to clean it up in the morning and post it.

Tech Box 2: evidence vs. proof

Here is where I need to stop and name the mistake I was making, because it took me most of the next day to see it.

Five observations that are consistent with a hypothesis are not the same as five observations that confirm it. They're just observations the hypothesis has survived. For confirmation, you need at least one experiment that the hypothesis could have failed — an experiment where, if the hypothesis is wrong, you'd see a specific result different from the one you're predicting.

Karl Popper made this point about scientific theories in the 1930s, and it's been part of the standard philosophy-of-science curriculum ever since. It's also one of those things that's easy to agree with in the abstract and hard to actually practice when you're tired and you have a hypothesis that feels right. Every scrap of evidence I'd collected was of the form "is this observation compatible with the Havok theory?" (yes, yes, yes, yes, yes). I hadn't asked "is there any observation that would force me to abandon it?"

Popper's test is sometimes called "risky prediction." A good hypothesis sticks its neck out: it makes a specific prediction that would look really bad for the hypothesis if it turned out wrong. My Havok theory made no such prediction. It just kept comfortably absorbing every new observation into its explanatory cushion.

This is, in debugging terms, the signature of a bad hypothesis that hasn't been challenged hard enough yet.

Before clicking post on the bug report, I decided to run one more experiment. I built a control panel inside ACI: a new /aci debug command that would, right at the moment of failure, exercise ten different string operations on the same rootPath values and log the results.

The tests ran from T1 to T10:

• T1: rootPath:find("/AddOns/") on the raw API return.
• T2: tostring(rootPath):find("/AddOns/").
• T3: (rootPath .. ""):find("/AddOns/") — force a concat copy.
• T4: table.p:find("/AddOns/") — stuff it into a table first, then search.
• T5: rootPath:match("/AddOns/(.+)") on the raw return, with a capture group.
• T6: Same match, but via the table field.
• T7: rootPath:gsub("/", "") on the raw return, count the replacements.
• T8: Same gsub via the table field.
• T9: name:find("a") — a sanity check on the addon name string.
• T10: #rootPath vs #t.p — length comparison. If the Havok hypothesis was right, the raw-API tests (T1, T5, T7) should fail or return nonsense, and the table-routed tests (T4, T6, T8) should succeed. That was the prediction. That was the risky part.

The results came in and they killed the hypothesis stone dead.

T1 through T10 all succeeded on the raw API return values. find worked. match worked — with the capture group. gsub worked. The length was identical to the table-copied version. type(rootPath) was "string". Every raw-vs-table comparison showed the two values were indistinguishable.

There was one oddity in T7: the raw gsub count came back as zero, even though T8 (the table version) correctly reported three. For about thirty seconds this looked like a last-minute save for the Havok hypothesis. Then I looked at the test code I had written for T7:

local _, rawGsub = rootPath and rootPath:gsub("/", "") or "", 0

Lua operator precedence: and and or bind tighter than the comma. So this parses as

local _, rawGsub = (rootPath and rootPath:gsub("/", "") or ""), 0

The entire and/or expression produces a single value (the substituted string), which gets assigned to _. The , 0 then assigns the literal zero to rawGsub. Every time. Forever. The "failure" in T7 was me writing a Lua gotcha into my own control experiment.

So: Havok's string handling was fine. The API return values were fine. My test code had one bug, which I fixed. And the hypothesis I was about to file a bug report about collapsed.

Which left me with an uncomfortable question. The original embedded-detection failure was still real. I'd seen it. So if Havok wasn't the cause, what was?

Part 3: the maze

By this point it was Wednesday, and I had a new kind of problem. Instead of one mysterious symptom, I suddenly had several.

Symptom two: some of my Core and Inventory code changes didn't seem to be taking effect in the running game. I'd edit a function, redeploy, reload the UI, and the new behavior wasn't there. Meanwhile, changes to Analysis and Commands reflected immediately. Why would two files out of five fail to update?

Symptom three: the load banner message — [ACI] v0.2.0 loaded. Hooks: ... — wasn't appearing in chat. It had never appeared, not once, in any of my test sessions. I'd been telling myself this was a minor display glitch. In hindsight, it was the loudest alarm in the whole building.

Symptom four: remember the Havok issue itself. Four attempts to compute isEmbedded, all failing. That was still unexplained.

I approached these one at a time, starting with the "code isn't reflecting" problem, because it presented the clearest debugging surface. The hypotheses were obvious and the debugging was mechanical.

I swept through the obvious candidates quickly. A PowerShell timestamp check ruled out a partial deploy: all six files had identical modification times, down to the second. A recursive Get-ChildItem turned up exactly one copy of the addon folder, not two. A search across C:\Users\user\ for any ZZZ_AddOnInspector.addon file came up clean — no OneDrive shadow copy intercepting the Documents path, despite real precedent for that on the ESOUI forums.

Hypothesis: BOM or encoding mismatch in the source files. This one was plausible enough that I read the first three bytes of every .lua file in the deployment folder. 83 89 78 — that's S Y N, which was the start of the word "SYNTAX" from a diagnostic line I'd just inserted (more on that in a second). No UTF-8 BOM. The manifest was clean too.

Hypothesis: the manifest doesn't list the files. I catted the .addon file. All five Lua files listed in the right order, spelled correctly, standard header lines, no weird control characters.

Hypothesis: the files are on disk but ESO isn't reading them. This one I could test directly. I inserted a line of intentionally invalid Lua — literally SYNTAX_ERROR_FORCE_CRASH as the first line — into the deployed ACI_Core.lua. If ESO was reading the file, this would produce a Lua error popup at startup. If it wasn't, nothing would happen.

Nothing happened.

No error popup. Game started clean. This seemed like a smoking gun for "ESO isn't reading this file at all," which would have meant the remaining explanation was some kind of addon-manager cache I hadn't found yet. I started researching AddOnSettings.txt, bytecode caches, and addon registration internals.

But I was wrong about the smoking gun, and in a way that took another hour to untangle. When I actually restarted the game after inserting the syntax error, /aci stopped working — which meant the file was being loaded, but the Lua error wasn't surfacing anywhere I could see. ESO has several ways for addon errors to be suppressed in the UI, and apparently my environment was suppressing this one silently. The file was being parsed, parsing was failing, the whole file was being skipped, and the only externally visible consequence was that the rest of Core never defined its functions — hence /aci breaking.

Tech Box 3: silent failures in the ESO addon lifecycle

There are at least three different ways an ESO addon can fail quietly, and understanding them would have saved me a lot of time:

1. Pre-chat d() calls disappear. ESO's d() function, the standard way to print to the chat window, is documented on the ESOUI wiki as only being reliably available "after the loaded events have been fired." If you call d() too early in the addon lifecycle — before the chat frame is initialized — the call silently drops. No error, no queued output. This bit me because my load banner was in the EVENT_ADD_ON_LOADED callback, and I kept telling myself "the chat just isn't ready yet, that's fine" whenever the banner didn't appear. It was not fine. The banner's absence was information, and I was ignoring it.

2. Lua errors during parse are suppressed in some environments. Depending on your ESO settings and whether you have a library like LibDebugLogger installed (or not installed), parse-time errors in addon files can be hidden from the user. The file is skipped, the functions it defines never get defined, and if you don't happen to call one of those functions immediately, you may not notice anything is wrong.

3. Event callbacks die silently mid-execution. This is the big one, and the one that finally cracked the case. When a function registered via EVENT_MANAGER:RegisterForEvent throws an error, ESO catches it internally. The callback doesn't get to finish. Any code after the error point doesn't run. But crucially, other code paths — slash commands, other registered callbacks, hooks installed before the failure — continue to work normally. The addon appears partially functional, which is worse than being obviously broken, because partial function disguises the failure.

If you combine all three of these, you get an addon that loads, registers its commands, partially collects data, prints nothing at startup, and appears to work correctly from the user's perspective — while actually being completely broken in the one codepath you care about. That was ACI, all week.

Part 4: the real cause

I'd been poking at this for hours. I'd ruled out deployment, duplicate folders, OneDrive, BOM, encoding, manifest, addon settings, and bytecode caches. I was starting to construct more exotic theories about Havok's Lua VM behavior when I finally stopped and did the thing I should have done much earlier: I traced through the PLAYER_ACTIVATED callback line by line, looking not at what each line did but at whether it ran at all.

The answer came almost immediately. I added a diagnostic field that got bumped at each step of the callback — _step = 1, _step = 2, and so on — and checked what value it reached when the callback "finished." It reached 1. The very first substantive line, a call to ACI.BuildEventNameMap(), was where everything died.

Here's the function, in its pre-fix form:

function ACI.BuildEventNameMap()
    local map = {}
    for k, v in pairs(_G) do
        if type(v) == "number" and type(k) == "string" and k:sub(1, 6) == "EVENT_" then
            map[v] = k
        end
    end
    return map
end

Completely standard Lua. Walk the global table, find everything that looks like an EVENT_* numeric constant, build a reverse lookup so we can turn event codes back into names for reporting. This pattern exists in many ESO addons. It's not even clever.

But ESO's global table contains ZOS-locked entries. The security system described in Tech Box 1 isn't just a documentation quirk — it actually intercepts Lua accesses on specific globals at runtime. When pairs(_G) walks into one of those during iteration, the access raises a Lua error. The for loop aborts. The error propagates up to the enclosing callback.

And where was this function being called from? Look at the PLAYER_ACTIVATED handler in Core.lua:

EVENT_MANAGER:RegisterForEvent(ACI.name, EVENT_PLAYER_ACTIVATED, function()
    EVENT_MANAGER:UnregisterForEvent(ACI.name, EVENT_PLAYER_ACTIVATED)
    EVENT_MANAGER:UnregisterForEvent(ACI.name .. "_LoadOrder", EVENT_ADD_ON_LOADED)
 
    -- Build event name lookup
    ACI.eventNames = ACI.BuildEventNameMap()    -- ← dies here
 
    -- Collect static metadata
    ACI_SavedVars.metadata = ACI.CollectMetadata()  -- ← never runs
    ACI_SavedVars.svHookOk = ACI.svHookInstalled
 
    ACI.PrintReport()                                -- ← never runs
    d("[ACI] |c00FF00/aci|r for the latest stats.")  -- ← never runs
end)

BuildEventNameMap was the first line in the post-activation callback. When it errored, the entire remainder of the callback was skipped. CollectMetadata never got called. PrintReport never got called. The final d() banner that was supposed to tell the user "ACI is ready" never got called.

Let me show you how a single pairs(_G) error produced four visually distinct symptoms.

One root cause, four symptoms

Every one of these symptoms had a perfectly plausible individual explanation. Every one of those individual explanations was wrong. The thing that unified them was invisible from any individual symptom's vantage point: the failing line lived in a different function from all of them.

The most humbling part, looking back, is that the dump-vs-inline discrepancy — the single most diagnostic clue in the whole investigation — was the one I interpreted most wrong. "Same logic, different result" should have made me ask "are they actually running in the same execution context?" Instead I read it as "there must be something different about the data." The data wasn't the problem. The context was. I was looking at the wrong axis.

Tech Box 4: the fix, and better patterns

The minimal fix is a pcall wrapper:

function ACI.BuildEventNameMap()
    local map = {}
    local ok, err = pcall(function()
        for k, v in pairs(_G) do
            if type(v) == "number" and type(k) == "string" and k:sub(1, 6) == "EVENT_" then
                map[v] = k
            end
        end
    end)
    if not ok then
        -- Iteration aborted partway; map may be partially populated.
        -- Log for diagnostics but don't propagate.
    end
    return map
end

This stops the error from killing the enclosing callback. It has a cost: because pairs(_G) aborts at the first protected global it hits, you only get entries enumerated before that point. If the protected entry comes early in the iteration order, your map will be mostly empty. In my environment the map ends up with 75+ entries, enough for the reporting code to produce readable output, but that isn't guaranteed across every machine or every ESO build.

A more robust approach is to guard each access individually:

function ACI.BuildEventNameMap()
    local map = {}
    for k in pairs(_G) do
        if type(k) == "string" and k:sub(1, 6) == "EVENT_" then
            local ok, v = pcall(function() return _G[k] end)
            if ok and type(v) == "number" then
                map[v] = k
            end
        end
    end
    return map
end

This pattern iterates keys only, which is typically safer, and wraps the value fetch in its own pcall. A single protected entry skips just itself instead of aborting the whole iteration. The downside is the per-access overhead, but for a one-time-at-startup call over a few thousand globals it's negligible.

The most robust approach is to not iterate _G at all. EVENT_* constants are well-defined; you can pre-enumerate the ones your addon actually uses:

local KNOWN_EVENTS = {
    "EVENT_ADD_ON_LOADED",
    "EVENT_PLAYER_ACTIVATED",
    "EVENT_PLAYER_COMBAT_STATE",
    "EVENT_COMBAT_EVENT",
    "EVENT_EFFECT_CHANGED",
    -- ...
}
 
function ACI.BuildEventNameMap()
    local map = {}
    for _, name in ipairs(KNOWN_EVENTS) do
        local code = _G[name]
        if type(code) == "number" then
            map[code] = name
        end
    end
    return map
end

This has zero chance of hitting a protected global (you're only reading ones you explicitly named) and the list doubles as documentation. The downside is maintenance: ESO adds events in patches, and your list will drift. For a diagnostic tool like ACI that wants to display any event some other addon might register for, full enumeration is actually useful, so the pcall version is a better fit.

For the record, I went with the pragmatic single-pcall-around-the-loop version. It's the smallest possible diff, it solves the problem, and if future ESO patches introduce additional protected globals that land in awkward iteration positions I'll revisit.

A broader lesson for ESO addon code: any time you're iterating _G, assume it can throw. Wrap it. This is a general pattern for working with sandboxed environments — the sandbox boundary isn't always where you expect, and you don't want your entire callback chain dying because you walked into a locked room during routine iteration.

Five lessons

I'm going to try to state these as cleanly as I can, because they apply to more than just ESO addon development.

1. Reproducing a bug is not the same as understanding it. I could reproduce the embedded-detection failure on demand, every single time, across four different code paths. That's usually considered a strong position to debug from. In this case the reproduction was itself a product of the bug — I was reliably triggering a non-execution, and mistaking it for a reliable execution with wrong output. "Reliable" is a property of the setup, not the hypothesis.

2. Controlled experiments kill hypotheses; observations grow them. The Havok hypothesis survived five consecutive pieces of "evidence" because each of those observations was me looking for consistency, not contradiction. The hypothesis died the moment I designed a test that could have failed it — the T5 match-with-capture-group experiment — and it failed it immediately. The ratio of hours-spent-gathering-support to hours-spent-trying-to-falsify is the clearest diagnostic for whether you're doing science or theater.

3. Cached data wears a mask of normalcy. One of the reasons I spent a week thinking ACI worked was that it kept producing plausible-looking output whenever I ran /aci. What I eventually realized is that this output was almost entirely being served from SavedVariables written by previous sessions. The initial collection pass on PLAYER_ACTIVATED had been silently failing since Phase 0, but the stale cached metadata from the one time it sort-of ran during development was persistent enough to hide the failure. If you have a system with persistent state and one of your pipelines can fail, make sure the failure surfaces loudly — or you'll be debugging the wrong version of the program.

4. Wrong-direction debugging is exponentially more exhausting than right-direction debugging. Twenty hours on Havok plus filesystem plus OneDrive plus BOM. Fifteen minutes on the actual cause once I stopped asking what the code was doing wrong and started asking whether it was running at all. This isn't because the bug was easy to find — it's because once you're pointed at the right function, the problem is immediately legible and the fix is mechanical. Mental energy spent on wrong hypotheses is the single largest cost in debugging and it's not a linear cost. Do whatever you can to shorten the "looking in the wrong place" phase, even if it means pausing to ask yourself "is there any part of the code I haven't questioned yet?"

5. Fail early, even if it hurts. The pairs(_G) bug had been in BuildEventNameMap since the Phase 0 proof-of-concept. It sat dormant for weeks because the symptoms were ambiguous and I'd conditioned myself to ignore them. Finding it in Phase 2 Step 0 — while doing unrelated cleanup — was almost entirely luck. If I'd added aggressive error logging to the callback earlier, or tested with an empty SavedVariables file more often, or just paid attention to the missing load banner, I would have caught it months earlier. Bugs that hide and mature are worse than bugs that blow up immediately.

Coda: the report that didn't ship

The ESOUI forum bug report draft is still sitting in my Notion. It's tagged WITHDRAWN now, with a final comment explaining the actual root cause and a link to this post. I'm going to leave it there instead of deleting it, because it's a useful reminder of how close I came to publishing a confident, well-written, superficially rigorous bug report about a bug that didn't exist.

If I'd shipped it, a reasonable ESOUI maintainer would have spent some nonzero amount of time trying to reproduce it, failed, asked me for more information, and eventually concluded that either I was mistaken or they were looking at a subtly different environment. That would have cost them time and me credibility. The fact that the report didn't ship isn't a near-miss — it's the most valuable outcome of the entire debugging session, because it's the outcome that preserves the option to file real bug reports in the future and be taken seriously.

The actual fix — the one-line pcall wrapper around a for loop — took about ninety seconds to write. ACI v0.2.0 now loads cleanly, reports 75 addons and 227 events, and prints the banner I spent a week not seeing:

I've looked at that banner a lot over the past 24 hours. It's the most satisfying six lines of diagnostic output I've ever produced.

ACI is a work-in-progress diagnostic tool for ESO addon environments, currently at v0.2.0. Source is local for now; a GitHub repository and ESOUI release are planned once Phase 2 stabilizes. If you're an ESO addon author and you've hit pairs(_G) errors yourself, I'd love to hear how you solved them — the set of "globals that throw on iteration" seems to vary between environments and I don't have a complete picture yet.

AddOn Conflict Inspector