themes from a name corpus to the shopper desk. Prospects with out transcripts get NULL. NULL will get full of zero, or with “no problem talked about,” or quietly omitted as a reference class. In a single line of preprocessing, the pipeline converts didn’t name assist into didn’t expertise billing frustration.
The regression that follows appears clear. The coefficient on “billing frustration” is critical, signed the best way the product workforce anticipated, massive sufficient to matter. It will get pasted right into a roadmap doc. No one asks the place the variable got here from.
This text is about what received smuggled in with that fill worth, and about three different strikes that look simply as innocuous in a pocket book however relaxation on assumptions the evaluation by no means names. The setup isn’t particular to assist calls. It applies to speak logs, ticket summaries, product evaluations, gross sales transcripts, and survey free-response fields, wherever a contemporary pipeline turns textual content right into a tidy column. The pipeline could possibly be a fine-tuned classifier, a zero-shot LLM, or an embedding-plus-cluster. The conceptual drawback is similar: the column isn’t an statement of a buyer attribute. It’s the output of a generative course of utilized to a self-selected subset of buyer conduct.
Practitioners more and more deal with outputs like these as in the event that they had been direct readings of buyer state. They don’t seem to be. They’re generated variables: measurements produced by a pipeline, conditional on a buyer doing one thing that left a textual hint, conditional on that hint surviving the extraction mannequin. Each step of that conditional has penalties for what the variable means in a downstream causal mannequin, and most of these penalties are invisible within the joined desk.
4 issues are inclined to go mistaken, and the NULL transfer makes all 4 seen directly.
Choice. A theme exists for a buyer as a result of that buyer referred to as, complained, posted, or replied. No matter drove that motion can also be in all probability correlated with the therapy, the result, or each. The NULL fill collapses “didn’t generate textual content” into the reference class, and the evaluation is now not estimating an impact over the shopper base. It’s estimating an impact over a redefined inhabitants, and the redefinition occurred in preprocessing.
Timing. Was the decision earlier than the therapy, throughout it, or after? Pre-treatment textual content is a candidate confounder. Submit-treatment textual content is a candidate mediator or end result, and treating it as a pre-treatment management is a basic supply of post-treatment bias. The joined desk not often makes this seen.
Measurement. The label “billing frustration” isn’t billing frustration. It’s what the pipeline detected as billing-frustration-shaped language. Classifier accuracy is finite, and accuracy can differ throughout therapy arms, as a result of a therapy that adjustments how prospects speak additionally adjustments how the mannequin reads them. The label noise isn’t orthogonal to the factor being studied.
Position. Is the theme appearing as a confounder, a mediator, a therapy, an end result, or a descriptive characteristic? The DAG decides this, not the column title. A variable that’s methodologically legitimate in a single function turns into a bias supply in one other.
These 4 issues will not be impartial. They work together. An LLM-detected theme inherits a range footprint from the channel it got here via, a timing footprint from when the textual content was generated, and a measurement footprint from the pipeline that extracted it. The downstream regression sees a column of zeros and ones.
The issue isn’t that the pipeline produced a nasty label. The issue is that the label inherited a data-generating course of the downstream evaluation by no means modeled.
The remainder of this text works via what which means in apply, the place the usual workflow goes mistaken, and what the minimal diagnostic appears like. We begin with the role-and-timing query, as a result of it’s the one analysts get mistaken first.
Position and timing are the identical query
The primary transfer an analyst makes with a transcript-derived theme is implicit: they deal with it as a covariate. Themes go into the right-hand aspect of the regression. The therapy is the variable of curiosity. The end result is on the left. The theme is “managed for.”
That phrase, “managed for,” is doing work the analyst hasn’t checked. Controlling for a variable adjusts away the a part of the treatment-outcome relationship that flows via it. Whether or not that adjustment helps or hurts relies upon completely on the place the variable sits within the causal graph, and that place is decided by timing.
Pre-treatment textual content, generated earlier than the therapy was assigned, can play the function of a confounder. If a buyer referred to as about billing in January and the retention supply went out in March, the decision captures one thing about buyer state that will affect each who received the supply and who churned. Conditioning on the theme right here can scale back bias from omitted variables, supplied the theme really proxies for the related assemble and the choice points within the subsequent part are dealt with.
Concurrent textual content, generated as a part of the therapy itself, isn’t a covariate in any respect. If the therapy is a name from a retention agent and the theme comes from that very same name, the theme is a part of the intervention. Conditioning on it doesn’t regulate for confounding; it removes a part of the impact the analyst is attempting to measure.
Submit-treatment textual content, generated after the therapy, is essentially the most harmful class, as a result of it’s the one more than likely to be misclassified as a confounder by an analyst working from a flat desk with no time index. A buyer who obtained a retention supply in March and referred to as complaining in April produced a transcript that displays, at the very least partially, their response to the therapy. Conditioning on a theme extracted from that decision is conditioning on a post-treatment variable. That may block mediation paths, induce collider associations, or in any other case shift the estimand away from the therapy impact the analyst thinks they’re estimating.
A labored instance makes this concrete. Take into account an artificial however business-realistic setup. Prospects are focused right into a retention supply primarily based on a mannequin that picks up value sensitivity. Each the supply project and buyer churn rely upon this underlying value sensitivity, which the analyst doesn’t observe. Prospects who’re extra price-sensitive usually tend to obtain the supply (as a result of the concentrating on mannequin chosen them) and extra more likely to churn regardless. They’re additionally extra more likely to name assist and specific invoice shock. The theme “invoice shock” is generated from these post-treatment calls.
The naive analyst joins the theme onto the shopper desk, fills NULL as zero, and runs a logistic regression of churn on supply plus bill-shock:
import numpy as np
import pandas as pd
import statsmodels.api as sm
rng = np.random.default_rng(7)
n = 20000
price_sens = rng.regular(0, 1, n)
supply = rng.binomial(1, 1 / (1 + np.exp(-(0.8 * price_sens))))
churn = rng.binomial(1, 1 / (1 + np.exp(-(-1.0 + 1.2 * price_sens - 0.5 * supply))))
referred to as = rng.binomial(1, 1 / (1 + np.exp(-(-1.5 + 0.7 * price_sens + 0.9 * churn))))
theme_prob = 1 / (1 + np.exp(-(-0.5 + 0.8 * price_sens)))
bill_shock = np.the place(referred to as == 1, rng.binomial(1, theme_prob), 0)
df = pd.DataFrame({"churn": churn, "supply": supply, "bill_shock": bill_shock})
X = sm.add_constant(df[["offer", "bill_shock"]])
naive = sm.Logit(df["churn"], X).match(disp=0)
print(naive.params)The true impact of the supply on churn is −0.50 in log-odds. The supply is meant to cut back churn, and within the data-generating course of it does. Here’s what 4 specs return:
Picture by Creator
| Specification | Supply coefficient | What it says |
| Naive (with bill_shock) | +0.12 | Supply seems dangerous |
| Dropped (no bill_shock) | +0.24 | Supply nonetheless seems dangerous |
| Oracle (with price_sens) | −0.55 | Supply reduces churn |
| True impact (DGP) | −0.50 | Supply reduces churn |
As a result of supply project is already confounded by value sensitivity, eradicating the dangerous management doesn’t make the design legitimate. It solely removes one further supply of distortion. Two observations from this desk.
First, the naive specification is mistaken in path. Including the bill-shock management to a mannequin that was already biased flips the signal on the supply coefficient. The product workforce studying this output would conclude that retention affords trigger churn. They’d be mistaken.
Second, dropping the bill-shock variable doesn’t repair the evaluation. The dropped specification can also be constructive, and solely the oracle specification, which circumstances on the unobserved confounder instantly, recovers the true impact. In an actual evaluation the analyst doesn’t have that column. That’s the level. Eradicating a nasty management is important however not adequate, and a post-treatment theme extracted from a self-selected calling subpopulation isn’t an alternative choice to identification.
The mechanism behind the signal flip within the naive specification is value strolling via. Churn impacts the chance of calling, as a result of prospects who’re leaving usually tend to name. Invoice-shock is barely noticed for patrons who referred to as, because the theme requires a transcript to exist. Conditioning on bill-shock due to this fact circumstances on a downstream consequence of churn. Amongst prospects with bill-shock equal to 1, the connection between supply and value sensitivity has been distorted, as a result of each variables now assist clarify why the shopper ended up flagged. The coefficient on supply absorbs that induced affiliation.
The methodological level generalizes. A transcript-derived variable has a place within the causal graph decided by when the textual content was generated relative to the therapy, who generated it, and what course of produced the label. Position and timing are the identical query considered via completely different lenses. These variables include a structural footprint the analyst is accountable for tracing, and the joined desk isn’t the place the tracing occurs.
The choice query
Most business analyses utilizing assist transcripts implicitly redefine the inhabitants from “prospects” to “prospects who generated assist language.” The estimand adjustments earlier than the regression even begins.
That is the half that tends to matter most in practitioner workflows, and it’s the place the usual workflow is most fragile.
The textual content exists as a result of the shopper did one thing: referred to as, posted, complained, replied. That one thing is a conduct, not a measurement. It’s influenced by buyer traits, by the channel that was obtainable, by the urgency of the underlying problem, and infrequently by the therapy itself. None of those are random. None are usually orthogonal to the result.
The NULL dealing with resolution is the place this turns into operational. There are three widespread strikes, and every carries an assumption.
Filling NULL as zero or “no problem talked about” assumes that not producing textual content is informative concerning the absence of the underlying assemble. The analyst is claiming that prospects who didn’t name didn’t expertise the factor the theme is detecting. For many themes value detecting, that is implausible on its face. Prospects who didn’t name might have skilled billing frustration and resolved it by canceling, by switching to a competitor, by complaining on social media, or by giving up. The zero-fill turns all of those into “no frustration.”
Dropping rows with NULL themes, proscribing the evaluation to the calling subpopulation, is at the very least trustworthy concerning the inhabitants, nevertheless it adjustments the estimand. The therapy impact amongst prospects who referred to as isn’t the therapy impact amongst prospects, and the distinction between the 2 is commonly all the level of the enterprise query. A retention supply’s impact on churn-prone callers is a helpful amount. It’s not the amount most analyses declare to estimate.
Treating text-presence as a missingness mechanism and making use of inverse chance weighting primarily based on a mannequin of who calls is, methodologically, the fitting form of transfer. The catch is the propensity mannequin itself. Modeling who generates textual content requires writing down what drives calling, and that mannequin is dependent upon demographics, tenure, prior points, therapy publicity, and unmeasured frustration, which is the assemble the theme was supposed to assist measure within the first place. The IPW transfer is principled, and additionally it is not often as principled because it appears.
The deeper level is that choice into textual content is a conduct that interacts with the therapy. A retention supply might change calling charges. A pricing change might change grievance charges. A characteristic launch might change the sorts of points prospects articulate. Any of those makes the choice mechanism itself treatment-dependent, which suggests even a superbly extracted, completely timed theme is being measured on a inhabitants whose composition shifts with the therapy. Customary observational corrections assume the choice mechanism is steady. When the therapy strikes the choice, the corrections don’t.
None of this implies transcript-derived variables are ineffective. It means the analyst owes the reader an specific assertion of which inhabitants the evaluation is estimating an impact over, what mechanism produced the textual content, and what assumption was made about everybody whose textual content doesn’t exist.
The measurement query
Outdated NLP outputs appeared noisy. TF-IDF weights, sparse key phrase counts, LDA matter vectors: none of them appeared like issues a buyer felt. Practitioners distrusted them by reflex, and that reflex saved lots of dangerous analyses.
LLM outputs don’t look noisy. They seem like latent constructs. A label like “billing frustration” or “belief erosion” or “renewal anxiousness” reads like an outline of a buyer’s psychological state. The label is articulate, the classes are semantically coherent, and the failure modes don’t announce themselves within the column. The persuasion drawback is actual earlier than the statistical drawback begins.
The statistical drawback is extra acquainted. An LLM theme is a loud proxy for the underlying assemble. The label “invoice shock” isn’t invoice shock. It’s what the mannequin determined was bill-shock-shaped language within the transcripts it processed. Classifier accuracy is finite even for clear duties, and the accuracy on the precise inhabitants, not the held-out analysis set, is commonly unknown. Plugging a loud proxy right into a regression instead of the true variable attenuates coefficients towards zero in some setups and distorts them in others, relying on whether or not the noise is differential.
Differential measurement error is the place the true harm lives. If a therapy adjustments how prospects speak, and most therapies value operating do, then the classifier’s accuracy on theme detection can differ between therapy and management. A retention supply that softens buyer sentiment might scale back the speed at which the mannequin flags “invoice shock” language with out lowering the underlying frustration. A pricing change that shifts how prospects articulate complaints might transfer classifier accuracy extra in a single arm than the opposite. The label noise is now not mean-zero. It’s correlated with the therapy, and conditioning on the noisy label biases the estimated therapy impact in a path the analyst can’t simply signal.
There’s a literature on correcting for classifier-induced measurement error. Egami and colleagues develop a split-sample workflow for causal inference with text-discovered measures as therapies or outcomes in “How to Make Causal Inferences Using Texts”. Mozer and colleagues apply text-augmented matching to digital well being data and present how text-based covariates change estimated results in an actual medical research in “Leveraging text data for causal inference using electronic health records”. For the broader panorama, Keith, Jensen, and O’Connor overview how textual content has been used to take away confounding throughout functions in “Text and Causal Inference: A Review of Using Text to Remove Confounding from Causal Estimates”. These strategies exist, and they’re value utilizing when the evaluation issues. In addition they require the analyst to acknowledge {that a} label is a measurement with error within the first place, which is the transfer most workflows skip.
The practitioner mistake isn’t utilizing the label. The practitioner mistake is treating a label that got here out of a generative mannequin as if it had been a column learn off a sensor.
A practitioner guidelines
A causal evaluation that makes use of a generated variable derived from transcripts can nonetheless be defensible. It simply has to reply 5 questions earlier than the regression runs.
1. What function am I assuming this variable performs?
Confounder, mediator, therapy, end result, or descriptive characteristic. The DAG decides. The column title doesn’t.
2. When was the textual content generated relative to the therapy?
Pre-treatment, concurrent, or post-treatment. If the analyst can’t reply this from the info, the variable doesn’t enter the mannequin as a confounder.
3. What choice mechanism produced the textual content, and what am I assuming about everybody whose textual content doesn’t exist?
Zero-fill, drop, IPW: every is an assumption. Decide one and state it.
4. How was the label produced, and will its reliability differ throughout therapy arms?
If the therapy plausibly adjustments how prospects specific the underlying assemble, classifier accuracy isn’t fixed throughout the comparability the evaluation is making.
5. What does the outcome seem like underneath a stress check?
Refit the mannequin with out the transcript-derived variable. If the headline coefficient is fragile, the outcome isn’t steady sufficient to hold a causal declare by itself.
These 5 questions will not be an answer. They’re a diagnostic. An analyst who can reply them isn’t assured an recognized impact. An analyst who can’t reply them is doing descriptive work with causal language connected.
The broader sample is older than LLMs. Generated variables are pipeline outputs that seem like observations however are literally mannequin outputs conditioned on choice. They present up in fraud scores, recommender relevance metrics, sentiment indices, propensity scores reused as covariates, and any latent-trait estimate produced by an upstream mannequin and consumed by a downstream evaluation. LLMs didn’t invent this error. They made it accessible at a scale and a fluency that older NLP outputs by no means reached. The labels seem like latent constructs, the columns seem like measurements, and the workflow appears like causal inference.
The assumptions didn’t disappear. They only moved upstream.
Workers Knowledge Scientist targeted on causal inference, experimentation, and resolution science. I write about turning ambiguous enterprise questions into decision-ready evaluation.
Extra like this on LinkedIn 👇
