Sentiment in AI Answers: How to Measure

Measure sentiment in AI answers by labelling every answer that mentions you as positive, neutral, mixed or negative, using written rules, and tracking the share in each group on a fixed prompt set. Label accuracy separately: an answer can be positive and wrong. Our view: check a sample by hand monthly and report sentiment with its sample size, because answers vary between runs.

Points marked "Our view" are interpretation; examples are illustrative. No public standard for AI answer sentiment exists, so the scheme below is ours.

On this page

• Why measure sentiment at all?
• What labelling scheme should you use?
• Should you label at answer level or mention level?
• How do you add accuracy?
• Can AI models label sentiment for you?
• How big does the sample need to be?
• How do you find the cause of negative sentiment?
• How should you report sentiment?
• Which pitfalls should you watch for?
• How does sentiment differ by engine?
• How do you set a target?
• How do you keep labellers consistent?
• How do you share findings with other teams?
• How often should you measure?
• Exercise
• Checklist
• From MentionX data: where sentiment questions arise
• Worked example: Ledgerline labels its first sentiment sample
• How do you write labelling rules?
• How do you check labeller agreement?
• What does an accuracy label add?
• How do you use AI to help label?
• How do you connect sentiment to sources?
• How do you report sentiment to leadership?
• When should you escalate?
• Common questions
• Related
• Sources

Checked on 31 October 2026. "Our view" marks MentionX opinion. Ledgerline is a fictional company used for illustration.

Why measure sentiment at all?

Being mentioned isn't the same as being recommended. An answer that says "Acme is popular but expensive and hard to set up" (fictional) names you and may still put buyers off. Our view: sentiment tells you how answers frame you, and the reasons behind negative framing usually point to specific sources you can investigate.

What labelling scheme should you use?

Write the rules down and give labellers five worked examples per label. Our view: four labels are enough. More categories make labels less consistent.

Should you label at answer level or mention level?

Our view: label at mention level when answers name you more than once, then roll up to answer level using the most negative mention. A single "but support is slow" in an otherwise positive answer is what buyers remember.

How do you add accuracy?

For each mention, also record whether the facts are correct: pricing, features, availability, company details. Use three labels: correct, incorrect, unverifiable. Our view: report "positive and accurate" as your headline sentiment number. A positive answer based on outdated facts can backfire when a buyer checks.

Can AI models label sentiment for you?

Yes, as a first pass. Give the model your written rules and examples, and ask for a label and a one-line reason. Then review a random sample of 50 labels each month by hand. Our view: models tend to mislabel hedged answers ("some users report…") and mixed answers. Keep the prompt and model version fixed so changes aren't caused by the labeller.

How big does the sample need to be?

Answers vary from run to run, so small samples swing. Our view: run each prompt at least three times per engine and aim for a few hundred labelled mentions per month before comparing periods. See statistical confidence in AI visibility tracking for how to judge whether a change is real.

How do you find the cause of negative sentiment?

1. Filter to negative and mixed answers.
2. Group the reasons: price, support, missing feature, reliability, outdated info.
3. Check which cited sources contain the same phrasing.
4. Decide whether the issue is real (fix the product), outdated (update sources) or wrong (correct the source).

How should you report sentiment?

• Share of mentions by label, with the sample size.
• "Positive and accurate" share as the headline.
• Top three negative reasons with example quotes.
• Change versus last month, flagged only when it's outside normal variation.

Which pitfalls should you watch for?

• Comparing sentiment across different prompt sets.
• Treating neutral as bad; listing you is often fine.
• Reporting tiny samples as trends.
• Ignoring accuracy.
• Changing labelling rules mid-quarter without noting it.

How does sentiment differ by engine?

Our view: each engine cites different sources, so sentiment can differ. If one engine is consistently more negative, check which sources it cites that others don't. The fix is usually a specific page or thread, not the engine itself.

How do you set a target?

Our view: set targets on the "positive and accurate" share and on the number of answers with incorrect facts, not on negative share alone. Some fair criticism will always appear, and that's healthy. A realistic quarterly goal is to remove the top one or two incorrect claims and see the share of accurate mentions rise on the same prompt set.

How do you keep labellers consistent?

Our view: have two people label the same 50 answers at the start of each quarter and compare. Where they disagree, discuss, then add the case to your examples. Track the agreement rate; if it drops, your rules need clarifying. Consistency matters more than any single label.

How do you share findings with other teams?

Our view: sentiment findings are most useful outside marketing. Send product the recurring weakness themes, support the complaints about service, and sales the objections that answers repeat. Include two or three real quotes per theme with the engine and date. Ask each team what they'd change, then check the same prompts after the change ships. Over a few quarters this turns sentiment tracking from a report into a feedback loop that improves both the product and the sources engines rely on.

How often should you measure?

Our view: monthly is enough for most brands. Weekly checks add noise without adding insight unless you're responding to a launch, a pricing change or a public incident. Keep the prompt set, engines and labelling rules fixed between runs so the comparison is fair.

Exercise

Run ten prompts three times in one engine. Label each mention of your brand as positive, neutral, mixed or negative, and as correct, incorrect or unverifiable. Calculate the "positive and accurate" share and list the top negative reason.

Checklist

• Written labelling rules with examples
• Accuracy labelled separately
• Multiple runs per prompt
• Monthly hand check of model labels
• Sample size reported with every number
• Negative reasons traced to sources

From MentionX data: where sentiment questions arise

Method. Read-only aggregate queries over MentionX scans from 22 August to 29 October 2026 (UTC): 2,217 engine runs (2,154 completed) across 18 anonymised projects and six engines.

Sentiment only matters where your brand is named. In our sample, mention rates differed sharply by prompt intent:

They also differed by engine:

Our view: measure sentiment separately for comparison prompts, where brands are almost always named and often contrasted, and for recommendation and best-of prompts, where being named at all is the first hurdle. Blending them hides where tone is the real issue.

Limits. A convenience sample of mostly B2B software projects, run through engine APIs, which can behave differently from the consumer apps. Perplexity, Copilot and DeepSeek figures come from a single day (26 October 2026), so treat them as a snapshot, not a trend.

Worked example: Ledgerline labels its first sentiment sample

Ledgerline is a fictional accounts-payable tool. Figures are illustrative.

Ledgerline pulls 150 answers that mention its brand from three engines and two labellers mark each mention as positive, neutral, mixed or negative, plus accurate or inaccurate.

Most mentions are neutral. Negative mentions cluster in one engine and repeat the same complaint about slow onboarding. The cited sources include a two-year-old forum thread and a review site page with old ratings. Several positive mentions are inaccurate, crediting Ledgerline with a feature it doesn't offer. The team updates its onboarding guide, asks recent customers to leave reviews on the cited review site and fixes the feature description on its own product page. It schedules a re-label in two months using the same rules.

How do you write labelling rules?

Define each label in one sentence and add examples. Positive: the answer recommends or praises the brand. Neutral: it names the brand without judgement. Mixed: it gives both praise and criticism. Negative: it criticises, warns against or ranks the brand poorly. Add rules for edge cases, such as a brand listed last in a ranking with no comment, or a mention that repeats a customer complaint. Write the rule, then stick to it.

How do you check labeller agreement?

Ask two people to label the same sample of 50 mentions independently and compare. Where they disagree, discuss the case and update the rules. Repeat until disagreement is rare. Then spot-check a share of later labels. Without this step, changes in sentiment over time may just reflect changes in who labelled.

What does an accuracy label add?

Sentiment tells you tone; accuracy tells you whether the facts are right. A glowing answer that says you offer a free plan when you don't will cause problems in sales. Label each mention as accurate, partly accurate or inaccurate, and note the wrong fact. Inaccurate facts usually trace back to an old page, a third-party article or a confusing product description.

How do you use AI to help label?

Language models can label large samples quickly, but they need the same rules and examples as human labellers. Test them against a human-labelled sample before relying on them. Check agreement, look at the cases they get wrong and adjust the prompt. Keep a human review of a sample every round. Our view: AI labelling is useful for volume, but humans should own the rules and the final numbers.

How do you connect sentiment to sources?

For each negative or inaccurate mention, list the cited URLs. Group them by domain and page. Often a small number of pages explain most of the negative tone. Some you can fix directly, such as your own outdated pages. Some you can influence, such as review sites where recent customers can post. Some you can only respond to, such as forum threads.

How do you report sentiment to leadership?

Show the share of mentions in each label by engine and by prompt intent, with ranges across runs. Add the top three causes of negative mentions and the actions taken. Include one example answer for each cause. Keep the chart simple; the actions matter more than the decimal points.

When should you escalate?

Escalate when negative mentions repeat a safety, legal or pricing claim, when inaccurate facts could mislead buyers, or when negative tone rises in every engine at once. Those cases need product, legal or support involvement, not just content changes. Agree the escalation route before you need it, so the right people see the issue quickly.

Common questions

Is sentiment the same as share of voice? No. Share of voice counts mentions; sentiment measures their tone.

How big should the sample be? Large enough that one or two answers don't swing the result; a few hundred mentions is a reasonable start.

Can I compare sentiment across tools? Only if the labelling rules match, which they rarely do.

Related

• AI visibility score: how it's calculated
• Tracking brand hallucinations over time

Sources

• Google, AI features and your website

All accessed 30 Oct 2026.