.svg)
We dropped a new episode of The Frictionless Experience, and it might change how you read your own dashboards. Nick Paladino and I sat down with Lane Greer from Fullstory, and we had about a dozen questions planned for him. We got through maybe a quarter of them, because the conversation about AI and KPIs alone took over half the episode — and by the end of it, I was questioning how I read every dashboard I've looked at this year.
Here's the short version: if you've rolled out AI anywhere in your customer experience and your numbers got worse, that's not automatically bad news. It might be the opposite. You just might be measuring the wrong thing.
Does AI Increase Support Ticket Handle Time? Yes, and That's a Good Sign.
Lane opened with ticket handle time, which is about as sacred a KPI as support teams have. Higher handle time has always meant one thing: something's wrong, agents are slower, fix it.
Then AI chatbots entered the picture, and Lane watched that assumption fall apart in real time with one of his customers:
"When you have an intelligent chatbot like Intercom Fin running on the site, and you've trained Fin on all of your support documentation, it can actually handle almost 80 to 90% of the level one support requests that come in. They never actually go to the support agent. So ticket handle time for your support team is actually going to go up. The reason for that is that far fewer support cases get escalated but the ones that do are the hard ones."
Sit with that for a second. The AI is quietly absorbing the easy tickets — the password resets, the "where's my order" questions — and only the genuinely hard cases make it to a human. Of course those take longer. You didn't get slower. You got a filter.
Anyone watching that dashboard without the context would panic. That's the whole problem: the number moved, but the story it's telling is backwards.
Why Does Conversion Rate Drop When AI Improves the Customer Journey?
Lane had a second example, live from a customer site visit earlier that same day. Organic traffic had been sliding for a year and a half. Marketing was panicking. But average order value and conversion rate kept creeping up the entire time.
The cause? Google's AI search cards were answering questions before people ever clicked through to the brand's site. Fewer visits, sure, but the visitors who did show up arrived already informed and closer to buying.
Nick raised a version of this from the other direction: AI that helps a shopper build the right cart on the first try means fewer people coming back for a second order. Multiply that across millions of sessions and conversion rate can dip 5 to 10%, even while average order value climbs and total orders stay healthy. The top-line number says you're losing customers. The revenue says otherwise.
Lane's framing for sorting this out:
"The North Star has to be whatever that global number is—revenue per visit, whatever the board cares about. That’s the thing you should continue to see trending up and to the right. What you have to rethink are all the KPIs downstream from the KPI the board cares about.”
5 Steps How to Measure AI's Impact on KPIs.
This is the part I wanted more of, so here's what I'm taking back to our own reporting after this conversation:
- Segment AI-touched sessions from everything else. If you can't tell which sessions had an AI assist, you can't tell whether a KPI shift is AI doing its job or something actually breaking.
- Pair every operational KPI with the business metric it's supposed to serve. Handle time exists to protect customer satisfaction and cost efficiency — not to look good on its own. If those improve while handle time rises, the KPI is lying to you.
- Stop scoring every non-conversion as a loss. A session where someone checks an order status and leaves in ten seconds satisfied is a win, even though it'll show up as a bounce in most tools.
- Watch the downstream numbers, not just the North Star. The board metric moving in the right direction doesn't mean every KPI feeding into it is telling the truth. Check them individually before you draw conclusions.
- Ask "was AI involved here?" before you diagnose a dip. It's a five-second question that would have saved Lane's customer a week of unnecessary panic about their traffic numbers.
Why Taste and Judgment Still Can't Be Automated.
None of this works without a human deciding what "success" even means for a given session, and that's the part Lane kept coming back to.
He described building two apps on the same day: a buttoned-up enterprise tool for his work calendar, and a comic-book-styled itinerary app for a family trip. Both correct for their context. AI wouldn't know the difference on its own.
"Claude doesn't know the difference. It just sees a user interface and it uses it as a pattern. It just lacks the judgment and the taste to know what is appropriate at given times."
That's the same muscle you need to correctly read a KPI dashboard after AI enters the picture. The data doesn't interpret itself. Someone still has to know what "good" looks like in context, and right now, that's still a human's job.
The Bottom Line: AI Doesn't Break KPIs. It Changes What They Mean.
AI doesn't make your KPIs simpler. It makes them require more context to read correctly. A number going the "wrong" direction isn't proof of a problem anymore — it might be proof AI is working exactly as intended, filtering out the easy cases, or shifting where value gets captured in the funnel.
That's exactly the gap Blue Triangle exists to close. We're the only revenue assurance platform that quantifies, prioritizes, and proves the ROI of removing friction from your customers' digital journeys. So when a metric moves after an AI rollout, you know whether it's a real problem or a false alarm, instead of guessing.
Catch the full episode wherever you listen to The Frictionless Experience.
.png)
