Inside how Gainsight AI works with Shantan Reddy
Watch the episode 👇
About the show
Check out more at www.bitsandbots.ai, or find the show on YouTube, Spotify, Apple, or your favorite podcast spot. We’re always curious for ideas and feedback, so please get in touch!
Episode Transcript
Here's a one minute teaser of this episode.
Shantan leads AI at Gainsight. Gainsight is the industry leader in customer success. The first feature we call meeting assist. We generate a summary of the meeting and push it to timeline automatically. The second use case is what we call the customer cheat sheet. Essentially an exec level overview of what is going on.
With that account, what you see right now is a RAG approach, but how it started out was a hierarchical summary. This is a third feature. This is called the takeaways. This is a one stop shop for you to understand every recurring piece of feedback coming from your customers. Reducing configuration efforts is an important goal for Gainsight.
People can chat and have their reports generated. So that config will set up a report in Gainsight. I can show you the architecture. There's a user coming, asking the question. Now we are contextualizing that question. Which tables is the question to this answer likely to reside in that context is passed up to this query generation model, which is basically a text to SQL engine.
And now for the episode.
David: Welcome to Bits and Bots, have an awesome guest today, Shantan leads AI at Gainsight. He's done some incredible things. I'm really excited about what they built to have him on the show and talk about it. Welcome. Thanks.
Shantan: Thanks, David. Excited to be here. For folks
David: who aren't familiar, maybe we could start with what's Gainsight?
Shantan: Gainsight is the category creator and industry leader in customer success. Given the explosion of SaaS in general, there is a movement needed to ensure that your customer is successful with your engagement with them. And increasingly we champion that there needs to be a team and, a very tailored process to make that happen.
And Gainsight is a tool for teams, to be able to manage your customers at scale, and ensure that they're successful and eventually renew and expand.
David: And how have you been using AI?
Shantan: So we have been, using AI before this whole Gen AI, or madness began, since 2019, we've been having AI.
We primarily had two, centers of AI. One was around, churn prediction along with reasons and more deterministic modeling. And of course doing it at scale, when you have a multi tenancy system where, you have 800 customers with very different scenarios around renewals and to have an auto ML framework that does it.
It took a while for us to build it and get it to the right place. And then we had another set of functionality around more around NLP related stuff. So basics like sentiment followed by impact analysis, what factors are impacting your NPS, themes in conversations, et cetera. So we were doing some work around NLP.
But with the advent of Gen AI, what we could do in the field of customer success is basically limitless. I can see us working for the next four or five years and still have a 10 year roadmap ahead of us.
There's also, some streams of work on how we can drastically reduce, the overhead of administering GainSight. It's a very flexible, complicated tool. And we think genAI can drastically reduce the effort required there. And then we also have some very interesting ideas on how our customers as customers, can benefit from this technology.
So aspects of self service and personalization, and big automation is the third, broad stream of work that we see lots of possibilities.
David: That's awesome. Can you show us one of your favorite AI features?
Shantan: So the very first feature that I wanted to talk about, we call it meeting assist. The idea is CSMs no longer have to take manual meeting notes. We take the meeting transcript and we have this feature called timeline which is essentially a record of every key customer interaction that you have.
And all the meetings that CSMs have with their customers, they need to be logged into timeline. And it's a huge overhead that CSMs don't really want to do. So from the transcripts, we auto generate a summary of the meeting and push it to timeline automatically. We have a view on what a summary for the CS team needs to look like. And we have, incorporated some of that fine tuning and precise altitude into the summary.
This is a bit of an older demo that you're looking at, but we also extract action items, we look for any risks. One of the reasons why we are also generating our own summaries is that we want to do a lot more with meeting data. It's a gold mine of information, as you can imagine about customers.
And the summary action items and risks are the very first thing we want to extract. So we want to start looking for a lot more things that are actually relevant for CS teams like strategic priorities and, key projects have been discussed and there's a home for each of these items.
Or for example, imagine a sponsor telling them that they are leaving the organization. It's an immediate risk that you would want to log, right? We plan to extract a lot more from meetings and the summaries and the action items are the very first thing that you see.
The second use case is what we call the customer cheat sheet. So the idea here is you have timeline, which is a record of all these key customer touch points and interactions. The average is about 80, 90 customer interactions in one year. It's emails, meetings, updates, support tickets, all of that. The cheat sheet is essentially an exec level overview of what is going on with that account, based on the last six months of everything that is there. So as you can see, the first thing we show is background. We also fetch information from the web as to what the company does.
And then we look for these very specific things, which we think are very important for an exec or a manager, let's say getting into a meeting or generally trying to understand what's going on with that account. So we have risks and issues, any key projects, that they have been doing, who are the key contacts.
If there were any renewal discussions or strategic priorities that were discussed or things that they liked, each of those things would be searched for. All of those things are captured and summarized.
Ankur: Can you click into these things and go to the source materials?
Shantan: And if you just wanted to, let's say, know more about this one particular risk that you had, we have this elaborate. What this does is it gives you a little more information about it and it also links you to the original source from which we found this.
Ankur: What kind of systematic issues, have you observed? As you've looked at the ratings and stuff.
Shantan: What we found was that, we really had to tune, for example, the risks aspect. We had a summarization approach where we were taking this six months of large volumes of data and recursively summarizing it. And then getting to a summary. Then we realized we were missing a lot of information based on just the likes and the dislikes that we found.
Then we went to this entire new approach of, searching for specific things and getting it back. And once we deployed that, we realized that there were a lot of false positives in risks and issues, for example. And we had to tune the prompt again, just to make sure that this was that for the systemic issue that we found a lot of dislikes in risks and issues because a lot of them were false positives.
Most of it was prompt tuning and getting it just right was very important for us.
Ankur: Yeah, I have a couple of questions. So one, when you mentioned scale, I think that's an issue that's like top of mind for a lot of people. Both cost and model availability. Are you using OpenAI's models, and if so, are you using them directly through OpenAI? How are you thinking about cost and scale?
Shantan: Right now, yes, we are using open AI APIs. They are hosted by Microsoft Azure, because at the time we felt like they had just a little bit better protection. Our customers would be a little more comfortable if they were hosted by MS Azure. So we are using the chat GPT APIs, but we did try out other alternatives as well. We tried out, Anthropic, we tried out, llama, both the 7B and the 7TB. Overall, once we gauged the cost versus the quality of the models, we decided to stick with OpenAI for now, but more and more, we are looking at situations where we would need a mix of models, to deliver a given feature.
Ankur: And why is that? Is that like a quality thing or a cost thing?
Shantan: In the meeting assist feature that we talked about, we needed an intermediary summary and from the intermediary summary, the final summary is built and intermediary summary, we still store it because it's useful for other things in the future, right?
In that scenario, we found that even a LLAMA7B was giving us pretty good, summaries for us to work off of. We didn't really need to use ChatGPT. So A, it shares the load and B, it's more cost efficient. Similarly, in the future, if there's bigger things to summarize, Claude might be the right option.
So it could just be for summarizing documents, for example, the higher context window, it's just the right tool for the job. So it's those kinds of situations that we foresee ourselves running into that will require us to always keep more than one option in mind.
David: Shantan, I was curious, given how much data each customer has, everything in their timeline, meeting upon meeting.
I think you alluded to it a minute ago, but how do you actually generate these summaries? Is it hierarchical summaries of summaries are using RAG under the hood or embeddings?
Shantan: So we started out with the hierarchical summarization for this feature. And then we realized that it's not the most efficient way and we just found that we found better results with, embedding and RAG approach.
So what you see right now is a rag approach, but how it started out was a hierarchical summary.
We have a chatbot that we are working on that will answer any and all questions about a customer or across your customer. So you could ask for references for one of the features that you're building or any issues. That have been recurring in the feedback or for a given customer, you might want to just know what the risks are.
For example, all of those things would get unlocked by having a chat bot on the timeline. You can imagine how timeline and GPT have so much value to generate.
This is the third feature that we have live already. This is called the takeaways. The idea here is this is a one stop shop for you to understand every recurring piece of feedback coming from your customers. You read takeaways and you would get a very fair idea of everything of value that your customers are talking about.
We have a series of key points or key recurring themes in feedback. So here are customer support issues. So for example, I have 270 on a customer support being good, right? And then you can see the summary of what these 270 responses are saying, and you can drill down into the individual responses as well.
For this, we are using two technologies. Yes, there is generative AI, a chat GPT that is generating these summaries. It's generating the categories, but the themes themselves, like the actual key points. We are using an IBM researches, a service, called Key Point Analysis.
What that does is it looks for opinionated text with a very clear stance, and then it aggregates them into this key points and this is a particular theme within the feedback. And here are the 270 responses that link to this particular key point. This is the base of the technology.
And then we have chat GPT to make it easier to consume.
Okay. So the idea here is we are targeting natural language reporting. This opens up a broader window into natural language configuration of Gainsight. We want to give an AI assistant of sorts to administrators of GameSight who can talk to the chatbot and the chatbot kind of does the configuration for you.
David: You guys are using, it looks like Streamlit as sort of a dev playground for AI prototypes. Is that right?
Shantan: Yes. We are doing a very basic alpha launch. It's not going to be our final UI. We will let our internal users ask questions. And only when we see where all it is failing, that we'll know what improvements need to be done to this entire framework.
Give me at risk customers.
David: Okay. So what's happening under the hood ?
Shantan: Maybe I can show you a bit of the architecture for this. There's a user coming, asking the question. Now we are contextualizing that question. What exactly does he mean by at address customers, for example. Based on the information, what is his likely intent, which tables is the question to this answer likely to reside in, can I get the metadata of the relevant tables so that the AI can understand how to build out the query and then all of that context is then passed on to this query generation model, which is basically a text to SQL engine. Oh, wow. And then this text to SQL engine, it sees if it has all the information that is required to answer the question, right? If it thinks that, it needs a clarification, it needs more information, it will ask the user a follow up question if it has the information, it builds the query, runs it through a validator that checks if the query is right, valid, and if it is in line with what the user has requested in the first place. And if it's valid, then you give the answer, the response back to the user. If it's not valid, you again start the entire query generation model along with fetching the context one more time with the error that the validator had given you.
David: Interesting. And the query that it's running is presumably in Gainsight's own kind of query language format, or is it literally just a SQL query against your production
Shantan: right now we are going with SQL because if we figure out the text to SQL bit and we have a separate engine for SQL to, let's say the config JSON, right? it will unlock many many more than just reporting. Different feature areas would have different kinds of JSON formatting and Unlocking this is the basis and then the last step can be different depending upon the implementation of this feature. GPT 4, it's good at SQL. So we are focusing on generating SQL and from SQL, , you'll see, once you see the response
List of all high revenue companies for which I'm a CSM error greater than a hundred K. It assumed that, um, this report, it gives an explanation of the report, the SQL and the report config.
David: I see. So that config will set up a report in Gainsight
Shantan: based on this. I'll read this and it will be, I'll, a user would see the report right alongside the chatbot with the UI. Wow. And now let's say the, it said error is greater than a hundred thousand, or is it a million?
David: For the SQL part, do you feed, do you have a prompt that basically feeds all of your databases to the AI? How does it know the set of what's possible?
Shantan: with GainSight, we have a set of standard and system objects. Our customers would also bring in a lot of custom data into custom objects. In the V1 of this, we plan on supporting all standard and system objects within GainSight and most of our rules, reports, et cetera, are on those objects.
So we will be supporting those. And for those, yeah, we had to give a description for every object. It took a decent amount of work to get it to work on this set of tables. We had to give it a decent amount of context on what it's looking at. This looks awesome. Super cool. And again, we're starting with reporting, but as you can imagine, this becomes a base for natural language configuration of GainSight.
And reducing the configuration effort of GainSight is an important goal for GainSight. So now I asked you to change the threshold. It changed. People can continue to chat and have their reports generated in the future rules, geo programs. We've been doing research on this for more than a quarter now, and we're going to do an alpha launch, very bare bones, chat bot, just for data gathering.
And we want to set this expectation with our own users as well, that, this is something that we need to do to make sure that we are shipping high quality product into production. And this is the first experiment of its sorts where it's the UI is not going to be what we will have in production. It's the fastest UI we can ship just for the internal instance of Gainsight.
David: Yeah, I love this. definitely resonates with some challenges we have at Coda as well, where you can configure Coda docs in a whole bunch of ways. We have large tables that feel like databases.
Another similar effort that we did. We were doing a RAG based AI bot for our customers's customers. We call it the smart search assist. Where the idea is our customers's customers can log into the community portal. Ask any questions about our customers product or services and the chatbot would answer.
Shantan: Let's say IBM is a customer of ours. You would come to IBM's portal. Any customer of IBM could ask questions about IBM's product services. We had two different search vendors that we wanted to evaluate. So we built this UI we put it on top of the Gainsight data. And then we asked, questions like, can I create custom CTAs?
And we would show both responses side by side. We gave access to this to a bunch of our own internal users and asked them to vote on which search service seemed to be better.
David: Oh, this is cool. What are the two different responses?
Shantan: So the search engine behind them is different. One was a vector based search engine and one was an existing search engine that we already had. We wanted to know if we needed to move to a different search service. We quickly wanted feedback on which one is better. So we just gave them a convenient UI to test which search service is better.
David: It reminds me of a side by side at Google. We had search raters go through two sets of results next to each other and
Shantan: vote on them. And that's the thing, I think the biggest thing we and everybody I'm guessing underestimated is how much additional work it takes to actually have a production ready system.
Even for the simpler implementations, getting it just right, takes a bit of work and a little bit of ingenuity.
David: Shantan, this was awesome. Really cool stuff you guys are doing, amazing what you've already shipped and some of the big swings you're working on. Really excited to see how it comes to life and thank you for sharing your practical tips and lessons learned from building these with us.
Ankur: Really great to chat with you. It looks like you guys have built some really cool
Shantan: stuff. Thank you.
David: Thanks so much. As always, this is an experiment. We'd love your feedback and what you'd like to see more of less of, and ideas for future episodes.
So check out bits and bots. ai and see you on the next episode. Bye