[00:00:00] [music]
[00:00:04] >> Welcome back to another episode of Tech
[00:00:06] on Hinged powered by Code District where
[00:00:07] you can get human. I’m your host Rabia
[00:00:09] Javed. My guest today is Randeep Bhatia,
[00:00:11] the CTO of Splash Music, creator of 10
[00:00:14] billion parameter AI model that turns
[00:00:16] become into a song which has been
[00:00:17] streamed hundreds of millions of times
[00:00:19] worldwide. Before Splash, he spent
[00:00:21] nearly two decades at Twitch, Audible,
[00:00:23] and EA Games. He holds multiple US
[00:00:25] patents and he’s given over a hundred
[00:00:27] talks around the world including the AWS
[00:00:29] Summit keynote. Welcome to the show,
[00:00:31] Randeep.
[00:00:31] >> Hi Rabia. Thanks for having me.
[00:00:33] >> You’ve sat through a lot of AI pilot
[00:00:36] pitches by now. What’s the phrase or
[00:00:38] buzzword that makes you quietly roll
[00:00:41] your eyes at this point that you’ve
[00:00:42] heard enough of it?
[00:00:44] >> I will say change management. It’s
[00:00:46] something that I hear a lot
[00:00:50] whether it is even an AI pilot pitch or
[00:00:53] not. And the beauty of change management
[00:00:55] is it hides everything. It hides the
[00:00:56] person who brought the changes. It hides
[00:00:59] the reason the changes were brought in
[00:01:01] and why some things are not working. So,
[00:01:04] it’s what I call actually a hidden
[00:01:06] jargon that everybody uses in the way of
[00:01:09] saying, “Okay, it’s going to make
[00:01:10] everybody’s job easier.” But, why? So,
[00:01:13] so yeah.
[00:01:13] >> So, Randeep, you know, everyone’s
[00:01:15] running an AI pilot these days. It’s
[00:01:17] become the boardroom word of the year.
[00:01:19] In your own words, what is a pilot
[00:01:22] really and what’s the biggest
[00:01:23] misconception people have about it?
[00:01:25] >> Absolutely. So, pilot is a new name for
[00:01:29] experiments, what we used to call before
[00:01:32] AI became a prominent word in the
[00:01:35] industry. Pilot basically has a goal, a
[00:01:38] mission, a metric that you’re trying to
[00:01:40] target for a bounded date. And it is not
[00:01:44] a permanent demo. That’s where the catch
[00:01:46] is. When AI pilots go on and on for
[00:01:50] months, years, and then that’s when they
[00:01:53] become permanent. That’s not an AI pilot
[00:01:56] anymore. That becomes more of a
[00:01:57] decision.
[00:01:58] >> But, you know, if you would have to sort
[00:01:59] of explain it in layman terms for
[00:02:01] probably someone who’s not from a
[00:02:03] technical background, how would you
[00:02:04] explain AI pilot? Are are are those the
[00:02:06] pilots in the planes or what?
[00:02:09] >> Yeah, so absolutely. So, let’s say that,
[00:02:11] you know, you are building a car and you
[00:02:15] want to launch a very specific feature
[00:02:18] of a car of automatic rolling up of
[00:02:21] windows and it up and down. And you want
[00:02:24] to see if people start to use that
[00:02:26] feature or not or they manually want to
[00:02:28] roll the windows up or down. Now, for
[00:02:30] some cars, you can release that feature
[00:02:32] as a pilot to see whether people start
[00:02:34] to use this. And while the other cars
[00:02:36] still have the manual up down, based on
[00:02:38] the adoption of the people, you will
[00:02:40] realize, okay, yeah, more and more
[00:02:42] people are finding it easy to roll the
[00:02:45] windows up and down. Are we getting any
[00:02:47] complaints of customers? Are things
[00:02:49] breaking down and people are unable to
[00:02:51] get out? Or is it really making people
[00:02:53] lives easier so they can actually focus
[00:02:55] on the driving? And with that, the goal
[00:02:58] is to basically have a very time-bounded
[00:03:01] releases, very time-bounded efforts.
[00:03:04] Okay, within the month number one, we’re
[00:03:05] going to measure the traction of the
[00:03:07] people using. Within the month number
[00:03:08] two, we’re going to measure the
[00:03:10] complaints that people are going to
[00:03:11] bring in. Or within the month number
[00:03:12] three, are our NPS scores going up for
[00:03:15] these services or systems that we have
[00:03:17] launched? And that’s what really pilot
[00:03:20] helps get information out to people. And
[00:03:23] what we call is an AI pilot as well
[00:03:25] today. So.
[00:03:26] >> So, there’s a much-quoted MIT finding
[00:03:28] that 95% of company AI pilots show no
[00:03:32] real return. From where you sit, is that
[00:03:34] the technology failing short or
[00:03:37] something else?
[00:03:37] >> What MIT has quoted, uh, the 95% of the
[00:03:42] technology is right and is very much in
[00:03:45] alliance with what I am seeing around
[00:03:48] myself as well. And I’ll give you an
[00:03:50] example. So, at Splash, you know, when
[00:03:53] we were trying to use technology to
[00:03:56] really empower Gen Z and other people to
[00:03:58] make music, we had a lot of problems in
[00:04:00] front of us. Should we go and make use
[00:04:03] of technology to solve this problem
[00:04:05] end-to-end? Should we have some humans
[00:04:07] in the loop? Should we go more
[00:04:09] heuristic-based approach? What the
[00:04:10] answer should be? And what we really
[00:04:12] focused on is the product user use case.
[00:04:16] Is this really going to make people’s
[00:04:19] lives easier? Is this going to become
[00:04:21] more delightful experience for the user?
[00:04:23] And that’s what truly is a challenge
[00:04:26] when people divert from there and focus
[00:04:28] truly on the technology. I often like to
[00:04:31] quote a quote from somebody that, you
[00:04:33] know, the best AI model is a model that
[00:04:35] works for your customers. And that’s
[00:04:37] what truly this is about. Really finding
[00:04:40] the technology that can make the
[00:04:42] people’s experiences much more
[00:04:44] delightful, useful, and coherent for
[00:04:46] them rather than just making a shiny
[00:04:48] technology that nobody uses, which is
[00:04:50] what MIT has been quoting the 95% has
[00:04:53] been.
[00:04:53] >> Well, you know, that makes a lot of
[00:04:55] sense because, you know, my next
[00:04:56] question is an extension of the last one
[00:04:59] that the answer we usually hear or
[00:05:01] always hear behind it is that, you know,
[00:05:04] our data wasn’t ready. So, what does
[00:05:06] that actually mean? You know, paint us a
[00:05:08] picture of what messy data looks like
[00:05:10] inside a company and why the AI just
[00:05:12] can’t cope with something like it.
[00:05:14] >> Uh this was actually one of the very
[00:05:15] funny uh cases that happened at Splash.
[00:05:18] I’m going to bring Splash again into
[00:05:19] this. You know, the using of a language
[00:05:22] that exists in a company needs to be
[00:05:24] same, and that is very, very important,
[00:05:27] especially when you don’t have humans in
[00:05:29] the loop. Even when you have humans in
[00:05:30] the loop, you don’t need that same. Now,
[00:05:32] by definition, a station in music
[00:05:35] industry is an open-ended playlist.
[00:05:37] Playlist is just a collection of songs.
[00:05:40] Now, all it takes is one developer
[00:05:43] naming station as a playlist in the
[00:05:45] codebase, and AI when the wipe coding
[00:05:48] ecosystems understands that it is a
[00:05:50] playlist. Now, I’m going to interact
[00:05:52] with this system as a playlist. That
[00:05:55] causes a whole lot of messiness in our
[00:05:58] experiences. Not because the data was
[00:06:02] like messy, but it was because we did
[00:06:04] not really name it well. The language
[00:06:06] was not same. So, AI is again a system
[00:06:09] that looks for patterns on how you are
[00:06:12] presenting your data. So, having first a
[00:06:15] foundation of what each system mean. I
[00:06:18] will call it more like an appendix. If a
[00:06:20] station means an open-ended playlist,
[00:06:22] have it like that. If a playlist means
[00:06:24] collection of songs, have it like that.
[00:06:27] Rather than using 10 different versions
[00:06:29] of a same system. And that’s where AI
[00:06:32] has hallucinates. That’s where AI gets
[00:06:34] confused and people get different
[00:06:36] results and not the results especially
[00:06:38] what they expect. Oh, the wipe coding is
[00:06:40] not working for me. Oh, the wipe coding
[00:06:42] has completely changed my experience. If
[00:06:44] I wipe my database or because the naming
[00:06:46] conventions and the data presentation
[00:06:48] was not there for you.
[00:06:49] >> So, you know, most companies drop AI
[00:06:52] into their existing way of working and
[00:06:54] hope for the best. Very few redesign the
[00:06:56] work um itself around it. As a leader,
[00:06:59] why does that redesign step keep getting
[00:07:02] skipped?
[00:07:03] >> Redesigning is not easy for any
[00:07:05] technology leader. Redesign does mean
[00:07:07] you do bring in change management, the
[00:07:09] very famous buzzword that we hear all
[00:07:11] the time. And it also requires a whole
[00:07:14] lot of changes in the existing
[00:07:16] ecosystem. Whether it is hiring more
[00:07:18] people, whether it is improving the
[00:07:21] processes that are existing right now,
[00:07:23] or even getting more head count.
[00:07:25] Whereas, you know, buying something off
[00:07:27] the shelf and just adding it as a line
[00:07:29] item on your P&L is a much more easier
[00:07:32] way for people to do it. Secondly,
[00:07:35] people do want to treat AI more like an
[00:07:38] innovation lab than a concrete business
[00:07:41] school for themselves. Uh you know, we
[00:07:43] do see a rise of chief AI officer or AI
[00:07:47] strategist jobs because that is more
[00:07:49] like again, open-ended experiments with
[00:07:52] no goals, no descriptions, no targets,
[00:07:56] no dates. They keep on running as an AI
[00:07:58] pilot. If they work well, yes, they will
[00:08:00] impact the business metrics. If they
[00:08:02] don’t, they kind of work with less of an
[00:08:05] authority on the overall business OKRs
[00:08:08] and metrics, but they do try to
[00:08:10] influence at times to see whether this
[00:08:13] has moved the needle or not. And
[00:08:15] slipping on two boards at the same time,
[00:08:16] that’s where the chaos happens as well
[00:08:19] when people try to see if AI is working
[00:08:22] for them while only treating it as an
[00:08:23] innovation open-ended experiment with no
[00:08:26] metrics to go after. So, that’s why
[00:08:28] redesign keeps on getting skipped and
[00:08:30] people try to find the shortcut. So,
[00:08:32] okay, let’s see if it is going to work
[00:08:34] or not because it does become very
[00:08:35] challenging.
[00:08:36] >> Yeah, absolutely. And you know, now I’m
[00:08:37] going to go once again back to the MIT
[00:08:40] study which says that buying beats
[00:08:42] building two to one. You ignored that,
[00:08:45] you know, when you built your own 10
[00:08:46] billion parameter model at a series A
[00:08:49] startup. Was MIT wrong or are you the
[00:08:51] exception? When should a company
[00:08:54] actually build?
[00:08:55] >> Honestly speaking, if I had the
[00:08:56] opportunity, I would have totally bought
[00:08:58] it or looked into the other I did look
[00:09:00] into the other solutions as well. So,
[00:09:02] music is a very unique case. There are a
[00:09:05] lot of models that exist to improve the
[00:09:08] quality of a sound or make sure that
[00:09:11] sound is coming out right, it has less
[00:09:13] disturbances. Even when we are doing
[00:09:15] podcast, you know, transcription models
[00:09:17] and other models exist. But there is no
[00:09:20] model that can tell you that something
[00:09:22] sounds good, is it musically good enough
[00:09:25] as well? Now, there are building blocks
[00:09:27] to it which means even in building the
[00:09:29] 10 billion parameter model, we did make
[00:09:32] use of some of the off-the-shelf
[00:09:33] solutions to clean out the
[00:09:35] interferences, to clean out the audio
[00:09:37] signals, extract all the signals. But
[00:09:40] then we had to take the matter into our
[00:09:42] own hands to see, can we come up with
[00:09:45] metrics first, and can we come up with
[00:09:47] mechanisms to convert the human input
[00:09:51] into a musical output but that is
[00:09:53] generative enough so it can be measured
[00:09:56] on an automated basis. And that’s where
[00:09:59] we had to go and build our own. And the
[00:10:01] journey was not okay to go and build the
[00:10:04] billion parameters from day one. The
[00:10:06] journey was very much to start with
[00:10:08] couple of million parameter models,
[00:10:09] start addressing the couple of genres of
[00:10:12] the music, and only with very specific
[00:10:14] keys to see, is it really going to
[00:10:17] transform the music or not? Or do we
[00:10:19] need to bring in more parameters, more
[00:10:21] parameters to increase the quality and
[00:10:24] make it sound musically enough?
[00:10:26] >> You know, it’s really it’s really hard,
[00:10:28] you know, to have certain levels of
[00:10:30] accountability when it comes to AI, you
[00:10:32] know? Having a human in the picture, you
[00:10:34] can have someone accountable, but having
[00:10:36] an AI in the picture and AI
[00:10:37] hallucinating, it’s really hard to
[00:10:39] track, you know, who’s done what wrong.
[00:10:41] So, when should a company start setting
[00:10:43] the rules and who’s accountable? What
[00:10:45] the AI is allowed to do and who checks
[00:10:48] its work? Who takes this responsibility
[00:10:50] once it’s deployed?
[00:10:51] >> Absolutely. So, I’ll I’m going to bring
[00:10:53] in a bit of an Amazonian flavor to this.
[00:10:57] At Amazon we used to call it it’s always
[00:10:58] day one, and
[00:11:00] I like to lay emphasis on it that
[00:11:03] accountability or and data governance is
[00:11:06] a day one check for any kind of AI work
[00:11:09] that you want to do. You don’t want to
[00:11:11] check it after something breaks. You do
[00:11:14] want to check it right at the beginning
[00:11:16] of the stage when you’re deploying it.
[00:11:18] You do have to build that trust
[00:11:20] foundation and ecosystem from the very
[00:11:23] day one because that is going to put
[00:11:25] things more into motion about your
[00:11:28] autopilot that you want to build. An
[00:11:30] example of this is, you know, waiting
[00:11:32] for an incident in any of your AI pilot,
[00:11:36] say oh, the data did not return, the
[00:11:38] data started hallucinating, is never
[00:11:40] going to scale. Nobody really is going
[00:11:42] to trust the output and it will be
[00:11:44] expanded because you need to build a
[00:11:46] trust in the engineering itself when the
[00:11:49] pilot is being built.
[00:11:50] >> When you view your LinkedIn says, you
[00:11:52] know, you got Splashes cloud bill from
[00:11:55] $5 million
[00:11:56] a month to a half a million a year. How
[00:11:58] did you do it and what did it buy you
[00:12:00] beyond the money?
[00:12:02] >> That was one of the big challenges that
[00:12:03] I was going after. Being a startup, it’s
[00:12:07] not easy to scale especially in the AI
[00:12:10] space. Yes, you do get funding, yes you
[00:12:13] do get resources, but you know, it’s a
[00:12:16] common notion, turn off the lights when
[00:12:18] not in use and try to use a better
[00:12:20] optimized energy efficient tools when
[00:12:22] you can. We use it at our home, so why
[00:12:25] not use it in the cloud? So, one of the
[00:12:26] things I did at Splashes to really open
[00:12:29] up the product ecosystem and see what
[00:12:32] exactly is our use case. Is our use case
[00:12:35] to go after every single genre of music
[00:12:38] and address it from day one or is the
[00:12:39] goal to really see are people liking our
[00:12:41] product from the very basic genre and
[00:12:44] they’re finding delight in that
[00:12:45] experience. And so from that, I started
[00:12:48] building the initial foundation model
[00:12:50] and couple of million parameters to see,
[00:12:53] okay, people come in, they make music,
[00:12:55] music is being shared across the Roblox,
[00:12:57] people are finding delight, good. Now,
[00:12:59] we can expand into it even further. Can
[00:13:02] we look into the alternate options to
[00:13:04] what we are using? We were using GPUs
[00:13:07] before, then moved into the Amazon
[00:13:08] custom silicon. Amazon had were really
[00:13:11] friendly with us to helping us carve out
[00:13:14] strategies that can fit into the startup
[00:13:16] ecosystem. Being a startup, you know,
[00:13:18] you’re not always training, you’re also
[00:13:21] doing inference at the same time and to
[00:13:22] inference, I’ll clarify in this is the
[00:13:25] output of your training model that can
[00:13:27] be used by your applications at scale.
[00:13:31] And that allowed us to really use our
[00:13:34] hardware that we reserved for our
[00:13:37] training purposes to be multi-purpose
[00:13:40] for our application as well. It unlocked
[00:13:42] the opportunity in half for our cost
[00:13:45] that okay, now we don’t have to go after
[00:13:47] two different sets of hardware, we can
[00:13:49] just use one to do both. Secondly, we
[00:13:51] also got the ability to scale the
[00:13:54] systems up and down on an on-demand
[00:13:56] basis, which is extremely hard,
[00:13:58] especially when you’re building an
[00:13:59] open-ended musical problems because the
[00:14:02] parameters get so huge over time. And
[00:14:04] the fourth one was looking out there for
[00:14:06] your adoption of the product. Yes, it’s
[00:14:09] a good idea to build as many billion
[00:14:11] parameters model as you can. Engineers
[00:14:14] love to boil the ocean. I’m an engineer
[00:14:16] at heart, but I’m also a product person
[00:14:17] at heart, and I really like to see,
[00:14:19] okay, is this really converting audience
[00:14:23] from being coming to the platform,
[00:14:25] consuming the information, and then
[00:14:27] inspiring others and themselves to come
[00:14:29] in again. And that moment really drives
[00:14:32] that AI pilot metric to say, okay, yes,
[00:14:36] this is good. Now, let’s move on to the
[00:14:38] next experiment. Let’s move on to the
[00:14:40] next experiment.
[00:14:41] >> So, you know, already, the idea of now
[00:14:43] engineering music, getting something out
[00:14:45] of it, was it like always a passion or
[00:14:48] did Splash turn it into one for you?
[00:14:50] >> Music is something that I’ve been
[00:14:52] working since pandemic. Uh I started
[00:14:55] working in Twitch music, working
[00:14:57] directly with the artist, and giving and
[00:14:59] building an ecosystem where we can have
[00:15:02] a fair systems, fair platforms for
[00:15:06] creators to come and perform and share
[00:15:09] their stories because, you know,
[00:15:11] musicians and any form of creator works
[00:15:13] really hard to present it in front of
[00:15:15] audience. And it takes a lot for them to
[00:15:18] have a career in this because
[00:15:20] competition, because of rights, because
[00:15:23] of so many streams you have to get to
[00:15:26] make even money in pennies. So, that’s
[00:15:29] when it hit me like, can we do something
[00:15:32] to change this ecosystem? And this
[00:15:34] change is even significantly with AI
[00:15:36] because we talk a lot about, okay,
[00:15:38] something is ethically trained,
[00:15:40] something is not ethically trained.
[00:15:42] Then, is it just about getting paid?
[00:15:45] It’s actually not because that’s where
[00:15:47] artists truly like to have is the sense
[00:15:50] of ownership of their own data
[00:15:53] themselves. It’s less about, okay, the
[00:15:55] artist knows, okay, we are getting paid.
[00:15:57] It is more about, okay, you’re using our
[00:15:59] data. How are you going to use our data?
[00:16:01] What will happen if I move away from
[00:16:03] you? What will happen if I use another
[00:16:05] platform? What will happen if I go and
[00:16:07] start doing tools? And these are all
[00:16:09] blurry line item, and especially the
[00:16:11] more indie artist, this gets more blurry
[00:16:13] because it’s about making a career into
[00:16:16] the field. And at the end of the day,
[00:16:18] the artist wants to have the the new
[00:16:20] ways to connect with their fans, their
[00:16:22] super fans. And in this case, you know,
[00:16:24] what we were really bringing with Splash
[00:16:27] is a more coherent experience with Gen
[00:16:30] Z, which is a very hard audience to
[00:16:32] target. And that audience treats
[00:16:34] creators with equity and transparency.
[00:16:37] And that’s what we wanted to have, and
[00:16:40] we called it out publicly that we work
[00:16:42] with Gen Z, we make the music available,
[00:16:45] we ship it to Roblox. And so, it is not
[00:16:47] just about building relations with
[00:16:49] artists. It is a whole ecosystem what
[00:16:52] Gen Z wants and what artists want, and
[00:16:55] can we be the connecting tissue over
[00:16:57] >> Yeah, because, you know, I see that
[00:16:59] after viewing and seeing your work and,
[00:17:01] you know, your contribution in the
[00:17:03] industry, you you did make ethically
[00:17:05] trained part of your brand. So, you
[00:17:07] know, for the sake of this conversation,
[00:17:09] I want you to go a bit deeper on this
[00:17:11] layer and explain to our listeners that
[00:17:14] what what did it really mean for you?
[00:17:16] And where’s the honest line on data
[00:17:19] ethics after AI?
[00:17:21] >> Absolutely. The goal is basically be up
[00:17:24] front, be honest about your data
[00:17:26] approach, whether you’re building AI
[00:17:29] solutions or non-AI solutions. It’s not
[00:17:32] just being under the blanket of yes,
[00:17:35] artist agreed and we paid them and it
[00:17:38] worked. No, it is not just like that. It
[00:17:40] is about can you bring them the audience
[00:17:42] that they’re looking for? Can you bring
[00:17:44] them the community that they’re looking
[00:17:46] for? And that’s where the higher bar
[00:17:48] associates, especially with Gen Z,
[00:17:50] because Gen Z does love to connect and
[00:17:53] be in a community where it is much more
[00:17:56] driven by transparency, honesty, and
[00:17:58] authenticity. Because at the end of the
[00:17:59] day, these artists are trying to be
[00:18:02] authentic on all of these platforms, and
[00:18:04] that is the connecting tissue with the
[00:18:06] newer generation that we are seeing over
[00:18:08] here.
[00:18:09] >> But before I jump onto my next question,
[00:18:10] Randeep, you know, as a CTO, how did AI
[00:18:13] affect your, you know, day-to-day work?
[00:18:17] >> It’s a blessing in disguise.
[00:18:19] Blessing in a way that, you know, I can
[00:18:21] do a lot more things. I can research
[00:18:24] into a lot more things effectively. I
[00:18:26] can pretty much take off my daily
[00:18:28] routine task and give it to AI, whether
[00:18:31] I’m talking about personal life,
[00:18:33] professional life, and any of those.
[00:18:35] Other side of the coin is it does
[00:18:37] increase much more problems because then
[00:18:39] I am also sharing my data at many
[00:18:41] different places. Professionally, it
[00:18:43] also means that I am now relying on
[00:18:47] tools that I expect them to work in a
[00:18:50] very explicit way. Third is now I am
[00:18:53] also getting a whole lot of tools in my
[00:18:57] company’s ecosystem that I may or may
[00:18:59] not have an idea about. So, it does
[00:19:01] increase the surface area of your
[00:19:04] exposure, but that also then comes up
[00:19:06] with a responsibility. Can you manage
[00:19:08] that exposure? Can you retain? Can you
[00:19:10] keep everybody in the company aligned
[00:19:12] because every single solution that we
[00:19:14] see in the market whether it is model A,
[00:19:16] model B, and it all changes every single
[00:19:18] day. Wait for an announcement from
[00:19:20] company or company B and you will see
[00:19:22] the model charts literally go up and
[00:19:24] down. But which model is really going to
[00:19:25] work? Which solution, which tool, or
[00:19:28] which co-working thing is going to
[00:19:30] really work for us? Is really the name
[00:19:31] of the game because the companies, even
[00:19:34] being a startup, change management is
[00:19:36] not that easy. You still have to go
[00:19:38] through every single person to see
[00:19:41] whether you drive adoption, whether you
[00:19:43] drive the company’s ecosystem on top of
[00:19:46] that. So
[00:19:47] >> You shipped AI inside Amazon and now
[00:19:50] inside a startup small enough to fit in
[00:19:52] one room. What can the small team do
[00:19:54] that the big enterprise simply can’t?
[00:19:56] And what did the big company get right?
[00:19:59] >> say there is one thing that the small
[00:20:01] companies can do is, you know, the speed
[00:20:05] matters a lot in the AI ecosystem. We
[00:20:08] talk about AI events for product
[00:20:11] management. We talk about very
[00:20:13] concentrated bets that companies can do
[00:20:16] and that’s where small companies
[00:20:18] startups really outshine is very focused
[00:20:21] pilot, very focused experiences slash
[00:20:24] experiments they can build rather than
[00:20:27] building the whole experiences which
[00:20:29] people have to go through. I’ll take an
[00:20:30] example in this is, you know, you can
[00:20:33] build a fully blown edtech system or you
[00:20:36] can just have a system that converts all
[00:20:39] of your learning materials into flash
[00:20:41] cards because people really love
[00:20:43] experiences of flash cards. They learn a
[00:20:45] lot. They catch up a lot and they pick
[00:20:47] up a lot. Now that can be an experiment
[00:20:49] to see, oh, do people want to use AI to
[00:20:52] build flash cards? Do people want to use
[00:20:53] AI to do learning? Is AI just a solution
[00:20:57] that people don’t want to use in
[00:20:58] education? So that can really help you
[00:21:01] prove versus when I was at Amazon, one
[00:21:04] of the things I really picked up a lot,
[00:21:06] especially in the leadership is being a
[00:21:08] customer obsession and having that built
[00:21:10] inside the metrics. And that is not an
[00:21:13] option. That is a requirement. It’s
[00:21:15] something that gets enforced in the
[00:21:18] writing ecosystem before it gets built.
[00:21:21] And that’s extremely powerful for big
[00:21:23] companies to have that narratives being
[00:21:26] presented and truly see what we are
[00:21:28] working against. So it does help you set
[00:21:31] up a guardrail ecosystem with checks in
[00:21:34] place. So there is less diversion. You
[00:21:37] can catch it early on before it becomes
[00:21:40] a massive problem versus startups
[00:21:42] {slash} small companies are more
[00:21:44] open-ended. They can pivot pretty fast,
[00:21:46] but then that also comes up with the
[00:21:47] responsibility.
[00:21:48] >> Most pilots get measured on their
[00:21:50] activity, the logins, number of
[00:21:53] questions asked. What should a tech
[00:21:54] leader measure from day one instead? Do
[00:21:57] you have your own kind of a checklist?
[00:21:59] >> From the very day one,
[00:22:01] the metrics is
[00:22:03] is the number one thing that you should
[00:22:04] start to measure. And it’s not really
[00:22:08] the number of logins or the DAUs or the
[00:22:11] MAUs. It is about the rework rate that
[00:22:15] you have to do from the very day one.
[00:22:17] Did the customers, did the user expect
[00:22:20] what they were expecting when we shipped
[00:22:22] the product or did somebody need to
[00:22:24] intervene, whether AI or whether human
[00:22:27] or any model to go and fix it. What was
[00:22:29] the cycle time to actually go and fix it
[00:22:33] and deploy? What are the net scores we
[00:22:37] improved in terms of the customer
[00:22:39] delight? And did that move the needle of
[00:22:41] customers coming back? Because at the
[00:22:43] end of the day, we are all trying to
[00:22:45] build daily habits, whether it is in
[00:22:47] music applications, e-commerce,
[00:22:49] education, health, fitness, you name it.
[00:22:52] That’s the name of the game. But the
[00:22:54] rework metrics usually gets measured at
[00:23:00] >> this gap. only there’s this phrase that
[00:23:02] I really like, verification tax. If you
[00:23:05] can’t trust the AI, checking its work
[00:23:07] eats every hour it saves you. So, how do
[00:23:10] you build something people trust enough
[00:23:12] to stop double-checking it, doubting it?
[00:23:14] >> You know, I I like to bring this with an
[00:23:16] analogy
[00:23:17] uh of a receipt that you get when you
[00:23:19] actually go to a restaurant or a grocery
[00:23:21] store. You first measure, oh, is this a
[00:23:24] receipt that I got? Is it from the right
[00:23:25] vendor? See it first time and then you
[00:23:28] start to ignore it. Then you observe,
[00:23:30] okay, the line items in your receipt.
[00:23:31] Okay, I got 10 items. Okay, good. 10 and
[00:23:35] 10, great. The third layer is you go and
[00:23:37] start to measure, oh, I saw that price
[00:23:40] for 4.99. Was it 4.99 on my receipt?
[00:23:43] Yes, it is 4.99. And you start to build
[00:23:45] the trust. And the last one is the
[00:23:47] total. Okay, I expect my bill to be
[00:23:48] around 100. It has been 100. And you
[00:23:50] wouldn’t notice until the bill spikes to
[00:23:53] 150. So, that’s a receipt framework that
[00:23:56] you need to use in your AI workflows is
[00:23:58] start from the very basics. Start to
[00:24:00] measure and then build the trust on
[00:24:02] automated basis. And earn your autonomy
[00:24:05] through that. And this is how humans
[00:24:07] have been operating since the beginning
[00:24:09] of the age. So, why let AI not be an
[00:24:13] operational like that? So, so that’s
[00:24:15] that’s how it
[00:24:15] >> So, Wendy, if a chatbot’s mistake is a
[00:24:17] bad answer and agent’s mistake is a bad
[00:24:19] action, how big a deal is that
[00:24:22] difference?
[00:24:23] >> I’ll agree that, you know, the agents
[00:24:26] have their own ecosystem. And if the
[00:24:28] companies can get the governance and
[00:24:31] verification right, the chatbot just
[00:24:33] answers, then it’s the same problem that
[00:24:35] every single organization is failing
[00:24:37] where the AI pilots are not really
[00:24:39] doing. And that’s where it supports the
[00:24:41] MIT number as well because there is no
[00:24:44] governance, there is no verification
[00:24:46] built into the ecosystem. So, agents
[00:24:48] don’t really go and fix that. They just
[00:24:51] amplify this problem even further. I’ll
[00:24:53] I’ll give you an example. I was advising
[00:24:55] one of the legal tech companies and they
[00:24:59] wanted to build any AI workflow for
[00:25:01] legal companies to come in and rely on
[00:25:04] the AI chatbots
[00:25:06] for this without having any kind of
[00:25:07] governance. And this was one of the
[00:25:09] things that I wanted to build is and
[00:25:11] suggest them that these governance and
[00:25:14] these verifications needs to be from the
[00:25:17] day one. Having an agent on top of a
[00:25:19] solution is not going to make the
[00:25:21] customer experience or customers come
[00:25:23] back to your platform just because
[00:25:24] everybody is using agent. But if you
[00:25:27] build a verification and governance from
[00:25:29] day one and maybe do it in a more like I
[00:25:32] will say hold the leash format. So you
[00:25:35] know, when you take your dogs to the
[00:25:36] park, you do hold their leash
[00:25:39] in the beginning and you see how they
[00:25:41] behave. So it becomes more of a
[00:25:43] suggestion you try to see okay, how much
[00:25:45] your dog is actually distracting from
[00:25:46] the path and then at certain stages do
[00:25:49] you you do let the leash little bit
[00:25:51] loose to see okay, is the dog really
[00:25:53] going to go out and do some funny stuff.
[00:25:56] Same thing with your agents and
[00:25:57] chatbots. See what the data is
[00:26:00] suggesting from your agents and
[00:26:01] chatbots. Do you want to collect the
[00:26:03] feedback? Yes. Do you want to build the
[00:26:04] trust layer on that? Yes. And then build
[00:26:07] the autonomy on top slowly and steadily.
[00:26:10] You know, there comes a time one day you
[00:26:12] trust your dog and you trust your agents
[00:26:14] so much that you do want to let the
[00:26:16] leash fully autonomous because you have
[00:26:19] built that ecosystem. Until that
[00:26:21] receives framework come in, your bill
[00:26:23] spikes to 150 and you’re like oh, what
[00:26:25] happened? So now you know, but even then
[00:26:27] you need to be aware that you should not
[00:26:30] have paid for that bill when it reached
[00:26:32] 150. You should know that if it was
[00:26:34] 1500, you would have already paid then
[00:26:36] that’s another blunder you would do. So
[00:26:38] have a guardrail to stop it right there
[00:26:41] when those thresholds are even getting
[00:26:43] more and more from the from the actual
[00:26:46] thresholds that they should be.
[00:26:47] >> So then we’re moving towards the end of
[00:26:49] our conversation here.
[00:26:50] What’s the one lesson that you had to
[00:26:52] learn the hard way in your leadership
[00:26:55] journey?
[00:26:55] >> Be biased and be empathetic at the same
[00:26:58] time. Bring your team along is the
[00:27:01] number one key to success.
[00:27:03] You are not building alone. I do see a
[00:27:06] lot of AI companies and AI pilots to
[00:27:10] really talk about one one person
[00:27:11] company, two person company. I’m of a
[00:27:13] strong believer that people bring in
[00:27:16] diversity and diversity bring in new
[00:27:18] ideas and more ideas meaning the more
[00:27:21] ground you can cover up in problem
[00:27:23] solving. Build your path of transition
[00:27:26] from an AI pilot to a full scale
[00:27:28] production ecosystem and set yourself a
[00:27:30] date. Don’t make it an open-ended
[00:27:32] problem. So
[00:27:33] >> That’s really important, you know, the
[00:27:35] thing that you said at the last that do
[00:27:37] not make it an open-ended ticket because
[00:27:40] it has to have a date to it or else
[00:27:42] you’ll keep stalling.
[00:27:43] >> [laughter]
[00:27:44] >> Well, Randeep, thank you so much for
[00:27:45] such a grounded and honest conversation
[00:27:47] with us today.
[00:27:48] >> Thank you so much for having me.
[00:27:50] >> [music]
