[00:00:00] [music]
[00:00:04] Welcome to another episode of TechUnhinged
[00:00:06] where tech gets human. I’m your host
[00:00:07] Rabia Javeed. Then joining me today is
[00:00:09] Chris Colinsky, CTO and principal AI
[00:00:12] engineer who’s been building for the web
[00:00:14] since 2003. He’s led engineering at
[00:00:16] companies like goop and me and he’s
[00:00:18] currently heads up the AI stack at how
[00:00:21] and when he’s not doing that he’s
[00:00:23] architecting open source frameworks for
[00:00:25] LLM powered pipelines. Welcome to the
[00:00:27] show Chris. All right, thanks.
[00:00:28] Appreciate you having me on.
[00:00:29] >> Well, Chris, you’ve been building for a
[00:00:31] web since 2003 for someone who hasn’t
[00:00:33] been tracking this closely. What
[00:00:36] actually is an LLM and why is everyone
[00:00:38] suddenly trying to wire them into their
[00:00:40] systems?
[00:00:41] >> Uh, good question. you know LLM large
[00:00:43] language model it’s just a new form of
[00:00:46] prediction but at at scale has an
[00:00:48] intention mechanism to better predict
[00:00:51] what what it should say next or what the
[00:00:53] next token is but it’s large because
[00:00:54] it’s you know billions of parameters
[00:00:56] unlike you know models in the past or
[00:00:58] maybe thousands or millions I think the
[00:01:00] other key thing is it’s trained on the
[00:01:02] readable internet so webcale data makes
[00:01:04] it a bit different you know and at that
[00:01:06] scale prediction looks like reasoning
[00:01:09] instruction followed knowledge and other
[00:01:11] things like that so it’s a bit of a
[00:01:12] illusion if you will. It’s not in my
[00:01:14] opinion at least true true knowledge or
[00:01:16] understanding. It’s just very deep
[00:01:18] prediction.
[00:01:18] >> All right. Um that’s
[00:01:20] >> why now I guess the tech’s not new. The
[00:01:22] ideas aren’t new. It’s just a slightly
[00:01:24] different approach to it. It’s just now
[00:01:26] it’s accessible at scale, right? The
[00:01:28] cost versus the ROI you get from it is
[00:01:30] debatable sometimes. But yeah, in the
[00:01:32] past, you know, 6 months for an ML
[00:01:35] project, you get some APIs. Today you
[00:01:37] can parse extract your data structure,
[00:01:40] translate, classify, summarize. It’s all
[00:01:43] pretty easy. I think that’s what makes
[00:01:44] it different for a regular software
[00:01:46] engineer. It’s just a new type of API.
[00:01:48] And I think that’s how most people use
[00:01:49] it, you know, and there’s different
[00:01:51] layers of usage. There’s those that use
[00:01:53] APIs, there’s those that like go a bit
[00:01:55] deeper and do like what I’ll call real
[00:01:57] AI engineering, and then there’s the
[00:01:58] people who build the foundation models
[00:02:00] themselves.
[00:02:01] >> Yes. And also what’s the simplest way to
[00:02:03] explain what these models can and cannot
[00:02:05] do especially for people who’ve been
[00:02:07] burnt by the hype.
[00:02:08] >> You know it’s pattern matching at scale
[00:02:10] not true understanding. So it’s uh I
[00:02:12] think the key thing to understand is
[00:02:13] it’s probabilistic not deterministic.
[00:02:16] You won’t get the same answer twice. You
[00:02:18] get similar answers if you do it right
[00:02:20] but not necessarily the exact same. I
[00:02:22] think the hardest part is that it’s
[00:02:24] competent even when it’s wrong. That can
[00:02:25] be the dangerous part of it right and
[00:02:27] it’s hard to spot that. Valuation is
[00:02:29] really important. At the end of the day,
[00:02:30] it’s what makes it reliable and usable
[00:02:32] is really the engineering around it.
[00:02:34] It’s the model is just one thing. And I
[00:02:37] think people um overvalue the model and
[00:02:39] what it can do and just you know I see
[00:02:41] this on the business side all the time.
[00:02:44] You know just LM can do this or you know
[00:02:46] use the Gemini API. It’s not actually a
[00:02:48] Gemini API. There’s you know you can
[00:02:50] call the models and get the raw language
[00:02:53] model output and use it that way but you
[00:02:55] can’t use the agent Gemini as an API.
[00:02:58] And I think people have a lot of
[00:02:59] misconceptions about how things, you
[00:03:01] know, work in that sense, you know,
[00:03:03] things that maybe it’s not good at, you
[00:03:04] know, or like deterministic correctness,
[00:03:07] math and counting, real-time facts cuz
[00:03:09] the training has a cut off. Like I was
[00:03:11] using Claude last night and it said,
[00:03:13] well, this thing happened, you know,
[00:03:15] recently and my training cut off was
[00:03:16] January of this year. Sorry, I have to
[00:03:19] look that up. You know, so they’re
[00:03:20] becoming at least aware of their limits
[00:03:22] of it, right? You know, I think that the
[00:03:24] most dangerous part is that subtle
[00:03:26] wrongness can compound over time and
[00:03:28] that’s that’s really uh challenging and
[00:03:31] they’re definitely not going to have,
[00:03:32] you know, any kind of novel reasoning
[00:03:34] beyond their training distribution.
[00:03:36] >> So, two decades in, what genuinely feels
[00:03:38] different about building with AI versus
[00:03:40] everything you’ve built before and what
[00:03:42] just looks new, but is actually the same
[00:03:44] problem engineers have dealt with
[00:03:46] previously, too.
[00:03:47] >> I don’t know if this is necessarily new,
[00:03:49] but everyone’s an expert at AI, right?
[00:03:51] And it’s really easy to fake your
[00:03:53] expertise more than past like to master
[00:03:55] like machine learning was really hard.
[00:03:57] You had to know the math. You had to be,
[00:03:59] you know, in the weeds, you know, with a
[00:04:01] like a scikitlearn or caris or something
[00:04:04] like that to, you know, really do the
[00:04:06] work and speak, you know, coherently
[00:04:08] about it, understand, you know, things
[00:04:10] like cosign similarity and stuff like
[00:04:12] that. It’s a cornerstone of of old
[00:04:14] school like recommenders and stuff. But,
[00:04:16] you know, that’s not the case today.
[00:04:17] It’s easy to be an expert. It’s easy to
[00:04:19] fake that you’re an expert. I mean you
[00:04:20] got people who have never written code
[00:04:22] writing code and winning competitions at
[00:04:24] OpenAI even right I mean take me for an
[00:04:26] example I participated in the OpenAI
[00:04:29] parameter golf contest which was like a
[00:04:31] pretty extreme model training exercise
[00:04:34] that was all about efficiency bytes per
[00:04:36] bit and using claude I think it on the
[00:04:38] leaderboard cuz it was first to lowest
[00:04:40] score or um soda state of the art but if
[00:04:43] you looked at it pure score-wise I would
[00:04:44] have got seventh out of 1300 and I had
[00:04:47] never worked with model architecture
[00:04:48] prior to that I’m technically minded and
[00:04:50] I I wrote my little like LinkedIn post
[00:04:53] about it was more about how my
[00:04:54] engineering manager skills is probably
[00:04:57] worked most. I could guide it, make
[00:04:58] judgment calls when this was the dead
[00:05:00] end, what was working, what was not,
[00:05:02] that kind of thing. And and I think
[00:05:03] that’s kind of what happened there. But
[00:05:05] I’m not an expert in model architecture,
[00:05:06] but I did well in the competition. And I
[00:05:08] think that’s definitely new. Like people
[00:05:10] weren’t able to do that before. You
[00:05:12] know, I would say old and new is
[00:05:14] security. totally different types of
[00:05:16] security problems that you never really
[00:05:18] thought about before. As I mentioned
[00:05:20] before, probabilistic outputs, which
[00:05:22] people are definitely not used to, you
[00:05:23] know, they’re they’re used to like more
[00:05:25] like item potency getting the same thing
[00:05:27] every time, you know, or it’s a expected
[00:05:30] change. Like if I ask for the time, I’m
[00:05:31] expecting at the time even though the
[00:05:33] answer might be different, you know, or
[00:05:34] the weather, you know, but the shape of
[00:05:36] it and the way it’s presented is the
[00:05:37] same. But I think the most uh the
[00:05:39] biggest difference or what’s new for me
[00:05:41] anyway is that it’s it’s a natural
[00:05:43] language interface. You’re not getting
[00:05:45] parameters, you’re getting a sentence.
[00:05:47] And that’s challenging for a lot of
[00:05:48] people, right? It’s not necessarily new.
[00:05:50] It’s just new to the masses and how we
[00:05:52] think about software engineering and the
[00:05:54] way we take input. Same problem, but
[00:05:55] rebadged. I would probably say just
[00:05:57] software engineering itself. I think
[00:05:59] everybody seems to forget that’s really
[00:06:01] what this is. You know, lab coding and,
[00:06:03] you know, I can do it and this and that,
[00:06:04] but like doesn’t mean you’re going to
[00:06:06] make something that’s going to go into
[00:06:07] production or be the one that wakes up
[00:06:09] at 3:00 a.m. when something doesn’t go
[00:06:10] right. And I think that’s the biggest
[00:06:11] challenge about, you know, kind of going
[00:06:13] back to my other new like people can
[00:06:14] fake it. If you don’t really understand
[00:06:17] the domain that you’re working on, like
[00:06:18] what it takes to really build a
[00:06:20] production app, you don’t really have to
[00:06:21] be like a software engineer to know
[00:06:22] these things. You could be a really
[00:06:24] technically minded product manager as an
[00:06:26] example, which you know, I think is a
[00:06:27] category where people are doing really
[00:06:28] well with these tools because they
[00:06:30] understand tech. They understand
[00:06:31] deployments. They understand the step
[00:06:32] and the process, what characteristics
[00:06:34] make up a good application in production
[00:06:36] like observability, security. they
[00:06:38] understand the checklist and they can
[00:06:39] make sure they happen even if they don’t
[00:06:41] ne understand exactly what they are you
[00:06:43] know but they’re still able to do
[00:06:44] software engineering in that sense but a
[00:06:46] lot of people don’t have that experience
[00:06:48] you know it’s uh and then I think team
[00:06:50] structure uh you know also you know
[00:06:52] people think oh it’s radically changing
[00:06:54] but no it still need the same skills
[00:06:56] it’s just different tools
[00:06:57] >> a lot of companies are sitting on
[00:06:59] systems that are 10 15 or 20 years old
[00:07:02] when leadership says we need to
[00:07:03] modernize and bring in AI what does that
[00:07:06] actually mean in practice It means they
[00:07:08] want to bring AI in and they have no
[00:07:09] problem definition. It’s a solution
[00:07:11] looking for a problem. Usually it falls
[00:07:13] apart because you have an existing bad
[00:07:15] process you’re trying to automate,
[00:07:16] right? It’s just going to amplify what
[00:07:18] you have more or less like your skills
[00:07:20] or your process, your code base, it’s
[00:07:23] going to amplify that, right? Unless you
[00:07:25] take the steps engineering minded to go
[00:07:27] fix that first, you know, you’re kind of
[00:07:29] just you’re piling this new shiny thing
[00:07:30] on top of whatever you got. I think the
[00:07:32] other biggest um problem and and I don’t
[00:07:35] really understand this is no good
[00:07:37] success criteria. I just want to use it.
[00:07:39] AI will make it better. Well, how and
[00:07:41] what? You know, I think people have lost
[00:07:43] sight of what business objectives really
[00:07:45] are. Like your business didn’t change
[00:07:46] because AI maybe if you’re adopting like
[00:07:48] more of like a AI product or something,
[00:07:50] but for the most part, your your goals,
[00:07:52] revenue, sales, whatever they are, those
[00:07:55] haven’t changed. And AI will maybe help
[00:07:58] you get there differently or better,
[00:07:59] slightly faster, more optimized. Well,
[00:08:01] whatever the case may be, still has to
[00:08:03] work with an existing system, work with
[00:08:05] your business. Your business didn’t
[00:08:07] change overnight because AI showed up,
[00:08:08] right? You know, other problems is your
[00:08:10] data is not ready. You don’t have APIs
[00:08:12] that it can utilize, or more
[00:08:14] importantly, it doesn’t have a true
[00:08:16] architectural home. you just kind of
[00:08:17] bolt it all on the side or use some
[00:08:19] service off to the, you know, out in the
[00:08:21] field and try to make it work with
[00:08:22] whatever you got going on, you know,
[00:08:24] like in AWS or something like that,
[00:08:25] right? There are a lot of areas where it
[00:08:27] can be really tough from the get- go.
[00:08:29] >> There’s a version of AI transformation,
[00:08:30] Chris, that’s mostly cosmetic, a chatbot
[00:08:33] on top of a broken [clears throat]
[00:08:34] system, some dashboards with, you know,
[00:08:36] AI powered in the label of it. What does
[00:08:38] real transformation look like versus the
[00:08:40] version that’s mostly just for the press
[00:08:42] releases?
[00:08:43] >> You got to modernize first, right? like
[00:08:45] your whole stack has to come along with
[00:08:47] it on the ride, right? It’s going to
[00:08:48] expose everything, every weakness you
[00:08:50] have in your foundation, you know, and
[00:08:52] as you said, they might skip the
[00:08:53] foundation and build a demo on top of
[00:08:55] it. Probably will never even reach
[00:08:56] production anyway. You know, it’s a lot
[00:08:58] about data. It’s not accessible. It’s in
[00:09:00] silos. It’s not queriable correctly. Low
[00:09:02] low latency. You know, those are all
[00:09:04] kind of problems you’re going to have to
[00:09:05] deal with. For me, you know, when I
[00:09:07] think about a project, it’s like the
[00:09:09] first part is predominantly planning.
[00:09:11] Then you have some execution and then
[00:09:13] now I layer on AI at the end. It’s like
[00:09:15] you know it’s it’s the end. It’s the
[00:09:17] last thing you think about usually. It
[00:09:19] it’s not my first thought like how do I
[00:09:21] solve this with AI like I solve the
[00:09:23] problem. If AI happens to be the right
[00:09:25] tool at any point or or component of the
[00:09:28] problem, then you can apply it, but it’s
[00:09:30] not what you reach for first, right? You
[00:09:32] know, if you’re deploying it, it’s uh
[00:09:34] you know, maybe you deploy it one piece
[00:09:36] at a time on top of the the legacy
[00:09:38] system. Take maybe more like a strangler
[00:09:40] fig approach like it’s not a rip and
[00:09:41] replace. you know, kind of pick away at
[00:09:43] your old systems one component at a time
[00:09:45] until you generally replace it over time
[00:09:47] or pick a workflow that AI might add
[00:09:50] value to or prove a new concept and then
[00:09:52] expand on it. Right? Start small and
[00:09:54] grow from there. Right. These are ways
[00:09:56] you can be successful with it and and at
[00:09:59] least in practice what I read, what I’ve
[00:10:00] experienced myself or you know how you
[00:10:02] approach it.
[00:10:03] >> Absolutely. All right. So Chris Stack
[00:10:05] Overflow recently put out a piece
[00:10:07] arguing that most AI failures aren’t
[00:10:10] really AI problems. they are data
[00:10:11] problems, bad metadata, um missing
[00:10:14] context definitions that quietly shifted
[00:10:16] over the years. Does that match with
[00:10:18] what you’ve seen?
[00:10:19] >> Yeah. Yeah, for sure. That is definitely
[00:10:21] the key issue. And there’s a couple of
[00:10:23] reasons. You know, you could be missing
[00:10:24] data like, you know, I want to build
[00:10:26] this thing and it requires, you know,
[00:10:28] three types of data. You have two. Maybe
[00:10:30] you can’t get the other one. Maybe it’s
[00:10:32] not licensable or accessible or just
[00:10:34] missing for some reason, right? You
[00:10:35] know, I see that happen a lot. Although
[00:10:37] not necessarily unique to AI, it becomes
[00:10:39] more apparent because they’re rushing
[00:10:41] into it. Like I want this thing now. You
[00:10:43] don’t have everything you need. Like why
[00:10:45] not? You know, there’s pressure there.
[00:10:46] You know, it’s in the wrong format. It’s
[00:10:48] not queryable. It’s low quality. You
[00:10:50] don’t know how to deal with those
[00:10:51] problems in the first place, right?
[00:10:52] Those are all issues. But, you know,
[00:10:53] then you come into like governance and
[00:10:55] lineage, you know, definitions, you
[00:10:57] know, all like the things nobody wants
[00:10:59] to think about, the un glorious stuff,
[00:11:01] right? Those are, you know, really all
[00:11:02] preai and the area that most people
[00:11:05] skip. And now with AI, it’s just
[00:11:07] becoming more apparent like, hey, you
[00:11:08] weren’t even really doing engineering
[00:11:10] correctly before in the first place. And
[00:11:11] I’m going to amplify that as I mentioned
[00:11:13] before, right? And then the other part,
[00:11:15] you know, is that the model doesn’t
[00:11:16] really have tribal knowledge of your
[00:11:17] business or or whatever it is. You know,
[00:11:20] the humans were always silently doing
[00:11:21] this in the background and dealing or
[00:11:23] working around the bad data themselves.
[00:11:25] Now the model just believes it and you
[00:11:27] have, you know, it doesn’t really work.
[00:11:28] You know, there’s also another research
[00:11:30] that says that 62% of the companies
[00:11:32] experience pipeline failure every single
[00:11:34] month, averaging three hours of downtime
[00:11:36] each time and AI powered self-healing is
[00:11:39] being sold as the answer. Is is that
[00:11:41] even a thing?
[00:11:42] >> It sounds like a vendor selling a cure.
[00:11:44] [laughter]
[00:11:45] But you know, it’s like um observability
[00:11:48] first. You know, you can’t heal what you
[00:11:50] can’t see, right? You know, into
[00:11:52] detection. So schema validation at the
[00:11:54] boundaries, health checks, abnomality
[00:11:57] detection, you know, using metrics, item
[00:11:59] potency checkpointing, you know,
[00:12:00] deterministic recovery playbooks. These
[00:12:02] are all core engineering things that you
[00:12:04] would do anyway, right? Like maybe um,
[00:12:06] you know, AI can help you reason a
[00:12:08] little bit better, do some human loop
[00:12:10] components for you, you know, in the
[00:12:11] middle of the night, help with triage,
[00:12:13] that kind of stuff. But, you know, I
[00:12:14] think that, you know, self-healing is
[00:12:16] something that most people are not
[00:12:18] doing, right? It’s it’s a lot harder to
[00:12:20] understand that it’s it’s maybe not like
[00:12:22] you know writing its own code
[00:12:24] self-healing but let’s say it’s like in
[00:12:25] the the lens of of quality assurance
[00:12:28] right you’re testing something it’s a
[00:12:30] it’s a live dynamic environment your
[00:12:32] test case isn’t perfect can it adjust
[00:12:34] its test case to understand is it really
[00:12:36] working or not is this temporary blip I
[00:12:39] think that kind of self-healing is where
[00:12:40] we’re going to see things first you know
[00:12:42] writing its own code over time I don’t
[00:12:44] know that’s a larger problem and most
[00:12:45] most offerings don’t even you know think
[00:12:47] about or offer for that kind of stuff.
[00:12:49] That’s really custom work that only
[00:12:50] probably the biggest places are doing. I
[00:12:52] don’t see it really coming up much. And
[00:12:54] if you’re, you know, most environments
[00:12:55] don’t even need that in the first place.
[00:12:57] It just depends what you’re talking
[00:12:58] about, right? You know, I work in like
[00:13:00] small to mediumsiz hund00 million
[00:13:02] companies. Not a concern more or less,
[00:13:04] right? They’re just trying to get cloud
[00:13:06] right in the first place, let alone
[00:13:08] layer on AI, you know? And so, uh, you
[00:13:10] know, those are all like the things that
[00:13:12] really matter and maybe what
[00:13:13] self-healing really looks like. It’s
[00:13:14] just a little bit more sophisticated
[00:13:15] version to what you should have been
[00:13:16] doing already. You know, this is part of
[00:13:18] the reason why I came about with open
[00:13:20] armature is that it’s uh it makes things
[00:13:22] like auto recovery possible. It’s not
[00:13:24] self-healing, but it’s part of that, you
[00:13:26] know, process. You know, recovery,
[00:13:28] orchestration, checkpoint retries,
[00:13:30] observability, they’re all first class
[00:13:32] primitives in the framework, right? It
[00:13:33] makes it easier to deal with because
[00:13:35] it’s it’s baked into it and not
[00:13:36] something you have to customize each
[00:13:38] time or or or whatever, right? you know
[00:13:40] triage or remediation is just another
[00:13:42] node in a graph ultimately and that’s
[00:13:44] kind of how you have to think about it.
[00:13:45] You know as I think I mentioned before
[00:13:47] it’s more about the engineering around
[00:13:48] it than it is the model itself.
[00:13:50] >> Of course traditional systems are
[00:13:51] deterministic same input same output
[00:13:54] every time. LLMs aren’t the same
[00:13:56] question can get a different answer
[00:13:58] depending on the model version or how
[00:14:00] you word it. How do you build something
[00:14:01] reliable on top of something that’s
[00:14:03] designed to be unpredictable?
[00:14:05] >> Yeah, I have a strategy catchphrase. I
[00:14:08] don’t know what you’re going to call it,
[00:14:09] but I I I talk about it as calculate
[00:14:10] first, reason second. Don’t let it do
[00:14:12] the math. Don’t let it make up facts.
[00:14:15] Like do all that pre-work first. Do it
[00:14:18] really well. I think about it as like
[00:14:20] ECR, extract, calculate, and reason. You
[00:14:22] know, retrieve, drive structured data,
[00:14:25] pre-calculate values, you know, just
[00:14:27] like you might in like an analytical
[00:14:28] pipeline or a financial workflow, right?
[00:14:31] Don’t don’t let it figure that stuff
[00:14:32] out. Do that up front because it’s
[00:14:34] totally possible. It’s just the way you
[00:14:35] always did it, right? you know, compute
[00:14:37] your totals, taxes, dates, counts,
[00:14:39] reconciliations, right? Math should
[00:14:41] never be the model’s job. And then
[00:14:42] lastly, you give it pre, you know, like
[00:14:45] uh assembled facts and it’s not just I
[00:14:48] retrieve some documents for you. You’ve
[00:14:50] actually done some work over that
[00:14:51] information that you pulled from the
[00:14:53] database or your vector store or
[00:14:54] whatever the case may be. And then the
[00:14:56] LM can interpret, summarize, side,
[00:14:58] summarize, you know what I mean? But
[00:14:59] it’s doing it over verified numbers and
[00:15:01] facts. It didn’t compute or have to
[00:15:03] figure anything out on its own. you gave
[00:15:05] it very clear direction with all the
[00:15:07] information it needs to succeed.
[00:15:08] >> Moving to the next question um rag
[00:15:10] retrieval augmented generation where the
[00:15:13] AI pulls um in relevant documents before
[00:15:16] it answers became the default almost
[00:15:18] overnight. You actually built a
[00:15:20] graph-based version of this at lunar
[00:15:22] command. What wasn’t working with the
[00:15:24] standard approach that pushed you in
[00:15:25] that direction?
[00:15:26] >> It’s a little bit different. I mean I
[00:15:28] did it cuz I needed a deterministic
[00:15:30] world state. It’s actually graph-based
[00:15:32] retrieval and action generation because
[00:15:34] it’s for a game. It’s a little bit
[00:15:35] different, but same same core
[00:15:37] principles, right? You know, in this
[00:15:39] case, I use the LLM as a textual
[00:15:41] renderer, not the game master itself.
[00:15:43] The graph is the source of truth, right?
[00:15:45] What wasn’t working, I guess, would be,
[00:15:47] you know, fuzzy semantics, similarity
[00:15:50] wasn’t really helping me. There was no
[00:15:51] structural guarantee of oral
[00:15:52] consistency. Like in the in this case
[00:15:54] like if a door was locked and now it’s
[00:15:56] suddenly open. An NPC forgot that he
[00:15:58] just talked to you. You know the
[00:16:00] physical laws of the world shift based
[00:16:02] on how it perceives it right doesn’t
[00:16:04] work. It makes for a bad game
[00:16:06] experience. And these same principles
[00:16:07] apply like customer service or anything
[00:16:09] else you know and so using uh a graph as
[00:16:12] a deterministic state engine and then
[00:16:15] layering on uh the LM as a generative
[00:16:17] narrative layer worked extremely well.
[00:16:19] As we mentioned most AI failers are
[00:16:21] related to data. um you know having a
[00:16:23] graph really works but in this case it’s
[00:16:26] not just you know a graph gives you more
[00:16:28] than just like document retrieval it can
[00:16:29] it can structure relationships um but in
[00:16:32] my case I use it you know not just for
[00:16:34] this thing’s related to that thing but
[00:16:35] it’s like a spatial relationship it’s
[00:16:37] near you it’s in the room with you you
[00:16:38] can actually touch it temporal changes
[00:16:40] things happen over time you can do that
[00:16:42] you can facilitate movement you can
[00:16:44] change edges right it’s a distributed
[00:16:47] state model um like for conversations as
[00:16:49] an example uh the NPC code will hold the
[00:16:52] conversation template and all the
[00:16:54] various pathways the conversation can
[00:16:56] go. But the history as it played out
[00:16:58] lives on the player over time. And then
[00:17:00] when they have a conversation, you
[00:17:02] interle the history with the template
[00:17:04] and now you have reconstructed the past
[00:17:06] and have a correct next step to, you
[00:17:08] know, you’re still building a message
[00:17:10] list at the end of the day, but you’re
[00:17:11] doing it correctly. And this, you know,
[00:17:13] ultimately helps long-term memory and
[00:17:15] keeps your contact window small. There’s
[00:17:17] all kinds of things you can do in this
[00:17:19] situation uh that I found to be work a
[00:17:22] lot better than like traditional rag.
[00:17:24] You know, you can have retrieval
[00:17:26] problems, stale data. You know, there’s
[00:17:28] a lot of reasons why a rag system may
[00:17:30] produce wrong answers. There are
[00:17:32] solutions for those. So, I say do your
[00:17:34] own research. My situation was a little
[00:17:36] bit differently because I had to create
[00:17:37] a consistent experience, right? A bit
[00:17:40] more challenging. I think
[00:17:41] >> teams are spending 15 to 25% of their AI
[00:17:44] budgets just on tooling to support the
[00:17:47] data layer underneath these pipelines.
[00:17:49] Is that the industry which is growing up
[00:17:52] or proof that the underlying problem was
[00:17:54] never really solved?
[00:17:55] >> I don’t think it was ever really solved
[00:17:57] and we sort of touched on this earlier,
[00:17:58] right? Data integration, quality,
[00:18:00] semantic drift, never solved it before.
[00:18:02] Never had to care about it maybe, right?
[00:18:04] Um it’s more important than it used to
[00:18:06] be. you know, things that you tucked
[00:18:08] away in your your data lake in, you
[00:18:10] know, your in Glacier, right? Like, oh,
[00:18:12] now I suddenly care about all this
[00:18:13] stuff. How do I get it back? What do I
[00:18:15] do with it? Nobody’s touched this stuff
[00:18:17] in a long time, right? AI didn’t really
[00:18:19] create this problem. Um, it just removed
[00:18:21] your ability to ignore it more or less,
[00:18:23] right? I think that’s the big thing. And
[00:18:24] it’s forcing teams to grow up. You know,
[00:18:26] you’re finally treating data like a true
[00:18:28] the true foundation that it could as a
[00:18:30] first class citizen. Uh, you know, you
[00:18:32] can’t just pretend the model is the
[00:18:33] product, right? your data, your
[00:18:35] information, your process. That’s your
[00:18:38] unique. Everybody has the model, right?
[00:18:39] I can access the latest and greatest
[00:18:41] foundation model same as you and same as
[00:18:43] everybody else, right? It’s what you do
[00:18:45] with it and what you feed it is what
[00:18:46] makes the difference.
[00:18:47] >> You also hinted upon open armature
[00:18:50] earlier, but we never really talked
[00:18:51] about it at length. So, it’s one of the
[00:18:53] frameworks that you’ve, you know, built
[00:18:55] on your own. what made you build um it
[00:18:58] rather than you know buy and what does
[00:19:00] it actually solve that the existing
[00:19:02] tools probably weren’t
[00:19:03] >> I struggled with some of the I won’t
[00:19:05] name them common tools you probably can
[00:19:07] guess uh you know they optimized for
[00:19:09] chat shaped agents right designed around
[00:19:12] conversation loops you know that’s very
[00:19:14] difficult when you needed a
[00:19:15] deterministic multi-stage pipeline you
[00:19:17] know LLM could be worked in the mix like
[00:19:19] at one step I might need it to reason or
[00:19:21] classify for me it’s not an agent per se
[00:19:23] right you know the these frameworks
[00:19:25] defaults will push you towards modeling
[00:19:27] everything as a message and LM control
[00:19:29] flow which is hard to you know shove a
[00:19:32] data pipeline into that. You know I need
[00:19:34] LLMs are a great tool for analytics and
[00:19:36] analysis. I use it for classification
[00:19:39] all the time. Um or even to like you
[00:19:41] know assess data quality or or you know
[00:19:43] there’s all kinds of really great things
[00:19:45] it’s good at but it’s not necessarily an
[00:19:47] agent. Everybody says I need an agent
[00:19:48] but you most likely don’t. You need a
[00:19:50] workflow that has LLM capabilities or
[00:19:53] reasoning capabilities baked into it. I
[00:19:55] also currently work and have in the past
[00:19:57] in heavily event driven environments.
[00:19:59] They lack first class support for that
[00:20:01] things that like you can’t interrupt it.
[00:20:03] If you do, you have to do it while
[00:20:04] you’re in your the current session or
[00:20:06] still in memory and wherever it’s
[00:20:08] running, right? You can’t save state,
[00:20:10] kill the instance later on, even a month
[00:20:12] later, spin it back up and start off
[00:20:14] where you get you started, right? Like
[00:20:16] most of them can’t handle that. They’re
[00:20:18] mainly designed for human in the loop
[00:20:20] right now. But, you know, you think
[00:20:21] about it like an agent handling like a a
[00:20:23] call center, helping someone over time,
[00:20:26] dealing with that in any reasonable way
[00:20:28] or dealing with a failure and, you know,
[00:20:30] and then and then recovering from that,
[00:20:32] not really happening there. You know, a
[00:20:34] lack of first class support for MCP in
[00:20:37] the real world, they assume you’re only
[00:20:39] connecting to MCP servers that are
[00:20:42] always on and probably hosted by someone
[00:20:44] else. when in reality when you’re
[00:20:45] building your own, you know, MCP layer
[00:20:47] on top of your existing fast API rest
[00:20:49] services, you don’t leave them on all
[00:20:51] the time. A lot of ways like, you know,
[00:20:53] they suspend when they’re not being
[00:20:54] used. You don’t want to pay for these
[00:20:56] things to be on all the time. So, you
[00:20:57] know, I had to write a custom wrapper
[00:20:59] around that fast MCP client to deal with
[00:21:02] wakeups and retries to get a machine out
[00:21:04] of a suspended state. You know, I ended
[00:21:07] up just writing a lot of custom code on
[00:21:09] top of these things. And I was doing the
[00:21:10] same thing over and over and over again.
[00:21:12] And so I took about 10 projects, looked
[00:21:14] at all the things that I kept doing over
[00:21:16] and over again and and then designed my
[00:21:19] own approach to it. It’s designed to not
[00:21:21] only overcome these shortcomings, it’s
[00:21:23] designed for LM power data pipelines and
[00:21:26] tool calling agents, right? And I think
[00:21:28] the key there for me is that pipelines
[00:21:30] agents now share the same primitives.
[00:21:32] The engine is agnostic about um if
[00:21:34] you’re doing control flow that’s
[00:21:36] deterministic or it’s an LLM kind of
[00:21:38] component, right? There’s you don’t have
[00:21:40] to have separate frameworks. So you
[00:21:41] don’t have to have awkward glue, none of
[00:21:42] that stuff. Stateless workers uh can
[00:21:45] cross process resume as I was just
[00:21:46] talking about. You know, work state is
[00:21:48] is durable. It facilitates state
[00:21:50] migrations. So things change over time
[00:21:52] if you upgrade your agent in between
[00:21:53] sessions. You can, you know, do a state
[00:21:56] migration and deal with that and recover
[00:21:58] and you remain in control about how that
[00:22:00] happens, right? I think the other thing
[00:22:01] that I did that was a bit different is I
[00:22:03] started with a language agnostic
[00:22:04] specification with conformance fixtures
[00:22:07] so that every language implementation
[00:22:09] would work exactly the same. but leave
[00:22:11] the implementation up to the language.
[00:22:13] Obviously, TypeScript and Python are
[00:22:14] going to be you know written a little
[00:22:16] bit differently and they have different
[00:22:17] um you know inherent capabilities. You
[00:22:19] don’t want to port a Python
[00:22:20] implementation to TypeScript. You want
[00:22:23] both to sort of like come from the same
[00:22:24] origin story if you will, right? So,
[00:22:26] that’s the approach I’ve taken. You
[00:22:28] know, I have a a very usable Python
[00:22:31] implementation today. I’m already using
[00:22:32] it in production. I’m working on getting
[00:22:34] support, getting others to to get into
[00:22:36] it. I’m really hoping it it gains
[00:22:38] traction. and it’s relatively new since
[00:22:40] sort of the first of the year and once
[00:22:41] it gets a little bit further then I’ll
[00:22:43] I’ll release a TypeScript version as
[00:22:45] well. It’s just that the ideas behind it
[00:22:47] are more enterprisegrade um if you will
[00:22:50] durable execution uh event driven
[00:22:53] capabilities really meant to fit in with
[00:22:56] how I build applications already but now
[00:22:58] I want to you know kind of I want to
[00:23:00] properly mix in an LLM component you
[00:23:03] know I felt like I just I didn’t want to
[00:23:04] reinvent that wheel every time
[00:23:05] >> also Chris you’ve built across
[00:23:06] e-commerce automative and creator
[00:23:09] platforms all with different rules
[00:23:10] around customer data when does the risk
[00:23:12] of sending sensitive data to a cloud API
[00:23:15] I tip the balance toward running the
[00:23:17] model on your own infrastructure. I
[00:23:19] would say in practice I don’t think we
[00:23:20] really felt we had to send sensitive
[00:23:22] data to the cloud really from a
[00:23:24] production perspective right in limited
[00:23:26] cases when we did need to it we then use
[00:23:28] self-hosted big fan of astral small
[00:23:30] three use it a lot we also use like GMA
[00:23:34] and things of that nature there’s a lot
[00:23:35] of really great small models I think the
[00:23:37] perception just as a quick tangent that
[00:23:39] everybody has to use the latest and
[00:23:41] greatest model whatever not true you
[00:23:43] know one of the things I think people
[00:23:45] also don’t think about or run into
[00:23:46] problems is cost optimization this gives
[00:23:49] be a really great way to do that. As
[00:23:51] long as your production decisions are
[00:23:53] clean, you know, and you you can control
[00:23:56] how it works. It’s your product, right?
[00:23:57] You don’t have to do these things. There
[00:23:59] are other ways to solve these problems.
[00:24:01] I think the bigger issue is on the
[00:24:03] business side, you know, these users are
[00:24:05] pulling CSVs out of your BI layer, you
[00:24:07] know, shoving them into like whatever
[00:24:09] tool. And from an IT perspective, you
[00:24:11] lose control over that. you know, some
[00:24:13] AI platforms that you pay for, uh, you
[00:24:15] know, like, you know, Chat GPT or
[00:24:17] Gemini, if you’re more on like an
[00:24:18] enterprise plan, are, you know, probably
[00:24:21] not going to use your data. But you’re
[00:24:22] still taking sensitive data out of your
[00:24:24] system. They’re just saying, “I won’t
[00:24:25] train on it.” But your users are now
[00:24:27] being creating new behavioral patterns,
[00:24:29] or at least they’re more obvious than
[00:24:30] they used to be, to like pull data out
[00:24:32] of things that they shouldn’t, and put
[00:24:33] it somewhere else that they shouldn’t,
[00:24:34] whether it trains on your data or not.
[00:24:36] And I think those are the bigger issues
[00:24:37] I see from a business perspective. you
[00:24:39] know, you can you can design the problem
[00:24:41] into a corner and and deal with it
[00:24:43] properly if if you think about it from a
[00:24:45] design flow perspective in your product,
[00:24:47] but it’s the non-product stuff that you
[00:24:50] don’t really have control over. And true
[00:24:52] human behavior of these tools, again,
[00:24:54] it’s going to amplify that. You know, a
[00:24:55] lot of users just want to pull it out
[00:24:57] and stick it in Google Sheets and send
[00:24:58] it around. You know, now they have more
[00:25:00] things than Google Sheets they can shove
[00:25:02] it into and get value out of it.
[00:25:03] spinning up free accounts and stuff like
[00:25:05] that to bypass maybe, you know, if
[00:25:07] you’re not like an enterprise structure,
[00:25:09] your it is not going to lock down your
[00:25:10] laptop or what you have access to.
[00:25:12] There’s a lot of holes there.
[00:25:13] >> Also, Chris, as AI goes deeper into how
[00:25:16] these systems are built, does the team
[00:25:18] that built the old system have what it
[00:25:20] takes to run the new one or does
[00:25:22] something have to change there?
[00:25:23] >> Yeah, I think it depends heavily on
[00:25:25] engineering leadership probably more
[00:25:26] than anything else. You know, how you
[00:25:28] approach it and how you lead lead
[00:25:30] through that. You know, if you invest in
[00:25:31] your people and want to continue to get
[00:25:33] return on that, then you got to put time
[00:25:35] into training them. Give them the tools.
[00:25:37] Help them understand how to use it. Bake
[00:25:39] it into your project plan. So, how you
[00:25:41] communicate, do your, you know, one-on-
[00:25:43] ons, your code reviews, your, you know,
[00:25:46] when you team up and work on stuff
[00:25:47] together, right? And you pair up. Baking
[00:25:49] that in as a leader or a mentor is
[00:25:51] really key. You know, I think a lot of a
[00:25:53] lot of teams can and should be able to
[00:25:55] evolve, you know, is are they given the
[00:25:57] chance or are they set up for success or
[00:25:59] not, right? you know mindset and
[00:26:01] practices and or structures might need
[00:26:02] to change a little bit or rethink how
[00:26:04] how you approach stuff but you know in
[00:26:06] reality usually non-engineering
[00:26:08] leadership will want like want teams to
[00:26:10] evolve they want to say their companies
[00:26:12] you know using AI but then they don’t
[00:26:14] provide any support to do so money for
[00:26:16] training time to learn just figure it
[00:26:18] out as you go can be dangerous and it
[00:26:20] all starts at the top on this one I feel
[00:26:22] like
[00:26:22] >> a recent study found that roughly one in
[00:26:24] five companies in retail finance and
[00:26:26] healthcare are actually running AI in
[00:26:28] production the other four are still
[00:26:30] watching. What are those four concretely
[00:26:32] falling behind on right now? Not in
[00:26:34] theory, but practically if we look at
[00:26:36] the bright side of AI,
[00:26:38] >> I mean, I think the reality is they’re
[00:26:39] probably using more AI than they think
[00:26:41] or realize because it’s through the lens
[00:26:43] of their vendors, right? Tools and
[00:26:44] platforms they already use and pay for
[00:26:46] are adopting quickly. Um, you know, if
[00:26:48] you use Figma now, you’re using AI,
[00:26:50] right? Like whether you realize it or
[00:26:52] not. And and I think that’s a great way
[00:26:54] to go, honestly. You know there’s
[00:26:55] different, you know, layers in which you
[00:26:57] can use AI and it really depends on your
[00:27:00] engineering culture. Do you have
[00:27:01] engineers in house? Do you outsource?
[00:27:03] Some companies, especially small ecom,
[00:27:06] have like no engineer or struggle to
[00:27:08] keep engineers on staff or they rotate
[00:27:10] heavily because they’re like on Shopify
[00:27:12] or whatever. No offense to Shopify. I
[00:27:14] think we’re not running AI usually means
[00:27:16] we’re consuming it but didn’t know it or
[00:27:18] realized it or thought about it at that
[00:27:19] level, right? I think there’s always
[00:27:21] going to be a gap between those that
[00:27:22] build and those that use ultimately. And
[00:27:24] that’s really what this one in five
[00:27:26] means to me.
[00:27:26] >> So Chris, last question for a CTO or an
[00:27:29] IT director listening to our
[00:27:30] conversation today. What advice would
[00:27:32] you give them when it comes to, you
[00:27:34] know, where from where should they start
[00:27:37] with AI? Perhaps
[00:27:38] >> you just got to start using it. I mean,
[00:27:40] use it daily. Use AI search mode in
[00:27:42] Google. Gemini, it’s baked into your
[00:27:44] workspace. Have it help you with your
[00:27:46] Google Sheets. Have it help you, you
[00:27:47] know, one of my new favorite things is
[00:27:49] just the Ask Gemini thing where it
[00:27:50] slides out, you know, and you it can
[00:27:52] like help you write an email real quick.
[00:27:53] It can um help you respond to something
[00:27:56] on LinkedIn. Maybe you’d be surprised
[00:27:58] the context it has because it can kind
[00:28:00] of see what you’re actively doing. I
[00:28:01] mean, you know, Perplexity and and
[00:28:03] others have that kind of functionality.
[00:28:04] I think that’s super helpful. It’s more
[00:28:06] of like a casual easy way to introduce
[00:28:08] yourself into what’s possible, right?
[00:28:09] And those are really great use cases to
[00:28:11] start to wrap your mind around about
[00:28:12] what works and what doesn’t, you know,
[00:28:14] or or dig into like a heavy chat
[00:28:15] interface. And it doesn’t have to be
[00:28:16] work orientated. I built a mountain bike
[00:28:19] recently using a really an older frame
[00:28:21] because I wanted to go full suspension
[00:28:23] and it had like all kinds of weird
[00:28:24] compatibility problems. Gemini worked me
[00:28:27] through parts compatibility, build a
[00:28:29] list of parts I needed. Every part I put
[00:28:31] on there worked and I had deep
[00:28:32] conversations with it about like how I
[00:28:34] was going to assemble this thing, you
[00:28:35] know, specifically for for downhill
[00:28:37] mountain biking and picking the parts
[00:28:39] and, you know, some I had to buy used,
[00:28:40] some I could buy new, how they can fit
[00:28:42] together, how I made, you know, parts
[00:28:44] from different eras work together, how I
[00:28:45] could bridge those gaps. I was kind of
[00:28:47] blown away at how it helped me in the
[00:28:48] real world doing something non-digital
[00:28:50] and so looking for ways in which it can
[00:28:53] you know help you out like build
[00:28:54] something at home do an electronics
[00:28:57] project with your kid it doesn’t have to
[00:28:59] be business right you can make it fun
[00:29:00] too you’re still learning how to use and
[00:29:02] work with AI and then from there you can
[00:29:05] get in you know like try to prototype
[00:29:07] something depends on your role I guess
[00:29:08] if you’re business focused you know
[00:29:10] maybe help it with writing you know you
[00:29:12] can have it help you at home
[00:29:14] conversational discovery is something
[00:29:15] I’ve been deeply be involved with at
[00:29:16] work and a lot of AI products are around
[00:29:18] providing data related to that and so
[00:29:21] you know using AI to help you shop and
[00:29:23] um you know getting my wife a perplexity
[00:29:26] uh subscription like was life-changing
[00:29:28] for her [laughter]
[00:29:30] and it’s just these you got to start
[00:29:32] small but start meaningful I guess you
[00:29:34] know it’s it’s like you know you want to
[00:29:37] learn don’t read the encyclopedia that’s
[00:29:38] tough right or you want to learn like a
[00:29:41] new technology or try a new framework if
[00:29:44] you don’t have something that’s
[00:29:45] interesting you to build a hello world
[00:29:46] or following some random tutorial is
[00:29:49] terrible. It’s hard to engage, right?
[00:29:51] So, make it personal. Make it, you know,
[00:29:53] something you care about and it’s, you
[00:29:55] know, it’s an easy fit.
[00:29:56] >> No, I think that’s that’s a great piece
[00:29:57] of advice, Chris. And thank you so much
[00:29:59] for this insightful and grounded
[00:30:01] conversation with us today.
[00:30:02] >> No, I enjoyed it. It was really great. I
[00:30:03] appreciate you having me on.
[00:30:04] >> Thanks a lot. And to everyone listening,
[00:30:06] thanks for joining us on Tech Unhinged.
[00:30:08] Until next time. [music]
