GenAIOps 102 — Or Why Your Agents Keep Breaking

0:04 Oh, I guess we’re I guess we’re ready. All right. Hi, guys. Um, quick question. Who was here last year at FGS for the data meetup? Can you raise your hands? One person. Okay. Not that many. Okay. Yeah. Cool. So, a bunch of newcomers. Um, nobody asked me for this, but I made a second series of a presentation that I held last year about uh, genops and ebels. Um, very familiar to very related to the topic that we had previously with a little bit of a different twist. So, I hope you’re going to enjoy it. Um, so this is titled Jenny Ops 102 or why your agents keep breaking a lot of experiences from practical experience. So, we’re starting off with a very simple scenario, right? We’re having a a customer that is asking a chatbot, hey, where’s uh the refund for my order? And the chatbot answers with a very generic and u not really helpful answer. So, what broke? Um the satisfaction score is low. The analytics say all good. Five different op uh five different opinions say very different things about the quality but nobody really owns the question and nobody really owns the problem. So we’re kind of like doubling in the dark and now what to do right this is essentially where we’re landing exactly at this domain geni ops derived from dev ops originally evolving later into ML ops and now we’ve got this term that I mean has different flavors you can also call it LLM ops you can call it agent ops yeah if you heard about it um it becomes harder the more you move to the right because we are essentially now testing non-statistical uh statistical models. We are testing statistical models that are uh not always answering the same way to the same questions kind of like humans do. But we want to measure quality and um there is different metrics than a pure code that we are looking at which is traces um signals and uh nuances.

2:28 Very tough stuff. Uh very tough But essentially we um we are landing at a spot where uh we are kind of like overwhelmed with this whole topic. We’re just going with prompting, improving our agent, and just pray that it doesn’t break in production. Great. Okay, let’s start from there. So, we noticed the bug. So, we’re just going and say, okay, we are doing a few prompt tweaks because we can already identify where in the system prompt the issue comes from. So, this in our smoke tests. But then two weeks later, we notice that a user is still complaining about the same issue. So, we didn’t essentially we didn’t essentially fix the problem. We just delayed it and did a few changes to a prompt that we didn’t exactly treat how we would treat normal code. We need to unit test it. And that’s why we’re landing at the unit test suite that you’ve never built and we call it eval. This is a problem that we had from 2003. So I’m basically comparing it with uh how we started with unit tests and nobody really wanted it because it slows us down. Uh but now we are essentially at the same problem but just evolved.

3:52 But we’ve been already doing this for quite a while. So there there is systems in place. We got a beautiful presentation earlier that uh showed how they do it in their company and uh so others are doing it and aligning on a certain workflow and how to continuously improve and stabilize your agents. And this is one way how you can test it offline. So you’ve got your case that you’re describing, then you are prompting it to the agent, you’re getting a response, and then you got a range of referees. we call them LLM judges that will provide you a verdict and that will be then dumped into an analytics layer that you can look at for your whole history and that is extremely powerful knowledge. So once we land the bearer, we can release our agent into the wild. And this is where people like to distinguish between the offline part of agent evaluation and the online part of agent evaluation, which are both fundamentally different but extremely relevant for your cause. Offline is what the user won’t see and you can control within your own sandbox environment. online is going to trace the signals, user satisfaction, uh, thumbs up, thumbs down, um, yeah, things like that, right? That you’re then, um, evaluating and essentially putting back in into your development environment if you see a case that you haven’t covered so far.

5:36 And this is a very simplified version of a agent development life cycle including your evaluation online and So the top part you can kind of see you’ve got the specification, the alignment, the building, the evaluation that is happening. Then we’re gating this in a PR. We release it when we’re satisfied or we go back to development. serve the new version of the agent and essentially evaluate it and triage the results. When we find something interesting, we bring it back into offline evaluation and that’s how we continuously evolve or improve our agent. Okay, so that sounds really good. And we also got a lot of tools that um that are part participating in that race. And here we’ve got the big winners of 2026. Some of them got even acquired by uh by bigger companies. On the left side we got um uh observability and telemetry tools like Langfuse Berlin company um Rise Phoenix that is using open telemetry as a standard and lang that is um yeah it’s just been there for so long that’s why it’s there and then we’ve got brain trust DP valas uh for different cases in in evaluation whether it’s data whether it’s like simple unit tests or like a whole suite that is providing you a a really big ecosystem. Um, one thing that I didn’t mention, yes, we need to version our agents how we version code. That is really a that is highly recommended.

7:25 Despite all of this, the complexity of the agent also kind of defines what kind of tool you want to go for. Going too simple and you might need to overengineer a lot of stuff. going um too defined, locking in to a specific tool might lead to missing out on specific signals that you want to catch that you might need to customize or wrap around if you already decided for something. And the hardest truth is evaluation or geniops is usually the question that’s asked after the team already decided for the agent framework they’re using. So in my case I’ve been working with Marstra in uh in the recent in the recent months a lot and that is a very opinionated agent framework that um carries his own e belts but for the use case that we had we needed to further evolve them. So yeah, these trade-offs, they need to be considered when using tools. It’s a it’s a whole different topic. So we’re not going to dive into this. The most important part about geni ops is actually not really the tech. It is a uh field, but uh often overwhelms engineers and kind of like needs everyone on board. And uh here we’ve got the team that I would build when uh when when I have a geni ops team. There’s one engineer and there’s two non-engineers.

9:03 One of them as the product guy says, “Okay, this is good enough. This is um what we’re going to uh what we’re going to deem successful in in our uh in our cases in our tests.” And we’ve got the domain expert who’s going to tear everything down and say, “Okay, what you built there kind of doesn’t make sense. in the middle is the engineer. So anyway, um it is thus a very complex topic, usually very underestimated. Uh usually people are like, okay, I don’t really see the point in this. So yeah, there there needs to be a lot of um alignment happening. uh maybe also not trying to overengineer uh things at the start and uh first proving some simple cases and then we can prove things such as we can actually ship confidently. We can treat our prompt sections as a um as a versionable code that we uh that we have full control over because we have 50 different cases that we run over a single section in theory.

10:21 uh and therefore we can make better judgments over whether our agent is actually improving or degrading. That’s great. And therefore we’re boosting productivity. Just saying that here. Now the hardest part is actually to getting the right insights. So what does that actually mean? What kind of signals do we want to go for? what are the relevant ones that we want to pick to determine whether our agent is uh is doing a good or a better job. I’m bringing up a negative example here from um a case that was in 2024 when uh 700 people got fired by CLA because uh they wanted to replace everybody with AI yeah didn’t really successfully work. Is anybody working at CLA here? What [laughter] are you working at? Okay, that’s good. All right. Yeah. Um yeah anyway so the story was like they wanted to improve cost they only measured for cost and then they uh kind of like dropped in quality. It’s a very very simple scenario but I mean you got to transcend this into into evaluation cases for for offline or online uh quality data as well right if you’re uh testing for one thing maybe you’re scrutinizing the other.

11:54 Anyway, this is my favorite slide because it’s um it’s telling so much about the things that you can actually run into. Famous last words. um we’re just going to do one tweak or one improvement and then it just has so many repercussions. Starting with it was never only the It really depends on what kind of platform or framework you’re using. Um if it’s um all provided by the platform, maybe there’s not so much to test for. If it’s purely custom, so let’s say you’re still brave enough to use lang chain in 2026, then um yeah, you have to test for memory, uh routing, um context, windows, and all of that stuff. Is my time up already? [laughter] That was quick. So why am I writing three sub uh suspects here? We are essentially testing for agent quality right so if we catch an agent degradation great sometimes we might even have eval so that’s uh that’s an uncomfortable truth that uh geni ops engineers need to face because eelss are code that you need to maintain from now on as well and the third one there’s also scorer degradation so the LLM judges that you’re building you also need to regularly test them if they’re still doing the thing. It’s like going to the doctor and, you know, going for a checkup. Those things can happen.

13:32 Then there’s cultural stuff that we talked about, right? We need to agree on the stop rule. When is something actually good? Do we uh want to overly fine-tune towards a specific uh direction or do we say okay we are good with 80% and we are going to revisit this uh the specific scenario later to uh to not get uh caught in on uh unnecessary details. I’m skipping things here. Adoption is the hard part I already mentioned. Um data literacy is is something that also needs to be understood across the practice group. So if you are tracing signals, you’re outputting data, you’re defining your uh your passcate, uh people still need to understand the data that you’re um providing in your analytics layer and um yeah, that’s where it can get tricky. So alignment needs to happen there as well and Um by evaluating online data, what many people don’t really talk about is the uh compliance issues that uh that you all face. So that is tough by itself to to find ways to to getting that user data that you desperately need to build your data sets. Um and among everything your tech can still suck.

14:58 that’s a job at my side actually. So um the last part Well, to put it in other words, the the dream that I have every alternating day is that my geni ops pipeline will do all the observability, all the fixing and all the agent improvement for myself. So, how would that work in in practice? Right now what we are at is that we are augmenting the uh the perspective for for our uh for our agent performance. Instead of doing one test, we’re doing 10 tests. The resulting benefit from that is that we accelerate our learnings and uh the uh uh the uh improvement cycles of uh of our of our agent product. The dream is that we can automate this whole life cycle. So when we uh triage cases that uh that users have uh have we put them automatically into our into our eval data sets and then [clears throat] automatically improve our um our agent based on these new uh new cases that comes with a caveat. So what can happen is that you are just looking for the wrong uh the wrong examples.

16:24 Essentially, if you’re taking a signal like thumbs up from a user chart, one bias that could happen is that you’re only taking at the single examples that were the final answer. And the final answer is the one where the user is usually getting a whole like a whole like summary or like a very very long output from a from their original request. Maybe an agent will then learn ah every output needs to be verbose. This is what the user wants. And in that case, we can create data sets that will uh sway our agent in a in a certain way that we we don’t really desire. So I guess my dream will never be fulfilled. But um I guess what you can do instead is you can harvest for candidates with an agent and then a um human will review this and approve of new cases. Oh, that was pretty fast. Okay, so yeah, we we’re kind of done. Um now I’m giving the conversation back to you guys. I’m very ready to hear some war stories from your side.

17:53 Anybody that wants to ask question or share stories, someone built an agent so now has a story. I mean I can share that definitely when there is no consistency in the knowledge that’s where the agents will fail because they don’t know exactly what the correct answer is and there is amb ambiguity in in in the knowledge and that’s definitely where we see the valators go down. So when when we notice this, we definitely had to go to the human part like why there is this inconsistency in the knowledge in the first place and basically was because you know probably there were two different domains that never talked to each other but the the project actually you know created that space so they could speak in the same room. That’s what I want to share. somebody else. Hi. Um regarding your last part about LLM in your dream that improve themselves it’s actually happening um in the cyber security field especially uh there is um how do you say um a field of info security that is called capture the flag and there is basically an LLM competing on itself on his previous interaction making reinforcement learning to actually extract uh the sensible information like I think the cyber security is quite a thing. Um especially since Fable 5 has been used on many of the major infrastructure for software as a service like IBM and stuff like that. So actually one of those your dream came true. [laughter] Thank you so much. Oh, okay. It’s not working anymore. Thank you. That’s cool.

19:56 Let’s chat later. That sounds Any question? Standard questions also are a load. Hey, um, one of the things that keeps bugging me and I can never decide what I think on it, you’ve got an agent running and then you test it with another model to see what you think of the answer. And then yeah, you might pin your versions, you might upgrade, but if something changes, is it the agent that’s now failing? Do you evaluate the agent? Do you evaluate the evaluator as well? Um, what are your thoughts on this? Uh, okay. So, we talked about like the the three culprits, right? Um, I want to comment on one thing actually. um testing the reliability of a score is fairly simple. Uh so that one is kind of like out of the game. The problem usually is with the last two candidates, whether it’s an evil that’s uh flaky or unreliable or it’s the agent. Um what you can test for is a confidence So when you’re having a little bit of fluctuation, which usually happens with your LLM judges, right? And an agent answer is always a little bit different to your question as well, then uh you can see over time how that has evolved.

21:34 And if you see a strong dip, then that is a suspect for a case that has degraded with your changes. What you also need to do though, you need to trace if the change that you did in in your feature is related to that specific eval. So there there needs to be some yeah some some things that you need to check to be fully sure otherwise you got humans to to check on it. And would you pin your model versions or would you upgrade? Would I pin my model versions or would I upgrade? So yeah, so for the models you’re using both for the agent and the evaluators would you pin the version? So I would version everything. Yeah. [snorts and laughter] Yeah. I mean like I said eels are also maintainable code at this uh this point. So you’re maintaining a data set versions too. Maintaining agents versions, data set versions, score versions if you want to. You can also over complicate these things quite easily. Just mentioning. Okay, no further questions. Let’s stop here for uh thank you.