Building Reliable Data Agents: Our Evaluation Framework

Introduction and context

0:04 Hello everybody. Sorry for the technical issues, but you know, usually these things happen. It was just a cable. Um, welcome. I’m Fabian Marino. I work in the for more than five years. I’m a software engineer and is Felix. He’s a cloud engineer. He’s been working with us for more than one year. And today we are going to present um this talk about building reliable data agents and how you evaluate those agents. So the first thing I want to ask is who who’s familiar with the dialogue? Okay. Okay. A lot of people cool I don’t have to present this slide maybe. Yeah. But in general for those who are not familiar with the is just a price comparison tool in which you go and you search for a product and if you want to know the best price you will get it from we are um um this company which also has a mobile app. So if you want to download it and if you don’t have it in your phone be careful because Black Friday is coming. So if you want to check the price on something, it’s a it’s a good thing to have in your mobile phone. So yeah, we are this company. We’ve been for more than 25 years in the market. We also one of the best price comparison platforms in Europe and we are in more than six six countries now and waiting for the next year to expand to more

Data agent use case

1:38 So let’s get this party started. Um Eidalo has data. We are a datadriven company and we have a lot of you know data facts but basically we have the muscle to process a lot of data and we connect merchants who publish officer offers and users who interact with the platform. So we just decided like hey you know we have the muscle to to build data products and internally and this is what we did. So we just come up with this hey we have this way of consuming data internally. So we have a business user who has a question about the data and you know this person’s asked like for example it’s in a meeting with uh a merchant and they you know the merchant want wants to know hey what is the best category for the last month or you know what’s was the best product for the last month. So these are normal questions that u you know data unit process but you know sometimes is it takes time you know if the question is not in a it will take time to to get the report so then they will ask an analyst and it will take some days to process the SQL and make sure that everything is correct but this is not you know the the main focus of the solution that we come up with the the focus is like what if you do follow-up questions then it will take more time. So this is what we think we putting AI. So now everybody puts AI on top of everything. We did something similar. So we created a conversational agent from which you can ask directly these type of questions. Our goal is not to replace analysts. That’s not the goal. But just you know the regular questions that you get every day. you know these are things that an agent can definitely ask uh faster. So you will get these answers in minutes and not in So but this is the thing with with an agent [clears throat] you can you can have like different knowledge right it has all the knowledge from the internet so but you need to know and you need to make sure that you know it’s answering questions about your business. So how do we enable this?

Agent architecture

4:15 So this is what we create. So now this is kind of the architecture of the solution. So now the business users goes to a chatbot UI, you know, just a regular chat and ask the questions in in English and behind the scenes is using AWS engine core runtime. So what is this? This is just a framework to manage agents. So you don’t have to think about you know keeping the agent session the uh and tracing everything what’s happening behind the scenes. So we use this because we didn’t wanted to focus on the whole architecture but actually making sure that the agent can response uh correctly. So behind the scenes the agent has access to Atheina it runs SQL queries and then it has access to the data lake. So we decide what exactly the agent has access to. We just give them like a specific tables, a specific metrics and this is the knowledge. So with the knowledge we control exactly what the agent uh will respond. So it doesn’t you know in you know invent metrics or if somebody wants to know something specific it doesn’t you know make things up. So in this way the knowledge is defined in markdown files. So this is very beneficial because analysts will um make changes to this and they will decide which knowledge comes into the agent and it’s just plain files so it’s easy to to change.

5:49 But this is not the most important thing about the agent. Uh the most important about this agent you know nowadays building UIs building agent stuff is easy right? You can just buy code stuff, but you need to make sure that the answers and the responses are actually correct. And this is where I’m going to pass the mic to Felix so he can explain how the valuation framework works.

Evaluation framework

6:16 Yeah, thanks Fabian for introducing the project. And Fabian raised the question, how can we make sure the agents actually correct? And we do this uh with uh and we can quickly explain how they So imagine you ask a question to ch in the yellow context. This could be something like um what’s the cheapest um offer for product XYZ and then the agent looks at the question um looks at the knowledge. Okay, it might be in this table. It builds the query and runs the query for Athena and comes up with the the cheapest offer is €99 and then we ourselves for that specific input we define like an expected response that we know to be true. So for example the cheapest offer is uh €899 and then we take the statement response and expected response and passing into an evaluator and the evaluator will say okay uh that was a pass or fail. Um this combination of input and expected uh u response is evaluation uh scenario and in our project we have around 45 scenarios right now and we can talk a bit more about the evaluator. Our evaluator is based on a technique called LLM as a judge which just means it basically called another model. We use a haiku based model because it had good evaluation performance and it was still quite affordable.

7:45 And by using a model um you also get a lot of flexibility. So you pass in the agent response and expected response but you can also pass in some instructions via the system prompt on how to evaluate. Um so this site also shows some flexibility. So the agent seems to have come up with the correct answer but it also seems to have rounded up by 1 cent. And for an analytics needs this is okay. So we tell the agent okay minor rounding is okay still pass the the evaluator. And just for your information, there’s another type of evaluators uh evaluator which is code- based ones uh where you call a lambda and you have a Python script and it uses that to evaluate. Um but there doesn’t work as well with conversational output because it’s unstructured. But if you have structured output, it can be like more deterministic and cheaper and faster to run. So now we can talk a bit about why we

Why and what to evaluate

8:37 evaluate. Um so it basically enables us to consistently measure agent quality. So in the software engineering world you have test-driven development where you might have a function and you write some unit tests to cover that function and you make sure it works and you could make some changes to the function and see if there are no regressions. Um and the basic the same thing applies here. So a valuation framework is kind of like a test suite but there’s a core difference is that the agent kind of is like a black box and um produces nondeterministic outputs. So this evaluation framework is like a way to bring back consistency and this plays out in two ways. So you can establish a baseline. So let’s imagine you create like a new agent and you have 10 scenarios and it’s scoring 9 out of 10, right? So you know, okay, it’s it’s doing fairly well. Um but then you can also understand the impact of further changes and we think this is really important. So for example, you add a new table definition to the knowledge because you think uh that will get you from a 9 out of 10 to a 10 out of 10 or but that could also have some negative imp uh uh impact. Uh for example, maybe you know your contact size is too big and the the the quality diminishes. Or another example, you want to save some costs or make the agent respond faster. could use the effort of the model maybe switch from high to medium and see is it still passing um the the n out of 10 uh checks. Uh and so this is really important to build up this dynamic in this practice to make some change on the agent side and the setup or the knowledge and understand how it impacts the output and get a feel for that uh as a developer of the agent.

Evaluation dimensions

10:20 Now we can talk a bit about uh what dimensions we evaluate. So we started this section kind of talking about correctness and fast correctness is the most important but you can also evaluate other things. So we have seven dimension we used in this first iteration of this project. Uh we split them into hardgate dimensions and softgate dimensions. The hardgate dimensions are a bit more important and the soft gate ones are a bit moreformational. With the hardgate ones when you make a change to the agent and there’s a regression there you won’t be able to u merge that change. And we can also define the hardgate ones real quick. So correctness we already talked about and we also focus mostly on uh numeric correctness. So you can imagine like okay what was the price trend of product this product XYZ in the last month and then it might be okay went up by €10. Now kind of checking for now the user asks okay but why was it this change explain this is a lot more difficult to assert because the knowledge you kind of need to build up a framework to explain reasoning also in the expected response you need to be really sure that’s that that’s actually the reason so we focus on numeric correctness then with SQL correctness we have a way to analyze the the quality of the agent the quality of the SQL the agent produces so in the knowledge we provide metric metric and table definitions but we also provide like example SQL so for example like an SQL bit of how to calculate a certain metric uh certain caveats data sources and with this evaluator is a bit different you know before I explained okay you have agent respond uh and uh the expected response in this case it’s more like you pass in the expected escrow it should run and then the SQL queries the agent actually run ran and this is again where the flexibility of this LLM as a judge technique comes in because now you can say okay if it used the correct tables get one out of five scores if it didn’t use any deprecated uh columns give it two out of five and so on. Uh grounding and completeness are a bit more simple uh simpler uh grounding is a way we can make sure that the agent is not hallucinating. We do this by checking that any facts mentioned in the output comes from some kind of Athena query tool call. The Athena uh in the SQL ran might be wrong but at least it’s not hallucinating not getting the information from the internet or it’s not using the set 5 um training data and completeness is a way we can make sure that the agent addresses all the user questions. So if the user asks okay what’s the price trend for product A and product B that it actually addresses it for both products not just one and then we can skip about the over the softgate dimensions and now we can quickly talk

CI/CD integration

13:02 about how we integrate evations into our development cycle it’s basically integrated into our CI/CD pipeline so let’s say you create a pull request making an agent change um you think maybe adding this table will bring it from a 9 out of 10 to a 10 out of Then when you open the PR basically uh temporary agent runtime gets deployed the evaluations are run the 45 scenarios and the results are published to a report um and then the PR author is able to look at the report we create this report it’s basically like a HTML page that we publish to GitHub pages and it contains all the important information so the input the agent output the expected output uh all the SQL queries that ran but also So with this element as a judge technique um the model actually explains how it came to that decision why it passed or fail and will explain itself. So the PR author considers that all sees okay the change I made has the expected out outcome or maybe some unexpected outcome uh will refine the PR if needed and if there are no regressions on the hardgates they’re able to merge and the temporary agent runtime gets uh destroyed again.

Lessons learned

14:13 So now to summarize, we kind of talked about what evaluations are, why we do them, what we evaluate on, and how we integrate it. And now we just want to share some lessons learned uh that hopefully you can also integrate into your project. So we recommend starting simple. Um well, starting simple will really allow you to do this kind of okay, you make a small change to the agent side and you see the effect it has on the output. And we recommend doing this in three ways. So start with one dimension for us. This is uh correctness. It’s [clears throat] always in most cases going to be some kind of correctness. But you could have for example a customer care agent where you think uh friendliness is the most important. But focus on one dimension in the beginning and then focus on outcomes before reasoning. This is again we focus on numeric correctness rather than reasoning but also efficiency. For example, we don’t care as much that it ran the most efficient SQL. If it got the answer right, it’s okay. And then you can worry about uh how well it created the SQL later. Another thing is pin your variables. So let’s say you want to don’t say like okay what was the price history of product XYZ last month. Say define it as like what was the price history between the 1st of January and the 31st of January. And that way uh that that scenario stays static. Uh and when you run your evaluations over and over the results will be more and then uh define your uh evaluators precisely. So we had some issues with this in our team when we started out which led to some confusion. For example, guardrails evaluator could mean many different things. could mean that you didn’t run select star on in a query or that you didn’t access outside internet or that you’re not talking about nonidial data and in the end we decide okay it means that when you talk about a metric that metric needs to be clearly defined in the knowledge don’t make up some kind of nonialo metric and when you define this precisely you can develop with a lot more confidence and also when you talk to your stakeholders you have more confidence because you can have a good better idea about the coverage the evaluation framework has and you can be more certain with its ability.

16:26 And lastly, yeah, build evaluators uh evaluations into your workflow early. Um so start early. Um don’t start when you have a huge knowledge base with thousands of ND files and a vector database because you will lose this ability to track. Okay, you make a change on the agent config side and you understand the output side or the impact it has on the output side. And lastly, um, build evaluation, uh, build into your workflow. So, make it part of your CI/CD pipeline. Um, and also we recommend creating this kind of report. That way you don’t have to go through cloudatch logs or open telemetry traces. Uh, it’ll be like a nice report that’s easy to use. Yeah. [clears throat] And that concludes our presentation. Fabian has one more thing to share before we get to our Q&A. Yeah. Before we get into the questions, we we are also it was mentioned at like you know third job covers. So also in in if you are interested in this type of projects or you want to build some new skills or you think you can improve these things definitely we have some open positions so you can go to jobs.com or you can reach to us so you we can

Q&A session

17:34 also refer to you. So thanks everybody this was the talk. actually. Yeah, that’s okay. Let’s start with the Q&A. We have a little helper here. I mean, it’s not that little. Uh, she doesn’t work for FGS. I mean, even if there is no sticker questions from [clears throat] the audience, I can also say about you. No, no, [clears throat] because it’s It’s streaming. Yeah. So now I’m quite scared that I know. Um what kind of agents did you develop in your use case and uh what were the results of your evaluation? Um so we just focused on one agent and this is like in bedrock agent core. So you have the runtime there and it uses like strands and it runs on sonet 5. And when you kind of set up the evaluation framework first, it’s good to okay, we have these 45 scenarios and before we started the testing phase and giving it out to test users, you want to make sure you have like a 100 out of 100 on those. Yeah, that’s kind of but what was the use cases behind it and the evaluation results you got? Oh, so the use cases um so basically it’s for analytic needs. So instead of asking the analyst, you can just ask it.

18:57 Um and yeah, we have those different dimensions I talked about. So one of them is correct. So like if a business user wants to okay you know what is this price trend is quite simple but on some kind of analysis ad hoc analysis uh we you know you need to know that it’s correct and that’s what this correctness about. So you will we have these scenarios and they kind of ask these kind of questions and we ourselves talk to the analysts they will write the results and then um yeah that’s how we cross reference this and make sure it’s correct. Hello. Yeah. Hi, I’m Adish. I have like one quick question on uh how do you uh handle uh you know uh from real life scenario people would ask vague questions or uh you you said you have the set of 45 scenarios and there might be like paraphrases for the same question itself. So do you check whether it’s robust to paraphrase vagueness and all of that? Uh you mean like checking consistency or like if I ask the same question in a different way. Sometimes the users would be very vague about what they want and then [clears throat] you know the system will have to ask for a clarification.

20:18 Yep. like some cache like the same answer is not um replied twice. So typically if it’s a very educated user uh who’s from the domain they might ask very precise question and agent might understand what what the intent of the user but sometimes they would just ask uh questions that can mean multiple I have an example of that dumb question. So when you said uh which is which is the best product that can mean best product in terms of visitors that can be best product in terms of sales. How do you handle such kind of a scenario? So yeah in this case in the agent we have we have introduced two domains inside the company and these were the things that we noticed when we just uh gathered the knowledge because these things are very clear like for one domain like you know for marketing it could be a different definition than for B2B. So with the knowledge since it’s we start seeing the evaluation going down when things were not clear like you know that specific definition uh has to be in the knowledge. So we have like a knowledge for one of the for both domains and if it’s not consistent the valuations definitely will go down because we have questions for both. And what we did is like hey you know both domains need to sit down and talk and have a single definition or they just can split. So the the questions are defined by the context like if you are asking by B2B then it will answer by B2B. So yeah, this is tricky but I was asking the follow-up questions because uh definitely we want to do like you know for the next iteration saving some time if if the answer is you know if the question is asked multiple times and it’s the same we can save some time but yeah regarding your question um yeah we we face that and everything is inside the knowledge.

22:22 Thank you. Uh hi, I have a question. How do you maintain the knowledge inside the agent and how do you make sure that these definitions are updated? Um so right now like Fab mentioned we only have like two dimensions and I don’t know under 20 markdown files. So it’s still quite easy to maintain and we worked with some data analysts and data engineers in this first iteration. But um if this goes to production, we need some kind of like uh semantic layer on a organizational level where analysts kind of build into their workflow to maintain this knowledge and then the the data agent would kind of just pull off that. But right now, yeah, it’s just MD files around 20 in a GitHub repo. But yeah, you need kind of organizational management around there if you want to uh have it or yeah, go into production. Thank you. Okay. So I want to know about more SQL correctness. So there could be multiple SQLs which can produce the same results but they can have different plans and cost behind the scene. Do you also consider that like one optimized SQL with partition can everything can give the right answer and then the second one with select star can also give you the same output probably but the cost behind the scene on Athena and other things will be too much. So do you also consider SQL plan or other things or no?

24:01 I mean um yeah I mean we consider it I mean we have this uh in the knowledge we kind of provide like the best way to write the query and then in the agent when we evaluate it we pass it in and this is really where this kind of like you make a change to the agent side and you observe the output comes in because um yeah you they’ll analyst will make a small change to to the knowledge and it will have certain impact and kind of understanding this but yeah in the second iteration we also want to focus more on Athena costs like we track them. So on the UI you can kind of see okay these are the costs that were produced by the token usage these are the costs that were produced by Athena um and with the select star you that’s I mean you could also have that on a two call level so whatever every query that gets run um before it gets run you maybe do like a regular expression for select star statements and already block them there cool thank you one last question. Thank you. Um hi, sorry it might be a vague question but um you said uh you’re building evaluations right and um my question was how do you build evaluation on something which hasn’t been asked yet. So you’re trying to like kind of cater um that anytime an analyst is asking a question it should be like valid right. So how do you foresee that like because uh to validate something you also need like that experience what is going to come. So what comes first like do you know these repetitive questions that you’re building evaluations on or like what’s the like you know it’s a vague question sorry but um it’s true but yeah we started with simple scenarios that you know this is why I explained like we are not made to replace analysts but these are things that happen every so often so these things you know you can give it to an AI so yeah we have the structure beforehand So we know the expected response. We have the golden SQL query so we can compare these things beforehand. Yeah, of course for unknown stuff that’s trickier. Uh but that’s the goal like at some point we’ll get into those things but right now it’s more about validating that the the agent can respond correctly things that we already know. But the things that we don’t know it’s maybe not not in this moment but yeah at some point it will become a goal. Sure.

26:33 Last question from Mark. Um, hi. Um, when you’re looking at your set of evaluation tests, [clears throat] how do you say, yeah, we’ve got enough tests. We’re covering everything that we need to. Um so on the one side you have like uh different kind of so on the knowledge you’ll have like certain type of queries or like some kind of metrics and analyses. So at idallo there’s some certain analysts kind of do the same deep dive over and over some certain yeah things they dive into and explain and you know so you can kind of check different metrics and just give it a few example but if you have if it works for three metrics you don’t need to do it for the 20 other metrics you have also done. So it’s more so like how how how in depth does it go? Is it just like a numeric answer or is it like analytical like if you want the agent to like provide analysis trends and things like that then that’s kind of like a new use case. Uh so you need to cover that and then also we we scale it with the domains we provide. So we talked about a semantic layer where the different analysts will maintain the semantic layer and they’re also supposed to uh maintain some kind of these evation scenarios. We’re not the one as a platform team writing these scenarios but uh so it scales with the size of the knowledge somewhat and also how deep it goes but yeah it’s it’s a good question.

28:11 That’s been [applause] Thank you. Thank you a lot.