From the Semantic Layer to the AI Analyst: Engineering Self-Serve BI That Actually Works

0:00 This evening we we have a um Hamza who is the founder and CEO of Leash and this evening is dedicated to you know some people that build and use light dash and do some AI stuff with that right so without further ado you want to come here you want to um cool hi everyone uh my name is Hamza this is what we’re going to be talking about today, but we’ll get into that in a second. Um, the topic, yeah, exactly, is meant to be going from the semantic layer to like an actual AI analyst. Hopefully, all these words in here, if you haven’t heard of them, we’ll talk about them in just a second. Um, don’t worry about that. But what we wanted to try and do is have a talk about what we’ve learned as a company at Light Dash and having actually built these AI analyst agents. And hopefully some of it is things that you can take away. And I’ll try to not make it a sales pitch, but I will make it a little bit of a sales pitch, but hopefully mostfully mostly it will be kind of useful things that you can take away to your own organization um and see if they work for you. And I’m also kind of curious to hear where people might disagree with some of the things that I talk about. But really, it’s just like I tried to sum up as many lessons as we had from having worked with lots of companies and trying to actually make AI work in a real world scenario, not like a nice pretty demo data set with like one model and one question. Um, and this is kind of some of the things that we came up with. Um, before we jump into it though, a quick intro about me. Um, like I mentioned, my name is Hamza. I’m one of the founders of Light Dash. Um, if you haven’t heard of us, Light Dash is a business intelligence platform. Um, we’re open source and the idea is that we work very closely with tools that are already in your stack. So, if you’ve used tools like DBT, we use pretty much any other tool like that. Light just plugs in natively on top of it. Um, actually just a show of hands. How many people here have heard of light dash before? Okay, that’s great. That’s we we we were here a few years ago. We did a talk and there were way less hands. So that’s that’s a good thing. Um, how many folks here use a tool like DBT every Okay, about 50%. Awesome. Um, how many people here would call them call themselves SQL proficient? SQL savvy.

2:17 Okay, I like it. Very nice. Okay, this is great. This talk is going to be amazing. Um, okay, cool. So, my background, by the way, before I built Light Dash, I was a software engineer. That’s what I did for a long time. Um, I got into the world of data and analytics because in the last company I was at, um, it was an working with insurance companies. I was trying to build machine learning models to predict the the likelihood of a building catching fire. It’s like what the likelihood this building would catch fire and if so, how bad would that fire be? Um, and then price that into the price of the insurance policy uh for the underwriters. Um, we used BI tools as the way to try and explain the outputs of these really complicated machine learning models to underwriters who are, if you’ve ever met an insurance underwriter, um, they’re not the most technical folk and BI tools are meant to be for folks who aren’t that technical. And if you don’t have a physics or maths degree, this is meant to be how you become data driven. Um, my experience using the BI tools that existed at the time was that that wasn’t really true. uh most tools kind of fell short of uh of kind of where we wanted them to be and this this hope for self-s serve kind of was always left lacking. Um but maybe AI is here to to save us for that and hope that’s what I want to talk to you a little bit about today. Um for anyone here who works in a data team uh something like this might look familiar to you. Does anyone have a data request channel in your Slack or Teams in your company somewhere where you can ask questions? I see some hands and heads, but just quick hand check. Any quick quick check of hands? Anyone have this in your company? Okay, not as many as I thought actually. I thought there’d be more. Um, for most data teams, this is actually the primary way. This is what self-s serve looks like in most companies is as an end user, I put a question in this Slack channel and an answer comes out. I don’t care how it happens in the back. This is self-s served. This this meets the criteria. Um the problem with it is that these these sort of questions add up pretty quickly.

4:15 Um and it ends up becoming the most timeconsuming part of a data team’s life and you end up dealing with these very small annoying problems that really should be able to be self-s served but they can’t. Um there’s an analogy that someone on the team used that I really like which is um the data team is kind of like a chef. Um, and ideally what you want is sometimes you want to go out to eat and you want a chef to cook for you and that’s kind of where you want to be really wowed by the most like the hardest problems that a data team should solve that a chef should be cooking. Um, but sometimes you you don’t want to ask a chef to like make you a ham and cheese sandwich. That seems like a bit of like a bad use of their time. Um, and so the idea is how do you help people kind of cook for themselves? Um, and in a real self-s served environment, hopefully we can teach people to make themselves a sandwich. And there will inevitably always be times where they need to come back and speak to the chef and they want to have like a proper meal. And there are things that will never take away from that. But there’s definitely a lot of room to improve, I think, from the status quo. Um, and so this is my my first statement of the evening. I think that these data request channels are basically a symptom of like a missing self-s serve experience. They’re kind of like a band-aid over a gunshot wound. Um, that most of the BI tools and the kind of self-s served experiences we have aren’t really fit for purpose because people have to resort to sending messages here rather than using probably the very expensive tools that we’ve bought and and installed and probably spent a lot of time on. Um, and yeah, so is there a way out of this?

5:46 Um, but to go forward, let’s go back a little bit first because I want to explain a couple things that I mentioned at the beginning um before we get into it. Does anyone here know has anyone here heard the term of a semantic layer or does it is it familiar? Okay, about like twothirds of folks. Um, so I’ll blast through this bit pretty quickly. Um, at Light Dash at least, we chose to build a BI tool that uses a semantic layer. The semantic layer basically is a place to, like it says here, map things that make sense in data to things that make sense in business. The most common example I use with folks is revenue. Um, everybody inside a company understands revenue as a word and probably what it means, but actually the number of people that might be able to calculate that is very different. Um, and so being able to translate the semantic definition into some sort of SQL code or something else is essentially what the semantic layer does. It is the trade-off point between the business and the technical users inside a company. Um, it’s not a new concept. It’s been around for a long time. Lots of companies have had different permutations of it. Um, but I think in this new AI world, it’s actually taking on a bit of a new form, at least the one that I’m the most excited about. Um, but there are some downsides to it. So, we’re going to kind of see a little bit about how it all maps out. Um, how semantic layers came about was basically from your warehouse, you would write some raw SQL. Um, some of that would turn into to some transform data or some final tables that you might want to use. We started calling that a semantic layer and then pumping it into all the downstream sort of governed systems that people might be ingesting.

7:17 So BI tools are one um notebooks all those things are on there. Actually the most common place isn’t on that list I realize which is probably a spreadsheet. Um the majority of semantic error uh metrics and outputs I think tend to go to spreadsheets. Um there’s nothing wrong with that. Spreadsheets are great. I have no no problem with them. I just left them off this slide. Um, but the idea is the semantic error was a good start, but it didn’t kind of it didn’t really like finish the whole picture for us. Um, here’s an example of what I mean. So, we had this idea that if we mapped every single semantic definition in a business, then everybody would be able to answer every question they need. You’re like, but there’s a definition for revenue and there’s a definition for cost of uh customer acquisition. Like, why do you need to talk to me as a data person? Um it turns out that actually good questions to ask is actually half of the problem. Um the semantic layer is one half of it which is it is the building blocks that you can use to answer the question. But if the question sucks then it doesn’t really matter. Um and as it turns out as data people one of our skills as analysts and analytics engineers is figuring out the right question to ask or helping someone figure out the question. What I mean by that is someone might say to you, “How much money did we make last week?” And normally as an analyst, you won’t just have like a number ready. You might ask a clarifying question and say like, “Do you mean revenue? Do you mean profit? Do you mean like the last seven calendar days? Do you mean Monday to Friday? Like what’s there’s that’s a very very simple definition, but the idea is that you see there’s a lot of complexity and often a very very simple statement that people ask.” Um, and this is kind of where semantic layers don’t help as much because they don’t, you know, they’re kind of static. They don’t really do anything. Um the other problem that I think comes out and I put a few of them here um is this third one which is a lot of the time even if something already exists and this is maybe a very very common problem you might have lots and lots of amazing dashboards and charts and reports that are like really beautifully built but if they’re hard to find or hard to read or hard to understand people just won’t use them.

9:21 Um, we did some analysis internally at Light Dash and basically found that if it takes more than four clicks for someone to find something, they’re not going to do it. And I’m talking about like an end user here. So, if they land on your landing page of whatever BI tool you’re using within four clicks, and that includes like the keystrokes, they can’t find the asset they’re looking for, they’re going to bounce and go straight to that Slack channel and just say like, “Hey, like so and so, who used to be on the data team two years ago built this report for me. Does anyone know where it is and have you been keeping it up to date?” that would be great. Please send it through to me. That’s kind of what happens. Um, and so there’s actually a lot of other problems that kind of fall around this self-s serve piece. And I think a semantic error is one part of it. It’s a big part of it. It’s not the only part. Okay, I’m just going to put this back up for a second. Um, cool. Okay. Well, that’s all the doom and gloom part of this this topic. Um, I’m now going to try and talk a little bit about what I think the answer might be for us. Um, I think a couple years ago when the first chat GBC was released, there was this amazing feeling in the world that this is it. Like we don’t need data teams anymore. We don’t need data analysts. I can just talk to my data all the time and it’s going to be correct. Um, it’s going to be brilliant. It’s going to be beautiful. Um, as it turns out, that wasn’t quite right. And I try to use the chef analogy here because I actually think what you really want from an AI tool is for it to take away a lot of what I call the busy work and leave folks with the leave data teams with the most complex problems. And I actually think what happened was people went straight with the hardest problems and tried to give those to all the LLMs and were like do this do the hardest thing for me um that normally takes a data analyst two or three weeks to do and that’s where we want to start rather than maybe the opposite way which is to say like what’s the thing that you do maybe 50 times a day that takes you a minute um that over time adds up and just becomes like a real grind on data Um, and essentially what we needed is an AI agent that acts a bit like an elite data analyst. Um, I have an example demo here. Maybe folks have seen demos like this before. Also, I apologize. All the demos today are recorded because I had a huge freakout this afternoon when Cloudflare went down. I don’t know if anyone else has been affected by this.

11:40 Um, and a load of the demos weren’t really that reliable. So, a lot of them are going to be videos. So, I apologize for that. But, if you want to see me do it live later, I will show it to you on my laptop. But I got a little bit of stage fright, so I had to record some of them. Um, but here’s an example maybe of like a demo um that I don’t think is that great, but maybe we’ve all seen before. I’m going to slow it down a bit so we can see it. So, we get shown Oh, no. We get shown a demo of a tool um that is something like this, which is I’ll pause it once it finishes the question so we can read it. Oh, no. Okay. So, um, ask a question like that and you might get like a really really good answer. Um, I’ve never met a stakeholder who’s asked me a question like that. Um, it would be great if they did. Um, but I think this kind of represents the problem. Um and then you know once I’ve asked my perfect question then yes I get the perfect answer with 100% success rate. Um the reality is this is not what anybody’s experience is with talking to any LLMs not even just in in you know most people don’t even use full sentences when we’re talking to these AI tools. So expecting folks to give this level of context I think is kind of insane. Um and it’s just not going to work. The way that I think about it is that there’s a few constituent parts to what you should be looking for for anyone here who’s thinking about implementing AI agents or LLMs into your data stack. And this bit the next bit I’m going to try I tried to oh I tried to diffuse it into two or three stories that I think are whoever you’re speaking to any vendor you should try and question them and say how do they handle these things? Um the first thing for me is that actually the best analyst agents should be coaches for your team. Um, so a really bad example is I ask for a conversion rate and I get given a number. Um, a better example might be a small amount of information that’s given to the user which says something like, “Hey, there’s actually a few different things that might mean conversion rate. Let me ask you some clarifying questions around that.” By the way, this stuff I know might seem really really simple, but it is insane how many tools that I’ve seen and played with and used that kind of miss these things which are like you’re trying to mimic a human interaction. So like make it like a human interaction. Don’t make it like a machine. Um so try and help coach people into getting better at data. For us at least, that’s the thing that I’ve seen has been the most helpful is having these agents try and act like coaches and try and help people get better so that every time they ask the question, they’re learning something.

14:22 The next time they come back, they ask a better question. Um and ideally you want that to be a virtuous cycle as part of the agents workflow. It’s not something that you have to do manually. Um it’s something that’s kind of built into how it works and and kind of how it Um the second one, this is actually a huge one. Um particularly if you are using a BI tool, this is very context specific. Um really when you’re using um when people ask questions in data teams, one of the things that I’ve seen is probably about 75% of the time the answer already exists in something that someone has created. So, it’s very rare that people ask a question that isn’t already covered by like an existing dashboard, an existing chart, an existing piece of content. Um, but finding it is the hard part and knowing that it’s the right thing that you need to look at is kind of the hard part. Um, this is actually where LLMs are probably better than humans. I had this conversation with someone earlier today where they said, even as the data team, sometimes we don’t know which table to look at, which chart to pull from. So, how could anyone else figure that out? And actually the answer is LM have far more context available to them. You can use use all kinds of things. You can use metadata to understand is it a dashboard that was built by the data team that’s someone that’s really trustworthy or is it built by me in which case maybe it’s less trustworthy and you don’t want to use it. Um there’s lots and lots of different things that can kind of improve the kind of what this looks like. Um so again here’s a terrible example. Um, and then maybe a slightly better example, which is an agent that actually before it decides to just jump in and run a query, its first protocol is to say like, “Let me go and look and see if there’s something that already exists.” Um, because I think BI slop is a very, very real thing. We’ve seen like folks developing thousands of new charts and dashboards because there’s just very little control on like how things get saved and how things don’t get saved.

16:10 Having a tool that has this sort of thing baked into how it functions, I think is super important, particularly if you’re a small team. Um, maybe even if you’re a big team, I think it affects Um, and then the third one, this one is subjective to me. I think it’s super important. Um, the reason why I think it’s most important is that if your agent isn’t actually contributing back to improving your underlying semantic layer, so that is the things, these building blocks that you’re using to answer these questions, um, it’s not really improving your system over time. Like if you think about an agent’s memory, its memory should be getting better as you ask it more questions. And a part of that is understanding how people ask questions or part of that is understanding the gaps and holes that you currently have in your data modeling. I’m yet to meet a team that has like the perfect data modeling set up on day one and they’re like there are no changes, everything is perfect. Uh the reality is data changes over time. Um no spec stays the same and having a system that has it baked into it that it can kind of make improvements, it can change things, whether it’s literally creating new metrics, whether it’s um updating descriptions, whatever it might be. So it’s helping your team get better over time. Those are all things that I think you want to look for when you’re thinking about building AI tooling, particularly for production. All this stuff is this point is probably the one that matters the most after the first iteration. You might get an agent that does something really good on Monday, but by the next Monday, all its information is out of date. And so kind of having this bit of virtuous cycle in there uh kind of really helps.

17:39 Um and yes, here’s an example. Um one of the most common things you’ll get is maybe the query is wrong, maybe the data is out of date. I don’t really know. I can’t really help you with that one. Um, a better example might be something that says, I’ve noticed that some of the descriptions are out of date here. I’m actually going to go ahead and update those for you. Now, if you think about it, it kind of becomes like a bit like um a bit like having like an AI agent assistant or an AI analyst that’s kind of just doing your busy work for you. These also for me fall into the category of things that people don’t love spending a lot of time on. You want to review this work, but you don’t want to do it yourself. So getting the agent or getting these agents to basically do a lot of the busy work and then your job as a data team is just to come in and verify and just edit things or review them um I think is far more exciting um than having to write everything from scratch. Um yeah. Okay. I know there was a fourth one. Um this one is probably I think most people are getting this and it’ be I haven’t seen a tool in a while that isn’t doing this sort of stuff but maybe you folks have. um is that it should explain its reasoning. The biggest problem people have with trusting answers from AI agents is that they don’t know where the answer came from. So they’re often like, “It gave me an answer, but it’s I don’t know if I can trust it. I don’t know where it came from.” It’s a bit like, you know, the computer just blurted out a bit of a blackbox answer. Um in reality, what we want it to do is show us our working. This is for two reasons, by the way. One is as an end user, yes, you want to be able to read it and say like, “Yeah, this seems like it’s probably the right answer.” more Um, as a data person, if you ever have to jump in and debug someone’s like chat session, you really want to be able to see what the agent was doing to understand if it’s picked the right data sets and working with the right metrics and explores that you were expecting.

19:27 So, I actually think this one is again more for the data team. Um, again, when we work with teams, the biggest question they have is they’re like, I need more context on what the end user was trying to do so that I can figure out how I can best help them. Um, and so just all these things I think maybe none of them are huge, but I think when you add them up, they really change the experience of what the agent interface is like versus just having like a black box. I think people think of it as like an oracle. They want to think of like AI agents, particularly when it comes to AI analyst agents as like oracles. And I think they should be more kind of like genies than oracles. They kind of like grant you wishes. Um, but they do things for you. It’s not just like a answer pops up out of nowhere and you just walk away with it and and believe it blindly. Um yeah. Um cool. Well, I wanted to share this because we don’t really hide it. Like like I mentioned, Light Dash is an open source company. Um now we’re getting into the sales pitch part of it, by the way. Um for a little bit at least. Um, but maybe this will be helpful if you are looking at other vendors and you want to figure out or even if you’re building your own actually, this would be useful to think about like what goes into the architecture of building one of these AI analysts under the hood. Um, for us, we kind of divide it into a couple of different categories. Um, the two most important ones are these kind of purple and yellow boxes here. The data plane and the memory plane. The data plane, I think, is the one that we’ve all understood. We’ve had that for 20 or 30, maybe 40 years. this notion of um some raw SQL database or warehouse with an abstraction layer called a semantic layer on top of it. I think what’s changing now or the new bit that we’re seeing a lot of is this thing called the memory plane. Um or at least that’s the the name we’ve given it.

21:11 Hopefully it it will catch on. Um and it encapsulates a couple of things alongside the semantic layer. It actually also takes what we call the agent memory and then here it’s called user profile but essentially information about the individual user. The really cool thing about these agents is they can be hyper specific to what a a particular kind of end user likes to receive. So for example, I’m on the exec team. I might actually only want highle KPIs and numbers. I never want to see a big table of information. I just want to see like highle KPIs and metrics and it knows to always return that kind of information. Um but if I was on the finance team, I might always want CSVs. I want everything with like three decimal places or never rounded. Um, and so the idea is that like it’s able to take that information, which isn’t important to answering the question correctly, but is very important to making the user trust the output. Um, and that’s kind of what the user profile and the agent memory makes up. Um, and for us that kind of comes from partly from the LLMs and partly that comes from that cycle of learning as you talk to it. The first time you talk to it should be the dumbest it ever is. Every conversation after that should get better and better. If it feels like it’s getting worse over time, you’ve got a problem. Um, but it should feel like it’s getting better and better over time. That’s like for me the hallmark of like a great AI agent system is that it’s getting better the more you use it rather than like it gets worse and worse and sloppier over time. That’s kind of the goal. Um, the other bits that we pull in to do this. Um, you can see over there that’s just some technical uh, but if you’re interested, we do some rag for existing content. So tries to like index all your existing content inside your project and figure out what might already be there. Um, and the other one is analyst instructions.

22:49 Unless instructions are kind of like things that you just want to tell it in the way that it returns answers. So your company you might say um always put things as bullet points, always include images, um tag people, whatever it might be. This is kind of like again not relevant to the correct data answer but very relevant to people wanting to use the tool. Um most people will know this as like prompt or like a prompt engineering something like that. This kind of where you fiddle and kind of tweak these agents to work for your team and and your workflow. Cool. So, does our AI analyst work? Um, yes it does. A sometimes, most of the time it does. Um, again, I’ve got some videos and I feel like a bit of a hypocrite after showing you the last video of the of one answering a perfect a perfect answer. But, I’ll show you one or two and hopefully try and highlight some things that we’ve tried to do and and see if any of these make sense for for kind of you and your teams. Um, ah, Cool. So, a typical question, someone asking something like, “How many orders did we get from I don’t remember the name, but there’s some partner that I think has something to do with ranches. This is a gardening data set that it’s asking questions about.” Um, and the kind of things that we want to see are it’s able to a use that context to pull something out and say like, “Okay, I think you meant this partner that’s in this case is called Redwood Ranch. Um, and you saw this field here called how it’s calculated. I’m going to take it back a second just to show you what you should be seeing.

24:32 Okay. So, the idea here is what I talked about that like make it explainable and make it easy to understand is you want to have something really simple even just like this that says like this is what I searched for. These are the fields I found. Here’s what I’m going to try and build. In this case, it’s going to try and build a table. Um, and I think once it builds the table, it’s going to try and I think we asked it to build a dashboard. there we go. Okay. And then um here’s a second one. So, in this case, I’ve asked a pretty simple question. How many visitors did we get to our marketing pages so far this quarter? You might argue actually that I put in far too much detail into this question and I should have just said how many unique visitors were there and then just let it figure out the rest of it. Um, but the idea is that you want to see the reason I wanted to show you guys this one is because here’s an example of something where in this case it figured out that actually the metric that we were using was incorrectly defined inside our semantic layer. So here it’s answered the question. Then it’s gone and said like, hey, I’ve actually gone ahead and updated the description of this particular metric that we had. Although it represented the right thing, it was poorly worded. So here I’m going to go through and provide this update. If you like the update, I’m going to accept it.

25:52 And then you can kind of carry on with your life. You’re still getting the answer, but it’s kind of improved your semantic layer for you on the fly. Um, and ideally this is the kind of workflow you want to have where you’re working with the agent and making it sound like you’re kind of like in this dance with it, but you kind of are where you kind of want to keep going back and forth. Um, trying to get some feedback from it, trying for it to suggest things for to you to improve. Um, and then kind of go from there. Um, and then the last demo, I’m actually not sure what this one is. Oh, okay. Well, I actually didn’t know what was on this this this last one, but it turns out it works in different languages. Um, if you’re interested in that, we can There you go. I can’t take credit for that one. I think that’s mostly Open AI who did the the heavy lifting on that one. Um, to to bring this back though a little bit, I want to talk about what this means for folks in the room. Does anyone here, by the way, call themselves an analytics engineer? Is that a term that folks might go by? Okay, there’s at least a few. Great. Um, okay. Well, I’m trying to coin this new term. Um, so analytics engineers, for those who don’t know, was a coined term by DBT back in 2018 or 19, something like that. Um, and it was meant to encapsulate exactly what we’re talking about here. Folks who are very, very data technical, so they’re really SQL savvy, Python dangerous. They know how to like work their way around the data schema, but they also have incredibly in-depth knowledge of the business as well. So they’re not just able to write code, they’re also able to understand deeply what the business is trying to do. And it’s kind of that marriage of the technical and the business information that makes that role so powerful. And for a lot of the people sitting in this room, that might describe what you do as well. Um, and my kind of hot take is that I think that role is changing in the next sort of well, I think it’s already changed. Um and the new term I think is going to be called the analytics intelligence engineer where the role of for us as data people is going to be maintaining these context stacks these memory planes for AI agents to be able to use and ingest. Um there will be some other tasks to deal with. So collecting feedback from stakeholders feeding it back into the AI analysts occasionally and then like I mentioned dealing with problems that only humans should. There are some things at the moment that really a human analyst should only ever do and you should never really offload to an AI agent because it just will not get it. Um but for the most part I think a lot of the job is going to move to building these like context rich semantic layers for these agents to work from. Um I think that’s really exciting because it just gives so much leverage to teams. Um we spoke to a data team earlier today that’s a data team of three that’s serving about a 100 people per data person. So about team of 300 people, three data, three three people and they’re not working 18 hours a day.

28:36 And a lot of it is because they’re like really cleverly leveraging um AI agents in their data questions channel. It’s not like everything goes to it though. They’ve kind of figured out how to triage what questions go to humans, what questions go to agents, the rest of it. And I think it’s super impressive. And a lot of the job of the actual data team now because they have lots of free time is focused on how do we make the system better? How do we focus on like making the actual agentic system better rather than just having to fight fires and deal with every single request that comes across our table and every you know exact report that needs to be built on the fly on the day. Um and then the final point for me is yeah on the semantic layer on the semantic layers the way I think about them is that they’re basically just rules of the road for your self-driving agents. I think in the last 10 or 15 years we had different reasons for why we had semantic layers. uh a few years ago was because people wanted to use lots of different tools. Before that, it was just to have a layer of governance. Um and I actually think now they’re taking on a new role. Um which is that they’re essentially the guard rails for these AI agents that we want to build. Um and they kind of have to evolve in that respect to be able to work for this new use case. some of the things that you that I hopefully showed you really briefly here. Um, things like having them be self-improving, having them very very closely tied to the user interfaces, whether it’s an AI chat interface, whether it’s a traditional BI tool, whatever it may be. Um, I think these semantic layers are going to stick around for a lot longer. I think there was a time a couple years ago where everyone was like, “That’s it. They’re over. We’re done with semantic layers.” Um, but I think they’re back. Um, and I think they’re here to stay. Um, which is why personally I think it’s worth investing in the right one. Um, there’s a lot out there. There’s loads of different permutations. I don’t think there’s a right one, but there’s definitely a right one for your company.

30:21 Um, so definitely have a yeah, have a look at them and kind of think long and hard before you decide to commit to one. Um, cool. That’s it folks. That’s everything I had for you. Thank you all for listening. I think we have some time for questions. Oh, okay. Hi. Um, great talk, Barry. Um, talking about semantic layers, have you got any thoughts on the new open semantic interchange that’s coming out? Yeah. Um, I think it’s probably the closest we have gotten so far to like a truly homogeneous semantic layer. Um, I think it’s worth remembering that the OSI at the moment is like a it’s a standard for the definition. It is not an engine for computation. So, I think there’s still going to be a little bit of I could see everybody having their own flavor of how they compute those metrics. Um, but I found out very recently this talk’s being recorded, right? Okay, great. Well, I’ll say it anyway. Um, so I I think that there is another large warehouse provider that might be joining the OSI. Um, it was launched by Snowflake for anyone who doesn’t know. Um, but there’s a rumor going around that another large warehouse provider might also join, which I think would be massive to actually pushing it forward. Um, the big break would be if it can be vendor agnostic. I think it’s the closest we’re going to get to one. I’m excited about Did they already announce it?

31:59 Oh, I was I was thinking of a different Star Starburst have said it. Okay, there’s another one that isn’t Starburst that I was thinking of. Okay, that’s great. Well, it seems like the ball’s already rolling on it. And which layer are you using at the moment? Which layer are we using? Um, so Light Dash uses an open source semantic layer. So, it lives inside DBT. Um, it is not DBT’s cloud semantic layer for anyone who’s familiar with it. Um, it’s basically our own flavor that we created about four years ago. Um, but it’s open source. It lives inside dbt. If you run core cloud, you can use it. It has a Python client, a rest API, all the rest of it. Um, but we’re very open and willing to partner with other semantic layers. So, we are signing up for OSI. Um, some folks might have seen that DBT recently made a change to their license to open source their previous proprietary semantic layer, metric flow. Um, so we’re planning on integrating with those. Our main thing is it has to be open source. I’m a little bit more bearish on paid for semantic layers. I’m less of a fan, but if it’s open source, we’ll we’ll probably connect with it. Hello. Uh, one of the problems with these language models is, you know, because they’re trained on on language, you know, sequence to sequence rather than ironically, you know, interaction like like we’re doing now. Um, and I noticed one of your transcripts the model asks a question back of the user, which they tend not to do. I just wondering how you did that. Is that through the system prompt or how do they know when to ask a question?

33:39 Yeah, it’s a really good question. Um, there is some system prompting going on there, but actually we look a lot at confidence scores. So with LLMs, basically there’s this thing that happens which is that they’re often very confidently wrong. I’m sure everyone here has experienced that at some point where it’s like this is the answer, you’re like nope. Um, and so what we try to do at light dash is make it so that if it’s not really that sure of the answer, it’s much better to just go back and ask a question. And so some of it is system prompting. So if we say that this question, if a question seems quite vague, here are some good things that you can go and clarify on. And a lot of it is actually not that hard if you think about it. It’s normally things around like time frames, date, timestamps, that sort of stuff that you want to figure out or like if there’s two data sets that look quite similar, what can I ask you to help you figure out which of the two you want to use? Um, so it’s a bit of a mixture, but there’s definitely some system prompting. Um, and then we built a sort of confidence scoring mechanism to try and give it some level of like here’s what I think the answer is. Um, yeah. Can I also ask? Great question by the way because I also had that one. So now I can ask the other one that I had. Um so I wonder when the user asks a question, what was the conversion rate this month or last month? Um when I get an answer in return, this is one number for sure I will have the next question. Ah is it more or less than the previous month? Is it more or less than the same month last year? And I will keep asking these questions one after another. And I wonder if it makes sense for the model to immediately when I’m just asking what is the conversion this month to immediately return to me a line chart with the like the conversions for the whole previous year with the deltas and all of that.

35:32 Yeah, it’s a really really good question. Um so in [clears throat] the example that I threw up which was just the slack version that was just like some messages but you are right I think like if you ask a question so I can talk about light dashes at least um what we try to do is two things one is return you the answer um and then try and give you like some next steps so we might say like hey here’s the number let me know if you want me to like segment it by different cohorts or if you want to see this broken down by region or different time frame and then we’ll use the kind of data context to try and think of a sensible next question. One of the things we found though is that if you give people everything, if you say like here’s the conversion rate, and you’re like great, here’s the number, here’s for it compared to last year, here’s compared to like the year before, it actually almost becomes too much. And so I think there’s like a bit of a balance. And I don’t I can’t say we always get it right, but I think it’s like a bit of a moving target because some folks are like, I just want the number and then I’m I’m done. Other folks are like, I want the number and then I want to know like what I might I want you to predict what I might want to do next and then like make it a button and I just click it and then you do the next thing. So I think it’s like how do you not overload people with too much information but definitely give them a step forward. Yeah, definitely a big advocate for Yeah, for sure. Um, so internally we benchmark across three providers. So like Anthropic, OpenAI, and Gemini. Um, funnily enough, like they’re all within quite like close realms to each other when it comes to this. I think Anthropic probably wins out like ever so slightly.

37:12 Um, but actually I’d say in the last like two or three weeks even, the space moves so quickly with the foundational models. Um, I’ve heard amazing things about the new Gemini product. Um, so we’re quite excited about that. Um, apparently it’s meant to blow a lot of anthropic models out of the water. Um, but I think the most important thing actually, we didn’t talk about this, uh, is that you have some sort of objective way to measure the performance of the models in your context. So, anyone here heard the concept evals? Is that like ring a Okay, cool. A couple people. Evals are basically like tests for LLMs. And so, it’s like if I ask the same question, you know, 10 times, do I get the same answer 10 times? um in light dash and in any a sort of decent AI agent platform you want to have eval so that when you consider changing models um you want to know if it’s getting better or worse at answering your questions this doesn’t just work for the foundational models by the way this is also for things like your semantic layer so you might say like I’ve made some changes in my semantic layer I want to know if it’s getting better or worse at answering questions so you run your evals again and it’s like adding a bit of determinism to an inherently non-deterministic sort of product like an LLM um so that’s super important Um, in terms of the the sort of context window, it is relatively tight. Um, but yeah, there’s nothing I wouldn’t say there’s anything too special there that we’re that we’re doing if you have the access. Yeah, not many people should have that access.

38:46 Hey, thanks for your talk. Uh I have a question like how do you deal with role based access to to data from the users right especially if it’s PII data and how do you constrain which users can get responses from which data and so forth. So in the context of the like an agent like an LLM you mean or just exactly? Yeah. Yeah. Um that’s a it’s a good question. Again I’ll answer it from the lens of light dash. Um the way that we built our AI agents is that anytime you ask a question of an agent, the agent basically impersonates you as a user. So if you don’t have the permissions to query the data set that you’re asking for, it’s not going to return you the answer. So the permission, it’s kind of like this. The permission model should not be on the agent. The permission model should be on the user. Um, and then what happens is at least in light dash I might ask a question behind the scenes everything is compiling down to a series of queries that it’s going to run against your your kind of data source. And what it’s going to do is it’s going to take Hamza’s credentials and say like okay well actually Hamza doesn’t have access to that like you know employee salary table that he’s asking for all the information on. So it’s just going to say like hey you don’t have access to that that data set or it doesn’t exist so I can’t share it with you. Um that for me has been we found the most straightforward way because it means that if you never had access to that data to begin with then the agent doesn’t grant you any new permissions.

40:11 It kind of mimics what the user already had. Does that make sense? Yeah, it makes sense. Uh and what if the user has for example some write Yeah. Yeah. Can you for example ask some ludicrous query to like run write a bunch of data to the database and if it has permissions it would execute. Yeah. So that’s a good question. So in light dash nobody has those kind of right permissions to basically try and you’re basically dealing with like an age-old problem of SQL injection but like happening in like prompt injection. Now, um I think the same protect the protection is the same, which is if you’re protecting for SQL injection, at least in our case, then you’re also protected against prompt injection because the user shouldn’t be able to do something that they’re not allowed. Generally, in a BI tool, even the most advanced permission set of a user won’t have right permissions. You might have those directly on the warehouse, but definitely not through the through the BI tool. We haven’t had that yet. We have we have quite a lot. You you’ve never written to the database through light dash, have you? No. Yeah. Okay. There you go. So, yeah. So, I was asking Sasha because they they’ve been using it. Um, you should you should never be allowed to basically I think for write permissions is something that shouldn’t exist in a BI tool anyway. Um, you might have that permission as a database admin, but that permission shouldn’t flow through to a end tool like a BI tool.

41:46 last two questions. Um yeah, thank you for a great presentation. Um so my question is more like on your kind of business I would say. So uh you probably heard about amplitude and also their queries and so on. They I think presented it somewhere in June in beta or something. So do you plan to have some more let’s say ad hoc things regarding the finetuning or something for your clients because your model is literally open source and then you have a features for your clients basically having more flexibility and what is the most popular integrations you have like request besides the multi-tenency or something like that. So on the finetuning do you mean like fine-tuning of the product or finetuning of like the agents and these LLMs for for for a client? Yeah. The model Um so the finetuning at the moment there the kinds of things that we let people do are kind of if we go back to the this um the bits that we let people inject kind of like their flavor into um it’s pretty much anything that’s in the purple and yellow you’re in control of.

43:04 So you can control the memory, how it responds to things, all that sort of stuff. You can kind of fine-tune quite a lot. Um, and we’ve seen some people take it to like um, for example, we have a company that um, they actually don’t want their agent to ever return a query. They wanted to only return existing content because they’re like, “Look, we’ve got 3,000 reports or something like we don’t need to ever write a new query. Um, so we want the agent to only ever return existing content.” um if you think about it, they’ve just recreated search, but like that’s uh that’s what they wanted. And so you’re able to kind of fine-tune that sort of thing into the behavior of the model. So there’s a fair amount of flexibility, but in terms of the actual foundational model itself, we haven’t seen the need to. We did a small PC a few months ago with building our own small language model because as it turns out for a data analytics, your LLM doesn’t need to know who like the fifth queen of England was. you can just have it scope down to the information that’s relevant today analytics. Um, but the improvement was very marginal. There was some performance gain, but it was so minor. I think betting against the foundational model providers just getting exponentially cheaper and faster seems seems unnecessary. That’s true. you won’t deploy frontier models, but even I’d say our biggest enterprises usually have agreements with most of the major model providers. So, as long as they’re using like a Google or Anthropic or OpenAI, um we have ways for them to inest inject their own model providers into it. Um yeah.

44:44 Oh, integration. Um that’s a good question. I think probably like we have two sets of uh requests. One is data sources and I probably I was talking to someone earlier um and I said the most popular data source at the moment for us is duct DB. So people are like please integrate light-d which we’re adding in the next sort of week or so. Um which is quite exciting. Um on the other side I think people are always interested like where data can push out to. So I’m a big advocate for like people not logging into a BI tool because lots of people that’s not where they want to do their work. Like we said they’re in spreadsheets, they’re in notion, they’re in Slack, they’re in Teams. Um, so we integrate with like as many of those places as we can. Um, and so I think I’m trying to think what the next pro maybe like Miro or something I think would be quite a popular one. Anywhere where people are like I want to put data. Actually, no, I know what it is. It’s a PowerPoint. PowerPoint and Google Slides where people want to have a live connection to a chart or a dashboard. That’s by far the most popular quarterly report. There you go. Hey, cool stuff. Um, my question is on the data protection side of things. So on one side you are pushing things to LLM let’s say um so you have to make sure via API uh you don’t leak company secrets for example you’re asking different questions on the other side you should also be improving your own product uh in the middle layer so how do you handle protection especially um you mean as a company that’s trying to improve our own product yes and also user data like that I’m trying to ask about my company I don’t want it to go to the LM models for its own um Yeah. So at least for the model providers that we work with, none of the data that we use is used for training, just the agreements that we have with the model providers. So the data is kind of secure. But to answer the other question, most of the time when we look at product improvements, you’d be surprised how little of it is actually because of the data like the query results itself. It is often all the supporting information around it, like the metadata, the schema, things that are typically less sensitive that we might have access to anyway. Um, and in the light dash system at least, you can choose for the agents and light dash to not have any access to the data, in which case they only operate using the schema and that’s kind of and metadata.

46:55 That’s all they use. If you want to, you can opt into sharing the data. I say not sharing the data for the LLM to use the data. In that case, like the results get a bit better because it’s able to look at the data, do things like forecasting, summarize, all that sort of stuff for you. Um but as a company itself we or we have like pretty much never used a customer’s data itself to improve the product. We use things like interactions. So we’ll look at like how many questions did you ask? How many threads did you have to go through before you got an answer you liked? That sort of stuff. But we don’t actually need to look at the data to understand what’s what’s normally going wrong. And then we have design partners. They help.