Operational AI in Business Intelligence

0:00 Welcome back. Let’s have our second talk. We have uh Sasha and Paula from uh Just Watch. I forgot the title of the uh operational AI in business So I leave the stage to you. Thank you very much. Few seconds more late ones all there. Good. Let’s start. So um that works. So me and Paula are going to talk about operational AI in business intelligence. So before we start in I would like to thank you all for coming. So we are very happy to see you here. I want to give a shout out to our office manager. She’s somewhere around for helping out organizing all the stuff. Uh it’s very great uh use of the this arena. So I mean mostly we use it for movie streaming and video games sometimes you know like internal team events but this is like really very nice opportunity to use it right so uh few words about just watch and what it is so just watch is as you can read it’s a streaming guide so it helps people to find the content they want to watch and they maybe they don’t know yet they want to watch so and they want to see why they want to watch it. It’s international uh application that you available in more than 140 countries where you can see uh all the available offers. Let’s say you want to see the movie but you don’t know where it’s available on your subscriptions you can find there. So we’re not really a streaming service though that’s about to change. If you stay tuned we are entering the game over streaming. So it’s not yet available in European Union but eventually going to be next year.

1:40 Stay tuned. And for the B2B side of the business, we are helping um streaming services and movie studios to promote their content uh running and strategizing uh digital media marketing campaigns. Right. So, what’s going to talk about today? Well, I could cover the agenda the usual way, but uh given the nature of the topic we are going to talk about today, let’s try not to anger the Skynet. And I want to change the style of the presentation to be a bit more robotic as a terminal output look. That’s going to be our style for today. I mean it’s also lazy way because uh basically I also used the basically to make all those all those presentations. This is this is text. So it’s not a picture it’s a text and you will see the text because again actually unreal like Rome by okay intro. I will also then explain how we get here and why we actually doing this. There will be a lot of demos today and at the end actually going to tell the secret source why and how this actually working what you need to do actually for this to work. So stay tuned till the First of all our journey and this actually going to be nicely connecting to to lightash um but it’s like recap for the TV show but it’s not going to be from the season one. It’s going to be from season five. I’m not going to talk about how we build all the pipelines and and and all the qualities and stuff but like year ago we had a presentation in this very arena for also for building data meetup and I was presenting about our um progress and success of the lie dash integration. So long story short, we solved our use case for self-service at scale and next step after that was basically okay cool what now and then the question was let’s go to next level and basically AI was already like ramping across the across the world and it was logical to say okay now let’s use AI we want not to just get the semantics and self-s service when you click around you want to ask a questions so we were exploring the um area and we were super happy to realize slide dash was also building the stuff And we became one of the first adopters of this particular application that Hamza where is he talked about a lot.

3:52 All right, that was kind of mid last year. Few months ago, just few months ago in August, our company got a cloud license from Anthropic. Till that moment, I was basically exploring and kind of watching the area what’s happening with the business intelligence and data in the there was not really much breakthrough like some data chats but we got covers with light dash and there was some basic analytics on the specialists mostly but not really much happening with the how you operate stuff. uh so once we had the license for the for the globe I decided okay let’s basically explore I went guns blazing and basically experimenting and brainstorming a lot and we tried to find out all the opportunities we had how we can optimize our processes uh our life of our stakeholders with the AI help so what we’ve done we built a lot of brainstorming sessions with the team everyone in the team even if it’s not super technical had to come up with some ideas and basically develop some particular application some particular process some improvement doesn’t matter what exactly but also still with the A. So today we’re going to show you a lot of examples of those of those products. Um most of them are already almost final in production and by the end of the year we’re going to definitely have them as our tool kit basis shortly about the format of this presentation. It’s going to be a bit of talking at the beginning at the end but mostly going to be presentation. So the focus is going to be all hands hands- on and like super u demos videos and and screenshots. So you’re going to be not theory at all.

5:29 But important catch up at the end there is a really key source how this method works. So stay still stay tuned till the All right, before we go to the presentations about the different tools, this is basically main areas of all the teams or the BI teams that actually doing right. So you data modeling in our case we use a lot of DBT. We have like more than 1500 models in 25 plus domains. Uh you handle all the reporting based on the data modeling of course uh reporting could be ad hoc, could be dashboarding, could be self-service. Well, we have already self-service sorted out. Then on top of that we also act as a consultancy uh for the company because we as BI involved in all the projects that’s happening the company. So we basically know if not the answer but at least the right person to answer. All right and last but not least all that’s fun it’s pretty exciting but there are also a lot of housekeeping activities you still need to do. So those are four main areas and I guess a lot of you can relate to those and for all of those we’re going to have some demos. We’re going to have several projects covering how we actually improving with our team uh all those particular activities. I’ll be mostly covering housekeeping and data modeling. So basically development operations Paulo later going to cover all the stakeholder operations how we basically expose it to our users.

6:54 Right. Let’s start with the less exciting one, housekeeping. Uh before we do that, uh just a few words. The first thing we actually had to develop is the way to can connect to our users and to ourselves. Uh JustWatch is a Slack first company. All our communications, all our decisions because we’re not really like completely in office. It’s still hybrid. So the Slack is very important for all our communications. So all the decisions have to be recorded there. You can find it. Um sometimes it’s like not super efficient, but at least you can find all the documentation in there. what is references to all the decisions, all the threats. That’s important that we develop our own u slack bot app that we’re using to expose it to our users. Dimma sitting here was the one to do all of that. That’s how he named it. Thank you. Right. So for housekeeping there are three activities you can think of. The monitoring let’s say have an alert, right? something you need to do. You need to go and investigate that maintenance. Let’s say you do all of the models. Sometimes you need to actually care about retiring them and optimizations. If you go crazy for the consumption of Snowflake, you need to care about that because you’re going to just burn your money. So those like three subcategories of housekeeping. And for all of those, we’re going to show you some demos right now. So really no theory slides anymore. Demos are coming.

8:20 First one. So this one developed by Rosella. She’s actually watching this right now from from the zoom as far as I know. She’s not in the office. She sits in Italy. So we using DBT as I said and I just watch. We are super good at building stuff and shipping stuff because there is always momentum. There is always importance. People always asking the things so we not that great in retiring because normally you don’t really have like an event to kill some particular analytics. Sometimes it just like fades you know like there is no usage really and over years you can accumulate a lot of models that you don’t really actually need but you still process them they still consuming some space some some processing power and that basically cost you money sometimes you actually drop the model but you don’t drop the objects from database which is also the case so what we built here this is like you you can see it November 13 it’s pretty fresh it was our first launch of the DPT blo scanner which uh identified a a lot of problems as you can see and you can see how many we can actually uh kill the orphan table here is like you can see it’s the objects you have in the base but you have on db the thing here this one is not only connecting to your repository it’s also connecting to the statistics of the processing of the models and the real queries that actually uh our users hit again them so we have some models that collect this all together and the agent is actually analyzing this and posting this back to us and as in the slack it’s super actionable so we can right away just go ahead And basically, yeah, you know, Paula kill this stuff and she’s reporting. Great. Now I don’t have 73 tables anymore or from once with just 16. Nice.

9:58 Next one. Similar fashion. Snowflake analysis. So obviously you don’t you don’t want to really discover like months later that you’re actually blowing up your snowflakes. Snowflake bill. We use snowflake as you already have guessed. Um here every week right now we’re running this fully automated uh analysis of all the different operations we run against the snowflake. It tells you how much is compared to the previous week. What’s the part traction like if there are any models that actually blowing up your bill right now for whatever reason if there are any anomalies and stuff and you can also again immediately throw it into action. As you can see here I’m pointing to dimma and bessa that there are some models that really is a match. And here you can see that I myself was the one responsible for a lot of All right, this one is a video. Um, let’s say so all those things were kind of proactive. Let’s say reactive one. It’s pretty typical you have a lot of alerts, right? Data alerts, whatever happens like you have multiple channels, whatever things. What you usually do? You got an alert. This is an example, one of many. we have like really hundreds different types. So what do you do? You jump into that. You do some explorative queries. Maybe by your gut feeling you tell it okay it’s not really like important. Let’s just let it be and going to be like result itself. But sometimes it’s a real problem and you need to spend some time to discover this and then basically escalate to the right person. But quite often those queries they’re pretty And in this particular message itself it’s already a lot of information. You you already know which particular model failed. You already know why. So why not just basically tell go ahead and explore what’s actually happening and report back to us. And as you can see it gets very creative in terms of how it visualizes. There is no real clear instructions that you actually have the country icons and stuff but you have instructions about okay this is a model go and check out what’s going on report back anomalies and you see here you can actually go interate with this. So the agent itself built by Mitri she’s here and the logic of the after discovery built by Maya somewhere she’s hiding up there.

12:10 All right and as also can see you can iterate you can engage in conversations directly in Slack and on each alert you can basically go investigate you don’t need to spend your time yourself. All right so that was about uh housekeeping. Next one is about building stuff. So what’s your usually usual routine about building the models? It’s probably something like this. I know that’s for me about 80% of the time you spend like writing code. Hopefully good one, right? So but by this moment you already know the specifications, you already know the structure of the the all the sources. Then you obviously testing your data, right? And in the end someone’s alo helping you code review or your code review. That’s how it used to be for me but for years till a few weeks now myself it’s not like happening yet for the whole team but I will definitely get them there. I completely change the way I work. So 60% of the time I’m I don’t really writing code myself though I I love it. That’s why 10% of the time I still query it for fun. And 30% of the time and this is crucial you really have to review what it’s doing for you and you have to correct it but ideally like imprompting. This is crucial. So though I spend the same amount of time, the nature of the time spent is different, but the output is in times more. Why? Well, because some of the things you can actually run in parallel. You can run multiple tasks at the same time and it’s only hitting your mental capacity. So like if you’re able to catch up, you can work on multiple projects. You have to prepare in advance all the specifications, but basically you can collect few days uh some insights how you prepare for this. Then you have like one evening session for development. You can ship in one evening something you could do before like in and that’s not actually theory that actually happened over the last few weeks or almost a month. I had a two very big projects I had to run. One was completely from scratch. I’m going to show you how I kicked it off and one was existing codebase. I had to really develop and um run a parallel migration for the business critical data set. So this actually helped me. I didn’t write a line of code but I did I did prompt a Okay. So results efficiency like seriously speaking like I don’t spend in the end the same amount of time for development. I spend less but my efficiency is still in times more than before because basically if you like compare one one hour to 1 hour is 10 times more they can develop.

14:38 Right? So that old stuff was about development which is just half of the things you do at business intelligence. The other half is more tricky sometimes because it involves people. It’s stakeholder management and how you actually deal with them. And for that there is Paula please. Okay. Hi. Um I’m going to cover the reporting part but first uh we need to talk about um consulting at scale as well. Uh we’ve built an agent actually uh Dimmitri buil it that connects to our entire technical infrastructure. uh is integrated with our git code repositories airflow and a snowflake and it contains all our technical documentation because it’s basically written um g it also interacts with a slack because as you’ve noticed already a slack is our main tool and and yeah this is how it works so basically here um da is asking a question about permanent audience and custom audience something that is really critical for us in in the company and GL It’s going to get and looking for the answer. And yeah, pretty good there. Yeah. Um but also you can do a bit more technical stuff. So in this case, um here you can also create a snowflake.

16:09 Yeah, I think it talks by itself. Um, so yeah, it goes to to airflow snowflake git again and retrieves the answer um the definition and then quering the What is the cool thing about this is um natural language. So that means that we can at some point maybe not the technical part but the conversational part we can open this to our internal users. So instead of um having a personal like can you please define me what is the permanent audience what is the custom audience because we have like 10 different definitions of audience now cloud can resolve it for us reading our g code now uh semantic layer I mean everything that I’m going to talk I mean hamsa covered it already um the semantic layer is crucial to our infrastructure uh we have all our data centralized in snowflake everything starts there we data coming from different sources but all it up ends up um in Snowflake. This data flows into the semantic layer and this is where the magic happens for us. Um why the sematic ledger is critical. We have all our KPI definition there but we not only have the lateral language definition we also have the the technical ones. So for example um I’m supporting the marketing team. I don’t know if you know what a video view is but some for example for Tik Tok is two seconds view and for uh YouTube is 30 seconds view. So if you go to our semantic layer you will see video view you will see just the definition that I tell you and also the SQL piece of query um that it will define it.

17:48 Also something really important for us is the role of security. Um we have users that can access some client data others that they cannot. So having everything control in the semantic layer there it’s crucial for us. All of this is maintained only by the VA team. Um for us is a single um source of truth and we contain all the business logic So the outcome uh we are using light dash uh for reporting uh all the dashboarding is happening there. This is reading the semantic layer and also the self- service. I have to say that all the self service is already covered. We’ve been using light for a couple of years and the addux requests are not that many are just a specific. So maybe some person is like I still don’t understand how this metric is calculated but it’s so little now and thanks to all of this then we are ready for AI. So um one of the things that we have building is um dashboard creation. We built an agent. The goal is like we wanted to create dashboards with just user instructions, not manual work. So how we build this? We download actual dashboards from our users, internal users to learn how they create dashboards, how they analyze the data, how they use it. We also download millions of charts and also some of the external dashboards that we have for the clients. Um then we start some patterns and train it. Also we had to give some of our best practices. So I imagine that you are prompting and then you ask for the time series and then you see a bar chart for a time series. You don’t want that. So we also explain that the same way that for example as as I told you we’re working with uh marketing YouTube should be red meta should be blue. Tik Tok should be black and stuff like that.

19:44 So the process how it works as I said someone says I need a dashboard. So they will give us the basic information that they want. How does it go in there? This reads our chart guide, the one that we built prompt. Then the dashboard guide, the same one we did prompting. And then it goes to the semantic layer and suit to some of domain files. Uh in those domain files we are explaining uh what the business is, what is just watch media, what’s just the streaming ad and stuff like that. And then after that the lightest generator creates the YML files because um light um has correct me if I’m wrong please all the charts and dashboards in YAML and then we automated validated and ensure that all the metric assist and after that it upload everything by itself to light generates the URL for us giving us the dashboard and then pretty easy so I’m going to show you conversation that I have with uh cloud the other day. It’s like I was working with um with a colleague. It’s like okay let me uh create a dashboard for you. It’s like okay I want to know you know this is morning uh Monday’s morning. I want to know how the internal um website that we have uh how many page views do we have who is visiting which section uh who is our most power user and stuff like that. And I want to see it like from the last week.

21:14 this is this is like the brief conversation that I have with Claude. So you can see it’s like yeah create a morning review dashboard blah blah blah. This is the requirements. I want two tabs the page views and the errors. I want the seven completed days filter. And something that I want is like okay apply to all the charts at the big numbers whatever. And then the the um agent replied to me okay I’m going to use light as generator. Is that okay? Cool. Do you want the charts to be stored within the dashboard, outside the dashboard? I said, I want them inside. I don’t want you too too many charts around. It’s like, and what do you want to save it? I said, PA, my personal space. And there it goes. It created all the charts for us. And then it gives me like, okay, this is how I I I filter the data. It’s complete. It’s a lie dash. And there’s your link. And here’s the dashboard. It’s not the perfect dashboard, but it took us five minutes. And it was from scratch. Also you need some touchins and but basically it’s like pretty fast thing that maybe my developer colleague would take 30 40 minutes maybe asking me pa where is this metric where’s the page views what is this so pretty and this is another example is also 95% done with AI that Sasha the project that Sasha was talking about this is the dashboard um for it again most of it I mean not the colors and some stuff but most of it was uh done with um light dash generator.

22:43 So here’s where we are today. Currently we have the light destinator working specifically for light dash and claude but we wanted to make it for every agent not only and we also wanted to make it um ascendable to hex but we we realized that this can be extendable to any uh tool that uses dashboard as code and again this is an MVP I’m pretty sure we’re going to change this entirely by the end of the year. So other things that every data team has to resolve a lot is the add of request. this is where I present you Noby. Noi is the name that we gave to our live dashboard. It’s so um so cute. It’s it’s integrated in Slack. Uh because again as I said Slack is our main tool is where everything is. Um, and it’s, as Ham already explained, we’ve been working with this for already a year, and it’s pretty simple. You ask in Slack, you tag Novi, and then it will give you an answer. You can interact in the threads, um, and it will it it works amazingly. Um, it’s conversational, no KPS understanding, no SQL, no technical knowledge. So imagine that you have a new colleague in the company and it he doesn’t understand what a page view, what a clickth through. Um it he doesn’t understand how to start dashboarding. So this is like a really nice entry point for it. So this is an example. For example, here I asked how many uh campaigns we’ve run in Spain um last week and you can see no replies. It gives me the answer but also it’s like you can explore it like dash. It will open the chart for you. You can also tell if it give you the right answer or not. And also you can keep interacting with it.

24:35 Yeah. Here for example I asked for definitions and as we have all the definitions in the semantic layer it provides like okay this is a page view and this is actually through and then you can keep asking questions and yeah all of this is thanks to to Also another thing um that is available is the light MCP but this is only for tech users because yeah it’s more technical sometimes and for example here what I asked from uh cloud AI I asked how many uh the evolution of our users in Eli dash month over month and it query the lightest API directly and I think it’s done for Thanks. So demos are done and now actually the prerequisites how this all works on based on top of what and there is a bit of secret here. But before we go into that I want to explain a bit of challenge. So why there is not so much you can probably find online about data in AI in projects and breakthroughs because data always adds complexity. So the good analogy here is DPT itself. So DPT introduced some good and best practices from software engineering in the data orchestration stuff engineering right. So like data test uh CI/CD uh agnostic to the to data warehouse but this came kind of almost like a decade after you always had it in software engineering because data adds complexity and why let’s compare data project to classical software engineering project.

26:11 So in a service like let’s say you have an app your current logic is what you care about. You don’t really care that much what was before. So what you have right now that’s it. That’s not the case for data because you have a state. You have a data that sits outside ideally not in your repository. If you do it is it’s not right. But you do care most of the cases what was before. So let’s say you have some tracking about some feature and it’s evolving. You cannot just say well that’s a new one. Let’s just report it from now on. What about the years before? That’s important. Okay, maybe not fully the same level of details but still important. So you cannot just neglect that and there are quite often like years of layer decisions about why things would develop this way and you cannot just neglect that and obviously if you change something like quite often there are a lot of downstream dependencies they not always even living inside your repository let’s say you have some reports and hacks and lie dash I mean you can make build the connection but things can break so this always adds a complexity and data itself is also a huge variable because well you can write the perfect model which like in wacom works nicely When the data is good but data can be messy there could be bugs right introduced like there could be noise let’s say if you deal with a real time data coming to you from through web of ad blockers they can mess up with the events that arriving so this is not going to be able to deploy to production so this also adds complexity to your data projects that’s why when you actually want to work with AI on data projects you need to care more just with your code that you have the repository you need to actually give all those insights that you are having in your head as data analysts and that’s basically the secret. There are two things first one as boring as it says the contact management basically documentation. So you probably already doing this if you work in software engineering but in data at least for us we took it next level.

28:00 Um in our DBT repository as already mentioned we have like 25 different business domains. Okay. And they have a lot of domain knowledge specificities like really a lot of terminology a lot of um I don’t know like KPIs like Paul was mentioning views video views they are different for different platforms. uh let’s say you have a streaming stuff or you have like okay this is like TV this is a a different definitions but in the end you need to combine them they have a different renders they have a different tracking this all lives there and so far I mean mostly you have it in your head over the years so maybe I remember some stuff I maybe I don’t you need to go like go on the slack some documentations ideally if you can find it but essentially that’s very difficult to keep up and especially imagine you have a very smart brilliant new hire are super senior and you want them to be efficient and it’s just like yeah this is a code make some feature it’s not going to work because they’re going to spend like weeks months explaining okay well what this is doing this because of three years ago we had to do that right so you’re doing this and obviously doesn’t matter how efficient how like how expert is the person you spend time on on boarding the person in your team it’s basically the same stuff you’re doing here with the AI you’re on boarding your agent with all the context but you do it by preserving all this context inside the repository in our case. So as I said we have 25 domains and what we’re doing for each of those domains we cover three files developed the diagnostic to AI uh one of them is as you can see domain MD it’s a super rich context file designed actually for for humans to read there is a AI version super condensed token efficient and there is a release notes so how we were doing this basically I forced everyone in the team they didn’t like it for several weeks basically sit we split the domains and like yeah for two weeks we just documented the enough. So I had to find all the documents uh all the Slack conversations if there were any. If not I was just sitting on the couch and basically talking to my laptop like an idiot explaining all my history knowledge from the from the past years and that’s basically in the end uh was documenting all the stuff in the repository and now only by doing this you can already do this consultation that Paul was showing right because AI is reading this docu domain documentation from the files itself.

30:13 Uh so what’s inside basically all those business decisions why it was happening because when you see the code you have no idea why this is happening you just see how it’s done but quite often you don’t you don’t know why. Okay next one next phase is specificationdriven development approach um you probably also heard about this it’s like all over the place in the in the world I kind of came to this myself and then I discovered it’s actually a big thing. So by the moment I discovered this thing, I already built a framework that actually helps you to develop your specifications, structure them into pieces. But then I figured out there are several frameworks already available. So what this one is about is basically nothing new really. It’s how project managers, architects I used to work like for decades already. You basically collect in all the inputs you want, requirements, record them somehow. Then you think for a long time I don’t know Google how to do things, right? So design it and then basically split into some actional pieces and you go and develop. That’s basically how it’s usually done. Same stuff here. You collect the requirements, you write specifications, you create a tasks and then execute them one by one. The cool part about this, this could be all done in matter of like hours not weeks. And I’ll show you like some of the examples how actually kicking this off. So as I said there are several already available pretty mature projects for like open source projects to facilitate this specificationdriven development approach. So you don’t need to really prompt it always the same. It’s always going to be like pretty detailed and done for you. Uh we tried several and as I said I actually tried to write myself which is kind of worked but I okay I didn’t want to maintain it and in the end I went for this uh agents framework from builder methods. You can go and check it out. Uh it’s also easily customizable. So we added some stuff on top of that like for example self documenting which is important because let’s say you spend those few weeks documenting the stuff for domains nice but what happens in a year it’s outdated. So I made uh as a requirement for the each feature we developing. In the end we call this extra step uh that we defined basically is look at your specification that you just developed and figure out if there is anything that you really uh need to add to the context of your domain because you’re developing new stuff. It’s here. You already know all the details. Let’s figure out if it makes sense to add it to the main uh documentation of the domain. And if it’s even not really like that much, it still ends up in the in the release notes. So why this is cool? Because actually it preserves all your decisions that were you went through uh while designing the features in repository as files. Those are the files. So again when you have a code you know how but you know why you don’t know what exactly with all those stuff if you actually keep doing this systematically you have all the history about how you get there. And imagine the scenario let’s say like in a year there is a new framework that’s doing something that you you couldn’t do before and you want okay cool do I have some features that already exists that I can do better now and you have it already commanded you have all these decisions from a year ago you just go and say like try again but with a different framework you don’t need to overexlain it again so those are real examples how I right now starting the conversations for specifications They were just slack threads. So basically I’m dumping all the information I have. Let’s say on left one there was a building new domain right we have like three or four different onboarding sessions kickoffs with different team leads with the business with the technical we were transcribing this we took the transcriptions fill it here we got the documents that are available some meeting notes that available mockups if they are and then it there and then you start actually engaging into with the interview. It ask you questions how it understood it. You ask it questions how you like want it to work and then in the end it documents it all like as I described with all those uh domain documentations files and then later you specific you split the tasks and then you can give to different team members even and the cool part about that you actually don’t really uh quite often you can run them in parallel again. So as I said those development sessions when you execute a task you can actually get to that phase multiple times for different tasks throughout the week and then like preserve like half a day let’s say and then you crush them all all those 10 different specifications in one session if you made your preparation made your homework very well before that.

34:40 So immediate benefits benefits just by uh writing the documentation for the context. Uh my team is still kind of having this habit like ask okay ask like yeah maybe know and few times already like did you ask actually I and they did not because they had this habit and now they actually do ask it first and if they cannot find the answer they ask me that’s also signal for me this is under recommended I need to go not answer them I need to go and answer actually to repository committed and then okay try ask again that’s a thing right maybe it’s a bit of overkill right now but eventually over time it’s going to really pay back so imagine you a new collect joining we can do it like in I mean we don’t really need to like babysit and like handhold they can just talk about particular things we can okay like your area is this this go read it up you don’t understand something ask back so it’s going to be much efficient they’re going to be independent for that um and as Paul showed already you can also apply it to some external stakeholders because they quite often having some questions that you know they don’t and then basically can answer them by not even engaging with them directly Uh yeah, as I already mentioned, every decision has a Y, which is important because quite often in gold base, you don’t know it. Um as also already mentioned here, it’s you can revisit the project anytime, which is super cool. And like I’m definitely waiting for some new stuff to be developed in in the data area and we can track we [clears throat] can really crush some models that we just developed right now uh with a new approach. Let’s say like I know stuff like release new SQL whatever function that’s doing stuff more efficient that we have right now.

36:14 we can go and redevelop it. And if you really want to get back to the project that you just dropped or post half a year ago, it’s also efficient because you just go and tell, okay, update me all the developments where you were, what’s next? Um, yeah. And obviously if you basically doing it well, you spend some time on specifications in the end the quality of that work when you review that when you bypassing your like sniff tests and your design test, you have to also like take a possibility for this. It’s very important. But in the end you can if you do it properly the quality is going to be higher. So like uh one example I was like I was asked to develop some particular uh alert and in the end in the same time I was usually doing this I made actually five because actually suggested me those and they were logical like okay that’s actually nice and they also made the message itself formatting well. So I spent the same time as before. I deliver the alert that I was asked to but on top of that practically I developed some other stuff because it was actually no really other time for me to do. Yeah. And the most probably like the biggest discovery for me personally that’s actually I don’t know like how how is for you but I always hated to do the documentation and now it’s actually fun. So yeah, as here the eye made it enjoyable and benefits appeared immediately.

37:37 Thank you. Okay, I think there are some questions. Uh Sasha will go reply to the repo and then we can uh No. [clears throat] Uh thank you for the presentation. Uh my question is about product analytics and I was wondering um how close is your team to the uh tracking infrastructure the tracking plan? Um can you change it? Are you involved when there is the instrumentation of your web and app? So at just watch BI is actually a gatekeeper for all the rocket definitions that happen on the platform. So we use a snowplow framework and basically we are the walls like define okay uh yeah this is new fish we are providing specifications why well because in the end we need to actually know what to expect and to give the insight so yeah so we are pretty close because we own it. Yeah, a follow-up question. Is the semantic layer also useful for having a tracking plan that is for filling the requirements faster? So, one of the projects I actually did not show but is one of actually we are making it’s exactly streamlining the everything about specifications of the of the tracking. So, we have this huge spreadsheet right which you maintaining manually over years. Uh why why spreadsheet? Because actually sometimes you need to still communicate to outsiders like developers, right? So they don’t want to go into SQL. They don’t always read that. And the idea of that particular project was actually they can go to our BIA bot give me specifications for this particular feature if it exists. It actually extracts the up to- date state. Uh create a small spreadsheet. Again the based give it them. We still like watching it. If it’s good enough they can actually suggest suggest changes right in this particular spreadsheet. If you verify it all good we can actually feel it back. That’s one of the projects we didn’t control because it’s kind of young. It’s probably going to be second layer of projects we’re going to be doing. Um yeah, so semantic layer we do obviously have a product analytics explorers based on on the data but they mostly kind of still like KPI based. So let’s say you have a very very detailed tracking about your streams right we have like right now doing the streams as well. So every 3 seconds you have like a like a point that there is like segment watched change of of the resolution and stuff like ads shown and stuff you don’t show those events you show like how many ads were done. So there is still like aggregation and and some modeling happening. There is explorer that you can go and just query all the snowflow events if you want to. I mean if if you really want to and you won’t know what you’re quiring. But there is right now the slide dash they don’t really have access to do documentation about the tracking but as I said we have in this project that I think actually eventually going to also expose it there.

40:32 uh wasn’t this is a classic not a question but a but a comment um but uh I wanted to say you know another big benefit of having uh all that institutional knowledge built into this chat uh uh interface for new onboarders not only does it does it help those onboarders but like I’m finding that um people are going to ask some chatbot why why code looks the way it does they’re going to open up your code and say why why does it look this way and it’s going to give them an answer. Uh it might as well be, you know, the correct answer, right? Like with your with the work you’re doing with that foundational stuff and capturing decision-m you’re offering what’s increasingly like a preferred way for people to get to know code. You’re offering one with some guard rails, right? like that that it’s not just going to make up some about why the code is the way it is. You actually have an authoritative source. Um so I think that will actually pay more and more as time goes on and you have new younger developers who that’s their first their first instinct is to is to go to a chat interface to ask about something.

41:52 Um I think you’ll see huge huge you know payoffs compounding. Yeah. As I said like those things we show today with the develop matter of weeks. So by the end of the year we’re going to finish those projects and we already have several like in um well in the backlog. One of the things actually it kind of already suggested this that was dangerous. It’s actually making PR already directly from the Slack when they say okay this is actually the bad alert. Let’s actually disable it and suggest the PR. Uh well we we not really feel comfortable for this yet but eventually definitely uh so lash again aent basically semantic and data and KPIs but all this like generic agent that we develop right now it has access to our code base right for read only right so it’s basically GitHub MCP just read the files on the go and there are obviously instructions what you do so actually one of the other things that uh we also have another project is not for BI it’s for the company in general so this context discovery if you think about for this you can apply the same approach to all the repositories not really necessary uh BI and then you can ask any questions about anything in your company front end back end doesn’t matter and then it’s like okay there’s going to be some references entry point which also could be automatically collected from all the repositories right so this is here this is here this is here you can navigate to repositories and read it out so basically you can ask any technical questions not only BI and this are also going to be pushing to other teams soon here Alex I don’t know if he’s here like yeah he’s listening Thank you. uh thank you for the nice presentation and I congratulate the team in thinking futuristic uh thanks for also selecting such advanced tools already uh which is making this happen so quickly right so in the in the right time um my questions are twofold one is you mentioned that for developing a new project you’re introducing or kind of enforcing uh teams to development teams or product teams to start with an so uh are Are you managing some kind of template specification for different projects? Um because that’s where the game changes if the specifications are defined differently for each of similar use cases I will say. So, so we using this framework that actually kind of uh unifies the approach of coming with two specifications right so you have like requirements you have specification file and like you have a tasks it’s like coming with aenttos you have also specit from github is doing a bit differently but it’s kind of the same there is no like really template how you define like each specification for each particular project so I don’t not enforce it actually maybe like good question for Paula because she was the one that I had to to push to do that and she was like really firsthand this learning curve and how actually it’s really getting into action actually we are doing it completely different so um he prepares the requirements on a way I do it in another way sometimes I copy paste them things from from the users um I maybe sometimes I write something super simple and then and then thanks to the framework um it iterates it asks me question it’s like no I don’t want this and then sometimes for example I go to I know this is this way. So instead of asking the AI to look for it’s like I want this uh do the joints in this and it’s pretty useful. I have to say that even if it was enforced, it was like really really useful and I myself not coding that much either lately.

45:24 And the second one is uh how do you manage this why uh into the metadata you writing howtos are very simple code can extract the documentation how it is done uh what has to be done of course from the specification but why some there are some design decisions or do you have a temp uh template or metadata structure for you don’t really have a template, you basically still review all the things documenting for you, right? You really have to do that. And if you don’t do it all on the half of the cases and correcting, you’re probably doing something wrong. You really have to um I’m pretty happy the way it’s basically I mean at least at least on for now doing this. Um why if you tell it why it will record it and it’s going to be just basically as artifact maybe not for the domain itself if it’s like not super crucial but at least in release notes, right? Because this all those things you develop specifications, task like implementation plans, validation plans they still live in repository you can find it. Then uh agents they also good into like finding git history. In g history you have also like kit like dit names whatever you can find easily where it’s coming from. It’s like doing super fast. Well actually like I didn’t show it here but uh like data exploration itself. I was like when I was building this dashboard that Paula was showing I was using the generator from her but I didn’t just tell what I want to build I also told that okay those are the models those are the questions go query those questions beforehand know the answers know the numbers build the dashboards open it dynamically in Corser all those charts compare not only that build that stuff but the figures also matching so I actually took it like next level and afterwards actually found some issues right I had to correct a little bit but yeah this is basically like really as I said like 10% of time I’m writing this for fun it’s actually true for fun because I’m missing sometimes like just today was a bug okay this figure wrong looks wrong like I was actually busy in the calls and I just like on the side you know like I just posted it and like while I was like talking actually already found the answer in two minutes and already on the on the call already presented it I mean took a look it’s actually was correct yeah thanks to the AI first approach I think that’s helping yeah well AI created approach you still have to like really review that Hi, I really like your textbased um presentation. Uh I also like it was efficient. I really didn’t spend time like in making those icons and like dragging them and coming up because actually it’s a text.

47:51 How did you how did you do it or did you use a specific tool? I use not clothes in this case but like yeah uh I was talking to it like so the whole by the way the whole presentation was also specification right I went with the specification with the task and develop all the chapters are done like this and then was validation so actually I was yeah and then then it produced me I mean I could probably even show it to you if you want but it’s actually those are just markdown files I copy the text and paste in Google slide it’s like career new it’s it’s a text box you just paste the text and it’s actually format nicely I mean applied some colors and bolt because you cannot have it in markdown but essentially yes that stuff I was cheating a little bit here I was thinking it’s like it’s pretty impressive what you can do like with this AI first kind of infrastructure design and I was thinking like it’s solving a really good problem which is like institutional knowledge and institutional memory and how that just disappears o over time especially when you have very high turnovers in in these tech companies and you’re making a kind of it’s almost like an ethnographer of the local institutional knowledge of your workplace and that’s really smart. I really like that. Um it was hard at the beginning because first of all I had to do it myself. I actually like caved in for several weekends and I was able to because the kids were like not with me. So I really and that was also engaging because the commentation was really fun and I had so much in my head. I really had to offload to somewhere and I finally found the Yeah. And then I kind of ask a team and they’re like, “Okay, so we have 25 demon, you do this three, probably do that.” And we spend time on this several If you do it right, then you can really leverage all that.

49:34 Yeah. Like I really like doing documentation, but I’m realizing that’s quite rare. It’s quite unusual actually. I’m not liking doing documentation. I hate that stuff. I always was saying because DBT like before that we didn’t do that much like code is your documentation. If you don’t know how to read the SQL, learn the SQL. That’s our way of calculating stuff, right? And now it’s actually yeah, now it’s mandatory. Now it’s easy to do and as I said it’s automated because we develop it’s also update documentation on the way. This is key in the in the end it’s like really adding this to the context. My question is did you have a legacy infrastructure to begin with or would were you creating this uh completely from scratch? We had like 1600 models in in repository with a lot of historical uh baggage. As you see the DBT float thing highlight how many actually we had like outdated they were like there are several folders with the tens of models inside for the things we were reporting like 3 years ago they’re not relevant anymore but because we always about like okay building new stuff shipping new stuff maybe they don’t even execute who knows but like why do you have them in repository and like this like scanner found them and like kill them okay fine we don’t need to document them stuff and now like we have this new domain developed I really spent like I don’t know half an hour like collecting all the things in all the threest documents talking to it and Now we have actually I can tell like I know Sana this is documentation here.

50:53 Build your model. Okay. Thanks. Um yeah and thanks again for for amazing amazing presentation. Uh first of all uh kudos for um yeah I think we all seen all this dooming news about oh we have an AI and nobody’s hiring but as we see they are hiring. That’s a kind of a you know controversial signal that uh yeah you can receive only in such a sessions. Um but my question is more [clears throat] related for the underlying thing. So you have now like a much better leverage on the things. Yeah. But probably you had some kind of agile ceremonies or some kind of frameworks you were using for development. How did it change for you like for a bigger company uh for the planning for I don’t know backlogs or something like that? Not clear yet because like it’s few weeks in and it’s really changing much like it’s very impressive. Not all the teams are like same advance even software engineer but they are also getting there like I know like one of the market like they have like really big advocates. Um some people are just really great in experimenting with new tools and they really like really progressing there. You can see that uh like I was still trying to figure out so like my team can definitely be more uh productive you as always like you spend more time on fun stuff like you know talking about projects designing them code review it’s actually also interesting effect like when you code review stuff like code of the agent you know like whenever someone else showing you the code you see issues right away because you didn’t write it same stuff here it tries like you see the issues right away and it’s more efficient even because yeah that’s a fresh one um about the height and of um companies definitely hiring right now for the data team. Actually, we’re not hiring because we’re trying to figure out the way to kind of reshape the the operations and we definitely want to scale productivity of the of the of the team, right? So, but I want them to spend less time on uh low-level coding. Not that it’s not cool because you still need to be able to read that code, but right now we trying to figure out how to really apply it and be more efficient.

53:01 Okay. And ju just another one uh do you use anything uh for the forecasting like maybe you kind of utilize the current uh data or something like that for developing of the new features or something like that. For forecasting you mean data Yeah. Yeah. Uh I would not say that. I mean there are some questions and that we’re going to be tackling for sure. I mean we have Nora she’s here. She’s our data scientist in the company. I definitely need to consult with her. So we’re not going to use her to code but definitely the approach have to be same right. So you still need to go like through the human if it’s like okay or not have enough things to do now. Yeah. Yeah. Last question maybe one one and a half if I may. Thank you for the presentation great talk and uh great uh call on the stateful aspect of the data transformations. I’m very and thanks for sharing the builder methods the repository like u I’ll make sure to check this out. I’m curious about this the context management. So I mean uh I would assume this is similar to like a bunch of like MCP like domain specific MCP servers like with some some rules and memories like I’ve been playing around with the MCP servers and I’m like I’m very much curious how do you yeah like known data quirks. How do you like uh write down all this history? Like how do you tell like the your agent, okay, this decision was like was a was a This PR was wrong. Like do you like would you literally store all the history of the communications? I don’t know stack slack threads, PRs like would would you vectorize it and somehow like how would you say like an agent like this is the the right decision and this is wrong and how would you prevent your agent from like hallucinating? I mean not hallucinating but actually using the the the the bad data from the history.

54:51 So there are like several sources for the context right. So the first one is git history for sure you still have it and obviously you don’t need to write yourself you need to document what’s happening and agents are really good in reading that second as I explained those domain documentations. So this is like this [clears throat] domain MD and domain AI is current state of the domain. It doesn’t have really a history because like we want to on board when we want to develop it’s like current state. The main release notes is like what happened when this also nicely connects to the artifacts from the specification development because those things like specifications decisions validations they still live inside your repository. They’re not part of the domain uh documentation. So whenever you need to they available and you can easily find them from the g history from the domain release notes. You can find those folders. can find out the the reason if you need to go you you look at the code okay why this is like that you can find the g history of the line okay this is a ticket the specification and [clears throat] there is a reason why so those like specific feature documentations they are pretty rich but they’re not loaded automatically only when you need them I see okay thank you and second question like we don’t have to go to detail just like very curious uh like in few sentences about this like cloud cloud bot Agent how does it work I mean like it sounds like like way too good to be true because I mean I I would know like is it like some some kind of like micros service with API it has to send like an API call write uh write a query I don’t know like talk talk with the like other different APIs GitHub dema probably yeah Sash [laughter] is looking at you so I’m curious how how how does it work a few sentences please yeah it’s a cloud code under the hood so with several MCPS so it has connection to GitHub to snowflake to our composer to airflow to read logs and uh investigate it to anything you want.

56:42 So it is actually like quite quite fast. Uh so it’s not so fast. So it takes seconds or minutes to make a response. So it’s a uh it is a complicated request. It could take maybe five minutes to digest it u practice different uh tools and collect all the Understood. Thank you so much. Thank