Stop Moving Your Data

0:04 This is actually this group right now. All of you, you are the first one that is actually going to get a presentation about a database completely. And Lucas, you are actually the first person that is actually probably recording a distributed database made in Berlin. Um, yeah. So it’s more than a database. Over the course of this whole presentation, I will tell you everything about it and also for all the previous presentations questions and stuff like that that was asked. Um I will answer them but actually database will answer them for you. So let’s start. So um who I am? Um I wrote the Rust compiler parts. Uh, if you know Rust, who knows Rust? God damn, that’s so nice. Um, so I wrote the Rust compiler parts. Um, that’s the good thing. Who knows Apache arrow? Who knows data fusion? Yeah, that’s memory layout of the arrow. I wrote it for Intel architecture and stuff like that. Oh, yeah. So, um, that’s the thing. and I built the databases in SAP and uh yeah also some American companies query engines I’m still building a database it’s called DERA you can check also there’s a separate one I’m a database guy so yeah and last nine years I built a dragon for you I’m not joking and where the dragon comes if you’re actually familiar with the compilers. Dragon is the signature of the compilers for the computer scientists. That is where it is coming from. It’s not a random thing. And uh we are compiling um SQL to native machine code and this is how it works. There are only four databases in the whole over the world.

2:15 This is the fourth uh and two of them are research databases. only one of them is in the production and they are 10 times slower than us and we will see this. So let’s start with it. So um everybody’s type everybody was like everyone typing SQL back in times now things are changing you know agents and stuff. So um thanks everybody is a data brick who who is datab bricks customer. Oh sad very sad how much money you are actually spending on that anyway. Yeah who is snowflake customer? Oh sad very sad again let’s go. This is free just saying and let’s go with it. So um agents we are not going to code them anything manually anymore and that’s how it rolls and um like Gartner is basically saying that everything that you are writing with the agents is going to die like 40% of it. So um the previous um Julian right so yeah you said that if you want a dream like you have a dream of doing the same thing it’s improving itself it will soon is this is the dream so um the thing is about that um everybody is basically um paying for this uh large enterprises are basically you know uh paying for getting paid for these kind of things from the medium-siz customers and um most of the times that your data is lying to you because what is happening is that you have a posgress somewhere and you have a allup database analytical database you have ETL who is maintaining a ETL pipeline let’s go who is maintaining yes thank you ladies and gentlemen that is the thing because you are spending your life for something that is not necessary you know, yeah, I mean, I hope that it wasn’t needed, right? So, 9 years ago when I see that everybody was actually doing this and nobody actually gives a damn about it. So, I said, why not? I’m actually building a database that is actually having a single snapshot every single time and always consistent whatever happens. And I also have the analytical side of it working. And you know what? If you want to know how fast we are, we are faster than duck DV. Who knows the Duck DV? Let’s go. We are faster than them. You can check clickbench benchmark and you will see that we are actually 10 10 times faster than duck DB. Okay. Goddamn. Let’s go.

5:07 Yeah. Half of your engineering capacity is on plumbing. I talk about this. This is the part that you are doing the plum plumbing which you should actually do something more s sensible developing the next generation of things but you are still spending this time on that and I think that after working on these ETL pipelines 5 years everybody will get tired of it. I hope so and everything is everybody is probably tired of that. So um yeah um this is most probably what it looks like right so there’s a postgress for transactional payloads transactional stuff you have a one warehouse and that warehouse has also something like quadrant blah blah something right next to it that you are ingesting right so the or like you try to use pg vector and right next morning your postgress is slowing down you don’t know Why? I know why. Uh and yeah and you are actually trying to implement some BM25 method and then suddenly you want to have a search index you included loose sand or something like this to your pipeline which wasn’t needed because I solved that problem for you too. So um and the famous last thing semantic AI everybody’s talking about it like build the semantic layer right like oh my god it’s so amazing I also built that for you [laughter] I’m not joking it’s like a fun talk that’s why I want to wake you up now you the question one of you asked the question I I think you you guys how I know that I’m actually knowing the AI model queries the correct thing.

6:57 We have a semantic AI layer. You can check on your phones and look at the blog post about the semantic AI layer. And it’s automatically defines the models for the idiola guys down there by the way. Uh it automatically optimizes the queries. Whomever asked that thing? Yes, it compiles the query to the machine code. So it’s actually 50 milliseconds to basically scan 100 billion rows not 1 billion you know it depends on how you are actually also sharding and all of that is not on a cluster on a single node single node so you are not building a warehouse you are just downloading a docker image putting onto a machine and it just works it’s not like that you are not paying $1 billion for something you are not going to pay like 20k per month and it will just work. Yeah. So um these are three numbers from your different three different things. For example, you’ll ask for the revenue first thing said 4.2 the other one said 4.6 six your growth dashboard your BI dashboard and suddenly like you asked to the agent which you asked the exact question that how agent can be trustful and it says 4.3 so you don’t trust that thing anymore right so yeah so um my number is always the same I want that so this a spider benchmark for the people that knows so spider 1.0 and spider 2.0 Z tries to do text to SQL. We don’t need that because we have a semantic AI layer automatically resolves the models. If a model is not showing a correct dimension or understanding it, it automatically getting a fact out of it. And yeah, we have branching. Who want? Yeah, you wanted to actually Julian wanted to have snapshots of your data set, right? With This is basically um the case. So what you are doing you are working on a live copy of your data and you know that agents you heard you hear that right? Every time you have something like, “Oh, agent removed the production database. Oh my god.” Yeah, let’s go. You know, um that thing is over. We are copyright branching from the existing database in immensely like you can create 10,000 copy of your data. It doesn’t cost you anything unless you write it or unless your agent writes to these branches.

9:54 Think like git. Git is inside this. Okay. So your agent your versions everything can be verged inside here and it can also every version can go to cloud and then come back like you can say that I want to store this versions changes I can push that to the cloud and then it just works like this. So this is the database that you are not paying still by the way you can still try with a single command and I will show that command to you. Yeah. So um this is like whole idea is one engine one line copy and no pipeline in the middle and this is done in Berlin not going to lie that’s the best thing and all these things yeah because like American companies are trying to do these kind of things be be aware of what you are actually listening right now because all the American companies got investment from all these you know big companies that we are paying in Europe and we are building this in here and I want my folks and people like you to do things with me and this is actually I think that what we call German innovation in it so yeah so um think that you can program your database do you see so um you have UDFs not the UDFs you’re thinking in eight languages You can write a command like literally program and you can program your own database. You don’t need to write like have a separate like container or anything you know everything is what you are seeing in here is inside that is the good part of it and you still don’t pay it guys. I’m not joking.

11:52 Yeah. And I have a very old PC. Yeah. Um everybody says that they splitted OLTP and OLAP from the uh one engine because they can’t do it. We did it with this is the time that stop being true. Everybody say that say that um HAB is dead like hitted analytical and transactional processing like I said it otherwise but yeah um it is written in u rust and fully It is using the cuttingedge research made in Germany in Technic University of And this was not actually this is built on top of it in a sense that research and some other research that I made through the my through the lifetime of my career basically and yeah so every query is a machine code that is the good part and it’s saved on the disk next time same query comes it is 2 milliseconds at first time it comes it’s 10 millconds for example it’s like complex XQ think about rollups and I don’t know whatever you can think of everything is basically I wrote a compiler that is actually a database okay so this is the whole idea of it and um next slide go yes and it speaks posgress you don’t need to do anything so whatever you have that talks postgress it will speak posgress there are additions to it that error codes for example for solving distributed uh transactions and stuff because you know Postgress doesn’t have distributed transactions you copy and replicate Postgress you don’t need to copy or replicate this it automatically it automatically replicates itself if you want sharding it automatically shards itself and the good part it has distributed commits if you have a cluster one node commits to all. Okay.

14:10 So actual distributed database based on a new uh method and stuff. So um let me go faster. This is the benchmark. You can find this online. So who which ones do you know? Yeah. Right. Yeah. I mean this is this is wrong. By the way, I I asked the cloud to make this nicer, but it can’t make it nicer when you it’s like needs to be put some colors in it and stuff like that. But yeah, so this is all the fanciness, you know. This is the one of the famous databases. Uh this is probably, you know, most of you probably paying for Starox, right? And uh most of the click house we are faster than most of the click house you can see on the internet like benchmarks are basically click house benchmarks like we beat them in their own game I presume. So you probably use polars and stuff like that. I’m not going to say say this. So this is um embedding like you know the embeddings vector generation and stuff.

15:28 We have automatic like embedded similarity generations and like calculations and stuff like that. I can skip these for the people. So um this is like there’s like a literally in your database semantic AI layer just what you need to do is point your harness to it like basically give an AI model because that is the thing. So it there’s like a tool to define a model data model uh look to the facts and check them these facts are actually the facts that you are looking for this is what database is doing I’m not doing anything this is what database is doing and it is correlating all these data like joins and like the scan question that you had about the how fast it’s going to be how you know efficient it is going to be all it’s inside of it. Okay, so this is the whole point of it and this is basically what is agent is handing you know you can see on the right side like sum total and it actually gives hints if you your harness is aware which is how it is becoming aware is basically by the database. it just looks to the you know the columns and it starts to save the hints save the AI decisions into metadata tables inside the database and all of the agents needs to have a explicit permissions all of these things that I’m talking about is already inside baked into database so there is no way that if you use properly the agent API like agent semantic AI layer you can’t get rid of your So there’s no way uh that you don’t know what your AI agent did did do you know.

17:18 So that is the whole point of it and it doesn’t write SQL queries database writes it. So that’s also another thing. So um yeah this is like basically what I explained to you like about like uh so the agents credentials are databases credentials role based access control RLS because you don’t want for example um the guy drinking coffee doesn’t want to see the guy drinking tea at the same time you know in the database terms let’s see so um that is the whole point of it And yeah, so let me wrap up on this. Um, you can think that you have a risk by a model, but you wrote a single Python function that runs with your database real time risk scores, writes the risk scores into another column and you don’t need to deal with it. This is possible with the just this function on the right side and like basically you are just writing a Python code and that’s it. it just runs you know you can use data frames with that Python code you can use anything basically and there’s a special CLI that you can compile this and put it inside the database so it can run and it is there for having this on the big query probably you will give like 10k per month good luck I am not asking for money still you know yeah, this is the column mechanism and point in time recovery. Let’s say that you still failed to make a you made a mistake on your main master branch of your data.

19:22 You can go back in time time travel and go to this time exactly with everything inside. You know, this can can be just something else. Um, and this is your agent. Let’s call it 1442, but it can be anything. Agent Teao, let’s go. I can clone at any point of time. Get a snapshot and use it. It’s instant. It’s not going to take time. It’s like a real database. It’s not a snapshot because we do some magic to do that. And it is the same machinery and it’s just pointed backwards. So time is running and every single time you can do a clone, we just see the deltas um in a very efficient way, you know. And I’m closing up. Don’t worry, I hope that you’re not bored. Um so, um it says 5,900. It is right now 10,000 tests plus tests are running for this database. And for the people that are uh model, have you ever heard model verification? Does anybody heard like TLA and stuff like that? Model verific that database is model verifi like model verified. So whatever it does, it is actually going to work in any real scenario hopefully.

20:53 I’m not saying that it is not bug-free. There will be bugs but specific parts about the consistency guarantees and everything is tested. So um that is one of the things let’s see. [sighs] So I am asking it was a very uh bad way of saying stop moving your data and naming the title stop moving your data. But I explain you something revolutionary the golden bullet in here. But this is the case. This is the time that you start branching your data because if you don’t agents will mess up your production and agents will do it instead of you. And this is the time for you to get ahead of what other people are doing on all over the world and what other big companies are making you pay. So this is the inception moment for all of us I think because this thing is started to be the reality of Germany. So hopefully we will do that. There are a handful of design partners. If you want to um like work with us, you can talk with Bashak and me and you know um that is going to be nice to work with you. Um and yeah, you can find me on LinkedIn.

22:22 Please add me on LinkedIn and tell me that how it is if you want to know. Uh but I hope that I will hear something nice. Uh and you can point like I forgot to show you but like you can literally point everything. So if you want to learn everything about scrum DB you can point um LM’s full which contains our documentation which contains our to your harness and ask every single question that you want to know. You want to design an AI? Yes. You want to design a harness system? Yes. You want to design a pipeline? I hope you don’t do it. But yeah, you can ask this and talk with conversationally with the AI and it will install and use and configure ask some questions to you. And these are all the references that you have seen and I think it’s time to Okay, there were some people that were Okay, now it’s working.

23:39 There were some people that were falling asleep before this talk, but now everybody’s awake. Yeah, let’s go. All right, so first off, I have to say the whole time I wanted to shout and say, “Shut up and take my money.” Yeah, take it. That was really convincing. I’m I’m just curious where do you want to take this? Like what are the next steps? Okay. Uh I’m not going to do what other companies like American companies did to That [laughter] is the first thing I would like to not do. Uh first thing I would like to do is altogether in this country and all over the Europe we use this and get advanced. This can be basically used as you can imagine for geni pipelines literally for the how fast it is. You can basically use this training llms and everything you know. So um I would like to make this thing used in everywhere. It’s not like going to be like my idea is not exit the company or anything. My idea is do something that is actually you know used by so many people but like built a next generation of everything this includes there’s a vectors index and stuff like that inside I’m not talking about it which is way faster than all the things that you know there’s also a blog post about it please check our blog about that so I just want to make this used that is the thing and build this together. You show me the varieties failing and not only me to the team that we are working we I want to have a nice team together and you know build this. I worked on this for 9 years guys. It’s like a not an easy thing to So uh I hope whomever works with me from now on like have an easy thing.

25:48 So um you say you want to go global but it’s not open source. What is your plan for expansion? Like how do you I’m not going to make this open source. I’m very sorry about it because if uh you think about this um how someone does work for 9 years on something that is working and you basically build a compiler from scratch and I’m not going to make this open source. If you want to use it right now it is free. You can like literally curl bash it. you know you can either download the binary that contains vector index um all these you know shenanigans that you see it is extensible but I’m not going to make this open source I’m sorry that’s not going to happen unless I’m very convinced that it needs to be thank you for for the talk so you listed a lot of features but you didn’t talk too much about the deployment model and the the architecture of Sure so like how do to scale and is there like master and slave and yeah um there is normally normal deployments if you know that there’s like a etc or something like some external thing is coordinating we don’t need to we use a epidemic algorithm that’s called swim uh that knows the membership membership changes and stuff like that automatically uh propagated uh one thing that I forgot to tell this database tested on one scale one scale means area network which means you can actually replicate this with America east coast U EU and I don’t know somewhere in our pack like you know so this is the whole thing that is working there’s a distributed setting that is where we are currently uh making money you know uh distributed set uh is the only gatekept feature um yeah so this is how it’s going to be.

27:51 Um yeah, that is like I can talk about like different deployment scenarios. You can deploy to the bare metal, you can deploy with Docker, you can run on your computer. Please don’t try to run this for 4 GB of memory. You it’s not going to work. It’s just a joke if you are going to do this. Um yeah, but it will work with 10 GB of memory and yeah with four CPUs approximately. So um yeah that is it so far. Hey first of all really cool product. Um I love it. Um I completely get pre-ompile JIT. Yeah that’s going to work. Um I’ve not seen anything that didn’t get faster when you did it in Rust either. So yeah. Um the one thing I’m not I don’t get I guess is that when I start to work with really big data sets is normally bounded either by something like a sort operation if you had to collect something in from multiple nodes or just the general IO bounds that you hit when you have really large amounts of data.

29:07 Um how do you overcome that obstacle? We have um I have invented a zero copy inmemory mechanism that’s thousand times faster than a that’s why data fusion is behind us um that’s called borax and that actually allowed the second problems to be solved is like IO problem and we have different levels of compression mechanisms on disk. um whole uh clickbench benchmark is 1 billion in very wide and u you know short columns so you can imagine how big the data sets is in a normal scenarios which is like it’s 100 gigabytes more more than that actually I don’t know it’s like compressed size is different but um we are whole like ingesting that it’s just on disk it’s 15 GB GB. So how much compressed you can imagine right now from 100 to 15 GB which is a which is also a good thing if you are going to run this on I don’t know in AWS you’re not going to give that much money to um disk because uh I don’t know um it’s like a thing I guess to make it um distributed comets I’m working on a research paper right now that um I invented a mechanism I also invented the vector index of this that is actually breaking right now the famous things that that you know uh I’m going to hopefully publish this research and hopefully you can read afterwards.

30:47 Thank you. So uh thank you for for the presentation. Um I remember there was um another uh project that was closed source and um they decided to uh make a commitment with the community of the and they said look we keep it closed but for any reason we decide to move on and the project is not in our scope anymore. Um there is a agreement with the community that we will release the the source code so the community can take ownership of the project and continue after the the first creators for any reason to decide to move on to other projects. So I was wondering if this something that you would accept. No I am not going to accept. I gave my 10 years to this and I will be successful in this in Europe and I’m not going to do this for free. I am going to make us all together like use this and make money and revolutionize the data science and data analytics with this. That’s the whole idea of it. Uh they did somebody did Reddus did they went to closed source they went to open core because they couldn’t make money and they went to close source again and again right do do you remember these times many companies did that what is what did get what did they get out they didn’t get out anything like you’re using still the radius and right now somebody is rewriting radius in rust and it’s called wy it’s like a so whole idea of the fanciness whole technology is gone already you know and now they are doing it a worse way from my point of view that’s how I see it you know so um yeah I hope I answered this question in this thank you any other question it’s time to go home So I wanted to thank you Teo.

33:04 [applause] Thank you to you all. Thank Thank you.