dltHub and the Future of Autonomous Data Platforms

Intro

dltHub and the agenda

0:05 Hello everyone. My name is Shreyas and I work as a solutions engineer at DLT Hub. And today’s talk I’m going to talk about DLT Hub and how we as a company envision future of autonomous data platforms. Uh so the agenda for today is that we will initially go through what DLT is, DLT Hub, and go through how we are trying to build an autonomous data platform and what makes it safe, and how we are applying the levels of autonomy. And in the end I’ll show a small live demo of a self-healing pipeline. Uh so in any Anyone in the crowd have ever heard about DLT DLT Hub? Yeah, okay. We have a few hands. And so

Data pipelines in Python

0:45 for those of you who don’t know, it’s a open-source Python library. Uh you can easily load data from like REST APIs, databases, cloud file systems, and custom sources. So it turns your messy data into like structured data sets. And uh it provides a Pythonic interface to extract, normalize, load, transform, [clears throat] and inspect data. And it’s like really reliable, flexible, and scalable to use. And what makes this uh production-grade ETL tool is that it implicitly normalizes nested data and it gives you control as to how much deep you want to un-nest your nested data. Uh you can it it can easily handle the schema evolution by setting schema You can enable incremental loading, use different write strategies, observable observability through loading for progress logs, and a built-in dashboard which we built using Marimo. And uh various data access through data frame, SQL or Ibis. Uh but this is not just a open-source side project but is actually used by data teams across various industries. So we have around 6 million downloads a month. Around 18K companies have running

Agents and production operations

1:55 DLT in production. And we have an open-source thriving community with over 200 contributors. And what’s the most interesting number isn’t just how many people are using DLT, but rather that um almost 91% of pipelines, what we observed from Jan 2025 to Jan 2026, is that 91% of the DLT pipelines were built by the agents. And so we established that building pipelines is really easy by the agents, like they build almost like 10 times what we can build generally, but what the hard part of data engineering is that how we keep, maintain, and make sure that the pipelines, the data which we get from them, is trusted enough. So what we are trying to do here at DLT Hub is that we are trying to make the operation aspects of data engineering like deployment, scheduling, observability, alerting, scaling, and governance as accessible as possible. So the core vision of DLT Hub is that we want to democratize data engineering for everyone with using AI. And so DLT Hub is like an AI-native data engineering platform. So what this does is it’s one platform to build pipelines, test, deploy, and observe all in one place. So this happens through you can do this with using your coding agent, which then works with the DLT Hub agentic layer, which is powered by DLT Hub AI harness and context graph, which I’ll explain more later.

3:28 And this can work with uh And this can work with the DLT Hub managed infrastructure. So basically, DLT Hub manages the infrastructure for you so that you can easily build pipelines from various sources, test, deploy on any warehouse of your choice, and serve the data to your end users. It may be agents or MCP clients or uh or data apps like Streamlit. So, we’ve already established that, you know, the agent can build uh pipelines in the platform. But, the question is what’s autonomy which you give to these agents?

From assisted to governed autonomy

4:05 And so, I would like to draw an analogy to self-driving cars. So, we never reached autonomous driving from manual driving to autonomous driving. We actually went through cruise control, lane assist, and then hands-off driving. And so, we believe data platforms can also reach the same level of autonomous by uh and what enables this is it’s bounded. So, engineers decide what an agent may do alone. And every action an agent takes is always recorded and uh with the reason, and it’s always learned. So, you build transparency and the autonomy grows with uh the agent’s track record. And here at DLT Hub, we achieve that in four levels of uh data platform autonomy. So, the first level is uh the L1 assisted authoring where uh the agents build the pipelines, validate the data, but a human approves it. And in the L2 level, it’s basically you have a background-verified agent which then diagnoses failed pipelines and gives you a suggested fix. And in L3 autonomy, where there’s an another agent which reads the diagnosis from the L2 agent and actually writes a fix. And uh L4 is a governed autonomy where uh you tell the agent all these pre-author- authorized fixes it can do, and then it just notifies you. So, right now we’ve shipped two products, which I’ll talk about later. And today I’ll be demoing the supervised autonomy and how a pipeline can actually fix itself.

5:35 Um and what powers all this is the context graph. So, it’s basically a graph of everything on the platform and

Context graph and safety bounds

5:43 how it connects to each other. So, DLT Hub falls in the data paths. So, when you load a data from sources, it goes to the pipeline data to the data data sets in the destinations. And so, every step is recorded onto the context graph. And so, in the future if a source changes or like a schema changes and like say you have a new business logic or you have a new different question, everything is written into context graph so that next time an agent tries to propose something, it actually reads the context graph first so that it has all the context necessary to give you the correct solution. And whatever action it takes is written back to the context graph. And then that’s the recorded part, but bounded part is how we enforce limitations on these agents because in the end, you know how agents can hallucinate and act on themselves independently. So, the way we try to limit this is not through prompts, but actually configuration files. So, we have different profiles where it can never touch your production data. It can only have access grants which can be scoped according to how you feel about it. Secrets are never touched by these agents. It It has only read-only access and the spend can be capped with the number of turns it can actually work on these issues.

AI Harness

7:00 And so, these are the foundations of which how the platform works. And now coming on to how we climb the autonomy levels. The first product we launched in March of 2026 was the AI Harness. So, basically AI Harness is nothing but toolkits, skills, rules, MCP tools all bundled together which your coding agent can use to build data pipelines. And the way it works is you basically in your cloud code or code X wherever you just prompt that you want to ingest data from like these two resources like request and issues from GitHub rest API. The DLT Hub router scale implicitly understands your intention and installs the necessary tool kits which comes with its own entry skill. And then you get a successful DLT pipeline within like the first And then as you can see the different development life cycle stage in data And for each stage you have tool kits which comes with its own skills. And so what you observe here in the life cycle feeds onto the next round. So it forms an iterative loop which is always self-improving. And here obviously the agent lives on your local computer. So

Job Inspector

8:16 in the next level where we move where it’s explained operations, we launched product called job inspector this month. So here the agent runs on the platform. And what this does is when a pipeline fails you can trigger this agent based on a pipeline and if it fails it reads the run records, the logs, the traces and it classifies the error into either it’s a configuration error, it’s a credentials error, code or an upstream data error. And then it gives you a report like you see here. Like what failed, why did it fail? And also gives you a confidence which it thinks it’s correct. And also it proposes a fix that in this file in line 40 you might have to change this. and you might be wondering how do you deploy this agent? It’s actually very easy as writing a pipeline job. It uses the same primitives like the same manifest file, triggers, schedules and the AI harness use. So it’s as simple as just defining defining this in Python. And you just define your model and the key and the DLT Hub platform takes care of the rest. So this is all explanation till now. So in level three we this is still in testing phase but this is also another verified agent which acts on the job inspector agent. The job inspector suggests a fix but the code fix agent actually takes the suggestion, makes a and in this case it actually opens a PR.

Self-healing pipeline loop

9:43 And I’ll talk about it more. And so this is the loop of a self-healing pipeline. So the GitHub ingest pipeline fails, the job inspector says, “Hey, with high confidence that this file and this line is what’s causing the issue.” The code then writes the fix, and in this case it opens a PR against the repository which holds your pipeline code. And here’s when a human steps in. He either accepts the fix and merge it. And then when you merge it, the GitHub action automatically deploys the fixed pipeline code back onto the DLT Hub platform. And then the next time it’s triggered but manually or by schedule, the pipeline is running as it’s supposed to run. And again, the whole loop here just lives in the manifest file, which I’ll probably touch upon later when I actually show you the code. And then basically this agent uh uh proposes of or writes a fix only when the confidence is really high, and it has a narrow scope that it only edits the file which the job inspector suggested it, and it never has direct access to the code on GitHub. It can only like create PRs for you. And so enough with the explanations, so let’s just take a to get a look at the live demonstration of

Live demo

11:04 All right. So let me All right. So this is my workspace locally. You can install DLT Hub by using the UVX command. So once you install it in installs all the files necessary for you, and I use the DLT Hub AI Harness to build me a GitHub pipeline to ingest the of a pull request. And I deliberately introduced a bug here where I’m using the field ID in the URL. So, this throws a 404 error as we are supposed to use the field numbers. And, I’ve already deployed this pipeline on the platform. All right. So, this is how the DLT Hub platform looks. Um and, uh we have the pipeline here, and I’m just going to trigger this manually. And, this will fail as we’ll see soon. And, by the time I can actually show you how you can actually see the logs, you can go to the run number, and you can see the pipeline just failed. It says, uh resource extraction error.

12:23 And, uh since it failed, you should as you can see, the job inspector was triggered because it’s based on GitHub ingest failing. So, what the job inspector is doing is going through the traces and logs of the failed pipeline, and then it proposes a fix. Um and, the code fix inspector should then take and create and write the fix in a PR, which we’ll see soon. And, so I want to show you where the definition of the background agent exists. At least here in our cloud agent, we have uh the DLT Hub, and agent.md is where the system prompts live. So, you can edit so that your background agent can act however you would like to as well. Um and now going back to the platform, So, you see, uh the code fix is working because it was triggered based on a job inspector run being successful. And, similarly, uh Uh, we can look at the logs, but I think we [clears throat] should soon have uh, this agent uh raising a PR against the repository. So, this is my repository where I have all my same pipelines which I just showed you locally. It is uh uh, remotely on this repository.

13:46 So, as you can see the summary here, it’s it’s like it found the problem. It gives you the fix. And it says it has opened a PR here. So, it says it the resource already pulls number, so it should be using the number. So, if I go to the PR and if I see the files it has changed, it has changed from using the field ID to number. So, all I need to do now is I just merge my merge this pull request because I think it’s given me a good explanation of what is wrong. And so, I’ve set up my GitHub actions in a way where it auto deploys the pipelines. So, this is my YAML file. And the way this auto deploys to the DLTAP platform is in the DLTAP platform in our workspaces, you can actually enable uh API keys. So, using these API keys, you can trigger jobs from your external functions and like say in this example GitHub actions. And uh and even in the environment variables is where I’ve uh uh, you know, specified all my agent API keys, GitHub keys, and agent model. So, if I go back to my workspace, uh let’s see what’s up with the So, once this is Okay, so now uh, my pipeline is deployed. So, if I go back to my workspace, go to my pipelines. So, the last run failed and it’s fixed itself.

15:22 So, I’ll trigger this manually and this uh pipeline should run successfully. All right, so So, as you can see our pipeline run successfully and uh uh this is how we are reaching the level autonomy with our data platform where not only a pipeline can suggest a fix, but also can fix itself with a human in the loop. And uh the next stage is obviously reaching the level four of uh um autonomous autonomy where, you know, an engineer pre-authorizes all the uh issues which you would like to uh fix without human intervention. So, with this, I’d like to uh end my demo. And uh if you have any questions, please feel free to ask. Thank you.

Q&A

16:41 Okay. Questions? Yeah, I have a question. Um so, actually I have two one is um how do you make sure that the code that the agent changes doesn’t do anything bad? And second question is when the code works in a fix environment, let’s say your pipeline is fetching data from an And I’m a malicious provider, and then I send you data with instructions. How do you make sure that these instructions don’t run in your code, not in your runner, or in your pipeline? Uh yeah, the first is to answer your first question. Uh when it raises a PR, you actually either check it manually that the code it’s changing is not breaking anything. And that uh either you with intuition, with experience, you accept the pull request that this code is not going to change is not going to break anything. If not, you can always uh uh pull the code back to your local workspace, test it out, see that it’s working, and you can you deploy back the corrected code yourself as well. So, you can test it locally. And uh uh and what was your second question So, the second question was on a similar line, right?

18:14 Um uh where where is the execution environment, right? So, because agent code needs to run somewhere, and you kind of answered it, right? But not to my full satisfaction, right? Because if you pull code that you haven’t reviewed it and you run it in your local machine, you’re susceptible to supply chain attacks, right? So, that is something I I would need a bit more of an extended answer. Uh yeah, so uh regarding uh the agent like So, the the only thing you can make changes to is the code, like an uh an external person cannot actually make changes to the system prompt, which exists in your local repository. So, the only person who can make changes to how the agent behaves is yourself. Uh and the agent runs in our uh cloud platform, where it’s a Docker container. So, and the only thing like say for malicious you’re trying to maybe steal some credits or use some resources make sure like your compute is maximized so you spend a lot of money. Uh, but we have enough guardrails to stop that to detect like uh, unusual activities which happens so uh, there are guardrails which actually stops from things happening like that.

19:36 Any else? Nobody is going to ask how these things I want to ask questions. There is one question. I have to pull people. I have a question. You mentioned that you already have this in production with some of the companies. But you also say at the same time that you need to run this locally. So how this works in production? Uh, you mean run locally? it’s so which product are you talking about? The the agents and fix pipelines. Oh, the agent the self-healing pipeline so that that we haven’t shipped yet. So I just showed a demo of what’s coming. So the only thing shipped is the agent which suggests a fix. Uh, and that is the job inspector which Yeah, so here it suggests a fix for you like and this is it says hey in line 40 you got to change uh, this thing. So you go back to your workspace. You change the code, you test it and then you deploy it to production. So you can test it against a database like DuckDB. And then one every you can validate your data. We also have skills which does validate validations and data quality checks. And then once you’re happy that like actual the data which you want to see is landing in the destination, you can just change the destination to your production destination and then deploy it Uh, to the DLTR platform. So, we do all your testing locally and only then you deploy it in production on the platform.

21:17 And uh And the DLTR platform isn’t the since this uh DLT is Python code, you don’t really um have to use our platform to deploy. You can also deploy DLT in lambda functions um or anywhere Python runs in a Docker So, it’s like very flexible in that way. Okay. Thank you. Yeah. Thank you. Uh I have a question about pricing. So, is the plan long-term to keep DLT, the library including transformations, including all these agent skills 100% open source and just monetize the platform or will we see something like MinIO where open source slowly devolves into this VC-funded uh paid platform? Yeah. Um I don’t know if you’re the correct person to answer that. No, right now uh DLT has been active for over three, three and a half years and it has always been open source and it’s a way for customers to really find out about DLTR Hub as a product. But, we also have a lot of features which falls behind the pricing model like the AI harness um and all the transformation tool kits, the data quality checks, which are like really important for production grade uh data engineering, which falls under uh our pricing model. So, there needs to be some form of features which we need still need to attract people to our commercial product. Uh and to be honest, a lot of people say DLT itself is such a great product and people do find ways to run it by themselves. Uh so, it’s always a challenge for us to like, you know, also attract them to use DLTR Hub uh with the features we offer as well.

23:14 Thanks for the great presentation. I’m a little bit more interested in the genetic or in the agents and the confidence that they that they assess these things. They either assess the fix, they assess the problems. How do you visualize it or make that transparent to the user or to the next agent? Um yeah, maybe just a little bit on the on the confidence assessment maybe. Yeah, so obviously it gives you a short summary of uh what at least with the job inspector. But you can always trace uh I can’t uh you can always go through the traces of how it arrived at such a uh result or like the suggested fix. And you also have logs and your uh coding agent sessions which always exist on your wherever you’re using. So uh and we have a mechanism by with which uh we rate the confidence of the fix. So and if it’s the confidence is low, obviously we are being transparent that the agent whatever particular agent you’re using. So different agents function different ways to particular types of problems. And uh yeah, uh we have the traces and we provide the confidence if we think it’s a good fix or not. But then ideally you would need to take the suggested fix, run it see if the fix is actually works, and then you deploy it. So we would never suggest anyone to just take the fix as it is and just run it to de- deploy it.

24:50 So you kind of have a proprietary like scoring framework of of confidence that you you apply? I think so, yeah. Behind on how we come up with the confidence level, but I can’t really tell you more about it. Not that I can’t talk about it, but I just don’t know the ins and outs of it, so I don’t really want to say things which I don’t really fully know. All right, thank you. All right. Now we have really no further questions. Okay, yeah. So, questions? All right. So, Shiraz? Yeah, thank you so much for your patience and listening to me. Thank you. See you later.