DAGWorks Inc.

DAGWorks Inc.

Standardizing Python Data, ML, & LLM Pipelines + Applications

At DAGWorks our mission is to standardize the way people write python code. Hamilton (https://github.com/dagWorks-Inc/hamilton) standardizes the way individuals and teams express data, machine learning, and LLM pipelines. Burr (https://github.com/DAGWorks-Inc/burr/) standardizes how stateful applications are written, executed, and observed. We want our projects to be 'the way' that enables engineers & data practitioners to get their job done: developing quickly, streamlining productionization and maintenance, agnostic to the infrastructure run underneath. By doing so, we can make these initiatives more human capital efficient, enabling businesses to get more ROI out of existing headcount and infrastructure, while also reducing the amount of tools and platforms that businesses need to maintain. We offer hosted observability, catalog, lineage, provenance, and execution features for our projects. Learn more at www.dagworks.io.

DAGWorks Inc.
Team Size:2
Location:San Francisco
Group Partner:Nicolas Dessaigne

Active Founders

Stefan Krawczyk

A hands-on leader and Silicon Valley veteran, Stefan has spent the last 15 years working on data and machine learning systems at companies like Stitch Fix, Idibon, Nextdoor, and LinkedIn. Before DAGWorks Inc., Stefan led the Model Lifecycle team at Stitch Fix, where its mission was to streamline the model productionization process for over 100+ data scientists. That's when he co-created https://github.com/dagworks-inc/hamilton, which he's excited to continue driving today with DAGWorks Inc.

Stefan Krawczyk
Stefan Krawczyk
DAGWorks Inc.

Elijah ben Izzy

Elijah has spent his career building tools for Data Scientists, Quantitative-minded engineers, and AI researchers. Prior to founding DAGWorks, Inc., Elijah built out Stitch Fix's machine learning platform, which was used and loved by over 100 data scientists. A core component of this was Hamilton (https://github.com/dagworks-inc/hamilton), an open-source framework for managing dataflows that DAGWorks's is now driving forward.

Elijah ben Izzy
Elijah ben Izzy
DAGWorks Inc.

Company Launches

👋 Hi, Stefan & Elijah here from DAGWorks.


(1) At DAGWorks Inc. our goal is to make Machine Learning (ML) initiatives more human capital efficient. (2) Our solution is an open source SaaS platform for Data Science teams to build and maintain model pipelines, plugging into as much of your existing MLOps and data infrastructure as you want. (3) Star our open source project Hamilton on github and get started with it via tryhamilton.dev.

❌ The Problem:

Ask a Data Scientist whether they like inheriting someone else’s ML pipelines. 99% of the time, they’ll say no. Why? Getting ML to production is easy when you’re small, but when your team and codebase start to grow, maintaining these pipelines becomes challenging:

  • 😕 understanding code you didn’t write
  • 🐛 debugging production issues
  • 🏗️ migrating to new infrastructure
  • 🥼 handling personnel changes

This causes teams to spend far more time keeping existing ML pipelines afloat than innovating and driving their business forward.

To get out of/avoid such a mess, you need a high (and expensive) software engineering skill set, but for many that’s not an option. In our experience, this phenomenon is the biggest blocker to leveraging machine learning in your business as it grows.

✨ Our Solution

(1) We leverage our open source project Hamilton, which is an opinionated way to write python code (akin to what DBT did for SQL), that naturally helps teams follow software engineering best practices.

(2) Using the DAGWorks Client on top of Hamilton, we can give you full insight into how your pipelines operate on the DAGWorks Platform, including:

A catalog of functions - feature catalog anyone?

Lineage - see upstream and downstream dependencies:

Debugging insights - e.g. know exactly what code failed given an airflow task failure:

And more like data observability, and links with materialized artifacts (like data sources, models, features, etc.).

(3) The DAGWorks Platform becomes your ML Pipeline control plane. Use it to manage integrations with your existing infrastructure, as well as providing any reporting and compliance insights you need.

⚙️ How it works

With Hamilton, users logically express what should happen, which helps avoid vendor and infrastructure lock in. Then with the DAGWorks Platform & Client, we make your Hamilton code run on your existing infrastructure, providing out of the box connectors to common systems, and provide insights post execution. By building on a solid foundation that clearly decouples concerns, we can avoid the problems that plague traditional ML Pipelines. This enables Data Science teams to move faster and do more themselves, making ML projects more human capital efficient.

👪 Founding Team

Stefan and Elijah worked together at Stitch Fix where they built the self-service ML Platform for over 100 data scientists. They learned firsthand the troubles that occur building and maintaining ML Pipelines over a diverse set of modeling disciplines. Their work has been recognized by industry and included in books, newsletters, and was even the topic of a lecture at Stanford. Previously Stefan built data and ML systems at Nextdoor and LinkedIn and graduated from the Stanford CS Masters program with a Distinction in Research. Elijah built tools for quants at Two Sigma and studied Math and CS at Brown.

🙏 Asks

  1. Start with Hamilton open source today!
    • ⭐️ Star Hamilton on Github! Download it from PyPI.
    • Checkout out www.tryhamilton.dev for an in browser overview of Hamilton. A common use case is to start using it for feature engineering.
    • Don’t know how to leverage Hamilton – book time and we’ll walk you through it!
  2. We’re looking for more leads to larger organizations that have ML Pipeline struggles, in particular if they have compliance/auditing needs for them. Know someone, or perhaps you’re the right person? Please email us - founders@dagworks.io or book 15 minutes here.
  3. We’re still currently in closed beta for the DAGWorks Platform, but you can sign up for early access via www.dagworks.io.
  4. Follow us on Twitter for updates:

Selected answers from DAGWorks Inc.'s original YC application for the W23 Batch

Describe what your company does in 50 characters or less.

Collaborative ML ETL platform for data scientists

What is your company going to make? Please describe your product and what it does or will do.

An open-core platform for data science teams to develop, productionize, and maintain machine learning ETLs, using a company’s existing MLOps & data infrastructure.