Revamping Sifflet's dbt Integration: A Unified, Smarter Experience

At Sifflet, we aim to empower you with end-to-end full-stack data observability. This makes integrating with data transformation tools a core part of our vision. With dbt being a cornerstone of many modern data stacks, we're constantly exploring ways to better support our customers who rely on it.

Today, we’re thrilled to announce a major update to our dbt integration. This revamp combines dbt models with the datasets they generate, streamlines lineage, and brings new dbt metadata to Sifflet — all in one seamless experience. The new features are designed to simplify workflows, reduce redundancies, and give you even more powerful dbt insights. Let’s dive into what’s new.

dbt Models + Datasets: A Unified Asset Experience

Previously, dbt models and the datasets they generated existed as separate entities in Sifflet, with distinct catalog entries and asset pages. With this release, we’ve combined them into a single asset and brought new dbt metadata to Sifflet. Here’s what this means for you:

In the Catalog: Enhanced Metadata for dbt-Generated Datasets

Datasets created by dbt now include key dbt metadata:

• Last Execution Timestamp: See exactly when the corresponding dbt model was last run.
• Last Execution Status: Quickly identify whether the most recent dbt model run succeeded, failed, or was skipped.

This enriched metadata is readily available throughout Sifflet (in the catalog, asset page, and even the lineage), giving you key information about your dbt models at a glance.

On the Asset Page: Introducing the New dbt Tab

The dataset’s asset page now features a dedicated dbt tab, consolidating previously scattered dbt information in one place. This tab will include insights like the model's group, its access modifier, and custom dbt metadata defined using the meta field - giving you all the dbt-related context in one place.

Streamlined Lineage Graph: Fewer Nodes, More Insights

Navigating lineage graphs can be challenging when there’s unnecessary complexity. In this update, we’ve streamlined the Sifflet lineage graph by merging dbt nodes with dataset nodes.

• What’s Changed: dbt models no longer appear as separate nodes in the graph. Instead, their metadata (including the model status) is now integrated into the dataset node itself.
• The Result: A cleaner, more intuitive lineage graph that eliminates redundancies while adding valuable context.

This update ensures that you can focus on the insights that matter without getting lost in a sea of nodes.

It is now easier than ever to navigate Sifflet's lineage graph to understand how data flows within your stack (and immediately identify issues) while also benefitting from built-in insights.

Looking Ahead: More dbt Features on the Horizon

This is just the first phase of our dbt integration revamp. Here’s a sneak peek at what’s coming in the next few weeks:

• Cost & Performance Monitoring (pictured below): Gain insights into the resource usage and efficiency of dbt runs via a dedicated tab.
• Leveraging Custom Metadata: Use dbt’s custom metadata directly in Sifflet for advanced configurations.
• dbt-based Monitor Setup: Define and configure Sifflet monitors directly within your dbt YAML files.

We’re excited about this leap forward and hope you are too. Want to see it in action? Reach out to our team to learn more!

‍