How Adaptavist Revolutionized Their Data Stack with End-to-End Lineage

Adaptavist is a London-based technology company that specializes in software solutions and consulting services. They help organizations implement, customize, and optimize Atlassian tools, improve software development, project management and collaboration processes. When it approached Sifflet, data engineering teams were working with a big, bloated stack and mono repo that was causing a lot of issues. On top of that, their plans to move to a poly repo faltered as DBT wouldn’t let them draw lineage between different repos.

Adaptavist needed to overhaul its data architecture and improve visibility into data lineage.

Adaptavist approached Sifflet to help with a key proof-of-concept project that involved changing data architecture to implement a poly repo. The team was also eager to put lineage solutions into place in order to better understand which data were feeding into dashboards, how it was being transformed, and generally benefit from better overall visibility into their data stack. 

Sifflet provided Adaptavist with a fast and effective solution, enabling seamless data lineage tracking and significantly reducing debugging time.

Sifflet’s data lineage solutions allowed Adaptavist to prove out their poly repo POC, and in record time.

"End-to-End Data Lineage in Just Three Days with Sifflet" "It took me about three days to set up the entire tool, onboard people, and set up lineage between all our tools. One piece where we couldn’t draw lineage was all the way up to S3 and the stored procedures and stages. Sifflet’s universal adapter and declarative lineage was super helpful. We built a mapping table ourselves, then parsed it into declared assets and declared lineages, then sent to Sifflet’s API to display these. Now we have lineage from S3 all the way through our stack."

Callum O'Connor, Senior Analytics Engineer

‍

Thanks to lineage, the team has massively reduced the amount of time it takes to debug an issue. Before, debugging dashboards would take up to two days and require finding the source system and dealing with nested CTEs. With Sifflet, Callum’s teams can simply start the report and work backwards to find the failure monitor, dive into one piece of code, and figure it out. 

An evolution which has saved hundreds of hours of engineering time, bringing root cause analysis down from days to hours.

Sifflet drastically reduced testing, monitoring, and refactoring time, saving Adaptavist hours of engineering effort.

Rapid implementation of data lineage which as saved hours of incident resolution time within the data engineering team, and implementation of new use cases for observability, such as monitoring at scale with data-quality-as-code.

Monitoring and Testing

+94% gain in testing efficiency. Daily testing runs now take 10 minutes instead of hours thanks to a shift from python tests to dbt tests & Sifflet monitors. 

‍Data Lineage

+92% efficiency gain in refactoring projects. Takes weeks now instead of months thanks to Sifflet’s ability to see what downstream actions needed to be done before decommissioning a repo.