Seamlessly validate code changes and platform upgrades/migrations in CI, to proactively prevent data incidents & performance degradations.
The challenge today: risky, lengthy, and heavily manual validation of code changes
For data engineers, validating pipeline code changes during development is a complex and time-intensive process. This is especially true for Spark pipelines, where workload complexity and non-linear transformations (and code to physical plan translation) make validation more challenging. Traditional validation methods face several challenges:
- Limited coverage: static code analysis and small-scale data tests provide only partial insights. While they can potentially detect schema, lineage changes, or even unfavorable joins, they fail to uncover critical issues such as data distribution and skewness, execution run-time increases, or inefficient shuffles; and ensure resilience at real-life volumes.
- Manual setup of testing env: simulating pipeline runs in a staging environment is error-prone, resource-intensive, and highly manual. For example, redirecting pipeline reads/writes from/to staging often requires extensive ad-hoc code changes.
- Partial behavior profiling: profiling pipeline behavior across data quality, execution health, and performance requires setting up tens or even hundreds of monitors – an overwhelming task without the proper infrastructure.
- Heavy-effort root-cause analysis (RCA): identifying behavior changes is one thing, but pinpointing their root cause – before deploying the new code – requires an intelligent and contextualized approach. Without coupling lineage, environment configuration, and code insights with monitoring, RCA becomes a tedious, time consuming task.
As a result, code changes are often deployed without proper validation, introducing vulnerabilities that lead to immediate data incidents, degraded performance, and increased costs. High-risk initiatives like platform upgrades and migrations are even more difficult, often taking months to complete and delaying cost savings, engineering efficiencies, and business use-case support.
A new approach: definity's CI/CD testing
To address these challenges, definity is introducing CI/CD Testing: a robust capability built on top of its unique full-stack observability, designed to help data engineers to validate pipeline changes seamlessly, efficiently, and comprehensively:
- Real-data testing in CI: validate pipeline code changes in CI using actual data inputs, to emulate real-life scenarios, ensuring deep, comprehensive coverage at production scale.
- Seamless staging: automatically reconnect pipeline inputs and redirect pipeline outputs to a staging environment with no manual setup or code changes.
- Out-of-the-box profiling: automatically build deep behavior profiles for baseline and release-candidate code versions, across deep data quality, pipeline health, infra performance and resource utilization, deep data+job lineage, environment configurations, and code versions.
- Intelligent RCA: with a fully contextualized intelligent comparative analysis, uncover behavior changes and pinpoint their root-cause in just a few clicks in a unified, intuitive visualization.
The future: accelerated, more reliable deployments
definity's unique CI/CD Testing capability empowers data teams to de-risk deployments and accelerate development cycles while maintaining high standards of data reliability and performance, across 3 key use-cases:
- Ongoing code changes: data engineers can seamlessly validate applicative code changes before deployment, proactively preventing 30-40% production incidents.
- Platform upgrades: data platform teams can holistically validate new Spark/Platform versions, and achieve horizontal upgrades 40-60% faster.
- Platform migrations: data platform and data engineers can easily ensure post-migration parity when moving workloads on-prem to cloud, between cloud providers, or across platforms – saving months of sisyphean testings.
And the best part? Data teams can get started with a central instrumentation in under 30 minutes and be ready for code changes validation in week-1 !
Why it matters
For teams working with Spark at scale, traditional validation methods often fail to capture the complexity of real-world data environments. By testing new code or platform version on real-data out-of-the-box, definity’s CI/CD Testing transforms how data pipelines are developed & validated, shortening dev cycles and reducing risk. Whether it’s a weekly code update, an upgrade to Spark 3.4, or a major platform migration, definity ensures your pipelines are faster, safer, and more reliable.
Ready to learn more?
Check out a short demo video or book a demo today to see how definity can transform your pipeline CI/CD process.