Location:
Remote, US (Pacific, Mountain or Central time zone preferred)
Schedule:
9:00am - 6:00pm, flexible
Duration:
4-month W2 contract
Pay Rate:$87.00 - $91.00/hr (W2)
Position Summary
Our client is a vehicle technology company. Its Core Data team builds and scales the data platform and analytics capabilities that power decision-making across the business. Within Core Data, the Vehicle Catalog team owns the vehicle domain, ingesting, curating and organizing vehicle data for analytics, ML and BI use cases relied on by hundreds of stakeholders across dozens of domain teams.
The team is migrating the Vehicle Catalog to a new Databricks account while building out a new vehicle catalog on that platform. As Senior Data Engineer, you will help refactor and move the existing ETL/ELT and analytics pipelines into the new environment, migrate the underlying data and validate that everything lands correctly.
This is hands-on migration work. You will work alongside the client's engineers on well-defined pipelines with clear acceptance criteria, and pick up related migration requests, including dashboard and reporting migrations, as they come up.
Essential Functions and Responsibilities
- Work with a Staff Data Engineer to understand the migration plan, sequence your work against it and provide regular progress updates.
- Refactor existing ETL/ELT code written in PySpark, DABs and dbt from the current Databricks environment to meet the patterns and standards of the new account.
- Migrate data pipelines and their underlying datasets, including backfills, and validate parity between source and target.
- Build and run reconciliation checks (row counts, schema conformance, freshness, completeness) to confirm migrated data matches the legacy environment.
- Migrate dashboards and downstream reporting assets to point at the new datasets, coordinating with the analysts who own them.
- Update orchestration, scheduling, alerting and runbooks so migrated pipelines are supportable after cutover.
- Troubleshoot pipeline failures and data discrepancies during and after cutover.
- Document what was migrated, what changed and anything the team needs to know to own it going forward.
- Take on adjacent migration and data engineering requests as the program evolves.
Education/Experience
- Bachelor's or Master's degree in Computer Science, Engineering, Mathematics or a related field, or equivalent practical experience.
- 6+ years of experience in Data Engineering, Analytics Engineering or Software Engineering working with production data systems.
- Hands-on experience building ELT pipelines on modern cloud data platforms (Databricks experience required).
- Prior data or platform migration experience.
- Experience designing and operating production-grade data pipelines with monitoring and observability.
Knowledge/Skills
- Strong expertise in SQL and Python for building scalable data pipelines and transformations.
- Deep experience with dbt, including model development, testing, documentation and CI/CD integration.
- Ability to build robust testing frameworks for data migration (completeness, quality, etc.).
- Strong familiarity with AI tools such as Claude Code or Devin in the software development process.
- Strong collaboration skills and ability to partner with engineering teams, analysts and operational stakeholders.
- Nice to have: familiarity with Fivetran connector management and ingestion architecture.
- Nice to have: experience with data contracts and schema governance.
- Nice to have: knowledge of streaming or near-real-time data pipelines.
...