Job ID:
33403
Published date:
22/10/2026
Summary
Job Title-Databricks Data Engineer
Client-Global Consulting Firm
Day Rate-£400-£450 (Outside IR35)
Duration-6 months (extendable)
Location-Manchester or Leeds (Mostly remote with occassional travel to the client site)
- 5-7 years in data engineering, with at least 4 years building production workloads on Databricks.
- Databricks Certified Data Engineer Professional is mandatory.
- A cloud certification (AWS, Azure or GCP) is an added advantage.
- Strong hands-on experience on Databricks and Apache Spark, including reading query plans and the Spark UI.
- Strong hands-on experience on Python, SQL, PySpark and Pandas.
- Solid grasp of ETL/ELT, data warehousing, dimensional modelling and lakehouse design.
- Production experience with both batch and streaming pipelines on large, distributed datasets.
- CDC patterns, SCD Type 2, deduplication and handling late or out-of-order data.
- Strong Delta Lake knowledge, with the ability to design and build reliable, optimised data pipelines that scale efficiently and keep storage and compute costs under control.
- Structured Streaming with Auto Loader and Kafka, watermarks and checkpoint recovery.
- Lakeflow Declarative Pipelines (DLT), Databricks Workflows and Databricks SQL.
- Unity Catalog: grants, row filters, column masks and lineage.
- Performance tuning: partitioning, liquid clustering, caching, broadcast joins, Photon and serverless compute.
- Working experience with a broad range of Databricks native connectors, including Lakeflow Connect managed ingestion (e.g. Salesforce, SQL Server, Workday, ServiceNow), Lakehouse Federation, JDBC/ODBC sources, cloud object storage, Kafka and other streaming sources, and Partner Connect integrations.
- Current knowledge across the breadth of the Databricks product and platform, not only data engineering features.
|
|
|