Job Title-Databricks Data Engineer

Client-Global Consulting Firm

Day Rate-£400-£450 (Outside IR35)

Duration-6 months (extendable)

Location-Manchester or Leeds (Mostly remote with occassional travel to the client site)

 

• 5-7 years in data engineering, with at least 4 years building production workloads on Databricks.
• Databricks Certified Data Engineer Professional is mandatory.
• A cloud certification (AWS, Azure or GCP) is an added advantage.
• Strong hands-on experience on Databricks and Apache Spark, including reading query plans and the Spark UI.
• Strong hands-on experience on Python, SQL, PySpark and Pandas.
• Solid grasp of ETL/ELT, data warehousing, dimensional modelling and lakehouse design.
• Production experience with both batch and streaming pipelines on large, distributed datasets.
• CDC patterns, SCD Type 2, deduplication and handling late or out-of-order data.
• Strong Delta Lake knowledge, with the ability to design and build reliable, optimised data pipelines that scale efficiently and keep storage and compute costs under control.
• Structured Streaming with Auto Loader and Kafka, watermarks and checkpoint recovery.
• Lakeflow Declarative Pipelines (DLT), Databricks Workflows and Databricks SQL.
• Unity Catalog: grants, row filters, column masks and lineage.
• Performance tuning: partitioning, liquid clustering, caching, broadcast joins, Photon and serverless compute.
• Working experience with a broad range of Databricks native connectors, including Lakeflow Connect managed ingestion (e.g. Salesforce, SQL Server, Workday, ServiceNow), Lakehouse Federation, JDBC/ODBC sources, cloud object storage, Kafka and other streaming sources, and Partner Connect integrations.
• Current knowledge across the breadth of the Databricks product and platform, not only data engineering features.