Job Description As a Lead Data Engineer specialising in Databricks, you will design, build, and optimise data pipeline solutions on Databricks and related cloud platforms. Working closely with data scientists, analysts, and engineers, you will ensure our data infrastructure supports advanced analytics and business insights across industries (including energy, resources, and mining). You will join a collaborative, agile team where continuous improvement, innovation, and knowledge sharing are part of the culture. Work closely with data scientists, analysts, and other stakeholders to understand data requirements and deliver effective pipeline solutions. Contribute to Data Architecture and Solution Design, helping to build Proof of Concepts. Design, develop, and maintain robust ETL/ELT pipelines on using Databricks along with AWS / Azure / GCP tools and services, to ingest, process, and transform large datasets. Implement data validation, cleansing, and governance procedures to guarantee data quality, integrity, and security. This includes enforcing data standards and addressing data quality issues proactively. Continuously improve the scalability, efficiency, and cost-effectiveness of data pipelines. Identify opportunities to enhance performance, reliability, and cost-efficiency across our data systems. Monitor data pipeline performance and promptly troubleshoot any issues or failures to ensure high data availability and consistency. Leverage observability tools and best practices to maintain reliable pipelines. Develop streaming or event-driven data processes as needed for real-time analytics, leveraging frameworks like Apache Kafka and Spark Structured Streaming. Maintain clear documentation of data pipelines, data models, and processes for transparency and team knowledge sharing. Follow best practices in coding, testing, and version control to ensure maintainable and auditable workflows.
