Databricks Data Engineer

SUMMARY:
-

POSITION INFO:

PBT Group has an opportunity for a skilled Senior Data Engineer with expertise in Databricks and a strong background in the financial services industry. This role will take ownership of data engineering within the IT department, specifically focusing on managing Databricks clusters and designing efficient data processing pipelines. As a Senior Data Engineer, you will play a crucial role in ensuring the reliability, scalability, and performance of our data infrastructure.

Skills and Experience Required:

• Proven experience as a Databricks Engineer, with a deep understanding of Apache Spark and the Databricks platform.
• Extensive knowledge of data processing concepts, ETL pipelines, and data warehousing principles.
• Hands-on experience in managing Databricks clusters, including provisioning, configuration, and optimization.
• Proficiency in programming languages commonly used with Databricks, such as Python or Scala.
• Familiarity with cloud platforms like AWS, Azure, or GCP, and a track record of deploying and managing Databricks in a cloud environment.
• Strong understanding of distributed computing concepts and parallel data processing.
• Solid knowledge of SQL and experience working with relational databases.
• Strong problem-solving skills, with the ability to troubleshoot and resolve issues related to data processing pipelines and cluster management.
• Excellent communication and collaboration skills, enabling effective teamwork in a cross-functional environment.
• Ability to thrive in a fast-paced, dynamic work environment and handle multiple priorities simultaneously.
• At least 5+ years of experience designing, developing, deploying, and supporting data pipelines.
• Minimum of 3 years of hands-on experience designing, developing, deploying, and supporting data pipelines in Databricks.
• Experience with machine learning frameworks and data science workflows on Databricks (preferred).
• Knowledge of big data technologies such as Hadoop, Hive, or Kafka (preferred).

Qualifications/ Certification:

• Bachelors degree in Computer Science, Engineering, or a related field. A relevant combination of education and experience will also be considered.
• Databricks Certified Developer or Databricks Certified Administrator certification (preferred).

* In order to comply with the POPI Act, for future career opportunities, we require your permission to maintain your personal details on our database. By applying for the vacancy via this job ad, you give PBT your consent.

NB! This job is now closed. You can apply for other jobs by uploading your CV.