Data Streaming Engineer

SUMMARY:
-

POSITION INFO:

PBT Group has an opportunity for a Data Streaming Engineer to build, enhance, and maintain real-time data pipeline. Work with various infrastructure and operations teams to maintain data infrastructure. As a senior engineer on the Data Streaming Platform team, you will be responsible for developing and maintaining data ingestion pipelines vital to the continued growth. You will collaborate with many teams to understand data needs and turn them into platforms and services, monitor and maintain the health of the data streams, and have an impact on the data ecosystem.

Duties:

1. Operational Delivery

• Build, enhance, and maintain our real-time data pipeline.
• Work with various infrastructure and operations teams to maintain our data infrastructure.
• Be self-driven in identifying and documenting feature gaps and designing and implementing solutions to them.
• Help build, modernize, and maintain services and tooling to ensure resiliency, fix data discrepancies, and enhance the customer experience.
• Monitor daily execution, diagnose and log issues, and fix pipelines to ensure SLAs are met with internal stakeholders.
• Mentor other engineers and help them grow; code reviews, guidance on best practices, leveraging your experience in the field.

2. Technical Leadership

• Participate in the engineering and other discipline’s community of practice.
• Share AWS knowledge and practical experience with community.
• Challenge and contribute to development of architectural principals and patterns.

3. Compliance

• Ensure solutions adhere to Olympus patterns, guidelines and standards.
• Operate within project environments and participate in continuous improvement efforts.

4. Delivery Management

• Follow and participate in defined ways of work including, but not limited to, sprint planning, backlog grooming, retrospectives, demos and PI planning.

Skills and Experience Required:

• Experience of developing solutions in the cloud
• At least 2-3 years' experience with designing and developing streaming Data Pipelines for Data Ingestion or Transformation using AWS technologies.
• Experience with distributed log systems, such as Kafka and AWS Kinesis
• Must have experience with DataBricks, Hadoop, Spark
• redPanda or Confluence experience advantageous
• Experience in developing data warehouses and data marts
• Experience in Data Vault and Dimensional modelling techniques
• Experience working with automated data warehousing solutions would be advantageous.

Qualifications/ Certification:

• Bachelor’s degree in computer science or similar fields like Information Systems, Big Data, etc.
• AWS Data Engineer Certification would be advantageous.
• Related Technical certifications.

* In order to comply with the POPI Act, for future career opportunities, we require your permission to maintain your personal details on our database. By completing and returning this form you give PBT your consent