Senior Data Engineer

SUMMARY:
Senior Data Engineer who will be responsible for implementation and improvements

POSITION INFO:
Responsibilities:

• Identify and Implement Process Improvements: Automate manual tasks, optimize data delivery, and enhance technology scalability.
• Enhance Data Systems: Implement new features and improvements across data platforms.
• Streamline CI/CD Processes: Optimize data pipelines within continuous integration and delivery workflows.
• Manage Complex Data Sets: Assemble large, complex datasets to meet business and technical requirements.
• Optimize ETL Processes: Ensure efficient data extraction, transformation, and loading.
• Conduct Testing for Data Consistency: Develop and execute unit tests to maintain data accuracy across pipelines.
• Automate Monitoring: Build and maintain automated solutions for tracking data pipeline performance and health.
• Support Analytics Infrastructure: Maintain and improve reporting and analytics systems.
• Ensure Data Quality and Governance: Uphold data governance standards and maintain infrastructure (AWS, database security, etc.).
• Manage Metadata and Documentation: Maintain the data warehouse, data lake metadata, catalogs, and user documentation.
• Ensure Best Practices in Database Management: Maintain optimal configurations, including collations, engines, and indexing strategies.

• 5+ years experience as a Data Engineer
• BSc degree in Computer Science / Information system / Engineering or related or equivalent experience
• Atleast 2 years experience builidng and optimising 'big data' data pipelines, architecture and maintaining data sets
• Experience in Python, SQL, AWS, PySpark (desirable), AWS, Airflow, Docker
• Proficiency in manipulating, processing and architecting large disconnected data sets for analytical requirements