Senior Data Engineer

SUMMARY:
Our client, based in Durban, is looking for a Senior Data Engineer to join their team of highly proficient and experienced Engineers. They have a implemented a hybrid working model with 2 days in the office and 3 days at home. You can also be based either in Durban OR in Somkhele (Richards Bay area).

You will be required to design, plan, setup, configure, support high performance data pipelines for scientific compute resources, working closely with the science team. The science team will have the science experience but not the technical experience and we are looking to bridge that gap. This person will sit within the IT team and provide assistance to the science departments.
 
Likely candidates may have past experience working at universities or other research and science related institutions, or some larger corporate type roles that do similar data heavy computation work.
 
They will be skilled in many niche technical aspects including databases, but this is not just a DBA only role. They must be skilled on the technical side of building the data pipelines which may involve Cloud and/or hardware and software skills and knowledge.

POSITION INFO:
You will help build and maintain scalable data pipelines and related systems in a research focused environment. You will be responsible for designing, developing, testing, and deploying data solutions that meet the business requirements and align with the scientific goals. You will collaborate with research scientists, internal IT and other stakeholders to ensure data quality, reliability, accessibility, security and governance, as follows:

• Design, develop, and maintain end-to-end technical aspects of all data pipelines required to support the research scientists and data managers
• Support ETL processes including, data ingestion, transformation, validation, and integration processes using various tools and frameworks
• Optimize data performance, scalability, and security
• Provide technical guidance and support to data analysts and research scientists.
• Design data integrations and data quality frameworks
• Work and collaborate with the rest of the IT department to help develop the strategy for long term scientific Big Data platform architecture
• Document and effectively communicate data engineering processes and solutions.
• Make use of and help define the right cutting edge technology, processes and tools needed to drive technology within our science and research data management departments.

1. Bachelor's degree or higher in Computer Science, IT, Engineering, Mathematics, or a related field
2. Industry recognized IT related certification and technology qualification such as Databases and Data related certifications.
3. This is a technical role so a strong focus needs to be on technical skills and experience

1. 7+ years’ experience in a Data Engineering, High Performance Computing, Data Warehousing, Big Data Processing
2. Strong experience with high performance computing environments including Unix, Docker, Kubernetes, Hadoop, Kafka, Nifi or Spark or Cloud-based big data processing environments like Amazon Redshift, Google BigQuery and Azure Synapse Analytics
3. At least 5 years’ advanced experience and very strong proficiency in UNIX, Linux, Windows
4. Knowledge of various data related programming, scripting or data engineering tools such as Python, R, Julia, T-SQL, PowerShell, etc.

• Strong Experience working with various relational database technologies like MS SQL, MySQL, PostgreSQL as well as NoSQL databases such as MongoDB, Cassandra etc.
• Experience of Big Data technologies such as Hadoop, Spark and Hive
• Experience with data pipeline tools such as Airflow, Spark, Kafka, or Dataflow
• Experience working with containerization is advantageous
• Experience with data quality and testing tools such as Great Expectations, dbt, or DataGrip is advantageous
• Experience working with Big Data Cloud based (AWS, Azure etc) technologies is advantageous
• Experience with data warehouse and data lake technologies such as BigQuery, Redshift, or Snowflake advantageous
• Strong Experience designing end-to-end data pipelines.
• Strong knowledge of data modeling, architecture, and governance principles
• Strong Linux Administration skills
• Programming skills in various languages advantageous
• Strong data security and compliancy experience
• Excellent communication, collaboration, and problem-solving skills
• Ability to work independently and as part of a cross-functional team
• Interest and enthusiasm for medical scientific research and its applications.