Data Engineer (JHB)

Johannesburg, Gauteng
Permanent
Full-time

2 months ago

Our client is a development firm focusing on the advancement of innovative agricultural technologies. They are in search of a proficient and enthusiastic Data Engineer to join their dynamic team in the role of integration specialist. Your duties will involve the design, implementation, and maintenance of the organization’s data infrastructure and pipelines. Close collaboration with data scientists, analysts, and software engineers is essential to ensure smooth and dependable data flows across the entire organization. The ideal candidate possesses a robust foundation in data engineering, exceptional problem-solving abilities, and a keen interest in working with extensive datasets.DUTIES:

Design, develop, and maintain scalable and efficient data pipelines and ETL processes to ingest, transform, and load data from various sources.
Develop and implement data warehousing solutions.
Collaborate with cross-functional teams to integrate various data sources.
Ensure data quality and consistency across different data systems.
Optimise data retrieval for dashboard/reporting solutions.
Optimise data infrastructure, including data storage, data retrieval, and data processing for enhanced performance and scalability.
Implement data quality and data governance processes to ensure accuracy, consistency, and integrity of data.
Monitor and troubleshoot data pipelines to identify and resolve issues in a timely manner.
Perform data profiling and analysis to identify data quality issues and propose improvements.
Collaborate with data scientists and analysts to provide them with the necessary data sets for analysis and reporting.
Stay up to date with emerging technologies and trends in data engineering and recommend new tools and frameworks to improve data infrastructure.
Willingness to actively contribute to BI analytics tasks, such as creating and maintaining reports.
This means being comfortable with hands-on work, including report development, SQL writing, and refactoring tasks.

REQUIREMENTS:Qualification:

Preferably a bachelor's degree in computer science, engineering, or a related field.
Certifications (AWS, GCP, Azure, Microsoft) a plus.

Knowledge, Skills & Experience:

Proven experience as a Data Engineer or similar role, with a strong understanding of data modelling, data warehousing, and ETL processes.
Proficient in SQL and experience working with relational databases (e.g., PostgreSQL, MySQL, SQL Server) and NoSQL databases (e.g., MongoDB, Cassandra).
Strong programming skills in at least one scripting language (e.g., Python) and experience with data manipulation and transformation libraries (e.g., Pandas, PySpark).
Comfortable working with cloud-based infrastructure and services provided by Amazon Web Services (AWS)/ Azure.
Familiarity with data pipeline orchestration tools (e.g., Apache Airflow, Luigi, Lambda) and workflow management systems.
Experience with real-time data streaming technologies (e.g., Apache Kafka, Apache Flink).
Knowledge of containerisation technologies and orchestration tools (e.g., Docker, Kubernetes).
Familiarity with machine learning concepts and frameworks.

Copyright 2016-2024 © Datafin. All Rights Reserved.|Manage Cookie ConsentTo provide the best experiences, we use technologies like cookies to store and/or access device information. Consenting to these technologies will allow us to process data such as browsing behaviour or unique IDs on this site. Not consenting or withdrawing consent, may adversely affect certain features and functions.Functional Functional Always activeThe technical storage or access is strictly necessary for the legitimate purpose of enabling the use of a specific service explicitly requested by the subscriber or user, or for the sole purpose of carrying out the transmission of a communication over an electronic communications network. Preferences PreferencesThe technical storage or access is necessary for the legitimate purpose of storing preferences that are not requested by the subscriber or user. Statistics StatisticsThe technical storage or access that is used exclusively for statistical purposes. The technical storage or access that is used exclusively for anonymous statistical purposes. Without a subpoena, voluntary compliance on the part of your Internet Service Provider, or additional records from a third party, information stored or retrieved for this purpose alone cannot usually be used to identify you. Marketing MarketingThe technical storage or access is required to create user profiles to send advertising, or to track the user on a website or across several websites for similar marketing purposes.

Intelligence Software

Apply Now