Data Engineer
EQPlus Technologies
- Johannesburg, Gauteng
- Permanent
- Full-time
- Responsible for solution design and development of Various Functionality in AWS for the project flow
- Develop and maintain automated ETL pipelines (with monitoring) using scripting languages such as Python, SQL and AWS services such as S3, Glue, Lambda, SNS, Redshift, SQS, KMS
- Develop Glue Jobs for batch data processing and created Glue Catalog for metadata sync up
- Develop Data Pipelines using AWS lambda, Step Functions for data processing
- Manage data coming from different sources
- Experienced with AWS services to smoothly manage application in the cloud and creating or modifying instances.
- Implement with Scaled Agile Framework (SAFe)
- Be involved with the performance and optimization of the existing algorithms in Hadoop using Spark Context, Spark
- Be involved in creating Hive Tables, loading with data and writing Hive queries
- Degree (Information Technology)
- 6 -7 Experience in the same role
- Talend
- SSIS, SSAS, and SSRS
- Clover ETL
- Strong SQL Background
- AWS
- Jupiter Notebook
- Data Warehousing
- Cloud Warehousing