Purpose

The Big Data Engineer will execute master data management policies developed by the data architect and perform the data quality evaluations. He/She is required to work closely with business representatives to improve the quality of data to the required levels.

Responsibilities

1. Responsible for the integration of large, structured and unstructured data volumes into the cloud platforms
2. Development of scalable end-to-end data pipelines for batch and stream processing
3. Execution of the datalake integration workflow and activities for populating the data lake and integrating diverse data sources
4. Execution an further development of the physical implementation of the logical data model into a physical implementation in the data lake
5. Implementation of solutions for reference data and master data management within the context of the mobility data business
6. Execution of data quality measurements and implementation of data quality improvement activities to the required levels of data quality
7. Support of build-up and maintenance of a data directory for all data relevant to the mobility data business
8. Representation of the Data Architecture team in selected data architecture, data modelling, and metadata management work teams inside Mobility

Requirements

1. Diploma or University degree in an appropriate area (e.g. informatics)
2. Relevant work experience
3. Experience with data ingestion tools like Nifi, Streamsets, Node Red
4. Experience with modern big data technologies like Hadoop, MapReduce, Kafka, Hive, Presto, Spark, Storm
5. Experience with cloud solutions like AWS, Azure
6. Experience with programming languages like R, SQL, Scala, Python, Java
7. Experience with NoSql and traditional databases like Mongodb, MSSQL, Hadoop
8. Experience with enterprise application integration and with approaches in one of the leading tool suites (e.g. Kibana, Solr, ElasticSearch, R, Python)
9. Strong technical design and analysis skill
10. Creativity and lateral thinking
11. Ability to deal with ambiguity and work in fast paced environment
12. Deep experience supporting mission critical applications quickly
13. Excellent communication skills, both through written and verbal channels
14. Excellent collaboration skills to work with multiple teams in the organization
15. Ability to understand and adapt to changing business priorities and technology advancements
16. Strong knowledge and technology trends in implementing of Big data ecosystem
17. Strategic thinking and critical problem solving skills