DATA ENGINEER
DATA ENGINEER
Job Description
Contribute to the business value of Data-oriented products based on on-premise Data lake or on cloud environments, by implementing end-to-end data processing chains, from ingestion to API exposure and data visualization.
General responsibility: Quality of data transformed in the Data lake, proper functioning of data processing chains and optimization of the use of resources of on-premise or cloud clusters by data processing chains
General skills: Experience in the implementation of end-to-end data processing chains and Big data architectures, mastery of languages and frameworks for the processing of massive data in particular in Streaming Mode (Spark / Scala). Practice agile methods.
Role
You will set up end-to-end data processing chains in cloud environments and in a DevOps culture, you will work on brand new products, for a wide variety of functional areas (Engineering, connected vehicle, Manufacturing, IoT, Commerce, Quality, Finance), with a solid team to support you.
Main responsibilities
- During the definition of the project
- Design of data ingestion chains
- Design of data preparation chains
- Design of basic ML algorithms
- Data product design
- Design of NOSQL data models
- Data visualization design
- Participation in the selection of services / solutions to be used according to usage
- Participation in the development of a data toolbox
During the iterative realization phase
- Implementation of data ingestion chains
- Implementation of data preparation chains
- Implementation of basic ML algorithms
- Implementation of data visualizations
- Use of ML framework
- Implementation of data products
- Exhibition of data products
- Configuration of NOSQL databases
- Distributed processing implementation
- Use of functional languages
- Debugging distributed processing and algorithms
- Identification and cataloging of reusable items
- Contribution to the evolution of work standards
During integration and deployment
- Participation in problem solving
During serial life
- Participation in the monitoring of Operations • Participation in problem solving
Required Skills
- Expertise in the implementation of end-to-end data processing chains
- Mastery of distributed development
- Basic knowledge and interest in the development of ML algorithms
- Knowledge of ingestion frameworks
- Knowledge of Spark and its different modules
- Mastery of Scala
- Knowledge of the use of Solace
- Knowledge of the ecosystem of NOSQL databases
- Knowledge in building data product APIs
- Knowledge of Data visualization tools and libraries
- Ease in debugging Spark and distributed systems
- Popularization of complex systems
- Control of the use of data notebooks
- Expertise in data testing strategies
- Strong problem-solving skills, intelligence, initiative and ability to resist pressure
- Excellent interpersonal skills and great communication skills (ability to go into detail)
- For the future architecture from next year:
- Nice to have knowledge of the GCP ecosystem DataProc, DataFlow, BigQuery, Pub-Sub, PostgreSQL/Composer, Cloud Functions, StackDriver)
- Nice to have knowledge of Beam and its different execution modes on DataFlow
- Nice to have knowledge of Java
Please send your CV and Address Letter (English) to:
Crystal System Group Ltd.
Attn.: HR Department
Str. Jiului 153 RO-013218 Bucharest
Email: talents@crystal-system.eu
Apply now
Please send your resumes at talents@crystal-system.eu