Data Engineer

Luxoft Poland
Mid
Online interview
B2B Employment contract
Krakow

Project description

Cloud-based reference data platform, a new Finance Risk And Data Analytics capability, that will provide data mastering and distribution capabilities for various reference data domains including Instruments, Ratings, Book Data, Product Taxonomy, Legal Entity, Industries, Countries, Currencies, etc.

As a service we provide cleansing, enriching and data quality in a centralized place and offering it to different systems, applications, or users, irrespective of where they are in the organization, or on the network.

As a Data Engineer, you will be building big data solutions to solve some of the organization's toughest problems and delivering significant business value. This is an exciting time to join as you will be helping to shape the Reference Data Mastering and Distribution architecture and technology stack within our new cloud-based datalake-house.

Your tasks

  • Model data landscape, obtain data extracts and define secure data exchange approaches
  • Acquire, ingest, and process data from multiple sources and systems into Cloud Data Lake
  • Create data structures optimized for storage and various query patterns
  • Implement pipelines integrating database management systems, cleaning data and improving its data quality
  • Collaborate with others to map data fields to hypotheses and curate, wrangle, and prepare data for use in their advanced analytical models
  • Shape the portfolio of business problems to solve by building detailed knowledge of internal data sources
  • Define, develop and maintain artifacts like technical design or user documentation and look for continuous improvement in software and development process within an agile development team
  • Help architect the strategic advanced analytics technology landscape
  • Build reusable code and data assets
  • Codify best practices, methodology and share knowledge with other engineers
  • Operate in the fast-paced, iterative environment while remaining compliant with bank's Information Sec policies/standards
  • Technology explorations, research & development, deep investigations & trouble-shooting

Who we're looking for?

MUST
  • Hands on experience with at least two of the following technologies: Scala and Spark
  • Experience and interest in Cloud platforms such as Azure (preferred) or AWS
  • Experience in software development, including a clear understanding of data structures, algorithms, software design and core programming concepts
  • Experience in Distributed Processing using Databricks (preferred) or Apache Spark
  • Ability to debug using tools like Ganglia UI, expertise in Optimizing Spark Jobs
  • The ability to work across structured, semi-structured, and unstructured data, extracting information and identifying linkages across disparate data sets
  • Expert in creating data structures optimized for storage and various query patterns for e.g. Parquet and Delta Lake
  • Meaningful experience in at least one database technology such as:
  • Traditional RDBMS (MS SQL Server, Oracle)
  • NoSQL (MongoDB, Cassandra, Neo4J, CosmosDB, Gremlin)
  • Experience in traditional data warehousing / ETL tools (Informatica, Azure Data factory)
  • Ability to clearly communicate complex solutions
  • Proficient at working with large and complex code bases (Github, Gitflow, Fork/Pull Model)
  • Working experience in Agile methodologies (SCRUM, XP, Kanban)
NICE TO HAVE
  • Understanding of Information Security principles to ensure compliant handling and management of data
  • Relevant certifications

Our company

Luxoft Poland

Krakow, Wrocław, Warsaw, Gdańsk 1500
Tech skills
  • Java
  • JavaScript
  • C++
  • C#
  • Python
  • Scala
  • Android
  • Android
  • iOS
  • Go

Check out similar job offers