The Data Ingestion and Transformation Leader will oversee data engineer scrum teams responsible for Infutor’s ingestion of external and customer’s data sources, transformation and cleansing of the data into a consolidated data model, and creation of output data to create customer data products and to feed customer applications. This role will report to the Chief Technology Officer, and be based in the corporate headquarters in Oakbrook Terrace, IL.
- Responsible for the success and management of the US and Costa Rica scrum teams that run legacy processes to ingest, transform and deliver data products to Infutor file install customers.
- Partner with the CTO and lead the evaluation and choice of new innovative Big Data technology, Data Lake approaches and commercial or open source ETL software to create a new future data ingestion platform running in AWS.
- This new ETL/ELT/Ingestion platform will be built by moving industry leading data matching and cleansing logic from the legacy applications into a Cloud Microservices architecture (Lambda/Docker), with all rules stored in a central data repository (not in the code) that can then be leveraged internally by the new platform, and eventually externally by our customers as data cleansing API’s.
- The new platform will feed a consolidated data model (InfutorGraphDB) in AWS (Redshift) as a source of truth for consumer and household identities along with other consumer attributes and intelligence.
- Transformation routines will need to be orchestrated in the new platform to ingest customer and external data sources from the Data Lake, as well as streaming data, and perform Master Data Management (MDM), cleansing and execution of Data Intelligence Microservices to feed the consolidated InfutorGraph DB.
- Transformation routines will also need to be created to create customer install files to be delivered to customers via SFTP and to feed legacy applications to replace the current Q Files process. The goal of the new platform is to eventually replace all current legacy data ingestion processes.
- Ensure the new platform is secure and engineered to meet high quality standards that not only meet business and continuity needs, but are designed to adapt and scale in the long-term.
- Recruit, coach and mentor a world-class, data engineering team based in Costa Rica to build this new platform. Provide clear direction, encourage professional development, and motivate the team to achieve success. Reinforce the cultural principles of Infutor and set the leadership tone for the team
- During the transition to the new platform, oversee continuous improvement of existing legacy data ingestion environments and processes.