Lead Data Engineer

The Role: 

Integrate’s data is treated as a critical corporate asset and is seen as a competitive advantage in our business. As a Lead Data Engineer you will be working in one of the world's largest cloud-based data lakes. You should be skilled in the architecture of data warehouse solutions for the Enterprise using multiple platforms (EMR, RDBMS, Columnar, Cloud, Snowflake). You should have extensive experience in the design, creation, management, and business use of extremely large datasets. You should have excellent business and communication skills to be able to work with business owners to develop and define key business questions, and to build data sets that answer those questions. Above all you should be passionate about working with huge data sets and someone who loves to bring datasets together to answer business questions and drive change. 

Responsibilities: 

• Design and develop workflows, programs, and ETL to support data ingestion, curation, and provisioning of fragmented data for Data Analytics, Product Analytics and AI. 
• Work closely with Data Scientists, Software Engineers, Product Managers, Product Analysts and other key stakeholders to gather and define requirements for Integrate’s data needs. 
• Use Scala, SQL Snowflake, and BI tools to deliver data to customers.  
• Understand MongoDB/PostgreSQL and transactional data workflows.  
• Design data models and build data architecture that enables reporting, analytics, advanced AI/ML and Generative AI solutions.  
• Develop an understanding of the data and build business acumen.  
• Develop and maintain Datawarehouse and Datamart in the cloud using Snowflake.  
• Create reporting dashboards for internal and client stakeholders.  
• Understand the business use cases and customer value behind large sets of data and develop meaningful analytic solutions.  

 Basic Qualifications:  

• Advanced degree in Statistics, Computer Science or related technical/scientific field. 
• 7+ years experience in a Data Engineer development role.  
• Advanced knowledge of SQL, Python, and data processing workflow.  
• Nice to have Spark/Scala, MLFlow, and AWS experience.  
• Strong experience and advanced technical skills writing APIs. 
• Extensive knowledge of Data Warehousing, ETL and BI architectures, concepts, and frameworks. And also strong in metadata definition, data migration and integration with emphasis on both high end OLTP and business Intelligence solutions.  
• Develop complex Stored procedure and queries to provide to the application along with reporting solutions too. 
• Optimize slow-running queries and optimize query performance. 
• Create optimized queries and data migration scripts 
• Leadership skillsets to mentor and train junior team members and stakeholders.  
• Capable of creating long-term and short-term data architecture vision and tactical roadmap to achieve the data architecture vision beginning from the current state 
• Strong data management abilities (i.e., understanding data reconciliations). 
• Capable of facilitating data discovery sessions involving business subject matter experts. 
• Strong communication/partnership skills to gain the trust of stakeholders. 
• Knowledge of professional software engineering practices & best practices for the full software development lifecycle, including coding standards, code reviews, source control management, build processes, testing, and operations. 

Preferred Qualifications: 

• Industry experience as a Data Engineer or related specialty (e.g., Software Engineer, Business Intelligence Engineer, Data Scientist) with a track record of manipulating, processing, and extracting value from large datasets. 
•  Experience building data products incrementally and integrating and managing datasets from multiple sources. 
•  Query performance tuning skills using Unix profiling tools and SQL 
•  Experience leading large-scale data warehousing and analytics projects, including using AWS technologies – Snowflake, Redshift, S3, EC2, Data-pipeline and other big data technologies.