Senior Data Engineer

The data engineering team is building Paradigm’s infrastructure for clinical care data. We ingest data from multiple data sources: Electronic Health Records, Lab providers, Health Information Exchanges (and more). We enrich, harmonize and make it available for multiple consumers and product verticals. As our population, research and health system footprint grows we will scale our architecture to support parallel data processing. 

ML & Data Science teams will use data to optimize and develop new classification models. Data Scientists and Analysts for defining better patient matching heuristics, answering trial & research site feasibility queries, and ultimately surface insights for clinical trial opportunities. 

We work with application consumer teams to optimize data for our Clinical Trial Matching workflows (matching patients to potential trials) and Study Conduct (capturing study data once a trial has started). 

As a Senior Engineer on the Data team, you’ll partner closely with product and data consumers to improve the breadth, quality and completeness of our data. You will own critical system components, designing them for performance, correctness and reliability. You will work on capabilities such as data lineage, data derivation algorithms, parallel processing, automated monitoring and data quality, data warehouse architecture and data modeling. 

What you’ll do:

• Partner closely with product managers, data scientists and analysts, and engineering colleagues to distill complex business problems into elegant technical solutions
• Build and support robust batch and streaming pipelines
• Build tools and processes to make data accessible across products and data consumers
• Evolve the maturity of our monitoring systems and processes to improve visibility and failure detection in our infrastructure
• Become a subject matter expert on our different datasets and tooling
• Work with privacy & security to ensure we govern data responsibly

What you’ll bring:

• 5+ years of experience in data engineering
• You have experience driving technical projects, prioritizing work, identifying dependencies, facilitating technical decisions and cross-functional team discussions
• Depth of expertise in SQL and data analysis skills
• System design and data workflow optimization
• Coding in Python or other languages
• Experience with ETL orchestration tools like Apache Airflow and Dagster
• Data transformation tools like dbt or dataform
• Experience with data warehouse DBs (such as BigQuery, Redshift, Snowflake, etc…)
• Experience designing and implementing systems that are scalable, available, maintainable, and performant

Nice to have, but not required

• Experience in the healthcare data domain
• Experience with Epic Systems