Lead Data Engineer
Motion Recruitment
- Atlanta, GA
- Contract
- Full-time
- 5+ years of hands-on Data Engineering experience with On-Prem and Cloud based Data tools/Platforms,
- 3+ years of experience in data engineering with expertise specifically in Azure Data Factory, Databricks, PySpark, Python and related services.
- Proven experience in leading and managing technical teams in data engineering and analysis.
- Experience in designing and developing data models and schemas.
- Strong proficiency in programming languages such as Python.
- Experience in developing and maintaining CI/CD pipelines for automated deployment and testing.
- Good understanding of cloud computing and its services (e.g., Azure, AWS, GCP).
- Excellent problem-solving skills and ability to work in a fast-paced environment.
- Experience with the some of the following concepts: Real-time & Batch Data Processing, Workload Orchestration, Cloud, Data lakes, Data Security, Networking, Serverless, Testing / Test Automation (Unit, Integration, Performance, etc.), WebServices, DevOps, Logging, Monitoring, and Alerting, Containerization, Encryption / Decryption, Data Masking, Cost & Performance Optimization
- Mentoring (20%) Manage and lead a technical team responsible for data engineering and analysis.
- Focusing on individual's professional development as well as overall team health and technical proficiency on their assigned project tasks.
- Conducts product work reviews with team members.
- Design and implement scalable data solutions on Azure platform using Data Factory, Databricks, PySpark, Python and other related services.
- Build data pipelines and workflows to ingest, transform, and load data from various sources.
- Develop and maintain data models and schemas for efficient data storage and retrieval.
- Develop and maintain CI/CD pipelines for automated deployment and testing of data solutions.
- Lead development and production deployment of analytic Data and BI products (also potentially pilots and proof of concepts), determining appropriate design strategies and methodologies
- Execute data strategies with an understanding of enterprise architecture, consumption patterns, platforms and application infrastructure.
- Collect weekly status updates from Dev teams across multiple Data POD teams and consolidate for leadership project health reporting.