Senior Azure Data Engineer
Enable Data Incorporated
- Hyderabad, Telangana
- Permanent
- Full-time
- Design, implement, and maintain cloud-based data pipelines to enable efficient storage, processing, and analysis of large volumes of data.
- Collaborate with data engineers, and other stakeholders to understand requirements and translate them into scalable and reliable data platform architectures.
- Evaluate and select appropriate cloud infrastructure and services to support data storage, processing, and analytics needs, considering factors such as scalability, performance, cost, and security.
- Implement data ingestion pipelines, integrating various data sources and ensuring data quality, integrity, and timeliness.
- Design and optimize data storage solutions, including data lakes, data warehouses, and data marts, to support different types of data processing and analysis.
- Implement data processing workflows using technologies such as Apache Spark, Apache Hadoop, or cloud-native data processing services.
- Develop and maintain data governance and security policies, ensuring compliance with data protection regulations and industry best practices.
- Monitor and optimize the performance of data platforms, identifying bottlenecks and implementing optimizations to improve data processing speed and efficiency.
- Collaborate with DevOps teams to automate deployment, monitoring, and management of data platforms using infrastructure-as-code and CI/CD practices.
- Stay up-to-date with emerging technologies and industry trends in cloud computing, big data processing, and data engineering.
- Provide guidance and mentorship to junior team members, fostering a culture of learning and innovation.
- Bachelor's or Master's degree in computer science, engineering, or a related field.
- 8+ years of proven experience as a Data Engineer, Software Engineer, or similar role, with a focus on building cloud-based data platforms.
- Strong experience in Microsoft Azure
- Proficiency in the latest Big Data tools/technologies like Hive, Hadoop, Yarn, Kafka, Spark Stream.
- Proficiency in data processing frameworks such as Apache Spark, Apache Hadoop, or cloud-native data processing services (Azure Data Lake, Azure Data factory, Azure Databricks, Azure Synapse, Snowflake, CosmosDB)
- Experience with data integration and ETL (Extract, Transform, Load) processes, including tools like Apache Airflow or cloud-native orchestration services.
- Experience with workload and job optimization for cost reduction
- Knowledge of database systems (relational and NoSQL) and data modeling principles.
- Familiarity with data governance, data security, and compliance frameworks.
- Understanding of distributed computing principles and scalable architectures.
- Experience with infrastructure-as-code tools like Terraform
- Excellent problem-solving and troubleshooting skills, with the ability to address complex data platform challenges.
- Strong communication and collaboration skills to work effectively with cross-functional teams