Senior Staff Software Engineer, Data Platform
Udemy
- Chennai, Tamil Nadu
- Permanent
- Full-time
- Design, build, enhance, and support self-service data infrastructure as a platform, which would be used for Udemy's data mesh.
- Build robust, scalable, cloud-native, cost-optimized real-time and batch data integration pipelines in AWS with Airflow, Kafka, Databricks, Delta Lake, AWS services like S3, EMR, Athena, RDS, Redshift, EKS, etc., and Snowflake.
- Contribute towards Udemy's comprehensive data quality initiatives like data contracts, data validation, observability, and alerting
- Support data platform privacy with RBAC and encryption
- On-call rotation business hours
- 8+ years of experience in building distributed systems with Java, Scala, or Python
- 5+ years of experience managing big data platforms with storage (AWS S3), compute (AWS EMR or Databricks with Spark, AWS Athena or Presto), data warehouses (Redshift or Snowflake), streaming (Kafka or AWS Kinesis)
- 5+ years of experience in agile development methodologies & test-driven development (TDD)
- 5+ years of experience building and maintaining resilient real-time & batch data pipelines.
- Experience in building or using any one of the data quality tools (in-house built, SODA SQL, Anomalo, Monte Carlo, Great Expectations, or any other)
- Expertise in cloud data privacy tools in the area of authentication (AWS IAM/OKTA), encryption (AWS KMS/Safenet), authorization (Apache Ranger/Privacera)
- Strong grasp of object-oriented design patterns, algorithms, and data structures