Data Engineer -US Power Markets
ICF International
- New Delhi
- Permanent
- Full-time
1. Data Acquisition and Processing:
- Extract, transform, and load (ETL) data from various sources including ISOs and public regulatory agencies.
- Develop and maintain web scrapers to gather real-time power market analytics.
- Implement RESTful services to fetch data and handle asynchronous tasks efficiently.
- Design and manage data warehouse solutions using Azure Data Lake
- Streamline data processes to ensure fast and reliable access to clean and accurate data.
- Design, build, and optimize ETL jobs leveraging Azure services.
- Integrate with various Azure services to enhance the data pipeline's scalability and resilience.
- Work closely with data SMEs, analysts, and stakeholders to ensure data availability and accuracy.
- Provide documentation and share knowledge on developed pipelines with the team.
- Keep abreast of the latest developments in Azure services and the power markets domain.
- Proactively identify opportunities for process improvements and introduce best practices.
- Bachelor's degree in Computer Science, Data Science, Operations Research, Engineering, or a related field.
- 2+ years of minimum experience in power markets or any other quantitative advisory role
- Familiarity with the Power markets and Energy Policy.
- Proven experience in web scraping, data warehousing, and streamlining data.
- Hands-on experience with RESTful service development and asynchronous functions.
- Prior experience in building and optimizing ETL jobs on cloud platforms.
- Extensive experience with Azure services, including but not limited to AKS, Azure Batch, Azure Data Factory, Azure Data Lake, Azure Functions, and Azure Blob Storage.
- Strong understanding of distributed systems and cloud-based architectures.
- Knowledge and experience with Operations Research techniques such as Linear Programming, Integer Programming, Nonlinear Programming, and other optimization models. Familiarity with tools like CPLEX, Gurobi, or similar optimization solvers is a plus.
- Demonstrated ability to apply OR techniques to real-world problems.
- Excellent problem-solving skills, combined with the ability to work in fast-paced environments.