Big Data Engineer
NorthBay Solutions
- Lahore, Punjab
- Permanent
- Full-time
NorthBay is looking for a Big Data Engineers with 2 to 3+ years of experience to join our team. The ideal candidate will have a proven track record in data engineering, with a focus on modern cloud-based solutions and a strong proficiency in SQL. This role requires expertise in designing, implementing, and maintaining data pipelines for both batch and streaming data processing, as well as the ability to troubleshoot data issues effectively. The Data Engineer will play a key role in supporting our data infrastructure and driving innovation in data processing and analytics.Location: Lahore, Karachi, Islamabad
Work Scope: HybridKey Responsibilities:
- Design, implement, and maintain ETL/ELT processes for batch and streaming data processing.
- Develop and optimize data pipelines using modern cloud data warehousing and data lake solutions such as Snowflake, BigQuery, Redshift, and Azure Synapse.
- Investigate and troubleshoot data issues, providing short-term fixes and proposing long-term solutions to prevent recurrence.
- Utilize AWS services including S3, DMS, Glue, and Athena to build scalable and reliable data solutions.
- Collaborate with cross-functional teams to understand data requirements and develop solutions that meet business needs.
- Employ data transformation tools such as dbt to efficiently transform and model data for analytics and reporting purposes.
- Bachelor's degree in Computer Science, Engineering, or related field.
- 3+ years of experience in data engineering.
- Proficiency in SQL.
- Experience with modern cloud data warehousing and data lake solutions.
- Excellent ability to investigate and troubleshoot data issues, providing fixes and proposing both short and long-term solutions.
- Knowledge of AWS services (like S3, DMS, Glue, Athena, etc.).
- Familiarity with dbt or other data transformation tools.
- Proficiency in Python programming language.
- Familiarity with infrastructure as code tools such as Terraform and Terragrunt.
- Experience with orchestration tools like Dagster, Airflow, and AWS Step Functions.
- Knowledge of pub-sub, queuing, and streaming frameworks such as AWS Kinesis, Kafka, SQS, and SNS.
- Understanding of CI/CD pipelines and automation practices.