Senior Data Engineer - DatabricksDescription -The HP Ecommerce CRM enablement team is seeking for its CDP program a proven and hands-on Databricks process architect and engineering leader to drive our Databricks transformation, pipelines architecture, governance and engineering patterns. Our CDP delivers a unified Customer data model across our global CRM operations. In this role, you will be responsible for driving our Databricks migration, designing and executing a comprehensive pipeline architecture strategy that aligns with CDP team's skillsets, business objectives and drives compliant, efficient and effective data management.ResponsibilitiesDrive our AWS Databricks transformation from current relational DB setup.Collaborate with our diverse teams, SQL developers/data engineers/CRM analysts, to develop, implement and maintain sustainable, high-performance, growth-ready data-processing and data integration systemsProvide best practice on CI/CD techniques with Git and DatabricksProvide best practice on Databricks cluster management.Reviews and evaluates designs and project activities for compliance with privacy, architecture, security and quality guidelines and standards; provides tangible feedback to improve product quality and mitigate failure risk.Design and maintain an appropriate metadata repository to enable the organization to understand its data assets.Builds and manages relationships throughout the organization.Drives innovation and integration of new technologies into projects and activities in the big data space.Collaborates and communicates with project team regarding project progress and issue resolution.Provides guidance and mentoring to less experienced staff members.RequirementsBachelor's or Master's degree in Computer Science, Information Systems, Engineering, or equivalent.10+ years of experience in data architecture, database design, and data management.Expert in AWS DatabricksExpert in SQL & relational database systems (PostgreSQL preferred)Proficient knowledge in working with distributed data processing frameworks like Apache Spark.Hands-on experience in Python and PandasExperience in overall architecture of big data systems within AWSExperience in Global Companies a great advantage.Knowledge & SkillsExtensive experience with designing data engineering solutions to manage data streams in complex, distributed and parallel systems.Excellent analytical and problem-solving skills. Ability to collaborate with and analyze source systems to drive data profiling/cataloging. Can present clear insights to colleagues to support the end use of the data.Extensive experience with data engineering pipelines, frameworks to mine, cleanse and explore data.Strong problem management. Initiating and monitoring actions to investigate patterns and trends to resolve problems. Determining preventative measures.Evaluating forms and processes for database architecture testing and methodology, including writing and execution of test plans, debugging, and testing scripts and tools.Excellent written and verbal communication skills; mastery in English and local language.ComplexityApplies advanced subject matter knowledge to solve complex business issues and is regarded as a subject matter expert.Frequently contributes to the development of new ideas and methods.Works on complex problems where analysis of situations or data requires an indepth evaluation of multiple factors.Acts as an expert providing direction and guidance to process improvements and establishing policies.Exercises significant independent judgment within broadly defined policies and practices to determine best method for accomplishing work and achieving objectives.Provide mentoring and guidance to lower level employees.Job - SoftwareSchedule - Full timeShift - No shift premium (Spain)Travel -Relocation -Equal Opportunity Employer (EEO) -HP, Inc. provides equal employment opportunity to all employees and prospective employees, without regard to race, color, religion, sex, national origin, ancestry, citizenship, sexual orientation, age, disability, or status as a protected veteran, marital status, familial status, physical or mental disability, medical condition, pregnancy, genetic predisposition or carrier status, uniformed service status, political affiliation or any other characteristic protected by applicable national, federal, state, and local law(s).Please be assured that you will not be subject to any adverse treatment if you choose to disclose the information requested. This information is provided voluntarily. The information obtained will be kept in strict confidence.If you’d like more information about HP’s or your EEO rights as an applicant under the law, please click here:
