Job Description
Job Description
We are looking for a passionate certified Data Engineer. The successful candidate will turn data into information, information into insight and insight into business decisions.
Data analyst responsibilities include conducting full lifecycle analysis to include requirements, activities and design. Data analysts will develop analysis and reporting capabilities.
They will also monitor performance and quality control plans to identify improvements.
Primary skillset : Python / Pyspark & Azure, ADF, SQL, SQL Server, Data Warehousing, ETL
Secondary : Databricks
Nice to have : Informatica / ETL.
Responsibilities
- Design and develop ETL processes based on functional and non-functional requirements in python / pyspark within Azure platform
- Understand the full end to end development activities from design to go live for ETL development and Azure platform
Recommend and execute improvements
- Document component design for developers and for broader communication.
- Understand and adopt an Agile (SCRUM like) software development mindset
- Follow established processes / standards, business technology architecture for development, release management and deployment process
- Execute and provide support during testing cycles and post-production deployment, engage in peer code reviews.
- Elicit, analyze, interpret business and data requirements to develop complete business solutions, includes data models (entity relationship diagrams, dimensional data models), ETL and business rules, data life cycle management, governance, lineage, metadata and reporting elements.
- Apply automation and innovation on new and on-going data platforms for those development projects aligned to business or organizational strategies.
- Design, develop and implement reporting platforms (e.g. modeling, ETL, BI framework) and complex ETL frameworks that meet business requirements.
- Deliver business or enterprise data deliverables (that adhere to enterprise frameworks) for various platforms / servers / applications / systems.
Requirements
- Proven working experience as a data engineer
- Bachelor degree or equivalent in Computer Science
- Skilled in Python object-oriented programming
- Skilled in AWS Compute such as EC2, Lambda, Beanstalk, or ECS
- Skilled in AWS Database products such as Neptune, RDS, Redshift, or Aurora
- Skilled in AWS Management and Governance suite of products such as CloudTrail, CloudWatch, or Systems Manager
- Skilled in Amazon Web Services (AWS) offerings, development, and networking platforms
- Skilled in SQL
- Skilled in Jenkins
- Skilled in JSON
- Skilled in discovering patterns in large data sets with the use of relevant software such as Oracle Data Mining or Informatica
- Skilled in cloud technologies and cloud computing
- Experience using software and computer systems' architectural principles to integrate enterprise computer applications such as xMatters, AWS Application Integration, or WebSphere
- Determining causes of operating errors and taking corrective action
- Experience in the process of analyzing data to identify trends or relationships to inform conclusions about the data
- Skilled in creating and managing databases with the use of relevant software such as MySQL, Hadoop, or MongoDB
- Programming including coding, debugging, and using relevant programming languages
- Communication including communicating in writing or verbally, copywriting, planning and distributing communication, etc.
- Ability to frame ideas as systems and analyzing the inputs, outputs, and process
- Experience helping an organization to plan and manage change in effort to meet strategic objectives
- Adept at managing project plans, resources, and people to ensure successful project completion
- Working with people with different functional expertise respectfully and cooperatively to work toward a common goal
Senior Software Engineer ML Data Pipeline
Key Responsibilities:Collaborate with perception experts and roboticists to scale solutions for L3 driving systemsArchitect and implement state-of-the-art solutions in machine learning domainsDevelop cloud services for data management, workflow orchestration, and safety validationRequired Qualificat...
Sr Data Engineer
Any experience in a minimum of two of the following technical disciplines: data warehousing, big data management, analytics development, data science, application programming interfaces (APIs), data integration, cloud, servers and storage, and database mgmt. Analyze, design and develop enterprise da...
Senior Data Analytics Engineer
A company is looking for a Senior Data Analytics Engineer. ...
Data Engineer III
Experience in: SAS Grid, Database: Snowflake, DB2 and Oracle Data Engineering, Data Architecture Methodologies: Agile, ETL and development. The data engineer designs and builds platforms, tools, and solutions that help the bank manage, secure, and generate value from its data. These solutions can in...
Senior Data Software Engineer
A company is looking for a Senior Software Engineer, DataKey Responsibilities:Building Data Platforms for the Research TeamBuilding highly scalable data pipelines on GCPContributing to building the multimedia AI LakehouseRequired Qualifications:5+ years of software engineering experience3+ years of ...
Senior Data Engineer - People & Finance Technology (REMOTE)
Experience developing new and enhancing existing data processing (Data Ingest, Data Transformation, Data Store, Data Management, Data Quality) components. Data processing/data transformation using ETL/ELT tools such as DBT (Data Build Tool), or Databricks. Our Senior Data Engineer is a key member of...
Data Engineer
We are seeking a skilled Data Engineer with a strong background in mortgage banking to join our IT team. This includes optimizing data flow and collection to ultimately support data analysis and decision-making processes. Integrate new data management technologies and software engineering tools into...
Senior Data Platform Engineer
Data Platform Engineer with experience on Building Data platforms of Cloud Data Platform provides platform capabilities to critical data warehousing and reporting systems in an AWS-based environment. Data Engineer with Dev Ops experience. Bachelor of Computer Engineering. The Engineer must be h...
Data Engineer Azure or GCP - Only Full time
Knowledge of Azure Services such as ADF Databricks EventHub Delta Lake Data Lake. Experiencein GitHub Git Action Jira end to end data pipeline. ...
AWS Data Engineer
Extensive experience in Databricks Data engineering Job Runs Data Ingestion and Delta Live Tables. Experienced in data ingestion using various methods API Direct database reads AWS Cross account data copy third party vendor data using FTP process etc. As well as previous experience data architecture...