Minimum years of experience : - years
Job Details :
Must Have Skills : Databricks, Spark, Python
Nice to have skills : Hadoop, Hive
Detailed Job Description
- years of experience in data engineering.
- Design, develop, and maintain scalable and robust data pipelines on Databricks.
- Optimize and troubleshoot existing data pipelines for performance and reliability.
- Proficiency with Databricks and Spark.
- Implement and optimize Spark jobs, data transformations, and data processing workflows in Databricks.
- Strong SQL skills and experience with relational databases.
- Experience with big data technologies.
- Ensure data quality and integrity across various data sources.
- Implement data security and compliance best practices.
- Monitor data pipeline performance and conduct necessary maintenance and updates.
- Document data pipeline processes and technical specifications.