Sr. Data Engineer
New York - NY Hybrid
- Job Description : The ideal candidate will have a strong background in data engineering and development, with expertise in various data skills and tools. The role involves working on complex data projects, collaborating with cross-functional teams, and ensuring the efficient processing and management of data.
Key Responsibilities :
Develop and maintain data pipelines using Python and Pyspark.Design and implement efficient SQL queries for data extraction and manipulation.Work with Aurora Postgres and Redshift for data storage and management.Utilize Informatica or other ETL tools for data integration and transformation.Implement and manage AWS cloud services, including Athena, S3 , ECS / Docker, EMR, EC2, Lambda, CloudWatch, and EventBridge.Use Airflow or other orchestration tools for workflow management.Ensure data quality and integrity through rigorous testing and validation.Collaborate with other teams to understand data requirements and deliver solutions.Required Skills :
Proficiency in Python and Pyspark.Solid experience in AWS GlueStrong SQL skills.Experience with Aurora Postgres and Redshift.Knowledge of Informatica or other ETL tools.Experience with AWS cloud services, including Athena, S3, Lambda, CloudWatch, and EventBridge.Preferred Skills :
ECS / Docker (or any docker experience / concepts)EMR, EC2 (or knowledge of any other distributed computing)Familiarity with Airflow or other orchestration tools.Diverse Lynx LLC is an Equal Employment Opportunity employer. All qualified applicants will receive due consideration for employment without any discrimination. All applicants will be evaluated solely on the basis of their ability, competence and their proven capability to perform the functions outlined in the corresponding role. We promote and support a diverse workforce across all levels in the company.