Data Platform Engineer Cloud Ops + Data Ops R1012521YASMESOFT INC • Plano, Texas, USA

Data Platform Engineer Cloud Ops + Data Ops R1012521

YASMESOFT INC • Plano, Texas, USA

1 day ago

Job type

Full-time

Temporary

Job description

Industry Group : Automotive.

Job Title : Data Platform Engineer Cloud Ops Data Ops - R1012521

Location : Plano TX (Local to Dallas area in client office 3days / wk.)

Duration : 12 Months Contract (Potential for extension)

Pay Rate : $70 - $75

Custom Skill Requirements :

Data Platform Engineer : Cloud Ops Data Ops
PySpark
AWS
Cloud
DevOps : CI / CD
Databricks administration

Qualifying Questions :

Have you worked on Kubernetes

Do you have PySpark

Do you have Cloud AWS experience

Are you able to work with offshore

Job Description :

As a Data Platform Engineer you will be responsible for the design development and maintenance of our high-scale cloud-based data platform treating data as a strategic product. You will lead the implementation of robust optimized data pipelines using PySpark and the Databricks Unified Analytics Platform-leveraging its full ecosystem for Data Engineering Data Science and ML workflows. You will also establish best-in-class DevOps practices using CI / CD and GitHub Actions to ensure automated deployment and reliability. This role demands expertise in large-scale data processing and a commitment to modern scalable data engineering and AWS cloud infrastructure practices.

Key Responsibilities :

Platform Development : Design build and maintain scalable efficient and reliable ETL / ELT data pipelines to support data ingestion transformation and integration across diverse sources.

Big Data Implementation : Serve as the subject matter expert for the Databricks environment developing high-performance data transformation logic primarily using PySpark and Python. This includes utilizing Delta Live Tables (DLT) for declarative pipeline construction and ensuring governance through Unity Catalog.

Cloud Infrastructure Management : Configure maintain and secure the underlying AWS cloud infrastructure required to run the Databricks platform including virtual private clouds (VPCs) network endpoints storage (S3) and cross-account access mechanisms.

DevOps & Automation (CI / CD) : Own and enforce Continuous Integration / Continuous Deployment (CI / CD) practices for the data platform. Specifically design and implement automated deployment workflows using GitHub Actions and modern infrastructure-as-code concepts to deploy Databricks assets (Notebooks Jobs DLT Pipelines and Repos).

Data Quality & Testing : Design and implement automated unit integration and performance testing frameworks to ensure data quality reliability and compliance with architectural standards.

Performance Optimization : Optimize data workflows and cluster configurations for performance cost efficiency and scalability across massive datasets.

Technical Leadership : Provide technical guidance on data principles patterns and best practices (e.g. Medallion Architecture ACID compliance) to promote team capabilities and maturity. This includes leveraging Databricks SQL for high-performance analytics.

Documentation & Review : Draft and review architectural diagrams design documents and interface specifications to ensure clear communication of data solutions and technical requirements.

Required Qualifications :

Experience : 5 years of professional experience in Data Engineering focusing on building scalable data platforms and production pipelines.

Big Data Expertise : Minimum 3 years of hands-on experience developing deploying and optimizing solutions within the Databricks ecosystem.

Deep expertise required in :

Delta Lake (ACID transactions time travel optimization).

Unity Catalog (data governance access control metadata management).

Delta Live Tables (DLT) (declarative pipeline development).

Databricks Workspaces Repos and Jobs.

Databricks SQL for analytics and warehouse operations.

AWS Infrastructure & Security : Proven hands-on experience (3 years) with core AWS services and infrastructure components including :

Networking : Configuring and securing VPCs VPC Endpoints Subnets and Route Tables for private connectivity.

Security & Access : Defining and managing IAM Roles and Policies for secure cross-account access and least privilege access to data.

Storage : Deep knowledge of Amazon S3 for data lake implementation and governance.

Programming : Expert proficiency (4 years) in Python for data manipulation scripting and pipeline development.

Spark & SQL : Deep understanding of distributed computing and extensive experience (3 years) with PySpark and advanced SQL for complex data transformation and querying.

DevOps & CI / CD : Proven experience (2 years) designing and implementing CI / CD pipelines including proficiency with GitHub Actions or similar tools (e.g. GitLab CI Jenkins) for automated testing and deployment.

Data Concepts : Full understanding of ETL / ELT Data Warehousing and Data Lake concepts.

Methodology : Strong grasp of Agile principles (Scrum).

Version Control : Proficiency with Git for version control.

Preferred Qualifications :

AWS Data Ecosystem Experience : Familiarity and experience with AWS cloud-native data services such as AWS Glue Amazon Athena Amazon Redshift Amazon RDS and Amazon DynamoDB.

Knowledge of real-time or near-real-time streaming technologies (e.g. Kafka Spark Structured Streaming).

Experience in developing feature engineering pipelines for machine learning (ML) consumption.

Background in performance tuning and capacity planning for large Spark clusters.

Key Skills

Apache Hive,S3,Hadoop,Redshift,Spark,AWS,Apache Pig,NoSQL,Big Data,Data Warehouse,Kafka,Scala

Employment Type : Full Time

Experience : years

Vacancy : 1

Create a job alert for this search

Data Engineer Data • Plano, Texas, USA

Related jobs

Data Engineer

Protagona • Dallas, TX, US

Full-time

Quick Apply

As a Data Engineer, you will be part of a talented team of engineers responsible for the deployment and configuration of cloud resources to meet individual client business needs in AWS.Client engag...Show more

Last updated: 30+ days ago

Software Development Engineer

Amazon • Celina, TX, USA

Full-time

Join Amazon's engineering team and help us build innovative solutions to complex problems.As a Software Development Engineer, you will design, develop, and test software applications and services.W...Show more

Last updated: 21 days ago • Promoted

IOS Engineer

Vytwo Technologies Inc • Prosper, TX, United States

Full-time

Position : IOS Engineer Hire Type : FTE (Full Time) only - No contractors Work Location : Austin, TX or Dallas, TX (3 days Onsite) Salary : 120k / yr. Must be legally authorized to work in US without need...Show more

Last updated: 6 days ago • Promoted

Cybersecurity Staff Engineer

Meritore Technologies • McKinney, TX, United States

Full-time

Data Protection to lead the design, implementation, and continuous improvement of data protection solutions.This role will focus on encryption standards, certificate lifecycle management, DLP, secu...Show more

Last updated: 5 days ago • Promoted

Lead Data Engineer

Coca Cola Southwest Beverages • Dallas, TX, United States

Full-time

This role oversees and manages the work of Data Engineer who design and develop ETL pipelines and processes in our Cloud Infrastructure. They are responsible for the overall data integrity and quali...Show more

Last updated: 30+ days ago • Promoted

Data Engineer-2

eTeam Inc • Plano, Texas, United States

Full-time

Quick Apply

Extensive experience in designing, configuring, deploying, managing and automating AWS Core Services like S3, IAM, EC2, Route53, SNS, SQS, ELB, CloudWatch, Lambda and VPC.Experience in automating c...Show more

Last updated: 30+ days ago

AWS Data Engineering Leader

Coforge • Dallas, TX, United States

Full-time

Role : Data Engineer Leader - AWS.Key Skills : AWS Glue, Lambda, DBT, Python, Data Engineering.We are seeking a dynamic and experienced. The ideal candidate will have deep expertise in.Lead the design...Show more

Last updated: 1 day ago • Promoted

Cloud Engineer

Precision Technologies • Dallas, Texas, United States

Full-time

We are seeking a skilled and motivated.The ideal candidate will have strong expertise in Microsoft Azure, cloud infrastructure, CI / CD pipelines, automation, and cloud migration.You will be responsi...Show more

Last updated: 2 hours ago • Promoted • New!

Azure Data Platform Architect

Walker Elliott • Dallas, TX, US

Full-time

Location : Dallas, TX, Type : Full-Time | On-Site.Our client, a leader in land surveying, planning and civil engineering, is looking for an Azure Data Platform Architect to help integrate a wide rang...Show more

Last updated: 6 days ago • Promoted

Technical Manager Cloud Operations and Support

Damco • Mckinney, TX, US

Full-time

Technical Manager Cloud Operations And Support Work Location.Primary Skills : Business Analyst, Project Management.Secondary Skills : Cloud Computing Specialization Project Management : Engineer, Clo...Show more

Last updated: 30+ days ago • Promoted

Junior Data Engineer

Akaasa Technologies • Dallas, TX, United States

Full-time

Quick Apply

Job : Junior Data Engineer (SQL, Python, GCP, SAS, Teradata) Location : Dallas area- must currently live there <...Show more

Last updated: 3 days ago

Data Engineer Lead : 24-02271 (No C2C)

Akraya Inc • Dallas, Texas, United States

Full-time

Quick Apply

AWS, SQL, Python, Redshift, S3.Dallas, TX OR Seattle, WA (Hybrid.Plan, design, implement, and manage a deployment of self-service data platform using native AWS services. Design, build, and maintain...Show more

Last updated: 30+ days ago

Lead Cloud Engineer

Selby Jennings • Dallas, TX, United States

Full-time

Lead Cloud Engineer - Contract-to-Hire (Hybrid, Dallas-Based).Dallas, TX (Hybrid - 3 days onsite per week).Confidential (Consumer Finance). We're seeking a Lead Cloud Engineer to join a high-impact ...Show more

Last updated: 1 day ago • Promoted

AWS Cloud Engineer

VDart Inc • McKinney, TX, United States

Full-time

Quick Apply

MsoNoSpacingCxSpFirst"> Role : AWS Cloud Engineer Location : McKinney, TX (Hybrid) &...Show more

Last updated: 3 days ago

Data Integration Engineer

Redolent • Dallas, TX, United States

Full-time

Title : Data Integration Engineer.Proficient in Apache Kafka concepts (producers, consumers, topics, partitioning, etc.Expertise in developing SSIS packages using the visual designer and advanced tr...Show more

Last updated: 8 days ago • Promoted

Cloud Engineer

EGB Systems & Solutions • Plano, Texas, United States

Full-time

Quick Apply

Must have : AWS (must have at least 3-4 skill - S3, EC2, VPC, EKS, ECS, CLOUDFORMATION, CLOUDWATCH, IAM).Big Plus : EKS / Kubernetes AND terraform. Deliver incident management and advanced-level support...Show more

Last updated: 30+ days ago

Lead Data Engineer - AWS

Tiger Analytics • Dallas, TX, US

Remote

Full-time

Quick Apply

Tiger Analytics is a fast-growing advanced analytics consulting firm.Our consultants bring deep expertise in Data Science, Machine Learning and AI. We are the trusted analytics partner for multiple ...Show more

Last updated: 30+ days ago

Cloud DevSecOps Engineer

Regions Bank • Dallas, TX, United States

Full-time

Thank you for your interest in a career at Regions.At Regions, we believe associates deserve more than just a job.We believe in offering performance-driven individuals a place where they can build ...Show more

Last updated: 17 days ago • Promoted