Data Platform Engineer Cloud Ops + Data Ops - R1012521YASMESOFT INC • Plano, TX, United States
Data Platform Engineer Cloud Ops + Data Ops - R1012521
YASMESOFT INC • Plano, TX, United States
4 hours ago
Job type
- Full-time
- Part-time
- Temporary
- Quick Apply
Job descriptionHave you worked on Kubernetes? Do you have PySpark? Do you have Cloud AWS experience? Are you able to work with offshore? Platform Development : Design, build, and maintain scalable, efficient, and reliable ETL / ELT data pipelines to support data ingestion, transformation, and integration across diverse sources. Big Data Implementation : Serve as the subject matter expert for the Databricks environment, developing high-performance data transformation logic primarily using PySpark and Python. This includes utilizing Delta Live Tables (DLT) for declarative pipeline construction and ensuring governance through Unity Catalog. Cloud Infrastructure Management : Configure, maintain, and secure the underlying AWS cloud infrastructure required to run the Databricks platform, including virtual private clouds (VPCs), network endpoints, storage (S3), and cross-account access mechanisms. DevOps & Automation (CI / CD) : Own and enforce Continuous Integration / Continuous Deployment (CI / CD) practices for the data platform. Specifically, design and implement automated deployment workflows using GitHub Actions and modern infrastructure-as-code concepts to deploy Databricks assets (Notebooks, Jobs, DLT Pipelines, and Repos). Data Quality & Testing : Design and implement automated unit, integration, and performance testing frameworks to ensure data quality, reliability, and compliance with architectural standards. Performance Optimization : Optimize data workflows and cluster configurations for performance, cost efficiency, and scalability across massive datasets. Technical Leadership : Provide technical guidance on data principles, patterns, and best practices (e.g., Medallion Architecture, ACID compliance) to promote team capabilities and maturity. This includes leveraging Databricks SQL for high-performance analytics. Documentation & Review : Draft and review architectural diagrams, design documents, and interface specifications to ensure clear communication of data solutions and technical requirements. Experience : 5+ years of professional experience in Data Engineering, focusing on building scalable data platforms and production pipelines. Big Data Expertise : Minimum 3+ years of hands-on experience developing, deploying, and optimizing solutions within the Databricks ecosystem. Deep expertise required in : Delta Lake (ACID transactions, time travel, optimization). Unity Catalog (data governance, access control, metadata management). Delta Live Tables (DLT) (declarative pipeline development). Databricks Workspaces, Repos, and Jobs. Databricks SQL for analytics and warehouse operations. AWS Infrastructure & Security : Proven, hands-on experience (3+ years) with core AWS services and infrastructure components, including : Networking : Configuring and securing VPCs, VPC Endpoints, Subnets, and Route Tables for private connectivity. Security & Access : Defining and managing IAM Roles and Policies for secure cross-account access and least privilege access to data. Storage : Deep knowledge of Amazon S3 for data lake implementation and governance. Programming : Expert proficiency (4+ years) in Python for data manipulation, scripting, and pipeline development. Spark & SQL : Deep understanding of distributed computing and extensive experience (3+ years) with PySpark and advanced SQL for complex data transformation and querying. DevOps & CI / CD : Proven experience (2+ years) designing and implementing CI / CD pipelines, including proficiency with GitHub Actions or similar tools (e.g., GitLab CI, Jenkins) for automated testing and deployment. Data Concepts : Full understanding of ETL / ELT, Data Warehousing, and Data Lake concepts. Methodology : Strong grasp of Agile principles (Scrum). Version Control : Proficiency with Git for version control. AWS Data Ecosystem Experience : Familiarity and experience with AWS cloud-native data services, such as AWS Glue, Amazon Athena, Amazon Redshift, Amazon RDS, and Amazon DynamoDB. Knowledge of real-time or near-real-time streaming technologies (e.g., Kafka, Spark Structured Streaming). Experience in developing feature engineering pipelines for machine learning (ML) consumption. Background in performance tuning and capacity planning for large Spark clusters.
Industry Group : Automotive.
Job Title : Data Platform Engineer Cloud Ops + Data Ops - R1012521
Location : Plano, TX (Local to Dallas area, in client office 3days / wk.)
Duration : 12+ Months Contract (Potential for extension)
Pay Rate : $70 - $75
Custom Skill Requirements :
- Data Platform Engineer : Cloud Ops + Data Ops
- PySpark
- AWS
- Cloud
- DevOps : CI / CD
- Databricks administration
Qualifying Questions :
Job Description :
As a Data Platform Engineer, you will be responsible for the design, development, and maintenance of our high-scale, cloud-based data platform, treating data as a strategic product. You will lead the implementation of robust, optimized data pipelines using PySpark and the Databricks Unified Analytics Platform-leveraging its full ecosystem for Data Engineering, Data Science, and ML workflows. You will also establish best-in-class DevOps practices using CI / CD and GitHub Actions to ensure automated deployment and reliability. This role demands expertise in large-scale data processing and a commitment to modern, scalable data engineering and AWS cloud infrastructure practices.
Key Responsibilities :
Required Qualifications :
Preferred Qualifications :
Create a job alert for this search
Data Engineer Data • Plano, TX, United States
Related jobs
This is a remarkable opportunity to join a team dedicated to revolutionizing healthcare through data-driven solutions.You will play a vital role in developing and maintaining data pipelines and inf...Show more
As a Data Engineer, you will be part of a talented team of engineers responsible for the deployment and configuration of cloud resources to meet individual client business needs in AWS.Client engag...Show more
Hybrid onsite at client's office \- 2\ / 3 days IN, but can transition to full remote once established (1\-2 months).Duration : start at 6 months, likely 12+.
We are seeking an experienced Data Enginee...Show more
Describe the role and team the candidate will be joining.Describe the specific responsibilities and job functions of the role.
Describe the experience and attributes of the ideal candidate.Show more
Do you love a career where you Experience.If so, we are excited to have bumped onto you.Learn how we are redefining the.Clients, Job-seekers and Employees.
AWS Cloud Developer .Position&n...Show more
Do you love building and pioneering in the technology space? Do you enjoy solving complex business problems in a fast-paced, collaborative, inclusive, and iterative delivery environment? At Capital...Show more
Overview / Job Responsibilities.We are seeking a highly experienced and skilled Senior Data Lake Engineer to join our team.
As the Senior Data Lake Engineer you will play a critical role in establish...Show more
Extensive experience in designing, configuring, deploying, managing and automating AWS Core Services like S3, IAM, EC2, Route53, SNS, SQS, ELB, CloudWatch, Lambda and VPC.Experience in automating c...Show more
Job Title : Senior Snowflake Data and Platform Engineer Location : Dallas, TX 5 days onsite Duration : Contract Key R...Show more
Data Engineer (Full Time Position) Dallas, TX.We offer solutions that are tailored to the needs of each individual cli...Show more
REQUIREMENTS : Qualified candidates must be legally authorized to be employed in the United States on a full-time basis for any position.
Omnitech will not provide sponsorship for employment visa sta...Show more
Allata is a global consulting and technology services firm with offices in the US, India, and Argentina.We help organizations accelerate growth, drive innovation, and solve complex challenges by co...Show more
Job : Junior Data Engineer (SQL, Python, GCP, SAS, Teradata) Location : Dallas area- must currently live there <...Show more
Senior Data Engineer (Python, SQL, AWS).Do you love building and pioneering in the technology space? Do you enjoy solving complex business problems in a fast-paced, collaborative,.At Capital One, y...Show more Do you love building and pioneering in the technology space? Do you enjoy solving complex business problems in a fast-paced, collaborative, inclusive, and iterative delivery environment? At Capital...Show more 
Distinguished Data Engineer - Capital One Software (Remote).Ever since our first credit card customer in 1994, Capital One has recognized that technology and data can enable even large companies to...Show more
At AECOM, we're delivering a better world.Whether improving your commute, keeping the lights on, providing access to clean water, or transforming skylines, our work helps people and communities thr...Show more
We are looking for a highly skilled Full Stack Data Engineer with expertise in Databricks to design, develop, and optimize end-to-end data pipelines, data platforms, and analytics solutions.This ro...Show more
Data Engineer
Rnrs Solutions • Arlington, Texas, United States
Remote
Full-time
Last updated: 30+ days ago • Promoted
Data Engineer
Protagona • Dallas, TX, US
Full-time
Quick Apply
Last updated: 30+ days ago
AWS Data Engineer
3Core Systems • Dallas, Texas, United States
Full-time
Last updated: 3 days ago • Promoted
Data Engineer
Ascentt • Plano, Texas, United States
Full-time
Last updated: 30+ days ago • Promoted
AWS Cloud Engineer
Alignity Solutions • Dallas, TX, us
Full-time
Quick Apply
Last updated: 8 days ago
Lead Data Engineer
Capital One • Plano, TX, US
Full-time +1
Last updated: 30+ days ago • Promoted
Senior Databricks Platform Engineer
Sev1 Tech • Arlington, Texas, USA
Full-time
Last updated: 21 hours ago • Promoted • New!
Data Engineer-2
eTeam Inc • Plano, Texas, United States
Full-time
Quick Apply
Last updated: 30+ days ago
Senior Snowflake Data and Platform Engineer
VDart Inc • Dallas, TX, United States
Full-time
Quick Apply
Last updated: 10 hours ago • New!
Data Engineer
Cg Infinity • Plano, Texas, United States
Full-time
Last updated: 30+ days ago • Promoted
Data Engineer
Omnitech, Inc. • Arlington, Texas, United States
Full-time
Last updated: 30+ days ago • Promoted
Lead Data Engineer Databricks
Allata • Dallas, Texas, United States
Remote
Full-time
Last updated: 30+ days ago • Promoted
Junior Data Engineer
Akaasa Technologies • Dallas, TX, United States
Full-time
Quick Apply
Last updated: 1 day ago
Senior Data Engineer (Python, SQL, AWS)
Capital One • Plano, TX, US
Full-time +1
Last updated: 30+ days ago • Promoted
Senior Lead Data Engineer
Capital One • Plano, TX, US
Full-time +1
Last updated: 30+ days ago • Promoted
Distinguished Data Engineer - Capital One Software (Remote)
Capital One • Plano, TX, US
Remote
Full-time +1
Last updated: 30+ days ago • Promoted
Lead Data Engineer - Hybrid (Houston or Dallas, TX)
Aecom • Dallas, Texas, United States
Full-time
Last updated: 30+ days ago • Promoted
(US) Fullstack Data Engineer - Databricks
Codvo.ai • Dallas, Texas, United States
Full-time
Last updated: 30+ days ago • Promoted