As a Principal Data Engineer, you will lead a team in building and maintaining scalable, reliable data pipelines that bridge on-premises and cloud environments. You'll leverage your expertise in data engineering, streaming technologies, and leadership to drive the team towards achieving business objectives.
Key Responsibilities:
Team Leadership: Lead, mentor, and manage a team of data engineers specializing in streaming technologies.
Data Pipeline Development: Design and implement high-throughput, low-latency streaming data pipelines using Apache Kafka, ensuring integration with cloud services (e.g., BigQuery, Looker).
Data Analytics: Oversee the development of stream processing applications using Apache Spark or Apache Flink, and implement real-time data transformations and analytics using KSQL.
Data Storage: Design and maintain scalable data storage solutions with ClickHouse for fast analytics on streaming data.
ETL Processes: Lead the design and implementation of ETL processes, extracting, transforming, and loading data into a data warehouse.
Data Quality: Ensure data integrity, consistency, and accuracy through robust data quality assurance.
Optimization: Optimize performance and implement best practices in data engineering, covering data quality, security, and efficiency.
Collaboration: Collaborate with stakeholders to gather requirements and align data strategies with business objectives.
Technology Updates: Stay current with emerging technologies in streaming and cloud environments, evaluating their potential application.
Qualifications:
5+ years of hands-on data engineering experience with Python, Scala, or Java
3+ years of experience with cloud vendors (AWS, Azure, GCP), data warehouse services (e.g., Redshift, Databricks), and cloud storage
Expertise in KSQL, ClickHouse, ETL/ELT tools (e.g., Airflow, ADF, Glue, NiFi), and orchestration
Proficiency in code versioning (Git) and CI/CD pipelines
Experience with stream processing tools (Apache Kafka, Apache Flink, Apache Spark Structured Streaming)
Strong understanding of data modeling, optimization techniques, and stream processing patterns
Excellent leadership and mentorship skills with at least 2 years in a leadership role
Nice to Have:
Experience with NoSQL databases, Kafka Connect, Kafka Streams, Superset for data visualization
Knowledge of integrating real-time machine learning models into streaming environments
Expertise with monitoring and observability tools for streaming systems
Why Join Overwatch Agriculture?
At Overwatch Agriculture, we prioritize people over processes, fostering a supportive and tech-savvy environment. Our customized benefits prioritize your well-being and professional growth, making this an ideal opportunity for tech enthusiasts.
Principal Data Engineer • Boulder, Colorado, United States
SciTec is a dynamic small business, with the mission to deliver advanced sensor data processing technologies and scientific instrumentation capabilities in support of National Security and Defense,... Mostrar más
When it comes to finding the perfect locums assignment, sometimes it is all about who you know.CompHealth has been around for a long time and have a vast network of connections to facilities across... Mostrar más
Culinary Specialists operate and manage Navy dining and lodging services, planning menus, preparing meals, managing inventories and records, and contributing directly to crew morale and readiness a... Mostrar más
SciTec, a wholly owned subsidiary of Firefly Aerospace, is a dynamic non-traditional defense contractor that delivers advanced technologies in support of U.For the past forty-five plus years, we ha... Mostrar más
At LifeStance Health, we believe in a truly healthy society where mental and physical healthcare are unified to make lives better.Our mission is to help people lead healthier, more fulfilling lives... Mostrar más
Information Technology Professional (IT/CTN/IS).Information Systems Technicians, Cryptologic Technician Networks, and Intelligence Specialists keep the Fleet connected, informed, and secure by oper... Mostrar más
Enterprise Architect – AI & Technology Governance | Charlotte, NC.We are building out an enterprise architecture function with a focus on AI adoption and governance, and we’re looking for a seasone... Mostrar más
SciTec has been awarded multiple government contracts and is growing our creative Team! SciTec, Inc.We support customers throughout the Department of Defense and U.Government in building innovative... Mostrar más
The Agent Platform team delivers a new AI‑focused experience for Workday customers and their employees.We are at the forefront of Workday’s AI & Agentic initiatives, delivering and integrating Agen... Mostrar más
HOERBIGER Safety Solutions, IEP Technologies (IEP) is seeking a Regional Sales Manager (RSM) to grow sales, build brand recognition and strengthen distribution sales channels for our Explosion Prot... Mostrar más
Cogent Biosciences is a publicly traded biotechnology company focused on developing novel precision therapies to treat a broad range of patients with unmet medical needs.Cogent’s lead program, bezu... Mostrar más
At Ambrosia Biosciences, our mission is to develop innovative, orally delivered small‑molecule therapies that address the global challenges of obesity and metabolic disorders.Founded in 2024 and he... Mostrar más
Be among the first 25 applicants.We support customers throughout the Department of Defense and U.Government in building innovative new tools to deliver unique world-class data exploitation capabili... Mostrar más
A technology company specializing in aviation solutions seeks a Senior Software Engineer to lead the development of core product features and optimize data transformation pipelines.Located in Bould... Mostrar más
Job Title : Information Technology Professional (IT/CTN/IS) Category / Component : Enlisted • Both Overview Information Systems Technicians, Cryptologic Technician Networks, and Intelligence Specia... Mostrar más
Principal / Senior Scientist page is loaded## Principal / Senior Scientistlocations: USA - CO - Bouldertime type: Full timeposted on: Posted 2 Days Agojob requisition id: R00008378At KBI Bi... Mostrar más
An international financial services firm is seeking a skilled sales professional who excels in driving revenue and exceeding targets.This role involves developing territory strategies, engaging pro... Mostrar más
We are looking for a passionate individual with a sense of responsibility and willingness to help create families to join us as a surrogate.In this role, you will be instrumental in supporting indi... Mostrar más
