As a Principal Data Engineer, you will lead a team in building and maintaining scalable, reliable data pipelines that bridge on-premises and cloud environments. You'll leverage your expertise in data engineering, streaming technologies, and leadership to drive the team towards achieving business objectives.
Key Responsibilities:
Team Leadership: Lead, mentor, and manage a team of data engineers specializing in streaming technologies.
Data Pipeline Development: Design and implement high-throughput, low-latency streaming data pipelines using Apache Kafka, ensuring integration with cloud services (e.g., BigQuery, Looker).
Data Analytics: Oversee the development of stream processing applications using Apache Spark or Apache Flink, and implement real-time data transformations and analytics using KSQL.
Data Storage: Design and maintain scalable data storage solutions with ClickHouse for fast analytics on streaming data.
ETL Processes: Lead the design and implementation of ETL processes, extracting, transforming, and loading data into a data warehouse.
Data Quality: Ensure data integrity, consistency, and accuracy through robust data quality assurance.
Optimization: Optimize performance and implement best practices in data engineering, covering data quality, security, and efficiency.
Collaboration: Collaborate with stakeholders to gather requirements and align data strategies with business objectives.
Technology Updates: Stay current with emerging technologies in streaming and cloud environments, evaluating their potential application.
Qualifications:
5+ years of hands-on data engineering experience with Python, Scala, or Java
3+ years of experience with cloud vendors (AWS, Azure, GCP), data warehouse services (e.g., Redshift, Databricks), and cloud storage
Expertise in KSQL, ClickHouse, ETL/ELT tools (e.g., Airflow, ADF, Glue, NiFi), and orchestration
Proficiency in code versioning (Git) and CI/CD pipelines
Experience with stream processing tools (Apache Kafka, Apache Flink, Apache Spark Structured Streaming)
Strong understanding of data modeling, optimization techniques, and stream processing patterns
Excellent leadership and mentorship skills with at least 2 years in a leadership role
Nice to Have:
Experience with NoSQL databases, Kafka Connect, Kafka Streams, Superset for data visualization
Knowledge of integrating real-time machine learning models into streaming environments
Expertise with monitoring and observability tools for streaming systems
Why Join Overwatch Agriculture?
At Overwatch Agriculture, we prioritize people over processes, fostering a supportive and tech-savvy environment. Our customized benefits prioritize your well-being and professional growth, making this an ideal opportunity for tech enthusiasts.
Principal Data Engineer • Boulder, CO, US
A dynamic small business is seeking a Senior Software Platform Engineer to contribute to a Platform as a Service (PaaS) team.This role focuses on building and evolving platform capabilities for con...Show more
Data Entry Specialist (Remote).We're looking for a detail-oriented Data Entry Specialist to join our team from home.In this role, you'll be responsible for accurately inputting, updating, and maint...Show more
DataAnnotation is committed to creating high-quality AI.Join our team to help train the next generation of AI while enjoying the flexibility of remote work and the freedom to set your own schedule....Show more
Responsibilities The primary responsibility of this position is to read meters and record consumption of the water used, cleaning of meter boxes, and removal of vegetation impeding access to meters...Show more
Who we are Green Mountain Animal Hospital is seeking an experienced and motivated Hospital Manager to lead our team and oversee the daily operations of our veterinary practice.We are committed to d...Show more
We're transforming one of the world’s oldest industries with cutting‑edge technology and an innovative approach.Backed by top‑tier investors and recognized by Time as one of the “best Inventions of...Show more
SciTec, a wholly owned subsidiary of Firefly Aerospace, is a dynamic non-traditional defense contractor that delivers advanced technologies in support of U.For the past forty-five plus years, we ha...Show more
At Synopsys, we drive the innovations that shape the way we live and connect.Our technology is central to the Era of Pervasive Intelligence, from self-driving cars to learning machines.We lead in c...Show more
A global private markets firm is seeking an Evergreen Portfolio Management Investment Data Associate to manage fund liquidity, support investment activities, and maintain data integrity.The role re...Show more
SciTec has been awarded multiple government contracts and is growing our creative Team! SciTec, Inc.We support customers throughout the Department of Defense and U.Government in building innovative...Show more
Join Our Team as a Territory Manager – Where Passion Meets Opportunity!.Are you ready to build relationships, drive sales, and make a meaningful impact in the foodservice industry? At US Foods, we’...Show more
Be among the first 25 applicants.We support customers throughout the Department of Defense and U.Government in building innovative new tools to deliver unique world-class data exploitation capabili...Show more
SciTech is a dynamic small business, with the mission to deliver advanced sensor data processing technologies and scientific instrumentation capabilities in support of National Security and Defense...Show more
We're transforming one of the world’s oldest industries with cutting‑edge technology and an innovative approach.Backed by top‑tier investors and recognized by Time as one of the “best Inventions of...Show more
A technology company specializing in aviation solutions seeks a Senior Software Engineer to lead the development of core product features and optimize data transformation pipelines.Located in Bould...Show more
The Automation & Controls Engineer supports TruStile’s manufacturing operations by maintaining,.Program, troubleshoot, and maintain industrial automation systems including Beckhoff, Allen-Bradley,....Show more
Shift Leader Evergreen, CO Title: Shift Leader State: Colorado Reports to: Restaurant Leader Supervises: Shift Team Members FLSA: Non-Exempt Department: Operations Revision/Creation Date: January 1...Show more
Amazon warehouse associates hiring in your area.If you are looking for outstanding pay, flexible part-time, full-time and seasonal schedules, plus benefits for you and your family, the Amazon Logis...Show more
%20Limited){kind=logo}
