Databricks Engineer

Job Description

Job Description

Triangle Cyber is seeking a highly skilled and motivated Databricks Engineer to join our team in northern Virginia for a federal contract engagement. As a key contributor, you will help design, implement, and optimize advanced analytics solutions on the Databricks platform to support federal agency objectives. The ideal candidate will have hands-on experience with cloud-based big data technologies, a strong understanding of data engineering principles, and the ability to work within the compliance constraints required for government contracts. You will collaborate with data scientists, analysts, and other stakeholders to deliver scalable, secure, and high-performance data solutions in a mission-critical environment.

Key Responsibilities

• Databricks Platform Development : Design, build, and maintain scalable data pipelines and analytics solutions leveraging Databricks and Apache Spark for federal agency projects.
• Data Integration : Develop and manage the ingestion, transformation, and integration of structured and unstructured datasets from various sources (on-premises, cloud, and external) into Databricks workspaces.
• ETL Process Automation : Implement automated ETL workflows using Databricks notebooks, orchestration tools, and CI / CD methodologies to ensure efficient and repeatable data processing.
• Optimization & Performance Tuning : Monitor, analyze, and optimize the performance of Spark jobs and Databricks clusters, including tuning configurations to meet demanding SLAs.
• Collaboration & Support : Work closely with data scientists, business analysts, and project managers to understand requirements, provide technical guidance, and support advanced analytics, machine learning, and reporting needs.
• Security & Compliance : Ensure all data engineering activities meet federal security and compliance standards (such as FISMA, FedRAMP, and other agency-specific requirements), including data encryption, access controls, and auditing.
• Documentation : Produce and maintain comprehensive technical documentation for data pipelines, architecture, processes, and compliance procedures.
• Troubleshooting : Diagnose and resolve issues related to Databricks environments, Spark applications, data pipelines, and integrations in a timely manner.
• Continuous Improvement : Stay current with emerging Databricks features, best practices, and federal data management requirements, and proactively recommend improvements to enhance the agency's analytics capabilities.
• Stakeholder Communication : Communicate project progress, technical challenges, and solutions to both technical and non-technical stakeholders.

Required Qualifications

Desired Qualifications