This is a 100% remote, TS / SCI required, mission-critical role focused on designing, building, and optimizing high-throughput, stateless data processing pipelines.
The successful candidate will develop a portable, scalable architecture that produces high-quality data streams from complex rule sets.
This data will be deployable on an infrastructure that is containerized, scales dynamically, and strictly adheres to government security protocols and quality standards.
Clearance :
- An active TS / SCI Security Clearance is required Responsibilities
- Pipeline development and orchestration : Design, implement, and maintain complex data workflows that use an "Extract, Load, and Enrich" (ELE) or a similar methodology to create high-fidelity data streams from foundational rules.
- You'll need to use distributed processing frameworks to achieve high computational efficiency and throughput.
This includes an expert-level proficiency in orchestration systems like Apache Airflow.
Quantitative fidelity engineering :Collaborate with quantitative analysts and data scientists to operationalize complex probabilistic models and statistical distribution rules.This includes a critical understanding of geospatial trajectories and activity patterns.
Architecture and scalability :Develop and maintain a resilient, portable, and infrastructure-agnostic data architecture.The design requires a stateless approach, optimized for in-memory processing.Expertise in containerization technologies like Docker and Kubernetes is mandatory for building single-container deployable solutions that can scale dynamically.
DevSecOps and quality assurance :Implement robust Continuous Integration / Continuous Deployment (CI / CD) pipelines to automate deployment and testing.You will also need to define source-to-target mapping directly as pipeline code to ensure auditable and verifiable software baselines.
Security and compliance integration :Seamlessly integrate data solutions with host-managed security, authentication, and authorization systems.Familiarity with relevant compliance frameworks for classified government operations (e.g., NIST 800-171) is a must.Requirements Education :
A Bachelor's degree in Computer Science, Data Engineering, Mathematics, or a related quantitative or technical field is required.Experience :
5+ years of demonstrated progressive experience in large-scale data engineering, specifically in pipeline development within secured or classified environments.Programming :
Expert proficiency in Python is mandatory for pipeline development.Advanced proficiency in other high-performance compiled languages like C, Go, or Rust is preferred.Distributed data processing :
Deep expertise in Apache Beam (mandatory), Apache Spark, or Dask for scalable stream / batch processing is required.Workflow orchestration :
Advanced proficiency in Apache Airflow (mandatory), Prefect, or Dagster for complex scheduling and monitoring is required.Data modeling & storage :
Expert SQL development and optimization skills are required, along with hands-on experience with both relational (e.g., PostgreSQL) and NoSQL databases (e.g., MongoDB).Containerization & deployment :
Expert-level experience with Docker and orchestration tools like Kubernetes or ECS / Fargate is mandatory.Cloud / Infrastructure :
Strong familiarity with cloud primitives (e.g., GCS, S3) or experience managing portable, secure on-premise / bare-metal environments.Quality & CI / CD :
Strong experience with CI / CD tools (e.g., GitHub Actions, Jenkins) and a commitment to unit and integration testing.About Torchlight Torchlight AI® provides Behavior Based Insights to decision-makers in government and corporate organizations to uncover emerging threats and new opportunities.Torchlight is an Equal Opportunity / Affirmative Action Employer.All qualified applicants will be considered for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, age, protected veteran status, disability status, or any other protected class.U.S.Citizenship is required for positions requiring government security clearance and for certain other restricted positions.Powered by JazzHR