Talent.com
Senior Data Engineer

Senior Data Engineer

Torchlight Analytics LLCUS
14 hours ago
Job type
  • Full-time
  • Quick Apply
Job description

This is a 100% remote, TS / SCI required, mission-critical role focused on designing, building, and optimizing high-throughput, stateless data processing pipelines.

The successful candidate will develop a portable, scalable architecture that produces high-quality data streams from complex rule sets.

This data will be deployable on an infrastructure that is containerized, scales dynamically, and strictly adheres to government security protocols and quality standards.

Clearance :

  • An active TS / SCI Security Clearance is required Responsibilities
  • Pipeline development and orchestration : Design, implement, and maintain complex data workflows that use an "Extract, Load, and Enrich" (ELE) or a similar methodology to create high-fidelity data streams from foundational rules.
  • You'll need to use distributed processing frameworks to achieve high computational efficiency and throughput.

This includes an expert-level proficiency in orchestration systems like Apache Airflow.

  • Quantitative fidelity engineering :
  • Collaborate with quantitative analysts and data scientists to operationalize complex probabilistic models and statistical distribution rules.
  • This includes a critical understanding of geospatial trajectories and activity patterns.

  • Architecture and scalability :
  • Develop and maintain a resilient, portable, and infrastructure-agnostic data architecture.
  • The design requires a stateless approach, optimized for in-memory processing.
  • Expertise in containerization technologies like Docker and Kubernetes is mandatory for building single-container deployable solutions that can scale dynamically.

  • DevSecOps and quality assurance :
  • Implement robust Continuous Integration / Continuous Deployment (CI / CD) pipelines to automate deployment and testing.
  • You will also need to define source-to-target mapping directly as pipeline code to ensure auditable and verifiable software baselines.

  • Security and compliance integration :
  • Seamlessly integrate data solutions with host-managed security, authentication, and authorization systems.
  • Familiarity with relevant compliance frameworks for classified government operations (e.g., NIST 800-171) is a must.
  • Requirements Education :

  • A Bachelor's degree in Computer Science, Data Engineering, Mathematics, or a related quantitative or technical field is required.
  • Experience :

  • 5+ years of demonstrated progressive experience in large-scale data engineering, specifically in pipeline development within secured or classified environments.
  • Programming :

  • Expert proficiency in Python is mandatory for pipeline development.
  • Advanced proficiency in other high-performance compiled languages like C, Go, or Rust is preferred.
  • Distributed data processing :

  • Deep expertise in Apache Beam (mandatory), Apache Spark, or Dask for scalable stream / batch processing is required.
  • Workflow orchestration :

  • Advanced proficiency in Apache Airflow (mandatory), Prefect, or Dagster for complex scheduling and monitoring is required.
  • Data modeling & storage :

  • Expert SQL development and optimization skills are required, along with hands-on experience with both relational (e.g., PostgreSQL) and NoSQL databases (e.g., MongoDB).
  • Containerization & deployment :

  • Expert-level experience with Docker and orchestration tools like Kubernetes or ECS / Fargate is mandatory.
  • Cloud / Infrastructure :

  • Strong familiarity with cloud primitives (e.g., GCS, S3) or experience managing portable, secure on-premise / bare-metal environments.
  • Quality & CI / CD :

  • Strong experience with CI / CD tools (e.g., GitHub Actions, Jenkins) and a commitment to unit and integration testing.
  • About Torchlight Torchlight AI® provides Behavior Based Insights to decision-makers in government and corporate organizations to uncover emerging threats and new opportunities.
  • Torchlight is an Equal Opportunity / Affirmative Action Employer.
  • All qualified applicants will be considered for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, age, protected veteran status, disability status, or any other protected class.
  • U.S.
  • Citizenship is required for positions requiring government security clearance and for certain other restricted positions.
  • Powered by JazzHR
  • Create a job alert for this search

    Senior Data Engineer • US

    Related jobs
    Data Engineer with IRS MBI Clearance

    Data Engineer with IRS MBI Clearance

    JobgetherUS
    Remote
    Full-time
    Quick Apply
    This position is posted by Jobgether on behalf of a partner company.We are currently looking for a Data Engineer with IRS MBI Clearance in the United States. We are seeking a Data Engineer to design...Show moreLast updated: 2 days ago
    Data Developer / Architect

    Data Developer / Architect

    Fathom Management LLCUSA
    Full-time
    Quick Apply
    Department of Veterans Affairs (VA).They are modernizing clinical contact centers to provide Veterans with additional choices for meeting clinical, pharmacy, scheduling, and administrative needs.VA...Show moreLast updated: 3 days ago
    Data Engineer

    Data Engineer

    Cloudious LLCUnited States
    Full-time
    Quick Apply
    Data Engineer Remote Contract-6 Months+ Qualificat...Show moreLast updated: 30+ days ago
    Senior Data Engineer

    Senior Data Engineer

    OpenkyberUnited States
    Full-time
    Quick Apply
    A Senior Data Engineer is required for a contract position at PSCI.Requirements : Must have 5-8+ years of IT experience in data engineering and DWH, ETL databases, ETL...Show moreLast updated: 6 days ago
    Data Engineer

    Data Engineer

    Inherent TechnologiesUnited States
    Full-time
    Quick Apply
    Position : Data Engineer Location : Denver, CO (Remote allowed) Show moreLast updated: 30+ days ago
    Databricks Architect

    Databricks Architect

    Techcliff GroupUnited States
    Full-time
    Quick Apply
    MessageBody"> Job Description Candidates MUST have overseen a migration from Hadoop to Databricks.Lead Databricks Architect Job Description<...Show moreLast updated: 7 days ago
    Databricks Architect

    Databricks Architect

    Tiger AnalyticsUS
    Remote
    Full-time
    Quick Apply
    Tiger Analytics is pioneering what AI and analytics can do to solve some of the toughest problems faced by organizations globally. We develop bespoke solutions powered by data and technology for sev...Show moreLast updated: 30+ days ago
    Databricks Engineer

    Databricks Engineer

    Akaasa TechnologiesUnited States
    Full-time
    Quick Apply
    Qualifications Bachelor's degree in Computer Science, Information Technology, or a related field, or equivalent professional experience. Databricks and Hadoop ecosystems....Show moreLast updated: 30+ days ago
    Data Engineer - FHIR Expertise

    Data Engineer - FHIR Expertise

    WalterUS
    Remote
    Full-time
    Quick Apply
    Our client is a forward-thinking company focused on creating cutting-edge medical AI assistants.Their mission is to revolutionize healthcare by integrating AI into health systems to improve efficie...Show moreLast updated: 7 days ago
    Databricks Architect

    Databricks Architect

    Stellent IT LLCUnited States
    Full-time
    Quick Apply
    Job Title : - Data bricks Architect Remote DURATION : - 3 months, possible extension Show moreLast updated: 7 days ago
    • Promoted
    Data & AI Engineer

    Data & AI Engineer

    Truity Credit UnionBartlesville, OK, US
    Full-time
    Truity Credit Union is seeking a data-savvy and forward-thinking.Rather than full-scale software development, this position emphasizes. You’ll collaborate with teams such as Software Engineeri...Show moreLast updated: 13 days ago
    Data Engineer II (Remote - US)

    Data Engineer II (Remote - US)

    JobgetherUS
    Remote
    Full-time
    Quick Apply
    This position is posted by Jobgether on behalf of a partner company.We are currently looking for a.The Data Engineer II will play a critical role in building and optimizing scalable data pipelines ...Show moreLast updated: 15 days ago
    Data Engineer

    Data Engineer

    OpenkyberUnited States
    Full-time
    Quick Apply
    The position is part of a data-focused team working on the "T-Bar" (Transaction Books and Records) product, a critical data store for positions, transactions, and tax lots.The team is tra...Show moreLast updated: 6 days ago
    Data engineer

    Data engineer

    Shrive Technologies LLCUnited States
    Full-time
    Quick Apply
    MessageBody" style="color : black !important; font-size : 12pt; font-family : Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif. align-self : start;"> 7+ y...Show moreLast updated: 6 days ago
    Data Engineer (Data Ingestion)

    Data Engineer (Data Ingestion)

    Akaasa TechnologiesUnited States
    Full-time
    Quick Apply
    Data Engineer (Data Ingestion) Location : Remote (No California OR New York Candidates) Must Have : LinkedIn with Picture Show moreLast updated: 1 day ago
    Data Engineer II

    Data Engineer II

    TekWissen LLCUnited States
    Temporary
    Quick Apply
    Overview TekWissen is a global workforce management provider headquartered in Ann Arbor, Michigan that offers strategic talent solutions...Show moreLast updated: 7 days ago
    Databricks Architect

    Databricks Architect

    Akaasa TechnologiesUnited States
    Full-time
    Quick Apply
    Table" style="border-collapse : collapse; border : solid #a3a3a3 1.Title : <...Show moreLast updated: 8 days ago
    Senior Software Engineer, Data

    Senior Software Engineer, Data

    Anvilogic IncUS
    Remote
    Full-time
    Quick Apply
    Anvilogic is a Palo Alto-based AI cybersecurity startup founded in 2019 by security veterans and data scientists from Fortune 500 companies. Our mission is to democratize threat detection and huntin...Show moreLast updated: 30+ days ago
    Data Engineer - Remote

    Data Engineer - Remote

    Lorven technologiesUnited States
    Remote
    Full-time
    Quick Apply
    Data Engineer Position : Contract (W2) Duration : 12+ Months Location : Remote< / b&g...Show moreLast updated: 6 days ago
    Data Engineer

    Data Engineer

    Care ContinuityUSA
    Full-time
    Quick Apply
    Care Continuity is redefining patient navigation.AI-driven insights, and compassionate human support to ensure patients receive the care they need - when and where they need it.Our solutions empowe...Show moreLast updated: 3 days ago