Job Title : Lead Databricks Data Architect
Location : Atlanta, GA (Onsite / Hybrid / Remote as per requirement)
Experience : 8 – 10 Years
Job Type : Full Time
Job Overview
We are seeking an experienced Lead Databricks Data Architect to design and implement robust, scalable, and secure cloud-based data solutions. This role involves working with Azure cloud services, leading development of data ingestion pipelines, and optimizing data architecture to meet evolving business needs.
Key Responsibilities
- Architect and build end-to-end data solutions using Azure Databricks, Azure Data Factory, and Azure SQL Database.
- Lead the development of real-time and batch data pipelines using Delta Live Tables (DLT) and Structured Streaming.
- Collaborate with stakeholders to translate business requirements into scalable data warehouse solutions on Azure.
- Provide strategic leadership for large-scale data programs and ensure alignment with organizational goals.
- Drive performance optimization of Spark jobs and Databricks workflows through advanced tuning techniques.
- Build and maintain metadata-driven ingestion frameworks to support automation and scalability.
- Work cross-functionally with data scientists, analysts, and other stakeholders to deliver business-aligned data solutions.
- Implement data governance, quality checks, and security measures to maintain data integrity and compliance.
- Monitor and enhance cloud-based data infrastructure for performance and cost-efficiency.
- Lead or support migration and upgrade efforts of legacy applications (e.g., .NET Framework) to cloud platforms.
- Utilize Azure DevOps tools including Git, pipelines, and work item management.
- Troubleshoot issues on Windows Server environments (e.g., IIS, Event Viewer, Performance Monitor).
- Participate in Agile / Scrum ceremonies including sprint planning, retrospectives, and daily standups.
Technical Skills Required
Strong proficiency in :Azure Databricks, ADF, Azure SQL
Apache Spark (job tuning and internals)Structured Streaming, Delta Live TablesExperience in :.NET Frameworks 4.8.2 and Core 6 / 8ASP.NET MVC, Blazor, WebFormsWeb services & API integrationsMicrosoft SQL Server (DDL, DML, stored procedures)Nota Fiscal integration (Preferred in finance / logistics / related domains)Familiarity with :Azure authentication and key vault
Agile / Lean methodologiesPreferred Qualifications
Proven experience designing and delivering enterprise-level cloud data platforms.Ability to lead cross-functional teams and mentor junior developers.Excellent communication, documentation, and stakeholder management skills.Familiarity with CI / CD and infrastructure-as-code in cloud environments.