Azure Databricks Architect - San Diego, CA

Overview

An Azure Databricks Architect is a technical role focused on designing and implementing scalable data solutions using Azure Databricks, acting as a subject matter expert in the platform to create efficient data pipelines, manage data lakes, and ensure data quality while leveraging the full capabilities of the Azure cloud ecosystem; essentially, they are responsible for architecting robust data solutions on Azure Databricks, considering data ingestion, processing, storage, and analysis needs within the context of an organization's business requirements.

Responsibilities

Solution Design : Develop and implement data architecture using Azure Databricks, including data ingestion pipelines, data transformation logic, and data warehousing strategies. Building and optimizing ETL pipelines, particularly in Databricks using PySpark, Scala, or Spark SQL. Ability to create and manage Databricks notebooks for data exploration, pipeline development, and data visualization. Familiarity with scheduling and automating Databricks jobs.

Big Data Processing : Design and optimize ETL / ELT pipelines using Apache Spark, Python, Scala, or SQL. Experience building ETL / ELT pipelines, data cleansing, and data transformation processes.

Data Technologies : Proficiency in setting up, configuring, and managing Azure Databricks clusters, as well as optimizing performance and costs. Understanding of how to manage, process, and store big data in Azure Data Lake or other storage solutions.

Technical Leadership : Guide development teams, data engineers, and business analysts on best practices for data modeling, data processing, and optimization within the Databricks platform.

Data Integration : Design and build data integration solutions to connect diverse data sources, including on-premises systems, cloud-based applications, and APIs.

Performance Optimization : Monitor and optimize data processing performance on Databricks clusters to ensure efficient data processing at scale.

Security and Governance : Implement data security measures and access controls to protect sensitive data within the Azure Databricks environment. Knowledge of networking concepts within Azure, such as virtual networks, firewalls, private endpoints, and network security groups. Ensure compliance with security and governance best practices, including RBAC, encryption, and data masking.

Collaboration : Work closely with data engineers, data scientists, business intelligence analysts, and business stakeholders to understand requirements and translate them into technical solutions.

Compliance

Adhere to company policy and procedure, complete annual compliance training, and comply with business continuity and vendor management policy. These are crucial to the success of the organization and have regulatory compliance and / or safety and soundness for the sustainability of the credit union.

Comply with all Anti Money Laundering laws such as the Bank Secrecy Act, OFAC and USA PATRIOT Act and all related policies and procedures including accurate Member / Customer Identification Program, member due diligence, accurate OFAC match processing, and reporting unusual activities and suspected fraud.

SUPERVISORY RESPONSIBILITIES

Effectively lead data teams, manage complex data projects, and ensure proper implementation of architectural solutions.

Qualifications

EDUCATION and / or EXPERIENCE

Bachelor's degree and 8+ years related experience required, or equivalent combination of education and experience. Credit union or financial institution experience highly preferred.

• 8+ years of experience in big data architecture and cloud solutions, preferably on Azure
• Experience using Databricks and expertise in Azure services such as Azure Data Factory, Azure Data Lake, Azure Synapse, Azure SQL, Azure Functions, and Apache Spark.
• Dax Programming for Power BI, CSharp.net preferred.
• Experience in a financial institution is highly preferred.
• Experience using Correlation Keystone is preferred, but not necessary.
• Proficient in Azure Data Factory (ADF), Azure Databricks, Azure Synapse Analytics, Aure SQL Database, Azure Data Lake Storage (ADLS), and Azure Blob Storage, Databricks Notebooks, Spark SQL, PySpark, and Scala, Databricks Workspaces, Unity Catalog, and role-based access (RBAC)
• Expertise in Azure data services like Azure Data Lake Storage, Azure Cosmos DB, Azure Synapse Analytics, and Azure Key Vault.
• Data Modeling - Strong skills in data modeling, including star / snowflake schemas, normalizations, denormalizations, and working with NoSQL and relational databases.
• Programming Languages - Proficiency in Python, SQL, Scala, or other relevant languages for data manipulation and analysis.

OTHER QUALIFICATIONS

BluPeak Credit Union is an Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex including sexual orientation and gender identity, national origin, disability, protected Veteran Status, or any other characteristic protected by applicable federal, state, or local law.

Employment may be contingent upon BluPeak Credit Union's receipt of an acceptable and job-related background check, drug screen and credit check, as applicable and permissible by law. BluPeak Credit Union is committed to working and providing reasonable accommodation to applicants with physical, mental disabilities and sincerely held religious beliefs. For more information, contact Human Resources.

Min

USD $130,000.00 / Yr.

Max

USD $150,000.00 / Yr.