Design and maintain enterprise-scale data pipelines using AWS cloud services, handling schema evolution in data feeds and delivering analytics-ready datasets to BI platforms. This role requires hands-on expertise with the full AWS data stack and proven ability to build enterprise-grade data solutions that scale.
Essential Functions
- Build and orchestrate ETL / ELT workflows using Apache Airflow for complex data pipeline management
- Develop serverless data processing with AWS Lambda and EventBridge for real-time transformations
- Create scalable ETL jobs using AWS Glue with automated schema discovery and catalog management
- Execute database migrations and continuous replication using AWS DMS
- Design and optimize Amazon Redshift data warehouses and Amazon Athena federated queries
- Implement streaming data pipelines with Apache Kafka for real-time ingestion
- Manage schema changes in data feeds with automated detection and pipeline adaptation
- Create data feeds for Tableau and BusinessObjects reporting platforms
Supervisory Responsibilities
No supervisory responsibilitiesRequired Skills / Abilities
Airflow : DAG development, custom operators, workflow orchestration, production deploymentLambda : Serverless functions, event triggers, performance optimizationEventBridge : Event-driven architecture, rule configuration, cross-service integrationGlue : ETL job development, crawlers, Data Catalog, schema managementDMS : Database migrations, continuous replication, heterogeneous database integrationRedshift : Cluster management, query optimization, workload managementAthena : Serverless analytics, partitioning strategies, federated queriesTableau (Expert Level)
Develop and maintain data analogs, data cubes, queries, data visualization and reportsAssist in testing code, governance, data quality, and documentation effortReported data and visualizations and reportsCollaborate with data stewards to test, clean, and standardizeEvaluate patterns and meaningful insights from data through qualitative and quantitative analysisData Technologies
Apache Kafka : Stream processing, topic design, producer / consumer developmentSQL : Advanced querying across multiple database platforms (PostgreSQL, MySQL, Oracle)Python / Scala : Data processing, automation, custom pipeline developmentSchema Management : Automated detection, evolution strategies, backward compatibilityTableau : data modelingAnalytics Platforms
Tableau : Data source optimization, extract management, connection configurationBusinessObjects : Universe design, report development, data feed creationTableau : Design and Code rewriteEducation and Experience
5+ years AWS data platform development3+ years production Airflow experience with complex workflow orchestrationProven experience managing high-volume data feeds (TB+ daily) with schema evolutionDatabase migration expertise using DMS for enterprise-scale projectsBI integration experience with Tableau and BusinessObjects platformsTableau : 2 plus yearKey Competencies
Design fault-tolerant data pipelines with automated error handling and recoveryHandle schema changes in real-time and batch data feeds without pipeline disruptionOptimize performance across streaming and batch processing architecturesImplement data quality validation and monitoring frameworksCoordinate cross-platform data synchronization and lineage trackingPreferred Qualifications
AWS Data Analytics Specialty or Solutions Architect Professional certificationExperience with Infrastructure as Code (Terraform, CloudFormation)Knowledge of DataOps practices and CI / CD for data pipelinesContainerization experience (Docker, ECS, EKS) for data workloads