Talent.com
Data Engineer - AI & ML
Data Engineer - AI & MLTEPHRA • San Francisco, CA, United States
Data Engineer - AI & ML

Data Engineer - AI & ML

TEPHRA • San Francisco, CA, United States
30+ days ago
Job type
  • Full-time
Job description

Description :

Location : San Francisco, CA

Responsibilities :

1.Design and Build Data Pipelines :

  • Develop, construct, test, and maintain data pipelines to extract, transform, and load (ETL) data from various sources to data warehouses or data lakes.
  • Ensure data pipelines are efficient, scalable, and maintainable, enabling seamless data flow for downstream analysis and modeling.
  • Work with stakeholders to identify data requirements and implement effective data processing solutions.

2. Data Integration :

  • Integrate data from multiple sources such as internal databases, external APIs, third-party vendors, and flat files.
  • Collaborate with business teams to understand data needs and ensure data is structured properly for reporting and analytics.
  • Build and optimize data ingestion systems to handle both real-time and batch data processing.
  • 3. Data Storage and Management :

  • Design and manage data storage solutions (e.g., relational databases, NoSQL databases, data lakes, cloud storage) that support large-scale data processing.
  • Implement best practices for data security, backup, and disaster recovery, ensuring that data is safe, recoverable, and complies with relevant regulations.
  • Manage and optimize storage systems for scalability and cost efficiency.
  • 4. Data Transformation :

  • Develop data transformation logic to clean, enrich, and standardize raw data, ensuring it is suitable for analysis.
  • Implement data transformation frameworks and tools, ensuring they work seamlessly across different data formats and sources.
  • Ensure the accuracy and integrity of data as it is processed and stored.
  • 5. Automation and Optimization :

  • Automate repetitive tasks such as data extraction, transformation, and loading to improve pipeline efficiency.
  • Optimize data processing workflows for performance, reducing processing time and resource consumption.
  • Troubleshoot and resolve performance bottlenecks in data pipelines.
  • 6. Collaboration with Data Teams :

  • Work closely with Data Scientists, Analysts, and business teams to understand data requirements and ensure the correct data is available and accessible.
  • Assist Data Scientists with preparing datasets for model training and deployment.
  • Provide technical expertise and support to ensure the integrity and consistency of data across all projects.
  • 7. Data Quality Assurance :

  • Implement data validation checks to ensure data accuracy, completeness, and consistency throughout the pipeline.
  • Develop and enforce data quality standards to detect and resolve data issues before they affect analysis or reporting.
  • Monitor and improve data quality by identifying areas for improvement and implementing solutions.
  • 8. Monitoring and Maintenance :

  • Set up monitoring and logging for data pipelines to detect and alert for issues such as failures, data mismatches, or delays.
  • Perform regular maintenance of data pipelines and storage systems to ensure optimal performance.
  • Update and improve data systems as required, keeping up with evolving technology and business needs.
  • 9. Documentation and Reporting :

  • Document data pipeline designs, ETL processes, data schemas, and transformation logic for transparency and future reference.
  • Create reports on the performance and status of data pipelines, identifying areas of improvement or potential issues.
  • Provide guidance to other teams regarding the usage and structure of data systems.
  • 10. Stay Updated with Technology Trends :

  • Continuously evaluate and adopt new tools, technologies, and best practices in data engineering and big data systems.
  • Participate in industry conferences, webinars, and training to stay current with emerging trends in data engineering and cloud computing.
  • Qualifications : (Please list all required qualifications) Click here to enter text.

    (Rationalizes basic requirements for candidates to apply. Helps w / rationalization when

    Requirements : -

  • Minimum of 7 years of total experience
  • 1.Educational Background :

    Bachelor's or Master's degree in Computer Science, Information Technology, Data Engineering, or a related field

    2.Technical Skills :

  • Proficiency in programming languages such as Python, Java, or Scala for data processing.
  • Strong knowledge of SQL and relational databases (e.g., MySQL, PostgreSQL, MS SQL Server).
  • Experience with NoSQL databases (e.g., MongoDB, Cassandra, HBase).
  • Familiarity with data warehousing solutions (e.g., Amazon Redshift, Google BigQuery, Snowflake).
  • Hands-on experience with ETL frameworks and tools (e.g., Apache NiFi, Talend, Informatica, Airflow).
  • Knowledge of big data technologies (e.g., Hadoop, Apache Spark, Kafka).
  • Experience with cloud platforms (AWS, Azure, Google Cloud) and related services for data storage and processing.
  • Familiarity with containerization and orchestration tools (e.g., Docker, Kubernetes) for building scalable data systems.
  • Knowledge of version control systems (e.g., Git) and collaboration tools (e.g., Jira, Confluence).
  • Understanding data modeling concepts (e.g., star schema, snowflake schema) and how they relate to data warehousing and analytics.
  • Knowledge of data lakes, data warehousing architecture, and how to design efficient and scalable storage solutions.
  • 3.Soft Skills :

  • Strong problem-solving skills with an ability to troubleshoot complex data issues.
  • Excellent communication skills, with the ability to explain technical concepts to both technical and non-technical stakeholders.
  • Strong attention to detail and a commitment to maintaining data accuracy and integrity.
  • Ability to work effectively in a collaborative, team-based environment.
  • 4.Experience :

  • 3+ years of experience in data engineering, with hands-on experience in building and maintaining data pipelines and systems.
  • Proven track record of implementing data engineering solutions at scale, preferably in large or complex environments.
  • Experience working with data governance, compliance, and security protocols.
  • 5.Preferred Qualifications

  • Experience with machine learning and preparing data for AI / ML model training.
  • Familiarity with stream processing frameworks (e.g., Apache Kafka, Apache Flink).
  • Certification in cloud platforms (e.g., AWS Certified Big Data - Specialty, Google Cloud Professional Data Engineer).
  • Experience with DevOps practices and CI / CD pipelines for data systems.
  • Experience with automation and orchestration tools (e.g., Apache Airflow, Luigi).
  • Familiarity with data visualization and reporting tools (e.g., Tableau, Power BI) to support analytics teams
  • 6.Work Environment :

  • Collaborative and fast-paced work environment.
  • Opportunity to work with state-of-the-art technologies.
  • Supportive and dynamic team culture
  • #LI-AD1

    Create a job alert for this search

    Ai Ml Engineer • San Francisco, CA, United States

    Related jobs
    AI / ML Engineer

    AI / ML Engineer

    Air Apps • San Francisco, CA, United States
    Full-time
    At Air Apps, we believe in thinking bigger-and moving faster.We're a family-founded company on a mission to create the world's first AI-powered Personal & Entrepreneurial Resource Planner (PRP), an...Show more
    Last updated: 30+ days ago • Promoted
    AI / ML Engineer

    AI / ML Engineer

    Reflective Inc • San Francisco, CA, United States
    Full-time
    Sunlight reflection may be the only available option, alongside dramatic emissions reductions, adaptation, and rapid scaling of carbon removal, to rapidly limit many climate impacts over the coming...Show more
    Last updated: 30+ days ago • Promoted
    Principal AI, ML Engineer

    Principal AI, ML Engineer

    Connvertex Technologies Inc. • San Francisco, California, USA
    Full-time
    Portland ME; Boston MA; Chicago IL; San Francisco CA (Locals only 20-40miles).Hybrid (1 3 days onsite per week).Have 15-20 years of software design and development experience at a large scale.Lea...Show more
    Last updated: 11 days ago • Promoted
    AI / ML Engineer

    AI / ML Engineer

    Brainco • San Francisco, CA, United States
    Full-time
    Applied AI startup founded by Elad Gil and Jared Kushner, and backed by many of Silicon Valley's leading builders - including Patrick Collison (CEO of Stripe), Andrej Karpathy (Cofounder of OpenAI)...Show more
    Last updated: 30+ days ago • Promoted
    Data Engineer AI & ML

    Data Engineer AI & ML

    Diverse Lynx • San Francisco, CA, United States
    Full-time
    Design, build, and train machine learning models using appropriate algorithms (e.Use various machine learning and AI frameworks (TensorFlow, PyTorch, Scikit-learn, Keras, etc.Experiment with differ...Show more
    Last updated: 16 days ago • Promoted
    AI / ML Engineer

    AI / ML Engineer

    Alation • Redwood City, CA, United States
    Full-time
    Alation is pioneering the next era of.Organizations around the world rely on.Our technology helps enterprises connect people and data to make faster, smarter, and more confident decisions.With more...Show more
    Last updated: 8 days ago • Promoted
    AI / ML Engineer - Known

    AI / ML Engineer - Known

    Pear VC • San Francisco, CA, United States
    Full-time
    You'll be the technical founder driving the machine learning and AI backbone behind.You'll design and ship the systems that make Known feel magical : personalized matching algorithms, adaptive recom...Show more
    Last updated: 7 days ago • Promoted
    ML / AI Engineers

    ML / AI Engineers

    Adidev Technologies • Oakland, California, United States
    Remote
    Full-time
    This role is open to US Citizens, Green Card holders, GC-EAD only.Adidev is looking for an adept Machine Learning Engineer to take the helm in deploying advanced machine learning models, with a spe...Show more
    Last updated: 30+ days ago • Promoted
    Founding AI / ML Engineer - LLMs, MLOps & Predictive Analytics

    Founding AI / ML Engineer - LLMs, MLOps & Predictive Analytics

    SherlockTalent • San Francisco, CA, US
    Full-time
    Join a pioneering team at a leading company as an AI / ML & Data Ingestion Engineer.You will engage in developing cutting-edge ML models and integrating advanced technologies while driving impact...Show more
    Last updated: 6 days ago • Promoted
    Senior ML / AI Engineer

    Senior ML / AI Engineer

    Reacher • San Francisco, California, United States
    Full-time
    Under Armour, Hanes, HeyDude, and Logitech scale their affiliate marketing.Youtube Shopping, Instagram Shopping, Shopify, Amazon). We're building key infrastructure for the creator economy and imple...Show more
    Last updated: 9 days ago • Promoted
    AI / ML Engineer

    AI / ML Engineer

    Known • San Francisco, California, United States
    Full-time
    You’ll be the technical founder driving the machine learning and AI backbone behind.You’ll design and ship the systems that make Known feel. LLM-based interactions that help users connect meaningful...Show more
    Last updated: 9 days ago • Promoted
    AI / ML Platform Engineer

    AI / ML Platform Engineer

    Whatnot • San Francisco, CA, United States
    Full-time
    Join the Future of Commerce with Whatnot!.Whatnot is the largest live shopping platform in North America and Europe to buy, sell, and discover the things you love. We’re re-defining e-commerce by bl...Show more
    Last updated: 7 days ago • Promoted
    Staff AI / ML Analytics Engineer

    Staff AI / ML Analytics Engineer

    Okta, Inc. • San Francisco, CA, United States
    Full-time
    Okta is The World's Identity Company.We free everyone to safely use any technology, anywhere, on any device or app.Our flexible and neutral products, Okta Platform and Auth0 Platform, provide secur...Show more
    Last updated: 16 days ago • Promoted
    Senior AI / ML Engineer - Product-Driven AI Workflows

    Senior AI / ML Engineer - Product-Driven AI Workflows

    TeamEx Inc. • San Francisco, CA, United States
    Full-time
    A technology company is seeking an AI Engineer to design and implement user-facing AI products.This role requires a blend of strong engineering and product thinking. Responsibilities include develop...Show more
    Last updated: 5 days ago • Promoted
    Data & AI / ML Engineer — Forecasting & GenAI

    Data & AI / ML Engineer — Forecasting & GenAI

    Verse Inc. • San Francisco, CA, United States
    Full-time
    A pioneering clean energy technology firm is seeking a Staff Software Engineer focused on Data & AI / ML to drive the development of data applications and tools. The ideal candidate will have 8+ years...Show more
    Last updated: 1 day ago • Promoted
    AI / ML Engineer

    AI / ML Engineer

    Brain Co. • San Francisco, CA, United States
    Full-time
    Applied AI startup founded by Elad Gil and Jared Kushner, and backed by many of Silicon Valley’s leading builders — including Patrick Collison (CEO of Stripe), Andrej Karpathy (Cofounder of OpenAI)...Show more
    Last updated: 30+ days ago • Promoted
    AI / ML Engineer

    AI / ML Engineer

    Rulebase • San Francisco, CA, United States
    Full-time
    This range is provided by Rulebase.Your actual pay will be based on your skills and experience — talk with your recruiter to learn more. We are building an autonomous factory for financial service a...Show more
    Last updated: 10 days ago • Promoted
    AI Engineer

    AI Engineer

    Confidential • San Francisco, California, United States
    Remote
    Full-time
    This role is open to US Citizens, Green Card holders, GC-EAD only.Adidev is looking for an adept Machine Learning Engineer to take the helm in deploying advanced machine learning models, with a spe...Show more
    Last updated: 30+ days ago • Promoted