Big data engineer Jobs in West Palm Beach, FL

We are seeking an experienced AI / LLM Data Engineer to build and maintain the data pipeline for our Generative AI platform. The ideal candidate will be well-versed in the latest Large Language Model (LLM) technologies and have a strong background in data engineering, with a focus on Retrieval-Augmented Generation (RAG) and knowledge-base techniques.  This role sits in the AI COE within DX Tech & Digital. As a AI / LLM Data Engineer (you will report into the Director, AI Solutions & Development who oversees the AI COE.

You will work on highly visible strategic projects, collaborating with cross-functional teams

to define requirements and deliver high-quality AI solutions.

The ideal candidate will have a passion for Generative AI and LLMs, with a proven track record of delivering innovative AI applications.

Responsibilities

• Design, implement, and maintain an end-to-end multi-stage data pipeline for LLMs, including Supervised Fine Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF) data processes
• Identify, evaluate, and integrate diverse data sources and domains to support the Generative AI platform
• Develop and optimize data processing workflows for chunking, indexing, ingestion, and vectorization for both text and non-text data
• Benchmark and implement various vector stores, embedding techniques, and retrieval methods
• Create a flexible pipeline supporting multiple embedding algorithms, vector stores, and search types (e.g., vector search, hybrid search)
• Implement and maintain auto-tagging systems and data preparation processes for LLMs
• Develop tools for text and image data crawling, cleaning, and refinement
• Collaborate with cross-functional teams to ensure data quality and relevance for AI / ML models
• Work with data lake house architectures to optimize data storage and processing
• Integrate and optimize workflows using Snowflake and various vector store technologies

Requirements

Preferred Skills

Benefits