Big Data & AI Engineer

Engagement Type

Contract

Responsibilities

• Design and develop data pipelines for agentic systems; develop robust data flows to handle complex interactions between AI agents and data sources.
• Train and fine-tune large language models (LLMs).
• Design and build data architecture, including databases and data lakes, to support various data engineering tasks.
• Develop and manage Extract, Load, Transform (ELT) processes to ensure data is accurately and efficiently moved from source systems to analytical platforms used in data science.
• Implement data pipelines that facilitate feedback loops, allowing human input to improve system performance in human-in-the-loop systems.
• Work with vector databases to store and retrieve embeddings efficiently.
• Collaborate with data scientists and engineers to preprocess data, train models, and integrate AI into applications.
• Optimize data storage and retrieval for high performance.
• Conduct statistical analysis to identify trends and patterns and create data formats from multiple sources.

Qualifications

Required / Desired Skills

Skill Required / Desired Amount of Experience Understanding the Big data technologies Required 5 Years Experience developing ETL and ELT pipelines Required 5 Years Experience with Spark, GraphDB, Azure Databricks Required 5 Years Experience training LLMs with structured and unstructured data sets Required 4 Years Experience in Data Partitioning and Data conflation Required 3 Years Experience with GIS spatial data Required 3 Years

Skills :

ETL and ELT pipelines,GraphDB, Azure Databricks