Job Description
Job Description
East Tennessee Department of Energy Facility is looking for a SIEM Engineer who would be responsible for the process of acquiring, importing, and preparing data from various sources into a data storage or processing system. This role is critical in ensuring that data is available, accurate, and ready for analysis. The specific duties and responsibilities include :
1.
- Data Acquisition :
- Identifying and connecting to various data sources (e.g., databases, APIs, file systems, streaming data sources).
- Developing and maintaining data pipelines to automate the ingestion of data from these sources.
2.
Data Transformation and Cleaning :Processing and transforming raw data into a usable format.Cleaning and validating data to ensure quality and consistency.Handling data format conversions, data parsing, and enrichment.3.
Data Integration :Integrating data from multiple sources to create a cohesive data set.Ensuring that data from different sources is correctly aligned and merged.4.
ETL Processes :Designing and implementing ETL (Extract, Transform, Load) processes to move data from source systems to target storage systems.Monitoring and optimizing ETL workflows to ensure efficiency and reliability.5.
Data Storage :Ensuring data is stored securely and can be accessed efficiently by downstream applications and users.6.
Performance Monitoring and Optimization :Monitoring data ingestion processes for performance and reliability.Troubleshooting and resolving any issues that arise during data ingestion.Optimizing data pipelines to handle increasing volumes of data and ensure scalability.7.
Documentation and Compliance :Documenting data ingestion processes, data sources, and data transformations.Ensuring compliance with data governance policies and regulatory requirements.Skills and Tools Commonly Used by Data Ingest Engineers :
Proficiency in programming languages such as Python, Java, or Scala.Experience with data integration and ETL tools like Apache NiFi, Talend, or Informatica.Familiarity with big data technologies such as Apache Kafka, Apache Spark, or Hadoop.Understanding of cloud platforms and services (e.g., AWS, Azure, Google Cloud).Strong problem-solving and analytical skills.