Actively seeking opportunities

Animesh Singh

Software Engineer (Data)

AWS Certified Data Engineer – Associate

iamanimesh11june@gmail.com

github.com/iamanimesh11 linkedin.com/in/animesh-singh11 Download Resume

iamanimesh11june@gmail.com

github.com/iamanimesh11 linkedin.com/in/animesh-singh11 Download Resume

Animesh Singh

Software Engineer (Data & AI )

AWS Certified Data Engineer – Associate

My Focus

I’ve always been curious about what truly powers AI. Beyond the models and algorithms, I realized its real strength lies in the data foundation. Data engineering isn’t just support for AI - it’s the infrastructure that determines whether intelligent systems succeed or fail.

That realization led me to dive deep into modern data systems - building reliable ETL pipelines, designing scalable architectures, and working with distributed platforms that power real-time and intelligent applications. I’m especially interested in how strong data infrastructure enables technologies like RAG, vector databases, and intelligent workflows to operate at scale.

I focus on designing systems where data flows reliably from source to decision. Strong data foundations don’t just enable AI - they empower organizations to make confident, data-driven decisions today while unlocking intelligence for tomorrow.

Work Experience

Software Developer (Data & Automation)

April 2024 – Present

LG Electronics India, Noida, India

Built and deployed ETL scripts integrated with SQL Server and internal APIs to automate daily build reporting, improving reporting speed by 70% and enhancing data accuracy across departments.
Optimized LG OBS (Online Business System) by automating order ID submissions using legacy tool UiPath and Python, reducing manual effort by 90% and eliminating 300+ repetitive entries daily.
Led automation of high-volume factory workflows, implementing scheduled job orchestration and validation frameworks, achieving an 80% boost in operational efficiency and maintaining 99% process uptime.
Enhanced data reliability and error traceability by developing custom validation, logging, and exception-handling modules, cutting failure rates by 40% across automated pipelines.

Education

Bachelor of Technology in Computer Science Engineering

Galgotias University, Greater Noida, UP, India

2020 - 2024

CGPA: 8.0/10
Major: Computer Science Engineering

Technical Skills

Languages

Python, SQL, .NET, Shell Scripting

Big Data Frameworks

Apache Spark, Databricks

Databases

PostgreSQL, SQL Server, Firebase

Cloud Platforms

Snowflake, Microsoft Azure, AWS S3, EC2, RDS

Workflow / Orchestration

Apache Airflow, AWS Step Functions, Azure Data Factory

DevOps / Monitoring

Docker, Grafana Loki

Messaging / Streaming

Kafka, MQTT, Redis

Resume

Download Resume

Projects

Production Ready

Real-Time IOT ETL for Predictive Maintenance

Updating

Scalable event-driven architecture with real-time stream processing, distributed ETL workflows, and ML inference pipelines for predictive maintenance analytics.

GitHub

Production Ready

Real-Time ETL Road Traffic & Weather Monitoring

Completed

Microservices-based architecture with orchestrated ETL pipelines, multi-source data integration, and real-time stream analytics for intelligent traffic optimization.

GitHub

Real-Time IOT ETL for Predictive Maintenance

Problem Statement

Traditional maintenance approaches are reactive, leading to unexpected equipment failures, costly downtime, and inefficient resource allocation. Organizations need a proactive solution to predict and prevent equipment failures before they occur.

Solution

Built an end-to-end real-time IOT data pipeline that ingests sensor data from multiple devices, processes it through scalable ETL workflows, applies machine learning models for predictive analytics, and provides real-time alerts and visualizations for maintenance teams.

Business Impact

Reduced unplanned downtime by 40% through predictive alerts
Decreased maintenance costs by 25% via optimized scheduling
Improved equipment lifespan by 30% with proactive care
Real-time monitoring of 1000+ IOT devices across facilities
Processing 10K+ events per second with sub-second latency

Technical Architecture

MQTT protocol for efficient IOT device communication
Apache Kafka for real-time stream processing
AWS S3 + Glue for data lake and ETL orchestration
PySpark for distributed data processing
ML models for anomaly detection and failure prediction
Real-time dashboards with Grafana and Prometheus

Python Apache Kafka PySpark AWS S3 AWS Glue MQTT Docker Grafana

View Live Project View on GitHub

Real-Time ETL Road Traffic & Weather Monitoring

Problem Statement

Urban traffic management systems lack real-time, integrated data combining traffic conditions with weather patterns. This results in poor routing decisions, delayed emergency responses, and inefficient traffic flow management.

Solution

Developed a comprehensive real-time ETL pipeline that integrates traffic APIs and weather data sources, processes millions of data points, and delivers actionable insights through interactive dashboards for traffic management and route optimization.

Business Impact

Reduced average commute time by 15% with optimized routing
Improved emergency response time by 20% with real-time alerts
Enhanced traffic flow efficiency by 25% through predictive analysis
Monitoring 500+ traffic sensors and weather stations
Processing 5K+ API calls per minute with 99.9% uptime

Technical Architecture

REST API integration for traffic and weather data sources
Apache Airflow for ETL workflow orchestration
AWS Lambda for serverless data processing
Snowflake data warehouse for analytics
Real-time streaming with AWS Kinesis
Interactive dashboards with Tableau

Python Apache Airflow AWS Lambda AWS Kinesis Snowflake PostgreSQL Tableau Docker

View Live Project View on GitHub

Certificates & Badges

AWS Certified Data Engineer - Associate

Completed a comprehensive learning path on AWS data engineering, covering end-to-end domains such as batch and real-time data ingestion, scalable ETL/ELT pipelines, data lakes and data warehouses, data modeling and analytics enablement, streaming architectures, and cloud-native security, monitoring, reliability, and cost optimization. Gained hands-on experience designing and operating production-grade data solutions on AWS.

View Certificate

Knowledge Badge - Data Ingestion with Lakeflow Connect

Earned the Data Ingestion with Lakeflow Connect badge by implementing batch, incremental, and streaming ingestion, leveraging Auto Loader, COPY INTO, and MERGE INTO, and managing scalable, production-ready data ingestion workflows in Databricks.

View Certificate

Knowledge Badge - Build Data Pipelines with Lakeflow Spark Declarative Pipelines

Earned the Lakeflow Spark Declarative Pipelines badge by building incremental batch and streaming pipelines, implementing CDC/SCD workflows, and managing production data pipelines in Databricks.

View Certificate

Master Retrieval-Augmented Generation (RAG)

Completed a comprehensive learning path on RAG, covering key topics like vector databases, fine-tuning models, and integrating large language models (LLMs). Gained hands-on experience building RAG solutions using LlamaIndex, enhancing AI systems with real-time data retrieval for improved performance.

View Certificate

Docker Foundations Professional Certificate

Validates core expertise in Docker containerization, including building, deploying, and managing container-based applications using industry-standard Docker tools and workflows.

View Certificate

Hands-On Essentials: Data Warehouse

Earned by demonstrating foundational knowledge and practical skills in Snowflake's data warehousing capabilities.

View Certificate

Hands-On Essentials: Data Lake Workshop

Gained hands-on experience with building and managing data lakes on Snowflake, including ingesting, organizing, and querying large datasets.

View Certificate

Hands-On Essentials: Data Engineering Workshop

Learned core data engineering skills in Snowflake, including data modeling, pipeline design, and best practices for scalable data workflows.

View Certificate

Hands-On Essentials: Data Application Builders Workshop

Explored Snowflake collaboration features, data marketplace integration, and cost estimation strategies for optimized data operations.

View Certificate

Blog

Thoughts, learnings, and technical deep-dives

Loading blog posts...