Loading…
Data Engineering
A collection of real-world Data Engineering experience - from building pipelines and migrating data warehouses, to leading teams and building industry-scale data platforms.
Diagrams on this page are illustrative recreations of systems I built — production dashboards and client data are omitted for confidentiality (NDA).
Overview of cross-industry data engineering patterns covered in these use cases
Use Cases
Scalable Data Collection for HORECA Market
Agri-techIn agritech HORECA, fast and massive data collection is required with distributed sources and no official APIs.
Data Pipeline Optimization & Cost Reduction
Agri-techOptimized data pipelines in an agritech environment to reduce BigQuery costs and improve performance through data usage audits, real-time vs batch segmentation, and query and architecture efficiency improvements.
Data Warehouse Migration to Hadoop Ecosystem
HealthcareLed migration from Oracle Data Warehouse to Hadoop Ecosystem at one of Indonesia's largest laboratory companies, without disrupting existing analytical workflows.
Building a Data Engineering Team from Scratch
Agri-techTransformed 5 System Analysts into productive Data Engineers while keeping operations running in a fast-paced agritech environment.
Industrial Data Platform for Mining Operations
MiningBuilt an end-to-end data platform on the Hadoop ecosystem to integrate SCADA, Kafka, and external data sources for mining operational analytics.
Scalable Data Collection for HORECA Market
The Story
The HORECA sector — hotels, restaurants, and cafés — holds enormous commercial data, but none of it is available through official APIs. The business team needed comprehensive market coverage, fast. I designed a distributed web crawling system using Python and Selenium, built to run across multiple devices simultaneously using multi-threading. Data flowed directly into PostgreSQL with consistent schema and deduplication. What used to take days completed in hours. The output — clean, structured CSV exports — went straight into the canvassing team's workflow.
In agritech HORECA, fast and massive data collection is required with distributed sources and no official APIs.
📊 Impact
- Data collection time reduced dramatically (days → hours)
- HORECA coverage increased
- Business canvassing becomes faster
- Opens pathway for future automation and analytics
🧩 Tech Stack
Python, Selenium, Threading/Concurrent Execution, PostgreSQL, CSV Export Pipeline
⚡ Problem Statement
- Collect HORECA business data quickly
- Maintain consistency & data quality
- Scale up without single-device bottleneck
- Output ready for business use (CSV / database)
🧠 Solution Overview
- Python + Selenium for dynamic website crawling
- Multi-threading + multi-device execution for scaling
- PostgreSQL as centralized data store
- CSV export pipeline for operations
🏗️ Architecture
- Data Crawling Layer: Selenium WebDriver + AJAX/lazy load handling
- Concurrency Strategy: multi-thread & multi-device task partitioning
- Data Pipeline: store results to PostgreSQL, schema for business fields
- Export Layer: CSV output for canvassing & analytics
🔥 Challenges & Solutions
- Anti-bot & dynamic content: adaptive delays, random interaction, scroll simulation
- Performance: multi-threading + horizontal multi-device
- Consistency: data validation + unique constraints in PostgreSQL
- Reliability: retry mechanism + logging
Data Pipeline Optimization & Cost Reduction
Optimize data pipeline in Agritech to lower cost and improve performance.
📊 Impact
- Cost reduction up to ~45%
- Query performance improved significantly
- Architecture more scalable and maintainable
- Reduced unnecessary compute/storage
🧩 Tech Stack
Python, BigQuery, Fivetran, Apache Airflow, Advanced SQL
⚡ Problem Statement
- BigQuery cost grows uncontrolled
- Redundant pipelines & scheduled queries
- Overuse real-time ingestion (not all need 5-minute sync)
- Suboptimal queries (large scans without filtering/clustering)
🧠 Solution Strategy
- Data usage audit with BigQuery INFORMATION_SCHEMA queries
- Pipeline segmentation: real-time (critical) vs batch (non-critical)
- BigQuery tuning: partitioning, clustering, pre-aggregation
- Remove redundant scheduled queries and improve SQL efficiency
🏗️ Architecture
- BigQuery as main warehouse
- Fivetran for critical tables (operational dashboard)
- Airflow + Python for non-critical batch pipelines
- Monitoring and cost control via usage queries and alerts
🔥 Challenges & Solutions
- BigQuery & storage cost without governance
- Redundant queries and overlapping workflow
- Non-scalable pipeline due to speed-first implementation
Data Warehouse Migration to Hadoop Ecosystem
I led as Lead Data Engineer on a data warehouse migration from Oracle to the Hadoop Ecosystem at one of Indonesia's largest laboratory companies, without disrupting existing analytical workflows.
📊 Impact
- Data warehouse successfully migrated to Hadoop ecosystem with zero data loss
- Query performance improved for large-scale processing
- More efficient in storage & compute
- Data lineage & governance more structured
- Strong foundation for future data platform (analytics & ML)
🧩 Tech Stack
Oracle Database, Hadoop (HDFS), Hive, Parquet, Oozie, Sqoop/NiFi
⚡ Problem Statement
- Maintain query result consistency between Oracle and Hive
- Convert complex SQL including cube logic
- Build a reliable pipeline from source to Hadoop
- Ensure data validity at every layer (staging → curated → cube)
- Provide clear data lineage & documentation
🧠 Solution Overview
- Extract data from Oracle, load to HDFS in Parquet format
- Rebuild query logic in Hive from Oracle SQL
- Orchestrate pipelines using Oozie
- Validate data via automated QA checks at each layer
- Comprehensive data lineage and structure documentation
🏗️ Architecture
- Data Ingestion Layer: Sqoop/custom tool → HDFS (Parquet, landing zone) for storage & query performance optimization
- Transformation Layer: Convert Oracle SQL → Hive SQL (aggregation, joins, cube queries)
- Pipeline Orchestration: Oozie for scheduling, dependency management, and workflow automation
- Data Validation & QA: Row count, aggregation check, Oracle vs Hive comparison at each layer
- Data Lineage & Documentation: Source → transformation → output, table & field mapping, inter-dataset dependencies
🔥 Challenges & Solutions
- Oracle vs Hive SQL dialect differences: manual rewrite + pattern-based conversion + aggregation validation
- Large data volume: Parquet format + Hive partitioning optimization
- Data consistency: QA checks at every layer, not just the final output
- Pipeline reliability: Oozie dependency & retry handling + logging for traceability
Building a Data Engineering Team from Scratch
During a scaling phase in agritech, I led the build-up of an internal Data Engineering capability - transforming 5 System Analysts into productive Data Engineers, while ensuring operations continued without disruption.
📊 Impact
- Successfully built a Data Engineering team from scratch
- Data pipelines became structured using the Medallion Architecture approach
- Operations continued running without major disruption
- Individual capabilities within the team grew significantly
- 🏆 Highlight: All mentored team members successfully landed Data Engineer roles at their next companies - a clear indicator of the mentoring system's effectiveness and industry-relevant skill development
🧩 Tech & Concepts
Data Engineering Fundamentals, Medallion Architecture (Bronze & Silver), Data Pipeline Design, Data Quality Validation, Schema Migration Handling, Mentoring & Team Building
⚡ Problem Statement
- No dedicated Data Engineering team existed
- 5 System Analysts with limited data exposure
- Growing demand for data pipelines, data quality, and governance
- Team also had to support operational data changes under high pressure
🧠 Approach
- Learning by Doing 70:30 - 70% project-based, 30% foundational knowledge
- Structured Knowledge Validation with pre-test & post-test to measure real improvement
- Medallion Architecture implementation (Bronze & Silver Layer) to standardize pipelines
- Dual role: Data Engineering + Data Operations - balancing pipeline building & operational support
- Intensive mentoring using real use cases, not just theory
🏗️ Architecture & Responsibilities
- Bronze Layer: raw data ingestion with minimal transformation (landing zone)
- Silver Layer: cleaned & structured data, ready for downstream usage (analytics & reporting)
- Pipeline Design: standardized ingestion → transformation → validation flow
- Schema Migration Handling: up/down migrations & data corrections to support application-side changes
- Data Quality Validation: layer-based checks from Bronze to Silver
🔥 Key Challenges & Solutions
- Skill Gap (System Analyst → Data Engineer): addressed with hands-on projects, intensive mentoring, and real use cases
- Context Switching (Operational vs Engineering): addressed with clear task prioritization and focus allocation within the team
- Data Quality in a Dynamic Environment: addressed with layer-based validation and pipeline standardization
Industrial Data Platform for Mining Operations
In the mining industry, operations rely heavily on real-time data from SCADA systems. I built an end-to-end data platform on the Hadoop ecosystem to integrate multi-source data and deliver analytical cubes that support operational decision making.
📊 Impact
- Integrated SCADA, streaming, and external data into one centralized platform
- Gained insights into production drop factors (weather impact)
- Violation & driver fatigue identification became more structured and actionable
- Solid data platform foundation for future industrial analytics
🧩 Tech Stack
Apache NiFi, Apache Kafka, Hadoop (HDFS), Hive, SCADA Integration, BMKG API, Analytical Cube Design, Virtual Machine Infrastructure
⚡ Problem Statement
- Data scattered across many sources: SCADA, Kafka streaming, external APIs, and BMKG weather data
- No integration exists for analysis and decision making
- Need to support operational use cases: violation analysis, driver fatigue monitoring, weather impact on production
- System must remain stable in a constrained environment (VM + isolated infrastructure)
🧠 Solution Overview
- Unified ingestion via Apache NiFi: consume Kafka topics, pull API data, routing & lightweight transformation
- Centralized storage in HDFS (vanilla Hadoop) deployed on Virtual Machines
- Data modeling as analytical cubes per operational use case
- Pipeline orchestration focused on reliability and flexibility in a constrained environment
🏗️ Architecture
- Data Sources: SCADA (conveyor metrics), Kafka (streaming events), BMKG (rainfall data), operational APIs
- Ingestion Layer: Apache NiFi as unified ingestion — centralized control, multi-source, easy flow monitoring
- Storage Layer: HDFS on vanilla Hadoop within isolated VM architecture
- Processing & Modeling: Analytical cube for Violation Analysis (detecting operational violations from sensor & event data)
- Processing & Modeling: Analytical cube for Driver Fatigue Monitoring (driver activity patterns + event data)
- Processing & Modeling: Analytical cube for Weather Impact Analysis (correlating rainfall with production drops)
🔥 Key Challenges & Solutions
- Multi-source integration: centralized ingestion in NiFi with standardized data flows
- Real-time vs Batch combination: Kafka for streaming, API for batch, unified in one pipeline
- Resource limitation (VM-based Hadoop): storage & processing optimization, critical pipeline prioritization
- Multi-domain data correlation (SCADA + weather + operational): use-case-driven data modeling with cross-domain joins
🧠 AI Pipeline & LLM
Growing FocusBuilding retrieval pipelines and AI-ready data infrastructure — a natural extension of data engineering into the LLM ecosystem.
RAG (Retrieval-Augmented Generation)
Context retrieval architecture for accurate LLM output
Vector Database
Embedding storage & similarity search (Qdrant, ChromaDB)
Text Chunking Strategy
Document splitting strategies for optimal retrieval quality
Embedding Pipeline
Text-to-vector transformation for semantic search
LLM Integration
Connecting data pipelines to language models (Ollama, OpenAI)
Explore other domains
To go back to profile, click here.