Sri Omkar D
Get in Touch
Open to Senior Data Engineer & ETL Roles

Engineering Reliable ETL Pipelines & Cloud Data Platforms.

I'm Sri Omkar D, a Senior Data Engineer with 7+ years of experience building enterprise ETL pipelines and leading cloud migration initiatives — IBM DataStage, SSIS, PySpark, and AWS-native data platforms.

View System Architectures GitHub LinkedIn Email Me

7+ Years

Enterprise Experience

25–40%

Runtime Latency Reduction

Multi-TB

Daily Pipeline Scale

100%

Reconciled SLA Integrity

01. Architectural Case Studies

Production data platforms — ETL modernization, streaming ingestion, and performance engineering. Every project answers: what breaks in production, and how was it designed to handle it?

Flagship Project
DATA PLATFORM Production System

Job Intelligence Data Platform — Ingestion to Serving

Engineered a production data platform ingesting high-volume job postings into PostgreSQL — schema design, dedup, fit scoring, and a JSON-RPC serving layer. The interesting parts are the failures it was designed around: provider quota exhaustion (multi-provider failover routing), bot-detection throttling (human-paced automation), and untrusted model output (human review gates before any write action).

Flow: Ingestion → PostgreSQL (Psycopg 3 Pool) → Fit Scoring → MCP API / Dashboard
PostgreSQL Psycopg 3 Starlette Model Context Protocol Docker GitHub Actions
1,300+ jobs in production · graceful degradation by design View Repo
ETL MODERNIZATION Legacy → Cloud

Legacy ETL Modernization: Ab Initio → AWS

Reverse-engineered legacy Ab Initio ETL workflows — mapping job dependencies and embedded business logic — and re-platformed them onto AWS-native services. Rebuilt transformations in PySpark, organized the S3 data lake across raw, cleansed, and curated layers, cataloged every asset in Glue Data Catalog, and held each pipeline to source-to-target reconciliation before decommissioning the legacy jobs.

Migration: Ab Initio Analysis → PySpark Rebuild → S3 Raw/Cleansed/Curated → Glue Catalog → Redshift
Ab Initio PySpark AWS S3 Glue Data Catalog Redshift Airflow
Legacy decommissioned · parity verified · audit-ready
STREAMING LAKEHOUSE Near-Real-Time

IoT Fleet Telemetry & Asset Engine

Ingested high-frequency JSON/XML telemetry payloads from 50,000+ rental units via AWS Kinesis. Orchestrated structured streaming via Databricks to feed S3 Delta Lake curated layers for operational Qlik Sense dashboards.

Pipeline: Kinesis → Databricks Streaming → S3 Curated → Qlik Sense
AWS Kinesis Databricks PySpark AWS S3
40% Faster executive dashboard refreshes
SPARK TUNING Performance Study

Spark Catalyst & Skew Mitigation

Benchmarked high-volume banking compliance jobs suffering from partition skew and shuffle overhead. Resolved bottlenecks via Spark UI analysis, salting keys, Adaptive Query Execution (AQE), and broadcast hash joins.

Optimization: Anti-Join Appends + Data Salting + AQE
Spark UI AQE Data Salting Redshift
35–40% Batch runtime reduction
INGESTION FRAMEWORK Enterprise ERP

Asynchronous ERP Ingestion Engine

Architected an API-driven extraction framework using Python, FastAPI, and SQLAlchemy to ingest multi-source enterprise client ERP files, enforcing strict schema validation and staging to Azure Data Lake Storage Gen2.

Stack: FastAPI → ADLS Gen2 → Azure Databricks → Snowflake
FastAPI ADLS Gen2 Snowflake Terraform
99.8% Schema reconciliation compliance

02. Professional Experience

Data Engineer @ Nortek Consulting INC — Client: Optum

Oct 2025 - Present

Florida, USA • Analytics & Reporting Platform

  • Own, monitor, and optimize Databricks + PySpark pipelines feeding daily KPI reporting layers.
  • Cut nightly load runtime 50% (7:30 AM → 3:00 AM) via re-partitioning and query tuning — dashboards refresh before the business day.
  • Tuned Delta Lake tables (OPTIMIZE, Z-Ordering) — Tableau workbook queries from 40s+ to under 4s.
  • Automated manual workflows with Python/PySpark: data archiving, schema validation for nested inputs, failure alerting.
  • Lead root-cause analysis on metric anomalies; automated source-to-target reconciliation to catch drift before downstream reports.

Data Engineer @ Akkodis (Herc Rentals Inc.)

Nov 2024 - Sep 2025

Florida, USA • Fleet & Rental Data Platform

  • Built Python ingestion from RentalMan, AS400, Oracle OLTP, and Teradata into an AWS S3 data lake.
  • Near-real-time telematics pipelines (Kinesis + Databricks Streaming) for GPS, engine, and fuel JSON/XML payloads.
  • PySpark/Spark SQL pipelines in Databricks — cut daily batch runtimes 25–30% via partition tuning and incremental loads.
  • Cut pipeline rerun effort 30–40% with modular, restartable stages and curated intermediate outputs.
  • Validation and reconciliation checks (counts, dupes, nulls) before business reporting; Airflow orchestration.

Data Engineer @ Quess Corp (Blue Yonder India Pvt Ltd.)

June 2022 - July 2023

India • Supply Chain Optimization

  • Engineered high-throughput Azure Databricks pipelines, reducing batch transformation runtimes by 25–30% through unified PySpark transformations.
  • Built custom Python, FastAPI, and SQLAlchemy ingestion frameworks to ingest complex ERP payloads safely into ADLS Gen2.
  • Optimized Snowflake and Exasol analytical layers via advanced query profiling, partition tuning, and SQL performance tuning.
  • Managed orchestration workflows using Airflow alongside secure Terraform-deployed Azure environments (Key Vault, RBAC).

Data Engineer @ Trigent Software (Accenture)

Jan 2020 - June 2022

India • Banking & Regulatory Compliance

  • Cut runtime by 35–40% on large-scale financial pipelines by diagnosing Spark UI bottlenecks and applying data salting, AQE, and Broadcast Joins.
  • Modernized legacy Ab Initio workflows into Python-based S3 data lakes cataloged via AWS Glue Data Catalog.
  • Designed audit-ready Amazon Redshift dimensional models and structured rigorous source-to-target reconciliation checks.

ETL Developer @ Experis IT (Thomson Reuters)

June 2017 - Sep 2019

India • Sales & Revenue Analytics

  • Extracted enterprise SQL Server legacy data into S3 data lakes using Apache Sqoop on AWS EMR.
  • Built AWS Glue and PySpark pipelines to process sales, subscription, and revenue data for Redshift and Athena analytics.

03. Technical Competencies

ETL & Data Integration

IBM DataStage Microsoft SSIS Ab Initio AWS Glue SQL Server Agent Cloud Migration Data Mapping T-SQL / PL-SQL

Cloud & Distributed Compute

AWS (S3, EMR, Glue) Azure (ADLS Gen2) Apache Spark PySpark Databricks AWS Kinesis Athena Delta Lake

Warehousing & Modeling

PostgreSQL (Psycopg 3) Snowflake Amazon Redshift Teradata SQL Server Star Schema SCD Type 2 CDC Data Lineage Data Masking HIPAA / HITRUST

DevOps & Data Platform

Model Context Protocol (MCP) Deterministic RAG Starlette / Uvicorn Python 3.12 Apache Airflow FastAPI Docker Terraform

04. Verified Certifications

AWS Certified Solutions Architect – Associate Verified

AWS Certified Solutions Architect – Associate

Amazon Web Services (AWS)

Verified via Credly • Active Credential

Verify Credential

05. Education

GRADUATED 2024

Masters in Information Technology Management

California Baptist University • Riverside, CA

GRADUATED 2017

B. Tech in Computer Science and Engineering

JNTU-K University • India