cat resume.txt

Senior Data Engineer, Infocepts

Apr 2024 – present (2y 6m) · Pune, India

Data Engineer, Infocepts

Aug 2022 – Apr 2024 (1y 8m) · Nagpur, India

Assistant Analyst, Infocepts

Feb 2022 – Aug 2022 (6m) · Nagpur, India

The landing zone

paid work at Infocepts · AWS Glue, PySpark, AWS Lambda, Amazon SQS, API Gateway, Amazon S3, Jenkins

The ingestion layer of a multi-source AWS platform: Glue and PySpark for batch, and one contract every source meets before anything downstream reads it.

scale

Several upstream systems on one contract — batch through Glue, and the platform's first event path behind the same landing zone

outcome

50% lower end-to-end processing time than the pipeline it replaced

the full write-up →

Warehouse and serving layer

paid work at Infocepts · Amazon Redshift, Amazon S3, Denodo, Amazon QuickSight, SQL

The serving layer — Redshift models, a Denodo virtualization tier, and the QuickSight dashboards the business actually opens on a Monday morning.

scale

One modelled Redshift layer behind a virtualization tier, serving every BI consumer

outcome

35% faster query execution after performance tuning

the full write-up →

Retail point-of-sale ETL

paid work at Infocepts · AWS Glue, PySpark, Amazon Redshift, Amazon S3, Python, SQL, pytest

Twenty-nine Glue processors that turn fourteen retail partners' sales and inventory files into one Redshift star schema, a feed at a time.

scale

29 processors (13 dimension, 16 fact) across 14 retail partner feeds, each with its own unit tests

outcome

A new retailer is a processor, a config entry and a test file, not a new pipeline

the full write-up →

SAP object onboarding

paid work at Infocepts · AWS AppFlow, AWS Glue, Python, pandas, Amazon S3, Parquet, Terraform, AWS CDK, Amazon Athena

Ninety SAP services taken from "it exists in SAP" to "it is a partitioned table in Athena", through a fleet of 400+ AppFlow flows and a parameterised Glue job.

scale

90 SAP objects onboarded and 140+ source-to-lake mappings added, in a fleet of 400+ flows on schedules from every twenty minutes to weekly

outcome

Delta-enabled feeds moved from one extraction a day to one an hour, with hour-level partitions to match

the full write-up →

Infrastructure as code

paid work at Infocepts · Terraform, AWS CDK, Jenkins, GitHub Actions, Python

Terraform and AWS CDK modules that moved environment setup out of the console and into reviewed pull requests.

scale

Every environment defined in code — no console-assembled infrastructure left

outcome

40% less manual effort per deployment

the full write-up →

The data catalog as code

paid work at Infocepts · Terraform, AWS Glue Data Catalog, Amazon Athena, Amazon S3, Parquet, HCL

More than 150 partitioned Glue catalog tables and 5,000 columns declared in Terraform, a third of the platform's catalog.

scale

154 tables and 5,000+ columns across 12 SAP functional areas; 166 Terraform modules in all, the most of any contributor

outcome

A table is reviewable in a pull request before it exists, and identical in every environment after

the full write-up →

Supply-planning ingestion

paid work at Infocepts · AWS Glue, Python, pandas, pandera, Terraform, Amazon S3, Parquet, Amazon Athena

Ten scheduled Glue jobs that check planning-system extracts against a declared schema and land them as partitioned tables.

scale

10 Glue jobs with catalog tables, on daily, weekday and weekly schedules, in a family of 32

outcome

A file that cannot be processed fails the run and names itself

the full write-up →

Analytics event API

paid work at Infocepts · Amazon API Gateway, AWS Lambda, Amazon SQS, AWS CDK, AWS Glue, Python, Amazon S3, OpenAPI, Terraform

A REST endpoint, a queue, two Lambdas and a Glue job that take usage events from client applications to a partitioned table, built alone in three phases.

scale

One API, two queues, two Lambda functions and a daily Glue job, consuming batches of up to 1,000 messages per invocation

outcome

Three phases shipped in under four months, from first endpoint to queryable table

the full write-up →

Flow operations utility

paid work at Infocepts · AWS Lambda, AWS AppFlow, Python, boto3, Amazon SNS, AWS CDK, pytest

A Lambda that suspends, reactivates, reschedules and reports on a fleet of more than 400 AppFlow flows, so nobody does it one flow at a time in the console.

scale

Four modes over a fleet of 400+ flows, selected by criteria, covered by 33 unit tests

outcome

A bulk suspend or reactivate is one invocation with a filter

the full write-up →

Daily snapshots

paid work at Infocepts · Amazon Athena, SQL, Jinja, AWS Step Functions, AWS Lambda, Amazon S3, Parquet, Terraform

Nine Athena queries that turn change-record tables into one current row per key, once a day, as partitioned Parquet.

scale

9 snapshot tables, deployed to two environments, on the platform's existing query runner

outcome

A consumer reads one day's de-duplicated partition in place of the full change history

the full write-up →

WarehouseGPT

independent · LangGraph, dbt, DuckDB, PySpark, Delta Lake, LiteLLM, sqlglot, LanceDB, FastAPI, OpenTelemetry, Python

A text-to-SQL agent, and an ablation suite built to measure why agents like it fail. Context took the smallest model from 63.5% to 98.0%.

scale

Nine source tables and about 1.5M rows — 99,441 orders, 96,096 customers, 3,095 sellers — into a seven-table star schema behind 17 governed metrics, 31 Spark data-quality checks and 49 dbt tests, on a $0 budget

outcome

Execution accuracy on 52 answerable golden questions, with 6 must-refuse ones scored separately: 63.5% on raw DDL, 98.0% (50/51) on the full context. Documentation alone was worth 10–19 points, and with everything switched on the smallest model beat both larger ones running on docs alone

the full write-up →

Leakproof

independent · LightGBM, Redpanda, Redis, DuckDB, FastAPI, Streamlit, Docker, Python

The same fraud model built carelessly and point-in-time correct, then both scored on live stream features. The careless one lost half its PR-AUC.

scale

The last 15% of transactions by time — 88,581 rows over 30.8 days — replayed through Redpanda and Redis, where all 14 streamed features matched the offline ones on every row, largest difference 1.8e-12

outcome

POST /score at p50 2.8 ms, p95 4.3 ms, p99 5.9 ms, down from 121 ms. The careless model fell from 0.999 to 0.901 ROC-AUC and from 0.986 to 0.513 PR-AUC between offline and live; the honest one went from 0.954 to 0.947 ROC-AUC and from 0.695 to 0.665 PR-AUC

the full write-up →

Lossproof

independent · PostgreSQL, Debezium, Apache Kafka, Apache Iceberg, Terraform, AWS Step Functions, AWS Lambda, dbt, Amazon Athena, AWS Glue, Amazon S3, Python

The same CDC pipeline built carelessly and correctly, then faulted. With every fault in one run, 69% of the careless lake was wrong. The correct one matched.

scale

100,000 source transactions and about 320,000 messages through one stream into two lakes, then every source row compared with every lake row by key: 53,785 of 77,962 wrong for the careless consumer, 0 of 77,962 for the correct one

outcome

A nightly Step Functions run compacts each table from 15 data files to one, reconciles 8,229 rows to zero differences, then builds 7 dbt models and 10 tests — and fails on LakeDiverged before building a single mart if one row disagrees

the full write-up →

Nightshift

independent · AWS CDK, AWS Lambda, Amazon SQS, Amazon S3, Amazon API Gateway, AWS Glue Data Catalog, Amazon Athena, Amazon CloudWatch, Parquet, MCP, Docker, Python

A receipt-extraction lake with an LLM as an ETL step and an agent on call, both measured. A 101-line script fixed 15 of 16 incidents. The agent fixed 11.

scale

100 receipts from the CORD test split for each of three models, scored against the dataset's labels, then 16 injected incidents handled by four responders through one policy layer, each judged by a reconciler that compares the lake with a fault-free run

outcome

Contracts and a repair loop removed 36% to 66% of wrong values for 24% to 35% more tokens, and 24% on a held-out split. On call, the scripted runbook got 15 of 16 right in a median 9 s; the agent got 11 in 37 s, and 9 without the runbook as text

the full write-up →