Technical Skills
Languages: Rust, Go, TypeScript, Python, C++, Zig
Infrastructure: Kubernetes, Docker, Helm
Systems and Performance: Linux, perf, pprof, flame graphs, jemalloc
Data Systems: Apache DataFusion, Arrow, Iceberg, PostgreSQL, Kafka, Redis
Real-Time Media: WebRTC, GStreamer, FFmpeg
Work Experience
e6data - Software Development Engineer (Sep 2024 - Present)
Rust - Java - DataFusion - Arrow - Iceberg - Kafka - FFI - NVMe - Kubernetes
- Proposed and led the migration from Java and hybrid Java-Rust database executors to an all Rust execution stack built on Apache DataFusion, Apache Arrow, delta-rs and iceberg-rs.
-
Built query performance monitoring and profiling infrastructure and used it to diagnose and resolve bottlenecks in production customer queries scanning multiple terabytes of data.
-
Optimized query execution across the TPC-DS benchmark suite, reducing total runtime by 70%, from vanilla Datafusion's 1023 seconds to 305 seconds, versus 341 seconds for Databricks Runtime under the same benchmark configuration.
-
Designed hybrid memory and NVMe disk based data-caching infrastructure supporting local and distributed execution, improving workload performance by 60-70%. Further work included reducing warm-up time through distributed cache bootstrapping, introducing sketch-based admission policies to prevent cache pollution and optimizing NVMe I/O by evaluating pread/pwrite, epoll and io_uring backends.
-
Led the redesign of logical table reads around immediately consistent metadata and table scan layer caching. This eliminated manual catalog refreshes, removed network hops, and retired the Storage and Schema microservices. Overall effort lead to improved query performance by 30% across various workloads.
-
Diagnosed and fixed cancellation-safety and task-lifecycle bugs in async Rust query execution, improving query-engine reliability during failures and interrupted queries. [blog]
-
Established an upstream contribution workflow that helped e6data become a leading Apache DataFusion contributor and me getting recognized individually as a major contributor. [report]
-
Built Linkstream, an Iceberg streaming-ingestion system built on RisingWave, that onboarded a customer at 50K messages/second with a 15-second ingestion-to-query SLA.
-
Added Glue and Hive catalog updates and AWS authentication to iceberg-rust, enabling production-grade ACID writes and replacing JNI-based catalog integrations.
-
Extended RisingWave's Kubernetes operator to support e6data's infrastructure and local-development workflows, adapting deployment and lifecycle management to internal platform requirements.
Dyte (acquired by Cloudflare) - Software Development Engineer (Sep 2021 - Sep 2024)
WebRTC - WebSockets - GStreamer - FFmpeg - Livestreaming - RabbitMQ - Golang - TypeScript - Rust - Kotlin - FFI
- Worked on scaling the socket layer, reducing worst case response times from 5-10s to sub-100ms alongside reducing failure rates by 85% drastically by optimizing data access patterns, evolving APIs and removing locking.
- Led the development of the livestreaming product from concept to launch. Took ownership of end-to-end development, turning it into a core revenue vertical eventually contributing up to 50% of the total revenue, i.e. 250K USD.
- Developed FFI bindings of libmediasoupclient for Android/iOS and a Kotlin Multi-Platform call statistics collector, forming the foundation of all enterprise mobile SDKs.
- Worked on server-side headless-browser-based recording service, fixing concurrency bugs, writing custom autoscaler, improving FFmpeg pipelines and integrating livestreaming.
- Worked on individual audio track recording, transcription and dev tooling in stream recording service using GStreamer and Rust.
- Actively participating in debugging production issues ranging from memory leaks, DB migration failovers, retry storms, etc. on multiple different services.
- Led work on stability of socket layer, backbone of media and room management core workflows. Established quick iteration development, testing suites, better ergonomics using incremental refactors.
- Built and actively helped maintain billing and usage tracking services in Typescript allowing automated payments and invoices.
Open Source
DataFusion
-
feat: selectivity metrics (for Explain Analyze) in Hash Join
[GitHub]
-
feat: allow pushdown of dynamic filters having partition cols
[GitHub]
-
feat: integrate batch coalescer with async fn exec
[GitHub]
-
Remove coalesce batches rule and deprecate CoalesceBatchesExec
[GitHub]
-
fix: predicate cache stats calculation
[GitHub]
Iceberg Rust
-
Implemented NaN value count statistic in parquet writer
[GitHub]
-
Developed Glue and Hive update table methods completing write support for iceberg-rs
[GitHub]
[GitHub]
-
Fixed incorrect key_metadata defaults issue
[GitHub]
-
Implemented AWS Authentication methods for different catalogs
[GitHub]
-
Wrote partition data write integration test
[GitHub]
rust-analyzer
- Developed a complex assist: Extract To Module and worked with the community to improve and maintain it.
- Contributed critical fixes to Rust Analyzer, improving refactoring, diagnostics, and code assists for thousands of Rust developers.
[GitHub]
Fenil Jain - Resume