Data Engineering Archives

Chunking and Reranking in RAG: The Two Dials That Matter More Than Your Model

Chunking and reranking in RAG are the two dials that decide retrieval quality. A practical guide to chunk size, strategies, and cross-encoder rerankers.

Afzal Malik August 5, 2026

Featured image for S3 Files as an alternative to internal SFTP workflows

AWS Data Engineering Technical Blogs

S3 Files Instead of SFTP: When It Works, When It Does Not, and How Data Teams Should Decide

S3 Files can replace SFTP in some internal AWS workflows, but not partner-facing secure file exchange. Here is how data teams should decide.

Ruby Malik May 17, 2026

Technical illustration showing legacy SFTP moving to AWS Transfer Family and S3 data pipelines

AWS Data Engineering Technical Blogs

AWS Transfer Family SFTP: Pricing, Limits, and the Management Case

A management-ready guide to AWS Transfer Family SFTP pricing, limitations, and when data engineering teams should replace traditional SFTP servers.

Ruby Malik May 17, 2026

Illustration of serverless AI agents using AWS Lambda and a shared S3 Files workspace

AI AWS Data Engineering

S3 Files for Lambda: a real workspace for serverless AI agents

S3 Files lets Lambda mount S3 buckets as file systems. Learn the architecture, IAM, VPC, agent workflow, and data pipeline trade-offs.

Ruby Malik May 16, 2026

AWS Data Engineering Technical Blogs

Amazon S3 Files: What Data Engineers Actually Need to Know

S3 Files went generally available on April 7, 2026. The AWS blog covered the setup steps. This post covers something more useful: what it actually changes for data engineers who…

Ruby Malik April 14, 2026

Graphviz Medallion Architecture Data Pipeline Diagram - Bronze Silver Gold layers

Data Engineering

Graphviz Tutorial: Install, Setup & Build Your First Diagram on Linux (Medallion Architecture Example)

Learn how to install Graphviz on Linux and create professional architecture diagrams using the DOT language. Includes a complete Medallion Architecture data pipeline example with Bronze, Silver, and Gold layers.

Afzal Malik March 15, 2026

Snowflake caching architecture diagram showing three cache layers

Data Engineering Technical Blogs

Snowflake Caching Explained: Query Results, Metadata and Warehouse Cache

Learn how Snowflake's three caching layers — Query Results Cache, Metadata Cache, and Warehouse Cache — work together to speed up queries and cut compute costs.

Afzal Malik March 14, 2026

AIML Backend Cloud Computing

How to Remove Duplicates from an Apache Iceberg Table in Athena (Without a Unique ID or Timestamp)

Duplicate data in data lakes is one of the most common and frustrating challenges for data engineers. But what happens when: You don’t have a primary key You don’t have…

Afzal Malik March 2, 2026

AIML AWS Backend

Building High-Performance Serverless Functions with Rust, Axum, and WebAssembly: The Future of Efficient Computing

Learn how to build high-performance serverless functions using Rust, Axum, and WebAssembly to solve the cold start problem and achieve superior performance compared to traditional serverless architectures.

Afzal Malik January 5, 2026

AWS Data Engineering

PySpark Performance Optimization: 10 Essential Best Practices for Data Engineers in 2025

Master PySpark performance optimization with 10 essential best practices for data engineers. Learn memory management, DataFrame API patterns, and advanced techniques to boost your Spark applications by 5-10x.

Afzal Malik December 14, 2025