Data Engineering Archives

Featured image for S3 Files as an alternative to internal SFTP workflows

S3 Files Instead of SFTP: When It Works, When It Does Not, and How Data Teams Should Decide

Ruby Malik May 17, 2026

S3 Files can replace SFTP in some internal AWS workflows, but not partner-facing secure file exchange. Here is how data teams should decide.

Technical illustration showing legacy SFTP moving to AWS Transfer Family and S3 data pipelines

AWS Data Engineering Technical Blogs

AWS Transfer Family SFTP: Pricing, Limits, and the Management Case

Ruby Malik May 17, 2026

A management-ready guide to AWS Transfer Family SFTP pricing, limitations, and when data engineering teams should replace traditional SFTP servers.

Illustration of serverless AI agents using AWS Lambda and a shared S3 Files workspace

AI AWS Data Engineering

S3 Files for Lambda: a real workspace for serverless AI agents

Ruby Malik May 16, 2026

S3 Files lets Lambda mount S3 buckets as file systems. Learn the architecture, IAM, VPC, agent workflow, and data pipeline trade-offs.

AWS Data Engineering Technical Blogs

Amazon S3 Files: What Data Engineers Actually Need to Know

Ruby Malik April 14, 2026

…

Graphviz Medallion Architecture Data Pipeline Diagram - Bronze Silver Gold layers

Data Engineering

Graphviz Tutorial: Install, Setup & Build Your First Diagram on Linux (Medallion Architecture Example)

Afzal Malik March 15, 2026

Learn how to install Graphviz on Linux and create professional architecture diagrams using the DOT language. Includes a complete Medallion Architecture data pipeline example with Bronze, Silver, and Gold layers.

Snowflake caching architecture diagram showing three cache layers

Data Engineering Technical Blogs

Snowflake Caching Explained: Query Results, Metadata and Warehouse Cache

Afzal Malik March 14, 2026

Learn how Snowflake's three caching layers — Query Results Cache, Metadata Cache, and Warehouse Cache — work together to speed up queries and cut compute costs.

AIML Backend Cloud Computing

How to Remove Duplicates from an Apache Iceberg Table in Athena (Without a Unique ID or Timestamp)

Afzal Malik March 2, 2026

…

AIML AWS Backend

Building High-Performance Serverless Functions with Rust, Axum, and WebAssembly: The Future of Efficient Computing

Afzal Malik January 5, 2026

Learn how to build high-performance serverless functions using Rust, Axum, and WebAssembly to solve the cold start problem and achieve superior performance compared to traditional serverless architectures.

AWS Data Engineering

PySpark Performance Optimization: 10 Essential Best Practices for Data Engineers in 2025

Afzal Malik December 14, 2025

Master PySpark performance optimization with 10 essential best practices for data engineers. Learn memory management, DataFrame API patterns, and advanced techniques to boost your Spark applications by 5-10x.

AI AIML Data Engineering

Youtu-Agent: The Open-Source AI Agent Framework Built for Real-World Performance

Afzal Malik December 14, 2025

Discover Youtu-Agent, an open-source AI agent framework from Tencent achieving 71% on WebWalkerQA using DeepSeek-V3. Learn how this framework enables cost-effective autonomous agent development with automatic tool generation and training-free optimization.