Every few years for the last fifteen odd years I find myself staring at the samme problem in a different costume. Server costs that don’t justify usage, a few extra milliseconds in response time, a feature that starts to stand out as a resource hog at scale - not broken, just hungry. Symptoms easy to miss if you weren’t looking real close. “When you hear hoofbeats, think horses, not zebras.” — Dr....
Late last year I was building a multi-model agentic pipeline. Not a demo — something I wanted to actually run: audio in, Whisper for transcription, a small intent classifier, a RAG retrieval step, and finally Llama 3.1 8B for the response. Five models, one machine. The GPU I had was a single RTX 4090 with 24GB of VRAM. That should’ve been enough. Spoiler: the way existing inference stacks work, it wasn’t....
diff-match-patch-rs A few years back, while building HandyTrain, we decided to build a collaborative content creation feature, among other things we needed a text-synchronization library - a WASM version for the client and a high-performance library for our Go and Rust services. With some research we landed on the fantastic diff match patch algorithms, the diff part is an implementation of this paper (often called the Myer’s Diff algorithm) by Eugene Myer’s....
A DIY style step-by-step guide to building your own cutting-edge GenAI-powered document QA desktop app with RAG.
I’ve been using WebAssembly aka WASM for a while in production to do some incredible stuff in the browser, things that would be prohibitive in terms of performance otherwise. Here are some real use-cases I’ve used WASM for: Running statistical processing 10 million rows (roughly 50 columns in each row) of CSV data in browser. This feature required us to create a temporary playground of reports generated for our clients where they could run their own analysis without the need of permanent storage or costly servers....