jdcsen Portfolio, projects, and other work by Joshua David Christensen

My name is Josh Christensen, a lifelong engineer living in Pittsburgh, PA with roots in Northern Utah. For the past several years I’ve been a software engineer at Amazon, building the production inference systems behind conversational AI, including work for Alexa and Amazon’s Nova line of generative models.

My most recent work surrounds productionalizing novel LLM inference and classification approaches coming out of Amazon applied science teams. I led a small team building a Triton inference backend that derives paralinguistic features like confidence, sentiment and language identification from the token-level outputs of a vLLM-served LLM. We took this from prototype to production: single-digit-millisecond p50 latencies, and a concurrency refactor that took sustained throughput from ~8 to ~80 requests per second on the same hardware. Along the way I’ve built retrieval-augmented personalization for speech recognition, LLM-agent tooling that cut our on-call investigation time from half an hour to minutes, and the AWS CDK infrastructure underneath all of it.

I got here by way of systems software. Before Amazon I spent four years at L3Harris writing safety-critical C++ for radio systems, and I hold a BS and an MS in Computer Science from the University of Utah, where I graduated cum laude through the Honors program as an undergraduate and led the controls team for Formula SAE Electric. Outside of work I enjoy a variety of personal engineering projects, including a self-hosted, multi-modal inference stack serving roughly 35 models across text, speech, image, and video on a single RTX 5090.

Featured Projects

 

Paralinguistic Signals from a Speech LLM

  • NVIDIA Triton Python backend fronting a vLLM-served speech LLM, taken from prototype to production.
  • Derives confidence, sentiment and spoken-language ID from the model’s own token-level outputs. No additional models to train or serve.
  • Orchestrates per-signal LLM queries over gRPC alongside ONNX Runtime inference for confidence calibration and forced alignment.
  • p50 latency of approximately 5 to 10 ms per signal.
  • Technical lead for a three-engineer team. A later concurrency refactor took sustained throughput from roughly 8 to 80 requests per second.

Read more →

gRPC Bidirectional Streaming for a C++ Node-Graph Framework

  • Nova Sonic needed the speech framework’s pipelines delivered as a containerized gRPC service: audio and system prompts streaming in, inference requests streaming out. The team’s code had only ever been called through JNI.
  • Mapped gRPC C++’s bidirectional-streaming reactor onto the framework’s pipe abstraction once, as a reusable layer covering session setup and teardown, signal handling, graceful error handling and logging.
  • A new service needs under 300 lines of integration code: which pipe input receives request messages, and which pipe outputs become response messages. Adopted org-wide as the standard way to deploy a pipeline.
  • Established Protobuf and gRPC generated code as first-class CMake libraries in the framework’s build, so the same generated types are consumed by the pipeline’s nodes and by the server without duplicate-symbol conflicts.

Read more →

Self-Hosted Multi-Modal Inference on One GPU

  • One OpenAI- and Anthropic-compatible endpoint fronting 36 model keys on a single RTX 5090 (32 GB): 14 LLM keys (Qwen3 coder, thinking and instruct tiers, a vision model, a captioner), 4 Whisper variants, 11 image generators (Flux, SDXL, Chroma, Qwen-Image), 4 Wan video models and 3 GPU feature-extraction sidecars.
  • Built on llama-swap, a Go router that starts and stops upstream inference processes on demand. Anything that speaks HTTP can be an upstream, which is what lets llama.cpp, whisper.cpp, a patched stable-diffusion.cpp and three PyTorch services share one card behind one API.
  • Co-residency is declared with set-algebra rules and eviction costs, but llama-swap does not measure VRAM, so I did: a sweep script that measures resident and peak footprints per model and per combination, which turned up a 6.7 GB transient VAE-decode spike as the binding constraint.
  • Every workhorse LLM has two keys: an exclusive full-context key and a co-resident “lite” twin, so a 256k-context 30B model and an image generator never fight for the card.
  • Heavy upstreams run as sibling containers launched on demand, so the router image rebuilds in seconds instead of recompiling sd-server and three multi-gigabyte venvs.

Read more →

RAG Personalization for Speech Recognition

  • Retrieval-augmented personalization for the same speech LLM behind the paralinguistic backend.
  • A user’s contacts, device names and music library are retrieved from their catalogs with approximate-nearest-neighbor search.
  • Retrieved entries are injected into the LLM’s prefill context, so the model transcribes the user’s own vocabulary instead of guessing at it.
  • The original design needed three round trips through the LLM per request. Working with the applied science team, we replaced LLM-generated query embeddings with an index keyed on voice features, eliminating one of them, a roughly 20 ms LLM round trip per request.
  • Keying retrieval on voice features also decoupled the lookup from the LLM’s context request, so retrieval runs speculatively, in parallel with the model, instead of waiting on it.
  • Led the effort end to end, from retrieval design to serving.

Read more →