HPC & AI Infrastructure Engineer
I run production GPU infrastructure. Sole administrator of three production clusters at Temple University — compute, storage and networking for 78+ researchers — and I spent a summer tuning GPU allocation across 356 H100 and A100 nodes on Mount Sinai's 11-petaflop Minerva supercomputer.
Three production clusters end to end — compute, storage, networking. Bare-metal PXE provisioning through Cobbler, configuration management in Ansible, monitoring in Ganglia, on AlmaLinux 9 / RHEL. Sole administrator: I own the full node lifecycle from rack to production scheduling.
Multi-partition Slurm with GPU scheduling, fairshare and prolog/epilog. I ran an internal security assessment treating the scheduler as an attack surface — prolog/epilog injection, munge key handling, module system and SUID escalation paths — and hardened the cluster configuration against what it found.
I work extensively with Kubernetes, including k3s for GPU clusters — GPU scheduling through the NVIDIA device plugin, node lifecycle, cordon and drain for maintenance, and reconciliation-based deployment of long-running services. Batch and orchestration are different tools for different shapes of work, and running both means I can put a workload where it belongs instead of where the scheduler forces it.
NVIDIA A100 and H100 fleets across heterogeneous hardware. Diagnosed and resolved a fleet-wide driver generation mismatch (570 vs 595), iDRAC event storms, and recurring GPU node scheduling failures — the unglamorous work that keeps utilization up.
I run the cluster and I am also a demanding user of one. GPU-accelerated ML at scale: JAX/CUDA and cuML/RAPIDS debugging, chunked UMAP transforms in 200K-row batches to work around GPU memory limits. I know what the workloads on my nodes actually need.
A GPU-accelerated behavioral analysis pipeline that discovers behavioral states in rodent recordings without predefined labels. DeepLabCut pose estimation feeds kinematic feature extraction, then UMAP dimensionality reduction and HDBSCAN clustering over 22M+ frames across 3,846 recordings. It identified a fear discrimination state invisible to traditional human scoring.
Unsupervised clustering is easy to fool, so I built a six-axis evaluation framework to prove the discovered states are real rather than artifacts:
It is in active use in live neuroscience experiments — which means it has to survive real data, real GPUs and real deadlines, not just a benchmark.
github.com/toucheLos/VIEBExtensible object-oriented scientific visualization software for neuronal modeling, supporting dynamic addition and removal of ion channel components. I implemented the Euler method solver for Hodgkin-Huxley action potential dynamics and cross-validated it against the Yale NEURON solver.
Batch. Jobs arrive with resource requests and wait in
PENDING until the backfill pass finds a node with room. Multi-GPU jobs
block until enough contiguous devices free up — which is exactly why a badly
balanced partition sits idle with a full queue. Drain a node and watch the
scheduler route around it.
Declarative. The same fleet under Kubernetes. You do not submit
work here — you declare that you want N replicas and a controller loop
reconciles reality toward that number. Kill a pod and it comes back. Cordon a node
and its pods are evicted and rescheduled somewhere with free
nvidia.com/gpu. Scale past the fleet's capacity and the surplus sits
Pending forever with Insufficient nvidia.com/gpu — the
scheduler will not queue it the way Slurm would, because a Deployment has no
concept of waiting its turn.
Occasional writing on cluster operations, scheduler internals and scientific computing. All notes →
