GPU-Accelerated Optimization of Transformer-Based Neural Networks for Real-Time Inference

View PDF

Abstract:This paper presents the design and evaluation of a GPU-accelerated inference pipeline for transformer models using NVIDIA TensorRT with mixed-precision optimization. We evaluate BERT-base (110M parameters) and GPT-2 (124M parameters) across batch sizes from 1 to 32 and sequence lengths from 32 to 512. The system achieves up to 64.4x speedup over CPU baselines, sub-10 ms latency for single-sample inference, and a 63 percent reduction in memory usage. We introduce a hybrid precision strategy that preserves FP32 for numerically sensitive operations such as softmax and layer normalization, while applying FP16 to linear layers. This approach maintains high numerical fidelity (cosine similarity >= 0.9998 relative to baseline outputs) and eliminates NaN instability. The pipeline is implemented as a modular, containerized system that enables reproducible benchmarking across more than 360 configurations. Cross-GPU validation on an NVIDIA A100 shows consistent FP16 speedup ratios between 1.84x and 2.00x, along with stable numerical behavior. Downstream evaluation on SST-2 demonstrates no accuracy degradation under hybrid precision. Validation on WikiText-2 shows that random inputs underestimate NaN instability by up to 6x for full FP16, while confirming the robustness of the hybrid approach (0.0 percent NaN, cosine similarity >= 0.9998). These results provide a detailed characterization of performance and accuracy trade-offs across GPU architectures and offer practical guidance for deploying transformer models in latency-critical environments.

Comments: 10 pages, 8 figures, 15 tables

Subjects:

Machine Learning (cs.LG); Distributed, Parallel, and Cluster Computing (cs.DC)

Cite as: arXiv:2603.28708 [cs.LG]

(or arXiv:2603.28708v1 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2603.28708

arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Sangwhan Cha [view email] [v1] Mon, 30 Mar 2026 17:27:33 UTC (713 KB)