Regarding the inference speed of the yolov 5 model

[isakkk3]

Search before asking

• I have searched the YOLOv5 issues and discussions and found no similar questions.

Question

Hello, my self-trained YOLOv5s model has an identical architecture to the official version and uses the official weights, yet the inference time of my implementation is 70 milliseconds, showing such a significant discrepancy compared to the official benchmarks. Why does this happen despite meticulously following the official setup steps?"

Additional

No response

[UltralyticsAssistant]

👋 Hello @isakkk3, thank you for your interest in YOLOv5 🚀! Please visit our ⭐️ Tutorials to get started, where you can find quickstart guides for simple tasks like Custom Data Training all the way to advanced concepts like Hyperparameter Evolution.

If this is a 🐛 Bug Report, please provide a minimum reproducible example to help us debug it.

If this is a custom training ❓ Question, please provide as much information as possible, including dataset image examples and training logs, and verify you are following our Tips for Best Training Results.

Requirements

Python>=3.8.0 with all requirements.txt installed including PyTorch>=1.8. To get started:

git clone https://github.com/ultralytics/yolov5  # clone
cd yolov5
pip install -r requirements.txt  # install

Environments

YOLOv5 may be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):

• Notebooks with free GPU:
• Google Cloud Deep Learning VM. See GCP Quickstart Guide
• Amazon Deep Learning AMI. See AWS Quickstart Guide
• Docker Image. See Docker Quickstart Guide

Status

If this badge is green, all YOLOv5 GitHub Actions Continuous Integration (CI) tests are currently passing. CI tests verify correct operation of YOLOv5 training, validation, inference, export and benchmarks on macOS, Windows, and Ubuntu every 24 hours and on every commit.

---

🔔 This is an automated response. An Ultralytics engineer will also assist you here soon!

[pderrenger]

Hi @isakkk3,

The inference time discrepancy is likely due to several factors that affect benchmarking results. The official benchmarks are typically measured under optimized conditions:

1. Hardware: Benchmarks often use high-end GPUs in controlled environments
2. Inference settings: Check if you're using half-precision (FP16) vs full-precision (FP32)
3. Measurement methodology: Official benchmarks may exclude model loading time and measure only the forward pass
4. Optimization: TensorRT export can significantly speed up inference compared to standard PyTorch

You can benchmark your model with the Ultralytics package using:

from ultralytics.utils.benchmarks import benchmark
benchmark(model="your_yolov5s.pt", imgsz=640, half=False, device=0)

Try optimizing with half=True and exploring different export formats like ONNX or TensorRT for better performance.

[isakkk3]

After converting the ONNX model to RKNN format and deploying it on an RK3588 platform, I observed a significant performance discrepancy: my model's inference time is 75ms compared to the official benchmark of 35ms, despite using the same YOLOv5s pretrained weights and exclusively applying ReLU activation functions during training. Which step in this pipeline might be causing this degradation?"

[pderrenger]

Hi @isakkk3,

When deploying to RK3588 with RKNN format, several factors can cause performance discrepancies:

1. RKNN conversion parameters - verify you're using optimal settings during ONNX to RKNN conversion
2. Quantization level - INT8 quantization can significantly improve NPU inference speed
3. NPU core allocation - check if you're fully utilizing the available NPU cores on RK3588
4. Pre/post-processing - ensure image processing is offloaded to hardware when possible
5. RKNN runtime version - latest versions often include platform-specific optimizations

You might want to profile each step of your inference pipeline to identify bottlenecks. The Rockchip official documentation provides specific optimization guidelines for RK3588 that could help close this performance gap.