Using any torchvision pretrained model as backbone for FasterRcnn

[mrinath123]

🚀 The feature

Adding only the name of the backbone network, loads the pretrained model in the FasterRcnn model as backbone.

Motivation, pitch

Its possible to use any backbone, but we have to specify many things like feature size, feature_name etc., wont be it easy for the user to just give the name of the backbone and we internally do everything.

Alternatives

No response

Additional context

No response

cc @datumbox @YosuaMichael

[mrinath123]

Need reviews for this, and thanks for listening to my idea
CC @datumbox

[datumbox]

Thanks for the recommendation. If I understand you correctly, you would like to make seamless the combination of different backbones with different detection heads, is that right?

The current API, allows you to provide a backbone on the constructor of the Detection models. This is the first argument that all Object Detection classes receive. Using it is a rather advanced API because one has to pass quite a few configurations to it such as AnchorGenerators. How one will configure these settings is non-trivial and often backbone and dataset specific. Some models even need to adjust the anchors depending on the input size. Here is an investigation we did previously around this which shows how sensitive this is: #3819 (comment). As a result it's rather difficult to provide one generic method that can attach any backbone to any head and expect it to work (at least that's true for Anchor-based techniques with FPNs etc).

This is the reason we only offer a limited number of models for Detection paired with specific backbones. Typically those are the backbones that their original paper used but sometimes we bended this rule (for example fasterrcnn_mobilenet_v3_large_fpn) to provide better support to for mobile specific applications. Having said that, there is interesting work that researchers are doing in the field to simplify the detection task. For example see this paper by @lyttonhao et al.

So your proposal makes sense, but it's quite hard to achieve with the current models. But it's something that there is active research on and might become possible sooner than later. If I misunderstood the question, please ignore the above and let me know what I missed. Thanks! :)

[mrinath123]

Yes, this is exactly what I am talking about. Like you said its not easy to like know some internal config. I think if we can solve this issue, it will be really helpful for the users.
If any work related to this comes I would definitely want to contribute.
Thanks for making it know why its a complex process and giving the link of the papers, I will try to study them.

[oke-aditya]

I had actually tried this with any configurable backbone in my own spinoff quickvision. (Shameless plug)

But as @datumbox mentioned out of box suport is hard as it needs proper tried and tested configuration of anchor generator, etc. Supporting FPNs etc is hard as it needs proper layer mapping etc.

See here for a naive implementation

https://github.com/Quick-AI/quickvision/blob/master/quickvision/models/components/torchvision_backbones.py

This proposal is often experimented.

Many such libraries have tried MixNMatch for allowing customisable heads, Necks, Backbones etc. See mmdetection / icevision etc. While it offers flexibility it sometimes leads too too many confusions on creating a proper architecture. Torchvision has to be rock solid in providing any implementation / possible methods to allow such combinations. So it's very hard to offer generic API as pointed before.

(Quickvision is just a pet project before we started batteries included / multi weight etc. So lot of features from them are old and moved to torchvision too)

[mrinath123]

Nice repo!
I too made something similar https://github.com/mrinath123/timmFasterRcnn I used timm backbone models rather than pytorch vision, but as you can see in the model config, there is a lot of manual work to do.
I hope there will be a easier solution to do this.