RegNet in torchvision ?

[blefaudeux]

🚀 Feature

Add RegNet trunks in torchvision

Motivation

RegNets were proposed in this paper https://arxiv.org/pdf/2003.13678.pdf, they're showing very interesting performance and speed. They have been open sourced already, but are not usable in a straightforward way for people used to having reference models in torchvision. Another implementation is available in ClassyVision (I'm a co-author of this one), but it does not cover all use cases.

Pitch

Start from the ClassyVision RegNet support and implement RegNets in torchvision.

Alternatives

Let users use RegNets from external implementations

Additional context

This has been discussed with @pdollar, one of the RegNet authors. CC @fmassa

cc @vfdev-5

[vfdev-5]

@blefaudeux thanks for the suggestion ! For which tasks you think about for this model: at least classification, right ?

Another implementation is available in ClassyVision (I'm a co-author of this one), but it does not cover all use cases.

Could you detail which use-cases ClassyVision implementation does not cover ?

Would you like to draft a PR for that ? Otherwise, me or someone else can do that.

[blefaudeux]

@blefaudeux thanks for the suggestion ! For which tasks you think about for this model: at least classification, right ?

Another implementation is available in ClassyVision (I'm a co-author of this one), but it does not cover all use cases.

Could you detail which use-cases ClassyVision implementation does not cover ?

Oh, I just meant that not everyone is using ClassyVision obviously, I for instance came across users telling me that they were sticking to EfficientNets or ResNets because they were only wiling to consider Torchvision.

Would you like to draft a PR for that ? Otherwise, me or someone else can do that.

I'm not sure what there is to know for a model to be supported by torchvision, apart from the raw code (which I can handle indeed, or anybody else, no preference). Are there some licence pre-requisites, pre-trained models, authorship constraints (validation from the original authors ?), things like that ? I don't have that much time right now, so if the requirements are clear (or minimal :)) I can handle that starting from the implementation in ClassyVision, else if there's some know-how required I would gladly stay around to assist but not do it myself

[vfdev-5]

Are there some licence pre-requisites, pre-trained models, authorship constraints (validation from the original authors ?), things like that ?

Excellent question. I'd say we have to provide model's implementation and ImageNet pretrained weights. In the docstring we provide some information about the model, link to the paper etc. For example, MNasNet:

vision/torchvision/models/mnasnet.py

Lines 204 to 208 in 190a5f8

	def mnasnet0_5(pretrained=False, progress=True, **kwargs):
	"""MNASNet with depth multiplier of 0.5 from
	`"MnasNet: Platform-Aware Neural Architecture Search for Mobile"
	<https://arxiv.org/pdf/1807.11626.pdf>`_.
	Args:

ImageNet pretrained weights are often coming from retraining, but there are cases when they were converted let's say from TF weights etc. If torchvision's implementation will be a copy from ClassyVision, maybe we can reuse their weights if there are ones...

@fmassa can you also comment out this question, please ?

I don't have that much time right now, so if the requirements are clear (or minimal :)) I can handle that starting from the implementation in ClassyVision, else if there's some know-how required I would gladly stay around to assist but not do it myself

No worries. I send a PR where I can copy and adapt the implementation from ClassyVision and you could check if we are correct etc.

[blefaudeux]

ImageNet pretrained weights are often coming from retraining, but there are cases when they were converted let's say from TF weights etc. If torchvision's implementation will be a copy from ClassyVision, maybe we can reuse their weights if there are ones...

I could provide reference weights from a ClassyVision ImageNet training, fairly easy to reproduce (if we're ok to limit this to some members of the RegNet family, probably not all of them :)). Another option is to translate the weights in the PyCls repo/model zoo, but that's some work because the model definition is not exactly the same (even if the actual underlying architecture is, of course). CC @mannatsingh from Classy

[pdollar]

Yes, all model weights are available in pycls, not sure how easy it is to convert them to classy vision format. Link: https://github.com/facebookresearch/pycls/blob/master/MODEL_ZOO.md

[vfdev-5]

@blefaudeux @pdollar thanks for the details ! So, we prefer to have here certain families of RegNet implemented as in Classy than the ones in pycls format right ? I let you define those families you think the most interesting for the users.
If there are families we would like to include, but you do not have the weights, we can retrain them too.

It would be, probably, helpful for the future to add some info, here in torchvision, on the reason why we prefered one implementation (classy) to another (pycls) if they are a bit different...

[blefaudeux]

@blefaudeux @pdollar thanks for the details ! So, we prefer to have here certain families of RegNet implemented as in Classy than the ones in pycls format right ? I let you define those families you think the most interesting for the users.
If there are families we would like to include, but you do not have the weights, we can retrain them too.

It would be, probably, helpful for the future to add some info, here in torchvision, on the reason why we prefered one implementation (classy) to another (pycls) if they are a bit different...

Sorry for the imprecision, I forgot that the context is clearly not trivial, trying to address that :

• the pycls implementation is the original one, straight from FAIR (research), it exposes a lot of features which were mostly there for experimentation (architecture search) but are not useful once the best models in the family were found
• the ClassyVision implementation was tentatively more production ready, a little easier to read, with all the code to reproduce the paper but not more
• what I meant by "not all the regnets" in the above was that RegNets define a family of models, each of them being fully defined by a couple of coefficients, loosely comparable with ResNext for instance (change the width, depth, etc.. even if the scaling is a little more complex here). I just meant that one has to decide on a couple of models whose weights would be provided, for instance the models comparable to RN50, RN101, things like that, but covering each and every data point may not be needed. Additionally RegNets can scale to very big models for vision, 128GF or more, and training these would be very costly and probably not needed for torchvision users, or at least that was my assumption.

[vfdev-5]

@blefaudeux thanks for the explanation ! I see better the context. Yes, this sounds good !

I just meant that one has to decide on a couple of models whose weights would be provided, for instance the models comparable to RN50, RN101, things like that

If you have an idea of which RegNetX, RegNetY it would be good to provide with pretrained weights. Maybe, those from the paper's tables 5 and 6 :

• REGNETX-3.2GF
• REGNETX-6.4GF
• REGNETX-12GF
• REGNETX-4.0GF
• REGNETX-8.0GF
• REGNETY-400MF
• REGNETY-600MF
• REGNETY-800MF
• REGNETY-1.6GF
• REGNETY-4.0GF
• REGNETY-8.0GF

What do you think ?

[blefaudeux]

Looks good to me ! Probably worth it writing a small tool to transcribe the weights in between pycls and this more streamlined implementation, it's something I could do. Not sure whether @mannatsingh would have something handy around to help on that ?

[mannatsingh]

@blefaudeux I don't have anything available to convert pycls weights to classy, unfortunately.
While you should be able to get similar results by training the models from scratch in Classy (we verified this for a few models), it might be easier to just convert the weights available in pycls.

[pdollar]

What is the motivation for only including a subset of the models? Is it because they have to be retrained?

The reason I ask is that the benefit of RegNets is they give good accuracy models across a wide range of flop regimes, as opposed to say ResNet which typically is only optimized for a narrow range of 4GF-12GF (ResNet50-ResNet152). On the other hand RegNets can be good at very small sizes (200MF) and very large sizes (32GF). The very small and very large models are potentially the most interesting (for say mobile and state-of-the-art research). So I would advocate including the full range of models if possible.

[pdollar]

It may be better to figure out how to convert pycls weights to classy vision weights. i don't know classy vision well, but i can't imagine it being that hard?!? (famous last words :P )