CI job `cmake_windows_cpu` failing

[datumbox]

🐛 Describe the bug

Using PyTorch Core 20220824, the job started failing with:

+ ./test_frcnn_tracing

Exited with code exit status 127
CircleCI received exit code 127

This is likely due to a change on Core, possibly on the jit side. It was originally masked by another similar issue at #6477 but this one seems unrelated.

Versions

Latest TorchVisio main branch and any Core nightly after 20220824

cc @seemethere

[janeyx99]

I'm having trouble figuring out where in the logs the CI fails. Could you give me a pointer? It looks like the build succeeds but something exits with error code 127.

[datumbox]

@janeyx99 here is the relevant part from the build scripts:

vision/packaging/build_cmake.sh

Lines 86 to 106 in 59ef2ab

	# Trace, compile and run project that uses Faster-RCNN
	pushd test/tracing/frcnn
	mkdir build
	# Trace model
	python trace_model.py
	cp fasterrcnn_resnet50_fpn.pt build
	cd build
	cmake .. -DTorch_DIR=$TORCH_PATH/share/cmake/Torch -DWITH_CUDA=$CMAKE_USE_CUDA
	if [[ "$OSTYPE" == "msys" ]]; then
	"$script_dir/windows/internal/vc_env_helper.bat" "$script_dir/windows/internal/build_frcnn.bat" $PARALLELISM
	mv fasterrcnn_resnet50_fpn.pt Release
	cd Release
	export PATH=$(cygpath "C:/Program Files (x86)/torchvision/bin"):$(cygpath $TORCH_PATH)/lib:$PATH
	else
	make -j$PARALLELISM
	fi
	# Run traced program
	./test_frcnn_tracing

We first build the test_frcnn_tracing example project that loads a jit-scripted model and tries to run it. If I had to guess there is probably some change on JIT coming from Core that make the program crash.

@atalman what's the best way to debug this error code 127 on Windows and get a more helpful error?

[atalman]

Error code 127 means file or program not found. The error seems to be related with wrong PATH config:

Time Elapsed 00:00:15.76
+ mv fasterrcnn_resnet50_fpn.pt Release
+ cd Release
++ cygpath 'C:/Program Files (x86)/torchvision/bin'
++ cygpath 'C:\tools\miniconda3\lib\site-packages\torch'
+ export 'PATH=/cygdrive/c/Program Files (x86)/torchvision/bin:/cygdrive/c/tools/miniconda3/lib/site-packages/torch/lib:/c/Program Files (x86)/Windows Application Driver:/c/Program Files (x86)/Microsoft Visual Studio/2019/Community/MSBuild/Current/Bin:/c/Program Files (x86)/Microsoft Visual Studio/Installer:/c/tools/ruby26:/c/tools/ruby26/bin:/c/ProgramData/nvm:/c/miniconda3/miniconda3/condabin:/c/tools/miniconda3:/c/tools/miniconda3/Library/mingw-w64/bin:/c/tools/miniconda3/Library/usr/bin:/c/tools/miniconda3/Library/bin:/c/tools/miniconda3/Scripts:/c/Program Files (x86)/Microsoft SDKs/Azure/CLI2/wbin:/c/Windows/system32:/c/Windows:/c/Windows/System32/Wbem:/c/Windows/System32/WindowsPowerShell/v1.0:/c/Windows/System32/OpenSSH:/c/ProgramData/GooGet:/c/Program Files/Google/Compute Engine/metadata_scripts:/c/Program Files (x86)/Google/Cloud SDK/google-cloud-sdk/bin:/c/Program Files/PowerShell/7:/c/Program Files/Google/Compute Engine/sysprep:/c/Program Files/Docker:/c/ProgramData/chocolatey/bin:/cmd:/mingw64/bin:/usr/bin:/c/Program Files/Git LFS:/c/Program Files/Amazon/AWSCLI/bin:/c/Program Files/Microsoft/Web Platform Installer:/c/Program Files/Microsoft Service Fabric/bin/Fabric/Fabric.Code:/c/Program Files/Microsoft SDKs/Service Fabric/Tools/ServiceFabricLocalClusterManager:/c/Program Files (x86)/vim/vim80:/c/Program Files/Go/bin:/c/Program Files/OpenJDK/jdk-12.0.2/bin:/c/ProgramData/nvm:/c/Program Files/nodejs:/c/Program Files/dotnet:/c/Program Files (x86)/Windows Kits/10/Windows Performance Toolkit:/c/Program Files/Microsoft SQL Server/Client SDK/ODBC/170/Tools/Binn:/c/Program Files (x86)/Microsoft SQL Server/150/Tools/Binn:/c/Program Files/Microsoft SQL Server/150/Tools/Binn:/c/Program Files/Microsoft SQL Server/150/DTS/Binn:/c/Program Files/Microsoft SQL Server/130/Tools/Binn:/c/Program Files (x86)/IncrediBuild:/c/Users/circleci/AppData/Local/Microsoft/WindowsApps'
+ PATH='/cygdrive/c/Program Files (x86)/torchvision/bin:/cygdrive/c/tools/miniconda3/lib/site-packages/torch/lib:/c/Program Files (x86)/Windows Application Driver:/c/Program Files (x86)/Microsoft Visual Studio/2019/Community/MSBuild/Current/Bin:/c/Program Files (x86)/Microsoft Visual Studio/Installer:/c/tools/ruby26:/c/tools/ruby26/bin:/c/ProgramData/nvm:/c/miniconda3/miniconda3/condabin:/c/tools/miniconda3:/c/tools/miniconda3/Library/mingw-w64/bin:/c/tools/miniconda3/Library/usr/bin:/c/tools/miniconda3/Library/bin:/c/tools/miniconda3/Scripts:/c/Program Files (x86)/Microsoft SDKs/Azure/CLI2/wbin:/c/Windows/system32:/c/Windows:/c/Windows/System32/Wbem:/c/Windows/System32/WindowsPowerShell/v1.0:/c/Windows/System32/OpenSSH:/c/ProgramData/GooGet:/c/Program Files/Google/Compute Engine/metadata_scripts:/c/Program Files (x86)/Google/Cloud SDK/google-cloud-sdk/bin:/c/Program Files/PowerShell/7:/c/Program Files/Google/Compute Engine/sysprep:/c/Program Files/Docker:/c/ProgramData/chocolatey/bin:/cmd:/mingw64/bin:/usr/bin:/c/Program Files/Git LFS:/c/Program Files/Amazon/AWSCLI/bin:/c/Program Files/Microsoft/Web Platform Installer:/c/Program Files/Microsoft Service Fabric/bin/Fabric/Fabric.Code:/c/Program Files/Microsoft SDKs/Service Fabric/Tools/ServiceFabricLocalClusterManager:/c/Program Files (x86)/vim/vim80:/c/Program Files/Go/bin:/c/Program Files/OpenJDK/jdk-12.0.2/bin:/c/ProgramData/nvm:/c/Program Files/nodejs:/c/Program Files/dotnet:/c/Program Files (x86)/Windows Kits/10/Windows Performance Toolkit:/c/Program Files/Microsoft SQL Server/Client SDK/ODBC/170/Tools/Binn:/c/Program Files (x86)/Microsoft SQL Server/150/Tools/Binn:/c/Program Files/Microsoft SQL Server/150/Tools/Binn:/c/Program Files/Microsoft SQL Server/150/DTS/Binn:/c/Program Files/Microsoft SQL Server/130/Tools/Binn:/c/Program Files (x86)/IncrediBuild:/c/Users/circleci/AppData/Local/Microsoft/WindowsApps'
+ ./test_frcnn_tracing

vs correct one:

++ cygpath 'C:/Program Files (x86)/torchvision/bin'
++ cygpath 'C:\tools\miniconda3\lib\site-packages\torch'
+ export 'PATH=/c/Program Files (x86)/torchvision/bin:/c/tools/miniconda3/lib/site-packages/torch/lib:/c/Program Files (x86)/Windows Application Driver:/c/Program Files (x86)/Microsoft Visual Studio/2019/Community/MSBuild/Current/Bin:/c/Program Files (x86)/Microsoft Visual Studio/Installer:/c/tools/ruby26:/c/tools/ruby26/bin:/c/ProgramData/nvm:/c/miniconda3/miniconda3/condabin:/c/tools/miniconda3:/c/tools/miniconda3/Library/mingw-w64/bin:/c/tools/miniconda3/Library/usr/bin:/c/tools/miniconda3/Library/bin:/c/tools/miniconda3/Scripts:/c/Program Files (x86)/Microsoft SDKs/Azure/CLI2/wbin:/c/Windows/system32:/c/Windows:/c/Windows/System32/Wbem:/c/Windows/System32/WindowsPowerShell/v1.0:/c/Windows/System32/OpenSSH:/c/ProgramData/GooGet:/c/Program Files/Google/Compute Engine/metadata_scripts:/c/Program Files (x86)/Google/Cloud SDK/google-cloud-sdk/bin:/c/Program Files/PowerShell/7:/c/Program Files/Google/Compute Engine/sysprep:/c/Program Files/Docker:/c/ProgramData/chocolatey/bin:/cmd:/mingw64/bin:/usr/bin:/c/Program Files/Git LFS:/c/Program Files/Amazon/AWSCLI/bin:/c/Program Files/Microsoft/Web Platform Installer:/c/Program Files/Microsoft Service Fabric/bin/Fabric/Fabric.Code:/c/Program Files/Microsoft SDKs/Service Fabric/Tools/ServiceFabricLocalClusterManager:/c/Program Files (x86)/vim/vim80:/c/Program Files/Go/bin:/c/Program Files/OpenJDK/jdk-12.0.2/bin:/c/ProgramData/nvm:/c/Program Files/nodejs:/c/Program Files/dotnet:/c/Program Files (x86)/Windows Kits/10/Windows Performance Toolkit:/c/Program Files/Microsoft SQL Server/Client SDK/ODBC/170/Tools/Binn:/c/Program Files (x86)/Microsoft SQL Server/150/Tools/Binn:/c/Program Files/Microsoft SQL Server/150/Tools/Binn:/c/Program Files/Microsoft SQL Server/150/DTS/Binn:/c/Program Files/Microsoft SQL Server/130/Tools/Binn:/c/Program Files (x86)/IncrediBuild:/c/Users/circleci/AppData/Local/Microsoft/WindowsApps'
+ PATH='/c/Program Files (x86)/torchvision/bin:/c/tools/miniconda3/lib/site-packages/torch/lib:/c/Program Files (x86)/Windows Application Driver:/c/Program Files (x86)/Microsoft Visual Studio/2019/Community/MSBuild/Current/Bin:/c/Program Files (x86)/Microsoft Visual Studio/Installer:/c/tools/ruby26:/c/tools/ruby26/bin:/c/ProgramData/nvm:/c/miniconda3/miniconda3/condabin:/c/tools/miniconda3:/c/tools/miniconda3/Library/mingw-w64/bin:/c/tools/miniconda3/Library/usr/bin:/c/tools/miniconda3/Library/bin:/c/tools/miniconda3/Scripts:/c/Program Files (x86)/Microsoft SDKs/Azure/CLI2/wbin:/c/Windows/system32:/c/Windows:/c/Windows/System32/Wbem:/c/Windows/System32/WindowsPowerShell/v1.0:/c/Windows/System32/OpenSSH:/c/ProgramData/GooGet:/c/Program Files/Google/Compute Engine/metadata_scripts:/c/Program Files (x86)/Google/Cloud SDK/google-cloud-sdk/bin:/c/Program Files/PowerShell/7:/c/Program Files/Google/Compute Engine/sysprep:/c/Program Files/Docker:/c/ProgramData/chocolatey/bin:/cmd:/mingw64/bin:/usr/bin:/c/Program Files/Git LFS:/c/Program Files/Amazon/AWSCLI/bin:/c/Program Files/Microsoft/Web Platform Installer:/c/Program Files/Microsoft Service Fabric/bin/Fabric/Fabric.Code:/c/Program Files/Microsoft SDKs/Service Fabric/Tools/ServiceFabricLocalClusterManager:/c/Program Files (x86)/vim/vim80:/c/Program Files/Go/bin:/c/Program Files/OpenJDK/jdk-12.0.2/bin:/c/ProgramData/nvm:/c/Program Files/nodejs:/c/Program Files/dotnet:/c/Program Files (x86)/Windows Kits/10/Windows Performance Toolkit:/c/Program Files/Microsoft SQL Server/Client SDK/ODBC/170/Tools/Binn:/c/Program Files (x86)/Microsoft SQL Server/150/Tools/Binn:/c/Program Files/Microsoft SQL Server/150/Tools/Binn:/c/Program Files/Microsoft SQL Server/150/DTS/Binn:/c/Program Files/Microsoft SQL Server/130/Tools/Binn:/c/Program Files (x86)/IncrediBuild:/c/Users/circleci/AppData/Local/Microsoft/WindowsApps'
+ ./test_frcnn_tracing
Loading model
Model loaded