IDLE editor crashes opening a file with million+ character lines

[AlexWaygood]

Bug report

IDLE crashes if you try to open a .py file with ridiculously long nested dicts in it.

To reproduce

1. Copy and paste the contents of this file into a .py file: repro.txt
2. Attempt to open the file with IDLE.

Observed behaviour

The first time I tried doing this, IDLE opened the file, but then immediately crashed. All subsequent times I have tried doing this, Python has hanged on me requesting IDLE to open the file. (I don't know how long it hangs for; I've interrupted it using CTRL-C every time.)

Comments

This is pretty weird code. The reason why it's so weird is that it's partially generated code: it's the result of a codemod by stubdefaulter.

IDLE is far from the only tool that crashes or hangs when encountering this code. This code also causes black, flake8 and mypy to hang indefinitely. VSCode does open the file, and has no performance issues, but it takes a very long time for pyright's analysis of the file to complete in the background when you open the file with VSCode.

Your environment

• CPython versions tested on:
  • 3.11.2
  • CPython main branch @ 7703def
• Operating system and architecture: Windows

[AlexWaygood]

Cc. @terryjreedy for IDLE

[terryjreedy]

The expert-IDLE tag emails me.

The file is 15 lines, not all the same length, with about 15 million chars. IDLE read() the file in iomenu.load_file, line 142, and inserts it into an empty tk text widget at line 170. Tk text widgets are known to get sluggish with lines somewhere around 2000 chars and unusable at about 10,000 chars. I expect the crash and hang are 3rd party issues. I am leaving this open to a) verify this and b) try to mitigate it a bit by checking chars for total length and longest and average line length and issuing a crash/hang warning when 'excessive'. (While doing so, I should check the effect of wrapping or not. Also consider same check for pasted text and reconsider squeezer check and warning.)

When you say "IDLE loaded the file" did you actually see 15 lines in a normal editor window? For your experiments, was the extension .py or .txt as loaded here? I believe IDLE will highlight syntax only for the former. (15M chars should take maybe a minute.)

EDIT: Alex actually said above 'opened', not 'loaded', which opening was followed by a crash. So I asked whether anything appeared before the crash, and he says 'yes' below.

[AlexWaygood]

When you say "IDLE loaded the file" did you actually see 15 lines in a normal editor window?

Yes.

For your experiments, was the extension .py or .txt as loaded here?

A .py extension, but GitHub doesn't let you upload .py files into bug reports. (This is why I said "copy and paste the contents of this file into a .py file" in my instructions to reproduce the issue.)

[StanFromIreland]

Tested on Linux. Takes well over 20 minutes to load (I forgot when exactly I started it) but it gets there (and then its very slow...). I would vote to close as I doubt this will ever realistically pose an issue. 1984 has roughly half a million chars.

[AlexWaygood]

I would vote to close as I doubt this will ever realistically pose an issue.

Well, it did pose a real issue for me. I opened this issue because I ran a codemod tool, found Black hung on my code after the codemod had been applied, and then tried opening the file with IDLE to see if I could figure out why Black was hanging -- only for IDLE to then also crash/hang. This was all pretty confusing and less than ideal!

I'm not suggesting that IDLE needs to be able to actually work well as a text editor when opening files like this. But a better behaviour might be to detect if the file is too long for IDLE to realistically cope with and refuse to open it, printing an error message to the terminal. Or, alternatively, to detect if trying to open the file is taking too long and abort. There are lots of behaviours that would be better than simply hanging without any explanation!

[terryjreedy]

Closing would be a legitimate response, but hanging is really annoying. So I will leave open for now.

The main issue with long lines is user code output, and we our current imperfect solution is the shell squeezer, inserted between fetching from a socket and inserting into a text widget. We could similarly insert something between chars = f.read() (inmenu 127) and self.text.insert("1.0", chars) (170). The virtue of a timed popup, self.text.after(1500, popup), would be not having to predict a problem. But I don't know if 'after' events can trigger during an insert call.

While pasting could be an issue, it is currently an atomic operation in that clipboard content is directly moved to the text widget. This might be separable, but a 15M paste first requires a 15M copy, which I think even less likely than a 15M file write.