Authoring guidelines: preloaded code

[hobovsky]

Breakdown of #139

Set of guidelines related to writing preloaded code.

Proposed guidelines:

• stick to general coding guidelines
• user solution is able to read a file with preloaded snippet, and call code present in it, access variables and types defined there. do not put reference solution, verification functions, or any information wchich should be hidden from users in preloaded because they can be called, read, or modified by user solution.
• can be used to reduce boilerplate, but remember it's not visible to users in UI. If necessary/convenient, consider putting it in description

[ggorlen]

Preloaded code is visible to users programmatically, see #146 (comment). This is implied by the "do not put reference solution or verification functions" bullet but might be worth making explicit since "remember it's not visible to users, consider putting it in description" seems to contradict this.

[error256]

Is visibility of preloaded different in any language from visibility of the tests? Tests can be read by solutions too. I thought it was so only for some interpreted languages, but it looks like they are available even in C.

[hobovsky]

by "remember it's not visible to users, consider putting it in description" I meant literally "cannot be seen". Authors sometimes happen to put some things there, like test helpers, some data, other things to make them common for sample tests and full tests. This leads to the fact that by just looking at sample tests users cannot recreate them locally. For example see "morse code" kata which keep morse dictionary in preloaded, or some C/C++/Java/C# kata which have return data types or input data types in preloaded, so user cannot see how its defined.

[Blind4Basics]

Is visibility of preloaded different in any language from visibility of the tests? Tests can be read by solutions too.

Yes. As long as we aren't talkng about "hacks/cheats" (those who go opend the tests file manually), there are languages where the tests cannot be programmatically accessed from the soltuion: python, for instance.

[ggorlen]

That's right, you can read the preloaded source file from any language that lets you run a subprocess or read a file which I assume is all languages. SQL often has the solution query in preloaded, making cheating easy. There was some talk of unlinking the preloaded source file by default before running the candidate solution but I doubt this has been implemented, or is needed assuming authors are aware that it's readable. Even after unlinking, data may be exposed accidentally.

I hadn't considered the use case of wanting to run tests locally; that makes sense. I generally avoid preloaded for reasons like this--it hides information. Having said that, if you have some setup code that preps a database or something, plopping 50-100 lines into the sample tests is pretty distracting and this is when preloaded is useful. Given a choice, I'd optimize for the in-browser user over the local development user, although maybe having all non-hidden files like preloaded downloadable as a zip or repo would make it convenient for both (that's another can of worms).

The important thing is that authors know that preloaded is not visible on the UI but is readable for all intents and purposes, which is why no secure information can be kept there.