Anatomy of a Submission

When a learner clicks the “Run” button in the problem interface, this assembles a number of different code snippets into one string and sends it to Judge0 to be evaluated. In order of insertion, these snippets include:

1.     Unit test boilerplate code

2.     Pre-code content (defined on the Problem Variant)

3.     The learner’s code

4.     Post-code content (also defined on the Problem Variant)

5.     Unit tests

The unit test boilerplate code defines a custom unit test class that is designed to run various assertion methods and track the results of all unit tests during the evaluation process. Currently, the only available assertion method is “assertEquals”, which checks for strict equality between the unit test’s input and expected output, based on the learner’s code. As long as there are no runtime errors, all unit tests will be evaluated; if one fails, that does not stop the others from being checked.

After the unit test boilerplate code comes the learner’s code, the content of which depends on the problem type of the submission. For Pseudocode and Parsons problem types, Snapshot Blocks are converted into the associated Parsons Block text, and given the appropriate level of indentation based on what the learner specified. Faded Parsons problem submissions work similarly, but for any Snapshot Block in the solution that expected faded text, the learner-entered text (or lack thereof) is used instead of the Parsons Block’s faded text. Fix code and write code problem types are more straightforward, as the respective fix code and write code contents are taken directly from the Snapshot.

Finally, unit test assertion statements are added in the order specified in the Problem Variant console. Their format is the same as what learners see on the “Tests” tab of the problem interface, with one small different: an additional argument is inserted into each assertion to track the unit test that is being evaluated. This allows Codespec to persist the results of each unit test evaluation after the results come back from Judge0.