How to Create a Problem

Before jumping into the steps for building a problem, let's start by reviewing the core pieces involved:

• Problem - the outer "shell" of the overall task that contains information common to all variants (such as the title and other metadata like disciplines, interests, and keywords).

• Problem variant(s) - the version of a problem that contains language-specific instructions, example code, and starter code for Fix Code and Write Code problem types. A problem can have 1 or more variants.

• Parsons problem(s) - a block-based way to solve a given problem variant, comprised of Parsons blocks and Parsons solutions. A variant can have 1 or more Parsons problems.

• Unit test(s) - rules for validating the correctness of a learner's solution for a particular variant.

Get started by logging into Codespec as an instructor, researcher, or peer instructor. Click "Problem" in the "Create" dropdown menu from any page, or "New Problem" on the "My Work / Problems" page.

Enter a title for the problem that is language-neutral (avoid putting a specific language name in the title as this name will appear across all variants of the problem).

From the list of provided disciplines and interests, choose any that are relevant to the context of your problem. These choices will act as metadata tags that help learners practice with problems that align with their own areas of study or hobbies.

From the list of keywords, search for matching results or add new ones. Any keyword you add will be available to all other peer instructors, instructors, and researchers to use in the platform going forward.

Then, click "Save and Create Variant" to continue the problem set-up process.

There are two ways to create a problem variant (or language version) of a problem: manually or with help from artificial intelligence. We'll describe each of them separately here.

Manual Creation

The manual option provides you the most control over the instructions and solution code from the start. After selecting a compiler (or language) for the problem, there are 5 additional fields to fill out:

• knowledge components (optional) - items you select will become another piece of metadata that helps match learners with problems most relevant to them. Knowledge components represent language specific concepts that the problem incorporates (under the hood, they are derived from node types in an abstract syntax tree for the chosen programming language).

• instructions (required) - content in this field will serve as the problem variant's instructions and example code. Instructions exist at the variant level, and not the parent/problem level because not all programming languages refer to similar concepts in the same way (for example, hashes vs dictionaries, or lists vs arrays).

• pre-code (optional) - code that will not be shown to the learner but will be added before their solution. This is useful for adding extra import statements or setting hidden variables for testing purposes.

• solution code (required) - code that solves the problem described in the instructions. This space provides a "shortcut" for populating the problem's first Parsons problem. Using the following syntax, problem authors can write each line's code, pseudocode, and faded text all at once.

  • Start by writing a line of code as you normal would in an editor

  • Add a comment symbol

  • To add pseudocode (plain English description of the preceding code for that line), enter ::pseudocode:: followed by the text you want to display on the Pseudocode problem type.

  • To add faded text for a line, enter ::faded_text:: followed by the text you want to hide on the Faded Parsons problem type.

  • To add fix-code text for a line, enter ::fix_code_text:: followed by the text you want to initially display for the block. This will be the buggy/incorrect text learners have to fix.

  • To add a distractor block, enter ::distractor_pseudocode:: AND ::distractor_text::followed by the respective content. This will create a distractor block and automatically associate it with the correct block.

  • Two important notes:

    • In order to submit the variant form, each line in the solution code must have accompanying pseudocode.

    • Faded text is optional, but has one limitation: Codespec will look for the specified faded text in the preceding code for that line (anything to the left of the comment symbol) and hide that. If you have multiple uses of the target faded text in the code and want to hide instances other than the first one, that is not possible at this time.

def example_function(arg): # ::pseudocode:: DEFINE the function and argument ::faded_text:: arg

• post-code (optional) - code that will not be shown to the learner but will be added after their solution.

Submitting this form will create the problem variant, as well as an initial Parsons problem, complete with Parsons blocks and a solution based on the provided solution code.

Creation with Artificial Intelligence

The second way to create a problem variant is with artificial intelligence, which requires the following input:

• compiler - the language you want the variant written in

• minimum and maximum solution length (optional) - the lower and upper range of lines you'd prefer for the auto-generated solution code to be. Disclaimer: we're currently fine-tuning the prompt and model, so results based on these parameters may vary.

• knowledge components (optional) - any particular programming concepts you'd like the auto-generated solution code to incorporate

Once you've provided the above criteria, click "Save and Create". Codespec will auto-generate instructions and solution code to the best of its ability.

On the following page, you'll be able to edit the instructions and solution code to your liking, or regenerate them.

When you are ready to finalize the variant, click "Save". Doing so will save the variant and create a Parsons problem with blocks and a solution that match the solution code.

If you wish to create additional Parsons problems for a given problem variant, you may do so on the Parsons Problems page for that variant.

Why create multiple Parsons problems for the same variant?

It's often possible to solve the same problem with different approaches, such as using a for-loop or a while-loop. Creating multiple Parsons problems allows you to help learners practice various equivalent concepts and strategies.

If multiple Parsons problems are available on a problem that a learner is viewing in a practice session, Codespec will randomly show one of the options on each problem type.

For problems a learner encounters in a workbook, the presence of multiple Parsons problems allows you to specify which one you want them to see for each problem type.

During the Parsons problem creation process, you may use the name field to internally distinguish between Parsons problems. For example, "For-Loop Parsons Problem" and "While-Loop Parsons problem".

In many cases, it's also possible for learners to solve the same Parsons problem in multiple ways (placing the same blocks in different order, and even different indentation).

For example, if the following code were a Parsons problem, it could be written in 6 different ways and still have the same result:

a = 1
b = 2
c = 3

If learners can theoretically solve a Parsons problem in multiple ways, there should ideally be a Parsons solution for each possibility.

Like with Parsons problems, while creating Parsons solutions you can use the name field to internally distinguish between solutions. Solution names are automatically given in an ascending numerical way (e.g., "Solution 1", "Solution 2", etc).

When you created the problem variant, Codespec used the solution code to populate a first draft of the code learners will see on the Fix Code and Write Code problem types.

Before publishing the problem variant, be sure to modify the fix code content so that it no longer reflects the correct solution code, but instead contains at least 1 error the learner must fix.

By default, Codespec uses the first line from the solution code as the starter content for the Write Code problem type. If you wish to modify this, you may do so on the Write Code page for the problem variant.

No problem variant in Codespec is complete without at least 1 unit test. These tests act as validation measures on all problem types, and are especially important for Fix Code and Write Code problem types, where learners can ultimately solve the problem in any way they want.

Unit tests in Codespec are run as part of a custom testing framework. Currently, the only available test allows for checking equality:

test.assertEquals(1, 1) => True

To add unit tests, visit the Tests page of the problem variant. Each unit test stores the following pieces of information:

• name (required) - a human-readable name for the test

• description (optional) - a plain text description of what the test does

• input (required) - the first argument of the unit test's assertion

• expected output (required) - the second argument of the unit test's assertion

• expected output type (required) - the data type of the expected output

Example Unit Test

If we imagine writing a unit test for the following solution code:

def addition(a, b):
  return a + b

Its components might be:

name:                  Check 2 + 2
description:           Returns 4
input:                 addition(2, 2)
expected output:       4
expected output type:  Integer

And the programmatic representation would look like:

test.assertEquals(addition(2, 2), 4)

At any time during the authoring process, you can see a problem variant will look like to a learner by visiting the Preview page, which provides an interactive playground for testing the problem variant and its various components (Parsons problems, solutions, fix code, write code, and unit tests).

Note: while the preview page is interactive at anytime, it may not behave as desired/expected until you've added all of the necessary components.

Once you are satisfied with the variant and have confirmed all of the problem types can be solved successfully, publish the variant to make it usable in a workbook and eligible for sharing.

By default, published variants are still only visible to/usable by the author. Continue reading to learn more about exposing it to other people.

There are several ways authors can share a problem in Codespec, all of which can be done on the Share page for a problem:

• practice session visible - this setting grants Codespec permission to randomly include this problem and any of its published variants in the pool of potential practice problems for a learner practice session

• template viewership - this collection of settings grants Codespec permission to list this problem and any of its published variants in the Problem Bank, an exclusive library of problems which peer instructors, instructors, and researchers can pull from to create content for their own learners.

• co-editors and co-investigators - these settings allow an author to grant other peer instructors, instructors or researchers the ability to edit or view data for a given problem, respectively.