Sources

Sources are types of tests that OK can recognize. Each source describes how to load, interact with, and dump tests from a given file.

Staff interface

Staff members can specify which sources they want to use for a particular assignment in the configuration file's tests field:

'tests': {
    'file pattern': 'source',
    ...
}

The value of tests is a mapping of file patterns to sources:

• file pattern: a UNIX-file pattern. Patterns may include wildcards *, ? and []-style ranges.
• source: All files in the current directory that are matched by the file pattern will be loaded as the specified source.

Here's an example. Suppose we want to load all Python files in the tests/ directory as OK tests. We would specify this behavior with the following snippet:

'tests': {
    'tests/*.py': 'ok_test'
}

The following is a list of currently supported sources:

• ok_test: tests using OK's test format
• doctest: doctests in a Python module

Developer interface

Sources can be found in the directory client/sources/:

client/
    sources/
        common/
        doctest/
        ok_test/

Each type of source is its own module (in the schematic above, doctest and ok_test are packages). Whether the source module is a standalone Python file (for lightweight sources) or a package (for more complex sources) is up to the developer. The only requirements are the following:

• The source is importable in Python code. This places restrictions on possible names for sources (e.g. no hyphens)

• The source module must have a function called load that returns a dictionary mapping test names to Test objects (or subclasses of Test). load has the following function signature:

    def load(file, parameter, cmd_args):
        """Loads the specified file as a particular source type.
  
        PARAMETERS:
        file      -- str; a filepath to a test. This file is not guaranteed to exist.
        parameter -- str; parameters that may be necessary for loading this source. The
                     source can ignore this parameter if it is not needed.
        cmd_args  -- Namespace; command-line arguments as returned by
                     argparse.ArgumentParser.parse_args().
  
        RAISES:
        client.exceptions.LoadingException; in cases where the test cannot be loaded.
  
        RETURNS:
        dict; a mapping of test names to client.sources.common.models.Test objects, or
        subclasses of Test.
  

The names in the name-to-Test mapping are only used for students to specify certain tests. Each source can choose what to name its tests.

Test objects

The abstract class client.sources.common.models.Test is a serializable object that defines an interface for Test objects.

• Fields:
  • name = core.String()
  • points = core.Float()
  • partner = core.Int(optional=True)
• Methods:
  • run(): Defines how to run tests
  • score(): Defines how to score tests
  • unlock(interact): Defines how to unlock tests. The interact parameter is a function used for processing student input
  • lock(hash_fn): Defines how to lock tests. The hash_fn parameter is the hash function used for generating the locked version of text
  • dump: Defines how to serialize the test. Test's are expected to retain information about where tests should be dumped -- this information is provided by the load function described above.