Table of Contents
The next few chapters describe the process of preparing and uploading project files through the Submit Server, which are then used by the Build Server to compile and grade student submissions. This chapter describes the files that make up this process and process of uploading them into the Submit Server.
The process of building and testing one submission requires two files: a submission archive containing an implementation of the specification and a test setup archive containing test cases that either pass or fail when run against a submission. Both of these files can be in the zip format or the Java jar format.
Some sample files are provided with the Marmoset Distribution and linked here:
test-setup.jar: Test cases for an example specification.
canonical.jar: A submission that completely implements the example specification.
skeleton.jar: The initial project files given to students. This is essentially an incomplete implementation of the specification.
For the full set of examples and templates provided with the marmoset distribution, download examples.jar.
The next two chapters: Chapter 12, Creating Java Projects and Chapter 13, Creating C Projects provide step by step instructions for creating the necessary files for Java and C projects respectively. The model used for C projects can also be used for other languages such as C++, OCaml and Ruby (see Section 6, “Applying The C Model to Other Languages”).
Student submissions are stored in zip/jar files, referred to as submission archives. They contain the source code files for the student's project. They may also contain other files: for example, test cases provided to the students, the student's own test cases, image or data files, etc.
When setting up a project, an instructor must make a submission that passes all the tests. This submission archive is referred to as the canonical solution. An example of a canonical solution is canonical.jar.
Submission archives are usually created by archiving the entire student folder. Marmoset then looks for the source files and compiles them. This process is quite robust but sometimes problems can occur if files are in unusual places. The following rules are helpful to avoid any problems:
In java projects, source files should be in a
srcdirectory and should be arranged in the standard directory structure. E.g. the class cs101.project1.HelloWorld should be located insrc/cs101/project1/HelloWorld.java.In C projects, source and make files should be at the top level.
Any non-source files referenced using relative paths should be in a directory structure that starts at the top level. This is because the programs are run from the top level. So if a program references
examples/input.txt, then the directoryexamplesshould be at the top level of the archive.![[Note]](note.png)
Note When student programs access non-source files, the test.runInInstructorDirproperty (in the test.properties file) determines which files are visible. If this property is true, then files the student's submission are not visible because the program is run from the instructor's directory, not the student's directory. See Section 2, “The Test Setup Archive” for more information about the test.properties file and see Section 3, “Test Properties” for more information about thetest.runInInstructorDirproperty.
If the project is done using the
Eclipse IDE, the student
can simply archive the entire project directory and submit this.
Marmoset uses Eclipse's .classpath file to figure
out where all the source files are. There is also an Eclipse plugin
provided by the University of Maryland that can be configured to
allow students to submit from within Eclipse (see
Chapter 14, The Eclipse Plugin).
All of the information about how to build and test submissions for a particular course project is contained in the test setup archive. This archive is a zip or jar file containing several elements:
- The test.properties file
This is a Java properties file defining metadata about how to build and test a submission for the particular project. This file must be in the root directory of the test setup archive. For a detailed description of the test.properties file, see Section 3, “Test Properties”.
- Test Cases and Supporting Source or Object Files
The test setup archive contains source and/or object files needed to compile and test the student source code.
For Java projects, only .class files need to be provided in the test setup archive. (However, we recommend including source files in the test setup because this is often the only record of the test cases in the Marmoset database. Instructors and researchers that review your course in the future often want to see the source of the test cases.)
The test setup jarfile is provided on the compile-time and runtime classpath for building/executing the student's code. For testing student code, JUnit test cases should be provided. More information is in Chapter 12, Creating Java Projects.
For C projects, either C source files or object files may be provided. (Again we recommend including the source files.) The individual test cases must be standalone executables linked against the student's modules. More information is in Chapter 13, Creating C Projects.
- Files needed by Test Cases at Runtime
If the test cases require files when they are executed, the files should be stored in the test setup jarfile. These files will be extracted into the testfiles directory, and will be available to the tests at testing time.
- The security.policy file (optional, Java-only)
For Java projects, a security policy file may be provided, specifying what privileges are granted to student code. It is important to prevent untrusted student code from forging test outcomes, sending secret test information over the network, etc. This file, if present, must be located in the root directory of the test setup jarfile.
If no security policy file is specified, a restrictive default security policy is used.
- Makefile (C-only)
Test setup jarfiles for C projects must include a Makefile capable of compiling the executables that will serve as the test cases. The Makefile must be in the root directory of the test setup jarfile.
Chapter 12, Creating Java Projects and Chapter 13, Creating C Projects provide more details on what needs to be in the test setup jarfile for Java projects and C projects.
This chapter describes how to define a test.properties file for a project. As noted in Chapter 10, Projects , this file must be placed in the root directory of the test setup jarfile.
The format of the test properties file is described in the Java API documentation for the load() and store() methods of the java.util.Properties class .
Table 11.1. Test properties
| Property Name | Value | Language | Meaning |
|---|---|---|---|
| build.language | "c","java","ruby", or "ocaml" | Any | Specifies the source language of the project. Case is ignored, so "Java", "java", and "JAVA" are all equivalent. Currently C, Java, Ruby and OCaml projects are supported. If this property is not specified, it defaults to "Java". |
| build.sourceVersion | "1.4", "1.5", etc. | Java | Specifies the value of the "-source" command line option used when invoking javac . If not specified, defaults to "1.5". |
| build.make.command | Path of make command | C | Specifies the full path of the make command used to build student submissions. If not specified, defaults to "/usr/bin/make". Instructors should not need to set this property unless they are administering their own buildserver. |
| build.make.file | Filename of makefile | C | Specifies the filename of the makefile. If not specified, the makefile is left implicit, meaning that the make utility will decide which one to use. If this property is specified, it will be passed as part of the "-f" option to the make command. |
| test.class.{public,release,secret,student} | Name of JUnit TestCase class (Java), or comma-separated list of test executables (C) | Java or C | For Java projects, specifies the full classname of a JUnit TestCase class which performs the public, release, secret or student tests. For C/Ruby/OCaml projects, specifies the names of the executables which represent the public, release, or secret tests. Note that C projects do not support student-written tests that are executed by the BuildServer, as this would be insecure and complicated. This property must NOT be left blank; i.e. if a project has no secret tests, then this property must be omitted. |
| test.timeout.testCase | Timeout in seconds | Java | Specifies the amount of time a single test method is allowed to execute before considering the test failed. If not specified, defaults to 30 seconds. |
| test.timeout.process | DEPRECATED: This field is ignored (Used to be Timeout in seconds) | Java or C | deprecated |
| test.output.maxBytes | Number of bytes | Java or C | Specifies the maximum number of bytes of output that will be captured from a test process. If not specified, defaults to 1 MB. |
| test.runInInstructorDir | true or false | Java | If true, test processes are run in the directory containing the extracted test setup jarfile (the test files directory containing instructor-provided files), rather than the build directory (containing student submission files). The default is to run in the build directory. |
| test.performCodeCoverage | true or false | Java | Should the buildServer try to perform code coverage? The BuildServer has Clover, a code-coverage tools, installed into it and can collect code coverage results for each test case. This is especially useful when combined with student-written tests because you can download the code-coverage stats from the server and use them for grading purposes to support test-driven development for students. Note that it is possible for students to submit a zip archive through the web interface in a format such that it can be built and tested but the code coverage tool will not work properly. |
Once you have created the three archives, you are ready to initialize your project on the Submit Server by uploading these files. Go to the Project Utilities page to upload all three archives. The test-setup and canonical archives can be uploaded at the bottom of this page. The skeleton is uploaded by selecting "Upload new project starter files" in the Project Maintenance section.
Submit Server sends the test-setup and canonical archives to a Build Server for testing. While this is happening, the state of your test setup should be set to pending. If this is not the case, contact your Marmoset administrator to ensure that the Build Servers have been activated. If any of the tests fail, the test-setup status will be set to "failed". Check to make sure the format of your archives is correct.
Once all the tests in your test-setup pass on the canonical solution, the test-setup status is upgraded to "tested". This may take a few minutes. You can now assign points to each of the tests by clicking the "Assign Points" link. When you assign points to a test-setup, it becomes the active test setup, meaning that all student submissions will be tested against this setup.
Once all this is done, you can make the project visible to students by selecting the "Make Visible" button at the top of the page.
This section describes how the build server uses the test setup to test the submissions.
Two directories are used by the BuildServer at runtime. The build directory is where the student submission is extracted. The test files directory is where the test setup jarfile is extracted.
The idea behind keeping these two directories separate is that it prevents commingling of student and instructor files. However, for C projects, a single directory is used as both the build directory and the test files directory.
The following steps are used when building and testing a student submission.
The submission zipfile is extracted into the build directory.
The test setup jarfile is extracted into the test files directory.
The submission is built: either by issuing a command to build all of the source files in the submission (for Java projects) or by invoking a make command (for C projects). If the build fails, the compiler error messages are recorded and a failed build test outcome is recorded.
The public, release, secret and student tests specified in the test setup jarfile's test.properties file are executed and test outcomes recorded.
The buildServer supports two categories of programming languages: Java
projects (which use a security manager and JUnit) and C or C-like
projects (which use the
make
utility to create executable files which are then run run by the
BuildServer.
The mechanism used for C projects is flexible enough that we've used it for Ruby and OCaml projects as well. Other langauges, such as C++ or Python, will also work using the same mechanism.
Thus when we write "C project", keep in mind that this mechanism is extremely flexible and is not certainly not limited to projects written in the C language.