CodeGrader is an automated grader for lisp programming practicum assessments.
- Follow the instructions available here to install the Lisp programming environment.
Type the commands below an a linux or Mac terminal:
cd ~/quicklisp/local-projects/
git clone https://github.com/marcus3santos/codegrader.git CodeGrader is able to mark students’ solutions that have been downloaded from D2L as a zip file, or solutions that have been saved in the computer where the student took the test. For this tutorial, we assume the latter. In this case, the IT Technicians have provided you with access to the solutions for each student of a given course section, stored in in specific folders named CPS305YY/ENGXXX-ZZ/pt/, where YY is the course section number, ENGXXX is the lab room number, and ENGXXX-ZZ is the computer’s ID number. For example, suppose one student from section 05 took the exam in lab room ENG203 on a PC whose ID is ENG203-03, and another student from section 08 took the same version of the exam in lab room ENG201 on a PC whose ID is ENG201-07. Then, the diagram below represents the corresponding file structure where the two Lisp files containing the solutions for each student have been stored:
- CPS30505/ENGENG203-03/pt/
- q1.lisp
- q2.lisp
- CPS30508/ENG201-07/pt/
- q1.lisp
- q2.lisp
- Create a zipped file containing the students’ solutions: Since
students from specific sections may have taken different versions
of the exam, it is crucial to obtain the sections-to-exam-versions
mapping in advance from the instructor to ensure you are running
CodeGrader on the solutions for the correct course sections and
using the correct test cases for the respective exam version. For
example, suppose students from sections 05 and 08 took Version 1 of
the exam, and students from section 10 took Version 2. You would
have to first create a zip file containing the students’ solutions
for sections 05 an 08. Assuming you want to store that zip archive
file in ~/Zipped-solutions/, the commands below show how to
create the zip file:
cd mkdir Zipped-solutions (cd CPS30505 && zip -r ~/Zipped-solutions/std-sol.zip *) && (cd CPS30508 && zip -r ~/Zipped-solutions/std-sol.zip *)
By zipping this way, you would create a zip archive that does not include the parent directories (CPS30505 and CPS30508); it only includes the ENGXXX-YY/ folders in your zip archive.
- Create a CSV file containing the mapping of students-to-computers: We assume the IT technicians have sent you CSV files containing the student-to-computer mapping for each of the course sections. Now, based on these CSV files and on the sections-to-exams-versions, you should create a CSV file that contains the mappings of all students who took a given test version. Each row in that spreadsheet should contain the following information: Student ID number, Student First Name, Student Last Name, and Room-PC ID
- Prepate the test cases: Prepare a folder containing the test cases lisp files for the assignment you want to mark. You can find folders with test case files for CPS305 Practice Lab Exercises in the Test-Cases directory of this repository.
- Create a folder where CodeGrader will store the results. You can give any name to that folder.
- [ This is step is not necessary if you are assessing an “ungraded” examination (i.e., an assessment
whose weight is zero) ] On D2L, export the students’ assignment grades to a CSV
file. When generating this file, select the following Export
Options:
- Key Field:
- Both
- Grade Values:
- Points grade
- User Details:
- Last name
- First name
Below is an example of a CSV file exported by D2L:
Username,Last Name,First Name,Lab 0X Points Grade <Course Data>,End-of-Line-Indicator #500583619,#TTiger,Tigertongue,Tim,,# #500585612,#Patrick97,Pearson,Patrick,,# #501585619,#Towhander,Twohands,Tony,,# #500586619,#Zain1997,Zodson,Zain,,# #500585619,#Coopercat,Cooper,Cain,,# #500585119,#Hammermann,Odinson,Thor,,#For more information, visit https://www.torontomu.ca/courses/instructors/tutorials/grades/grades-export-import/
- Key Field:
- Launch sbcl from the command line
rlwrap sbcl --dynamic-space-size 20480 - To load the codegrader, type the following commands on the REPL:
(ql:quickload :codegrader)
- To run the students’ solutions through CodeGrader, type the command
below on the CodeGrader REPL: (NOTE: once you launch CodeGrader, it
will start executing the students’ solutions; consequently, it will
display on the REPL buffer all error/warning messages and output
generated by the student’s solution. CodeGrader will be done
marking when you see the message
Exam grading complete!displayed on the REPL window buffer.)(cg:grade-exam submissions map tests-folder results-folder exam-grades-export-file)
where:
submissionsis a string representing the full path and name of the zipped file containing the students’ solutions, e.g.,/Users/johndoe/Zipped-solutions/std-sol.zipmapis a string representing the full path and name of of the csv file storing the student-to-pc mapping.test-folderis a string representing the full path for the test cases folder.results-folderis a string representing the full path for a folder where you want codegrader to store the results (the students’ marks and log files). For example, if you provide the path"/Users/johndoe/A1/"then CodeGrader will create its files/subfolders inside folder
/Users/johndoe/A1/.- (optional)
exam-grades-export-fileis a string representing the full path for the D2L exam grades exported by D2L
CodeGrader generates the following files in the results folder (see above):
- A csv spreadsheet file called
grades.csvThis is a D2L-importable grades file and it is created based on theexam-grades-export-fileargument optionally provided by the user (see items 1 and 2 above). Below is an example of such files:Username,Last Name,First Name,Lab 0X Points Grade <Course Data>,End-of-Line-Indicator #TTiger,Tigertongue,Tim,100.0,# #Patrick97,Pearson,Patrick,72.5,# #Towhander,Twohands,Tony,100.0,# #Zain1997,Zodson,Zain,95.5,# #Coopercat,Cooper,Cain,100.0,# #Hammermann,Odinson,Thor,0.0,#Note:
- If a student exists in the exported file but not in the submissions folder, then the respective grades will not be included in the generated in the respective csv files.
- A Feedback folder that holds feedback files for the students. The
general structure is like this: Consider Timb Handerson who did not
get a full grade. His feedback file will be as such:
Feedback on your assignment solution Unit test results: ((Pass TEST-DEPOSIT (EQUAL (DEPOSIT 20) 130)) (Pass TEST-DEPOSIT (EQUAL (DEPOSIT 10) 110)) (Pass TEST-DEPOSIT (NOT (DEPOSIT 10001))) (Fail TEST-WITHDRAW (EQUAL (WITHDRAW 60) 10)) (Pass TEST-WITHDRAW (NOT (WITHDRAW 80))) (Pass TEST-WITHDRAW (NOT (WITHDRAW 10001))) (Fail TEST-WITHDRAW (EQUAL (WITHDRAW 20) 70)) (Fail TEST-WITHDRAW (EQUAL (WITHDRAW 10) 90))) - A zipped version of the feedback folder. To be uploaded into D2L.
The log file codegrader-history/log.txt located in the root of the user’s home directory contains historical information about the evaluation of students’ assignments.
Test cases must follow a specific format and have a specific file name
in order to be used within CodeGrader. As an example, suppose the
exam requires the students to submit a file called q1.lisp that
includes two functions: a fact function that gives the factorial of
a number, and a avg function that gives the average of a list of
numbers. Moreover, suppose also that in certain questions, the use of
specific Lisp symbols is restricted, e.g., FIND and
COUNT. If students use any of these prohibited symbols, a penalty of
90% will be deducted from their total marks for that question. Then,
the test cases lisp file will be something like this:
(forbidden-symbols :penalty 0.90 :symbols '(find count))
(deftest test-fact ()
(check
(equal (fact 5) 120)
(equal (fact 6) 720)
(equal (fact 7) 5040)
(deftest test-avg ()
(check
(equal (avg '(5 8 10 2 12)) 7.4)
(equal (avg '(0 0 0 0 0 0)) 0)
(equal (avg '(1 2 0)) 1)
(defun unit-test ()
"Calls the test cases and 'forgets' the functions that were tested."
(test-fact)
(fmakunbound 'fact) ; Removes the function definition from the global environment,
; so the next time around the unit test is done on a freshly loaded version of this function.
(test-avg)
(fmakunbound 'avg))
(unit-test) Notice: you can include more complex forms of tests, but the general idea is that each argument of CHECK has to be a selfcontained form, i.e., any variables used in it should be defined within the form. For example, below is a test case for a function HT-DELETE that deletes an item from a hash table
(deftest test-ht-delete ()
(check
(equal (let ((*ht* (ht-create '((1 1) (2 2) (3 3) (4 4) (5 5) (6 6)))))
(ht-delete 4 *ht*)
(ht-get 4 *ht*)) ; accessing a deleted item
nil)
(equal (let ((*ht* (ht-create '((1 1) (2 2) (3 3) (4 4) (5 5) (6 6)))))
(ht-delete 4 *ht*)
(ht-delete 4 *ht*)) ; deleting an already deleted item
nil)
(equal (let ((*ht* (ht-create '((1 1) (2 2) (3 3) (4 4) (5 5) (6 6)))))
(ht-add "a" 44 *ht*)
(ht-delete "a" *ht*))
44)))
Any errors that the student’s solution could raise during runtime will be handled by CodeGrader and reported as appropriate.
In case you wish to mark one specific submission or test your test case file, you can use the following function:
evaluate-solution (student-solution test-cases-dir)
---------------------------------------------------
Description: Loads the student-solution file, loads the test cases, runs
the test cases, and returns the percentage of correct results over total results
Inputs: 1) student-solution [string]: The directory for the solution of the student.
2) test-cases-dir [string]: The directory for the test cases file. This will be used to test the solution of the students for the current assignment.
Outputs: [list] A list of the following:
1) [string] The grade of the student.
2) [string] A comment that describes if there was a runtime error while loading the student submission or not
3) [string] A description of what happened during runtime (from exceptions to conditions to whatever)
4) [list] The results of marking each of the test cases.
Side-effects: This function utilizes the global variable *results* while running. In the beginning by reseting it to nil, and at the end by updating it with the current
student's submission results.
---------------------------------------------------
Usage Example: Say there was a student that you want to mark their
submissions independantly from the other students. You can simply take
their lisp submission file, say "/home/John/mysol.lisp" , and the
test cases lisp file “/home/john/test-cases.lisp”~. You would use
CodeGrader as follows: (assuming you have already installed CodeGrader
as shown above)
CL-USER> (ql:quickload :codegrader) ; Loading the codegrader
CL-USER> (cg:evaluate-solution "/home/John/mysol.lisp" "/home/John/test-cases.lisp")
("100.0" OK "No runtime errors"
(("Pass" T TEST-DEPOSIT (EQUAL (DEPOSIT 20) 130))
("Pass" T TEST-DEPOSIT (EQUAL (DEPOSIT 10) 110))
("Pass" T TEST-DEPOSIT (NOT (DEPOSIT 10001)))
("Pass" T TEST-WITHDRAW (EQUAL (WITHDRAW 60) 10))
("Pass" T TEST-WITHDRAW (NOT (WITHDRAW 80)))
("Pass" T TEST-WITHDRAW (NOT (WITHDRAW 10001)))
("Pass" T TEST-WITHDRAW (EQUAL (WITHDRAW 20) 70))
("Pass" T TEST-WITHDRAW (EQUAL (WITHDRAW 10) 90))))
GRADER> (in-package :CL-USER)
CL-USER> See LICENSE for usage permissions. See AUTHORS for credits.