Goals

• More expereince writing an open ended C++ program!
• Direct expereince with how processes could be practically used

Collaboration

For assignments in CIS 3990, you will complete each of them on your own. However, you may discuss high-level ideas with other students, but any viewing, sharing, copying, or dictating of code is forbidden. If you are worried about whether something violates academic integrity, please post on Ed or contact the instructor(s).

Setup

You can downlowd the starter files into your environment by running the following command. You should run it from inside your git repository since that is how you will submit your code.

curl -o autograder_starter.zip https://www.seas.upenn.edu/~tqmcgaha/cis3990/current/projects/code/autograder_starter.zip

You can also download the files manually here if you would like: autograder_starter.zip

From here, you need to extract the files by running

unzip autograder_starter.zip

You should now have a folder called autograder that has the starter files. We expect this to be a folder inside the base of your github repository, similar to cowsay and check-setup. If you accidentally put the folder somewhere else, you can move it with the mv command in the terminal.

You also need to install somethings, notably python3, pycodestyle and cowsay (if you didn’t keep it from HW0)

To do this you should run:

sudo apt install python3

then run the next two commands to install pip and pycodestyle:

python3 -m ensurepip --upgrade
python3 -m pip install pycodestyle

sudo apt update
sudo apt upgrade
sudo apt install pip
python3 -m ensurepip --upgrade
python3 -m pip install pycodestyle

and lastly for cowsay:

sudo apt install cowsay

Overview

In this assignment you will be building an autograder in C++! Notably, you will be implementing an autograder for cowsay, but the “student submissions” are in python and the autograder is written in C++!

You will do this mostly from scratch, but we will give you a few files that will be useful to you:

• Json.hpp and Json.cpp contain the implementation for a simple object that can be used to represent a json object. JSON is a very common way to store data in a file. Each “object” can be thought of as a map where the keys are strings and the values are either: integers, doubles, strings, booleans, another JSON object or a list containing any of the previous types as elements. You are not required to learn JSON in-depth, this (and a little more later in the specification) should be all you need for this assignemnt. You can look back at the code from HW05 as an example of how to use the json object and print it.
• submissions is a directory containing a lot of “student submissions” that your code is expected to autograde. For example submissions/travis is a submission and submissions/herborb is another.
• cowsay_cases.py is the file from the course HW0 autograder containing the test cases for cowsay. You should use these cases in your autograder, but you are not expected to have your autograder read or parse this file directly. You may take this data and include it in your source code in whichever way you would like.
• .clang-tidy which contains the clang-tidy format rules

These are all the files we give you. The rest of the file is up to you to implement :)

You will generate an executable called autograder. It takes a single command line argument, which is the path to a student submission. So for example, you would run ./autograder submissions/travis. Each submission will have a cowsay.py file that your autograder can invoke by running something like python3 submissions/travis/cowsay.py.

The results of running the autograder are then stored in the file results.json. An example for what this file looks like can be found in a section below.

Requirements

Makefile

You wil have to make your own Makefile for this assignment. You can look to the Makefile from previous assignments and use those as inspiration if you want. You should generate an executable called autograder, and have a rule to clean your code. You also would find it useful to create a rules for formating, and running tidy-check (even if you don’t make this rule, the autograder will check these and yell a for mistakes)

Your makefile should compile with the flags -g3 -gdwarf-4 --std=c++2b -Wall -Wextra among other flags to make sure you get debug information, use the modern C++ version, and that a bunch of warnings are turned on.

File Layout

You should have a file called called autograder.cpp that contains your main function. You should have at least one .cpp (other than autograder.cpp and Json.cpp) that is part of your autograder implementation. The rest is up to you! Just try and follow good practices since we are grading on code quality in this assignment.

Command Line Args

Your program takes the directory to the student submission. If no submission is specified, or it does not lead to a valid path with a cowsay.py file, then your code should print an error message of some sort to standard error, and have a non-zero exit status.

Autograder Test Cases

As mentioned in the overview section above, we provide the test cases that your autograder should test on in a file called cowsay_cases.py. You are not expected to have your autograder read or parse this file directly. You may take this data and include it in your source code in whichever way you would like. Do not add any other cases.

Style Check

Your autograder should also do an automated style check on the user code. this can be done by running pycodestyle on the submission’s cowsay.py file. Your test case for this should be called "Style Check" and you should incldue the output of pycodestyle in the “output” field of the test case.

Extraneous Output

Your autograder should not print anything unecessary or create any output files other than results.json. By output files, we mean things that are generated when you run your code. This is not the same as files that are used to build your autograder.

Output Format

Your autograder should store the results of running in a file called results.json. Below we have included an example of this file when run on the submissions/crash example.

Note that you should match this pretty close. The only fields that can have some difference is how you format the value of output for each test. The test output should always start with the test description with a ...\n but what is stored after that can vary. If a test does fail for a submission, you should put a descriptive error message there and include the word “error” somewhere. (We will be checking this, but have some leniency here. Don’t just put “ERROR: case failed”.)

{
  "output" : "",
  "tests" : [
    {
      "output" : "submissions/crash/cowsay.py:6:1: E302 expected 2 blank lines, found 1\nsubmissions/crash/cowsay.py:9:1: E302 expected 2 blank lines, found 1\nsubmissions/crash/cowsay.py:12:1: E302 expected 2 blank lines, found 1\nsubmissions/crash/cowsay.py:15:1: E302 expected 2 blank lines, found 1\nsubmissions/crash/cowsay.py:22:1: E302 expected 2 blank lines, found 1\nsubmissions/crash/cowsay.py:32:1: E302 expected 2 blank lines, found 1\nsubmissions/crash/cowsay.py:36:1: E302 expected 2 blank lines, found 1\nsubmissions/crash/cowsay.py:62:1: E302 expected 2 blank lines, found 1\nsubmissions/crash/cowsay.py:87:1: E305 expected 2 blank lines after class or function definition, found 1\n",
      "max_score" : "15",
      "score" : "0",
      "name" : "Style Check"
    },
    {
      "output" : "multiple lines long moo...\nERROR: student code exited with non-zero exit status",
      "max_score" : "5",
      "score" : "0",
      "name" : "Long 3"
    },
    {
      "output" : "Very long poem...\nERROR: student code exited with non-zero exit status",
      "max_score" : "5",
      "score" : "0",
      "name" : "Complex: Jabberwock"
    },
    {
      "output" : "multi line long...\n",
      "max_score" : "5",
      "score" : "5",
      "name" : "Long 2"
    },
    {
      "output" : "looooooooooong moo...\n",
      "max_score" : "5",
      "score" : "5",
      "name" : "Long"
    },
    {
      "output" : "Long(er) input...\nERROR: student code exited with non-zero exit status",
      "max_score" : "5",
      "score" : "0",
      "name" : "Simple 3"
    },
    {
      "output" : "two long moooos...\nERROR: student code exited with non-zero exit status",
      "max_score" : "5",
      "score" : "0",
      "name" : "Long 4"
    },
    {
      "output" : "multiple word single line...\n",
      "max_score" : "5",
      "score" : "5",
      "name" : "Simple 2"
    },
    {
      "output" : "A simple moo...\n",
      "max_score" : "5",
      "score" : "5",
      "name" : "Simple 1"
    }
  ]
}

Tips

• Break your code up into helper functions. A helper function that makes a lot of sense would be something like Python’s Suprocess.run, which takes in something specifying a command to run via fork() and exec() and returns CompletedProcess struct containing any relevant output from having run that command (this can include the commands output among other things).
• You should also make use of pipe() and dup2() for this assignment.
• Handling file paths and whether a file path refers to an existing file can be done with the filesystem header. In particuilar filesystem::path and functions relating to the path may be useful to you. You can also use strings and other mechanisms if you want, but these are more portable and readable.
• Some may find the tuple struct from <tuple> useful. It allows you to store multiple different types of data in one struct. For example tuple<string, string, int> stores a tuple containing two strings and an integer.

Grading & Testing

Compilation

You have to write your makefile yourselves. It will automatically be evlauated to make sure it rebuilds your code correctly whenever a source file is updated. Your makefile should also use the flags mentioned above in the requirements section. We will also check that the clean rule exists and works properly.

Your submission will be partially evaluated on the number of compiler warnings. You should eliminate ALL compiler warnings in your code.

Valgrind

We will also test your submission on whether there are any memory errors or memory leaks. We will be using valgrind to do this. To do this, you should try running: valgrind --leak-check=full --trace-children=yes ./autograder submissions/joel/

If everything is correct, you should see the following towards the bottom of the output:

 ==1620== All heap blocks were freed -- no leaks are possible
 ==1620==
 ==1620== For counts of detected and suppressed errors, rerun with: -v
 ==1620== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

If you do not see something similar to the above in your output, valgrind will have printed out details about where the errors and memory leaks occurred.

Test Cases

There is no catch2 test suite like there in previous homeworks. Instead you should compare the JSON file your code generates with the one we provided above. We will have the gradescope autograder automaitcally try and detect whether your autograder behaves correctly, but let us know if something looks off.

Code Quality

For this assignment, we will be checking the quality of your code in two ways:

• Through automated checks with clang-format & clang-tidy
• Through manually reviewing your code for any style issues.

Reminder to check our code quality document to see what we are looking for in your code.

Submission

Please submit your completed autograder.cpp, Makefile, and other files to Gradescope via your github repo. We will be checking for compilation warnings, valgrind errors, making sure the autograder runs correctly, that you did not violate any rules established in the specification, and code quality.