PA5 — The Interpreter

Programming assignments 2 through 4 involved the constructed of the front-end (lexer, parser) and gatekeeping (semantic analyzer) stages of an interpreter. In this assignment you will write the code that performs the execution and interpretation of valid programs.

You may complete this assignment in OCaml, Haskell, JavaScript, Python or Ruby.

You may work in a team of two people for this assignment. You do not need to keep the same teammate. The course staff are not responsible for finding you a willing teammate.

Goal

For this assignment you will write an interpreter. Among other things, this involves implementing the operational semantics specification of Cool. You will track enough information to generate legitimate run-time errors (e.g., dispatch on void). You do not have to worry about "malformed input" because the semantic analyzer (from PA4) has already ruled out bad programs.

You will also write additional code to unserialize the class and implementation maps produced by the semantic analyzer and the parse tree produced by the parser.

The Specification

You must create three artifacts:

1. A program that takes a single command-line argument (e.g., file.cl-type). That argument will be an ASCII text Cool class map, implementation map, and AST file (as described in PA4). Your program must execute (i.e., interpret) the Cool program described by that input. If your program is called interp, invoking interp file.cl-type should yield the same output as cool file.cl. Your program will consist of a number of OCaml files, a number of Python files, or a number of Ruby files.
   • You will only be given .cl-type files from programs that pass the semantic analysis phase of the reference interpreter. You are not responsible for correctly handling ill-typed (e.g., 1+"hello") programs.
2. A plain ASCII text file called readme.txt describing your design decisions and choice of test cases. See the grading rubric. A few paragraphs should suffice.
3. Testcases test1.cl, test2.cl, test3.cl and test4.cl. The testcases should exercise interpreter and run-time error corner cases.

Error Reporting

To report an error, write the string

to standard output and terminate the program. You may write whatever you want in the message, but it should be fairly indicative. Example erroneous input:

Corresponding example error report output:

Commentary

You will have to handle all of the internal functions (e.g., IO.out_string) that you first encountered in PA4.

You can do basic testing as follows:

Note that this time, whitespace and newlines do matter for normal output. This is because you are specifically being asked to implement IO and substring functions.

You should implement all of the operational semantics rules in the Reference Manual. You will also have to implement all of the built-in functions on the five Basic Classes.

PA5t — Creating PA5 Tests

PA5t is a preliminary testing exercise that introduces a form of test-driven development or mutation testing into our software development process and requires you to construct a high-quality test suite.

The goal of PA5t is to leave you with a high-quality test suite of Cool programs that you can use to evaluate your own PA5 Interpreter. Writing an interpreter requires you to consider many corner cases (perhaps even more than in PA4!) when reading the formal operational semantics rules in the Cool Reference Manual. While you you can check for correct "positive" behavior by comparing your interpreter's output to the reference interpreters's output on the usual "good" Cool programs, it is comparatively harder to check for "corner case" behavior.

If you fail to construct a rich test suite of semantically-valid tricky programs you will face a frustrating series of "you fail held-out negative test x" reports for PA5 proper, which can turn into unproductive guessing games. Because students often report that this is frustrating (even though it is, shall we say, infinitely more realistic than making all of the post-deployment tests visible in advance), the PA5t preliminary testing exercise provides a structured means to help you get started with the constuction of a rich test suite.

The course staff have produced 22 variants of the reference compiler, each with a secret intentionally-introduced defect related to Interpretation. A high-quality test suite is one that reveals each introduced defect by showing a difference between the behavior of the true reference compiler and the corresponding buggy verison. You desire a high-quality test suite to help you gain confidence in your own PA5 submission.

For PA5t, you must produce syntactically valid Cool programs (test cases). There are 22 separate held-out seeded interpreter bugs waiting on the grading server. For each bug, if one of your tests causes the reference and the buggy version to produce difference output (i.e., different stdout/stderr), you win: that test has revealed that bug. For full credit your tests must reveal at least 17 of the 22 unknown defects.

The secret defects that we have injected into the reference compiler correspond to common defects made by students in PA5. Thus, if you make a rich test suite for PA5t that reveals many defects, you can use it on your own PA5 submission to reveal and fix your own bugs!

Video Guides

A number of Video Guides are provided to help you get started on this assignment on your own. The Video Guides are walkthroughs in which the instructor manually completes and narrates, in real time, the first part of this assignment — including a submission to the grading server. They include coding, testing and debugging elements.

If you are still stuck, you can post on the forum or approach the professor. The use of online instructional content outside of class weakly approximates a flipped classroom model. Click on a video guide to begin, at which point you can watch it fullscreen or via Youtube if desired.

What to Turn In For PA5t

For PA5t you should turn in (electronically):

• A set of .cl (and possibly corresponding .cl-input) files: Cool interpreter testcases.
  • Each testcase you submit must be semantically valid (i.e., must pass cool --type).
  • Each testcase you submit may have a corresponding input file. For example, if you submit wes.cl you may also submit wes.cl-input, and that test will be run with a command like cool wes.cl-type < wes.cl-input >& output.txt.
  • Each testcase you submit must be at most 2048 characters (i.e., wc -c yourtest.cl says 2048 or less). You want each of your testcases to be meaningful so that it helps you narrow down a bug later (cf. Delta Debugging).
    • Q: Ha ha! I will get around this edict by removing comments and compressing whitespace and variable names.
    • A: Remember, the point of PA5t is not to make me happy, but to provide you with a high quality test suite. You're not helping yourself later by making low-quality tests now. When your interpreter fails one of your own tests later you'll end up shrinking the test down as part of debugging, so you're not actually saving yourself any time.
  • Each testcase you submit may run to completion or it may trigger some run-time error. None of the seeded defects you are trying to uncover are intentionally related to infinite loops. Do not spam our server or submit long-running tests.
    • Clarification: Similarly, none of the errors are related to large memory allocations or exhausting the heap at run-time (e.g., by making arbitrarily large strings, etc.).
  • No testcase should be named bug... or ref... because the testing server uses those prefices internally. If you submit a test case with such a name it will be ignored. (Limited course resources can either be spent making better lectures, grading your assignments, etc., or ironing out wrinkles such as these from grading scripts. We have chosen to focus on pedagogy.)

Hint: All of the usual tactics apply (from randomly-generating programs to permuting one symbol in each operational semantics rule to reading the prose descriptions in the CRM and looking for words like "must" to digging through the reference compiler binary).

PA5c — Checkpoint

PA5c is a checkpoint for PA5. The Interpreter is a large and complicated assignment; we do not want you to fall behind.

For the PA5c checkpoint you will only be tested on something akin to hello-world.cl. If you can interpret that, you pass the checkpoint. (You can "cheat" the checkpoint by hard-coding output for that single test case, but you're ultimately only hurting yourself!) The goal of the checkpoint is not to do the minimal amount of work possible for this program, but instead to do the greatest amount possible now so that you have plenty of time for the rest of the features later.

You must turn in these files:

1. source_files — your implementation, including
   • main.rb or
   • main.py or
   • main.js or
   • main.hs or
   • main.ml

What to Turn In For PA5

You must turn in these files:

1. readme.txt — your README file
2. test1.cl — a novel testcase
3. test2.cl — a novel testcase
4. test3.cl — a novel testcase
5. test4.cl — a novel testcase
6. source_files — your implementation, including
   • main.rb or
   • main.py or
   • main.js or
   • main.hs or
   • main.ml

Language-Specific Hints

Python hint #1:

I was performing the computation 4 / -59. In a very strange turn of events, this computation returns 0 in a simple C program (and, apparently, in OCaml), but -1 in Python. Bwah?

Python's division truncation works differently from C's. This is more than a little annoying, and certainly a "gotcha".

previous student Charles Eckman

Charles reports that he was able to solve the problem by adding from __future__ import division to the top of his Python source, which makes the division operator always return a floating-point result. That result can then be cast to an integer via the int() constructor.

Python hint #2:

If you are programming PA5 in Python and are unable to pass test cases that have lots of nested expressions, it may be because Python's default max recursion depth is set to 1000 which stops your program from reading/evaluating them fully. To surpass this, you can write sys.setrecursionlimit(10000) in your main() function.

anonymous Bowdoin student(s)

Grading Rubric

PA5 Grading (out of 100 points):

• 66 points — for autograder tests
• 5 points — for a correct PA5t submission
• 5 points — for a correct PA5c submission
• 8 points — for a clear description in your README
  • 8 — thorough discussion of design decisions (e.g., the handling of let and new and dispatch) and choice of test cases; a few paragraphs of coherent English sentences should be fine
  • 4 — vague or hard to understand; omits important details
  • 0 — little to no effort, or submitted an RTF/DOC/PDF file instead of plain TXT
• 8 point — for valid and novel test1.cl, test2.cl, test3.cl and test4.cl files
  • 8 — wide range of test cases added, stressing most Cool features and some error conditions, novel files
  • 4 — added some tests, but the scope not sufficiently broad
  • 0 — little to no effort, or parts of course files resubmitted as tests
• 8 point — for code cleanliness
  • 8 — code is mostly clean and well-commented
  • 4 — code is sloppy and/or poorly commented in places
  • 0 — little to no effort to organize and document code