Code Runner: Solution for Recognition and Execution of Handwritten Code

Wenxiao Du
wxdu@stanford.edu

Motivation
In technical interviews, interviewees are sometimes asked to write code on white board or
paper. Interviewer can see the general logic of the code but cannot actually run it and
prove its correctness. Therefore, the goal of my project is to provide a solution that would
allow people take a picture of a piece of code by android phone and get its output. For
now the target code is python. More programming languages will be included if time
allows.

Implementation details
The entire project would follow the client-server model. Therefore, the server side can do
more complex computation and involve more programming language. Here is the detail
of workflow:
1. Client (Android Phone) takes a picture and sends it to server
2. Preprocessing the image
In this step, size normalization is first conducted on image. Then I would denoise the
image use median filter and remove the small regions, which should be dust or noise.
After that, the image is binarized using Otsu Method.
3. Recognize the code
I would use Tesseract-OCR engine in this Part. Tesseract is originally designed for
printed text recognition, but after proper training, it can also be used for handwritten
text recognition.[1][2][3][6] Some datasets have already been collected for training
purpose.[4]
4. Post-processing the result
Since there should be some common key words for a certain programming language
e.g. def in Python, I would build a dictionary contains such key words and run the
spell corrector on each recognized term. The general idea is that I would search for all

candidate words that are within the certain edit distance from the given word and see
whether any candidates match words in dictionary. [5]
5. The result is then sent back to the client. Client can now manually check whether
there is still any mis-recognition. After that the client can compile the final-version
code and produce the output. For this part, I havent decided whether let server or
client to do the compilation. The decision would be made when we come to this step.

References:
[1] Training Tesseract 3 Code.google.com. Google, June 2013. Web. 06 June 2013
[2] Recognition of Handwritten Textual Annotations using Tesseract Open Source OCR
Engine for information Just In Time (iJIT), Sandip Rakshit, Subhadip Basu, Hisashi
Ikeda 03/2010;
[3] Recognition of Handwritten Roman Script Using Tesseract Open source OCR Engine
Sandip Rakshit, Subhadip Basu 03/2010;
[4] IAM Handwriting Database
Sam Roweis Dataset
UJI Pen Characters Data Set
[5] How to Write a Spelling Corrector. Peter Norvig.
[6] "An Overview of the Tesseract OCR Engine." Smith, Ray. Research at Google.
Google, 2007. Web. 6 June 2013.