Build tools that bridge OpenAPI and gRPC

GSoC 2019
API Client Tools at Google
Lorenz Hofmann-Wellenhof

Graz, Austria, Central European Time github.com/LorenzHW linkedin.com/in/lorenzhofmannwellenhof

+43 664 88715585 lorenzhofmann.w@gmail.com lorenzhofmannw.com
Table of Contents
1 Abstract...................................................................................................................................................3
2 Statement of Motivation..........................................................................................................................3
3 Benefits to the Community.....................................................................................................................5
4 Project details..........................................................................................................................................6
4.1 Problem description.................................................................................................................6
4.2 Implementation Plan................................................................................................................7
5 Project Schedule......................................................................................................................................9
5.1 Deliverables.............................................................................................................................9
5.2 Timeline.................................................................................................................................10
5.3 After GSoC............................................................................................................................11
6 Related Work & Future work................................................................................................................12
7 About me...............................................................................................................................................13
1 Abstract
Creating APIs should be as easy as writing a “Hello World” program. Therefore, the Google APIs team is
working on different tools to accomplish that goal. One of those tools is gnostic which converts JSON or
YAML OpenAPI descriptions to equivalent Protocol Buffer specifications.
This project idea aims at creating a plugin for gnostic that generates a Protocol Buffer specification that
can then be used to generate a gRPC API.

2 Statement of Motivation
The first time I heard about GSoC was 20 th of March 2016. My brother texted me about it and said that it
would be a good thing if I participate. However, for students from the University of Passau this is the
worst possible time, because we got lectures and exams in exactly that period.
After my exchange semester in NYC in 2018 where I went to a lot of tech meet ups and got more familiar
with the idea of open source, I realized that I have one last opportunity for GSoC in 2019. Technically, it
is the perfect summer, because I only have my master thesis left to write, quit my job, and moved back
home to Austria to my parents (so they can cook for me while I code 😄).).

Illustration 1: Text conversation between my brother and me on March 20th

2016. I replied: “In the middle of the semester + exams”.
Why did I choose API Client Tools at Google?
I won’t lie, just like everyone else I would like the prestigious title of a GSoCer that is definitely
impressive on the resume. Additionally, it is a great opportunity to get to know people and network.
However those are only my secondary goals. I quickly realized what GSoC is actually about...
On the day the organization were announced I checked the technologies of API Client Tools at Google
and didn’t know a single technology keyword (except REST). There were a couple of other organizations I
was interested in, because I knew the technology stack. I asked myself what are the benefits of working
with an organization where I already know the technologies. This made me realize that GSoC is about
learning. It is an amazing opportunity to get insights from the best in the tech industry. Additionally, the
trend over the last years that more and more code is open sourced is surely a step into the right direction,
and being part of this community is definitely a good feeling.
Even if I don’t get the stipend, I appreciate the time I spend on constructing this proposal, because I
already learned a lot about new technologies. And the next time I need to design an API I definitely gonna
start by describing it with OpenAPI.
To sum it up, I want to quote Richard P. Feynman: “I was born not knowing and have had only a little
time to change that here and there”.

What are my expectations of GSoC ?

• Apply effective communication on a remote project.
• Advance my coding and software architecture skills.
• Create software that makes other peoples life easier.
• Fun.
3 Benefits to the Community
The best case and (expected scenario) is that I create a tool which takes in an OpenAPI description and
outputs a complete Protocol Buffer specification. This project could be merged inside of the gnostic
repository as a plugin.
The worst case is that I fail the project (unlikely) in the specified time. I would be willing to go the extra
mile after GSoC and complete it anyway. I think of myself as an easy going individual, open to
constructive criticism, but determined in the execution of a plan. So I am sure we will have a good time
nonetheless.
4 Project details

4.1 Problem description

The ‘API Client Tools at Google’ organization hosts tools, technical articles, and issues related to Google
API design and practices. Quoting the Google API Design Guide a networked API is described as
follows:
Application Programming Interfaces (API) that operate across a network of computers. They communicate
using network protocols including HTTP, and are frequently produced by different organizations than the
ones that consume them.
gRPC and REST are two different approaches for creating APIs. gRPC is based on the way Google builds
APIs internally and it depends on HTTP/2 and Protocol Buffers. REST on the other hand is based on
HTTP/1 and JSON. Both have their advantages and disadvantages. With HTTP/JSON transcoding a
gRPC API can also be provided in RESTful style via a reverse proxy. This is accomplished by using
HTTP annotations inside the Protocol Buffer specification. Therefore, it is possible to take advantage of
both approaches.
The OpenAPI specification is a structured way of describing REST APIs. OpenAPI descriptions are used
for better documentation, API validation, code generation, mocking, and testing.
This project aims at taking an OpenAPI description and generate a valid Protocol Buffer specification
with according HTTP annotations. This Protocol Buffer specification can further be used to generate
gRPC server and client stubs in any programming language that support Protocol Buffers. The problem
description can be seen in the illustration below.

Illustration 2: Problem description

4.2 Implementation Plan
This project consists out of three components that have to be build from scratch or updated:

OpenAPI description to Protocol Buffer Specification Generator:

The generator will be made available as plugin in gnostic. Therefore, a package with the name gnostic-
protoc-generator will be created (full path: gnostic/plugins/gnostic-protoc-generator). This generator will
be modeled after the already existing gnostic-go-generator plugin that takes in an OpenAPI description
and generates a Go client library and a scaffolding for a Go server. Modules that are used in both
generators should be factored out to avoid ambiguity. The package gnostic-protoc-generator will be build
from scratch. Additionally, there will be changes made inside the surface package to obtain the necessary
information for the generator. For (my) better understanding the workflow of gnostic can be seen in
illustration 3 (high resolution).

Illustration 3: Workflow gnostic

OpenAPI description to Protocol Buffer Specification Checker:

The checker will be made available as plugin in gnostic. Therefore, a package with the name gnostic-
protoc-checker will be created (full path: gnostic/plugins/gnostic-protoc-checker). The necessary
information is contained in ‘openapiv3_message’ (see illustration above). The checker then gives detailed
information about which parts of the OpenAPI description will not be represented inside of the generated
.proto file. The checker works in collaboration with a list that defines which parts of the OpenAPI
specification is translatable. Ideally, updating that list results in updating the checker without any
additional coding. This package will be build from scratch. Additionally, this may require changes inside
the OpenAPIv3 package.

Surface Model:
The surface package is an already created package in experimental state. It serves as model for the
generator plugins. It provides information in a structured way about the types (request parameters,
response paramaters) and methods (GET, POST, ...) used in the OpenAPI description. Additions have to
be made to meet requirements of the generator. Technically, information for the checker could also be
made available inside the surface model.
I already started implementing the generator. The progress can be seen here. An example of a
generated .proto file can be seen here for this OpenAPI description. Additionally, I created a repository
which takes an .proto file and generates + starts a reverse proxy and a gRPC server. Maybe this could
come in handy at the end of the project.
5 Project Schedule

5.1 Deliverables
• A list that defines what parts of the OpenAPI specification are transcodeable to a gRPC API.
◦ The list should be maintainable and extendable.
• A checker based on the defined list. The input of the checker is an OpenAPI description. The
output gives information about which parts of the OpenAPI description can be made available
with gRPC and which parts would not be represented.
◦ Implemented and distributed as Golang package. Available as plugin for gnostic, hence it is
executable as command line tool.
• A generator based on the defined list. The input of the generator is an OpenAPI description. The
output is a Protocol Buffer specification (.proto file).
◦ Implemented and distributed as Golang package. Available as plugin for gnostic, hence it is
executable as command line tool.
• Proper documentation and tests for the above mentioned components.
◦ Unit test cases for specified rules, documentation of source code, usage of how to use the
generator and checker.
5.2 Timeline
May 6 – May 27 (Before the start of GSoC)
• Spend more time on debugging through gnostic to get a deeper understanding of the code base.
• Read documentation, set up development environment, get familiar with the testing system used.
• Solve some LeetCode problems with Golang to get more familiar with the language.
May 27 – June 2 (Official start)
• Come up with the list that specifies which parts of the OpenAPI spec are transcodeable to gRPC.
June 3 – June 9
• Discuss with mentors which parts of the OpenAPI specification are must haves and which are optional.
• Start working on generator.
June 10 – June 16
• Work on the generator to satisfy most crucial requirements. Keep in mind that changes to other components have to
made (e.g.: surface model).
June 17 – June 23
• Keep working on generator. Commit progress continuously to verify that Travis build is not breaking.
June 24 – June 30 Phase 1 Evaluation: June 24 – June 28
• Discuss further steps with mentors.
• Code review existing progress.
July 1 – July 7
• Implement additional features for the generator.
July 8 – July 14
• Work on checker.
July 15 – July 21
• Code.
July 22 – July 28 Phase 2 Evaluation: July 22 – July 26
• Discuss final steps with mentors.
• Code review existing progress.
July 29 – August 4
• Proper source code documentation.
• Proper documentation on how to use the parser and checker.

The last three weeks are not included: A buffer of two weeks has been kept for unpredictable delay and
another week for testing. I plan to apply test-driven development practices. So there won’t be a dedicated
week only for testing. Test cases will be created during development.
During the entire timeline I will continuously be available on Slack or up for a scheduled video call to
discuss ideas, progress, further steps, and everything else that is related to this project.

Anticipated time investment per week in hours:

• Average case: O(40).
• Worst case: O(40 + 2x8) if needed I can spend additional time on the weekend for the project.
Other responsibilities/vacations:
I don’t have any university classes left. I might spend here and there a few hours for my master thesis but
my main focus would definitely be GSoC, because I can also start working extensively on my thesis after
GSoC.
Additionally, I booked a flight to visit my friend who is doing an exchange semester in San Sebastián from
May 20th to May 27th. So I would start one day late with GSoC. However, I would be willing to drop the
vacation in favor of GSoC if the vacation prevents me from being selected.

5.3 After GSoC

I am happy to keep an eye on open issues concerning the plugins that will be created by this project and
propose a solution on how to fix this issue or implement the fix by myself. However, my main focus will
shift towards my master thesis. In case I fail the project in the given time I would be willing to invest more
time until my mentors are satisfied with the result.
6 Related Work & Future work
A similar project is the Swagger Codegen project. It allows generation of API client libraries, server stubs
and documentation automatically given an OpenAPI description. However, it is not supported to generate
a Protocol Buffer specifications, and therefore gRPC is not supported.
Another project is the openapi-generator. It is forked from the Swagger Codegen project. This community
driven version provides similar functionalities and can be used as drop-in replacement. The major
differences between Swagger Codegen project and openapi-generator can be found here.
Googles Discovery format is a simpler version of the OpenAPI specification. It is used by Google
internally to describe APIs. Future work of this project could aim at generating a Protocol Buffer
specification from Google Discovery format.
7 About me

What are my hobbies, aside from coding?

Aside from coding I like to spend my time with physical activities. This is typically some kind of sport
where my friends and I can challenge each other (e.g.: squash, beach volleyball, tennis, soccer).
Additionally, we do endurance sport together like running, or going on a tour with the racing bikes. To
give our adrenaline a kick, we like to go downhill mountain biking and skiing.

What do you want to do with your life?

In terms of career it will (hopefully) go something like that: I will start out as engineer and try to soak in
best practices from my peers. On my way to becoming more senior, I will focus on qualifications like
leadership, management, and public speaking. At some point in my life I have to make a decision on how
I want to proceed. I either start freelancing, if the situation is right I would also consider starting a
business, or work in a more senior position at some company.
In terms of location I am open to new opportunities. I believe that starting in the USA as software
engineer would be more of a benefit than in the EU. However, I realize that it gets increasingly difficult as
long as Trump is president, and I think I can’t delay my master thesis that long. So I would only really
consider it if I would be able to get a decent offer from one of the big companies. For freelancing I would
choose Europe though.
In terms of my own personality trades I want to be the best version of myself that I can be. This includes
openness, mindfulness, being humble, determination, and just being an easy going individual. Additional
information about my mindset can be found at lorenzhofmannw.com.

What was my proudest moment as software engineer?

I deployed an Alexa Skill that lets people send and listen to new messages they receive on the Telegram
Messenger. One of the best moments was seeing actual people interact with my software that I created
with my bare hands.
Another awesome moment was when an Italian software engineer contacted me and said that he
appreciates the Skill and would like to have an Italian version of it. He opened a pull request with the
translated output for Alexa.
Lorenz Hofmann-Wellenhof
Graz, Austria github.com/LorenzHW linkedin.com/in/lorenzhofmannwellenhof
+43 664 88715585 lorenzhofmann.w@gmail.com lorenzhofmannw.com

EDUCATION
M. Sc. in Computer Science University of Passau, Germany 02/2017 - 02/2020
Emphasis: Programming and Software Systems

Education abroad in Computer Science Stevens Institute of Technology, USA 08/2018 - 12/2018
Grade: 4.0 (GPA)

B. Sc. in Mobile and Embedded Systems University of Passau, Germany 10/2014 - 09/2017
Grade: 1.8 (German System) ~ 3.2 (GPA)

Courses:
• Algorithms and Data Structures • Machine Learning • Programming in Java
• Agile Software Development • Enterprise Software Architecture • Mathematics in Technical Systems I-III
• Computer Networks • Computer Architecture • Computer Engineering

EXPERIENCE
Creator – My Telegrams 10/2017 – 01/2019
• Created an open source Amazon Echo Skill that enables users to send and to listen to messages via the Telegram
Messenger
• Accumulated over 1000 users in English and German
• Check out: bit.ly/my-telegrams-video to see the Skill in action
• Used technologies: Python, AWS
Creator – lorenzhofmannw.com 03/2018 – 01/2019
• Implemented a web server for the usage with Amazon Echo Skills
• Set up REST API with Django and MySQL DB, used Angular as front end framework, deployed the application with
AWS, added social authentication, progressive web app support, OAuth, implemented push notification via Firebase,
and used a Python wrapper to communicate with the Telegram API
• Used technologies: Python, TypeScript, HTML5, CSS, Sass, Django, MySQL, Angular, AWS
Software engineer (working student) – csneovias GmbH 03/2016 – 07/2018
• Designed and implemented front end applications (course registry, courses overview) for the University of Münster
and the University of Zurich
• Assisted development and design of back end interfaces to provide applications with mock data
• Used technologies: JavaScript, ABAP, SAPUI5, OData protocol
Creator – Bing Bong / Stretch Out 02/2018 – 07/2018
• Launched a brain game as Amazon Echo Skill (Bing Bong)
• Created an Amazon Echo fitness Skill that lets user create their own workouts (Stretch Out)
• Accumulated over 1400 users
• Used technologies: Python, AWS

ACTIVITIES & AWARDS

Coding competitions
• Google Hash Code 2018: Rank 1545 out of 3012
Awards
• Top 20% of 2017 graduation class
• 6 up votes and and an accepted answer on stackoverflow
• Top 3 project in Machine Learning class
Activities:
• Student council (University of Passau)
• TCS New York City Marathon Volunteer
• German American Conference participant

SKILLS
Core programming languages: Python, JavaScript, TypeScript
Languages: German (native), English (fluent)