The Magic Behind Heroku’s “git push” Deployment

In my spare time I help out with a little app called Greg’s Toolkit that was built before I knew about Heroku. The app runs on EC2 and deploying new versions of the app is pretty tedious. Here is the deployment instructions copied directly from the project’s wiki:

./gradlew war
scp  ./gft_server/build/libs/gft_server.war api.gregstoolkit.com:
ssh api.gregstoolkit.com
cd /opt/apache-tomcat-7.0.21
sudo -u www-data bin/shutdown.sh
cd webapps/ROOT
sudo rm -r *
sudo -u www-data jar -xvf ~/gft_server.war
sudo -u www-data sed -i 's/dev/prod/' WEB-INF/web.xml
cd ../..
sudo -u www-data bin/startup.sh

That certainly isn’t as cumbersome as some deployment methods but it is time consuming, error-prone, and causes downtime.

If you’ve used Heroku one of the most magical things you first come across is Instant Deployment. With Heroku you just upload your code to Heroku and that kicks off a process that results in the new version of the app being deployed. Heroku uses the Git tool as a way for developers to upload an app’s source to Heroku. So to deploy a new version of an app on Heroku (from the command line), you just run:

git push heroku master

It’s magic! But what actually happens on Heroku when you “push” a Git repository to Heroku? Anything you want! A “Buildpack” translates whatever you push to Heroku into something that can be run on Heroku. Heroku provides a number of out-of-the-box Buildpacks like the Maven Buildpack (for Java projects), the NPM Buildpack (for Node.js projects), and many others.

Now here is the really cool thing… The Buildpack system on Heroku is totally open and pluggable. There is even a documented API for the Buildpack system. So if you want to modify the process the takes your code from the “git push” to a runnable application, you can either: customize an existing Buildpack, create an entirely new Buildpack from scratch, or use a third party Buildpack.

Lets try a really simple example using a third party Buildpack. The Null Buildpack literally does nothing on a “git push”. Whatever you push gets deployed. Lets try it out. I’m going to use Python since it includes a simple web server in the base system and doesn’t need to be compiled. Follow along:

1. Install Git and the Heroku Toolbelt
2. Login to Heroku from the command line:

   heroku login

3. In a new directory create a new file named “web.py” containing:

   import os from wsgiref.simple_server import make_server   port = os.environ.get("PORT", "5000")   def hello_world_app(environ, start_response): status = '200 OK' headers = [('Content-type', 'text/plain')] start_response(status, headers) return ["hello, world"]   httpd = make_server('', int(port), hello_world_app) print "Serving HTTP on port " + port + "..."   httpd.serve_forever()

4. If you want to test the app locally run:

   python web.py

   Then in your browser go to http://localhost:5000 and you should see “hello, world”.

5. To run this simple app on Heroku you need to let Heroku know what command to run on the system. To do this create a file named “Procfile” (in the same directory as “web.py”) containing:

   web: python web.py

   This tells Heroku that for the process named “web” run “python web.py” on the system. The managed systems that run apps on Heroku are called “Dynos“. You can allocate as many Dynos as needed to each process in your app.

6. Since Heroku uses Git for file upload and download you need to create a new Git repository, add your files to it, and commit them:

   git init git add Procfile web.py git commit -m init

7. Now provision (i.e. create) a new app on Heroku but instead of using the default set of Buildpacks, explicitly tell Heroku to use the Null Buildpack:

   heroku create --buildpack https://github.com/ryandotsmith/null-buildpack.git

   This provisions the app on Heroku and sets a new Git remote named “heroku” in the configuration for your Git repository. The “heroku” remote is just a name that points to the Git URL for your app on Heroku.

8. Now push your app to the “heroku” Git remote repository:

   git push heroku master

Once the files are uploaded, Heroku’s Slug Compiler uses the specified Buildpack to perform the build, converting source code into something runnable. Everything that is needed to run the app (with the exception of the base system) goes into a “slug”. Then the slug is copied to a Dyno and the process named “web” is run. Here is the output from my “git push”:

Counting objects: 4, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (4/4), 522 bytes, done.
Total 4 (delta 0), reused 0 (delta 0)
 
-----> Heroku receiving push
-----> Fetching custom buildpack... done
-----> Null app detected
-----> Nothing to do.
-----> Discovering process types
       Procfile declares types -> web
-----> Compiled slug size is 4K
-----> Launching... done, v4
       http://blazing-waterfall-8915.herokuapp.com deployed to Heroku
 
To git@heroku.com:blazing-waterfall-8915.git
 * [new branch]      master -> master

Check out the app in your browser by running:

heroku open

That’s it! Even though it feels magical, it is really a pretty simple (and totally customizable) process.

Lets recap what happens when you do a “git push heroku master”:

1. The Git repository is uploaded to Heroku.
2. The Slug Compiler uses either a custom specified Buildpack or a default Buildpack to transform the source that was pushed into something runnable. This process can be as simple as doing nothing or as complex is resolving dependencies, compiling source, packaging assets, running tests, etc. All files in the project directory when the Buildpack finishes are put into the slug.
3. The slug is copied to new Dyno(s). (Usually the default is 1 Dyno for the “web” process.)
4. The defined processes are started on the Dynos.

Pretty simple! Here is another cool feature of Heroku’s Buildpack system… If the Buildpack returns an error then the process stops. So with compiled languages you get some automatic “testing” of the code before it is deployed.

To recap: A Buildpack translates whatever you push to Heroku into something that can be run on Heroku. There are tons and tons of Buildpacks out there (mostly on GitHub). Next time you need to deploy something out of the ordinary, see if there is a Buildpack for it already. If not, then why not create your own? It’s actually really simple. My first custom Buildpack simply converts Markdown files to HTML so that I can instantly deploy Markdown on the web. It is under 40 lines of code and was pretty simple to write. If I can create a Buildpack then you probably can too! :)