Optimizing Play 2 for Database-Driven Apps

Last week Matt Raible and I presented the Play vs. Grails Smackdown at ÜberConf. The goal of the session was to compare Play 2 + Java with Grails by creating the same app with each framework. We used a number of criteria for the comparison including some benchmarks. You can read more about the results in Matt’s session recap blog. After presenting the results we learned that it’s pretty important to optimize Play 2′s Akka threading system in these types of applications. Play 2 is optimized out-of-the-box for HTTP requests which don’t contain blocking calls (i.e. asynchronous). Most database-driven apps in Java use synchronous calls via JDBC so Play 2 needs a bit of extra configuration to tune Akka for these types of requests.

Let me illustrate this with a simple example. In order to simulate a real-world scenario where the database blocks for a realistic amount of time I will run this benchmark on Heroku instead of locally. I’ll be using the “java-ebean” branch from my “play2bars” example app and the shared Heroku PostgreSQL database.

To run Apache Bench and send 10,000 requests with 100 concurrent connections to the JSON service in the app, I ran the following command from an EC2 server located in the same AWS region as my app on Heroku:

ab -n 10000 -c 100 http://falling-dusk-7291.herokuapp.com/bars

The results of the first, un-optimized run were:

Concurrency Level:      100
Time taken for tests:   10.272 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Total transferred:      1560000 bytes
HTML transferred:       510000 bytes
Requests per second:    973.53 [#/sec] (mean)
Time per request:       102.719 [ms] (mean)

At 973 requests per second this isn’t too bad but in some test runs I was getting errors like:

[error] play - Cannot invoke the action, eventually got an error: Thrown(akka.pattern.AskTimeoutException: Timed out)

This meant that Akka was blocked for too long so I tuned the Akka settings in Play to better handle synchronous code and re-ran the tests:

Concurrency Level:      100
Time taken for tests:   7.274 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Total transferred:      1560000 bytes
HTML transferred:       510000 bytes
Requests per second:    1374.70 [#/sec] (mean)
Time per request:       72.743 [ms] (mean)

This time I got 1,374 requests per second! The simple optimization is to give Akka more threads to work with and longer timeouts so when threads gets blocked by synchronous database calls there are still more to use for request handling. Here is the Play configuration I used:

play {
 
    akka {
 
        actor {
 
            deployment {
 
                /actions {
                    router = round-robin
                    nr-of-instances = 100
                }
 
                /promises {
                    router = round-robin
                    nr-of-instances = 100
                }
 
            }
 
            retrieveBodyParserTimeout = 5 seconds
 
            actions-dispatcher = {
                fork-join-executor {
                    parallelism-factor = 100
                    parallelism-max = 100
                }
            }
 
            promises-dispatcher = {
                fork-join-executor {
                    parallelism-factor = 100
                    parallelism-max = 100
                }
            }
 
        }
 
    }
 
}

The number of threads Akka will use is determined by the number of CPUs multiplied by the “parallelism-factor” setting, up to the “parallelism-max” setting. The default “parallelism-factor” is “1″ meaning the number of threads is equal to the number of CPUs with a default “parallelism-max” of “24″. The Play documentation provides more details about these settings.

The full config files are at available in the play2bars project.

As with all benchmarks, take this with a grain of salt, run your own benchmarks, and do your own tuning for your environment. Let me know if you have any questions.