Web frameworks benchmarks

In this article, an overview of the main web frameworks benchmarks is presented. To begin, it is analysed a presentation by Van den Enden, Ward Vijfeijken and Guy Veraghtert which is probably the best web framework benchmark done. Only Java-based frameworks are compared but  the methodology and results are still valuable. Next, there is a set of small web frameworks benchmarks posted on blogs, despite the fact that they could be more precise, the presence of close results in them leads to conclusions which could help in taking decisions on which framework to choose.

World Wide Wait

Stijn Van den Enden, Ward Vijfeijken and Guy Veraghtert made a presentation at Devoxx 2011 where they compared different Java frameworks.  They measured for every framework the requests per second that it was able to accept, the load in different CPUs, the mean page load time, the cost of scale and other features.

Compared web frameworks were:

• GWT
• Spring
• JSF
• Wicket

The study is scientific and precise. A web application based on common patterns was built and a set of tests were sent to the application. It consists of more than 700 hours of test run and more than 16 GB analysed. The results were validated. The database was simulated to avoid bottlenecks, it had a 40 milliseconds response time.

In figure 1, it is shown the mean page load time versus concurrent users, ThreadCount. In the case of GWT, DOM updates executed after onLoad event are not counted. Therefore, it should be a little higher.

Figure 2 is a brief diagram which shows framework cost of scale for 10000 users and a limit of 5 seconds to answer. GWT is the cheapest to scale despite being the second faster in average mean page load time due to the fact that GWT sends JSON to the client and it renders it. Because of this, server has a lower load and it is cheaper to scale, while the client has a higher load because of the rendering. Furthermore, research authors said that GWT was the most difficult to develop with.

In conclusion, Spring is the fastest framework but it costs around 50% more to scale with it than with GWT. GWT is the cheapest framework to scale but it gives a slower response time. Wicket and JSF are slow and expensive.

The Great Web Framework Shootout

Seth Davis made a more imprecise benchmark. According to his words “it would be foolish to consider these results as scientific in any way”. He tried to test frameworks performance just out-of-box. Data collected are not as precise as World Wide Wait data are but he compares different web frameworks based on PHP, Python and Ruby. The frameworks which have been compared are:

• Bottle
• Flask
• Sinatra
• Web.go
• Pyramid
• Django
• Rails
• Kohana
• TG
• Yii
• CakePHP
• Symfony

In case you are interested, the methodology and setup are well defined in his blog. Furthermore, there are more graphics than the one shown here.

Figure 3 shows requests per second for a template with a DB query. Bottle, Flash, Sinatra and Web.go are microframeworks which do not have so many capabilities as full stack frameworks have. The full stack frameworks which have higher requests per second are Pyramid, CodeIgniter, Django and Rails.

Benchmarking Web Applications and Frameworks

Paul M. Jones made a benchmark for PHP frameworks. The PHP frameworks which have been compared are:

• Aura
• CakePHP
• CodeIgniter
• Lithium
• Solar
• Symfony
• Zend

First, he tested the maximum responsiveness of the benchmark environment without any framework. The tests were a base line HTML page with “hello world” and a PHP page with “Echo hello world”. Afterwards, he tested each framework rendering the HTML page. Results in figure 4 show the requests per second in relative percentage to the PHP page rendering time.

 CodeIgniter presents the best performance.

Rails, Wicket, Grails, Play, Tapestry, Lift, JSP, Context

JT Dev published a benchmark for Java web frameworks and Ruby on Rails. The frameworks which have been compared are:

• Grails
• JRails
• JSP
• Lift
• Play
• Rails
• Wicket
• Context
• Tapestry

The test is a single html page where a list of products objects with associated category are rendered. All data is in memory to avoid the use of a database.

Figure 5 shows response time versus number of users. The best performance is presented by Play rendering with Japid and Netty, followed by Play with Japid, Play with Scala and Netty, Tapestry, JSP, Grails and more. Rails has a poor performance compare to rest. JRails presents the worst performance.

Conclusions

It is important to notice that each benchmark measures different features. In the first one, it has been measured mean page load time. In the second and the third one, request per second. While, in the last one response time.

The best benchmark approach is the World Wide Wait. It shows that Spring is the fastest framework but it costs a 50% more to scale with it than with GWT. GWT is the cheapest framework to scale but it gives a slower response time. Wicket and JSF are slower and more expensive.

The second and the third benchmarks show CodeIgniter as the leader in PHP. Pyramid and Rails are the leaders in Python and Ruby languages. Django is following Pyramid in Python language and close to Ruby on Rails.

Wicket is presented in the first benchmark and the last one. It could be inferred that all the frameworks which last longer than Wicket on response time in the last benchmark have worse performance than Spring or GWT.