Faster Testing with Rails 1.0
(Mon Oct 24, 2005) [/Rails] #
Rails makes doing the right things easy and the wrong things a bit more difficult. One of those right things is testing, and the Rails 1.0 release candidate has a new set of defaults and a couple new goodies to help your tests go faster. And when tests go faster, they tend to get run more often.
Herewith, an overview of the new testing stuff, a how-to for upgrading your existing tests (though you don't have to), an obligatory math formula or two, and a performance comparison that's worth about as much as you paid to read this.
What's New?
A quick peek inside the test/test_helper.rb file that's generated for all new Rails apps reveals shiny new defaults:
ENV["RAILS_ENV"] = "test" require File.expand_path(File.dirname(__FILE__) + "/../config/environment") require 'test_help' class Test::Unit::TestCase # Turn off transactional fixtures if you're working with MyISAM # tables in MySQL self.use_transactional_fixtures = true # Instantiated fixtures are slow, but give you @david where you # otherwise would need people(:david) self.use_instantiated_fixtures = false # Add more helper methods to be used by all tests here... end
Here we see the Test class being opened and two defaults being set.
self.use_transactional_fixtures = true self.use_instantiated_fixtures = false
In previous releases, transactional fixtures were turned off by default. Going forward, they're turned on by default. Conversely, instantiated fixtures were turned on by default in previous releases. Now they're turned off for all new Rails apps. In other words, the defaults for the two primary Rails testing options have been flipped.
All of your existing unit and functional tests are potentially affected by this change because they ultimately extend the Test class. And if your existing tests rely on the old defaults, as mine did, then upgrading your Rails app may break your existing tests. That said, you don't have to upgrade your tests and simply running gem update rails won't break your app. More on that later.
So why the complete reversal of defaults? One word: performance.
The Old Way
To appreciate what's new, let's recap how the old defaults worked.
Consider the following unit test:
require File.dirname(__FILE__) + '/../test_helper'
class CartTest < Test::Unit::TestCase
fixtures :products
def setup
@cart = Cart.new
end
def test_add_one_product
@cart.add_product @version_control_book
assert_equal 1, @cart.items.size
end
def test_add_duplicate_products
@cart.add_product @version_control_book
@cart.add_product @version_control_book
@cart.add_product @automation_book
assert_equal 2, @cart.items.size
end
end
The tests use the @version_control_book and @automation_book instance variables. These variables are instantiated automatically when the products fixture is loaded. The test/fixtures/products.yml fixture data file describes three products, as follows:
version_control_book: id: 1 title: Pragmatic Version Control description: How to use version control image_url: http://.../sk_svn_small.jpg price: 29.95 automation_book: id: 2 title: Pragmatic Project Automation description: How to automate your project image_url: http://.../sk_auto_small.jpg price: 29.95 unit_testing: id: 3 title: Pragmatic Unit Testing description: How to write better code image_url: http://.../sk_ut_small.jpg price: 29.95
Now let's see what happens behind the scenes when we run the test. (I had to make a tiny hack to Rails to get the fixture-specific output.)
SQL (0.000296) BEGIN Fixture Delete (0.000778) DELETE FROM products Fixture Insert (0.000828) INSERT INTO products (...) # first product Fixture Insert (0.043348) INSERT INTO products (...) # second product Fixture Insert (0.004889) INSERT INTO products (...) # third product SQL (0.000923) COMMIT Product Load (0.000992) SELECT * FROM products WHERE (products.id = 3) LIMIT 1 Product Load (0.000884) SELECT * FROM products WHERE (products.id = 2) LIMIT 1 Product Load (0.000949) SELECT * FROM products WHERE (products.id = 1) LIMIT 1 # setup runs # test_add_one_product runs SQL (0.000263) BEGIN Fixture Delete (0.001630) DELETE FROM products Fixture Insert (0.000754) INSERT INTO products (...) # first product Fixture Insert (0.000863) INSERT INTO products (...) # second product Fixture Insert (0.000992) INSERT INTO products (...) # third product SQL (0.002623) COMMIT Product Load (0.000887) SELECT * FROM products WHERE (products.id = 3) LIMIT 1 Product Load (0.000890) SELECT * FROM products WHERE (products.id = 2) LIMIT 1 Product Load (0.000830) SELECT * FROM products WHERE (products.id = 1) LIMIT 1 # setup runs # test_add_duplicate_products runs
Interesting. There's a lot going on as a result of including the following line in the test case:
fixtures :products
Consequently, Rails automatically does three things before each test method:
- Deletes all the test data in the products table of the test database. (DELETE FROM products)
- Inserts a row into the products table of the test database for each product listed in the fixture data file. (INSERT INTO products)
- Finds the instance corresponding to each product in the test database and assigns it to a instance variable of the same name. (SELECT * FROM products)
This is good because it means that each test method is isolated from database changes made by other test methods. That is, the fixtures are restored to their original state in the test database before each test method runs. What's the cost of this isolation? Let's do the math.
minimum # of SQL calls per test case = T * (F * (1 DELETE + R INSERTS + R SELECTS))
where T = # of test methods
F = # of declared fixture files
R = # of records specified in each fixture
So, for the example test case above, the math works out as:
2 * (1 * (1 DELETE + 3 INSERTS + 3 SELECTS)) = 14 SQL calls
That's not a huge number, but the example test case is a pip-squeak. Any respectable test case starts to rack up SQL calls faster than you can scream "In-memory database!". And pretty soon you aren't running the tests any more because you feel like you don't have time to test.
It doesn't have to be that way. Let's chip away some SQL calls before the tests get too slow.
Transactional Fixtures
Transactional fixtures use database transactions to isolate tests. Rather than deleting and re-inserting fixtures for each test method, transactional fixtures are loaded once at the beginning of the test case. The fixture data in the test database is restored to its original state after each test by doing a transaction rollback.
Let's re-run the test case, this time with transactional fixtures enabled, per the new defaults:
self.use_transactional_fixtures = true
The test log is noticeably shorter:
SQL (0.000493) BEGIN Fixture Delete (0.000805) DELETE FROM products Fixture Insert (0.001194) INSERT INTO products (...) # first product Fixture Insert (0.000824) INSERT INTO products (...) # second product Fixture Insert (0.000793) INSERT INTO products (...) # third product SQL (0.000982) COMMIT SQL (0.000208) BEGIN Product Load (0.002081) SELECT * FROM products WHERE (products.id = 3) LIMIT 1 Product Load (0.007071) SELECT * FROM products WHERE (products.id = 2) LIMIT 1 Product Load (0.001213) SELECT * FROM products WHERE (products.id = 1) LIMIT 1 # setup runs # test_add_one_product runs SQL (0.015225) ROLLBACK SQL (0.000236) BEGIN Product Load (0.000963) SELECT * FROM products WHERE (products.id = 3) LIMIT 1 Product Load (0.000793) SELECT * FROM products WHERE (products.id = 2) LIMIT 1 Product Load (0.000787) SELECT * FROM products WHERE (products.id = 1) LIMIT 1 # setup runs # test_add_duplicate_products runs SQL (0.002332) ROLLBACK
This time the test data in the products table is deleted and re-inserted from the fixture file exactly once. A transaction is started before each test method (BEGIN), then rolled back at the end (ROLLBACK).
Here's the math for transactional fixtures:
minimum # of SQL calls per test case = (F * (1 DELETE + R INSERTS)) + (T * R SELECTS)
where T = # of test methods
F = # of declared fixture files
R = # of records specified in each fixture
So we've spared ourselves the trouble of 4 extra database hits.
(1 * (1 DELETE + 3 INSERTS)) + (2 * 3 SELECTS) = 10 SQL calls
To take advantage of transactional fixtures, your database must support transactions. I realize this seems obvious, but it tripped me up because MySQL uses the MyISAM database type by default, as far as I can tell. And my favorite MySQL database demo tool (YourSQL) follows suit. Unfortunately, MyISAM doesn't support transactions. So if you're using MySQL, then you'll need to make sure your tables use the InnoDB table format. Here's an example of how to convert a table from MyISAM to InnoDB:
alter table products type=InnoDB;
The only drawback to using transactional fixtures is when you actually need to test transactions. Since your test is bracketed by a transaction, any transactions started in your code will be automatically rolled back.
On-Demand Instantiated Fixtures
By default in Rails 1.0, fixtures aren't instantiated and assigned to instance variables until you need them. Whereas previous releases would automatically create a @version_control_book instance variable, for example, before each test method, you can now use the fixture accessor method products(:version_control_book) to load fixture data into variables on demand.
So before running the test with instantiated fixtures disabled, we need to change the tests to use fixture accessor methods:
def test_add_one_product @cart.add_product products(:version_control_book) assert_equal 1, @cart.items.size end def test_add_duplicate_products @cart.add_product products(:version_control_book) @cart.add_product products(:version_control_book) @cart.add_product products(:automation_book) assert_equal 2, @cart.items.size end
Calls to products(:version_control_book), for example, are cached within a specific test method. That is, the first time you call it in a test method, a SQL query loads the corresponding model. The second time you call it within the same test method, it returns a cached result. Calling products(:version_control_book, :refresh) will force a reload.
OK, now let's run the test again, this time using both new defaults in Rails 1.0—transactional fixtures enabled and instantiated fixtures disabled:
self.use_transactional_fixtures = true self.use_instantiated_fixtures = false
The test log is even shorter:
SQL (0.000296) BEGIN Fixture Delete (0.001028) DELETE FROM products Fixture Insert (0.001111) INSERT INTO products (...) # first product Fixture Insert (0.003815) INSERT INTO products (...) # second product Fixture Insert (0.000737) INSERT INTO products (...) # third product SQL (0.000864) COMMIT SQL (0.000227) BEGIN # setup runs # test_add_one_product begins Product Load (0.000807) SELECT * FROM products WHERE (products.id = 1) LIMIT 1 # test_add_one_product ends SQL (0.001663) ROLLBACK SQL (0.000209) BEGIN # setup runs # test_add_duplicate_products begins Product Load (0.000850) SELECT * FROM products WHERE (products.id = 1) LIMIT 1 Product Load (0.001064) SELECT * FROM products WHERE (products.id = 2) LIMIT 1 # test_add_duplicate_products ends SQL (0.001645) ROLLBACK
Just as before, the test data in the products table is deleted and re-inserted from the fixture file exactly once. As well, database transactions still bracket each test method to keep them isolated. But notice what happens inside of each test method. Rather than going to the trouble to instantiate all three products before each test method, Rails simply hands you the reigns. If you use a fixture accessor in a test method, the respective fixture data is loaded and cached. If you don't use a specific piece of fixture data in a test method, you don't pay to have it loaded.
I'll spare you the math on this one, and instead show the results on a real-world project a bit later.
Upgrading Your Tests
First, you don't necessarily have to change your tests when you upgrade to Rails 1.0. You can simply choose not to have Rails update your test/test_helper.rb file when you run the rails command in your project directory.
Or you can simply change the generated test/test_helper.rb file and reset the new defaults back to their old settings, like so:
self.use_transactional_fixtures = false self.use_instantiated_fixtures = true
Or you can add those two lines to the top of any existing test case and override the new defaults just for that test case, like so:
require File.dirname(__FILE__) + '/../test_helper' class CartTest < Test::Unit::TestCase fixtures :products self.use_transactional_fixtures = false self.use_instantiated_fixtures = true # Your tests go here. end
Finally, you can bite the bullet and upgrade all of your tests by:
- Making sure you're using transactional database tables and
- replacing all occurrences of fixture instance variables with fixture accessor methods. Example: change @fred to users(:fred).
What's the Bottom Line?
You've been patiently waiting for a performance comparison on a real-world project, if only because you'd enjoy something to throw rocks at. Let me help you keep your arms pointed at the keyboard by saying that the numbers that follow don't relate to your project. As you've seen in the parenthetically-challenged math formulas, there are a few variables. Thus, you will get different results depending on how many fixtures you have, how much data is in each of those fixtures, how you write your tests, and how hard you press the Enter key to run the tests.
With that disclaimer behind us, I present you the performance numbers from a Rails project I'm working on.
Using the Old Defaults
> rake (unit tests) Finished in 13.426457 seconds. 57 tests, 134 assertions, 0 failures, 0 errors (functional tests) Finished in 20.938738 seconds. 53 tests, 183 assertions, 0 failures, 0 errors
Transactional Fixtures On, Instantiated Fixtures On
> rake (unit tests) Finished in 7.942351 seconds. 57 tests, 134 assertions, 0 failures, 0 errors (functional tests) Finished in 17.832574 seconds. 53 tests, 183 assertions, 0 failures, 0 errors
Transactional Fixtures On, Instantiated Fixtures Off
(This is the new default in Rails 1.0.)> rake (unit tests) Finished in 5.667295 seconds. 57 tests, 134 assertions, 0 failures, 0 errors (functional tests) Finished in 14.663263 seconds. 53 tests, 183 assertions, 0 failures, 0 errors
Unit tests are almost 3x faster; functional tests are about 25% faster. I suspect it would be a lot more dramatic on bigger suites and fixtures. I've heard rumblings of total times being increased as much as 5x.
Summary
Simple: The combined effects of the new testing defaults in Rails 1.0 make your tests go faster. And when you're running tests after every change, every second counts...
(We'll be talking about this and other tantalizing Rails goodies in the Pragmatic Studio.)
