I’ve been fortunate to spend the last month as the sole developer of greenfield Rails 4.1 app. As someone who’s spent quite a bit of time maintaining existing code, the freedom to establish patterns and choose tools is a highly welcomed change. One of the choices I made was to use Minitest and Rails fixtures.

The short is…it’s been great! So great that I’m having trouble imagining myself using anything else going forward.

Background

I got started with Rails in 2009. At the time, it felt like no one but the Rails-core team used test_unit. While I know this isn’t true, Rspec felt like the overwhelmingly popular testing framework from my perspective. As someone who didn’t have experience with testing prior to learning Ruby/Rails, I was looking for as much educational material as possible to learn what to test and how to test.

At the time (…and even now), there weren’t many books focused on Ruby testing techniques and specifically what and how to test a Rails application. So when I came across The Rspec Book I was excited there was finally some formal writing that would help me understand the best practices and concepts around testing with Ruby. Additionally, it seemed like the first steps of just about every Rails tutorial started with installing Rspec. Because I didn’t have an opinion about why or how, I went with the flow and forged ahead with Rspec at the center of my testing toolbox.

I admit, though, I never gave Test::Unit, or Minitest a fair shot. I immediately gravitated towards Rspec, for whatever reason, and didn’t consider doing otherwise until recently. Fortunately, it seems that more people are aware of Minitest and considering it a worthy option when starting out.

My Previous Setup

Here’s what a typical Gemfile looked like in one of my previous projects:

group :test do
  gem "rspec-rails"
  gem "factory_girl_rails"
  gem "capybara"
  gem "selenium-webdriver"
  gem "database_cleaner"
  gem "shoulda-matchers"
end

Aside from the additional gems, my spec_helper.rb had a number of settings like turning off transactions (in favor of database_cleaner) that went against the standard Rails testing conventions.

This setup can get so complicated, there’s even gems to manage the complication. For a framework that has so many conventions, this never felt right to me.

Below are my observations (in no particular order) after having spent 1 month using Minitest and fixtures in a Rails 4.1 application:

1. Fixtures force you to test against “real” data

Fixture data isn’t real. It’s staged, however, you have control over what and how much you add. I’ve found that creating structures around common subjects like familiar TV shows or movies allows you to move faster inside the app as the characters involved already have a preconceived hierarchy in your head.

Below is a sample from my users.yml fixture:

fred:
  first_name: Fred
  last_name: Flintstone
  email: fred@flintstone.com
  title: CEO
  password_digest: <%= ActiveRecord::FixtureSet.default_password_digest %>
  company: flintstone
  confirmed_at: <%= Chronic.parse("1/1/2014") %>
  invitation_token: <%= SecureRandom.urlsafe_base64 %>

wilma:
  first_name: Wilma
  last_name: Flintstone
  email: wilma@flintstone.com
  title: COO
  password_digest: <%= ActiveRecord::FixtureSet.default_password_digest %>
  company: flintstone
  supervisor: fred
  invitation_token: <%= SecureRandom.urlsafe_base64 %>

An interesting note is that while associations might have been painful in the past, notice how you can use flintstone as the company name and it will refer to the company fixture of the same name:

flintstone:
  name: Flintstone Inc.
  phone: 888-555-1212
  updated_at: <%= 6.months.ago %>
  created_at: <%= 6.months.ago %>

At every point in the application, I have well-structured data to test against. Using Factory Girl is frustrating if you need to create any kind of complex data structure with associations and seed data. When ActiveRecord associations get complex, I’ve found it frustrating and time consuming to bootstrap the initial data — largely because it’s painful to do in the setup of each test. While there are ways around this using factories with pre-established associations, I believe it pushes you to reach for stubs sooner. I’ve seen this first hand where the stubbing was too closely tied to implementation and tests fails with a single change to query conditions — even though the query returns the same data. The recent discussions about TDD have generated more conversation about similar topics. Overall, I’ve felt more confident in my tests when using fewer mocks and stubs, and when using fixtures, I’ve felt that need less frequently. My data is predictable and I’m confident in my tests.

Lastly, when Factory Girl inserts data before each test, there’s a cost associated with communicating with the database. Multiply that cost by many thousands and you have a slower test suite. Fixtures are inserted before the test suite runs, so outside of any test-specific mutation, typically no additional inserts are necessary. While this many seem trivial at first, the benefits will multiply over time and you’ll be left with a more performant test suite — and higher likelihood to run the tests more frequently.

Note: I’m aware of methods like build_model and others that create AR objects without touching the database, but there are times when testing using data from the database is necessary (ie. scopes, mutation methods, etc.)

Not to mention, you use easily load your fixture data in to your development environment.

2. Rspec provides more than one way to do something

Rspec provides a number of ways to do the same thing. This leads to confusion around supposedly-simple topics like how to assert two things are equal. Do you I use eq() or ==, or perhaps eql()? Who knows???

What about the fancy syntax around methods ending with “?”.

post.should be_active

Wait, so where’s the be_active method? Nope! Rspec parses the method due to the fact that active? is a legitimate method in the application under test. At first, I was enamored with the magic. However, later, I found thinking too much about how and what to write, when I should’ve just been typing out active? and asserting it’s false or true (which of course is an entirely other way to write it):

post.active?.should be_true

…or what about:

post.active?.should == true

Does that even work? I’ve learned to appreciate simplicity and knowing that there’s generally only one way to write an assertion:

assert post.active?

Aside from not floundering over the right assertion to make, I’ve found it leads to less syntax errors in my tests. Going through 2 or 3 iterations of test errors before the test actually gets to the application code is frustrating and a waste of time. Having fewer ways to do the same thing has led me to make fewer syntax errors when writing tests.

3. Setting up Capybara is trivial

If you’ve used Capybara in the past, you know that using it in conjunction with FactoryGirl is….interesting.

Avdi’s post on configuring database_cleaner has been my goto configuration:

RSpec.configure do |config|

  config.before(:suite) do
    DatabaseCleaner.clean_with(:truncation)
  end

  config.before(:each) do
    DatabaseCleaner.strategy = :transaction
  end

  config.before(:each, :js => true) do
    DatabaseCleaner.strategy = :truncation
  end

  config.before(:each) do
    DatabaseCleaner.start
  end

  config.after(:each) do
    DatabaseCleaner.clean
  end

end

This is after you first disable transactions in the spec_helper.rb, of course:

config.use_transactional_fixtures = false

Why is all this necessary?

When you use Factory Girl and you insert data in to the database during a test, it’s done so in a transaction. At the end of the test, the transaction is rolled back so the data doesn’t persist and the next test begins with a clean slate. All is well right? Not entirely…

When you use a javascript-enabled driver like selenium to run integration tests, browser actions run in a different thread. This removes the ability to see data setup within another thread/transaction. Because of this, you have to resort to a truncation strategy instead. Hence the following config option shown above:

config.before(:each, js: true) do
  DatabaseCleaner.strategy = :truncation
end

This is complex and not obvious for newcomers. Not to mention the complexity that arises if you actually use multi-threaded code or gems.

If you choose to use fixtures instead, the data is inserted at the beginning of each test run (not in an isolated transaction), so it’s available to any subsequent thread — browser action or not.

Using fixtures instead of factories removes the need for database_cleaner entirely under normal circumstances. Additionally, the only change that’s necessary to setup Capybara is the following addition to test_helper.rb:

require "capybara/rails"

class ActionDispatch::IntegrationTest
  include Capybara::DSL
end

That’s it…seriously…the crazy part to me is this configuration is largely commonplace in Rails test suites that use Capybara and Rspec. We’ve been largely spoiled by Rails in that the framework provides a solution for just about every common problem we encounter. The fact that a complex setup with database cleaner and deciding on truncating vs. transactions ever seemed reasonable seems counter-intuitive to the Rails experience. I’m, frankly, surprised that it’s seemed reasonable for so long.

Is it possible that practices/tools like Factory Girl are hurting the community more than they’re helping?

4. Lack of complex stubbing/mocking constructs simplifies code

Rspec makes it easy to drop in a stub or mock wherever/whenever (see #1 above). While there is some value in this, it makes it easy to abuse.

In fact, I’ve found that using fixtures has caused me to less frequently reach for stubs at all. When a customer comes to the application and has a good experience, the last thing they’re going to care about is whether boundary data was stubbed out. The fact that my tests are running against data the application will see on a daily basis gives me the confidence that all will be well when new code is sent to production.

Minitest does have a mocking library that’s easy to use and read. It’s not as extensive as what you get from Rspec out of the box, but nothing is stopping you from including the mocha gem or some other equivalent, if you need additional functionality (I haven’t had the need).

5. Snippets can help the uncertainty about Minitest assertion order

When I first approached Minitest, one of the long-standing questions in my head was the order in which the expected and actual value appear. At this point, I think it’s familiar enough to know without assistance, but why type more if you don’t have to.

I rely heavily on these Ruby snippets for vim, which takes away the pain of knowing the order of arguments.

6. Minitest is just Ruby

To be fair, Rspec is just Ruby too. But generally Rspec seems to have magic ways to do just about everything - shared examples, test setup, configuration. All these things have an “Rspec way”.

Minitest deals with this by just using Ruby. If you need shared examples, why not include a module that includes the shared tests?

I noticed during the first few days of my Minitest experience, I was looking for the “right” way to do something. With some urging from people I respect in the community, I realized it was just Ruby. That mindset allowed me to do just about anything I wanted using the language itself, rather than some magic from the Rspec DSL.

In some ways, I think too much magic gives us tunnel vision. After a few niceties are used (and abused), we start to believe that whatever tool we’re using will solve ALL our problems. Using Minitest has allowed me break out of this mindset and rely on the Ruby skills that I’ve developed to solve my testing challenges.

7. Deviating from Rails defaults doesn’t always provide value

Sure, not everything in Rails is ideal. In fact, it’s admirable to think about how many people actually get value from something so opinionated. After relying on the Rails default stack for the last month, I’ve realized how much simplicity I’ve been missing due to my choice of tools. I assumed that because the community was largely using Rspec and Factory Girl (what it seemed like from my perspective), it was a good idea. And while arguments can be made for either side, I wasn’t using Rspec because I was convinced it was better — I just didn’t know any better.

The ease at which I was able to get going with the Rails default stack using Minitest and fixtures has made me a convert. There was minimal setup and largely required very little additional configuration.

I have yet to feel the pains from using fixtures that some talk about. But I’ve been careful not to introduce large scale changes in the data without thinking through them before-hand.

Perhaps the app isn’t big enough yet? Maybe my data isn’t complicated enough? Or maybe I’m paying close enough attention to the effects that changes in data will have?

Whatever it is, it’s working for me right now. Part me feels like I’m going to have a moment where I say, “OHHHHH! This is why everyone uses Rspec and Factory Girl.” Although, it’s hard for me to imagine at this point. The default stack is working for me and staying out of the way, which is what I prefer.

Summary

I intentionally chose to leave out comments about performance of one framework vs the other. However, this deck provide great benchmarks on the topic.

I’m excited to add Minitest and fixtures to my toolkit. With the benefits I’ve seen so far, it’s hard for me to imagine using anything else going forward — assuming I have the choice.

If you have beef over the default Minitest assertion syntax, you’ll be happy to know that Minitest comes with the option to use a spec-style syntax. While it’s not identical to the Rspec syntax, it gets you closer to the natural language syntax, if that’s important to you.