On tests

Testing · 30 Mar 2025

After years of writing code and seeing how different testing approaches affect the quality and maintainability of software, I’ve developed some strong opinions about testing. While this post is about testing in general, I’ll focus on examples in Ruby on Rails, which is where I’ve spent my time lately.

We’ll explore various testing approaches, from the fundamental unit tests to the powerful stateful property testing. I’ll share practical examples, discuss how to handle common testing challenges like flaky tests and state management, and explain why I believe testing is a team responsibility rather than just a developer’s task.

Let’s dive in!

TL;DR

Integration tests are crucial - they catch the most challenging bugs that occur when components interact
Stateful property testing is a powerful technique for finding complex bugs in systems with internal state
Flaky tests should be fixed immediately - they erode trust in your test suite
Testing is a team responsibility - developers who write the code should write the tests
Focus on testing critical paths and complex business logic, not just coverage numbers
High test coverage isn’t always better - focus on meaningful tests that catch real issues

Unit Tests

Unit tests are like building blocks - they’re the foundation of your testing strategy. They test individual components in isolation, making them fast and reliable. In Ruby, we often use RSpec for unit testing. Here’s a simple example from a project I worked on:

# app/models/user.rb
class User
  def full_name
    "#{first_name} #{last_name}".strip
  end
end

# spec/models/user_spec.rb
RSpec.describe User do
  describe '#full_name' do
    it 'combines first and last name' do
      user = User.new(first_name: 'John', last_name: 'Doe')
      expect(user.full_name).to eq('John Doe')
    end
  end
end

While unit tests are great for testing business logic, I’ve learned the hard way that they can’t catch issues that arise from the interaction between components. That’s where integration tests come in.

Integration Tests

I like integration tests a lot, and I’ll tell you why - they test how different parts of your application work together, which is where most bugs actually occur. In my experience, the most challenging issues I’ve encountered weren’t in individual components but in how they interacted with each other.

Here’s an example:

# spec/requests/users_spec.rb
RSpec.describe 'Users API', type: :request do
  describe 'POST /api/users' do
    let(:valid_params) do
      {
        user: {
          email: '[email protected]',
          password: 'password123'
        }
      }
    end

    it 'creates a new user and sends welcome email' do
      expect {
        post '/api/users', params: valid_params
      }.to change(User, :count).by(1)
        .and have_enqueued_mail(WelcomeMailer, :welcome_email)

      expect(response).to have_http_status(:created)
    end
  end
end

Integration tests are particularly valuable because they:

Test real-world scenarios where components interact
Catch issues that unit tests might miss
Provide confidence that your features work end-to-end
Serve as living documentation of how your system should behave

Managing Test State

Managing test state is one of the most challenging aspects of testing. I’ve seen many projects where state management issues led to unreliable tests and debugging nightmares. Here are the key principles I’ve learned:

Keep Tests Independent: Each test should be able to run in isolation. Avoid sharing state between tests or using instance variables across test cases.
Handle Global State Explicitly: When dealing with global state or singletons, always reset them before and after tests. Make these dependencies clear in your test setup.
Use the Right Tools: For database state, use transactions for most tests (they’re fast) but switch to truncation for JavaScript tests or when you need a clean slate.
Document Complex State: When dealing with complex object relationships, use shared contexts and traits to make the setup clear and maintainable.

Here’s a simple example of these principles in action:

# spec/models/order_spec.rb
RSpec.describe Order do
  # Use shared context for complex setup
  include_context 'with a complete order'

  # Each test is independent
  it 'calculates total correctly' do
    expect(order.total).to eq(product.price)
  end

  it 'handles payment processing' do
    expect {
      order.process_payment
    }.to change(order, :status).to('paid')
  end
end

The key is to make state dependencies explicit and ensure each test can run independently. This makes your tests more reliable and easier to maintain.

The Flakiness Problem

Ah, flaky tests - the bane of every developer’s existence. In my experience, flakiness usually comes from two main sources:

Time-related issues: Tests that depend on specific timestamps or durations
Asynchronous operations: Jobs, background tasks, or external service calls

Here’s how I usually handle time-related issues:

# spec/support/time_helpers.rb
module TimeHelpers
  def freeze_time
    travel_to Time.current
  end

  def unfreeze_time
    travel_back
  end
end

# spec/requests/notifications_spec.rb
RSpec.describe 'Notifications', type: :request do
  include TimeHelpers

  it 'sends notification after 24 hours' do
    freeze_time
    user = create(:user)
    
    expect {
      NotificationJob.perform_later(user.id)
    }.to have_enqueued_job(NotificationJob).at(24.hours.from_now)
    
    unfreeze_time
  end
end

Another big challenge comes when dealing with enqueued jobs that aren’t actually executed during the test. In Rails, we have a few ways to handle this:

# spec/requests/orders_spec.rb
RSpec.describe 'Orders', type: :request do
  describe 'POST /orders' do
    it 'enqueues order processing job' do
      expect {
        post '/orders', params: { order: { items: [...] } }
      }.to have_enqueued_job(ProcessOrderJob)
    end

    it 'processes order synchronously' do
      # Sometimes you want to actually run the job
      perform_enqueued_jobs do
        post '/orders', params: { order: { items: [...] } }
      end
      
      expect(Order.last).to be_processed
    end
  end
end

The key is to be explicit about whether you want to:

Just verify that a job was enqueued (using have_enqueued_job)
Actually run the job (using perform_enqueued_jobs)
Run the job at a specific time (using have_enqueued_job.at)

The key lesson I’ve learned about flaky tests is to fix them immediately. It’s like the broken window theory - if you let one broken window stay unrepaired, more will follow. The same applies to flaky tests. They erode trust in your test suite and can lead to developers ignoring test failures.

Team Responsibility

Testing isn’t just the responsibility of the person writing the code - it’s a team effort. I’ve seen too many teams where testing is treated as an afterthought or delegated to a separate QA team. In my experience, this approach rarely works well.

The developers who write the code should also write the tests. They understand the implementation details and edge cases better than anyone else. This doesn’t mean QA isn’t valuable - they bring a different perspective and can help identify gaps in test coverage.

Test Coverage: Quality Over Quantity

I’ve worked in places where there was a strict requirement for 90%+ test coverage. While this might sound good on paper, it often leads to meaningless tests that don’t add value. Instead, I focus on testing the critical paths and complex business logic.

For example, in a payment processing system, I’d prioritize testing:

Payment processing flows
Error handling
Edge cases in business rules
Integration with payment providers

Rather than testing simple getters and setters or trivial methods.

Stateful Property Testing: A Powerful Testing Technique

While regular property testing is great for pure functions, it falls short when testing systems with internal state. This is where stateful property testing comes in - it generates sequences of operations and verifies that the system maintains its invariants throughout these state changes.

Here’s a practical example from a project where I used stateful property testing to verify a simple counter system (similar to a database counter):

# spec/models/counter_spec.rb
RSpec.describe Counter do
  it 'maintains consistency across operations' do
    property_of {
      array(one_of([
        [:increment],
        [:decrement]
      ]))
    }.check do |operations|
      counter = Counter.new(0)
      expected_count = 0
      
      operations.each do |op|
        case op
        in [:increment]
          counter.increment
          expected_count += 1
        in [:decrement]
          # Only decrement if we have a positive count
          if expected_count > 0
            counter.decrement
            expected_count -= 1
          end
        end
        
        # Verify invariants after each operation
        expect(counter.count).to eq(expected_count)
        expect(counter.count).to be >= 0
      end
    end
  end
end

This example demonstrates several key aspects of stateful property testing:

State Management: We maintain both the actual counter and an expected state
Preconditions: We only allow decrementing when the counter is positive
Invariants: We verify that the counter never goes below zero and matches our expected state

The beauty of stateful property testing is that it can find complex bugs that would be nearly impossible to catch with traditional unit tests. For example, it might discover that:

A specific sequence of operations leads to inconsistent state
Certain operations don’t properly maintain system invariants
Edge cases in state transitions that weren’t considered

I’ve found it particularly useful for testing:

Database operations and transactions
Caching systems
State machines
Complex data structures
Distributed systems

The key is to define good invariants that should always hold true, regardless of the sequence of operations performed on the system.

Best Practices I’ve Learned

Over the years, I’ve picked up several best practices that have served me well:

Keep Tests Independent: Each test should be able to run in isolation
Use Meaningful Descriptions: Your test descriptions should tell a story
Follow the DRY Principle: Use shared contexts and helpers when appropriate
Regular Maintenance: Keep your tests up to date with your code changes
Fix Flaky Tests Immediately: Don’t let them accumulate
Focus on Critical Paths: Test what matters most to your business
Use (Stateful) Property Testing: For complex logic and edge cases

Conclusion

A well-thought-out testing strategy is crucial for maintaining a healthy codebase. While unit tests are essential, don’t underestimate the value of integration tests. They provide confidence that your features work together as expected and catch issues that unit tests might miss.

Remember that testing is an investment in your codebase’s future. The time you spend writing tests today saves you debugging time tomorrow and helps maintain a stable, reliable application.

Happy testing!