Use another framework for the tests/specs of mspec itself

[eregon]

Currently RSpec 2 is used but this is really not ideal as it is very easy to accidentally override some RSpec methods by require-ing the full mspec. It's also confusing as both frameworks use mostly the same methods.

Tests for mspec itself should be written in a very simple test harness like minitest, or for instance mruby's assert.rb, so that newer implementations can easily run them.

The current tests are heavily relying on mocks, so that might be non-trivial to replace.
In general I'm in favor of removing these white-box mock tests and instead test at a higher level for the functionality and not for a specific implementation.

[ksss]

How about testing mspec by mspec?

[eregon]

@ksss Right, this could be one way and it would probably be relatively easy as syntax is mostly compatible. I didn't think about it!

The drawback is if mspec is broken in such a way that for instance it does not run some specs then we might have a hard time to notice it. I would think the probability of that is low, but ideally it's better to cross-check with some really simple testing tool.
As a nice side effect it also tests running mspec and not just the internals.

[ksss]

I think self testing is popular technique.

The rspec has been tested by the rspec itself.
The minitest has been tested by the minitest itself.
And as you know, The CRuby has been tested by the CRuby itself.

[eregon]

I tried it last night and a few tests mostly pass as is, except for some advanced mock support like
should_receive(:foo).with(:bar, an_instance_of(String)).

[elia]

To me there's a case for using assert.rb.

MSpec is dedicated to testing implementations and assert.rb lowers by an order of magnitude the barrier for new implementations.

Of course the burden would probably shift to the tester, but given the purpose of mspec I think it's worth considering this option.

[headius]

MSpec is dedicated to testing implementations and assert.rb lowers by an order of magnitude the barrier for new implementations.

Yes, but MSpec itself requires significantly more than assert.rb, so new implementations will still have to meet a high bar to even run ruby/spec. This isn't an argument against assert.rb...just pointing out that simpler tests in MSpec won't translate to much for a new Ruby impl.

Self-hosting tests (i.e. using mspec) can be really hard to figure out bugs, unless you pin the specs to an earlier known-working version...and then I don't know how you avoid conflicts between old and new. If you run the tests for mspec with the same checked-out version of mspec and something breaks, the tests may not even run far enough to tell you the problem.

I'd say I'm an advocate for testing with something other than mspec. I also agree with @eregon that testing with rspec is more confusing and tricky than it needs to be.

I also don't contribute much to mspec, so take my opinion for what it's worth.

[elia]

Yes, but MSpec itself requires significantly more than assert.rb, so new implementations will still have to meet a high bar to even run ruby/spec. This isn't an argument against assert.rb...just pointing out that simpler tests in MSpec won't translate to much for a new Ruby impl.

Completely agree. What I meant is that a new implementation could:

• first, dedicate itself to make assert.rb run properly (incredibly low barrier)
• then, it can start making the mspec suite pass (somewhat low barrier)
• finally, having implemented everything needed to run mspec, starts approaching ruby/spec

The final bar is the same, but it's easier to make progress in small bites. That said it shouldn't be necessarily assert.rb, but I see that level of simplicity as particularly useful in this context.

[eregon]

I decided to migrate MSpec's own specs to RSpec 3 with:

RSpec.configure do |config|
  config.disable_monkey_patching!
end

So then it's quite a bit clearer what's RSpec and what's MSpec in the specs now, and it's possible to test some MSpec constructs directly in the test without an extra process.
We can now load the entire mspec in spec_helper.rb without worrying about conflicts anymore.

It was relatively easy with transpec but there were still quite a few incompatibilites between RSpec 2 and 3 that I had to fix or workaround: a40a674...2d09cb5