Add URL wildcard feature (fixes #47) (#53)

* Add URL wildcard feature (fixes #47)

* Replace confusing/inaccurate test

* Add another illustrative test of basic auth behavior

* Correctly describe test

* collapse unused constant

* Rename general purpose boundary chars away from wildcard

* Simplify boundary chars

* Fix bug with trailing slashes

* extra newline

* test double-star behavior

* moar tests

* Note breaking changes in post_install_message

* Bump version to 2.0

* Clarify language

* enshorten

* Update lib/webvalve/version.rb

Co-authored-by: Nathan Griffith <nathan@betterment.com>

* Better section title

* Switch to Addressable::Template strategy

* Factor out `#template` for legibility

* Fix URL edge case and add tests

* Typo in code comments

* more succinct

* also more succinct

* Update README to match Addressable::Template capabilities

* Better error message

* Better comment

* Update spec/webvalve/service_url_converter_spec.rb

* Match multiple query params!

* Revert "Match multiple query params!"

This reverts commit 878fea0.

* Revert "Better comment"

This reverts commit d1f8cb5.

* Revert "Better error message"

This reverts commit 6ed0170.

* Revert "also more succinct"

This reverts commit e20c482.

* Revert "more succinct"

This reverts commit 96c0aed.

* Revert "Typo in code comments"

This reverts commit cb35fe3.

* Revert "Factor out `#template` for legibility"

This reverts commit ad2775c.

* Revert "Switch to Addressable::Template strategy"

This reverts commit 2deb420.

* Don't rely on Addressable for tests when it's not a dependency

* 2.0.0 changelog entry

---------

Co-authored-by: Nathan Griffith <nathan@betterment.com>
Co-authored-by: Sam Moore <sam@betterment.com>

CHANGELOG.md

@@ -9,6 +9,10 @@ and this project aims to adhere to [Semantic Versioning](http://semver.org/spec/
 ### Added
 ### Removed
 
+## [2.0.0] - 2023-07-20
+### Added
+- Dynamic URL support via wildcards, Regexps, and Addressable::Templates
+
 ## [1.3.1] - 2023-07-20
 ### Changed
 - Replace usage of deprecated `File.exists?` in generator

README.md

@@ -218,6 +218,43 @@ WebValve.register FakeBank, url: ENV.fetch("SOME_CUSTOM_API_URL")
 WebValve.register FakeBank, url: "https://some-service.com"
 ```
 
+## Dynamic URLs
+
+If the service you are interacting with contains dynamic elements, e.g.
+an instance-specific subdomain, you can specify a wildcard in your url
+with the `*` character to match a series of zero or more characters
+within the same URL segment. For example:
+
+```bash
+export BANK_API_URL=https://*.mybank.com/
+```
+
+or
+
+```ruby
+WebValve.register FakeBank, url: "https://*.mybank.com"
+```
+
+Note: unlike filesystem globbing, `?` isn't respected to mean "exactly
+one character" because it's a URL delimiter character. Only `*` works
+for WebValve URL wildcards.
+
+Alternatively you can use `Addressable::Template`s or `Regexp`s to
+specify dynamic URLs if they for some reason aren't a good fit for the
+wildcard syntax. Note that there is no `ENV` var support for these
+formats because there is no detection logic to determine a URL string is
+actually meant to represent a URL template or regexp. For example:
+
+```ruby
+WebValve.register FakeBank, url: Addressable::Template.new("http://mybank.com{/path*}{?query}")
+```
+
+or
+
+```ruby
+WebValve.register FakeBank, url: %r{\Ahttp://mybank.com(/.*)?\z}
+```
+
 ## What's in a `FakeService`?
 
 The definition of `FakeService` is really simple. It's just a

lib/webvalve/manager.rb

@@ -1,6 +1,7 @@
 require 'webmock'
 require 'singleton'
 require 'set'
+require 'webvalve/service_url_converter'
 
 module WebValve
   ALWAYS_ENABLED_ENVS = %w(development test).freeze
@@ -148,7 +149,7 @@ def allowlist_service(config)
     end
 
     def url_to_regexp(url)
-      %r(\A#{Regexp.escape url})
+      ServiceUrlConverter.new(url: url).regexp
     end
 
     def ensure_non_duplicate_stub(config)

lib/webvalve/service_url_converter.rb

@@ -0,0 +1,24 @@
+module WebValve
+  class ServiceUrlConverter
+    TOKEN_BOUNDARY_CHARS = Regexp.escape('.:/?#@&=').freeze
+    WILDCARD_SUBSTITUTION = ('[^' + TOKEN_BOUNDARY_CHARS + ']*').freeze
+    URL_PREFIX_BOUNDARY = ('[' + TOKEN_BOUNDARY_CHARS + ']').freeze
+    URL_SUFFIX_PATTERN = ('((' + URL_PREFIX_BOUNDARY + '|(?<=' + URL_PREFIX_BOUNDARY + ')).*)?\z').freeze
+
+    attr_reader :url
+
+    def initialize(url:)
+      @url = url
+    end
+
+    def regexp
+      if url.is_a?(String)
+        regexp_string = Regexp.escape(url)
+        substituted_regexp_string = regexp_string.gsub('\*', WILDCARD_SUBSTITUTION)
+        %r(\A#{substituted_regexp_string}#{URL_SUFFIX_PATTERN})
+      else
+        url
+      end
+    end
+  end
+end

lib/webvalve/version.rb

@@ -1,3 +1,3 @@
 module WebValve
-  VERSION = "1.3.1"
+  VERSION = "2.0.0"
 end

spec/webvalve/manager_spec.rb

@@ -38,6 +38,9 @@
   end
 
   describe '#setup' do
+    let(:wildcard_substitution) { WebValve::ServiceUrlConverter::WILDCARD_SUBSTITUTION }
+    let(:url_suffix_pattern) { WebValve::ServiceUrlConverter::URL_SUFFIX_PATTERN }
+
     context 'when WebValve is disabled' do
       around do |ex|
         with_rails_env 'production' do
@@ -94,10 +97,17 @@
 
       it 'allowlists configured urls in webmock' do
         allow(WebMock).to receive(:disable_net_connect!)
-        results = [%r{\Ahttp://foo\.dev}, %r{\Ahttp://bar\.dev}]
+        results = [
+          %r{\Ahttp://foo\.dev#{url_suffix_pattern}},
+          %r{\Ahttp://bar\.dev#{url_suffix_pattern}},
+          %r{\Ahttp://bar\.#{wildcard_substitution}\.dev#{url_suffix_pattern}},
+          %r{\Ahttp://bar\.dev/\?foo=bar#{url_suffix_pattern}}
+        ]
 
         subject.allow_url 'http://foo.dev'
         subject.allow_url 'http://bar.dev'
+        subject.allow_url 'http://bar.*.dev'
+        subject.allow_url 'http://bar.dev/?foo=bar'
 
         subject.setup
 
@@ -115,7 +125,7 @@
           subject.setup
         end
 
-        expect(WebMock).to have_received(:stub_request).with(:any, %r{\Ahttp://something\.dev})
+        expect(WebMock).to have_received(:stub_request).with(:any, %r{\Ahttp://something\.dev#{url_suffix_pattern}})
         expect(web_mock_stubble).to have_received(:to_rack)
       end
 
@@ -153,7 +163,7 @@
           subject.register other_disabled_service.name
 
           expect { subject.setup }.to_not raise_error
-          expect(WebMock).to have_received(:stub_request).with(:any, %r{\Ahttp://something\.dev}).twice
+          expect(WebMock).to have_received(:stub_request).with(:any, %r{\Ahttp://something\.dev#{url_suffix_pattern}}).twice
         end
       end
     end
@@ -208,8 +218,8 @@
             subject.setup
           end
 
-          expect(WebMock).to have_received(:stub_request).with(:any, %r{\Ahttp://something\.dev})
-          expect(WebMock).not_to have_received(:stub_request).with(:any, %r{\Ahttp://other\.dev})
+          expect(WebMock).to have_received(:stub_request).with(:any, %r{\Ahttp://something\.dev#{url_suffix_pattern}})
+          expect(WebMock).not_to have_received(:stub_request).with(:any, %r{\Ahttp://other\.dev#{url_suffix_pattern}})
           expect(web_mock_stubble).to have_received(:to_rack).once
         end
 
@@ -261,9 +271,9 @@
           subject.setup
         end
 
-        expect(WebMock).to have_received(:stub_request).with(:any, %r{\Ahttp://something\.dev})
-        expect(WebMock).to have_received(:stub_request).with(:any, %r{\Ahttp://something\-else\.dev})
-        expect(WebMock).to have_received(:stub_request).with(:any, %r{\Ahttp://other\.dev})
+        expect(WebMock).to have_received(:stub_request).with(:any, %r{\Ahttp://something\.dev#{url_suffix_pattern}})
+        expect(WebMock).to have_received(:stub_request).with(:any, %r{\Ahttp://something\-else\.dev#{url_suffix_pattern}})
+        expect(WebMock).to have_received(:stub_request).with(:any, %r{\Ahttp://other\.dev#{url_suffix_pattern}})
         expect(web_mock_stubble).to have_received(:to_rack).exactly(3).times
       end
     end

spec/webvalve/service_url_converter_spec.rb

@@ -0,0 +1,194 @@
+require 'spec_helper'
+
+RSpec.describe WebValve::ServiceUrlConverter do
+  let(:url) { "http://bar.com" }
+
+  subject { described_class.new(url: url) }
+
+  describe '#regexp' do
+    it "returns a regexp" do
+      expect(subject.regexp).to be_a(Regexp)
+    end
+
+    context "with a regexp" do
+      let(:url) { %r{\Ahttp://foo\.com} }
+
+      it "returns the same object" do
+        expect(subject.regexp).to be_a(Regexp)
+        expect(subject.regexp).to equal(url)
+      end
+    end
+
+    context "with an empty url" do
+      let(:url) { "" }
+
+      it "matches empty string" do
+        expect("").to match(subject.regexp)
+      end
+
+      it "matches a string starting with a URL delimiter because the rest is just interpreted as suffix" do
+        expect(":do:do:dodo:do:do").to match(subject.regexp)
+      end
+
+      it "doesn't match a string that doesn't start with a delimiter" do
+        expect("jamietart:do:do:dodo:do:do").not_to match(subject.regexp)
+      end
+    end
+
+    context "with a boundary char on the end" do
+      let(:url) { "http://bar.com/" }
+
+      it "matches arbitrary suffixes" do
+        expect("http://bar.com/baz/bump/beep").to match(subject.regexp)
+      end
+    end
+
+    context "with multiple asterisks" do
+      let(:url) { "http://bar.com/**/bump" }
+
+      it "matches like a single asterisk" do
+        expect("http://bar.com/foo/bump").to match(subject.regexp)
+      end
+
+      it "doesn't match like a filesystem glob" do
+        expect("http://bar.com/foo/bar/bump").not_to match(subject.regexp)
+      end
+    end
+
+    context "with a trailing *" do
+      let(:url) { "http://bar.com/*" }
+
+      it "matches when empty" do
+        expect("http://bar.com/").to match(subject.regexp)
+      end
+
+      it "matches when existing" do
+        expect("http://bar.com/foobaloo").to match(subject.regexp)
+      end
+
+      it "matches with additional tokens" do
+        expect("http://bar.com/foobaloo/wink").to match(subject.regexp)
+      end
+
+      it "doesn't match when missing the trailing slash tho" do
+        expect("http://bar.com").not_to match(subject.regexp)
+      end
+    end
+
+    context "with a totally wildcarded protocol" do
+      let(:url) { "*://bar.com" }
+
+      it "matches http" do
+        expect("http://bar.com/").to match(subject.regexp)
+      end
+
+      it "matches anything else" do
+        expect("gopher://bar.com/").to match(subject.regexp)
+      end
+
+      it "matches empty" do
+        expect("://bar.com").to match(subject.regexp)
+      end
+    end
+
+    context "with a wildcarded partial protocol" do
+      let(:url) { "http*://bar.com" }
+
+      it "matches empty" do
+        expect("http://bar.com/").to match(subject.regexp)
+      end
+
+      it "matches full" do
+        expect("https://bar.com/").to match(subject.regexp)
+      end
+    end
+
+    context "with a TLD that is a substring of another TLD" do
+      let(:url) { "http://bar.co" }
+
+      it "doesn't match a different TLD when extending" do
+        expect("http://bar.com").not_to match(subject.regexp)
+      end
+    end
+
+    context "with a wildcard subdomain" do
+      let(:url) { "http://*.bar.com" }
+
+      it "matches" do
+        expect("http://foo.bar.com").to match(subject.regexp)
+      end
+
+      it "doesn't match when too many subdomains" do
+        expect("http://beep.foo.bar.com").not_to match(subject.regexp)
+      end
+    end
+
+    context "with a partial postfix wildcard subdomain" do
+      let(:url) { "http://foo*.bar.com" }
+
+      it "matches when present" do
+        expect("http://foobaz.bar.com").to match(subject.regexp)
+      end
+
+      it "matches when empty" do
+        expect("http://foo.bar.com").to match(subject.regexp)
+      end
+
+      it "doesn't match when out of order" do
+        expect("http://bazfoo.bar.com").not_to match(subject.regexp)
+      end
+    end
+
+    context "with a partial prefix wildcard subdomain" do
+      let(:url) { "http://*baz.bar.com" }
+
+      it "matches when present" do
+        expect("http://foobaz.bar.com").to match(subject.regexp)
+      end
+
+      it "matches when empty" do
+        expect("http://baz.bar.com").to match(subject.regexp)
+      end
+    end
+
+    context "with a wildcarded basic auth url" do
+      let(:url) { "http://*:*@bar.com" }
+
+      it "matches when present" do
+        expect("http://bilbo:baggins@bar.com").to match(subject.regexp)
+      end
+
+      it "doesn't match when malformed" do
+        expect("http://bilbobaggins@bar.com").not_to match(subject.regexp)
+      end
+
+      it "doesn't match when missing password part" do
+        expect("http://bilbo@bar.com").not_to match(subject.regexp)
+      end
+    end
+
+    context "with a wildcarded path" do
+      let(:url) { "http://bar.com/*/whatever" }
+
+      it "matches with arbitrarily spicy but legal, non-URL-significant characters" do
+        expect("http://bar.com/a0-_~[]!$'(),;%+/whatever").to match(subject.regexp)
+      end
+
+      it "doesn't match when you throw a URL-significant char in there" do
+        expect("http://bar.com/life=love/whatever").not_to match(subject.regexp)
+      end
+    end
+
+    context "with a wildcarded query param" do
+      let(:url) { "http://bar.com/whatever?foo=*&bar=bump" }
+
+      it "matches when present" do
+        expect("http://bar.com/whatever?foo=baz&bar=bump").to match(subject.regexp)
+      end
+
+      it "doesn't match when you throw a URL-significant char in there" do
+        expect("http://bar.com/whatever?foo=baz#&bar=bump").not_to match(subject.regexp)
+      end
+    end
+  end
+end

webvalve.gemspec

@@ -30,4 +30,21 @@ Gem::Specification.new do |s|
   s.add_development_dependency "rails"
 
   s.required_ruby_version = ">= 3.0.0"
+
+  s.post_install_message = <<~MSG
+    Thanks for installing WebValve!
+
+    Note for upgraders: If you're upgrading from a version less than 2.0, service
+    URL behavior has changed. Please verify that your app isn't relying on the
+    previous behavior:
+
+    1. `*` characters are now interpreted as wildcards, enabling dynamic URL
+       segments. In the unlikely event that your URLs use `*` literals, you'll need
+       to URL encode them (`%2A`) both in your URL spec and at runtime.
+
+    2. URL suffix matching is now strict. For example, `BAR_URL=http://bar.co` will
+       no longer match `https://bar.com`, but it will match `http://bar.co/foo`. If
+       you need to preserve the previous behavior, you can add a trailing `*` to
+       your URL spec, e.g. `BAR_URL=http://bar.co*`.
+  MSG
 end