Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

replace the :continue option with a #continue_lex method #1151

Merged
merged 3 commits into from
Jun 3, 2019
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 21 additions & 6 deletions lib/rouge/lexer.rb
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,14 @@ def lex(stream, opts={}, &b)
new(opts).lex(stream, &b)
end

# In case #continue_lex is called statically, we simply
# begin a new lex from the beginning, since there is no state.
#
# @see #continue_lex
def continue_lex(*a, &b)
pyrmont marked this conversation as resolved.
Show resolved Hide resolved
lex(*a, &b)
end

# Given a name in string, return the correct lexer class.
# @param [String] name
# @return [Class<Rouge::Lexer>,nil]
Expand Down Expand Up @@ -405,15 +413,22 @@ def reset!

# Given a string, yield [token, chunk] pairs. If no block is given,
# an enumerator is returned.
#
# @option opts :continue
# Continue the lex from the previous state (i.e. don't call #reset!)
def lex(string, opts={}, &b)
return enum_for(:lex, string, opts) unless block_given?
def lex(string, opts=nil, &b)
if opts
warn 'the :continue option to Formatter#lex is deprecated, use #continue_lex instead.'
end

return enum_for(:lex, string) unless block_given?

Lexer.assert_utf8!(string)
reset!
pyrmont marked this conversation as resolved.
Show resolved Hide resolved

continue_lex(string, &b)
end

reset! unless opts[:continue]
# Continue the lex from the the current state without resetting
def continue_lex(string, &b)
return enum_for(:continue_lex, string, &b) unless block_given?

# consolidate consecutive tokens of the same type
last_token = nil
Expand Down
4 changes: 2 additions & 2 deletions lib/rouge/lexers/console.rb
Original file line number Diff line number Diff line change
Expand Up @@ -118,7 +118,7 @@ def process_line(input, &output)
$' =~ /\A\s*/
yield Text, $& unless $&.empty?

lang_lexer.lex($', continue: true, &output)
lang_lexer.continue_lex($', &output)
elsif comment_regex =~ input[0].strip
puts "console: matched comment #{input[0].inspect}" if @debug
output_lexer.reset!
Expand All @@ -129,7 +129,7 @@ def process_line(input, &output)
puts "console: matched output #{input[0].inspect}" if @debug
lang_lexer.reset!

output_lexer.lex(input[0], continue: true, &output)
output_lexer.continue_lex(input[0], &output)
end
end
end
Expand Down
10 changes: 5 additions & 5 deletions lib/rouge/regex_lexer.rb
Original file line number Diff line number Diff line change
Expand Up @@ -352,10 +352,10 @@ def groups(*tokens)
end
end

# Delegate the lex to another lexer. The #lex method will be called
# with `:continue` set to true, so that #reset! will not be called.
# In this way, a single lexer can be repeatedly delegated to while
# maintaining its own internal state stack.
# Delegate the lex to another lexer. We use the `continue_lex` method
# so that #reset! will not be called. In this way, a single lexer
# can be repeatedly delegated to while maintaining its own internal
# state stack.
#
# @param [#lex] lexer
# The lexer or lexer class to delegate to
Expand All @@ -365,7 +365,7 @@ def delegate(lexer, text=nil)
puts " delegating to #{lexer.inspect}" if @debug
text ||= @current_stream[0]

lexer.lex(text, :continue => true) do |tok, val|
lexer.continue_lex(text) do |tok, val|
puts " delegated token: #{tok.inspect}, #{val.inspect}" if @debug
yield_token(tok, val)
end
Expand Down
2 changes: 1 addition & 1 deletion spec/lexers/markdown_spec.rb
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@

it 'recognizes code block when lexer is continued' do
subject.lex("```ruby\n").to_a
actual = subject.lex("@foo\n```\n",continue: true).map { |token, value| [ token.qualname, value ] }
actual = subject.continue_lex("@foo\n```\n").map { |token, value| [ token.qualname, value ] }
assert { ["Name.Variable.Instance", "@foo"] == actual.first }
end

Expand Down