NEWS

Release 0.11 (not yet released)

* Major incompatible change

  A children parameter has been added to the lambda list of the generic
  function ECLECTOR.PARSE-RESULT:MAKE-SKIPPED-INPUT-RESULT so that results
  which represent skipped material can have children. For example, before this
  change, a ECLECTOR.PARSE-RESULT:READ call which encountered the expression
  #+no-such-feature foo bar potentially constructed parse results for all
  (recursive) READ calls, that is for the whole expression, for
  no-such-feature, for foo and for bar, but the parse results for
  no-such-feature and foo could not be attached to a parent parse result and
  were thus lost. In other words the shape of the parse result tree was

    skipped input result #+no-such-feature foo
    expression result    bar

  With this change, the parse results in question can be attached to the parse
  result which represents the whole #+no-such-feature foo expression so that
  the entire parse result tree has the following shape

    skipped input result #+no-such-feature foo
      skipped input result no-such-feature
      skipped input result foo
    expression result    bar

  Since this is a major incompatible change, we offer the following workaround
  for clients that must support Eclector versions with and without this change:

    (eval-when (:compile-toplevel :load-toplevel :execute)
      (let* ((generic-function #'eclector.parse-result:make-skipped-input-result)
             (lambda-list      (c2mop:generic-function-lambda-list
                                generic-function)))
        (when (= (length lambda-list) 5)
          (pushnew 'skipped-input-children *features*))))
    (defmethod eclector.parse-result:make-skipped-input-result
        ((client client)
         (stream t)
         (reason t)
         #+PACKAGE-THIS-CODE-IS-READ-IN::skipped-input-children (children t)
         (source t))
      ...
      #+PACKAGE-THIS-CODE-IS-READ-IN::skipped-input-children (use children)
      ...)

  The above code pushes a symbol that is interned in a package under the
  control of the respective client (as opposed to the KEYWORD package) onto
  *FEATURES* before the second form is read and uses that feature to select
  either the version with or the version without the children parameter of the
  method definition. See Maintaining Portable Lisp Programs by Christophe
  Rhodes for a detailed discussion of this technique.

* The new condition type ECLECTOR.READER:STATE-VALUE-TYPE-ERROR can be used to
  indicate that a value of an unsuitable type has been provided for a reader
  state aspect.

* The reader state protocol now provides the generic function
  (SETF ECLECTOR.READTABLE:STATE-VALUE) which allows clients to set reader
  state aspects in addition to establishing dynamically scoped bindings.

* The macros ECLECTOR.READER:UNQUOTE and ECLECTOR.READER:UNQUOTE-SPLICING now
  signal sensible errors when used outside of the lexical scope of a
  ECLECTOR.READER:QUASIQUOTE macro call. Note that the name of the associated
  condition type is not exported for now since quasiquotation will be
  implemented in a separate module in the future.

  Such invalid uses can happen when the above macros are called directly or
  when the ,, ,@ and ,. reader macros are used in a way that constructs the
  unquoted expression in one context and then "injects" it into some other
  context, for example via an object reference #N# or read-time evaluation
  #.(...). Full example:

    (progn
      (print `(a #1=,(+ 1 2) c))
      (print #1#))

  Another minor aspect of this change is that the condition types
  ECLECTOR.READER:UNQUOTE-SPLICING-IN-DOTTED-LIST and
  ECLECTOR.READER:UNQUOTE-SPLICING-AT-TOP are no longer subtypes of
  COMMON-LISP:STREAM-ERROR. The previous relation did not make sense since
  errors of those types are signaled during macro expansion.

* Eclector now uses the reader state protocol instead of plain special
  variables to query and track the legality of quasiquotation operations and
  the consing dot. The additional reader state aspects are documented but
  remain internal for now.

  The (internal) macro ECLECTOR.READER::WITH-FORBIDDEN-QUASIQUOTATION is
  deprecated as of this release. Clients which really need a replacement
  immediately can use the new (internal) macro
  ECLECTOR.READER::WITH-QUASIQUOTATION-STATE.

Release 0.10 (2024-02-28)

* The deprecated generic functions ECLECTOR.PARSE-RESULT:SOURCE-POSITION and
  ECLECTOR.PARSE-RESULT:MAKE-SOURCE-RANGE have been removed. Clients should use
  ECLECTOR.BASE:SOURCE-POSITION and ECLECTOR.BASE:MAKE-SOURCE-RANGE
  respectively instead.

* The new reader ECLECTOR.BASE:RANGE-LENGTH can be applied to conditions of
  type ECLECTOR.BASE:STREAM-POSITION-CONDITION (which includes almost all
  conditions related to syntax errors) to determine the length of the
  sub-sequence of the input to which the condition in question pertains.

* Minor incompatible change

  The part of the labeled objects protocol that allows clients to construct
  parse results which represent labeled objects has been changed in an
  incompatible way. The change allows parse results which represent labeled
  objects to have child parse results but requires that clients construct parse
  results which represent labeled objects differently: instead of
  eql-specializing the result parameters of methods on
  ECLECTOR.PARSE-RESULT:MAKE-EXPRESSION-RESULT to
  ECLECTOR.PARSE-RESULT:**DEFINITION** and ECLECTOR.PARSE-RESULT:**REFERENCE**
  and receiving the labeled object in the children parameters, the result
  parameters now have to be specialized to the classes
  ECLECTOR.PARSE-RESULT:DEFINITION and ECLECTOR.PARSE-RESULT:REFERENCE
  respectively. The object passed as the result argument now contains the
  labeled object so that the children parameter can receive child parse results.

  This change is considered minor since the old mechanism described above was
  not documented. For now, the new mechanism also remains undocumented so that
  the design can be validated through experimentation before it is finalized.

* The new syntax-extensions module contains a collection of syntax extensions
  which are implemented as either mixin classes for clients or reader macro
  functions.

* The extended package prefix extension allows prefixing an expression with a
  package designator in order to read the expression with the designated
  package as the current package. For example

    my-package::(a b)

  is read as

    (my-package::a my-package::b)

  with this extension.

* A new syntax extension which is implemented by the reader macro
  ECLECTOR.SYNTAX-EXTENSIONS.S-EXPRESSION-COMMENT:S-EXPRESSION-COMMENT allows
  commenting out s-expressions in a fashion similar to SRFI 62 for scheme. One
  difference is that a numeric infix argument can be used to comment out a
  number of s-expressions different from 1:

    (frob r1 r2 :k3 4 #4; :k5 6 :k6 7)

* The concrete-syntax-tree module now produces a better tree structure for
  certain inputs like (0 . 0). Before this change the produced CST had the same
  CONCRETE-SYNTAX-TREE:ATOM-CST object as the CONCRETE-SYNTAX-TREE:FIRST and
  CONCRETE-SYNTAX-TREE:REST of the outer CONCRETE-SYNTAX-TREE:CONS-CST node.
  After this change the CONCRETE-SYNTAX-TREE:FIRST child is the
  CONCRETE-SYNTAX-TREE:ATOM-CST which corresponds to the first 0 in the input
  and the CONCRETE-SYNTAX-TREE:REST child is the CONCRETE-SYNTAX-TREE:ATOM-CST
  which corresponds to the second 0 in the input. In contrast to the previous
  example, an input like (#1=0 . #1#) continues to result in a single
  CONCRETE-SYNTAX-TREE:ATOM-CST in both the CONCRETE-SYNTAX-TREE:FIRST and
  CONCRETE-SYNTAX-TREE:REST slots of the outer CONCRETE-SYNTAX-TREE:CONS-CST
  object.

Release 0.9 (2023-03-19)

* The deprecated function ECLECTOR.CONCRETE-SYNTAX-TREE:CST-READ has been
  removed. Clients should use ECLECTOR.CONCRETE-SYNTAX-TREE:READ instead.

* ECLECTOR.READER:FIND-CHARACTER receives characters names with unmodified case
  and is also called in the #\<single character> case so that clients have more
  control over character lookup.

* The new generic function ECLECTOR.BASE:POSITION-OFFSET allows interested
  clients to refine the source positions of errors obtained by calling
  ECLECTOR.BASE:STREAM-POSITION.

* Some condition and restart reports have been improved.

* A discussion of the relation between circular objects and custom reader
  macros has been added to the manual.

* Problems in the ECLECTOR.READER:FIXUP method for hash tables have been fixed:
  keys were not checked for circular structure and circular structures in
  values were not fixed up in some cases.

* Eclector provides a new protocol for handling labeled objects, that is the
  objects defined and referenced by the #= and ## reader macros respectively.

* Eclector now avoids unnecessary fixup processing in object graphs with
  complicated definitions and references.

  Before this change, cases like

    #1=(1 #1# #2=(2 #2# ... #100=(100 #100#)))

  or

    #1=(1 #2=(2 ... #2#) ... #1#)

  led to unnecessary and/or repeated traversals during fixup processing.

* Fixup processing is now performed in parse result objects.

  Before this change, something like

    (eclector.concrete-syntax-tree:read-from-string "#1=(#1#)")

  produced a CST object, say cst, which failed to satisfy

    (eq (cst:first cst)       cst)
    (eq (cst:raw (first cst)) (cst:raw cst))

  The properties now hold.

* Clients can use the new mixin classes
  ECLECTOR.CONCRETE-SYNTAX-TREE:DEFINITION-CSTS-MIXIN and
  ECLECTOR.CONCRETE-SYNTAX-TREE:REFERENCE-CSTS-MIXIN to represent labeled
  object definitions and references as instances of
  ECLECTOR.CONCRETE-SYNTAX-TREE:DEFINITION-CST and
  ECLECTOR.CONCRETE-SYNTAX-TREE:REFERENCE-CST respectively.

* The stream position in conditions signaled by
  ECLECTOR.READER::SHARPSIGN-COLON is now always present.

* When Eclector is used to produce parse results, it no longer confuses
  end-of-input with having read NIL when nil is used as the eof-value (nil
  makes sense as an eof-value in that case since nil is generally not a
  possible parse result).

* A detailed description of the constraints on return values of the generic
  functions in the Reader behavior protocol has been added to the manual.

* The eclector-concrete-syntax-tree system now works with and requires version
  0.2 of the concrete-syntax-tree system.

* Eclector provides a new protocol for querying and binding behavior-changing
  aspects of the current state of the reader such as the current package, the
  current readtable and the current read base.

  Clients can use this protocol to control the reader state in other ways than
  binding the Common Lisp variables, for example by storing the values of
  reader state aspects in context objects.

  Furthermore, implementations which use Eclector as the Common Lisp reader can
  use this protocol to tie the CL:*READTABLE* aspect to the CL:*READTABLE*
  variable instead of the ECLECTOR.READER:*READTABLE* variable.

  The new protocol subsumes the purpose of the generic function
  ECLECTOR.READER:CALL-WITH-CURRENT-PACKAGE which is deprecated as of this
  Eclector version.

* Eclector now provides and uses by default a relaxed version of the
  ECLECTOR.READER::SHARPSIGN-S reader macro function which requires the input
  following #S to be read as a list but not necessarily be literally written as
  (TYPE INITARG₁ VALUE₁ …).

  A detailed discussion of the topic has been added to the manual.

Release 0.8 (2021-08-24)

* The default ECLECTOR.READER:READ-TOKEN method and the functions
  ECLECTOR.READER::SHARPSIGN-COLON and ECLECTOR.READER::SHARPSIGN-BACKSLASH are
  now more efficient as well as less redundant in terms of repeated code.

* The feature :ECLECTOR-DEFINE-CL-VARIABLES now controls whether the file
  code/reader/variables.lisp is loaded and thus whether the variables
  ECLECTOR.READER:*PACKAGE*, ECLECTOR.READER:*READ-EVAL*, etc. are defined.

Release 0.7 (2021-05-16)

* The incorrectly committed generic function ECLECTOR.READER:CHECK-SYMBOL-TOKEN
  has been fixed.

* Empty escape ranges like || are no longer interpreted as potential numbers.

* The default ECLECTOR.READER:INTERPRET-SYMBOL method now signals specific
  conditions and offers restarts for recovering from situations related to
  non-existent packages and symbols as well as non-exported symbols.

  The default error recovery strategy for invalid symbols now constructs an
  uninterned symbol of the given name instead of using NIL.

* The "consing dot" is no longer accepted in sub-expressions of
  ECLECTOR.READER::LEFT-PARENTHESIS.

  At the same time, it is now possible to recover from encountering the
  "consing dot" in invalid positions.

* The default ECLECTOR.READER:INTERPRET-TOKEN method has been optimized
  substantially.

* The ECLECTOR.READER:*CLIENT* variable and the source location protocol (that
  is the generic functions ECLECTOR.PARSE-RESULT:SOURCE-POSITION and
  ECLECTOR.PARSE-RESULT:MAKE-SOURCE-RANGE) have been moved to a new base module
  and package eclector.base which the reader module and the eclector.reader
  package can use. This structure allows code in the reader module to work with
  source locations.

  The name ECLECTOR.BASE:*CLIENT* remains exported as ECLECTOR.READER:*CLIENT*.

  The old names ECLECTOR.PARSE-RESULT:SOURCE-POSITION and
  ECLECTOR.PARSE-RESULT:MAKE-SOURCE-RANGE still exist but are now deprecated
  and will be removed in a future release.

* Conditions signaled by code in the reader module now include source positions
  which are obtained by calling ECLECTOR.BASE:SOURCE-POSITION.

Release 0.6 (2020-11-29)

* Bogus nil parse results are no longer generated by
  ECLECTOR.PARSE-RESULT:MAKE-SKIPPED-INPUT-RESULT calls when CL:*READ-SUPPRESS*
  is true.

* The new generic functions ECLECTOR.READER:READ-MAYBE-NOTHING and
  ECLECTOR.READER:CALL-AS-TOP-LEVEL-READ give clients additional entry points
  to the reader as well as customization possibilities. With these functions,
  the chain of functions calls for a READ call looks like this:

    eclector.reader:read
      eclector.reader:call-as-top-level-read
        eclector.reader:read-common
          eclector.reader:read-maybe-nothing
            ...
              eclector.reader:read-char
              eclector.reader:peek-char

  Diagrams which illustrate the relations between the new and existing
  functions have been added to the manual.

* The function ECLECTOR.READER::READ-RATIONAL now better respects the value of
  *READ-SUPPRESS*.

* Fix return value of ECLECTOR.READTABLE:SET-SYNTAX-FROM-CHAR, fix
  (SETF ECLECTOR.READTABLE:SYNTAX-FROM-CHAR) to also copy the macro character
  information.

* The semicolon reader macro now consumes the terminating newline character.

* Eclector now provides the generic function ECLECTOR.READER:WRAP-IN-FUNCTION.

* Reset ECLECTOR.READER::*LIST-READER* around recursive read in
  ECLECTOR.READER::SHARPSIGN-DOT.

* Implement and default to relaxed syntax for ECLECTOR.READER::SHARPSIGN-C. The
  strict version is still available as ECLECTOR.READER:STRICT-SHARPSIGN-C and
  can be installed into a custom readtable.

  A detailed discussion of the topic has been added to the manual.

* Eclector can now recover from reading invalid inputs like .. and ....

* Implement and default to relaxed syntax for
  ECLECTOR.READER::SHARPSIGN-SINGLE-QUOTE. The strict version is still
  available as ECLECTOR.READER:STRICT-SHARPSIGN-SINGLE-QUOTE and can be
  installed into a custom readtable.

  A detailed discussion of the topic has been added to the manual.

* Eclector now provides the generic function ECLECTOR.READER:CHECK-SYMBOL-TOKEN.

* Input of the form PACKAGE::|| is now correctly read as a symbol.

* Eclector can now recover from reading the invalid input :.

Release 0.5 (2020-06-09)

* The generic function ECLECTOR.READER:CALL-WITH-CURRENT-PACKAGE has been added.

* The previously missing functions
  ECLECTOR.PARSE-RESULT:READ-PRESERVING-WHITESPACE and
  ECLECTOR.PARSE-RESULT:READ-FROM-STRING have been added.

* The previously missing functions
  ECLECTOR.CONCRETE-SYNTAX-TREE:READ-PRESERVING-WHITESPACE and
  ECLECTOR.CONCRETE-SYNTAX-TREE:READ-FROM-STRING have been added.

* The function ECLECTOR.CONCRETE-SYNTAX-TREE:CST-READ has been renamed to
  ECLECTOR.CONCRETE-SYNTAX-TREE:READ. ECLECTOR.CONCRETE-SYNTAX-TREE:CST-READ
  still exists but is deprecated and will be removed in a future version.

* Quasiquote and unquote are now opt-out instead of opt-in. This allows
  quasiquotation in custom reader macros by default. The new macro
  ECLECTOR.READER::WITH-FORBIDDEN-QUASIQUOTATION is used by Eclector (and can
  be used in custom reader macros) to control this behavior.

* A method on ECLECTOR.READTABLE:READTABLEP for the simple readtable
  implementation has been added.

* The condition type ECLECTOR.BASE:END-OF-FILE is now a subtype of
  CL:STREAM-ERROR but not of CL:READER-ERROR.

* An error is now always signaled independently of the the value of the
  eof-error parameter when the end of input is encountered a after single
  escape or within a multiple escape. The new error conditions
  ECLECTOR.READER:UNTERMINATED-SINGLE-ESCAPE and
  ECLECTOR.READER:UNTERMINATED-MULTIPLE-ESCAPE are signaled in such situations.

* The set invalid sub-characters for # now conforms to the specification.

* The value of CL:*READ-BASE* is now used correctly when distinguishing numbers
  and symbols.

* When a number with a denominator of zero is read the new condition
  ECLECTOR.READER:ZERO-DENOMINATOR is signaled.

* The function ECLECTOR.READER:READ-DELIMITED-LIST has been added.

* The reader macro function ECLECTOR.READER::SHARPSIGN-S now accepts string
  designators as slot names.

* The reader macro functions ECLECTOR.READER::SHARPSIGN-EQUALS and
  ECLECTOR.READER::SHARPSIGN-SHARPSIGN respect the value of CL:*READ-SUPPRESS*.

* The default methods on the generic function ECLECTOR.READER:FIXUP now works
  correctly for STANDARD-OBJECT instances with unbound slots.

* The reader macro function ECLECTOR.READER::LEFT-PARENTHESIS now always reads
  until #\), not some "opposite" character.

* ECLECTOR.READER:*SKIP-REASON* is now set correctly when a line comment at the
  end of input is read.

* In almost all situations in which Eclector signals a syntax error, a restart
  named ECLECTOR.READER:RECOVER is now established which, when invoked performs
  some action which allows the remainder of the expression to be read. The
  convenience function ECLECTOR.READER:RECOVER can be used to invoke the
  restart.

Release 0.4 (2019-05-11)

* The reader macro function ECLECTOR.READER::SHARPSIGN-PLUS-MINUS now sets
  ECLECTOR.READER:*SKIP-REASON* so that parse results can be created with an
  accurate "reason" value.

* Constituent traits are now represented and used properly.

* The lambda lists of the functions ECLECTOR.READER:READ-CHAR and
  ECLECTOR.READER:PEEK-CHAR have been fixed.

* The function ECLECTOR.READER::READ-RATIONAL now respects CL:*READ-SUPPRESS*
  and handles inputs of the form 1 2 correctly.

* The reader macro function ECLECTOR.READER::SHARPSIGN-R now handles
  CL:*READ-SUPPRESS* better.

* The default method on the generic function ECLECTOR.READER:INTERPRET-TOKEN
  now distinguishes positive and negative float zeros and uses radix 10 instead
  of the value of CL:*READ-BASE* for float digits.

* The input .|| is now interpreted as a symbol instead of the "consing dot".

* Long lists are now read into concrete syntax tree results without relying on
  unbounded recursion.

* Syntax errors in the initial contents part of #A expressions now signal
  appropriate errors.

* Source ranges of parse results no longer include whitespace which followed
  the corresponding expression in the input.

* The lambda list of the function ECLECTOR.PARSE-RESULT:READ is now accurate.

Release 0.3 (2018-11-28)

* The function ECLECTOR.READER:PEEK-CHAR has been added. The new function is
  like CL:PEEK-CHAR but signals Eclector conditions and uses the Eclector
  readtable.

* The function ECLECTOR.READER:READ-FROM-STRING has been added. The new
  function is like CL:READ-FROM-STRING but uses Eclector's reader
  implementation.

* The reader macro function ECLECTOR.READER::SHARPSIGN-S and the generic
  function ECLECTOR.READER:MAKE-STRUCTURE-INSTANCE have been added. Eclector
  does not define any methods on the latter generic function since there is no
  portable way of creating a structure instance when only the symbol naming the
  structure is known.

* The generic function ECLECTOR.READER:INTERPRET-SYMBOL is now called when the
  reader creates uninterned symbols.

* The generic function ECLECTOR.READER:FIXUP now accepts a client object as the
  the argument.

* In the generic functions ECLECTOR.READER:WRAP-IN-QUASIQUOTE,
  ECLECTOR.READER:WRAP-IN-UNQUOTE and ECLECTOR.READER:WRAP-IN-UNQUOTE-SPLICING,
  the client parameter is now the first parameter.

* The generic function ECLECTOR.READER:WRAP-IN-QUOTE has been added.

Release 0.2 (2018-08-13)

* The concrete-syntax-tree module has been generalized into a parse-result
  module which provides a protocol for constructing arbitrary custom parse
  results. The concrete-syntax-tree module is now based on this new module but
  can be used as before by clients.

* The default value of the eof-error-p parameter of the
  ECLECTOR.READER:READ-CHAR function is now true.

Release 0.1 (2018-08-10)

* Eclector was created by extracting the reader module from the SICL repository.

* The initial release includes many improvements over the original SICL reader,
  particularly in the areas of customizability, error reporting and
  constructing parse results.