Lift Framework 3.0-M8: Security and milestone release

This release features a dependency update to fix a security issue. See below.

It also features a large number of bug fixes and targeted improvements
based on feedback from actual use of Lift 3. It is expected to be the last Lift 3
milestone release, with the next build expected to be RC1. We don't expect
to land any more feature work before releasing Lift 3.0.0, and would like folks
to start hammering this release to find any lurking bugs that haven't yet been
caught.

Security

Lift 3.0 milestones were found to be using an outdated version of Apache
Commons FileUpload, which had an unpatched DoS vulnerability.

Lift 3.0-M8 bumps the dependency to the latest version, 1.3.1, which fixes
this vulnerability. Additionally, if you're using sbt, you can directly depend
on commons-fileupload 1.3.1 to evict the Lift dependency without upgrading
Lift. The versions are compatible with respect to Lift's usage.

Compatibility Notes

Lift 3.0-M7 was the last Lift build built with java 7. Future builds and the
final Lift 3.0.0 release will be built on Java 8. If you use Java 7, please
make sure to add the commons-fileupload 1.3.1 dependency mentioned
above so you are protected from the DoS vulnerability.

Also, earlier builds of Lift 3.0 defaulted the Content-Security-Policy to
report only in dev mode (which will produce error messages and logs, but
not prevent anything from actually loading/running), but actually enforced
the security policy in production. For the initial Lift 3.0.0 release, we are
relaxing this to allow for an easier transition: the default will be report-only
in both dev mode and production mode. Lift 3.1 will return to a secure
production default.

You can get the previous behavior by setting:

LiftRules.securityRules = () => SecurityRules(enforceInOtherModes = true)

You can also enforce in dev mode by passing the enforceInDevMode = true
parameter.

Note about M7

We cut M7 a little earlier than we wanted to announce another milestone,
so rather than announcing it we left the artifacts but made no official
announcement. The release notes below include the changes in M7.

Improvements

• (#1726) LiftRules.autoIncludeAjaxCalc, which decides whether or not to
  include Lift's JavaScript, now makes no attempt to include the lift.js file if the
  current request is stateless. Serving lift.js triggers session creation, so a
  page that was stateless for performance reasons but included lift.js would
  still cause session churn. The new default avoids this, though you can return
  to the old behavior if preferred.
• (#1710) Until now, render produced a Document, which is a now-deprecated Scala
  intermediary type. Passing that Document to pretty or compact would produce
  the desired kind of JSON output. We now drop Document from the pipeline,
  with compact(render) and pretty(render) producing deprecation warnings.
  These forms will be removed in 3.1, in favor of compactRender and prettyRender.
  The new serialization code is also both faster and more customizable, allowing
  rendering to any Java Appendable and some more customization on how
  renderings are formatted.
• (#1736) JSON serialization is well-specified, but when serializing for parsing
  by a JS language parser rather than a JSON parser (e.g., in JSON-P), escaping
  works a little differently. Lift now supports changing characters that are escaped
  and provides JS-specific convenience configurations for serializing JSON in
  ways that won't break for these cases.
• (#1741) RestHelper will now properly handle async responses wrapped in
  Scala Futures. Before, it only handled LAFuture. These async responses
  are handled using the same continuation mechanism that powers Lift's comets.
• (#1717) Deserializing JSON to a List[Option[T]] did not work. It now works
  correctly, deserializing null to None.
• (#1719) lift-json's write function can now take primitive values to serialize (e.g.,
  Serialization.write(3)). Before, it only supported object types.
• (#1752) Lift 3.0 finds on* attributes and other embedded JS and separates
  it out into actual JS that is invoked separately. This helps to enforce tighter
  Content-Security-Policy restrictions. Before, in AJAX responses that included
  HTML, we were not performing this event extraction. Now, event extraction
  happens for both pages and AJAX responses uniformly.

Fixes

• (#1711) S.appendJs could behave strangely when invoked from within
  JsCmd construction, and particularly in Wiring and in certain comet contexts.
  It should now behave predictably across the board.
• (#1714) MatchWithoutCurrentValue did not work correctly for Menus
  that did not include a *. This is now fixed.
• (#1718) Deserializing an incorrect type to an Option[T] would produce a
  None. Now, only a null produces a None, something that can deserialize
  to a T produces a Some[T], and a non-null value that cannot deserialize
  to a T produces a MappingException.
• (#1721) Lift 3's default client-side logError implementation was broken and
  caused a stack overflow. It is now fixed.
• (#1733) lift-markdown was failing to correctly break up parsing by newline.
  This was only on Java 8 due to a change in regex behavior in Java; it is now
  fixed.
• (#1734) CssUrlPrefixer, which prepends the context path of the application
  to served CSS, was failing to properly parse certain URLs, particularly those
  that included , and %. It was also always emitting URLs wrapped in single
  quotes irrespective of the source CSS's style. All of these issues are now fixed.
• (#1737) Msgs now appends any JS it needs to the page JS, rather than
  inserting it into the page markup.
• (#1744) Fixes for IE8 in lift.js. We were relying on some functionality that
  does not exist in IE8. We now have a polyfill for this functionality.
• (#1748) Lift's non-jQuery implementation of internal functionality now properly
  sets X-Requested-With for AJAX requests, thus allowing Lift to internally
  recognize AJAX requests (via Req's xhr_? method).
• (#1749) Our mongo dependencies were upgraded to the latest driver that
  still provides support for some of our older customization strategies. The
  older customization is now deprecated for removal in 3.1, so that we can
  catch up to the latest mongo driver. Note that the driver we currently depend
  on is still compatible with the latest server versions.
• (#1750) When trying to use Menu.item, we were trying to identify the current
  page using S.request. This failed when we were trying to build menu HTML
  from an AJAX callback, since S.request no longer referred to the current page,
  but rather the AJAX request. We now use S.originalRequest, which does refer
  to the current page even in an AJAX callback.

Deprecations

• (#1710) In lift-json, compact and pretty (previously used in conjunction with
  render to serialize JSON to a String) are now deprecated in favor of compactRender
  and prettyRender. They will be removed in Lift 3.1.
• (#1742) FocusOnLoad is now deprecated in favor of S.appendJs(Focus(...))
  and/or using the HTML5 autofocus attribute.
• (#1749) Our mongo dependencies were upgraded to the latest driver that
  supports some of our older customization strategies. The older customization
  is now deprecated for removal in 3.1. In particular, these are defineDbAuth
  and useSession wrappers.

Lift 2.6.3: Security Release

Lift 2.6 was found to be using an outdated version of Apache Commons
FileUpload, which had an unpatched DoS vulnerability.

Lift 2.6.3 bumps the dependency to the latest version, 1.3.1, which fixes
this vulnerability. Additionally, if you're using sbt, you can directly depend
on commons-fileupload 1.3.1 to evict the Lift dependency. The versions
are compatible with respect to Lift's usage.

Lift 2.5.4: Security Release

Lift 2.5 was found to be using an outdated version of Apache Commons
FileUpload, which had an unpatched DoS vulnerability.

Lift 2.5.4 bumps the dependency to the latest version, 1.3.1, which fixes
this vulnerability. Additionally, if you're using sbt, you can directly depend
on commons-fileupload 1.3.1 to evict the Lift dependency without upgrading
Lift. The versions are compatible with respect to Lift's usage.

Lift Framework 3.0-M6

This release features a good bit of cleanup from the previous one. It's been a busy
few months for the committers so the work is mostly in targeted improvements and
bug fixes:

Improvements

• (#1706) Thanks to @chriswebster, an up to 90% performance increase in JSON
  serialization for lift-json!
• (#1693) ParamFailure now behaves more correctly with ~>. That operator, when
  used on a Failure, returns a ParamFailure with the passed parameter, while
  preserving the rest of the original Failure's information. Before, invoking ~> with
  a parameter on a ParamFailure would replace the existing parameter with the new
  one, thus losing the previous parameter altogether. Now, we produce a copy of the
  ParamFailure with the new parameter, but we chain the previous ParamFailure
  to it so that it the original parameter is still accessible.
• (#1698) Lift 3 sessions allow an actor proxy for client code to be created via
  serverActorForClient. This release includes some new setup configuration
  when creating those actors, like providing setup and teardown functions as well
  as ways to convert from application messages into client JS content.
• (#1701) There is now a LiftRules.attributeForRemovedEventAttributes. When
  Lift strips event-related attributes (e.g. onclick, onsubmit, etc), if this rule is
  set to a String, it will tag the element whose event attributes were removed
  with the attributes that were removed. For example, setting it to Some("data-lift-removed-attributes")
  might result in a data-lift-removed-attributes="onclick onmouseover" attribute
  in your HTML if that element had its click and mouseover event handlers moved
  into page-specific JS. This allows you to make styling decisions based on these
  attributes, if you want to. See the rule documentation for more.
• (#1705) Until now, jsonCall would fail silently on both client and server if the
  submitted data from the client wasn't valid JSON. Now, it will log (at an ERROR
  log level) on the server if invalid JSON is sent. Whether this log info will be printed
  or not is configurable using usual logger configuration.
• (#1709) You can now register to find out when a function owner no longer has
  available functions bound in a session. Function owners typically (though not always)
  correspond to server-side functions bound or a page rendering (e.g., form fields).
  These usually expire ~2 minutes after the user is no longer on the page, but until
  now there was no way to find out that a given page was no longer “in scope”. In
  certain cases, that could be interesting to clean up page-related temporary assets
  or resource allocations. We now provide LiftSession.onFunctionOwnersRemoved,
  which can be set during application startup so that any new sessions can register
  those listeners. @andreak posted an excellent example of how they are using this
  on the mailing list.

Bug Fixes

• (#1694) When generating post-page JavaScript (mostly used by Wiring), there
  were some cases where null pointer exceptions could be thrown. These should
  be gone.
• (#1692) When binding onclick to an element that had an href, the new code
  that extracts event handlers into page-specific JS was failing to bind the click
  handler. This is now fixed, as is a related bug where elements that didn't need
  handlers still had a random id assigned to them.
• (#1677) When sending messages to comet actors before they are created (via
  LiftSession.sendCometActorMessage), they would arrive in reverse order.
  This is now fixed, and the first message sent using sendCometActorMessage
  arrives first.
• (#1703) When we moved away from bind, the CSS bindings in ProtoUser
  that checked password changes were incorrect, in that they never saw the
  user having submitted two versions of the same password. This is now fixed,
  and the change password stuff provided by ProtoUser is now correct.

Lift 2.5.3: Security Release

Lift 2.5.1 was found to be vulnerable to XML External Entity attacks,
which can leak private files through your application when parsing certain
types of XML. In the process of communicating the vulnerability to Typesafe,
they referred us to a more-restricted version of XML parsing used to prevent
additional vulnerabilities like the billion laughs vulnerability and its sibling
quadratic blowup vulnerability.

Lift 2.5.2 fixed the XXE attacks, but Lift 2.5.3 was released to shore up the
additional vulnerabilities we were made aware of. This is done by disabling
inline DTDs altogether. We also enable secure processing, which adds restrictions
to supporting parsers to prevent denial-of-service style attacks on XML parsing.
Anyone who switched to use net.liftweb.util.Helpers.secureXML will automatically
get these additional protections.

To do this, Lift 2.5.3 depends on Apache Xerces version 2.11. Some concerns
have been brought up about this additional dependency and how it may interact
with other XML parsing that may occur in the application; please provide feedback
on the mailing list if you run into any issues related to this new dependency.

Lift 3.0-M5-1: Security Snapshot Release

Lift 3.0-SNAPSHOT was found to be vulnerable to XML External Entity attacks,
which can leak private files through your application when parsing certain
types of XML. In the process of communicating the vulnerability to Typesafe,
they referred us to a more-restricted version of XML parsing used to prevent
additional vulnerabilities like the billion laughs vulnerability and its sibling
quadratic blowup vulnerability.

Lift 3.0-M5 fixed the XXE attacks, but Lift 3.0-M5-1 was released to shore up the
additional vulnerabilities we were made aware of. This is done by disabling
inline DTDs altogether. We also enable secure processing, which adds restrictions
to supporting parsers to prevent denial-of-service style attacks on XML parsing.
Anyone who switched to use net.liftweb.util.Helpers.secureXML will automatically
get these additional protections.

To do this, Lift 3.0-M5-1 depends on Apache Xerces version 2.11. Some concerns
have been brought up about this additional dependency and how it may interact
with other XML parsing that may occur in the application; please provide feedback
on the mailing list if you run into any issues related to this new dependency.

Lift 3.0-M4-1: Security Milestone Release

Lift 3.0-M3 was found to be vulnerable to XML External Entity attacks,
which can leak private files through your application when parsing certain
types of XML. In the process of communicating the vulnerability to Typesafe,
they referred us to a more-restricted version of XML parsing used to prevent
additional vulnerabilities like the billion laughs vulnerability and its sibling
quadratic blowup vulnerability.

Lift 3.0-M4 fixed the XXE attacks, but Lift 3.0-M4-1 was released to shore up the
additional vulnerabilities we were made aware of. This is done by disabling
inline DTDs altogether. We also enable secure processing, which adds restrictions
to supporting parsers to prevent denial-of-service style attacks on XML parsing.
Anyone who switched to use net.liftweb.util.Helpers.secureXML will automatically
get these additional protections.

To do this, Lift 3.0-M4-1 depends on Apache Xerces version 2.11. Some concerns
have been brought up about this additional dependency and how it may interact
with other XML parsing that may occur in the application; please provide feedback
on the mailing list if you run into any issues related to this new dependency.

Lift 2.6.2: Security Release

Lift 2.6 was found to be vulnerable to XML External Entity attacks,
which can leak private files through your application when parsing certain
types of XML. In the process of communicating the vulnerability to Typesafe,
they referred us to a more-restricted version of XML parsing used to prevent
additional vulnerabilities like the billion laughs vulnerability and its sibling
quadratic blowup vulnerability.

Lift 2.6.1 fixed the XXE attacks, but Lift 2.6.2 was released to shore up the
additional vulnerabilities we were made aware of. This is done by disabling
inline DTDs altogether. We also enable secure processing, which adds restrictions
to supporting parsers to prevent denial-of-service style attacks on XML parsing.
Anyone who switched to use net.liftweb.util.Helpers.secureXML will automatically
get these additional protections.

To do this, Lift 2.6.2 depends on Apache Xerces version 2.11. Some concerns
have been brought up about this additional dependency and how it may interact
with other XML parsing that may occur in the application; please provide feedback
on the mailing list if you run into any issues related to this new dependency.

Lift 3.0-M5: Security Snapshot Release

Lift 3.0-SNAPSHOT was found to be vulnerable to XML External Entity attacks,
which can leak private files through your application when parsing certain
types of XML.

Lift 3.0-M5 introduces net.liftweb.util.Helpers.secureXML, an analogous object to
Scala's scala.xml.XML that is secured against XXE attacks by disabling external
entities in doctypes. If you are parsing untrusted user-provided XML using
scala.xml.XML, it is recommended that you switch to secureXML instead.

Lift 3.0-M5 was rapidly superseded by Lift 3.0-M5-1, which secures the secureXML
object against a few additional XML-based attacks.

Lift 3.0-M4: Security Milestone Release

Lift 3.0-M3 was found to be vulnerable to XML External Entity attacks,
which can leak private files through your application when parsing certain
types of XML.

Lift 3.0-M4 introduces net.liftweb.util.Helpers.secureXML, an analogous object to
Scala's scala.xml.XML that is secured against XXE attacks by disabling external
entities in doctypes. If you are parsing untrusted user-provided XML using
scala.xml.XML, it is recommended that you switch to secureXML instead.

Lift 3.0-M4 was rapidly superseded by Lift 3.0-M4-1, which secures the secureXML
object against a few additional XML-based attacks.