Merge branch 'develop'

nbbrd · Nov 23, 2021 · 2d96ced · 2d96ced
2 parents e44a811 + 2bfea5c
commit 2d96ced
Show file tree

Hide file tree

Showing 243 changed files with 5,772 additions and 2,720 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,8 +1,9 @@
 # Changelog
+
 All notable changes to this project will be documented in this file.
 
-The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
-and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres
+to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 ## [Unreleased]
 
@@ -12,29 +13,81 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Fixed
 
+## [3.0.0-beta.5] - 2021-11-23
+
+This is the fifth beta release of **sdmx-dl**.   
+sdmx-dl follows [semantic versioning](http://semver.org/).
+
+_Note that sdmx-dl is still in heavy development and might change a lot between versions, so you shouldn't use it in
+production._
+
+This release adds the support of most SDMX time formats.
+The execution time of CLI is reduced greatly by using a smarter cache and by avoiding to load unnecessary resources.
+A few sources are also added, including a random data generator that can be used for tests/demos.
+
+### Added
+
+- Add support of codelist in API [#158](https://github.com/nbbrd/sdmx-dl/issues/158)
+- Add native driver for INSEE [#157](https://github.com/nbbrd/sdmx-dl/issues/157)
+- Add native driver for Eurostat [#155](https://github.com/nbbrd/sdmx-dl/issues/155)
+- Add second layer of cache in CLI
+- Add availability check on driver SPI [#162](https://github.com/nbbrd/sdmx-dl/issues/162)
+- Add optional RNG driver [#163](https://github.com/nbbrd/sdmx-dl/issues/163)
+- Add cache-folder-path option in CLI
+- Add source from Statistics Canada [#165](https://github.com/nbbrd/sdmx-dl/issues/165)
+- Add support of non-calendar reporting periods
+
+### Changed
+
+- Refactor network API to avoid unnecessary resource loading
+- Invert default activation of auto-proxy option in CLI
+- Simplify name of auto-proxy option in CLI
+- Replace multiple parameters with `DataRef` in API 
+- Change `SdmxWebSource#endpoint` type to URI
+- Change `SdmxWebSource#monitor` type to URI
+- Change cache default folder to `java.io.tmpdir/sdmx-dl/VERSION`
+- Refactor parsing of SDMX time formats
+
+### Fixed
+
+- Improve error reporting in CLI
+- Restore setup-completion command
+- Fix setup commands names
+- Fix datetime parsing of undefined freq
+- Fix dimension ordering when used in key
+- Fix external references in flows
+- Migrate ABS source to the new server [#168](https://github.com/nbbrd/sdmx-dl/issues/168)
+
 ## [3.0.0-beta.4] - 2021-10-21
 
 This is the fourth beta release of **sdmx-dl**.   
 sdmx-dl follows [semantic versioning](http://semver.org/).
 
-_Note that sdmx-dl is still in heavy development and might change a lot between versions, so you shouldn't use it in production._
+_Note that sdmx-dl is still in heavy development and might change a lot between versions, so you shouldn't use it in
+production._
 
-This release simplifies the CLI options and fixes web monitors by using a [new self-hosted provider](https://nbbrd.github.io/sdmx-upptime/).  
-The CLI binary is now also available in a [JBang catalog](https://github.com/nbbrd/jbang-catalog) and in a [snapshot repository](https://s01.oss.sonatype.org/content/repositories/snapshots/com/github/nbbrd/sdmx-dl/sdmx-dl-cli/).
+This release simplifies the CLI options and fixes web monitors by using
+a [new self-hosted provider](https://nbbrd.github.io/sdmx-upptime/).  
+The CLI binary is now also available in a [JBang catalog](https://github.com/nbbrd/jbang-catalog) and in
+a [snapshot repository](https://s01.oss.sonatype.org/content/repositories/snapshots/com/github/nbbrd/sdmx-dl/sdmx-dl-cli/)
+.
 
 ### Added
+
 - Add a link to documentation in CLI [#137](https://github.com/nbbrd/sdmx-dl/issues/137)
 - Add support of average response time in monitors [#144](https://github.com/nbbrd/sdmx-dl/issues/144)
 - Add Maven deployment of CLI
 - Add distribution to JBang [#141](https://github.com/nbbrd/sdmx-dl/issues/141)
 
 ### Changed
+
 - Rename command check.properties as check.config [#136](https://github.com/nbbrd/sdmx-dl/issues/136)
 - Migrate web monitors to Upptime [#142](https://github.com/nbbrd/sdmx-dl/issues/142)
 - Refactor cache to handle web monitors
 - Simplify CSV command options [#88](https://github.com/nbbrd/sdmx-dl/issues/88)
 
 ### Fixed
+
 - Fix missing content in subsequent calls to fetch-meta command [#154](https://github.com/nbbrd/sdmx-dl/issues/154)
 - Fix detection of console encoding
 
@@ -43,19 +96,22 @@ The CLI binary is now also available in a [JBang catalog](https://github.com/nbb
 This is the third beta release of **sdmx-dl**.   
 sdmx-dl follows [semantic versioning](http://semver.org/).
 
-_Note that sdmx-dl is still in heavy development and might change a lot between versions, so you shouldn't use it in production._
+_Note that sdmx-dl is still in heavy development and might change a lot between versions, so you shouldn't use it in
+production._
 
 This release adds new sources and improves feedback on request parameters such as Key and Flow.  
 The CLI binary is now available in some popular package managers.  
-Finally, sdmx-dl advertises itself on web queries with the user-agent `sdmx-dl/3.0.0-beta.3`. 
+Finally, sdmx-dl advertises itself on web queries with the user-agent `sdmx-dl/3.0.0-beta.3`.
 
 ### Added
+
 - Add support of multiple media-types in HTTP request header (RI)
 - Add support of media-type charset in HTTP response header (RI)
 - Add response event `#onSuccess(MediaType)` (RI)
 - Add default user-agent `sdmx-dl/version` (RI) [#117](https://github.com/nbbrd/sdmx-dl/issues/117)
 - Add source from Bundesbank [#104](https://github.com/nbbrd/sdmx-dl/issues/104)
-- Add source from Economic and Social Commission for Asia and the Pacific [#99](https://github.com/nbbrd/sdmx-dl/issues/99)
+- Add source from Economic and Social Commission for Asia and the
+  Pacific [#99](https://github.com/nbbrd/sdmx-dl/issues/99)
 - Add source from National Statistical Institute of Cambodia [#98](https://github.com/nbbrd/sdmx-dl/issues/98)
 - Add source from SDMX Global Registry [#97](https://github.com/nbbrd/sdmx-dl/issues/97)
 - Add source from Pacific Data Hub [#96](https://github.com/nbbrd/sdmx-dl/issues/96)
@@ -64,12 +120,14 @@ Finally, sdmx-dl advertises itself on web queries with the user-agent `sdmx-dl/3
 - Add distribution to Homebrew (macOS & Linux) [#84](https://github.com/nbbrd/sdmx-dl/issues/84)
 
 ### Changed
+
 - Enforce https on ABS source [#108](https://github.com/nbbrd/sdmx-dl/issues/108)
 - Improve CLI version option [#79](https://github.com/nbbrd/sdmx-dl/issues/79)
 - Refactor SdmxWebAuthenticator as an SPI
 - Improve feedback on missing data flow [#123](https://github.com/nbbrd/sdmx-dl/issues/123)
 
 ### Fixed
+
 - Fix parsing of blank labels
 - Update ILO source with new endpoint [#107](https://github.com/nbbrd/sdmx-dl/issues/107)
 - Fix key parsing when time dimension is not last in data structure [#110](https://github.com/nbbrd/sdmx-dl/issues/110)
@@ -81,12 +139,15 @@ Finally, sdmx-dl advertises itself on web queries with the user-agent `sdmx-dl/3
 This is the second beta release of **sdmx-dl**.   
 sdmx-dl follows [semantic versioning](http://semver.org/).
 
-_Note that sdmx-dl is still in heavy development and might change a lot between versions, so you shouldn't use it in production._
+_Note that sdmx-dl is still in heavy development and might change a lot between versions, so you shouldn't use it in
+production._
 
 This release adds new sources, the support of attributes and modify the CLI commands.  
-These command modifications are quite extended and concern [command names, overall structure and output](https://github.com/nbbrd/sdmx-dl/wiki/cli-usage).
+These command modifications are quite extended and
+concern [command names, overall structure and output](https://github.com/nbbrd/sdmx-dl/wiki/cli-usage).
 
 ### Added
+
 - Add FileDriver to allow demo data
 - Add initial support of native image
 - Add file reading feedback
@@ -100,19 +161,23 @@ These command modifications are quite extended and concern [command names, overa
 - Add [basic documentation](https://github.com/nbbrd/sdmx-dl/wiki/)
 
 ### Changed
+
 - Modify CLI [command name, structure and output](https://github.com/nbbrd/sdmx-dl/wiki/cli-usage)
 
 ## [3.0.0-beta.1] - 2020-09-25
 
 This is the first beta release of **sdmx-dl**.   
 sdmx-dl follows [semantic versioning](http://semver.org/).
 
-_Note that sdmx-dl is still in heavy development and might change a lot between versions, so you shouldn't use it in production._
+_Note that sdmx-dl is still in heavy development and might change a lot between versions, so you shouldn't use it in
+production._
 
 ### Added
+
 - Initial release
 
-[Unreleased]: https://github.com/nbbrd/sdmx-dl/compare/v3.0.0-beta.4...HEAD
+[Unreleased]: https://github.com/nbbrd/sdmx-dl/compare/v3.0.0-beta.5...HEAD
+[3.0.0-beta.5]: https://github.com/nbbrd/sdmx-dl/compare/v3.0.0-beta.4...v3.0.0-beta.5
 [3.0.0-beta.4]: https://github.com/nbbrd/sdmx-dl/compare/v3.0.0-beta.3...v3.0.0-beta.4
 [3.0.0-beta.3]: https://github.com/nbbrd/sdmx-dl/compare/v3.0.0-beta.2...v3.0.0-beta.3
 [3.0.0-beta.2]: https://github.com/nbbrd/sdmx-dl/compare/v3.0.0-beta.1...v3.0.0-beta.2

diff --git a/pom.xml b/pom.xml
@@ -5,7 +5,7 @@
 
     <groupId>com.github.nbbrd.sdmx-dl</groupId>
     <artifactId>sdmx-dl-parent</artifactId>
-    <version>3.0.0-beta.4</version>
+    <version>3.0.0-beta.5</version>
     <packaging>pom</packaging>
 
     <name>sdmx-dl</name>
@@ -46,7 +46,7 @@
             <dependency>
                 <groupId>org.checkerframework</groupId>
                 <artifactId>checker-qual</artifactId>
-                <version>3.18.1</version>
+                <version>3.19.0</version>
             </dependency>
 
             <!-- test libraries -->
@@ -65,7 +65,7 @@
             <dependency>
                 <groupId>io.github.hakky54</groupId>
                 <artifactId>logcaptor</artifactId>
-                <version>2.7.0</version>
+                <version>2.7.2</version>
                 <exclusions>
                     <exclusion>
                         <groupId>org.slf4j</groupId>
@@ -166,7 +166,7 @@
                 <lombok.version>1.18.22</lombok.version>
                 <java-service.version>1.4.0</java-service.version>
                 <java-design.version>1.1.3</java-design.version>
-                <picocli.version>4.6.1</picocli.version>
+                <picocli.version>4.6.2</picocli.version>
             </properties>
             <dependencyManagement>
                 <dependencies>

diff --git a/sdmx-dl-api/pom.xml b/sdmx-dl-api/pom.xml
@@ -6,7 +6,7 @@
     <parent>
         <groupId>com.github.nbbrd.sdmx-dl</groupId>
         <artifactId>sdmx-dl-parent</artifactId>
-        <version>3.0.0-beta.4</version>
+        <version>3.0.0-beta.5</version>
     </parent>
 
     <artifactId>sdmx-dl-api</artifactId>

diff --git a/sdmx-dl-api/src/main/java/internal/sdmxdl/web/spi/FailsafeSdmxWebConnection.java b/sdmx-dl-api/src/main/java/internal/sdmxdl/web/spi/FailsafeSdmxWebConnection.java
@@ -151,15 +151,13 @@ public DataStructure getStructure(DataflowRef flowRef) throws IOException {
     }
 
     @Override
-    public Collection<Series> getData(DataflowRef flowRef, Key key, DataFilter filter) throws IOException {
-        Objects.requireNonNull(flowRef);
-        Objects.requireNonNull(key);
-        Objects.requireNonNull(filter);
+    public Collection<Series> getData(DataRef dataRef) throws IOException {
+        Objects.requireNonNull(dataRef);
 
         Collection<Series> result;
 
         try {
-            result = delegate.getData(flowRef, key, filter);
+            result = delegate.getData(dataRef);
         } catch (RuntimeException ex) {
             throw unexpectedError("Unexpected exception while getting data", ex);
         }
@@ -172,15 +170,13 @@ public Collection<Series> getData(DataflowRef flowRef, Key key, DataFilter filte
     }
 
     @Override
-    public Stream<Series> getDataStream(DataflowRef flowRef, Key key, DataFilter filter) throws IOException {
-        Objects.requireNonNull(flowRef);
-        Objects.requireNonNull(key);
-        Objects.requireNonNull(filter);
+    public Stream<Series> getDataStream(DataRef dataRef) throws IOException {
+        Objects.requireNonNull(dataRef);
 
         Stream<Series> result;
 
         try {
-            result = delegate.getDataStream(flowRef, key, filter);
+            result = delegate.getDataStream(dataRef);
         } catch (RuntimeException ex) {
             throw unexpectedError("Unexpected exception while getting data stream", ex);
         }
@@ -193,15 +189,13 @@ public Stream<Series> getDataStream(DataflowRef flowRef, Key key, DataFilter fil
     }
 
     @Override
-    public DataCursor getDataCursor(DataflowRef flowRef, Key key, DataFilter filter) throws IOException {
-        Objects.requireNonNull(flowRef);
-        Objects.requireNonNull(key);
-        Objects.requireNonNull(filter);
+    public DataCursor getDataCursor(DataRef dataRef) throws IOException {
+        Objects.requireNonNull(dataRef);
 
         DataCursor result;
 
         try {
-            result = delegate.getDataCursor(flowRef, key, filter);
+            result = delegate.getDataCursor(dataRef);
         } catch (RuntimeException ex) {
             throw unexpectedError("Unexpected exception while getting data cursor", ex);
         }

diff --git a/sdmx-dl-api/src/main/java/internal/sdmxdl/web/spi/FailsafeSdmxWebDriver.java b/sdmx-dl-api/src/main/java/internal/sdmxdl/web/spi/FailsafeSdmxWebDriver.java
@@ -1,17 +1,17 @@
 /*
  * Copyright 2019 National Bank of Belgium
- * 
- * Licensed under the EUPL, Version 1.1 or - as soon they will be approved 
+ *
+ * Licensed under the EUPL, Version 1.1 or - as soon they will be approved
  * by the European Commission - subsequent versions of the EUPL (the "Licence");
  * You may not use this work except in compliance with the Licence.
  * You may obtain a copy of the Licence at:
- * 
+ *
  * http://ec.europa.eu/idabc/eupl
- * 
- * Unless required by applicable law or agreed to in writing, software 
+ *
+ * Unless required by applicable law or agreed to in writing, software
  * distributed under the Licence is distributed on an "AS IS" basis,
  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the Licence for the specific language governing permissions and 
+ * See the Licence for the specific language governing permissions and
  * limitations under the Licence.
  */
 package internal.sdmxdl.web.spi;
@@ -31,7 +31,6 @@
 import java.util.logging.Level;
 
 /**
- *
  * @author Philippe Charles
  */
 @lombok.extern.java.Log
@@ -42,8 +41,8 @@ public static SdmxWebDriver wrap(SdmxWebDriver obj) {
         return obj instanceof FailsafeSdmxWebDriver
                 ? obj
                 : new FailsafeSdmxWebDriver(obj,
-                        FailsafeSdmxWebDriver::logUnexpectedError,
-                        FailsafeSdmxWebDriver::logUnexpectedNull);
+                FailsafeSdmxWebDriver::logUnexpectedError,
+                FailsafeSdmxWebDriver::logUnexpectedNull);
     }
 
     @lombok.NonNull
@@ -84,6 +83,16 @@ public int getRank() {
         }
     }
 
+    @Override
+    public boolean isAvailable() {
+        try {
+            return delegate.isAvailable();
+        } catch (RuntimeException ex) {
+            onUnexpectedError.accept("Unexpected exception while getting availability", ex);
+            return false;
+        }
+    }
+
     @Override
     public SdmxWebConnection connect(SdmxWebSource source, SdmxWebContext context) throws IOException, IllegalArgumentException {
         Objects.requireNonNull(source);

diff --git a/sdmx-dl-api/src/main/java/sdmxdl/Attribute.java b/sdmx-dl-api/src/main/java/sdmxdl/Attribute.java
@@ -16,6 +16,8 @@
  */
 package sdmxdl;
 
+import org.checkerframework.checker.nullness.qual.Nullable;
+
 import java.util.Map;
 
 /**
@@ -31,13 +33,12 @@ public class Attribute extends Component {
     @lombok.NonNull
     String id;
 
-    @lombok.NonNull
-    @lombok.Singular
-    Map<String, String> codes;
-
     @lombok.NonNull
     String label;
 
+    @Nullable
+    Codelist codelist;
+
 //    @lombok.NonNull
 //    RelationShip relationShip;
 //

diff --git a/sdmx-dl-api/src/main/java/sdmxdl/Codelist.java b/sdmx-dl-api/src/main/java/sdmxdl/Codelist.java
@@ -0,0 +1,21 @@
+package sdmxdl;
+
+import java.util.Map;
+
+@lombok.Value
+@lombok.Builder(toBuilder = true)
+@lombok.EqualsAndHashCode(callSuper = false)
+public class Codelist extends Resource<CodelistRef> {
+
+    @lombok.NonNull
+    CodelistRef ref;
+
+    /**
+     * Non-null map of code description by code id that represents a codelist
+     * (predefined sets of terms from which some statistical coded concepts take
+     * their values).
+     */
+    @lombok.NonNull
+    @lombok.Singular
+    Map<String, String> codes;
+}