Releases: dimajix/flowman
Releases · dimajix/flowman
0.21.1
0.21.0
0.20.1
0.20.0
- Fix detection of Derby metastore to truncate comment lengths.
- Add new config variable
flowman.default.relation.input.columnMismatchPolicy
(default isIGNORE
) - Add new config variable
flowman.default.relation.input.typeMismatchPolicy
(default isIGNORE
) - Add new config variable
flowman.default.relation.output.columnMismatchPolicy
(default isADD_REMOVE_COLUMNS
) - Add new config variable
flowman.default.relation.output.typeMismatchPolicy
(default isCAST_ALWAYS
) - Improve handling of
_SUCCESS
files for detecting (non-)dirty directories - Implement new
merge
target - Implement merge operation for Delta relations
- Implement merge operation for JDBC relations (only for some databases, i.e. MS SQL)
- Add new config variable
flowman.execution.target.useHistory
(default isfalse
) - Change the semantics of config variable
flowman.execution.target.forceDirty
(default isfalse
) - Add new
-d
/--dirty
option for explicitly marking individual targets as dirty
0.19.0
- Add build profile for Hadoop 3.3
- Add build profile for Spark 3.2
- Allow SQL expressions as dimensions in
aggregate
mapping - Update Hive views when the resulting schema would change
- Add new
mapping cache
command to FlowShell - Support embedded connection definitions
- Much improved Flowman History Server
- Fix wrong metric names with TemplateTarget
- Implement more
template
types forconnection
,schema
,dataset
,assertion
andmeasure
- Implement new
measure
target for creating custom metrics for measuring data quality - Add new config option
flowman.execution.mapping.parallelism
0.18.0
- Improve automatic schema migration for Hive and JDBC relations
- Improve support of CHAR(n) and VARCHAR(n) types. Those types will now be propagates to Hive with newer Spark versions
- Support writing to dynamic partitions for file relations, Hive tables, JDBC relations and Delta tables
- Fix the name of some config variables (floman.* => flowman.*)
- Added new config variables
flowman.default.relation.migrationPolicy
andflowman.default.relation.migrationStrategy
- Add plugin for supporting DeltaLake (https://delta.io), which provides
deltaTable
anddeltaFile
relation types - Fix non-deterministic column order in
schema
mapping,values
mapping andvalues
relation - Mark Hive dependencies has 'provided', which reduces the size of dist packages
- Significantly reduce size of AWS dependencies in AWS plugin
- Add new build profile for Cloudera CDP-7.1
- Improve Spark configuration of
LocalSparkSession
andTestRunner
- Update Spark 3.0 build profile to Spark 3.0.3
- Upgrade Impala JDBC driver from 2.6.17.1020 to 2.6.23.1028
- Upgrade MySQL JDBC driver from 8.0.20 to 8.0.25
- Upgrade MariaDB JDBC driver from 2.2.4 to 2.7.3
- Upgrade several Maven plugins to latest versions
- Add new config option
flowman.workaround.analyze_partition
to workaround CDP 7.1 issues - Fix migrating Hive views to tables and vice-versa
- Add new option "-j " to allow running multiple job instances in parallel
- Add new option "-j " to allow running multiple tests in parallel
- Add new
uniqueKey
assertion - Add new
schema
assertion - Update Swagger libraries for
swagger
schema - Implement new
openapi
plugin to support OpenAPI 3.0 schemas - Add new
readHive
mapping - Add new
simpleReport
andreport
hook - Implement new templates
0.17.1
0.17.0
- New Flowman Kernel and Flowman Studio application prototypes
- New ParallelExecutor
- Fix before/after dependencies in
count
target - Default build is now Spark 3.1 + Hadoop 3.2
- Remove build profiles for Spark 2.3 and CDH 5.15
- Add MS SQL Server plugin containing JDBC driver
- Speed up file listing for
file
relations - Use Spark JobGroups
- Better support running Flowman on Windows with appropriate batch scripts
0.16.0
0.15.0
- New configuration variable
floman.default.target.rebalance
- New configuration variable
floman.default.target.parallelism
- Changed behaviour: The
mergeFile
target now does not assume any more that thetarget
is local. If you already
usemergeFiles
with a local file, you need to prefix the target file name withfile://
. - Add new
-t
argument for selectively building a subset of targets - Remove example-plugin
- Add quickstart guide
- Add new "flowman-parent" BOM for projects using Flowman
- Move
com.dimajix.flowman.annotations
package tocom.dimajix.flowman.spec.annotations
- Add new log redaction
- Integrate Scala scode coverage analysis
assemble
will fail when trying to use non-existing columns- Move
swagger
andjson
schema support into separate plugins - Change default build to Spark 3.0 and Hadoop 3.2
- Update Spark to 3.0.2
- Rename class
Executor
toExecution
- watch your plugins! - Implement new configurable
Executor
class for executing build targets. - Add build profile for Spark 3.1.x
- Update ScalaTest to 3.2.5 - watch your unittests for changed ScalaTest API!
- Add new
case
mapping - Add new
--dry-run
command line option - Add new
mock
andnull
mapping types - Add new
mock
relation - Add new
values
mapping - Add new
values
dataset - Implement new testing capabilities
- Rename
update
mapping toupsert
mapping, which better describes its functionality - Introduce new
VALIDATE
phase, which is executed even beforeCREATE
phase - Implement new
validate
andverify
targets - Implement new
deptree
command in Flowman shell