Releases: ptarmiganlabs/butler-spyglass
butler-spyglass: v2.1.0
butler-spyglass: v2.0.4
2.0.4 (2023-03-12)
Build pipeline
- Failing Docker image build (ff8d030)
🛡 VirusTotal GitHub Action analysis:
butler-spyglass: v2.0.3
butler-spyglass: v2.0.2
2.0.2 (2023-03-12)
Documentation
Build pipeline
- Disable MQTT messages from Docker build workflow (f6b1712)
🛡 VirusTotal GitHub Action analysis:
butler-spyglass: v2.0.1
butler-spyglass: v2.0.0
2.0.0 (2023-03-10)
⚠ BREAKING CHANGES
- New config file structure. Not backwards compatible!
Features
- Create pre-built binaries for Windows, macOS and Linux (2be9842), closes #69
- Improved bug reporting & feature suggestion (1122f27)
Bug Fixes
- New config file structure. Not backwards compatible! (9885d68)
- Update package.json & package-lock.json to reduce vulnerabilities (40845c7)
Refactoring
- Various code cleanup (5c1901b)
Documentation
Miscellaneous
- deps: update docker/build-push-action action to v4 (8b41abf)
- Update code linting setup (ab931f0)
- Update dependencies to stay safe and secure.
Build pipeline
Not too many parallel tasks, please
- Fix bug that in some cases caused too many parallel requests to the Sense server. Now at most concurrentTasks extracts are done in parallel.
Adding the missing pieces
This version adds a few features and improvements that make Butler Spyglass more useful in large Qlik Sense environments. The most notable ones are:
- Added configurable extract length (max number of characters) for each row of lineage data.
- Tweaked log messages, making info-level logging more relevant.
- Add possibility to enable/disable scheduled extraction runs. Opens up for using external schedulers (e.g. cron).
- Add configurable logging to disk. If turned on, the log entries shown on the console are also written to disk.
Full documentation in the repository's readme file.
More is more - parallel extraction of app metadata
Version 1.0 was great, but lacked in terms of performance when applied to very large Qlik Sense environments.
It would take hours and hours to extract metadata for thousands of apps.
Time for version 1.1 thus.
In this version there is a new config option that lets you control how many concurrent extractions should be allowed - this pretty much cuts extraction time linearly. I.e. 5 concurrent extracts will reduce the total extraction time with a factor of 5. Nice!
Disclaimer: If you run Butler Spyglass against a small Qlik Sense server your might need to dial down the concurrency to less than 5. On a large server you can probably use a value higher than 5. You will need to test what works in your environment.
Qlik Sense app metadata FTW!
Long time coming, this idea has brewed for a long time.
With this first release of Butler Spyglass it becomes very easy to extract both data lineage and
load scripts for all apps in a Qlik Sense Enterprise environment.
In large, complex Qlik Sense environments (hundreds or even thousands of apps!) it is impossible to manually keep track of which apps use what data sources - automated data lineage generation is a must.
The same thing goes for app load scripts. By automatically extracting them daily, it is then easy to have a scheduled task zip them into an archive that can be saved for future reference. A kind of backup thus, but way easier to use than traditional disk backups. It is also possible to do various analysis of the script files, such as counting lines of code, assessing whether coding standards are followed etc.
The thing missing in this first version are front-end Sense apps for doing the actual analysis.
The data lineage CSV file should be loaded into a Sense app, to enable later analysis of the lineage data.
Such app(s) are left as an exercise for you to create.. Feel free to contribute good analysis apps back to the main project though, pull requests are encouraged and appreciated!