Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hilbert YAML configuration parser and Hilbert Server/Client-side tools #34

Merged
merged 45 commits into from
Dec 9, 2016
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
8685e9a
Initial version of General Configuration Validation
Nov 7, 2016
7360603
Fixing first issues and extending
Nov 7, 2016
711b100
Added `supported_types` field to `Profile`
Nov 7, 2016
f9559e6
Improved detection of parsed YAML data types, e.g. in GroupList, Base…
Nov 8, 2016
2b5c8ed
Updated the ConfigurationDD due to feedback
Nov 8, 2016
07f6af3
Accamulated update of config/hilbert-cli-config.py + infrastructure
Nov 15, 2016
0e2cad6
Added another unittest for checking warnings due to shaddowed mapping…
Nov 15, 2016
a6e0f04
Renamed `name` -> `ref` in `DockerComposeService` to avoid conflict w…
Nov 15, 2016
3b778ee
Added complete configuration with docker-compose resources
Nov 15, 2016
e23485a
`Station::client_settings` is optional due to standalone stations
Nov 15, 2016
ab3922d
Added unit test for parsing the result
Nov 16, 2016
9bf3cf5
Minor fix
Nov 16, 2016
23a582d
Update Configuration model due to the meeting on 15.11.2016
Nov 16, 2016
d3d52c1
Starting to separate hilbert-cli-config.py into smaller units:
Nov 18, 2016
92b6179
Separated the main Hilbert tool from the library: /tools/hilbert.py
Nov 18, 2016
850ec47
Added `cfg_show [-o <obj>]` subcommand to display (a part of) the loa…
Nov 18, 2016
1542e87
Add more API to Validators (and keep them) (e.g. Hilbert)
Nov 19, 2016
39cb7e2
Keep Validators to handle configuration objects
Nov 20, 2016
e570cf5
Added generic Quering and Data-Dumping to Validators + corresponding …
Nov 21, 2016
a385fa7
Addded stubs of required sub-commands (for the Dashboard) to
Nov 21, 2016
710e24a
Repair `extends` mechnismus for stations
Nov 21, 2016
44bbadc
Updated General System Design Diagram
Nov 21, 2016
0134132
Fixing hilbert CLI
Nov 22, 2016
1a84b6f
Notes about missing things
Nov 22, 2016
c5409aa
Minor data cleanup
Nov 22, 2016
384b384
Fixed hilbert's CLI according to review comments:
Nov 24, 2016
9d4eba0
Moved tests to the top level + adaptation for searching for python li…
Nov 25, 2016
847647b
Added Config API ID to Validator objects
Nov 25, 2016
f73ea76
Enable using **previously checked** dump to check _compatible__ parsi…
Nov 25, 2016
51ade4c
Big internal changes to Validators to improve API + corrected tests d…
Nov 26, 2016
d5934ef
Added Requirements + removed some unnecessary imports
Nov 26, 2016
056ab1e
Looking for a minimal CLI: no app_start/app_stop ATM
Nov 26, 2016
a3b3800
Better automated doc generation + source verification (pep8/pylint)
Nov 26, 2016
8d43978
Added simple initial setup.py and tox.ini
Nov 27, 2016
fd0fefb
Initial configuration for testing via TravisCI
Nov 27, 2016
31c32f5
More .gitignore + fix a few minor warnings due to PEP8
Nov 27, 2016
78a5169
Added remote execution via ssh/scp + Dummy client script
Dec 1, 2016
fd3fcaf
Merge pull request #39 from hilbert/feature/hilbert_client_usage
malex984 Dec 1, 2016
fc5a172
Added dummy CLI to the client side tool
Dec 1, 2016
7131db8
Improved hilbert tool (Server-side)
Dec 6, 2016
7ddb638
Client-side hilbert-station tool
Dec 6, 2016
e7e8593
Merge pull request #40 from hilbert/feature/hilbert_client_scripts
malex984 Dec 6, 2016
b9d2828
Updated Station part of Hilbert
Dec 7, 2016
a7d2407
Updates/Fixes due to small real-life testing at HITS
Dec 8, 2016
6454c5d
Merge pull request #43 from hilbert/feature/hilbert_client
malex984 Dec 9, 2016
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
config/hilbert_cli_config.py ident
tools/hilbert.py ident
tools/hilbert-station ident

7 changes: 7 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -2,3 +2,10 @@ STATIONS*/
templates/compose
bashttpd.log
lastapp.cfg
*~
*.pyc
__pycache__
.idea
.tox/
.cache/
MANIFEST
27 changes: 27 additions & 0 deletions .travis.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
language: python
sudo: false

cache:
directories:
- $HOME/.cache/pip

matrix:
include:
- python: 2.7
env: TOXENV=py27
- python: 3.4
env: TOXENV=py34

install:
- ./.travis/install.sh

script:
- source ~/.venv/bin/activate
- tox

notifications:
email:
recipients:
- malex984+travis.cli@gmail.com
on_success: never
on_failure: always
1 change: 1 addition & 0 deletions .travis/install.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
pip install --upgrade tox
240 changes: 240 additions & 0 deletions Doxyfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,240 @@
USE_MDFILE_AS_MAINPAGE =
DOXYFILE_ENCODING = UTF-8
PROJECT_NAME = Hilbert
PROJECT_NUMBER = 0.0.1
PROJECT_BRIEF = "Hilbert CLI (server part)"
PROJECT_LOGO =
OUTPUT_DIRECTORY = docs/doxy
CREATE_SUBDIRS = NO
OUTPUT_LANGUAGE = English
BRIEF_MEMBER_DESC = YES
REPEAT_BRIEF = YES
ABBREVIATE_BRIEF =
ALWAYS_DETAILED_SEC = NO
INLINE_INHERITED_MEMB = NO
FULL_PATH_NAMES = YES
STRIP_FROM_PATH =
STRIP_FROM_INC_PATH =
SHORT_NAMES = NO
JAVADOC_AUTOBRIEF = YES
QT_AUTOBRIEF = NO
MULTILINE_CPP_IS_BRIEF = YES
INHERIT_DOCS = YES
SEPARATE_MEMBER_PAGES = NO
TAB_SIZE = 4
ALIASES =
TCL_SUBST =
OPTIMIZE_OUTPUT_FOR_C = NO
OPTIMIZE_OUTPUT_JAVA = YES
OPTIMIZE_FOR_FORTRAN = NO
OPTIMIZE_OUTPUT_VHDL = NO
EXTENSION_MAPPING =
MARKDOWN_SUPPORT = YES
BUILTIN_STL_SUPPORT = NO
CPP_CLI_SUPPORT = NO
SIP_SUPPORT = NO
IDL_PROPERTY_SUPPORT = YES
DISTRIBUTE_GROUP_DOC = NO
SUBGROUPING = YES
INLINE_GROUPED_CLASSES = NO
INLINE_SIMPLE_STRUCTS = NO
TYPEDEF_HIDES_STRUCT = NO
SYMBOL_CACHE_SIZE = 0
LOOKUP_CACHE_SIZE = 0
EXTRACT_ALL = YES
EXTRACT_PRIVATE = YES
EXTRACT_PACKAGE = NO
EXTRACT_STATIC = YES
EXTRACT_LOCAL_CLASSES = NO
EXTRACT_LOCAL_METHODS = NO
EXTRACT_ANON_NSPACES = NO
HIDE_UNDOC_MEMBERS = NO
HIDE_UNDOC_CLASSES = NO
HIDE_FRIEND_COMPOUNDS = NO
HIDE_IN_BODY_DOCS = NO
INTERNAL_DOCS = NO
CASE_SENSE_NAMES = NO
HIDE_SCOPE_NAMES = NO
SHOW_INCLUDE_FILES = YES
FORCE_LOCAL_INCLUDES = NO
INLINE_INFO = YES
SORT_MEMBER_DOCS = YES
SORT_BRIEF_DOCS = NO
SORT_MEMBERS_CTORS_1ST = NO
SORT_GROUP_NAMES = NO
SORT_BY_SCOPE_NAME = NO
STRICT_PROTO_MATCHING = NO
GENERATE_TODOLIST = YES
GENERATE_TESTLIST = YES
GENERATE_BUGLIST = YES
GENERATE_DEPRECATEDLIST= YES
ENABLED_SECTIONS =
MAX_INITIALIZER_LINES = 30
SHOW_USED_FILES = YES
SHOW_FILES = YES
SHOW_NAMESPACES = YES
FILE_VERSION_FILTER =
LAYOUT_FILE =
CITE_BIB_FILES =
QUIET = NO
WARNINGS = YES
WARN_IF_UNDOCUMENTED = YES
WARN_IF_DOC_ERROR = YES
WARN_NO_PARAMDOC = NO
WARN_FORMAT = "$file:$line: $text"
WARN_LOGFILE =
INPUT = config tools
INPUT_ENCODING = UTF-8
FILE_PATTERNS = "*.py" "*.md"
RECURSIVE = YES
EXCLUDE =
EXCLUDE_SYMLINKS = NO
EXCLUDE_PATTERNS =
EXCLUDE_SYMBOLS =
EXAMPLE_PATH = scripts
EXAMPLE_PATTERNS =
EXAMPLE_RECURSIVE = NO
IMAGE_PATH =
INPUT_FILTER = doxypy
FILTER_PATTERNS =
FILTER_SOURCE_FILES = YES
FILTER_SOURCE_PATTERNS =
SOURCE_BROWSER = NO
INLINE_SOURCES = NO
STRIP_CODE_COMMENTS = YES
REFERENCED_BY_RELATION = NO
REFERENCES_RELATION = NO
REFERENCES_LINK_SOURCE = YES
USE_HTAGS = NO
VERBATIM_HEADERS = YES
ALPHABETICAL_INDEX = YES
COLS_IN_ALPHA_INDEX = 5
IGNORE_PREFIX =
GENERATE_HTML = YES
HTML_OUTPUT = html
HTML_FILE_EXTENSION = .html
HTML_HEADER =
HTML_FOOTER =
HTML_STYLESHEET =
HTML_EXTRA_FILES =
HTML_COLORSTYLE_HUE = 220
HTML_COLORSTYLE_SAT = 100
HTML_COLORSTYLE_GAMMA = 80
HTML_TIMESTAMP = YES
HTML_DYNAMIC_SECTIONS = NO
HTML_INDEX_NUM_ENTRIES = 100
GENERATE_DOCSET = NO
DOCSET_FEEDNAME = "Doxygen generated docs"
DOCSET_BUNDLE_ID = org.doxygen.Project
DOCSET_PUBLISHER_ID = org.doxygen.Publisher
DOCSET_PUBLISHER_NAME = Publisher
GENERATE_HTMLHELP = NO
CHM_FILE =
HHC_LOCATION =
GENERATE_CHI = NO
CHM_INDEX_ENCODING =
BINARY_TOC = NO
TOC_EXPAND = NO
GENERATE_QHP = NO
QCH_FILE =
QHP_NAMESPACE = org.doxygen.Project
QHP_VIRTUAL_FOLDER = doc
QHP_CUST_FILTER_NAME =
QHP_CUST_FILTER_ATTRS =
QHP_SECT_FILTER_ATTRS =
QHG_LOCATION =
GENERATE_ECLIPSEHELP = NO
ECLIPSE_DOC_ID = org.doxygen.Project
DISABLE_INDEX = NO
GENERATE_TREEVIEW = YES
ENUM_VALUES_PER_LINE = 4
TREEVIEW_WIDTH = 250
EXT_LINKS_IN_WINDOW = NO
FORMULA_FONTSIZE = 10
FORMULA_TRANSPARENT = YES
USE_MATHJAX = NO
MATHJAX_RELPATH = http://cdn.mathjax.org/mathjax/latest
MATHJAX_EXTENSIONS =
SEARCHENGINE = YES
SERVER_BASED_SEARCH = NO
GENERATE_LATEX = NO
LATEX_OUTPUT = latex
LATEX_CMD_NAME = latex
MAKEINDEX_CMD_NAME = makeindex
COMPACT_LATEX = NO
PAPER_TYPE = a4
EXTRA_PACKAGES =
LATEX_HEADER =
LATEX_FOOTER =
PDF_HYPERLINKS = YES
USE_PDFLATEX = YES
LATEX_BATCHMODE = NO
LATEX_HIDE_INDICES = NO
LATEX_SOURCE_CODE = NO
LATEX_BIB_STYLE = plain
GENERATE_RTF = NO
RTF_OUTPUT = rtf
COMPACT_RTF = NO
RTF_HYPERLINKS = NO
RTF_STYLESHEET_FILE =
RTF_EXTENSIONS_FILE =
GENERATE_MAN = NO
MAN_OUTPUT = man
MAN_EXTENSION = .3
MAN_LINKS = NO
GENERATE_XML = NO
XML_OUTPUT = xml
XML_SCHEMA =
XML_DTD =
XML_PROGRAMLISTING = YES
GENERATE_AUTOGEN_DEF = NO
GENERATE_PERLMOD = NO
PERLMOD_LATEX = NO
PERLMOD_PRETTY = YES
PERLMOD_MAKEVAR_PREFIX =
ENABLE_PREPROCESSING = YES
MACRO_EXPANSION = NO
EXPAND_ONLY_PREDEF = NO
SEARCH_INCLUDES = YES
INCLUDE_PATH =
INCLUDE_FILE_PATTERNS =
PREDEFINED =
EXPAND_AS_DEFINED =
SKIP_FUNCTION_MACROS = YES
TAGFILES =
GENERATE_TAGFILE =
ALLEXTERNALS = NO
EXTERNAL_GROUPS = YES
PERL_PATH = /usr/bin/perl
CLASS_DIAGRAMS = YES
MSCGEN_PATH =
HIDE_UNDOC_RELATIONS = NO
HAVE_DOT = YES
DOT_NUM_THREADS = 0
DOT_FONTNAME = Helvetica
DOT_FONTSIZE = 10
DOT_FONTPATH =
CLASS_GRAPH = YES
COLLABORATION_GRAPH = YES
GROUP_GRAPHS = YES
UML_LOOK = NO
UML_LIMIT_NUM_FIELDS = 10
TEMPLATE_RELATIONS = NO
INCLUDE_GRAPH = YES
INCLUDED_BY_GRAPH = YES
CALL_GRAPH = YES
CALLER_GRAPH = YES
GRAPHICAL_HIERARCHY = YES
DIRECTORY_GRAPH = YES
DOT_IMAGE_FORMAT = png
INTERACTIVE_SVG = NO
DOT_PATH = dot
DOTFILE_DIRS =
MSCFILE_DIRS =
DOT_GRAPH_MAX_NODES = 50
MAX_DOT_GRAPH_DEPTH = 0
DOT_TRANSPARENT = NO
DOT_MULTI_TARGETS = NO
GENERATE_LEGEND = YES
DOT_CLEANUP = YES

37 changes: 37 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
## ---------------------------------------------- [ Makefile for Python Checks ]

# Simple Makefile used during development to check compliance with
# pep8 and to generate documentation

SRC=config/*.py tools/*.py
NAME=hilbert

.PHONY: usage pep8 apidocs clean pylint install build all_docs

usage: # Print Targets
@grep '^[^#[:space:]].*:' Makefile

check: # Run the tests
/bin/bash -c 'py.test -v -l --tb=auto --full-trace --color=auto tests/test_*.py'
/bin/bash -c 'py.test-3 -v -l --tb=auto --full-trace --color=auto tests/test_*.py'

tox: tox.ini setup.py # Run clean testing via tox
tox

pep8: ${SRC} # Check for PEP8 compliance
pep8 --first --show-source --show-pep8 --statistics --max-line-length=100 --format=pylint ${SRC} > docs/pep8.report.txt 2>&1 || echo $?

pylint: # Analyse Source
pylint --rcfile=rcfile.pylint -f html --comment=y --files-output=y ${SRC} 2>&1 || echo $? # --full-documentation
mv pylint_*.html docs/

all_docs: apidocs epydoc pep8 pylint

apidocs: ${SRC} # Build API Documentation with doxygen
doxygen Doxyfile 2>&1 || echo $?

epydoc: ${SRC} # Build API Documentation with epydoc
epydoc --html -o docs/epydoc --inheritance=listed --show-imports --graph=all ${SRC} 2>&1 || echo $?

clean: # Clean Project
rm -rf *~ docs/doxy docs/epydoc docs/pylint_*.html docs/pep8.report.txt
6 changes: 6 additions & 0 deletions config/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
from __future__ import absolute_import
from __future__ import unicode_literals

# __version__ = '0.2.2-dev' # TODO: add git commit id?

# from hilbert_cli_config import *
24 changes: 24 additions & 0 deletions config/helpers.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
# -*- coding: utf-8 -*-
# encoding: utf-8
# coding: utf-8

from __future__ import absolute_import, print_function, unicode_literals

# from .hilbert_cli_config import * # load_yaml # Hilbert # VerboseRoundTripLoader,

###############################################################
# import pickle
# import cPickle as pickle

def pickle_dump(fn, d):
import dill as pickle # NOTE: 3-clause BSD
with open(fn, 'wb') as p:
# NOTE: Pickle the 'data' dictionary using the highest protocol available?
# pickle.HIGHEST_PROTOCOL = 4 added in Python 3.4
pickle.dump(d, p, 2) # 2nd PROTOCOL was introduced in Python 2.3.

def pickle_load(fn):
import dill as pickle # NOTE: 3-clause BSD
with open(fn, 'rb') as p:
d = pickle.load(p)
return d
Loading