Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Rework analysis #347

Merged
merged 68 commits into from
May 26, 2019
Merged

Rework analysis #347

merged 68 commits into from
May 26, 2019

Conversation

kengz
Copy link
Owner

@kengz kengz commented May 26, 2019 •
edited
Loading

Rework analysis

  • Include new metrics: strength, sample efficiency, training_efficiency, stability, consistency
  • rewrite analysis, retro_analysis, viz modules with major simplification
  • use nested folder to organizing data
  • separate eval loop to analysis.gen_avg_return

Refactor and improvements

  • rename total_t to frame, standardize clock to always use frame as main tick, and remove extraneous settings
  • refactor control.py, search.py
  • remove evolutionary search deap to prepare for future ray.tune update
  • various small performance optimization
  • make unit tests more robust for testing specs; ensure it runs through analysis
  • update _random_baselines.json to be more comprehensive
  • rename training_epoch -> training_iter and training_batch_epoch -> training_batch_iter for accurate semantics.
  • propagate changes to specs

kengz added 30 commits May 21, 2019 20:04
@kengz
remove is_unfit
4bb8a54
@kengz
implement gen_avg_return
b294cf6
@kengz
add calc metrics sub methods
ae99462
@kengz
update get_random_baseline method
33b9127
@kengz
add calc_session_metrics
3d59ca2
@kengz
add calc_trial_metrics method
b00d1c2
@kengz
move save_spec method to spec_util.save
a505e9e
@kengz
cleanup data variables in control
37b06d2
@kengz
rename metrics vars
bb2ecb1
@kengz
update session analysis
d14a92e
@kengz
rename session output var
b898625
@kengz
fix plot session
c9db86b
@kengz
csv write no index
058c05a
@kengz
rename vars for clarity; carry time unit series
01dedb7
@kengz
pass series list in trial; add calc methods in util and viz
0f56dae
@kengz
add prepath to spec and use it automatically
546385c
@kengz
add plot_trial
6910a3e
@kengz
finish session and trial analysis
9f0edda
@kengz
avoid ckpt at init
c405b1d
@kengz
cleanup analysis
9be9985
@kengz
vastly simplify session and trial plots
2909e8e
@kengz
add pickle read write
f0c3b64
@kengz
simplify viz
24c80f3
@kengz
rename session graphs to group
d347d4b
@kengz
move plotters to viz
f9f42e8
@kengz
assert cols, remove min stre
0ad8b44
@kengz
remove plotly render for non notebook
f64100d
@kengz
make plot square
d62e479
@kengz
refactor analyze experiment
9da4e51
@kengz
add final strength to metric
5b9d1c6
kengz added 28 commits May 25, 2019 11:52
@kengz
propagate max_frame to spec
db9be6d
@kengz
fix read pickle typo
46a4c04
@kengz
reorder log
11b8b95
@kengz
guard ckpt against 0 frame or opt_step
3102db3
@kengz
guard session metrics save with df_mode
41149e6
@kengz
fix cond typo
e3c4e6a
@kengz
tune demo spec
f762dd7
@kengz
save with .pt extension
9b4dfce
@kengz
avoid trial analysis on eval run
5c8d8f0
@kengz
test spec
256c722
@kengz
refactor ckpt
182339c
@kengz
use tuple for most_recent
4b2c526
@kengz
fix unit tests
2ff1147
@kengz
make analysis methods args explicit
86eee66
@kengz
write trial_data_dict
5f029ac
@kengz
update retro_analysis module
4d5393d
@kengz
remove evolutionary search
8f277c0
@kengz
fix missing session_df save
7ac693e
@kengz
simplify and refactor search
dff48c0
@kengz
standardize to use parallelize session
6a73f60
@kengz
cleanup
a80814f
@kengz
correct epoch terminology to iter
ae43db8
@kengz
restore log session metrics
3049680
@kengz
add insert_folder, add prepath method tests
8f14bfa
@kengz
split save into folders
8f75e52
@kengz
rename data subfolder to info to prevent conflict
60d08ca
@kengz
add no_grad to analysis eval loop
14e4e52
@kengz
guard retro analysis parallel
5c825b1
@kengz kengz merged commit cb2258f into v4-dev May 26, 2019
@kengz kengz deleted the ret branch May 26, 2019 03:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@kengz