Model improvements6

[detlefarend]

Description

Background

Types of changes

• Bug fix
• New Example, please read the detailed guideline here
• New Feature
• Documentation

New Feature

• Model (Improvement on the Core Model)
• Environment (New Environment), please read the detailed guideline here
• Policy (New Policy), please read the detailed guideline here
• .....

Checklist:

• Read the CONTRIBUTION guide.
• Update the history on the source code (required).
• This change requires a change to the documentation.
• Update the tests accordingly.
• Update the documentation accordingly.

[detlefarend]

Hi Rizky and William, classes Training and RLTraining are reworked and enhanced as described in #161 and demonstrated in new Howto 17. Please take a look. Thx!

[rizkydiprasetya]

@detlefarend could you please check there are still some errors.

[detlefarend]

Hi @rizkydiprasetya, I guess the problem is caused by this code

rew is not a scalar but a np array. The method Reward.set_overall_reward() in turn expects a scalar.

Then a parameter of class RLTraining changed it's name. I already fixed and pushed it:

[rizkydiprasetya]

why should be a scalar? it used to work with np array

[detlefarend]

why should be a scalar? it used to work with np array

A reward is a scalar value, not an array. That's why both methods Reward.set_overall_reward() and Reward.add_agent_reward() expect scalars.

Fyi: I edited my last comment (parameter of class RLTraining)

[rizkydiprasetya]

ok, and why is it working without this stagnation and evaluation and other stuff that you added?

[detlefarend]

ok, and why is it working without this stagnation and evaluation and other stuff that you added?

It is a simple thing: you use the method beyond its specification. If you do so the behavior is undefined. The new evaluation stuff expects scalar values and gets arrays. We can surely improve both methods by adding try/except logic. But it's not a bug. Please adjust your code or add the logic to Reward by yourself.

[rizkydiprasetya]

ok, then we need to change all of the environment. Since all of them based on numpy. You can check them.

[rizkydiprasetya]

ok, and why is it working without this stagnation and evaluation and other stuff that you added?

It is a simple thing: you use the method beyond its specification. If you do so the behavior is undefined. The new evaluation stuff expects scalar values and gets arrays. We can surely improve both methods by adding try/except logic. But it's not a bug. Please adjust your code or add the logic to Reward by yourself.

its not that beyond. Just about either using NumPy or not.

[detlefarend]

I have no problem to solve this in Reward - except of time. Feel free to do it if you prefer this solution.

[rizkydiprasetya]

done

[rizkydiprasetya]

@detlefarend why?

[detlefarend]

@detlefarend why?

What is the problem? Give me a chance to understand.

[rizkydiprasetya]

@detlefarend why?

What is the problem? Give me a chance to understand.

Why are you using try and except, but there is no error that you want to catch?

[detlefarend]

Because you can use an env as one (or all) of the afcts in an envmodel. But an env is not adaptive and raises an exception, if you try to switch the adaptivity. See EnvBase.switch_adaptivity().

[rizkydiprasetya]

I see, ok. It is fine like that, but I would prefer to check for the inheritance of the object for the adaptivity.

Oh just check again, EnvBase inherits also the Model class. That is why you put the raise on the function.

[detlefarend]

What is your problem? Does something behave different from your expectation?