[MXNET-651] MXNet Model Backwards Compatibility Checker

[piyushghai]

Description

This PR aims to check whether models trained on earlier versions of MXNet are loading fine on the latest version or the latest release candidate. It also aims to do a check for consistency in the inference results on these trained models.

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

• The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
• Changes are complete (i.e. I finished coding on this PR)
• All changes have test coverage:
  The PRs purpose itself is to add more tests for backwards compatibility.
• Code is well-documented:
• A README.md has been added to explain the what Model Backwards Compatibility is and how it can be run.
• To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

TODO

• Check the tasks using an actual Jenkins Slave. @marcoabreu Can you help me out with that ?
• Backfill the S3 bucket with models trained on earlier versions of MXNet.

[piyushghai]

Pinging @marcoabreu @ThomasDelteil @piiswrong for review.
@marcoabreu Please let me know if I did the CI integration part correctly on this PR or if I need to make more changes to it.

[lupesko]

Thanks for this important contribution!

A not about the description, which is lacking, and should be updated as needed, rather than be left with un-relevant stuff from the template. A few examples:

• the issue description says "The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)"
  This issue is not tiny, and yet it has no JIRA. Please follow the guidelines and file a JIRA and update the issue name.
• "All changes have test coverage" is not checked. I think it should be checked, or else you should add the relevant coverage.
• I recommend you go over the issue description and update as needed. No need to leave things that are not relevant for the PR, and on the other hand add more details on bullet points that are relevant.

[szha]

Could you clarify what this is for?

[piyushghai]

@szha Here's what this PR does :
"Backward compatibility aims to check whether models trained on earlier versions of MXNet are loading fine on the latest version or the latest release candidate. It also aims to do a sanity check for consistency in the inference on these trained models. "

Here's the relevant JIRA Issue for it : https://issues.apache.org/jira/browse/MXNET-651

[szha]

What kind of "loading" are you testing? The statement, as is, is underspecified.

[piyushghai]

I am checking the following scenarios to save/load models :

1. Hybridized models import/export API from Gluon Package
2. Gluon Models load_params/save_params API from Gluon Package
3. Gluon Models load_parameters/save_parameters API from Gluon Package
4. Declarative Models load_checkpoint() from Model API

The Models are trained and saved on older MXNet versions, while the loading and inference part is performed on the newer versions.

[szha]

Any reason not to do it in unittest, using a smaller model?

[szha]

2 is deprecated so there's no need to test.

[szha]

How do you test load_parameters and save_parameters? What additional case would the new test case for gluon's save_parameters/load_parameters cover that's not already covered by https://github.com/apache/incubator-mxnet/blob/master/tests/python/unittest/test_gluon.py#L1093 ?

[piyushghai]

We cannot unittest it as we are not testing the various model save/load APIs on the same version. We are trying to check whether models trained (and saved) on older MXNet versions are still loading correctly and give the same inference results on the latest release candidate of MXNet and thus any new changes that might have crept in remain backwards compatible.
@marcoabreu can give more context on this change as well.

[marcoabreu]

Hello,
the following error is shown at http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/restricted-test-backwards-compatibility-checker/detail/restricted-test-backwards-compatibility-checker/4/pipeline/4:

Installing collected packages: graphviz, numpy, mxnet

  Found existing installation: graphviz 0.8.4

    Uninstalling graphviz-0.8.4:

      Successfully uninstalled graphviz-0.8.4

  Found existing installation: mxnet 1.2.0

    Uninstalling mxnet-1.2.0:

      Successfully uninstalled mxnet-1.2.0

Successfully installed graphviz-0.8.1 mxnet-1.1.0 numpy-1.13.3

==========================

Running training files and preparing models

==========================

Traceback (most recent call last):

  File "model_backwards_compat_train.py", line 20, in <module>

    from common import *

  File "/work/mxnet/tests/nightly/model_backwards_compatibility_check/common.py", line 22, in <module>

    import mxnet as mx

  File "/work/mxnet/python/mxnet/__init__.py", line 24, in <module>

    from .context import Context, current_context, cpu, gpu, cpu_pinned

  File "/work/mxnet/python/mxnet/context.py", line 24, in <module>

    from .base import classproperty, with_metaclass, _MXClassPropertyMetaClass

  File "/work/mxnet/python/mxnet/base.py", line 197, in <module>

    _LIB = _load_lib()

  File "/work/mxnet/python/mxnet/base.py", line 187, in _load_lib

    lib_path = libinfo.find_lib_path()

  File "/work/mxnet/python/mxnet/libinfo.py", line 74, in find_lib_path

    'List of candidates:\n' + str('\n'.join(dll_path)))

RuntimeError: Cannot find the MXNet library.

List of candidates:

libmxnet.so

/usr/local/lib/libmxnet.so

/work/mxnet/python/mxnet/libmxnet.so

/work/mxnet/python/mxnet/../../lib/libmxnet.so

/work/mxnet/python/mxnet/../../build/libmxnet.so

../../../libmxnet.so

==========================

Successfully trained files for the previous two MXNet release versions

It's strange that an error occurs but the run sitll continues. Could you please have a look?

[marcoabreu]

Missing set -ex

[piyushghai]

Done ceac705.

[marcoabreu]

It seems like that this pipeline has revealed some slight changes between 1.0.0 and our master:

INFO:root:Performing inference for model/API lenet_gluon_save_params_api
INFO:root:Fetching files for MXNet version : 1.0.0 and model lenet_gluon_save_params_api
/work/mxnet/python/mxnet/gluon/block.py:420: UserWarning: load_params is deprecated. Please use load_parameters.
  warnings.warn("load_params is deprecated. Please use load_parameters.")
Traceback (most recent call last):
  File "model_backwards_compat_inference.py", line 135, in <module>
    test_lenet_gluon_load_params_api()
  File "model_backwards_compat_inference.py", line 72, in test_lenet_gluon_load_params_api
    assert_almost_equal(old_inference_results.asnumpy(), output.asnumpy())
  File "/work/mxnet/python/mxnet/test_utils.py", line 493, in assert_almost_equal
    raise AssertionError(msg)
AssertionError: 
Items are not equal:
Error 1.969787 exceeds tolerance rtol=0.000010, atol=0.000000.  Location of maximum error:(5, 0), a=0.003014, b=0.003014
 a: array([[ 0.01743407, -0.30903903],
       [ 0.08352755, -0.365019  ],
       [ 0.06324662, -0.4323489 ],...
 b: array([[ 0.01743408, -0.3090391 ],
       [ 0.08352771, -0.365019  ],
       [ 0.06324662, -0.4323488 ],...

I will still merge this PR because fixing regressions is outside the scope of this PR.

[ChaiBapchya]

Q1 I can see this added as part of nightly test. However, the bucket contains models from 1.0 to till 1.6 Is it discontinued for subsequent minor versions of mxnet? 1.7 & master[2.0]?

Q2. The bucket for this test "mxnet-ci-prod-backwards-compatibility-models" was public. Our AppSec team reviewed it and asked to block public access. I've made that change. Why was this bucket made publicly accessible if its run on nightly within CI account? Was it required somewhere else as well? or was it kept public so that non-AWS folks can test it out as well?

@piyushghai @marcoabreu @szha @josephevans @sandeep-krishnamurthy

[ChaiBapchya]

Checked the pipeline: https://jenkins.mxnet-ci.amazon-ml.com/job/restricted-backwards-compatibility-checker/
It's been failing since months. We don't have a ticketing mechanism for this so its a feature that's untracked. Pl advise.

[marcoabreu]

Q1: No, it was not discontinued. It seems like it was just not caught. I remember that I created a sevice which sends out a daily email to an email list at Amazon with the results. Maybe somebody turned it off or everybody ignored the email..

Q2: It has to be public as otherwise the results are not reproducible for others. Hence it has to be on public read. Don't forget that this is an Apache and not an Amazon project. So yes, your question at the end is the answer :)

[marcoabreu]

I think we need to fix it. That'd be my advice.

[marcoabreu]

Btw, thanks for catching it!