Resolve pyspark / numpy conflicts

[loomlike]

Signed-off-by: Jun Ki Min 42475935+loomlike@users.noreply.github.com

Description

Pyspark is still relying on the old numpy api, referring np.bool that has been deprecated.
Because of that, when calling sparkDF.toPandas(), it throws AttributeError: module 'numpy' has no attribute 'bool'.
We have to upgrade pyspark version when they release new patch.

How was this PR tested?

Does this PR introduce any user-facing changes?

• No. You can skip the rest of this section.
• Yes. Make sure to clarify your proposed changes.

[blrchen]

I am having concern to do version pin in Feathr. Its okay to do version pin in Registry as its a standalone app. But for feathr it's a library and normally used in an environment with other python libraries in same context. Introduce a version pin might introduces the risk for package installation error or incompatible issues with other python packages.

I actually already experienced some issue when numpy was pin earlier

• Nightly notebook test fails with error ImportError: this version of pandas is incompatible with numpy < 1.20.3
• Installation issue on python 3.10 RuntimeWarning: NumPy 1.20.3 may not yet support Python 3.10

And seems this is already fixed in Spark apache/spark#37817, probably we can just set a min version for pyspeak instead?

[loomlike]

@xiaoyongzhu @blrchen I verified that unless we explicitly call sparkDF.toPandas() on the dataframe that include boolean type features, we can avoid the pyspark's bug. Once pyspark has new release, let's change the pyspark dependency to >new_version to address this.

Until then, we can stick with the current version. I'll put the comment on our setup.py instead of pinning numpy.