Update config files for g4

[ChaiBapchya]

Config files for G4 instance on MXNet CI [unix-gpu slaves]

UNIX AMI Creation changes

G4 instances have Tesla T4 drivers

1. refactored setup
2. updates

• nvidia-driver 418->440†,
• cuda driver 10.1->10.2,
• docker version 18->19, docker compose

3. upgrade host OS 16.04 -> 18.04
4. update instance type [p3/g3 -> g4]

† G4 instances require driver version 418.87 or later.
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/install-nvidia-driver.html

Autoscaling Lambda function changes

1. Added environment variables [test + prod] for G4

• launch template
• EXECUTORS_PER_LABEL
• WARM_POOL_SIZE
• MINIMUM_QUEUE_TIMES_SEC
• CCACHE_EFS_DNS
• MAXIMUM_STARTUP_TIME_SEC
• MANAGED_JENKINS_NODE_LABELS

2. Reduced executor per label for linux-cpu from 3 to 2
   This resolves the cant connect to linux-cpu error
   by reducing number of parallel jobs per instance
3. Update state from 'startingtopending` [as starting is incorrect state]

[leezu]

Suggest to have a single folder for all GPU instances. Not introduce a new folder for G4, but rather consolidate existing 2 folders into a single one.

[ChaiBapchya]

Sure makes sense!

[marcoabreu]

Why is it necessary at all? You can create an image on one machine type and just run it on all.

[leezu]

g4 needs a more recent driver. Therefore AMI requires update. Yes, there should be only a single AMI.

[marcoabreu]

Sure, but why not just simply update the existing setup script instead of introducing a new one?

[leezu]

Yes, that's what I suggested in the first comment above #20 (comment)

[ChaiBapchya]

@marcoabreu @leezu Updated with a single folder for all GPU instances.
Made a note of what needs to be updated in the README.md file of slave-creation-unix folder.

[josephevans]

@ChaiBapchya After our testing on Friday, I think we should also disable automatic Ubuntu updates. We know there is some fragility around the nvidia driver (if gcc is updated, for example, the driver stops working on the DLAMI based on Ubuntu.)

[ChaiBapchya]

https://askubuntu.com/questions/1059971/disable-updates-from-command-line-in-ubuntu-16-04

Adding

systemctl disable --now apt-daily{,-upgrade}.{timer,service}

[ChaiBapchya]

@leezu @josephevans Plz help review/merge.
Thanks!