add gpu uuid to the fingerprinted attributes

[shumin1027]

I want to use gpu uuid when configuring affinity or constraint，but there is no such attribute in fingerprinted attributes，how should it be achieved?

Fingerprinted Attributes

[lgfa29]

Hi @shumin1027 👋

I believe the UUID is just missing from the documentation as the fingerprinter does report the UUID.

Could you check if the GPU UUID is present when you run nomad node status -verbose <node ID>?

[shumin1027]

Hi @shumin1027 👋

I believe the UUID is just missing from the documentation as the fingerprinter does report the UUID.

Could you check if the GPU UUID is present when you run nomad node status -verbose <node ID>?

@lgfa29
Thank you，
It's not that the documentation is missing，all fingerprinted attributesare defined here：

nomad-device-nvidia/fingerprint.go

Lines 13 to 24 in 541c8ca

	const (
	// Attribute names and units for reporting Fingerprint output
	MemoryAttr = "memory"
	PowerAttr = "power"
	BAR1Attr = "bar1"
	DriverVersionAttr = "driver_version"
	CoresClockAttr = "cores_clock"
	MemoryClockAttr = "memory_clock"
	PCIBandwidthAttr = "pci_bandwidth"
	DisplayStateAttr = "display_state"
	PersistenceModeAttr = "persistence_mode"
	)

And here's how to populate the attribute data, it can be seen that there is indeed no GPU UUID

nomad-device-nvidia/fingerprint.go

Line 167 in 541c8ca

Attributes: attributesFromFingerprintDeviceData(deviceList[0]),

nomad-device-nvidia/fingerprint.go

Line 181 in 541c8ca

func attributesFromFingerprintDeviceData(d *nvml.FingerprintDeviceData) map[string]*structs.Attribute {

[shumin1027]

@lgfa29
By the way，when I try to solve this problem by adding GPU UUID to fingerprinted attributes here:

nomad-device-nvidia/fingerprint.go

Line 181 in 541c8ca

func attributesFromFingerprintDeviceData(d *nvml.FingerprintDeviceData) map[string]*structs.Attribute {

But a new problem was found：
When a device like the NVIDIA Tesla K80 with a Dual GPU，there will be a conflict :

nomad-device-nvidia/fingerprint.go

Lines 164 to 167 in 541c8ca

	Devices: devices,
	// Assumption made that devices with the same DeviceName have the same
	// attributes like amount of memory, power, bar1memory etc
	Attributes: attributesFromFingerprintDeviceData(deviceList[0]),

Two GPUs with different UUIDs will be treated as one device with the same fingerprinted attributes

I still don't know how to solve this problem elegantly

[lgfa29]

Oh, you're right @shumin1027, I misunderstood the code. I don't think it's possible to fix this at the plugin level, it seems like a limitation within Nomad.

I opened hashicorp/nomad#15455 to try and fix this. I'm building custom binaries for you to test if you have the chance and I will post them here once they're ready.

[lgfa29]

Here are the custom binaries: https://github.com/hashicorp/nomad/actions/runs/3604896100#artifacts

[shumin1027]

@lgfa29 It's great, I will continue to test

[lgfa29]

Nice! If the fix for you feel free to close this issue. The expect the PR to be merged soon and for it to be released in the next version of Nomad.

[ruspaul013]

Hello @shumin1027, @lgfa29 !
I tried using device.ids as constraint in a job file and every time I get random GPUs instead of the ones I set the UUIDs. Here is the link to nomad forum post that I have made, where are more details about the issue.

Thank you!

[lgfa29]

Hi @ruspaul013 👋

I answered in your post, but TL;DR: yes, you need to upgrade your Nomad clients to a version that includes this change.

[ruspaul013]

Hello @lgfa29 👋
Thank you so much for the answer. I posted an update on the forum.