backend : add eval callback

[ggerganov]

ref: #4931

# Metal
make -j && ./simple ./models/llama-7b/ggml-model-q4_0.gguf "Hello, my name is" 1

# CUDA
LLAMA_CUBLAS=1 make -j && ./simple ./models/llama-7b/ggml-model-q4_0.gguf "Hello, my name is" 1

The callback currently observes the softmax results in the attention, but can be customized in any way:

llama.cpp/examples/simple/simple.cpp

Lines 9 to 34 in 81d4162

	// a function that can be called for every computed node during graph evaluation
	// the user can choose to whether to observe the data of the node depending on the tensor parameters
	static bool observe_compute(int node_index, struct ggml_tensor * t, bool ask, void * user_data) {
	GGML_UNUSED(user_data);
	// the scheduler is asking us if we want to observe this node
	if (ask) {
	// check if name contains soft_max
	return strstr(t->name, "soft_max") != 0;
	}
	// print the node data
	printf("%s: node_index = %5d, t->name = %32s, t->op = %12s, [%5d, %5d, %5d, %5d]\n",
	__func__, node_index, t->name, ggml_op_name(t->op), (int) t->ne[0], (int) t->ne[1], (int) t->ne[2], (int) t->ne[3]);
	std::vector<float> t_data(ggml_nelements(t));
	ggml_backend_tensor_get(t, t_data.data(), 0, ggml_nbytes(t));
	// print first row
	for (int i = 0; i < t->ne[0]; i++) {
	printf("%8.4f ", t_data[i]);
	}
	printf("\n");
	return true;
	}

Skip last CLI arg (or set to 0) to disable the callback

[slaren]

Is the ask callback really necessary here?

[ggerganov]

I've changed the implementation to ask only once per node in a split

[slaren]

I think this is fine, we don't need to clear the callbacks here, the reset function is meant to prepare the sched for the next graph evaluation, resetting the allocators and the backend assignments (similar to ggml_allocr_reset).