Access normal world memory from OPTEE core - translation fault

[saeedm92]

Hi all,

I'm trying to copy a large memory buffer in the normal world to a buffer allocated in TA (using Tee_Malloc) at OPTEE kernel but it crashed with this error:
E/TC:6 0 Core data-abort at address 0x3c400000 (translation fault).

I'm doing memcpy() for size of 8294400 bytes. Here is the code:
core_mmu_add_mapping(MEM_AREA_RAM_NSEC, paddr, 2025);
va = phys_to_virt(paddr, MEM_AREA_RAM_NSEC);
tee_svc_copy_to_user(TA_va, va, 2025*4096); //memcpy()

The starting virtual address for the normal world buffer (in the OPTEE) is 0x3c100000 and the memcpy crashes at the address: 0x3c400000. Looks like that it can only copy for 0x300000 size. Also, I can read the normal world memory addresses with no problem by using the same mapping and phys_to_virt() technique. The code above works fine if I copy a smaller size. Seems like it has a problem when I'm copying a large size.

Does anybody have an idea what's the problem? I looked into discussions and tried some solutions such as increasing the PGT_CACHE_SIZE or CFG_CORE_HEAP_SIZE but it didn't change anything. I tried to break down the copy into multiple smaller chunks but the same issue. I'm trying on Hikey board.

Thanks a lot!

[etienne-lms]

The starting virtual address for the normal world buffer is 0x3c100000 (...)

1. Is this the address from from Linux side (from a linux driver or a userland application) to access the buffer? (from a kernel driver or a user application?).
2. Does the Linux word register the shared buffer using TEEC_RegisterSharedMemroy()?

If answer to 1st question if yes and to the 2nd is no, than you are attempting to use in the secure world a virtual address related to the non-secure world. This is not valid.

For a TA to access a (non-secure) shared memory buffer, the client application shall pass the buffer reference as an argument of type TEEC_MEMREF_TEMP_{INPUT|OUTPUT|INOUT} or TEEC_MEMREF_PARTIAL_{INPUT|OUTPUT|INOUT} or TEEC_MEMREF_WHOLE.
On secure side, TA shall:

• check that invocation argument is of expected type TEE_PARAM_TYPE_MEMREF_{INPUT|OUTPUT|INOUT}, see docs/checking-ta-parameters;
• use buffer address range referenced by <param-ref>[<param-index].memref.buffer and <param-ref>[<param-index].memref.size.

[saeedm92]

@etienne-lms Thanks for the reply!

1. The address is not from normal world. It's the virtual address that I got from phys_to_virt(paddr, MEM_AREA_RAM_NSEC) after mapping the paddr to the secure world using core_mmu_add_mapping().

2. I'm not registering the buffer as shared memory using TEEC_RegisterSharedMemroy(). Do I have to register the buffer as shared memory in the host? The buffer is the framebuffer which is already allocated. I prefer not to allocate a shared memory buffer and use the already allocated framebuffer.

The code above works fine if I copy a smaller size. Seems like it has a problem when I'm copying a large size. Do I have to increase something like CFG_TEE_RAM_VA_SIZE or something else?

[etienne-lms]

I understand that you register an non-secure physical memory area in OP-TEE core through core_mmu_add_mapping(). This maps the memory in the core. Ok.
But note that TAs cannot access memory mapped in core space. TAs access their own memory (code, data, stack, heap: these are persistent mapping in the TA) and the shared buffers passed as invocation arguments (memref parameters are temporary mapping in the TA, valid only for a single invocation occurrence).

For your TA to access your non-secure frame buffer, the standard way would be:

• client maps the frame buffer into its own env (application? kernel drivers?)
• client registers the frame buffer as a shared memory buffer (TEEC_RegisterSharedMemory())
• client invokes the TA with the frame buffer passed as invocation parameter
• TA, when invoked, gets a valid mapping for the frame buffer and get the buffer address/size from the invocation parameter arguments.

And in this case, if CFG_DYN_SHM_CAP=y, the invoked TA will read/write into the effective frame buffer.

Without CFG_DYN_SHM_CAP, Linux kernel will allocate a temporary buffer in the statically defined shared memory area and invoke the TA with a reference to that temporary buffer. Linux kernel will copy effective frame buffer content to the temporary buffer before invoking the TA, and will update effective frame buffer content from temporary buffer content once TA invocation returns with a successful status.

[saeedm92]

@etienne-lms Thanks a lot!

I used shared memory and I could read the whole buffer in the TA. But I had to allocate a new buffer in the host application and copy the whole framebuffer to it. Then passed the newly allocated buffer to the shared memory. When I used passed a pointer from my mmap to the buffer, the TEEC_InvokeCommand fails. The pointer from mmap is fine cause I can read the values from the address.
//This fails (TEEC_InvokeCommand() fails):
fbp = (char *)mmap(0, screensize, PROT_READ | PROT_WRITE, MAP_SHARED, fbfd, 0);
buf_shm.buffer = (void*)fbp;
// I can read the values of fbp in here.

//This works:
char* tmp = (char*)calloc(screensize, 1);
memcpy(tmp, fbp, screensize);
buf_shm.buffer = (void*)tmp;

Also, I still don't know how I can access the NW buffer in TA without using the shared buffer. I think using the shared buffer causes some overhead.

[etienne-lms]

When I used passed a pointer from my mmap to the buffer, the TEEC_InvokeCommand fails.

I think the base flow for sequence should be like this (edited: there were obvious mistakes):

	TEEC_SharedMemory shm = { 0 };
	TEEC_Operation op = { 0 };

	shm.buffer = <your-buffer-pointer>;
	shm.size = <your-buffer-byte-size>;
	shm.flags = TEEC_MEM_INOUT;

	rc = TEEC_RegisterSharedMemory(&context, &shm);
	if (rc)
		goto error;
	(...)

	op.params[0].memref.parent = &shm;
	op.params[0].memref.size = 2942;     // some value (byte size)
	op.params[0].memref.offset = 234;    // some value (offset from shm base)
		
	rc = TEEC_InvokeCommand(&session, SOME_TA_CMD, &op, &ret_orig);
	(...)

	TEEC_ReleaseSharedMemory(&shm);

I was to point you at https://github.com/linaro-swg/optee_examples but I see there is no example on shared memory. Here, 32 * 1024 is arbitrary. TEEC_MEM_INOUT makes TA have read/write access, see alternates from optee_client lib header files.

Also, I still don't know how I can access the NW buffer in TA without using the shared buffer.

OP-TEE is based on the GPD TEE specification that describes the interface between CAs & TAs. Use and management of shared is described.

Technically speaking, is it is possible to have a persistent mapping of non-secure memory in a TA. OP-TEE does not propose such hack/support.

I think using the shared buffer causes some overhead.

Yes, you are right. But OP-TEE may already have something for your case.

When OP-TEE is built with CFG_DYN_SHM_CAP=y, the non-secure world can register a non-secure memory (physically contiguous or not) into OP-TEE core and use buffer references from that buffer. No copyback, the TA will get a mapping for the buffer chunk. OP-TEE takes care that TA receives reliable invocation arguments.

[saeedm92]

@etienne-lms Thanks! But, I still don't know why the mmap()ed memory fails but if I allocate a new buffer it works. My code sequence is exactly like your code. Basically, when I use the fbp pointer as the buffer Invoke fails and when I change it to the tmp (which I got from calloc()) it works. I don't know why there should be any difference for the Invoke. Also, checked the libteec library and looks like that the ioctl() fails.

Here is the error code:
ERR [2979] TEEC:TEEC_InvokeCommand:611: TEE_IOC_INVOKE failed
optee_example_hello_world: TEEC_InvokeCommand failed with code 0xffff0000 origin 0x2

Thanks!

[etienne-lms]

I think you need to enable debug traces in OP-TEE OS/core and maybe in OP-TEE Linux driver.
It seems strange to me that the API successfully registered a buffer and refuses to use it afterward when invoking a TA.

What is possible is that the buffer you try to register fails to be used by OP-TEE Linux driver or by the secure world. For example, if the buffer was allocated from Ion, or some specific memory management stack, maybe OP-TEE Linux driver fails to construct a OP-TEE description of the buffer (list of physical pages used), or maybe OP-TEE core rejected the listed physical pages as being used as shared memory.

[saeedm92]

@etienne-lms Thanks! You're right. I saw that the TEEC_RegisterSharedMemory() failed before I did the Invoke. I debugged in the Linux driver and looks like that the tee_shm_register() fails. Inside this function the get_user_pages_fast() fails. What do you think might be the cause of the problem and how can I get around with it? Also my mmap() uses "dma_mmap_writecombine()" to create the mapping in the driver. Thanks!

[etienne-lms]

If your buffer is related to a dma-buf reference, maybe you can use TEEC_RegisterSharedMemoryFileDescriptor().
You can find an example of use in sdp_basic.c#L218 to register a buffer reference allocated from ION which produces a dma-buf file descriptor.

[github-actions]

This issue has been marked as a stale issue because it has been open (more than) 30 days with no activity. Remove the stale label or add a comment saying that you would like to have the label removed otherwise this issue will automatically be closed in 5 days. Note, that you can always re-open a closed issue at any time.