Discussion on ptr_pkg structure and naming

[bradleyharden]

@LarsAsplund and @umarcor, here is the discussion I promised on Gitter.

In their current form, I find the storage data structures for string_ptr and integer_vector_ptr a bit confusing. I think I can simplify them.

So far, you have kept the original definition of string_ptr_t, containing only the ref element. The value ptr.ref indexes into st.idxs which retrieves a storage_t element containing the mode and another reference id. The value of id then indexes either st.ptrs or st.eptrs, depending on mode. The types and shared variables in question are given below.

type string_ptr_t is record
  ref : index_t;
end record;

type storage_t is record
  id     : integer;
  mode   : storage_mode_t;
  length : integer;
end record;

type storage_vector_t is array (natural range <>) of storage_t;
type storage_vector_access_t is access storage_vector_t;

type ptr_storage is record
  idx   : natural;
  ptr   : natural;
  eptr  : natural;
  idxs  : storage_vector_access_t;
  ptrs  : vava_t;
  eptrs : evava_t;
end record;

shared variable st : ptr_storage := (0, 0, 0, null, null, null);

I see a few different problems with this. First, you have to index into an array twice for every access of a string_ptr. That's a bit awkward. Second, you are storing a useless length parameter for every internal string_ptr.

I think I can improve the structure by shifting things around slightly.

The mode should be added to string_ptr_t, because it represents an intrinsic attribute of a string_ptr. The mode of a string_ptr will never change after its creation, so this isn't a problem when treating string_ptr values as constants.

Storing a length attribute is only necessary for external pointers. Because of this, the storage requirements for internal and external string_ptrs are fundamentally different. Therefore, they should use distinct storage systems.

Below, I propose a new set of definitions to resolve these problems.

type string_ptr_t is record
  ref  : index_t;
  mode : storage_mode_t;
end record;

type ext_storage_item_t is record
  acc    : extstring_access_t;
  length : integer;
end record;

type ext_storage_vector_t is array (natural range <>) of ext_storage_item_t;
type ext_storage_vector_access_t is access ext_storage_vector_t;

shared variable int_storage : vava_t := null;
shared variable int_index : natural := 0;

shared variable ext_storage : ext_storage_vector_access_t := null;
shared variable ext_index : natural := 0;

Now, you only need to perform one indexing operation when accessing a string_ptr, because the mode attribute is able to break the ambiguity immediately.

If you agree with this new structure, I will make these changes while I make the other modifications necessary for my new queue implementation.

[umarcor]

Hi @bradleyharden! Overall, I'm good with improving the structure to make it easier to understand and, hopefully, more efficient. However, I am afraid that your current proposal does not preserve all the features.

We can rewrite the four shared varibles you propose as a single record:

type string_ptr_t is record
  ref  : index_t;
  mode : storage_mode_t;
end record;

type ext_storage_item_t is record
  acc    : extstring_access_t;
  length : integer;
end record;

type ext_storage_vector_t is array (natural range <>) of ext_storage_item_t;
type ext_storage_vector_access_t is access ext_storage_vector_t;

type ptr_storage is record
  
  int_index   : natural;
  ext_index   : natural;
  
  int_storage : vava_t;
  ext_storage : ext_storage_vector_access_t;
end record;

shared variable st : ptr_storage := (null, 0, null, 0);

Now, if we compare it side by side, the most meaningful differences are:

• The current codebase contains three arrays of indexes in ptr_storage (idxs, ptrs and eptrs) and you propose to handle two only.
• Apart from moving mode from storage_t (now ext_storage_item_t) to string_ptr_t, you propose to change id: integer to acc: extstring_access_t.

The point is that there are two types of external modes. Therefore, we do need three different 'shared' arrays (either 3/6 different variables or a record with as many fields). Apart from that, the id is required for extfnc (it is used as a parameter in each call to read_*/write_*) and it might be required for extacc (it would be used to update the pointer in C if an array is reallocated in VHDL). Note that 'id' is not the same as field ref in string_prt_t. The latter is the index in the corresponding shared variable in VHDL, and the former is the index in the shared array of pointers between VHDL and C. E.g., a user can declare three extacc arrays with ref values 0, 1 and 2, which point to pointers x, y and z in uint8_t *D[256];.

I propose the following definitions:

type string_ptr_t is record
  ref  : index_t;
  mode : storage_mode_t;
end record;

type extacc_item_t is record
  acc    : extstring_access_t;
  id     : index_t;
  length : natural;
end record;

type extfnc_item_t is record
  id     : index_t;
  length : natural;
end record;

type extacc_vector_t is array (natural range <>) of extacc_item_t;
type extacc_vecaccess_t is access extacc_vector_t;

type extfnc_vector_t is array (natural range <>) of extfnc_item_t;
type extfnc_vecaccess_t is access extfnc_vector_t;

type ptr_storage is record
  int_idx : natural;
  acc_idx : natural;
  fnc_idx : natural;
  int_vec : vava_t;
  acc_vec : extacc_vecaccess_t;
  fcn_vec : extfcn_vecaccess_t;
end record;

shared variable storage : ptr_storage := (null, 0, null, 0);

If you agree with this new structure, I will make these changes while I make the other modifications necessary for my new queue implementation.

It's ok if you want to make them. However, since I'm more familiar with the three types, it might be faster for me. It's up to you.

[bradleyharden]

@umarcor, yes, I realized in my sleep that I had misunderstood. I didn't fully understand the eid parameter.

I re-read the code, and I think I follow everything now. However, one of your statements confused me:

Apart from that, the id is required for extfnc (it is used as a parameter in each call to read_/write_) and it might be required for extacc (it would be used to update the pointer in C if an array is reallocated in VHDL).

In the existing code, the id parameter in extacc mode is only ever used to index into the st.eptrs array. The element of st.eptrs is the pointer you get from C using eid, correct? The id in extacc mode could be used to uniquely identify the corresponding pointer, but it still must always act as an index into st.eptrs, right?

If that's the case, then id already represents a VHDL index for internal and extacc modes. For extfnc mode, it acts as an identifier for C, but it is still restricted to the range of index_t by the definition of eid in new_string_ptr. So, in all cases, id is already restricted to the range of index_t.

Since the ref and id parameters are both restricted to the range of index_t, then do we really need to store them as separate values? I have two questions that could affect that answer:

1. What is the meaning of eid = -1 for extacc and extfnc. Is that equivalent to a null pointer?
2. Do users need to know the precise value of eid before calling new_string_ptr, or does eid simply act as a handle for external pointers in VHDL? Are you expecting users to manually sync eid values between VHDL and C?

If the answers to those questions are favorable, then why not combine ref and id? For the extacc case, id is already determined for you by the VHDL, and eid is only used at the initial allocation. Couldn't we just use the chosen id as the eid? If so, couldn't we do the same for extfnc? We could provide a VHDL get_eid function that returns the value of ref which is effectively eid for external modes.

If that would work, then this is how I would implement it. Here, I'm giving a "verbose" definition of the storage types to highlight the differences in their storage requirements. Mode internal only needs to store an array of pointers. Mode extacc needs to store an array of pointers and an associated array of lengths. And mode extfnc only needs to store an array of lengths.

type string_ptr_t is record
  ref  : index_t;
  mode : storage_mode_t;
end record;

type internal_item_t is record
  acc : string_access_t;
end record;

type extacc_item_t is record
  acc    : extstring_access_t;
  length : natural;
end record;

type extfnc_item_t is record
  length : natural;
end record;

type internal_vector_t is array (natural range <>) of internal_item_t;
type internal_vector_access_t is access internal_vector_t;

type extacc_vector_t is array (natural range <>) of extacc_item_t;
type extacc_vector_access_t is access extacc_vector_t;

type extfnc_vector_t is array (natural range <>) of extfnc_item_t;
type extfnc_vector_access_t is access extfnc_vector_t;

In reality, we would probably implement it as

type string_ptr_t is record
  ref  : index_t;
  mode : storage_mode_t;
end record;

type extacc_item_t is record
  acc    : extstring_access_t;
  length : natural;
end record;

alias internal_vector_access_t is vava_t;

type extacc_vector_t is array (natural range <>) of extacc_item_t;
type extacc_vector_access_t is access extacc_vector_t;

alias extfnc_vector_access_t is integer_vector_access_t;

We would also need to store the current index for each of these storage types. You like to group them into a single record, but I want to split them into three separate records, because I will be adding more to each storage as part of my code. I can get into that if you would like, but that's a minor point compared to the above discussion.

[umarcor]

In the existing code, the id parameter in extacc mode is only ever used to index into the st.eptrs array. The element of st.eptrs is the pointer you get from C using eid, correct? The id in extacc mode could be used to uniquely identify the corresponding pointer, but it still must always act as an index into st.eptrs, right?

Your are correct. Currently eid is only used in get_ptr(eid) and it is then lost. However, deallocate, reallocate and resize are not yet implemented for external modes. All of these will require to know eid, because VHDL will change the value of the pointer and that change needs to be communicated to C. Hence, I was anticipating that:

• Internal requires a single id (VHDL) and an access.
• Extacc requires two ids (VHDL and C) and an access.
• Extfnc requires two ids (VHDL and C).

If that's the case, then id already represents a VHDL index for internal and extacc modes.

When internal vectors are deallocated, reallocated or resized, the old pointer is explicitly replaced. Therefore, any other part of the VHDL design that is using the access will be implicitly aware of the changes. However, extacc requires set_ptr(eid, <new_pointer>) to be executed (not implemented yet https://github.com/VUnit/vunit/tree/master/vunit/vhdl/data_types/src/external) to update *D in C.

For extfnc mode, it acts as an identifier for C, but it is still restricted to the range of index_t by the definition of eid in new_string_ptr. So, in all cases, id is already restricted to the range of index_t.

Currently, the shared buffer between C and VHDL is defined as *D[256]. That's why the range of eid lies in the range of index_t.

1. What is the meaning of `eid = -1` for `extacc` and `extfnc`. Is that equivalent to a `null` pointer?

The meaning is 'you did something wrong': https://github.com/VUnit/vunit/blob/master/vunit/vhdl/data_types/src/string_ptr_pkg-body-2002p.vhd#L131-L135. It is an invalid default, to fail in case the user used extacc or extfnc without specifying eid. I forgot to add those assertions to string_ptr_pkg-body-93, integer_vector_ptr_pkg-body-93 and integer_vector_ptr_pkg-body-2002p.

2. Do users need to know the precise value of `eid` before calling `new_string_ptr`, or does `eid` simply act as a handle for external pointers in VHDL?

I'd say yes, although I am not sure to understand the question. How can eid be used as a handle for external pointers if it is not defined in advance?

Are you expecting users to manually sync eid values between VHDL and C?

I expect users to define eids in VHDL as:

constant eids: byte_vector_ptr_t := new_byte_vector_ptr( 256, extacc, 0);
constant eid_lengths: integer_vector_ptr_t := new_integer_vector_ptr( 256, extacc, 1);

-- external modes enabled
constant eid_first_vector := natural := eids(0);
constant eid_fifo_input   := natural := eids(1);
constant eid_RAM_content  := natural := eids(2);

-- external modes disabled
constant eid_first_vector := natural := -1;
constant eid_fifo_input   := natural := -1;
constant eid_RAM_content  := natural := -1;

and, in C:

D[0] = (uint8_t *) malloc(3);
D[0][0] = x; //assign id that will be accessed from VHDL as eid_first_vector
D[0][1] = y; //assign id that will be accessed from VHDL as eid_fifo_input
D[0][2] = z; //assign id that will be accessed from VHDL as eid_RAM_content

D[1] = (uint8_t *) malloc(256*4);
D[1][0] = xx; //assign maximum length of eid_first_vector
D[1][1] = yy; //assign maximum length of eid_fifo_input
D[1][2] = zz; //assign maximum length of eid_RAM_content

D[x] = (uint8_t *) malloc(xx * sizeof(X_TYPE));
D[y] = (uint8_t *) malloc(yy * sizeof(Y_TYPE));
D[z] = (uint8_t *) malloc(zz * sizeof(Z_TYPE));

where x, y and z are in the range [2,255].

Then, assuming that eid_* VHDL constants are defined in some (global) package, users can use them anywhere in the hierarchy.

Of course, I am interested on hearing any thoughts you might have about this procedure.

If the answers to those questions are favorable, then why not combine ref and id? For the extacc case, id is already determined for you by the VHDL, and eid is only used at the initial allocation. Couldn't we just use the chosen id as the eid? If so, couldn't we do the same for extfnc?

Even if id and eid are restricted to the same range, it is not possible to use the exact same value for both contexts. Say we have 5 arrays in C and we want to access 0, 2, 3 as extacc and 1, 4 as extfnc:

C	extacc_vector_t	extfnc_vector_t
0	0
1		0
2	1
3	2
4		1

Since the ref and id parameters are both restricted to the range of index_t, then do we really need to store them as separate values? I have two questions that could affect that answer:

If we wanted to use the id from the first column (eid) as the id of the third (ref), extacc_vector_t would need to be of size 5, even though 3 of the pointers would be null. Conversely, we cannot use the id from the third column to access the first one, because we would be interacting with a different vector, not the one we want.

We could provide a VHDL get_eid function that returns the value of ref which is effectively eid for external modes.

Let's assume that all the external functions that we define are callbacks. In this context, a get_eid function in VHDL is not really useful, because VHDL sources can already access ref and foreign languages cannot execute it. The closest approach would be a set_ptr(eid, <ptr>), as commented above.

If that would work, then this is how I would implement it. Here, I'm giving a "verbose" definition of the storage types to highlight the differences in their storage requirements. Mode internal only needs to store an array of pointers. Mode extacc needs to store an array of pointers and an associated array of lengths. And mode extfnc only needs to store an array of lengths.

Although it might be possible to achieve this, I see two major drawbacks:

• We would need to implement housekeeping features to ensure that h in *D[h] is always larger than the largest id assigned in VHDL when declaring/allocating a new vector. I don't think it is a problem to allocate a large *D, if most of the pointers are null. But we need to take into account that the same *D contains arrays that are accessed as strings, byte_vectors or integer_vectors from VHDL. Therefore, none of the VHDL packages contains all the information. Housekeeping would need to be done in C.
• It would not be possible to define different external vectors in VHDL to access the same array in C. Currently, it is possible to access the same D[n] array as a byte_vector in a VHDL module and as an integer_vector in another module, at the same time. This is because each VHDL array has a different id and both of them share the same eid (or access/pointer). If VHDL provides id numbers for the list of each type and mode independently (now sequently and, after you implemented your enhancements, any unused id), it is almost imposible to make them match.

We would also need to store the current index for each of these storage types. You like to group them into a single record, but I want to split them into three separate records, because I will be adding more to each storage as part of my code. I can get into that if you would like, but that's a minor point compared to the above discussion.

I don't think this is relevant at all. It's ok if you want to split them into separate records, specially if you are going to add more content. I wrote it again as a single record in order to not get distracted with cosmetic changes, because I wanted to focus on id vs eid and the fact that we are dealing with three types, not two.

[bradleyharden]

@umarcor, thanks for the example in both VHDL and C. It was very helpful to understand how you intended the interface to work. Also, thanks for taking the time to explain this all to me. It's very relevant to my additions, so I want to make sure string_ptr is stable before I try to build on top of it.

Regarding my comment

If the answers to those questions are favorable

They were not, so none of the comments that followed apply. However, now that I understand the system better, I have some additional thoughts. But before I speak, I would like to make sure I really understand the whole system and your design goals when creating it. I have a few questions for you:

1. Am I correct that vhpidirect_user.h is intended to be the implementation of the VHPIDIRECT functions? You did not intend for users to define these functions themselves, right? Users should include this file and work with directly with the predefined array D, yes?

2. What was your goal in defining the two external modes extfnc and extacc? Right now, the only difference seems to be who does the pointer dereferencing. In extacc mode, VHDL dereferences the pointer, but in extfnc mode, C dereferences the pointer. Am I missing something? Is there some future addition that will make the distinction more substantial?

[umarcor]

Also, thanks for taking the time to explain this all to me. It's very relevant to my additions, so I want to make sure string_ptr is stable before I try to build on top of it.

I understand it and I agree on it being important. Hence, I'm ok with discussing this as long as you wish, and I don't mind slightly repeating ourselves until we get it clear.

1. Am I correct that vhpidirect_user.h is intended to be the implementation of the VHPIDIRECT functions? You did not intend for users to define these functions themselves, right? Users should include this file and work with directly with the predefined array D, yes?

vhpidirect_user.h is the default C implementation of the interface. Users, specially those that are new, are expected to include it and use *D[256] as the shared type between C and VHDL. However, this header file is not used in VUnit's codebase at all. I.e., users needs to explicitly include it in their main.c file. This is because I/we expect intermediate and advanced users to require custom versions of it. Say you want to share 768 pointers between C and VHDL. You copy the header file along with your own testing VHDL sources, you modify whatever, and you import it instead of the default. All you need to guarantee is that the functions that are used in external_* VHDL sources (https://github.com/VUnit/vunit/tree/master/vunit/vhdl/data_types/src/external) are actually implemented in your main or in the headers you import.

Furthermore, if you have multiple tests in a run.py file (i.e. handled by the same VUnit object), and some of them do not use external modes, you do need to provide some dummy C code to those. https://github.com/VUnit/vunit/blob/master/vunit/vhdl/data_types/src/external/ghdl/stubs.c is provided for that purpose. It is another default implementation of the C interface. This is a very particular implementation that allows to compile VHDL designs but which will produce errors if external modes are used.

Last, although reference VHDL packages are provided in https://github.com/VUnit/vunit/tree/master/vunit/vhdl/data_types/src/external, it is possible to provide custom versions through vu.add_builtins. This feature is mostly meant for users that want to implement external modes with other simulators (say QuestaSim and FLI).

2. What was your goal in defining the two external modes extfnc and extacc? Right now, the only difference seems to be who does the pointer dereferencing. In extacc mode, VHDL dereferences the pointer, but in extfnc mode, C dereferences the pointer. Am I missing something? Is there some future addition that will make the distinction more substantial?

With mode extacc, VHDL retrieves a pointer once, and then data is read/written directly. This requires the pointer to point to a memory region that is accesible to both the VHDL and the C source. The expected use case is to wrap GHDL in C, C++ or Python, so that the VHDL is a child of the main process and the memory space is common.

Conversely, mode extfnc is to be used when data is not available in the same memory space. Each time a value needs to be read/written, a function is executed (read_*/write_*). Actually, using these functions as in vhpidirect_user.h is stupid (not useful). When extacc is suitable, users should use it. When it is not, they should copy the reference header file and adapt it. Examples of these advanced uses are not contributed yet.

One use case is to define from Python a callback that is executed in write_char. As a result, when the GHDL simulation is loaded from Python as in https://github.com/dbhi/vunit/blob/vunitcosim/examples/vhdl/external_buffer/cosim.py, it is possible to make VHDL write a message directly to a Python queue. This procedure reduces the importance of C sources so that the relevant logic is described/programmed in VHDL and/or Python. This is specially pertinent in the context of VUnit, because communication/queue libraries for VHDL are provided and Python already includes queue types that are easier to use than C equivalents.

Another use case is to run a service that accepts HTTP requests and contains the virtual model of a memory. Then, multiple simulations can be executed in several boards/workstations, all of them sharing the virtual memory by using read_* and write_* function calls to handle HTTPS requests. https://github.com/dbhi/gRPC is an example of such a service. It is a prototype, and virtual streams/FIFOs are supported only. However, since it is written in golang, it is an interesting example to show that extfnc is easier to use when interfacing different languages.

Precisely, I am very interested on your contributions regarding the new queue implementation, because I think that providing a generic solution to plug a queue/vector of type extfnc to any foreign language/queue and transfer arbitrary types of data (say records) can be a really powerful feature.

[bradleyharden]

@umarcor, sorry for the delay. I haven't had much free time lately. I wanted to go through your examples in more detail before responding, but I don't know how long it will be before I have the time. Instead, I think I will lay out my ideas and see what you think. If possible, I would also like feedback from @LarsAsplund and @kraigher, because I think my ideas are far-reaching and would require a lot of work. In the end, though, I think it would be worth it.

Overall, I like the new external pointer system, but I think I see some opportunities to simplify the interface for users and more fully integrate external pointers with other VUnit data structures. That being said, my C is a bit rusty, so I might be missing something.

In the current system, users must manage external pointers manually. For instance, in the example you provided above, users must allocate two arrays to store the eid and length of each pointer. Furthermore, they must also manually track which pointer is assigned to which index within the eids vector. I think this is ultimately prone to mistakes. Furthermore, I think it deviates from the existing VUnit model. Right now, users don't have to think about or manage string_ptrs at all.

Instead, I would like to see VUnit manage pointers for the user. We already do this for users in VHDL, why not mimic the same interface in C? Essentially, my thought is this:

Create a new C library, ext_ptr.c, that implements pointer management in the same way as string_ptr_pkg. Expose the exact same set of functions as string_ptr_pkg (e.g. new_ptr, resize, reallocate, etc.) in C. Then, provide a wrapper library in VHDL, ext_ptr_pkg, that calls these functions through VHPI direct.

On the VHDL side, the ext_ptr record would look just like the old string_ptr, i.e.

type ext_ptr_t is record
  ref : index_t;
end record;

Here, ref indexes into a data structure in ext_ptr.c.

On the C side, each ext_ptr struct would look something like this

struct ext_ptr_t {
  int32_t   ref;
  uint32_t  length;
  uint8_t  *ptr;
};

I don't see any need to separate pointers by type (i.e. uint8_t * vs. int32_t *), because users can always cast the bare pointer as needed.

On the VHDL side, I think we could find a way to implement something like a pointer type cast from ext_ptr to string_ptr and integer_vector_ptr. For example, we could overload the get and set functions of ext_ptr_pkg to return and accept both characters and integers. Then, we could provide to_string_ptr and to_integer_vector_ptr functions to convert ext_ptr to either. Finally, we could implement string_ptr and integer_vector_ptr as

type *_ptr_t is record
  ref  : index_t;
  mode : storage_mode_t;
end record;

If the string_ptr_pkg get function were called with ptr.mode = external, then it would know to "cast" the ptr back to ext_ptr_t before calling get to return a character.

So far, I've addressed the management problem, but I haven't addressed the syncing problem. For that, I would propose that we allow users to provide a unique name as an additional argument to the new_ext_ptr functions in both VHDL and C. There is precedent for this concept in both VUnit and in Python. In VUnit, we do the exact same thing to identify actors in the com library. And in Python, the newly introduced shared_memory library uses this approach for the exact same purpose. In both cases, we use a unique name to allow concurrent actors to identify a shared resource within a managed data structure.

What I've proposed seems to align well with the use cases for mode extacc, but I don't think I've addressed mode extfnc. However, based on your description of extfnc, it sounds less like shared memory and more like a data stream or socket. If that was your intent, then I think it would be better implemented by leveraging VUnit queues.

With the current external pointer system, I don't see an easy way to implement queues between an external actor and a VHDL actor. We could alter the new_queue function to accept a string_ptr or even an eid, but that leaves even more management to the user, and doesn't provide an easy way to push to a queue from C.

Alternatively, we could use an approach similar to what I proposed above. We could implement a C library, ext_queue.c that implements the same interface as queue_pkg. It would allocate pointers using ext_ptr.c, just like queue_pkg does with string_ptr_pkg. With this structure, users could push data into a queue in C, and that data would appear in VHDL. We would then add a VHDL package ext_queue_pkg that would be a wrapper for ext_queue.c. We could also allow users to provide a unique name for each ext_queue, so it could be identified by name on either side.

This structure would also mesh well with my new, proposed type system. Right now, queue_pkg tracks the type of each pushed element within queue_pkg-body. I want to separate that type system into its own package for use elsewhere in VUnit. We could sync the type marks between VHDL and C, so that users could create valid VDHL data structures in C and pass them to VHDL through an ext_queue.

Finally, we could do the same for the com library. If we implemented part of it in C, we could allow users to send messages to external actors. We could even find a way to interface VUnit message passing with gRPC, so that we could send messages to any language with a gRPC implementation. It looked like you were working on something like this in one of your examples, but I haven't had the chance to read and understand it yet.

If we followed this approach, I think the result would be something that is extremely easy to work with in VHDL. Essentially, all of the current features and data structures in VUnit could be extended into C for co-simulation.

What do you think?

[umarcor]

I'm the one who is sorry now. I completely overlooked your last comment.

Instead, I would like to see VUnit manage pointers for the user. We already do this for users in VHDL, why not mimic the same interface in C?

I do partially agree, but there some corner cases. In the current example users must allocate two arrays to store the eid and length of each point, but the pointer does not need to be explicitly allocated, it might be given. For example, when using Octave + VUnit + GtkWave, data is allocated in Octave, and VUnit/C receives a struct with the pointer and some fields describing the size (be it a vector, a matrix, a cube). Assuming that the data is saved by Octave as 32bit IEEE integers, C and/or VHDL can use the pointer/data without needing to explicitly allocate another buffer. This is opposite to current VUnit's built-in and internal integer vector types, where allocation is hidden. Should we reproduce the API in C, it would not be possible to implement the Octave example I just explained.

I believe that this is not critical, but we need to consider that new_vector in C can accept a pointer as an argument, and that the type of the content might be constrained (within some limits). Actually, this is how string_ptr works, but not integer_vector_ptr.

I would propose that we allow users to provide a unique name as an additional argument to the new_ext_ptr functions in both VHDL and C.

Totally agree.

What I've proposed seems to align well with the use cases for mode extacc, but I don't think I've addressed mode extfnc. However, based on your description of extfnc, it sounds less like shared memory and more like a data stream or socket. If that was your intent, then I think it would be better implemented by leveraging VUnit queues.

I am not sure... The model you explained is based on doing a one-by-one equivalence of VHDL and C functions, so that callbacks are used. Although ext_ptr_t resembles extacc, because it contains a pointer, if strings/names are used to identify each vector, it seems that no access types are bind. Hence, I think that the implementation addresses extfnc only.

My idea was that extacc was for local shared resources (pointers allocated in the same memory space) and extfnc was for remote resources (no pointer available). If we implement ext_ptr through callbacks only, the distinction does not apply: we assume that VHDL will never know the real pointers in C; it will only be possible to get/set values through functionc. I think this is not critical, though. Using VUnit's built-in VHDL vectors it is already not possible to do direct assignments.

As commented in #603, we might later use the VHPIDIRECT feature of binding access types to implement custom records/structs.

Regarding memory vs stream, I agree that streams can be better mapped to queues. That's why I have not tried to extend external strings/integer_vectors to AXI Stream verification components. I think it will be better done after these enhancements to support external queues are merged.

We could even find a way to interface VUnit message passing with gRPC, so that we could send messages to any language with a gRPC implementation. It looked like you were working on something like this in one of your examples, but I haven't had the chance to read and understand it yet.

Currently, users are expected to copy and modify vhpidirect_user.h to replace extfnc value get/set functions with their own implementation, which would be gRPC calls. This is precisely the difference between extacc and extfnc. With extacc there is no explicit C callback when VHDL modifies a value in a shared buffer. However, with extfnc VHDL cannot modify a value in a shared buffer without using a callback.

Therefore, we need to provide a mechanism in the C sources of ext_ptr, ext_queue, etc. to allow users to provide an alternative implementation of the callbacks that get/set or push/pop values. We provide the signature of a hook function, we accept a pointer to a function as an argument, and we use it if not null. We need to also allow users to decide if they want to execute the default body after the hook or if they want to completely replace it. Moreover, I don't know if the hook should be defined per ext_ptr or if it should be a single hook for all of them.