Parallelism to improve grouping / sorting / filtering performance

[mickeyl]

One of my bottle-necks here (in the range of 100.000 elements in a collection) is applying the grouping, sorting, and in particular filtering blocks. As far as I can see we're only using one thread here, I wonder whether it would be possible to distribute that workload.

Applying a filtering block to one key, element, and/or metadata has no effects whatsoever on other filter blocks, hence I would think they could run in parallel.

Is this something to pursue further or not an issue at all for you guys?

[dezinezync]

I was going through the source and this is a snippet from the FilteredViewTransaction code:

if (block == NULL) return;
	
if ((options & NSEnumerationReverse) == 0)
{
	// Forward enumeration
	
	std::vector<int64_t>::iterator iterator = vector->begin();
	std::vector<int64_t>::iterator end = vector->end();
	
	NSUInteger index = 0;
	BOOL stop = NO;
	
	while (iterator != end)
	{
		int64_t rowid = *iterator;
		
		block(rowid, index, &stop);
		
		if (stop) break;
		
		iterator++;
		index++;
	}
}
else
{
	// Reverse enumeration
	
	std::vector<int64_t>::reverse_iterator iterator = vector->rbegin();
	std::vector<int64_t>::reverse_iterator end = vector->rend();
	
	NSUInteger index = vector->size() - 1;
	BOOL stop = NO;
	
	while (iterator != end)
	{
		int64_t rowid = *iterator;
		
		block(rowid, index, &stop);
		
		if (stop) break;
		
		iterator++;
		index--;
	}
}

I'm not sure I fully understand how vectors work in this case. We can convert this to use - [NSArray enumerateWithOptions: usingBlock:] with the NSEnumerationsConcurrent option, however, that has been reported to be not thread-safe so I'm not sure how it affects YapDB in this case.