Filter Discussion - where (value = x or value is null)

[seanmalbert]

Our next step on this work will be to figure out the interaction between sorting and filtering. The high-level idea is: if a request indicates that we're filtering on column X and some listings have data in column X while others have null, we'd like to display the listings with data in that column above those that have nulls. If you all have thoughts on that, I'd be happy to discuss. Otherwise I'll send a PR for that change once it's ready.

---

Hey @anders-schneider , I have a question, if you're filtering on column X, you also want to return data where column X is null? It makes sense that if we're sorting on column X that we would want to show the data where column X is not null before the ones with null, but if we're filtering on it, shouldn't we return the ones that only have the requested filter value? I suppose there could be valid use cases where you'd want that, but I think we'd want to make that clear to the end user that that's what's getting returned.

Good question, and yes agreed that for some filters (maybe zip code? neighborhood?) if a listing has a null value, we probably don't want to include that listing in the results. But for others (rent range? unit size?) we may want to include listings that don't have that data at the bottom of the search results. Our plan was to consider each type of filter and make a reasonable decision.

And to your point ("make that clear to the end user that that's what's getting returned"), @avaleske suggested adding a UI component with some explanatory text as a "break" between the matching listings (above the component) and the "we don't know if these match" listings (below the component).

We are still thinking about the right approach here, and would appreciate any thoughts/ideas from your side!

Originally posted by @anders-schneider in #1817 (comment)

---

@avaleske suggested adding a UI component with some explanatory text as a "break" between the matching listings (above the component) and the "we don't know if these match" listings (below the component).

Adding to this, we will be launching with a lot of incomplete listings data, especially around accessibility. To avoid filtering out listings that might match, but for which we don't have data, we plan to add some filtering options such as "include listings with missing accessibility data" with a prompt to contact the property manager.

[seanmalbert]

@anders-schneider and @avaleske ,
Moved the discussion to this thread, so we can continue if necessary. I like the idea of adding an option for the user to select/check to include listings that don't have a value for that field. Each field that had this as a possibility could have a corresponding checkbox underneath or next to it, that says "Include listings with missing data", as @avaleske suggested.

A possible way to handle this with the updates to the filter logic to allow for arrays of objects, we could allow for a new key like $include_null: boolean, so before stringifying, an example would be:

filter: [
    {
        $comparison: "=",
        zipCode: <zipCode>
    },
    {
        $comparison: "=",
        unitSize: <unitSize>,
        $include_null: true
    },
],

[avaleske]

Oh I like this. We had been starting to build this by adding additional comparisons that specify "allow results with nulls". So "=" is regular equality, and the comparison "=U" is "equality including missing values."

But I like that this is more explicit, and we can easily add this to the BaseFilter class.

(@pfeeny5 since you're currently working on this. We can chat more tomorrow in our meeting.)