k6/html: Have a way to get the selection out of an element

[mstoykov]

Feature Description

As reported in https://community.k6.io/t/create-selection-from-element/4643 it is comment in jquery to do selection.each and then get the selection of the element that each will give you and continue selecting.

This is currently not possible in k6, but we do have the selection of the element as part of the element so it shouldn't be that hard.

This likely though needs some though on the API

Suggested Solution (optional)

We can just have new Element method selection() and be done with it. But it likely won't work a lot like jquery, how important is that is different question.

We can defined $ to mean something in k6 - maybe by calling a method on k6/html. And then use that 🤷.

But it will be confusing as $('li') will not really mean anything as we don't know which page to query.

Maybe it can only be difined in each 🤷

Already existing or connected issues / PRs (optional)

No response

[Azhovan]

@mstoykov i would like to work on this issue. can you please assign it to me?

[Azhovan]

I would like to go ahead! and add my proposal here, but before that I think it is also beneficial to discuss the first proposal by @mstoykov :

Challenges

• This solution might not match the actual functionality of the k6/html module with the user experience and expectations. And the reasoning behind it is because jquery has many features and methods that are not available in the k6/html module yet and it is not clear how they could be implemented.

• The second challenge has been already mentioned in the proposal above, and Im not going to add more to it:

  But it will be confusing as $('li') will not really mean anything as we don't know which page to query.

• The third challenge with the proposed solution is that it mentions that maybe $ can only be defined in each method, However, this might limit the usability of the $ symbol, because it would only work inside the each method, and not in other places(where the user might want to use it). For example, if the user wants to get the text of the first <li> element, they would have to write something like:

 $('li').each(function(index, element) {
   if (index == 0) 
      console.log(element.text()) 
})

instead of simply writing $('li').first().text().

Alternatives

As an alternative to the suggested solution, I would propose to use a different syntax for getting the selection out of an element, instead of adding a new method to the Element type. For example, instead of writing element.selection('p'), which might be confusing with the existing selection function, I would write something like element.$('p'), which is more similar to the jquery syntax and less ambiguous.

The $ symbol would be a property of the Element type, not a global variable or a function, and it would take a selector string as an argument and return a Selection object that contains the element itself and any other elements that match the selector. e.g. if element is an Element that represents a <div> element, then element.$('p') would return a Selection object that contains all the <p> elements inside the <div> element.

Challenges

Even though this solution won't have the downsides of the first proposal, it would have some drawbacks too:
This syntax only works when there is an Element object to call the $ property on. It does not work when there is no Element object, such as when the user wants to get the selection out of the document, which is the default scope of the $ symbol in jquery. For example, in jquery, the user can write $('li') to get a collection of all the <li> elements in the document. Therefore user would have to write something like parseHTML(response.body).$('li'), which is more verbose and less convenient.

Improvements

• I think one way to improve this solution and make it more more convenient is to use a global variable or a function to store the parsed HTML document, and then use the $ property on it to select elements. For example, instead of writing parseHTML(response.body).$('li'), which is verbose and inconvenient, the user could write something like doc.$('li'), where doc is a global variable or a function that returns the parsed HTML document.

• The second solution to improve this solution and make it more more convenient is to use a different symbol or a keyword to select elements from the document, instead of using the $ symbol. For example, instead of writing parseHTML(response.body).$('li'), which is confusing and inconsistent, the user could write something like html('li') or select('li'), where html or select is a function that takes a selector string as an argument and returns a Selection object that contains all the elements that match the selector in the document.

what do you think?

[mstoykov]

I am glad you want to work on this @Azhovan 🎉

I am not certain we can use $ at all too, but looking at some of your other comments, namely:

It does not work when there is no Element object, such as when the user wants to get the selection out of the document

I have to reiterate that k6 just isn't a browser and does not open pages or run it's javascript within their "scope". And you can have multiple pages that you have just "downloaded". Namely using somethign like http.batch.

So in those cases a global documetn is nto ... possible.

All in all I would prefer if we do not even try to make a global document as that will just make things more confusing.

Given the changes in the last 1 year around async and group and ... else (see #2728 for a very long discussion) ... I don't think having magical $ or anything esle within the each callback is good idea either.

Arguably a $ (or any other identifier for example selection) can get the selection out of an element as in

response.html().find('li').each(function (index, element) {
  let container = $(element).closest('div.listContainer');
  ...
});

But again response.html() or somethign similar where response specifies the page or html that it needs to be parsed is required as there is no global page in which k6 runs code.

I am not certain how much better this is from element literally having a .selection(), but it might be more jqurey like 🤷

[allansson]

The $ symbol would be a property of the Element type, not a global variable or a function, and it would take a selector string as an argument and return a Selection object that contains the element itself and any other elements that match the selector. e.g. if element is an Element that represents a div element, then element.$('p') would return a Selection object that contains all the p elements inside the div element.

I've never seen a $ being used in a method/property name unless to mark it as an observable or something internal/hidden. And, in my opinion, it reads horribly. It's hard to for the eye to parse and the non-descriptive name means I would have to look it up in the docs in order to understand what it does.

We can defined $ to mean something in k6 - maybe by calling a method on k6/html. And then use that 🤷.

But it will be confusing as $('li') will not really mean anything as we don't know which page to query.

I see no need to try and be jQuery. We have the Selection API. Reading the forum post, I don't think the author even expects that the API would be using $. They're just using jQuery as a reference.

Why not just a method Selection.fromElement(element: Element)?:

doc.find("li")
  .each((_, element) => {
    const listEl = Selection.fromElement(element)
      .closest('ul')
  })

We're not married to WHATWG or W3C either, so maybe Element extends Selection?

doc.find("li")
  .each((_, element) => {
    const listEl = element.closest("ul")
  })

@mstoykov's alternative is also good:

element.selection()
element.toSelection()
element.asSelection()

[w1kman]

Would be neat if DOM traversal in k6/html and k6/experimental/browser had similar APIs.

[mstoykov]

@w1kman can you expand on this, please.

[Azhovan]

@w1kman

import http from 'k6/http';
import { Browser } from 'k6/experimental/browser';

// Using k6/html module
export default function () {
  let res = http.get('https://abcd.com');
  // ...
  // ...  
  let dom = res.html();
  let title = dom.querySelector('title');
  let description = dom.querySelector('meta[name="description"]');
  console.log('Title: ' + title.text());
  console.log('Description: ' + description.attr('content'));
}

// Using k6/experimental/browser module
export default function () {
  let browser = new Browser();
  let page = browser.open('https://abcd.com');
  page.wait();
  let title = page.querySelector('title');
  let description = page.querySelector('meta[name="description"]');
  console.log('Title: ' + title.text());
  console.log('Description: ' + description.attr('content'));
  browser.close();
}

If I'm not mistaken you mean that e.g. the codes above are very similar the only diff is between let dom = res.html(); and let page = browser.open(...) right?

Im not sure how much this experimental api is being used by users. but overall I think this is a good idea!

[Azhovan]

what do you think @mstoykov ?

[w1kman]

@Azhovan @mstoykov

In k6-browser the user can use page.locator(<css-selector/xpath>) to find elements in the current Document. It would be great if the custom selectors in k6-browser and k6/http. parseHTML would return the same-ish (k6-browser can interact with the DOM, so I'd expect it to have an extended Element API) Element.

Personally I'd rather use vanilla Document and Element, but for those who want synthetic sugar, it would be nice if they didn't have to learn two different flavors to use k6.

[Azhovan]

@w1kman @mstoykov
I just created a PR! 👆🏻

[Azhovan]

@mstoykov
i think we can close this?

[mstoykov]

@Azhovan, ah yeah, I thought it was marked in the PR/commit, but apparently not.

closed with #3519