Create New page for /tools/

[iamVP7]

Latest description for the issue -> - list can still be enhanced manually as some tools are not on GitHub, there are some on GitLab

Reason/Context

• Why we need this improvement?
  Creating new index page for tools page.

• How will this change help?
  We can say what each and every tool does.

• What is the motivation?
  We are moving pages related to tools under /tools/* so a dedicated index page for tools alone

Description

• What changes have to be introduced?
  A new page.

• Will this be a breaking change?
  New page; mostly changes are restricted to new page. If broken it wll be mostly this page.

• How could it be implemented/designed?

[ritik307]

@iamVP7 can you provide the mockup for the page. I would love to work on this

[iamVP7]

Raised this feature request during code refactor (#378 )

I feel @derberg will be good person to share the thoughts on designing new page.

[quetzalliwrites]

I would ask our design contributor extraordinaire, @mcturco for design contributions. 😀👍🏽

Lukasz is awesome, but not in design 😄🙊

[derberg]

but not in design 😄🙊

I could not agree more 🤣

---

I'm not sure we are at the stage where we can start working on it. @alequetzalli isn't it somehow connected also with new docs page? I mean there will be docs restructuring, and we also want to incorporate docs from tools, and we also want to clean up that some tools are in one location and some are now described in the documentation. A lot of factors that can influence design, @mcturco will ask questions I personally do not have a clear answer to yet. This might be a good candidate for GSoC related task, as it requires discussions with community, and driving it end to end. Don't know 🤷🏼

[mcturco]

@derberg I was about to say the same thing. Before we can start developing a new page we definitely should discuss what the purpose is if different from the tooling docs and what content we would need to present. Once we have a clear direction and content written, we can begin designing a layout for this (again, if it will be different than the docs)

[quetzalliwrites]

I'm not sure we are at the stage where we can start working on it. @alequetzalli isn't it somehow connected also with new docs page?

I don't think so? I think I see the Docs as a different item than what I understood @iamVP7 was asking to do in this issue. I understand this issue to mean he wanted to know if we could add a page to our website to mention Tools. I guess I thought he opened this issue to ask if we could consolidate all our diverse tool landing pages into a single one. We currently have several individual pages or links to see the tools, but not a single one in the website just briefly summarizing each one. Thus, I thought it would be a question for say, @derberg, @mcturco, or maybe @magicmatatjahu to start a discussion w/ community on the why for this page.

That said, if I misunderstood his question, and he was instead asking about having the Tools documented, then yes.... this would be a question for Docs.

@iamVP7, can you confirm to us if you were wanting to document the Tools (for AsyncAPI Dev Docs), or if you were asking about making a landing page in our website for all the Tools?

[iamVP7]

As per @derberg comment, we were in need of landing page for tools.

Even I felt, it might be helpful to have landing page (say something like feature page for random project), as it can help with SEO.

[quetzalliwrites]

Got it, he may have forgotten that is what you were referring to! Thank you for clarifying @iamVP7.

So if you're proposing a landing page, which is not related to Docs, then I will let @derberg and @mcturco take over this with you. You'll prob work with them to propose your solution to the community and see what the community thinks this page would need. 😀👍🏽

[mcturco]

Great thank you both for the clarification @alequetzalli @iamVP7 I am understanding the idea now 😄

I think this is a great idea! It definitely will help to have that main /tools slug not reach a 404 😅 good catch! We will first have to start thinking about what kind of content we would need to include on this new page. Are there any other sites that we can use for inspiration? I think generally we can use this new page to showcase and link out to each tool, but just thinking about what else we can include on that page.

[quetzalliwrites]

@mcturco What if it had cards in a grid, and each card represented a tool?

[mcturco]

@alequetzalli yeah I think that's a great idea! Let's do it!

[derberg]

This issue is about having a single landing page that lists all the tools for AsyncAPI, not only the ones that are maintained by AsyncAPI:

We have tools:

• here https://www.asyncapi.com/docs/community/tooling
• here under Tools button/dropdown
  

We still need pages like:

• https://www.asyncapi.com/tools/cli
• or pages with full documentation, like we plan for generator for example

We should have one single landing page under https://www.asyncapi.com/tools where:

• we have a list of all the tools for AsyncAPI
• we have nice filtering and not hard to navigate tables like https://www.asyncapi.com/docs/community/tooling
• we can still have a category that given tool is maintained by AsyncAPI, so I can filter out only tools from AsyncAPI initiative
• filters should be enabled on URL level, so I can have for example a link like https://www.asyncapi.com/tools?asyncapi to get a list of tools that are owned by AsyncAPI Initiative
• I should be able to filter out tools by languages (JS, Java), area (CI&CD, testing) or type (library, cli, web)

Key - discoverability:

• now I can add a tool by opening a PR to the website repo but it requires extra effort that could be automated. Manual == boring and pushing away
• list should not be a markdown file like now but json or yaml file that is built by some GH workflow
• list is build based on GitHub search. Take this example. Arjun should just be required to only add certain topics to the repository, maintain accurate description, and that is it. We will automatically pull the tool to the list
• list can still be enhanced manually as some tools are not on GitHub, there are some on GitLab
• we should have some alerts when a new tool gets discovered? Slack channel alert? a maybe automated Tweet?

For now I think about simple repo description and topics, might be that long term better to introduce .asyncapi file that Arjun would need to maintain in his repo and all metadata for the tooling list would be provided there in the file.

For GSoC participants

This issue is interesting for GSoC because:

• you get to do some frontend work,
• JS coding and GitHub API usage is included
• You get to learn basics of GitHub Actions
• you will need to have discussions with different folks from AsyncAPI that will show you the real job work when in theory simple solution requires discussion with multiple folks in a company. The difference is that in open source it is even more complicated 😃

[akshatnema]

@derberg Hmm 🤔 this idea looks fascinating and interesting to me. I am designing the solution in my plan(for gsoc) and looking to make a proposal on it. But, I want more information regarding this idea. As discussed by you on Slack, you proposed me this:

To make a .asyncapi file from which data is extracted using GitHub API and is updated manually by organization members. So, where should we place this file? in website repo or .github repo so that it can be accessed to every repository and can be fetched easily from other repos?

Also, later we will have a prebuilt json file built using scripts but will be under .gitignore because the file will be built during build time and then rendered on the website (Same what we do in posts.json). This file will be updated using GH workflows on the production side. This is the overall plan you give me and I thought of. Also, I am actively looking for some more suggestions on this approach from everyone 😄. We can have other approaches too for the listing of data. If you have anything more to correct me or suggest me, do tell me. It will be my pleasure to discuss with you.

Regarding filtering of tools in the list and design of the page, it is pretty much straightforward that whenever a user will filter out some tools, those filters will be updated on URLs as well as query params to search in the list. We can have a customized search bar at top of list, where categories are mentioned in a dropdown for users to search. New page (or route) /tools/ will be made easily in the repo and it can be further extended in the future to add more information related to it.

[akshatnema]

Hey @derberg, I am willing to continue to work on this issue in Mentorship program. Kindly look on my previous comment and do give your suggestions on how I should improve it.

[magicmatatjahu]

@akshatnema Hi!

To make a .asyncapi file from which data is extracted using GitHub API and is updated manually by organization members. So, where should we place this file? in website repo or .github repo so that it can be accessed to every repository and can be fetched easily from other repos?

As I understand that file should be located on the tool's repository side and then in website workflow we should filter all repos with that file and make appropriate logic. Let me know if you understand. Also I prefer .asyncapi-tool or .asyncapi-webiste due to simple fact - we can introduce .asyncapi config file in our CLI, so it's better to have a different name.

Also, later we will have a prebuilt json file built using scripts but will be under .gitignore because the file will be built during build time and then rendered on the website (Same what we do in posts.json). This file will be updated using GH workflows on the production side. This is the overall plan you give me and I thought of. Also, I am actively looking for some more suggestions on this approach from everyone 😄. We can have other approaches too for the listing of data. If you have anything more to correct me or suggest me, do tell me. It will be my pleasure to discuss with you.

We'd have to hear from Łukasz here, but from what I understand it's a good approach. However, as Łukasz mentioned you need to figure out a way to retrieve this data from the repositories via Github API, this will probably be the hardest part here.

Regarding filtering of tools in the list and design of the page, it is pretty much straightforward that whenever a user will filter out some tools, those filters will be updated on URLs as well as query params to search in the list. We can have a customized search bar at top of list, where categories are mentioned in a dropdown for users to search. New page (or route) /tools/ will be made easily in the repo and it can be further extended in the future to add more information related to it.

👌🏼 great.

[akshatnema]

you need to figure out a way to retrieve this data from the repositories via Github API, this will probably be the hardest part here.

@magicmatatjahu Yep, it is pretty much difficult to filter out the repos of the specific tools, but what I think is that if we assign specific tags to the repositories having tools, we can make a json fiile according to it. But the condition is, we need to specify tag, category or topics on the repo and each repo should contain exactly one tool in it. Using GitHub GraphQL API, we can extract the tags attached to each repository and store there information in the prebuilt.json file. The .asyncapi-tool file will contain the name of the topics, and categories that will be used to filter out the tools from the repositories. Additionally, some manual tools can also be added to asyncapi-tool file, which will be later added to the prebuilt.json file during build time.

What I didn't get is the GitHub search that Lukasz mentioned in his previous comment. Some Github search implementation has been made in the cupid repository by Arjun, but I wasn't able to find in it. If you know, kindly send me the reference link of that particular function in the codebase.

[magicmatatjahu]

@akshatnema

The .asyncapi-tool file will contain the name of the topics, and categories that will be used to filter out the tools from the repositories.

Yep, but it should contain also another info like description, title, maintainer etc. Of course we can also extract that info package.json, but we should discuss it.

What I didn't get is the GitHub search that Lukasz mentioned in his previous comment. Some Github search implementation has been made in the cupid repository by Arjun, but I wasn't able to find in it. If you know, kindly send me the reference link of that particular function in the codebase.

Łukasz specified cupid as repo for searching - given repo will have topics and .asyncapi-tool file and based on that we will filter repos :) It's only an example for your task.

[derberg]

@akshatnema I love how far detailed you went with the research. The final solution is not written in stone, like file names and stuff like that. The only strong requirement:

• not PR based but discovery based
• enable an easy way to add support to other platforms for discovery, like GitLab (I know about at least one AsyncAPI tool that is hosted there, docs generator for Asciidoc)

I'm not sure if topics is a better solution over .asyncapi file. IMHO best to propose 2 alternative solutions and show to some tooling owners (that will be users of this solution) to say what is the best for them. For sure .asyncapi is easier to extend in future with more metadata, more than you can get with topics

Super happy to see you engaged here so much. Will advertise it among TSC to make sure it is selected as it is going to be a great addition for the community IMHO

[akshatnema]

Hey @derberg, I understood your previous comment and thus, came up with another approach to target this. Do give your views on it. Here it is as follows:

• First part of the approach - We will be using Web scraping here on the JSON data provided by GitHub and GitLab APIs because we can only rely on these APIs since most of the tools are present here only (Do tell me if we have a centralized place/file to collect metadata of all tools we want to target in this issue). Using GraphQL APIs, we will get the description and other required information of all the repositories in AsyncAPI. This data(in JSON) will not be stored permanently but will be used for scraping purpose.

• Second part of the approach - The .asyncapi-tool file will contain the necessary tooling words which will be used to identify the required repositories containing the tools of AsyncAPI. The words or the keywords in the file will be manually added using the PRs and can be changed in future to more customize our scraping process as well as categorizing the various tools in different sections/topics. The file will also contain the other necessary information like the category names in which the tools are to be categorized on the website and the schema of the details needed for each tool that will be required while rendering it on website. In short, this file contains all the metadata related to the scraping process and the tools.

• Third part of the approach - The description (if needed, the topics) of all repositories containing the tool will be updated in such a way that they describe the tool as well as they get identified the scraper function to add its details (acquired from the API) in the tools.json (which will be built on the build time). The text in the description will be scanned on the given keywords in the .asyncapi-tool file and hence will be added in tools.json. This file is then used to render data in the respective components in the website.

Key points to consider

• Since, we are using web scraping, it's not going to be a fast process. Scanning around 60+ repositories in the JSON will take some time but since we are using JS, it will not affect the build time. (:thinking: Maybe it will take less time than build time of posts.json).
• This approach will not make some weird additions on the repository. As mentioned by you earlier, we won't be adding too many topics in the repository to identify the tool. We are using the Description of each repo for the identification and categorization.
• The description of each repository needs to be precise so that the scraper won't identify a non-tooling repository as a tool and add it in the tools,json.\
• I made the approach mentioned above keeping in mind that all the tools are present either in GitHub or GitLab. If it is not so, do tell me about the other sources. OR do we have any centralized file/place which keeps account of every tool used in AsyncAPI.
• An alternative for the description can be to use a certain prescribed file in the repositories that describes a tool and the file contains all the necessary topics/tags needed to identify and categorize it.

Waiting for an awesome reply from Lukasz (@derberg). I will be happier if @magicmatatjahu reviews this one as it deals with some code and algorithms as well 🙂.

[magicmatatjahu]

I think you guys like to make life difficult for yourselves. When I read about the .asyncapi-tool approach I immediately had in mind situations where we use this file on the repository side to describe the tool (its metadata), and you describe about scraping data and identifying by description etc...

(Maybe you have the same thing in mind as me, but reading your previous comments I'm confused and don't know if I understood your suggestions correctly)

My suggestion:

• the owner of a given repository adds a topic (tag) to his/her repository asyncapi-tool.
• he/she creates an .asyncapi-tool file where provides information about the project/tool like description, title, maintainers, href, docs (documentation link), category (of tool, like template, parser etc), maybe also website etc.
• on building the website we download all the repositories from github using the asyncapi-tool topic, download the contents of the .asyncapi-tool file and we have all the metadata - for other sources like gitlab we should check if there is possibility for similar solution.

If someone provides additional topics he/she will be able to do that, but the most important thing is that there should be asyncapi-tool.

[derberg]

Definitely let us not consider web scrapping as a possible solution. GitHub search allows you to search for specific file in repos and as result, you get a list of repos. Then, afaik API allows you to just grab the content of the specific file, without cloning (something to confirm)

tl;dr

• we do not need topics at all
• all info for list of tools should be in .asyncapi-tool file
• just take advantage of GitHub API

[akshatnema]

Yep, we can extract the file contents but not directly through the Github search. What you mentioned in your message are the features used to find files/repos using the Github search bar but we need want it using Github API. I found this in the Github REST API. I think using the above-mentioned API, we can find the file present in the respective repo but need to access the content which is quite hard as the API is not returning the contents of the file directly. We need to discuss regarding this on how to extract the contents of the file using the present JSON.
For finding the AsyncAPI repositories using API, we can use this GET request of GitHub API.

As mentioned by you, if we only need to add .asyncapi-tool file in each tool repository and can extract data from it, then it is quite an easy and straightforward implementation that can be implemented inside scripts/ folder. We only need a addition of .asyncapi-tool file added to each tool and if it is not possible, the tools will be manually added to the /tools/ in the website.
Do you need to add anything more in this that I missed and is important?

[derberg]

• you have some example of search in action here: https://github.com/derberg/npm-dependency-manager-for-your-github-org/blob/main/lib/api-calls.js
• search returns 2 props, repository with typical repository object where you can grab org/profile and repo name + path where found file is actually located in repo, root? maybe .github?
• then you call for example https://api.github.com/repos/asyncapi/website/contents/README.md
• decode content string with base64

and enjoy life 🍹 🏖️

the only thing now is I think to agree on the structure of the .asyncapi-tool file

[akshatnema]

then you call for example api.github.com/repos/asyncapi/website/contents/README.md
decode content string with base64

As what I see in the API docs and its schema, we get a url variable containing the link of the contents inside the file (in base64 encoded form). The schema of the url is as follows:

 "url": {
            "type": "string",
            "format": "uri"
          },

We can call the GET request on this url to get the contents and then convert them into the normal text then. Do confirm if you want to convey this only?

the only thing now is I think to agree on the structure of the .asyncapi-tool file

I will try to make one draft schema in the upcoming days but in the format of a javascript object (that's my most preferable way 😅). If you want it in another format, do tell your suggestions.

[derberg]

We can call the GET request on this url to get the contents and then convert them into the normal text then. Do confirm if you want to convey this only?

sounds good to me

I will try to make one draft schema in the upcoming days but in the format of a javascript object (that's my most preferable way 😅). If you want it in another format, do tell your suggestions.

yeah, leave out schema for now, first focus on example JSON objects. Let us agree on the structure, and then to get schema out of it later is easy. Definitely do not start with schema first

[akshatnema]

@derberg @magicmatatjahu I'm not sure if you have read my previous comment #383 (comment) or not. Do check that out also and give your opinion. Also, I have some queries:

• We decided that we will have final JSON file created by backend and it's format will be on the basis of the categories. In this case, we have to specify what will be categories, we will be allowing for tools to get added in UI. We have to specify some terminologies like generator, cli, etc. Can you specify these categories?
• as discussed, a github action will be there to run the script and add the tools information in the repo, my question how we are gonna manage this action. Will this be a part of existing meetings and newsroom github action or a new action has to be made for this?

[akshatnema]

@mcturco as I can see in the design, you have shown the Free or Paid tag on a tool by making a tag-type component placed over the icon. But if a case arrives that there is no tool icon, what should be the card design then?

[akshatnema]

@derberg @magicmatatjahu I have a suggestion in my mind right now. We discussed that we will have specified background color and border color for each tag we will be using in dashboard. So, initially, we came up with the idea during the meeting that we create the final JSON in the backend side and then do the assigning of tags with colors in the Frontend side using Fuse and the defined color-schema file. Now, the drawback of this implementation is that every time a user goes to the page, the app will do the same computations to find the tags and their respective colors again and again, for each user. What if we come across a situation where we have enough number of tools and the tags inside them so that this computation take > 0.5 seconds to get executed and show up on the website? It may can slow the loading of the webpage. So, I get here that we are doing the same stuff repeatedly for each website loading.

My new idea is, inside each tool element, we can expand the language and technologies to the array of javascript object, instead of just a string. Inside the object, we will specify the correct name of the tag, its background color (in Hex) and border color. Also, the assigning of correct tags to each tool, using Fuse js, will also be implemented in the backend, not in next js side. All these computations will be done in scripts (backend side) only and the tools JSON data will be formed completely UI ready here.

Give your reviews on this whether we should do this on backend side or frontend?

[mcturco]

@mcturco as I can see in the design, you have shown the Free or Paid tag on a tool by making a tag-type component placed over the icon. But if a case arrives that there is no tool icon, what should be the card design then?

Good point @akshatnema!

What if we do something like this:

I am also thinking that it might be better just to leave the avatar out altogether. What do we think about that?

[akshatnema]

I am also thinking that it might be better just to leave the avatar out altogether. What do we think about that?

Ahh, this can be a good way but I can't decide of my own 😅. At the initial, I also didn't came up with the inclusion of tools icon but @derberg suggested me in the meeting to include it. I still ask, whether we should have it? Also, I like the simplified design of right side, and the FREE tag is also properly placed according to me.

[mcturco]

Yeah I am just wondering how many will actually have an avatar, and even then it might look a little cluttered. I would say if we don't need them then we should remove it and go with new card (on the right in the screenshot above)

[magicmatatjahu]

@akshatnema Yeah, you should compute colors in the backend.

About categories and workflows, please wait for @derberg answer.

[derberg]

what categories and workflows? 😄

@mcturco yeah, every dev has avatar, but I do agree, getting these in the UI will be hard, especially if there are many maintainers. The only solution is to have maintainers icon and on click getting a modal with all details. But IMHO this can be done as a followup from initial implementation

[mcturco]

The only solution is to have maintainers icon and on click getting a modal with all details. But IMHO this can be done as a followup from initial implementation

@derberg Agreed!

[akshatnema]

what categories and workflows?

I'm talking about the categories field in the .asyncapi-tool file, which is inside the filters and secondly, I was talking about the github action we will use to update the tools.json file via the functionality added inside the scripts folder.

[derberg]

workflows

can be done as you do it now, separate workflow, just make sure there is a 10-30min difference between the new workflow schedule and tools, as both end up with PR, so yeah, we just need to make sure they are not created at the same time, as they can create timeouts

categories

just the ones that are in current list on website:

• code first
• code generators
• converters
• validators
• directories
• documentation generators
• dls
• frameworks
• ui components
• mocking and testing
• diff (old compare)
• ci&cd (old GitHub Actions)
• editors (not on website but we need it as studio will go here, also vs code plugin and others)

now, I do not have a good idea how we will qualify the ones that are not here yet, like bundler, cupid, optimizer... AsyncAPI documents management? I'm not 100% sure

[akshatnema]

can be done as you do it now, separate workflow, just make sure there is a 10-30min difference between the new workflow schedule and tools,

The workflow has been designed in the PR. You can review it.

[derberg]

@akshatnema looks perfect

[akshatnema]

Hey @mcturco, can you please update the tool card in the figma according to the right side card in image #383 (comment). After having a discussion with @derberg and @magicmatatjahu #939 (comment), we decided to exclude the field of iconUrl from the schema and hence, we will not be extracting avatar of tools to show it in UI.

Regarding the maintainers, we have excluded the maintainers information for the initial release of this implementation. We can later discuss how to add it and handle the maintainers in the Tools card.

Also, can you please make the tags for languages and technologies so that I could take their backgroundColor and borderColor to use them in UI?

[akshatnema]

@mcturco since we have so many filters in a single dropdown, should we have a Apply button at the bottom to apply the filters when the user is final about the filters he wants to make? Presently, according to UI, if a user selects a filter, the cards will be sorted automatically on each selection.

[akshatnema]

Summary of Meeting held on 3rd October 2022

Participants: @akshatnema @Mayaleeeee

Agenda

• Discuss the project and it’s goal
• Discuss the addition of Apply Filters button
• Discuss on adding all Languages and Technologies tags in Figma
• Any suggestions regarding UI in Figma

Meeting Notes

• Explained most of the parts of the project to @Mayaleeeee. Hope she has understood everything.
• We discussed the designs and come up with the conclusion to do the changes given above.
• I have to give the list of Languages and Technologies to @Mayaleeeee for the creation of tags.
• @Mayaleeeee will be contributing more to the design and UI of the page, if needed further from reviews.

cc: @derberg @magicmatatjahu

[devarsh10]

Okay. Thanks for replying.

…

On Sat, 27 Aug 2022, 12:09 am Akshat Nema, ***@***.***> wrote: Is the new page for tools is created or not? If not, I want to add a new page. Hey @devarsh10 <https://github.com/devarsh10>, this project idea is part of my AsyncAPI Mentorship program. So, I've been assigned to make the /tools/ page. It will be better if you look at other issues to contribute to the repository. Thanks! — Reply to this email directly, view it on GitHub <#383 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AT4MQN3RVR2AXNKLYR4YGFLV3EFPHANCNFSM5EVV7WCA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[akshatnema]

@derberg As we can see that the functionality to extract the information of tools via GitHub has been added, can we now start to make it public to Tool Owners to make .asyncapi-tool file in their repositories? And also, we do need official documentation on how the contributors will make the file in their repositories and the key terms for fields. Should we start working on that also?

[derberg]

@akshatnema I think we first need:

1. UI ready
2. Get current list of tools on the new Tools UI
3. Get stuff documented, yup

then we can do some promotion on different channels