How to implement monitors systems ?

[syphax-bouazzouni]

Currently, at Agroportal, we have an intern @haddadzineddine for 9 months, and his first mission would be to implement a monitor system for our systems (Agroportal and SIFR bioportal)

The goal here is not just to implement a specific tool, that will only be for us (Agrportal). But instead, have something that can be added directly to the appliance and works directly with just configuration.

This monitor tool, need to

• Be free and easy to deploy and maintain
• Give the system status
• Is a log aggregator ( a common place for all our log files )
• Give the system performance and usage
• Give request errors

The following questions will be mostly meant for the @ontoportal/bioportal team because I think that you are the only ones to have it implemented.

• Do you have an idea of which tool we should use? (I know from our later discussions, that for example, NewRelic is already in the appliance)
• Can we get help from your team on this topic? (especially @alexskr) . I think about setting up one meeting to discuss how it works and how to implement it.

Summary of previous discussions on this topic

From Slack (16/11/2021) :

• John (@graybeal) :
  • Starting from scratch, I would set up StatusCake (or any simple monitoring tool) and have it monitor some basic APIs and UIs (see bioportal statuscake)
  • Nagios and Monitis are more complicated to use, but because they are free for many more tests, it's a logical next step if you have time but not money.
  • NewRelic is very sophisticated and valuable for deeper-dive troubleshooting (as ELK would be if we had time to maintain it)
  • You may choose not to worry about that too much unless you are seeing significant reliability/troubleshooting issues
  • Use Google Analytics for traffic monitoring
• Alex (@alexskr ) :
  • Newrelic is integrated into the ontoportal to some extend; all we need to do is create a newrelic account and add API keys to the appropriate configuration files

[graybeal]

A key question: Does the solution need to be compatible with distributed portals (as in, REST calls and logs spread across 3 different machines, UI logs on 2 different machines, etc.) or is it only for 1-machine systems? Because BioPortal needs, you know, the distributed kind. But I bet nobody else does yet.

In any case, I'm sure we're happy to share insights, and setting up a call is a good idea. It would also be nice to have a place where we all could collect requirement ideas.

In a related note, Misha has the job of preparing some 'real-time monitoring' tests that could be used (in a months or two) to evaluate BioPortal production performance with 4store, and then with Allegrograph. This wouldn't be a huge project but some of the ideas he's collecting may be relevant to this task.

[mdorf]

I am working on a simple script (or a set of scripts) that would execute queries against known OntoPortal endpoints (possibly allowing a list of ontologies as an input parameter) and time the results. We will use this to capture some baseline stats about the system as well as means of comparing the performance of our system under 4store vs AllegroGraph.

[alexskr]

BioPortal relies on a number of monitoring solutions that we have deployed over the years. 


1. Nagios. It is in place for basic infrastructure monitoring and alerting on the problems with CPU, RAM, disk, and unresponsive services.
2. NewRelic APM. APM collects telemetry from the API and UI, including web response time, throughput, error rate, traces, logs, and provides useful dashboards that provide better application visibility. We can use it to analyze which transactions take too much time; what types of errors are on the rise and should be addressed. 
- It is relatively easy to set up. NewRelic gem is already bundled in with UI/API so all you need to do is to create an account at NewRelic and provide API key. NewRelic is one of the top leaders in the APM market but costs $$$$$$. We deployed it a long time ago when NewRelic was offering free but limited licensing to academic projects. NewRelic has a free plan with up to 100G/Month of data ingest rate which could work for small deployments. 
- NewRelic has an infrastructure monitoring feature which collects telemetry on things like Redis, Memcached. We have also experimented with sending 4store statistics (running and outstanding queries) to NewRelic. While we do not have data ingest limits for NewRelic, we do not have access to some of the advanced features such as infrastructure monitoring.
3. External uptime monitoring services which do full page loading for set UI pages as well as API requests from multiple locations around the world. We used to use monitis for it until it was acquired by teamviewer so are planning to replace it with something like StatusCake. Web Monitoring sites like StatusCake offer free limited tiers; however, you are likely to pay for advanced features such as full page load or multi location tests.
4. We used to have an on-premises ELK stack in the past for aggregating API, UI, and 4store logs and are considering replacing NewRelic with ELK on elastic cloud/SAS since it could be more cost-effective. ELK stack is OSS so you can run it yourself if you have sufficient compute/storage resources. It would also be a good idea to look into opentelemetry tools and use them with things like SigNoz.io

[haddadzineddine]

Hello @alexskr ,

Thanks for all this useful information, I think in our case we will use NewRelic & StatusCake,
Can you please go into more detail on how to use these APMs step by step with ruby on rails?

Or perhaps a useful resource that can help.

Thanks in advance.

[alexskr]

1. Register for a new NewRelic account and create two services, one for UI and the second one for the API.
2. To create a new service you need to click on "Add data", select Ruby
3. In the 'Add your Ruby application data' give your application a name(i.e MyOntoPortal_UI), click on download your custom configuration file (newrelic.yml) and save it in the appliance in the /srv/ontoportal/virtual_appliance/appliance_config/bioportal_web_ui/config directory.
4. Optionally you can install newrelic infrastructure agent with a provided Linux install script.
5. Repeat steps the same steps for the API, creating a second service, downloading newrelic.yml and placing it in the /srv/ontoportal/virtual_appliance/appliance_config/ontologies_api/config
6. Perform UI and API application deployment described in the ontoportal admin guide

[graybeal]

StatusCake is just an external API tester, as we use it. It's trivial to configure for whatever tests you want to use it for, you can sign up for a free account and try it out. https://uptime.statuscake.com/?TestID=uR5YnbjHjw shows all the StatusCake tests we use for BioPortal (they are the APIs used by CEDAR). Unfortunately I can't make the download option work to share those configuration, though StatusCake says I can... Anyway, it's not a very long list and I think you should consider making your own, but here are the URLs corresponding to these tests:

• http://data.bioontology.org/ontologies/GAZ/classes/?include=prefLabel,hasChildren,created,synonym,definition&page=1&pagesize=50
• http://data.bioontology.org/ontologies/ICD10/classes/http%3A%2F%2Fpurl.bioontology.org%2Fontology%2FICD10%2FA69.2/tree
• http://data.bioontology.org/ontologies/ICD10/classes/http%3A%2F%2Fpurl.bioontology.org%2Fontology%2FICD10%2FA69.2/children
• http://data.bioontology.org/provisional_classes/?page=1&pagesize=50
• http://data.bioontology.org/ontologies/CEDARPC/provisional_classes/?page=1&pagesize=50
• http://data.bioontology.org/search?q=disease&also_search_provisional=true&valueset_exclude_roots=true&ontologies=ICD10&include=prefLabel,definition&page=1&pagesize=500&display_context=false&display_links=true
• http://data.bioontology.org/ontologies/FMA/classes/roots?include=prefLabel,hasChildren,created,synonym,definition
• http://data.bioontology.org/ontologies/BIBFRAME/properties/roots
• http://data.bioontology.org/search?q=delivery+procedures&also_search_provisional=true&ontology_types=VALUE_SET_COLLECTION&valueset_roots_only=true&include=prefLabel,definition&page=1&pagesize=500&display_context=false&display_links=true
• http://data.bioontology.org/search?also_search_provisional=true&valueset_roots_only=true&ontology_types=VALUE_SET_COLLECTION&ontologies=CEDARVS&page=1&pagesize=50
• http://data.bioontology.org/ontologies/CEDARVS/classes/http%3A%2F%2Fwww.semanticweb.org%2Fjgraybeal%2Fontologies%2F2015%2F7%2Fcedarvaluesets%23Study_File_Type?include=prefLabel,hasChildren,created,synonym,definition
• http://data.bioontology.org/ontologies/NLMVS/classes/http%3A%2F%2Fpurl.bioontology.org%2Fontology%2FNLMVS%2F2.16.840.1.113762.1.4.1045.59/children?page=1&pagesize=5000&include=prefLabel,hasChildren,created,synonym,definition
• http://data.bioontology.org/property_search?q=title&page=1&pagesize=500&display_context=false&display_links=true
• http://data.bioontology.org/ontologies/BIBFRAME/properties/http%3A%2F%2Fid.loc.gov%2Fontologies%2Fbibframe%2FmainTitle
• http://data.bioontology.org/ontologies/BIBFRAME/properties/http%3A%2F%2Fid.loc.gov%2Fontologies%2Fbibframe%2Fdate/tree
• http://data.bioontology.org/ontologies/BIBFRAME/properties/http%3A%2F%2Fid.loc.gov%2Fontologies%2Fbibframe%2Fdate/children