DataONE CN indexing on k8s

[gothub]

@mbjones @taojing2002

What are the steps needed to setup k8s for CN indexing? I'll start with a few basic items:

• determine k8s namespace and setup authentication
• setup persistent volume
  • how much space is needed?
  • should it use a static CephFS subvolume or dynamic RBD volume?
• install Solr Cloud on k8s
  • this Lucid doc shows a setup for Google GKE, but this could be adapted for our bare-metal setup.
• how will indexing be split among k8s pods?

[taojing2002]

Hi Peter: Thank you for your thoughts about this. I am thinking the first step is to set up solr on K8s. Running indexer on k8s needs the code modification and it should be the next step. I think we need three solr instances on k8s: - cn-sandbox-solr - cn-stage-solr - cn-production-solr The first two would be in the dev cluster and the last one would be in the production cluster. I checked the sizes of the directory of /var/solr on cn-stage-ucsb-1 and cn-ucsb-1. They are 47G and 179G respectively. So I think 100G will be good for the test clusters and 500G for the production cluster. What do you think? Thanks, Jing

…

On Wed, Jan 19, 2022 at 10:12 AM Peter Slaughter ***@***.***> wrote: @mbjones <https://github.com/mbjones> @taojing2002 <https://github.com/taojing2002> What are the steps needed to setup k8s for CN indexing? I'll start with a few basic items: - determine k8s namespace and setup authentication - setup persistent volume - how much space is needed? - should it use a static CephFS subvolume or dynamic RBD volume? - install Solr Cloud on k8s - this Lucid doc <https://lucidworks.com/post/running-solr-on-kubernetes-part-1/> shows a setup for Google GKE, but this could be adapted for our bare-metal setup. - how will indexing be split among k8s pods? — Reply to this email directly, view it on GitHub <#22>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB5QQDBMW54JO6LWG44SHX3UW35JLANCNFSM5MKUO3BA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>