Pinned Loading

1. creating_Phi2_MoE_using_mergekit.md creating_Phi2_MoE_using_mergekit.md

   1

   # microsoft/phi-2 for creating Mixture of Experts (MoE)

   2

   The [microsoft/phi-2](https://huggingface.co/microsoft/phi-2) is a small language model with 2.7billion parameters. Because of its small size, opensource license and thanks to finetuning techqniques like QLoRA, one can (fairly) quickly finetune a base model for performing downstream tasks and creating an expert phi-2 model. It would be interesting to combine the individual experts into a Mixture of Experts (MoE) to make the MoE perform the tasks of the individual experts. Follow the steps below to create your own version of a MoE based out of phi-2.

   3

   - Special mention to [Maxime Labonne](), [Aratako](https://github.com/Aratako), [Paul Ilioaica](https://github.com/paulilioaica) for showing the opensource community that the mergekit can be tweaked to make a MoE out of phi-2 experts.

   4

   - Big shoutout to [Charles O. Goddard](https://github.com/cg123), the author of mergekit for creating and letting us play with [mergekit](https://github.com/arcee-ai/mergekit)

   5

2. mergekit mergekit Public

   Forked from arcee-ai/mergekit

   Tools for merging pretrained large language models.

   Python 19

3. berts.cpp-on-android berts.cpp-on-android Public

   Forked from yilong2001/berts.cpp

   CPP inference of Bert-model family based on GGML, supporting sentence classification models, seq2seq text generation models, etc.

   C

4. Object-Detection---SSD-in-Tensorflow Object-Detection---SSD-in-Tensorflow Public

   Single Shot Multibox Detector implementation in Tensorflow for custom object detection.

   C++ 2 1