- Welcome to the InstructLab Taxonomy
- Learning
- Getting Started with Skill Contributions
- Getting Started with Knowledge Contributions
- Taxonomy tree layout
- Contribute knowledge and skills to the taxonomy!
InstructLab 🐶 uses a novel synthetic data-based alignment tuning method for Large Language Models (LLMs.) The "lab" in InstructLab 🐶 stands for Large-Scale Alignment for ChatBots [1].
The LAB method is driven by taxonomies, which are largely created manually and with care.
This repository contains a taxonomy tree that allows you to create models tuned with your data (enhanced via synthetic data generation) using the LAB 🐶 method.
[1] Shivchander Sudalairaj*, Abhishek Bhandwaldar*, Aldo Pareja*, Kai Xu, David D. Cox, Akash Srivastava*. "LAB: Large-Scale Alignment for ChatBots", arXiv preprint arXiv: 2403.01081, 2024. (* denotes equal contributions)
In general, we use the Dewey Decimal Classification (DDC) System to determine our domains (and subdomains) in the taxonomy. This DDC SUMMARIES document is a great resource for determining where a topic might be classified.
If you are unsure where to put your knowledge or compositional skill, create a folder in the miscellaneous_unknown folder under the knowledge or compositional_skills folders.
Learn about the concepts of "skills" and "knowledge" in our InstructLab Community Learning Guide.
Skills require a much smaller volume of content than knowledge contributions. An entire skill contribution to the taxonomy tree can be just a few lines of YAML in the qna.yaml file ("qna" is short for "questions and answers") and an attribution.txt file for citing sources.
Your skills contribution pull requests must include the following:
- A
qna.yamlthat contains a set of key/value entries with the following keys- Each
qna.yamlfile requires a minimum of five question and answer pairs.
- Each
- An
attribution.txtthat includes the sources for the information used in theqna.yaml
Tip
The skill taxonomy structure is used in several ways:
- To select the right subset of the taxonomy to use for data generation.
- To determine the interpretability by human contributors and maintainers.
- As part of the prompt to the LLM used to generate synthetic samples.
Important
There is a limit to how much content can exist in the question/answer pairs for the model to process. Due to this, only add a maximum
of around 2300 words to your question and answer seed example pairs in the qna.yaml file.
Compositional skills can either be grounded (includes a context) or ungrounded (does not include a context). Grounded or ungrounded is declared in the taxonomy tree, for example: linguistics/writing/poetry/haiku/ (ungrounded) or grounded/linguistics/grammar (grounded). The qna.yaml is in the final node.
Taxonomy skill files must be a valid YAML file named qna.yaml. Each qna.yaml file contains a set of key/value entries with the following keys:
version: The value must be the number 2. Requiredtask_description: A description of the skill. Requiredcreated_by: The GitHub username of the contributor. Requiredseed_examples: A collection of key/value entries. New submissions should have at least five entries, although older files may have fewer. Requiredcontext: Grounded skills require the user to provide context containing information that the model is expected to take into account during processing. This is different from knowledge, where the model is expected to gain facts and background knowledge from the tuning process. The context key should not be used for ungrounded skills.question: A question for the model. Requiredanswer: The desired response from the model. Required
Other keys at any level are currently ignored.
To make the qna.yaml files easier and faster for humans to read, it is recommended to specify version first, followed by task_description, then created_by, and finally seed_examples.
In seed_examples, it is recommended to specify context first (if applicable), followed by question and answer.
Example qna.yaml
version: 2
task_description: <string>
created_by: <string>
seed_examples:
- question: <string>
answer: |
<multi-line string>
- context: |
<multi-line string>
question: <string>
answer: |
<multi-line string>
...Then, you create an attribution.txt file that includes the sources of your information. These can also be self authored sources.
Example attribution.txt
[Link to source]
[Link to work]
[License of the work]
[Creator name]
For more information on what to include in your attribution.txt file, see For your attribution.txt file in CONTRIBUTING.md.
If you have not written YAML before, don't be intimidated - it's just text.
Tip
- Spaces and indentation matter in YAML. Two spaces to indent.
- Don't use tabs!
- Be careful to not have trailing spaces at the end of a line.
- Each example in
seed_examplesbegins with a "-". Place this "-" in front of the first field (questionorcontext). The remaining keys in the example should not have this "-". - Some special characters such as " and ' need to be escaped with backslash. This is why some of the lines for keys in the example YAML start the value with the '|' character followed a new line and then an indented multi-line string. This character disables all of the special characters in the value for the key. You might also want to use the '|' character for multi-line strings.
- Consider quoting all values with " to avoid surprising YAML parser behavior
(e.g. Yes answer can be interpreted by the parser as a boolean of
Truevalue, unless "Yes" is quoted.) - See https://yaml-multiline.info/ for more info.
It is recommended that you lint, or verify, your YAML using a tool. One linter option is yamllint.com. You can copy/paste your YAML into the box and click Go to have it analyze your YAML and make recommendations. Online tools like prettified and yaml-validator can automatically reformat your YAML to adhere to our yamllint PR checks, such as breaking lines longer than 120 characters.
version: 2
task_description: 'Teach the model how to rhyme.'
created_by: juliadenham
seed_examples:
- question: What are 5 words that rhyme with horn?
answer: warn, torn, born, thorn, and corn.
- question: What are 5 words that rhyme with cat?
answer: bat, gnat, rat, vat, and mat.
- question: What are 5 words that rhyme with poor?
answer: door, shore, core, bore, and tore.
- question: What are 5 words that rhyme with bank?
answer: tank, rank, prank, sank, and drank.
- question: What are 5 words that rhyme with bake?
answer: wake, lake, steak, make, and quake.Seriously, that's it.
Here is the location of this YAML in the taxonomy tree. Note that the YAML file itself, plus any added directories that contain the file, is the entirety of the skill in terms of a taxonomy contribution:
[...]
└── writing
└── poetry
| └── haiku <=== here it is :)
| | └── qna.yaml
| | attribution.txt
[...]
└── prose
| └── debate
| | └── qna.yaml
| | attribution.txt
[...]
[...]
Remember that grounded compositional skills require additional context and include a context field.
This example snippet assumes the GitHub username mairin and shows some of the question/answer pairs present in the actual file:
version: 2
task_description: |
This skill provides the ability to read a markdown-formatted table.
created_by: mairin # Use your GitHub username; only one creator supported
seed_examples:
- context: |
| **Breed** | **Size** | **Barking** | **Energy** |
|----------------|--------------|-------------|------------|
| Afghan Hound | 25-27 in | 3/5 | 4/5 |
| Labrador | 22.5-24.5 in | 3/5 | 5/5 |
| Cocker Spaniel | 14.5-15.5 in | 3/5 | 4/5 |
| Poodle (Toy) | <= 10 in | 4/5 | 4/5 |
question: |
Which breed has the most energy?
answer: |
The breed with the most energy is the Labrador.
- context: |
| **Name** | **Date** | **Color** | **Letter** | **Number** |
|----------|----------|-----------|------------|------------|
| George | Mar 5 | Green | A | 1 |
| Gráinne | Dec 31 | Red | B | 2 |
| Abigail | Jan 17 | Yellow | C | 3 |
| Bhavna | Apr 29 | Purple | D | 4 |
| Rémy | Sep 9 | Blue | E | 5 |
question: |
What is Gráinne's letter and what is her color?
answer: |
Gráinne's letter is B and her color is red.
- context: |
| Banana | Apple | Blueberry | Strawberry |
|--------|------------|-----------|------------|
| Yellow | Red, Green | Blue | Red |
| Large | Medium | Small | Small |
| Peel | Peel | No peel | No peel |
question: |
Which fruit is blue, small, and has no peel?
answer: |
The blueberry is blue, small, and has no peel.[...]
grounded
└── technology
└── machine_learning
└── natural_language_processing
| | └── information_extraction
| └── inference
| | └── qualitative
| | ├── sentiment
| | | └── qna.yaml
| | | attribution.txt
│ ├── quantitative
│ │ ├── table_analysis <=== here it is :)
│ | | | └── qna.yaml
│ │ │ | attribution.txt
[...]
While skills are foundational or performative, knowledge is based more on answering questions that involve facts, data, or references.
Knowledge is supported by documents, such as a textbook, technical manual, encyclopedia, journal, or magazine.
Knowledge in the taxonomy tree consists of a few more elements than skills:
- Each knowledge node in the tree has a
qna.yaml, similar to the format of theqna.yamlfor skills. - ⭐ Knowledge submissions require you to create a Git repository, can be with GitHub, that contains the markdown files of your knowledge contributions. These contributions in your repository must use the markdown (.md) format.
- The
qna.yamlincludes parameters that contain information from your repository.
Tip
Guidelines for Knowledge contributions
- Submit the most up-to-date version of the document
- All submissions must be text, images will be ignored
- Do not use tables in your markdown freeform contribution
The qna.yaml format must include the following fields:
version: The version of the qna.yaml file, this is the format of the file used for SDG. The value must be the number 3.created_by: Your GitHub username.domain: Specify the category of the knowledge.seed_examples: A collection of key/value entries.context: A chunk of information from the knowledge document. Eachqna.yamlneeds fivecontextblocks and has a maximum word count of 500 words.questions_and_answers: The parameter that holds your questions and answersquestion: Specify a question for the model. Eachqna.yamlfile needs at least three question and answer pairs percontextchunk with a maximum word count of 250 words.answer: Specify the desired answer from the model. Eachqna.yamlfile needs at least three question and answer pairs percontextchunk with a maximum word count of 250 words.
document_outline: Describe an overview of the document your submitting.document: The source of your knowledge contribution.repo: The URL to your repository that holds your knowledge markdown files.commit: The SHA of the commit in your repository with your knowledge markdown files.patterns: A list of glob patterns specifying the markdown files in your repository. Any glob pattern that starts with*, such as*.md, must be quoted due to YAML rules. For example,"*.md".
version: 3
domain: astronomy
created_by: juliadenham
seed_examples:
- context: |
**Phoenix** is a minor constellation in the southern sky. Named after the mythical
phoenix, it was first depicted on a celestial atlas by Johann Bayer in his 1603
*Uranometria*. The French explorer and astronomer Nicolas Louis de
Lacaille charted the brighter stars and gave their Bayer designations
in 1756. The constellation stretches from roughly −39 degrees to −57 degrees
declination, and from 23.5h to 2.5h of right ascension. The constellations Phoenix,
Grus, Pavo, are known as the Southern Birds.
questions_and_answers:
- question: |
What is the Phoenix constellation?
answer: |
Phoenix is a minor constellation in the southern sky.
- question: |
Who charted the Phoenix constellation?
answer: |
The Phoenix constellation was charted by french explorer and
astronomer Nicolas Louis de Lacaille.
- question: |
How far does the Phoenix constellation stretch?
answer: |
The phoenix constellation stretches from roughly −39° to −57°
declination, and from 23.5h to 2.5h of right ascension.
- context: |
Phoenix was the largest of the 12 constellations established by Petrus
Plancius from the observations of Pieter Dirkszoon Keyser and Frederick de
Houtman. It first appeared on a 35cm diameter celestial globe published
in 1597 (or 1598) in Amsterdam by Plancius with Jodocus Hondius. The first
depiction of this constellation in a celestial atlas was in Johann Bayer's
*Uranometria* of 1603. De Houtman included it in his southern star catalog
the same year under the Dutch name *Den voghel Fenicx*, "The Bird Phoenix",
symbolising the phoenix of classical mythology. One name of the brightest star Alpha
Phoenicis—Ankaa—is derived from the Arabic: العنقاء, romanized: al-‘anqā’,
lit. 'the phoenix', and was coined sometime after 1800 in relation to the constellation.
questions_and_answers:
- question: |
What is the brightest star in the Phoenix constellation
called?
answer: |
Alpha Phoenicis or Ankaa is the brightest star in the Phoenix
Constellation.
- question: Where did the Phoenix constellation first appear?
answer: |
The Phoenix constellation first appeared on a 35-cm diameter
celestial globe published in 1597 (or 1598) in Amsterdam by
Plancius with Jodocus Hondius.
- question: |
What does "The Bird Phoenix" symbolize?
answer: |
"The Bird Phoenix" symbolizes the phoenix of classical mythology.
- context: |
Phoenix is a small constellation bordered by Fornax and Sculptor to the north,
Grus to the west, Tucana to the south, touching on the corner of Hydrus to the
south, and Eridanus to the east and southeast. The bright star Achernar is
nearby. The three-letter abbreviation for the constellation, as adopted by the
International Astronomical Union in 1922, is "Phe". The official constellation
boundaries, as set by Belgian astronomer Eugène Delporte in 1930,
are defined by a polygon of 10 segments. In the equatorial coordinate system, the right
ascension coordinates of these borders lie between 23h 26.5m and 02h 25.0m,
while the declination coordinates are between −39.31° and −57.84°. This means it remains
below the horizon to anyone living north of the 40th parallel in the Northern
Hemisphere, and remains low in the sky for anyone living north of the equator.
It is most visible from locations such as Australia and South Africa during
late Southern Hemisphere spring. Most of the constellation lies within, and
can be located by, forming a triangle of the bright stars Achernar, Fomalhaut
and Beta Ceti—Ankaa lies roughly in the centre of this.
questions_and_answers:
- question: What are the characteristics of the Phoenix constellation?
answer: |
Phoenix is a small constellation bordered by Fornax and Sculptor to
the north, Grus to the west, Tucana to the south, touching on the
corner of Hydrus to the south, and Eridanus to the east and southeast.
The bright star Achernar is nearby.
- question: |
When is the phoenix constellation most visible?
answer: |
Phoenix is most visible from locations such as Australia and
South Africa during late Southern Hemisphere spring.
- question: |
What are the Phoenix Constellation boundaries?
answer: |
The official constellation boundaries for Phoenix, as set by Belgian
astronomer Eugène Delporte in 1930, are defined by a polygon of 10
segments.
- context: |
Ten stars have been found to have planets to date, and four planetary
systems have been discovered with the SuperWASP project. HD 142 is a yellow
giant that has an apparent magnitude of 5.7, and has a planet HD 142b 1.36
times the mass of Jupiter which orbits every 328 days. HD 2039 is a yellow
subgiant with an apparent magnitude of 9.0 around 330 light years away which
has a planet HD 2039 b six times the mass of Jupiter. WASP-18 is a star of
magnitude 9.29 which was discovered to have a hot Jupiter-like planet taking
less than a day to orbit the star. The planet is suspected to be causing WASP-18 to
appear older than it really is. WASP-4 and WASP-5 are solar-type yellow stars around 1000
light years distant and of 13th magnitude, each with a single planet larger
than Jupiter. WASP-29 is an orange dwarf of spectral type K4V and visual magnitude
11.3, which has a planetary companion of similar size and mass to Saturn. The planet
completes an orbit every 3.9 days.
questions_and_answers:
- question: In the Phoenix constellation, how many stars have planets?
answer: |
In the Phoenix constellation, ten stars have been found to have
planets to date, and four planetary systems have been discovered
with the SuperWASP project.
- question: |
What is HD 142?
answer: |
HD 142 is a yellow giant that has an apparent magnitude of 5.7, and
has a planet (HD 142 b) 1.36 times the mass of Jupiter which
orbits every 328 days.
- question: |
Are WASP-4 and WASP-5 solar-type yellow stars?
answer: |
Yes, WASP-4 and WASP-5 are solar-type yellow stars around 1000 light
years distant and of 13th magnitude, each with a single planet
larger than Jupiter.
- context: |
The constellation does not lie on the galactic plane of the Milky Way, and there
are no prominent star clusters. NGC 625 is a dwarf irregular galaxy of apparent magnitude 11.0
and lying some 12.7 million light years distant. Only 24000 light years in
diameter, it is an outlying member of the Sculptor Group. NGC 625 is
thought to have been involved in a collision and is experiencing a burst
of active star formation. NGC 37 is a lenticular galaxy of apparent magnitude
14.66. It is approximately 42 kiloparsecs in diameter and about 12.9 billion years old.
Robert's Quartet , and three spiral galaxies NGC 88 and NGC 92) is a group of
four galaxies located around 160 million light-years away which are in the process of colliding
and merging. They are within a circle of radius of 1.6 arcmin, corresponding to about
75,000 light-years. Located in the galaxy ESO 243-49 is HLX-1, an intermediate-mass
black hole intermediate-mass_black_hole —the first one of its kind identified.
It is thought to be a remnant of a dwarf galaxy that was absorbed in a collision
with ESO 243-49. Before its discovery, this class of black hole was only hypothesized.
questions_and_answers:
- question: |
Is the Phoenix Constellation part of the Milky Way?
answer: |
The Phoenix constellation does not lie on the galactic plane of
the Milky Way, and there are no prominent star clusters.
- question: |
How many light years away is NGC 625?
answer: |
NGC 625 is 24000 light years in diameter and is an outlying
member of the Sculptor Group.
- question: |
What is Robert's Quartet composed of?
answer: |
Robert's Quartet is composed of the irregular galaxy NGC 87,
and three spiral galaxies NGC 88, NGC 89 and NGC 92.
document_outline: |
Information about the Phoenix Constellation including the
history, characteristics, and features of the stars in the constellation.
document:
repo: https://github.com/juliadenham/Summit_knowledge
commit: 0a1f2672b9b90582e6115333e3ed62fd628f1c0f
patterns:
- phoenix_constellation.mdExample attribution.txt file
Title of work: Phoenix (constellation)
Link to work: https://en.wikipedia.org/wiki/Phoenix_(constellation)
Revision: https://en.wikipedia.org/w/index.php?title=Phoenix_(constellation)&oldid=1237187773
License of the work: CC-BY-SA-4.0
Creator names: Wikipedia Authors
This knowledge example references one markdown file: phoenix_constellation.md. You can also add multiple files for knowledge contributions.
Note
Due to the higher volume, it will naturally take longer to receive acceptance for a knowledge contribution pull request than for a skill pull request. Smaller pull requests are simpler and require less time and effort to review.
What might these markdown files look like? They can be freeform. Here's what a
snippet of phoenix_constellation.md might look like in your Git repository.
# Phoenix (constellation)
**Phoenix** is a minor constellation in the southern sky. Named after the mythical
phoenix, it was first depicted on a celestial atlas by Johann Bayer in his 1603
*Uranometria*. The French explorer and astronomer Nicolas Louis de
Lacaille charted the brighter stars and gave their Bayer designations
in 1756. The constellation stretches from roughly −39 degrees to −57 degrees
declination, and from 23.5h to 2.5h of right ascension. The constellations Phoenix,
Grus, Pavo, are known as the Southern Birds.
The brightest star, Alpha Phoenicis, is named Ankaa, an Arabic word meaning 'the Phoenix'.
It is an orange giant of apparent magnitude 2.4. Next is Beta Phoenicis, actually a
binary system composed of two yellow giants with a combined apparent magnitude of 3.3. Nu
Phoenicis has a dust disk, while the constellation has ten star systems with known planets and the recently
discovered galaxy clusters El Gordo and the Phoenix
Cluster—located 7.2 and 5.7 billion light years away respectively, two of the largest objects in the visible
universe. Phoenix is the radiant of two annual meteor showers: the Phoenicids in December, and the July
Phoenicids.In the taxonomy repository, here's what the previously referenced knowledge might look like in the tree:
[...]
└── knowledge
└── science
├── astronomy
│ └── constellations
│ └── Phoenix <=== here it is :)
│ | └── qna.yaml
| | attribution.txt
│ └── Orion
│ └── qna.yaml
| attribution.txt
[...]
For more information on what to include in your attribution.txt file, see For your attribution.txt file in CONTRIBUTING.md.
You can organize the knowledge markdown files in your repository however you want. You just need to ensure the YAML is pointing to the correct file.
The taxonomy tree is organized in a cascading directory structure. At the end of each branch, there is a YAML file (qna.yaml) that contains the examples for that domain. Maintainers can decide to change the names of the existing branches or to add new branches.
Important
Folder names do not have spaces. Use underscores between words.
Below is an illustrative directory structure to show this layout:
.
└── linguistics
├── writing
│ ├── brainstorming
│ │ ├── idea_generation
| │ └── qna.yaml
│ │ attribution.txt
│ │ ├── refute_claim
| │ └── qna.yaml
│ │ attribution.txt
│ ├── prose
│ │ ├── articles
│ │ └── qna.yaml
│ │ attribution.txt
└── grammar
└── qna.yaml
│ attribution.txt
└── spelling
└── qna.yaml
attribution.txt
For an extensive example of this layout see, taxonomy_tree_layout in the documentation folder.
The ability to contribute to a Large Language Model (LLM) has been difficult in no small part because it is difficult to get access to the necessary compute infrastructure.
This taxonomy repository will be used as the seed to synthesize the training data for InstructLab-trained models. We intend to retrain the model(s) using the main branch following InstructLab's progressive training on a regular basis. This enables fast iteration of the model(s), for the benefit of the open source community.
By contributing your skills and knowledge to this repository, you will see your changes built into an LLM within days of your contribution rather than months or years! If you are working with a model and notice its knowledge or ability lacking, you can correct it by contributing knowledge or skills and check if it's improved after your changes are built.
While public contributions are welcome to help drive community progress, you can also fork this repository under the Apache License, Version 2.0, add your own internal skills, and train your own models internally. However, you might need your own access to significant compute infrastructure to perform sufficient retraining.
You can contribute to the taxonomy in the following two ways:
- Adding new examples to existing leaf nodes:
- Adding new branches/skills corresponding to the existing domain:
For more information, see the Ways of contributing to the taxonomy repository documentation.
To contribute to this repo, you'll use the Fork and Pull model common in many open source repositories. You can add your skills and knowledge to the taxonomy in multiple ways; for additional information on how to make a contribution, see the Documentation on contributing. You can also use the following guides to help with contributing:
- Contributing using the GitHub webpage UI.
- Contributing knowledge to the taxonomy in the Knowledge contribution guidelines.
This taxonomy repository will be used as the seed to synthesize the training data for InstructLab-trained models. We intend to retrain the model(s) using the main branch as often as possible (at least weekly). Fast iteration of the model(s) benefits the open source community and enables model developers who do not have access to the necessary compute infrastructure.