Tutorial restructure draft

[brandenchan]

Creating a draft of Tutorial 1 in a new tutorial style. To summarize the main design changes:

• Adhering to the conventional idea of a tutorial being a hands on and step by step lesson towards creating something concrete
• Shifting the large majority of the conceptual explanation from markdown text blocks to links to documentation pages
• Making it clearer and cleaner by removing large majority of code comments, and also optional code blocks
• Adding opening bullet points on level, time to complete, prerequisites, nodes used and goal

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[ZanSara]

Sorry in advance if the comments are too drastic! Let's discuss if there's a way to address them. I won't block this refactoring for them.

[bilgeyucel]

I loved how everything is well structured and goes step-by-step! Such an improvement (at least for me 😅)

[bglearning]

Minor comment, just thinking out loud: for Tut 1, what is the reason we use ElasticSearch (as opposed to InMemory)? Is it to start off with a good default?

The "QA System" -> "QA System without ES" rather than "QA System (InMemory)" -> "QA System with ES" has always felt a bit odd to me. 😄

Just a hunch: someone not familiar with ES might find all the ES setup intimidating (albeit it's just running the given code/commands).

[brandenchan]

Minor comment, just thinking out loud: for Tut 1, what is the reason we use ElasticSearch (as opposed to InMemory)? Is it to start off with a good default?

The "QA System" -> "QA System without ES" rather than "QA System (InMemory)" -> "QA System with ES" has always felt a bit odd to me. smile

Just a hunch: someone not familiar with ES might find all the ES setup intimidating (albeit it's just running the given code/commands).

I actually agree with you on this one! I think ES is a complicating factor this early on. And they aren't working at the scale where they will be getting the benefits of an ES server instead of an InMemoryDS. One thing though is that InMemoryDS will require using the TFIDFRetriever instead of the BM25Retriever which is not a recommended default.

[ZanSara]

We could also consider adding support form BM25 in InMemory in the near future if there's a quick and easy way to do that. There might be libraries for it, so we wouldn't need to implement BM25 from scratch (for example https://pypi.org/project/rank-bm25/).

[anakin87]

To give a general opinion, I really appreciate this restructuring of the tutorials: it seems to me that they lower the entry barrier a little, without selling overly pre-packaged solutions. 💪

Just two small comments on the fine-tuning tutorial:

• I would like the user to be able to directly experience the positive effect of fine-tuning (some examples, metrics...), but I realize that it is difficult to do so if the evaluation topic is not introduced first
• I understand that using the SQuAD dev set for fine-tuning is simple and makes sense for demonstration purposes.
  On the other hand, I don't want the user to think this is the norm. (I'm not an expert on tuning QA models, so what I said might be nonsense...)

[brandenchan]

deepset-ai/haystack#3561 will add support for BM25Retriever in InMemoryDocumentStore.
I think it is already well advanced and I hope to complete it within a maximum of two days. So maybe we could wait for it to be merged...

Thanks @anakin87 for the heads up! Having BM25 in the first tutorial working with the InMemoryDocumentStore will be a big step up!

[brandenchan]

• I would like the user to be able to directly experience the positive effect of fine tuning (some examples, metrics...), but I realize that it is difficult to do so if the evaluation topic is not introduced first

• I understand that using the SQuAD dev set for fine-tuning is simple and makes sense for demonstration purposes.
  On the other hand, I don't want the user to think this is the norm. (I'm not an expert on tuning QA models, soso what I said might be nonsense...)

Thank you very much for taking the time to have a look at the tutorials! Your feedback is definitely really helpful. And I'm glad you picked up on this point. Further training the QA model on SQuAD dev is definitely not ideal because you cannot evaluate it anymore on SQuAD. Theoretically it should be generally better just because it has seen more data. But I will ask internally whether we have any more appropriate datasets to work with in this tutorial

[ZanSara]

@brandenchan deepset-ai/haystack#3561 is merged, we can now use BM25 with InMemoryDocumentStore. Let's update the tutorial to do so!

[anakin87]

@brandenchan to build the BM25 representation, the document store must be initialized as follows:
document_store = InMemoryDocumentStore(use_bm25=True)

The BM25 representation is not computed by default: if you just want to use the InMemoryDocumentStore for dense retrieval or for TF-IDF retrieval, it is not needed and would make operations slower.

[brandenchan]

@brandenchan to build the BM25 representation, the document store must be initialized as follows:
document_store = InMemoryDocumentStore(use_bm25=True)

Thanks for the heads up! Committing that change now

[TuanaCelik]

@brandenchan - the slug structure is now ready for you to use. Including the auto-generated download buttons. You can merge master and we can take it from there.

[TuanaCelik]

As I will be on PTO next week I will add this note here just in case it is needed. And @julian-risch tagging you here too as it is about tracking in telemetry and the files in the new s3 bucket as well. fyi: we've spoken about this with docs, just putting it in writing.
3 things:

• I think we should have a shift in mentality with regards to the numbers. We might want to start thinking of the number at the front of a tutorial as 'identification' numbers. Because the order is not linked to them. And could change in the future too. Depending on what we decide makes sense as a learning track. Otherwise we risk having to re-number tutorial files a lot and also have to adjust how we look at telemetry data..
• If a tutorial is the 'same'(ish) just being restructured, I suggest we keep the number in front of the name the same as before, this way we can look at telemetry, see the tutorial number there, and we know which notebook it corresponds to.
• With that, in this PR, let's make sure that the dataset used in the tutorial is the correct one for telemetry to id it properly (I think this is simply a path name check)

And with that, suggestion:
Keep the fine-tuning tutorial id'ed 02. This way we can see the difference of usage from before, for the same tutorial.
Meaning 'Build your first Question Answering System' and 'Build a scalable question answering system' can be 01 and 03 (or 03 and 01, I leave this to you :) )- The only trick is to then set their 'order' properly in index.toml

[TuanaCelik]

So now that we no longer have and no longer want to maintain "03_build_a_scalable_question_answering_system" --> where do you want the people looking for this to go to? do you want them to be directed to the 1st tutorial? Or this tutorial? We can add the old slug to one of the aliases accordingly.

[agnieszka-m]

the first tutorial, I'd say

[TuanaCelik]

@brandenchan here are some preview links for you. Please disregard the url structure in these previews. The slugs in this PR are correct, I just had to make the previews like this because I'm hacking it :) We don't have a proper previewing structure in place yet.
Tutorial 1: https://haystack-home-bpgtoj8a9-deepset-overnice.vercel.app/tutorials/1
Tutorial 2: https://haystack-home-bpgtoj8a9-deepset-overnice.vercel.app/tutorials/2
Tutorial 3: https://haystack-home-bpgtoj8a9-deepset-overnice.vercel.app/tutorials/3
Tutorial 21: https://haystack-home-bpgtoj8a9-deepset-overnice.vercel.app/tutorials/21

[anakin87]

Standing to the preview, BM25 doesn't appear in Tutorial 1.

[agnieszka-m]

Standing to the preview, BM25 doesn't appear in Tutorial 1.

I think it was mistakenly updated in the .md file instead of the notebook file. Fixed it now. Thanks for being alert! :)

[TuanaCelik]

This has to change to _txt3.zip

[TuanaCelik]

This dataset can be moved from tutorial 2 to 21 for telemetry @julian-risch ?

[julian-risch]

Yes can be down. To make this process less error-prone maybe we can have a list with old tutorials dataset URLs and what they become? This is not the only one that needs to change and it would be great if we could do it in one larger batch if possible.

[TuanaCelik]

At this point I am baffled and I think I need help figuring this out @julian-risch
Right now, we look at this dataset to count the times tutorial 2 has been run. Now we're decoupling the distillation bit so this line is out. However, we don't at any point for the training step have users download the data/squad20 "train_filename="dev-v2.0.json"? But the tutorial works. Does the train() function pull it by name from somewhere? Is it possible to once we merge these tutorial restructures that we track that? @agnieszka-m tagging you here for visibility

[julian-risch]

That can't work. Only possible explanation is that you already have the data_dir = "data/squad20"directory with some data in it. For example if you ran another tutorial before. We need to also fetch_archive_from_http the data in tutorial 2. So this needs to be added here again. And I could upload a/ datasets/documents/squad_small2.json.zip" or something like that with a different dataset name than before for telemetry.

[julian-risch]

Ah, I'm wrong.
I just tried running the new tutorial here: https://colab.research.google.com/github/deepset-ai/haystack-tutorials/blob/tutorial_restructure_draft/tutorials/02_finetune_a_reader.ipynb and it does download the dataset.
The explanation is that the processor has two datasets hardcoded here: https://github.com/deepset-ai/haystack/blob/e266cf6e29f78df751d9dbe7a505886579233aa5/haystack/modeling/data_handler/processor.py#L42 One is squad

Here is the logic that decides about downloading that data: https://github.com/deepset-ai/haystack/blob/e266cf6e29f78df751d9dbe7a505886579233aa5/haystack/modeling/data_handler/processor.py#L2250

[TuanaCelik]

Thanks, in that case for now, until we have any decisions to change or keep the way we do telemetry for tutorials, I will add a fetch_archive_from_http to tutorial 2 as well. As you suggested in the comment above. I will create a list of datasets for tutorials in the meantime. Until and if there is a change, it will be healthier to have.

I've already decoupled tutorial 1 and 3 into a separate PR as there's no need for those 2 to wait while we're fixing datasets etc for Tutorial 2 and 21..