Folder Structure

[EstherPlomp]

Share your Folder Structure based on the CookieCutter assignment or manual folder creation:

[ignasisg]

My current folder structure is basically as described in the video. That is a hierarchy of different subfolders, with a decent amount of files (not too many) per folder. I've been naming the folders so far using the following terminology: YYYYMMDD_InformationOnSample. I don't give information about the experiment because the subfolders are different experiments/instruments. For instance, under my main folder PhD, I have subfolders: Graduate School; Lab; Paperwork; Presentation, meetings, and notes. Under lab, I have three subfolders: data; safety, protocol, and risk assessments; Setting of Goals. Under data, I have a different folder for each type of experiment. That is Ultrasound Experiment, Steward Assay, etc. Under each of those, I have the file names with the convention above. One thing I'd like to add, however, is adding a readme file in each of the folders under data, to explain for instance where to find the exact protocol I followed and other important information.

[EstherPlomp]

Sounds like a clear overview! Just curious: Do you have 'setting of goals' under lab because this contains the goals for the lab for a certain period?

And I think you also did some programming/analysis with code - is that in the data folder or is that stored elsewhere?

[MaxThFe]

My project folder structure is very similar to the one presented in the cookie cutter template. In fact, we have a group template for projects that we set up inspired by the cookie cutter template. Since we work with code a lot, the src folder is much more extensive than the one in the video shown, with subfolders for different aspects of the code. My folder structure is further geared towards AI development, as that is what I do in my PhD. For instance, I have a folder called trained_models that contains trained AI models. The models are named by their training date, e.g. YYYYMMDD_modelname. Those project folders are also backed up on our group's GitHub organization as well.

Apart from my project folders, I also have a hierarchy of folders for other aspects of my PhD, e.g. Literature, Publications, Students, Travel, and Graduate School. In those folders, I save information related to the respective topics. These folders are on the same level as my overall Project folder.

[EstherPlomp]

Nice to hear that you're already implementing a group template for projects! That will save a lot of headaches in the future trying to collaborate on projects!

Everything else sounds good, so no feedback from my side :)

[vgualtieri]

I have a simulation package whose outcome is simulated data/plots.
I realized my folder structure was totally a mess: I have .py files, .ipynb files, and code output (.pdf, .npy etc) all in the same folder with no folder structure, mostly because I am lazy to add the file path to the code. I followed the instructions and just uploaded the .py files and one example jupyter notebook file (just a sample to show how the code works), disregarding all the other files/results that are not really needed.

[EstherPlomp]

It sounds like the exercise was helpful! 👍

[xxmeryl]

After watching the video, I have started with reorganizing my folder structure. I am still working on it (still thinking of where to put certain documents and whether I need extra categories). Overall I have divided my project in the years of my PhD at first. Then you have five folders: 1 - Project Management, 2 - Literature, 3 - Graduate School, 4 - Experiments, 5 - Dissemination and 6 - Teaching. From then on it depends on each folder of course. For the experiments I have now organized it as shown in the video (inputs, data, data analysis and outputs). Still working on further organization, but I already like the changes I have made and the new structure :)

[EstherPlomp]

I'm glad the video inspired you to work on the folder structure! It sounds very organised to me, so no feedback from my side :)

[lettema1]

On the Project Data (U:) drive, we have different folders. When we perform measurements, data will be stored in the Measurements folder, after which there is a division into the setup used, followed by the sample measured with the setup. The processed data is also saved in this folder (it has the same parent folder as the raw data). In addition, copies of the scripts used for running the measurement and processing the data are saved. This way, everyone in the group can access the raw and processed data. Topics that are not interesting for the others in the group (e.g. graduate school, literature, meeting notes, etc.) are stored in my personal folder on the U: drive.

[EstherPlomp]

Sounds very organised! Do you use a U: for your whole research group or just a smaller portion of the group? (Just curious to see if that works well for an entire research group if this is the case!)

[lettema1]

Yes, everyone in our research group uses the U: drive, including the Bachelor and Master students. However, our research group is very small (1 PI, 1 postdoc, 1 PhD student, and 5 students)

[EstherPlomp]

Makes sense! That is still manageable. Thanks for sharing!

[maartjespaans]

I have folders which I use and edit regularly (Projects, Notes, Literature, Meetings/Presentations) and folders that I use for storing general files that I use regularly (Genomes, Protocols, Plasmids etc.). In the projects folder I have a subfolder for each project, within these subfolders I have generated a structure of subfolders that is similar for each project (Data, Figures, Sequences, Results). At first I didn't have a separate data and results folder, since I haven't generated that much data yet. But in the future I will keep my analysis (results) better separated from my raw data.

[AimeeKok]

This sounds really organized already! Very nice 🙌

[EstherPlomp]

Sounds organised indeed - glad to hear that you'll separate the raw data and results!

[SIM1-S]

I previously structured my documents in such a way that it was easily accessible and i could find my most used documents easily. However, the resources provided in the assignment did point me to some mistakes i was making. I found that a combination of Barbara's Vreede Cookie cutter and Nikola Vukovics structure work best for me. I have mostly literature, large amounts of different experimental data, and a bit of code.
I am still in the process of rearranging the structure (in all versions/backups), but currently it is as follows; Main folders for Teaching, Graduate school, Project folder. The project folder then consists of folders for Data, Project management, Results, Presentations, and Reports. The Project management folder is the most extensive folder, containing documentation for experiments, notes, protocols, literature and planning.
The majority of the Experimental data (Belonging to the Data folder) is stored on the Project U drive as I generate large amounts of data weekly. This is structured similarly.

[EstherPlomp]

I'm glad to hear that you found the example folder structures helpful! Hopefully after the reorganisation it will safe you time in the future!

[AimeeKok]

Currently my folder structure is uh a bit of a mess, a lot of stuff is all over the place, partially stored between my home laptop (which I used the first few weeks before my work laptop arrived), my work laptop, OneDrive (and now SURFdrive, but that is at least linked to the main project directories on my work laptop). At least for the actual data/coding for the two projects I have started now I at least have a separate data, data_analysis and results folders that are consistent thanks to github but that is really the bare minimum haha. Notes, presentations, etc are really all over the place currently. 😳 This section of the course was really a good reminder to do better, especially now that I am still at the start and it is easy to adjust, and to prepare for my main project!

I made a mishmash of two cookiecutter templates (Reproducible Science and Good Enough Project) and added some own stuff. Maybe I will reconsider some things but I will remake my current project directories, and try to keep to this generally (changes depending on the project) for new projects. Within the private github repository it will look a bit different since I will be ignoring a large part of it.

.PROJECT_DIRECTORY
├── README.md		<- General project documentation file
├── administration	<- Administrative files for the project             
├── data
│   ├── processed	<- The final data
│   ├── raw		<- The original, immutable data dump. 
│   ├── temp		<- Intermediate data that has been transformed.
│   └── test		<- Subset of data for testing/example purposes (in notebooks and tests)
├── docs               	 
│   ├── manuscript	<- Manuscript source, including model description, bib file
│   ├── notes 		<- Copies of personal notes and summaries on the project
│   ├── notebooks 	<- Jupyter notebooks to show workflows / data analysis pipelines
│   └── presentations	<- PPTs, both for result summaries and meetings
├── results
│   ├── figures		<- Figures for the manuscript or reports
│   └── output 		<- Other output for the manuscript or reports
├──src         	
|   ├── data		<- Scripts to process data
|   ├── external	<- Any external source code / tools 
|   ├── model 	 	<- Source code for my own model/simulation
|   └── tools		<- Any helper/visualisation scripts go here
└── tests		<- Code pertaining to testing the source code 

[EstherPlomp]

I'm glad the course is of help here - if you were already implementing all the recommended practices the course would be a bit of a waste of time! And very nice that you used the cookie cutter templates to match your own work/style - it looks very clear and organised to me!

[lenafasching]

After starting different projects and collaborations, I had to restructure some of my folders.I started to organize per project with a very similar substructure that is divided in background, experimental data and output. I think I still need to optimise this strutures a bit bc some of the literature and experimental results for example can be used in different projects. Then it still gets a bit messy were to put it and find it again. For now I just link the data, but that has the a bit of a differnet file name structure. But I think it gets better with time.
In case someone has some experience, I would be happy to get some inspiration about how I can homogenise the data nomenclature.

[EstherPlomp]

I hope some of the structures shared here serve as some inspiration! Do ask your group in session 2 for any tips/help!

It is a bit difficult to provide you with any feedback on it from the information you're providing here - if you can share a screenshot or some more details I'm also happy to provide some more input!

[olivierwitteveen]

This is broadly what my current folder structure looks like in my OneDrive. There are certainly places it needs improving.

OneDrive -- Delft
├── administrative			<- anything administrative (contracts, forms, declarations, ...)
│   └── PhD contract 			<- example folder
├── posters and presentations 		<- powerpoint slides and posters for meetings and conferences
│   └── NWO Physics 2024  		<- example folder
├── textbooks, lecture notes     	<- mostly pdf files of textbooks and notes I consult, not very organised.     	 
├── graduate school courses 		<- files for graduate school courses
│   └── OnBoarding Module A		<- example folder
├── projects				<- folder containing all projects
│   ├── ribosome project		<- example project
│   │   ├── data 			<- folder containing all raw data provided by collaborators
│   │   ├── scripts 			<- scripts for data analysis and processing
│   │   ├── write ups 			<- LaTeX documents or PowerPoint slides, e.g. for meetings with collaborators
│   │   └── old 			<- old scripts that I no longer use but want to keep just in case
│   └── useful things 			<- data / scripts I produce that are important but not directly related to a project (bit unorganised...)
└── TA courses         			<- material related to courses I TA for
    └──Soft matter fall 2024		<- lecture notes, problem sets, marked work, etc.

A few comments:

• I'm working to get rid of this "old" folder I have within each project -- thanks for the help with this on the previous assignment! I need to integrate git into my workflow (indeed, in some way that makes sense with Jupyter notebooks), and then this will become unnecessary.

• Within a project directory, the folders "scripts" and "data" can get unorganised very quickly -- I likely need to add some further structure here. I really like the look of the Reproducible Science template. In fact it appears that I happen to use a structure that is quite similar. But I really like how they split up the data folder (external, interim, processed, and raw) as well and the additional "src" folder. I think I will give this a go!

• In parallel, I have got the handwritten notes on my iPad on iCloud that are also organised by project. But this is detached from this folder structure. Not sure if there is a need to integrate it, I've been coping fine.

[EstherPlomp]

This looks like a clear structure @olivierwitteveen! And I'm glad to hear that the course materials are of use!

I like the folder 'useful things' ;) Do be careful with spaces in folder names and file names - that may complicate your life at some point!

If your workflow with your notes work for you, then you may not need to integrate it. You may want to pilot it for a week to see how it would work if you integrate it - and if it doesn't work go back to how you were running things before?

And apologies for the slow response here - I overlooked a couple of shared folder structures earlier!

[adammubeen]

I currently used a project based hierarchical structure that is very similar to what is presented. When using models, I also separate input, intermediate and final output files, with and without pre/post processing separately. My file naming conventions are also quite similiar to what is presented. However, one thing I will adopt and include from now is a Readme that includes the filenaming convention and structure I am using and a file in each folder that lists and provides a directory of all files included. These will be in txt and csv formats.

[EstherPlomp]

I'm glad to hear that you're already implementing recommended practices AND that you also still found space for improvement with the README files!

[mcesarcarvalho]

My folder structure is divided in Teaching, BEP-MEP Supervision, Graduate School, Research Deliverables, Meetings, Other. The video really made me think about it and I'll definitely try to improve it as I think it still has room for improvement. Specially for lab results it's really important to define a better structure because it turns into a chaos really quickly with all the different files acquired with different lab equipment and experiments. I am considering implementing inputs, data, data analysis, and outputs, as suggested in the video on data organization. Also, something that I would like to implement is some kind of periodic cleaning to get rid of folders/files that I don't need anymore (for example, once a month).

[EstherPlomp]

There's probably always room for improvement! I like a monthly cleaning moment - that will indeed help you to keep on top of the increasing data accumulation :) Glad to hear that the course is inspiring you!

[SiemEerden]

Currently my folder structure has room for improvement. Yes I separated things like administrative documents, graduate school, bioinformatics, metabolic modelling, teaching, students, literature, etc, and I made subdivisions. But having seen this course, I want to restructure it in such a way that it reflects my subprojects/prospective chapters, which should be rounded-off bodies of work.

So in that sense it would be like a folder for each chapter or student project, with in it folders for documentation, raw data, scripts, and output data, much like shown in the presentations. Things that do not fit within a subproject/chapter (teaching, GS courses, administrative documents) of course get their own tree, with a structure that fits the logic of that subject.

[EstherPlomp]

Sounds great @SiemEerden! I'm glad you found the examples helpful!

[FolakeBabalola]

I have been using a hierarchical folder structure that allows me to organise my data in folders (e.g. literature, methodology, data collection, reference folder) and has all embedded in a main folder with the RISK-WASH project name.
It is nice to be aware of the README text file through this course, and I would like to implement it going forward in my data management so that the contents of my research project folder can be easily accessible.

[EstherPlomp]

Thanks for sharing! I'm glad you found the README files helpful, especially since you already implemented recommended folder structure practices 👍

[ivyliang98]

I structure my folders in a hierarchical way. First, I have some 'general' folders, that are numbered (0. PhD, 1. Courses, 2.PhD official forms, 3.Meetings, etc.). Then, I have subfolders. For example, my 0.PhD folder has subfolder for each year of my PhD. Within each subfolder, I have 0. Raw data, and 1. Data analysis folders to keep the raw data and analysed data separated. My raw data and data analysis folders are divided into subfolders for each experiment. For all data/files, I use the terminology YYYYMMDD_Experimentcode_Infoaboutthisfile.

This way of structuring works well for me. However, a problem is when I for example have a student. I then don't really know whether I should have a separate "student" folder to keep all the data, or to integrate it in my own folders.

[EstherPlomp]

Sounds good @ivyliang98! Thanks for sharing!

I think I would make a separation in the data that you generated and the data that student(s) generate - this way data access management is also easier (as in, the student might not need edit access to your generated data, but they obviously do need to do things to their generated data!). It may also become more clear with this separation who generated what data. But there's not really a single right way to do this of course - if you find it easier to incorporate everything together than this is also not a wrong way to do it, as long as it is clear who generated what because that might make a difference in your interpretations!

[Adarsh-Shajimon]

My folder structure is also hierarchical but I see that I need to make some changes to have a solid structure. I have a ReThink (project name), Graduate School, HR and Miscellaneous folder. The project folder is divided into subfolders namely, Research, Project symposiums and Symposiums. The Research folder then has subfolders for Reading Materials, Data (sorted based on the equipment used), Analysis and Presentations (Reports, results). My naming convention for files is YYYYMMDD_Info_on_experiments.

[EstherPlomp]

Thanks for sharing @Adarsh-Shajimon! This looks very clear! I might advise to move your research folder up in your structure so that it will be easier to access. It sounds like you'll need to use that folder a lot!

[Adarsh-Shajimon]

Thank you for the advice! Yes, I need to make some changes to the structure so that the research folder is up the order.

[ezgidemiroz]

I also made a hierarchical structure for my experimental data. In subfolders there are Users, Procedures, Methods, Data and Data Analysis. This way all information will be there for another person to reproduce or reuse.

I named the folders as

Material Preparation
Material Characterization
Electrochemical test
Product analysis
Visualization tests
Presentations

[EstherPlomp]

Looks clear Ezgi! 👍

[nicolastravnicek]

My folder structure is divided into Literature, Reports, Graduate school, Presentations, stocks (for proper documentation of the produced genes etc.), and Lab results which are further categorized. I think it is helpful to keep regularly track of the completeness of the folders to facilitate life and save a lot of time. However I think that I can further improve my documentation to facilitate and improve further reproduction experiments for example.

[EstherPlomp]

Thanks for sharing Nicolas! Then the next question would be how you will improve the documentation to indeed further clarify your procedures!

what are the sub folders in the Lab results folder?

[archana2810]

I am not very happy about the current folder structure that I maintain, but I hope to be more organised this year. Similar to @olivierwitteveen's I have folders for administrative paperwork, Graduate school, TA work, and projects. Under projects, i have three different folders for the three projects i am currently involved in. In each project, i have separate folders for literature, Experiments, and protocols. Under experiments, i have Microscopy and Biochemical assays. Under microscopy, I have folders listed for each date of experiments in the format (YYYYMMDD_MainAttributeOfExperiment). Inside these folders, I have raw data and analysis folders.

[EstherPlomp]

How you describe it looks like a clear structure - is this the new one that you want to use or the one that you're currently using?

[JiLi24]

My folder structure is based on the type of my experiments and output. The main folders are divided into courses, conferences, thesis, meetings, etc. Additionally, I have subfolders within these main folders. For example, within the thesis folder, there are subfolders for chapters, forms, research proposals, and methods and protocols. Each chapter folder contains files related to specific experiments, such as Gel Formation Tests, FT-IR, yields, output images, etc. While this structure is convenient for organizing and locating data, sometimes it can be frustrating to navigate through multiple layers of folders to find what I need.