Data Challenge

[EstherPlomp]

What is your main challenge regarding your data management? Please respond below with your data challenge!

Please read your peers’ responses and reply to any you find interesting.

[federa7]

Standardizing my data management practices and sticking to them once I have designed them and started working with them. So, in a way, consistency.

[EstherPlomp]

Thanks @federa7 for sharing! It sounds like the consistency is more related to sticking to what you said would do. Is it challenging because the new practices are very different? Or too complex? Or would it help to plan in a recurring moment in your calendar to remind yourself of reviewing your practises? In sum: What would help you to remain more consistent with this?

[federa7]

What I mean is more of standardizing the way in which I name my data, the format of the mini-report documents I do, the way I organized files, etc. So, more small details that should make my life easier. Maybe I should create cheat-sheets or something of that nature.

[EstherPlomp]

I like the ideas of cheat-sheets! If you do set something up like that, please share - I'm sure others could learn from that too 👍

[cfprimavera]

My main challenge is building a framework for data management which works for me, while also being consistent with what others use that work in the same lab.

[EstherPlomp]

Indeed a challenge @cfprimavera! It is perhaps easier to adjust your own workflows than to do the same for others and to create greater consistency in the lab. In a way it is also not your responsibility: ideally a lab manager or principal investigator plays a role in these processes. Perhaps you can nudge other individuals in the lab to review the current framework for data management? Let me know if you would appreciate any help with that.

[msafariii]

During ny works, we occasionally receive sensitive data from oil companies that they consider highly confidential, prohibiting us from sharing it elsewhere.

[EstherPlomp]

What is your challenge here? Making sure it is safely managed or are you concerned that you can't share any of that data more widely as part of any articles?

[nnadalalemany]

My biggest challenge related to data management is learning more modern tools instead of Excel, such as Python or R.

[ZQYZQYZQ]

Do use python, you can learn it from the help of Chatgpt

[Lloyd537]

I didn't put this in my comment, but as I'm reading this I fully agree! I actually started an online (self-paced) Python course recently on Udemy. I can highly recommend it if you have very little or no experience with coding. Right now it's 80 euros, but if you wait until Black Friday or some other sale, it should be very cheap - I bought it for 5 euros!

[nnadalalemany]

Hey Lloyd, thanks for the recommendation!! I should definitely do it too. Do you know if you can get credits from it?

[Lloyd537]

I'm not able to get credits from it, since it's not super-relevant to my project. However, if you can prove to the Graduate School that this is an important skill for you to have in your field; maybe they will agree to it? Worth a shot!

[EstherPlomp]

Learning programming is definitely no small feat!

The Graduate School also offers beginning courses for programming - for example the training with a focus on R as I mentioned during the first session (ignore the social sciences part there - you will just work with some tabular data). The other training includes an intro to Python but also includes an introduction to the command line and Git.

These courses also offer their materials as self paced: R and Python. I don't think the faculty graduate school will allow credits for that, so if you're short on credits for research skills it would be easier to follow one of the courses.

[msegami1]

My main challenge is keeping all collected data in an organized and understandable way (for me and for others). Often I like to work and try stuff quickly, but later it is hard to remember all the details of the experiments I tried so I need to keep good notes and descriptions of my work. However, organizing all the data I generate in a way that is sustainable long-term can be tricky.

[RiccardoDalMas]

Hi! I can relate to your challenge. Sometimes it is very tempting to try something quickly, maybe with only a small change to a previous set up, but then it is hard know which conditions were used for the results you record. This might give troubles with the reproducibility of the results themselves.

[TimTheChemist123]

I especially agree with the challenge of long-term sustainability with the way I organize my data. Since I do not yet know all the sorts of experiments I will be doing in the coming years, unforeseen issues with data labelling/storage folders etc may arise. Therefore I think improving data organization iteratively over time is inevitable.

[EstherPlomp]

As @TimTheChemist123 is mentioning, data management is an ongoing process so I'm afraid we will not solve all of your challenges with the course. Maybe a question to consider is how to make time for the organisation @msegami1? What would help you to ensure that you have the time to record everything that you need at the end of the day? Block 15 minutes in your calendar? Set up a checklist that you can follow to ensure that you documented everything?

[RozaliaP]

I can also relate to this challenge. Considering also that time is limited sometimes you are too busy, and you want to do something fast. Organizing can be very time-consuming.

[msegami1]

I will definitely try to block 15 min for this :) and with a checklist

[RiccardoDalMas]

It is very important for me to be able to trace all the inputs and outputs to the simulations I perform, keeping track in an efficient way of the scenarios analyzed. It is challenging to structure a workflow that allows me keep track of the correct inputs and outputs for each simulation and I need to be disciplined and document all the changes. This would make my life much easier down the road when I need to access results from months prior and be able to efficiently trace them to their source.

[EstherPlomp]

A challenge indeed! Something I have come across that might help here is: Dataled. I still need to checkout their handbook, but another PhD candidate was enthusiastic about it - you do need some understanding of Git to make it easier to try it out!

[Pjieter]

I will create a lot of different data, including lots of images files. Keeping the different files with different samples together will be a challenge.

[EstherPlomp]

A challenge indeed. Hopefully the sections on FAIR organisation provided some information on this. You may also want to check out the Dataled initiative, which may make it easier to track any changes to scripts/images etc.

[eugenhu]

Keeping experimental data organised and easy to retrieve, combine and analyse.

We collect a lot of multi-dimensional parameter sweeps, e.g. voltage output vs some frequency or some bias level, and the same sweeps are repeated ad-hoc over different regions of interest. The data is currently saved in a data folder with a unique id (this is done by our measurement framework), but there's no easy way to retrieve the data by asking for example: I want a point cloud of all the measurables for when the setpoints are in these intervals.

[EstherPlomp]

This sounds like something that could be 'fixed' by using a new storage solution that Arthur, your Data Steward, is working on. Maybe you've already heard about this and you've been in contact with him? It may be worthwhile to check out!

[PJL-vandenberg]

A challenge in RDM that I struggle with concerns physical data. When I work in the lab any deviations from the protocol or unexpected observations have to be logged in my logbook, but as I am usually busy dealing with the steps while this occurs, I log them late or forget (potentially losing important information). I've currently implemented a change where I have to write in the logbook, even if I had no deviations or observations, and so far that has helped.

[msegami1]

have you heard of electronic lab notebooks? ELNs? it could be an easy way to log normal and abnormal events, specially if you are working with a protocol, you could probably copy/paste a defined format and just add small comments/changes when something weird happens.

[EstherPlomp]

Thanks for sharing the ELNs @msegami1! Especially if you're using the same experimental set-up this might be a solution as you can work with templates and save minor changes in different copies of the template, as @msegami1 already explained.

It also sounds like you may have already found a solution to your challenge @PJL-vandenberg - so well done!

[giacomolastrucci]

Working with machine learning models, I need training data from the task I should perform, which must be appropriately stored for easy retrieval and labelled according to the origin, this is a crucial point that I want to improve with respect to what I have done so far.
Luckily in the last period, I started using cloud tools to store information about my experiments. However, still, there are several parameters to record in a proper manner. Lastly, I need to really improve my software design skills, together with appropriate documentation, which is crucial for future users.

[EstherPlomp]

You may want to bring up this challenge during the last session on the 18th to see how your peers are dealing with this! Otherwise there are also the training materials from the Code Refinery on documentation that I already mentioned a couple of times - sorry for the repeat!

[ZQYZQYZQ]

Perhaps the most challenging aspect for me is data management. In the field of experimental physics, we work with a variety of samples, parameters, and conditions. As a result, I have to meticulously label each file with relevant information in the header. Failing to do this can lead to a disorganized dataset that becomes a headache to navigate later on. Additionally, we're generating new data practically every second, so the demands on storage space are substantial. If the computer crashes, it's essentially a disaster for me!

[Lloyd537]

That sounds very familiar to me. I think for these kinds of projects it's very important to have a clear labeling strategy that you apply consistently. Of course, if along the way you come up with a better and more efficient way to label your samples/experiments, you can do so, but you would need to have a clear overview of the samples/experiments coming before those; or even relabel the old samples.

[EstherPlomp]

It indeed helps to think about this from the start: but perhaps you won't know everything from the start and you'll need to update indeed. Unfortunately, research data management is more of an ongoing process instead of a solution I'm afraid! I hope some of the course materials gave you some ideas of how to handle this - otherwise please do bring it up for discussion in your groups in the second session on the 18th!

[HuaChenTUDelft]

I am always scared of losing all my data, especially hearing from some of my colleagues about their own horrible stories. I do have several ways to make back-up but there is still chance that all of them go wrong.

[EstherPlomp]

Decreasing the chance that something will go wrong is all we can do sometimes! Hopefully the course will help you there. This is also another argument to make the data/code underlying articles publicly available - that way you're certain you won't loose the information :)

[FanJ-TUD]

1. Synchronize raw data and processed images on different devices; 2. remember the meaning and purpose of every data that I obtained

[EstherPlomp]

Thanks for sharing! For your second challenge it might help to set up short README files so that you don't have to remember the meaning and purpose of the datasets, because then it is recorded in a text document. You can find more about this in the FAIR documentation module.

[Lloyd537]

The main challenge I think is to effectively organize all my created/collected data in a way that is still easy to find and recognizable in the coming few years. I will have data from a lot of different measurement techniques for different parts of the project, and if I don't organize this well in a structured manner, I'm afraid I (and people after me) might lose track of it all.

[EstherPlomp]

Thanks for sharing! I hope some of the information in FAIR organisation and FAIR documentation modules will help with this challenge.

[Upsiton]

I have a challenge with storing data on many devices. Since I need to collect experiment data and analyze them, I store data on the Lab PC, my extension drive, my office laptop, and TUD network drives. It is difficult to manage, such as synchronization between devices, and avoiding overwrite data by mistake.

[EstherPlomp]

It could help here to set up a clear strategy of which version of the data is stored where, as well as to perhaps decrease the amount of storage spaces if you can. For example, your office laptop and the TUD network drives could be combined to the TUD network drives (which are automatically backed up). I hope assignment 2 was of some help to think about this!

[ieacdroste]

My biggest challenge regarding data management is the flow of data through my image processing pipeline. Different consecutive processing steps are applied to my data. The output of one step needs to match the input format of the next step. Also, some steps I perform on my laptop, some on my linux pc and some the HPC cluster or DelftBlue. The am improving the pipeline along the way, therefore the old data sometimes does not fit the format anymore. I spend a lot of time on putting the data in the correct format and moving the data around.

[EstherPlomp]

I'm not sure if the course materials provided help with your challenge. Something I have come across that could maybe help with keeping track of things is Dataled - but this does require some familiarity with Git to make it easier to use.

Please also bring up your challenge again during the group discussion on the 18th! Maybe your peers have some other suggestions that can help!

[QingLi-11]

My biggest challenge regarding the data analysis. While using equations for calculation, I sometimes use a shortcut method to replace the constant value in the equation. However, after a while when I want to re-calculate the numbers, I do not remember which numbers are calculated based on the shortcuts.

[EstherPlomp]

What can you do to either document the shortcut method better, or to shorten the normal method but to ensure that you have all the information you need? In sum: how do you avoid this in the future?

[enyasb]

I work with data from both myself and others. Keeping everything consistently organised (naming conventions, synchronising between devices, etc.) is then a challenge.

[EstherPlomp]

Indeed! Do you have any thoughts/ideas that could help here? Does your PI need to set up basic requirements? Do you need to have a discussion about this with the group?

[TimTheChemist123]

@EstherPlomp One other thing I find tricky sometimes is having a clear definition of what an experiment is. To illustrate, in my lab journal I create daily entries for the experiments I am conducting, but the end of one experiment and the start of another sometimes seems a bit arbitrary to me. Especially if I have several experiments at once, or if I continue with the experiment a few weeks later, I do not yet have a clear rule to split them up consistently. E.g. I prepare an agar plate on day 1, but use the colonies from the plate a few weeks later for an inoculation. In principle you could see it as all part of one experiment, or you could see them as different. And another example would be And another example would be doing a synthesis reaction overnight, and then doing the workup with column chromatography a week later. Or a multi step reaction where you purify the intermediate compound but use it later for the subsequent step, which can take place on the same day. etc etc.

Do you have advice for this or a clear distinction I can make?

[EstherPlomp]

This indeed sounds very complicated.. Is there any way you can assign codes/identifiers to each of the components of your experiments so that you can keep better track of it? That could be the date with an added name/number to it so you can easily trace it back. For some of this it may not be needed to connect all the dots if you've done each separate step correctly and there is no variation in between the steps that could influence the next step. But sometimes you only find out too late that it could have influenced things and then some extra recording may be a good fail safe. I think this might also be a good challenge to bring up during your discussion this afternoon @TimTheChemist123!

[EstherPlomp]

(and I think you already highlight the distinctions yourself: so there's an agar plate step, then the using the colonies for inoculation, and then synthesis reaction is another separate step from the column chromatography. Perhaps each of the separate steps does not need to be distinguished during the process, but at the end of it you should know all of the steps that have been taken. I think that is the most important information - so as long as you get there, you would be good to go!)

[Aleartulon]

my biggest challenge is data storage and quick access to it: working with Machine Learning models I both need a lot of data and quick access to it during the training. While the first issue can be solved by bigger clouds, the second one is more subtle as transferring files from one cluster to another is not always fast.

[EstherPlomp]

Thanks for still sharing your challenge @Aleartulon ! I'm not sure if there are ways to work around this challenge unfortunately! Have you already tried using Cloud4Research?