Daniel you told that we should first do some small experiments and then do bigger experiments...

[Programmer-RD-AI]

Daniel, you told that we should first do some small experiments and then do bigger experiments,
So does it mean by lowering the amount of data?
If yes how can I remove a percentage of data from my panda's data frame?

• Ranuga

[mrdbourke]

Hey Ranuga,

That's correct, you can do experiments by lowering the amount of data you use.

For example, if you have 100,000 rows, you might try to get something working on 1,000 rows (1% of the data) and then scale up to the full 100,000 when you find it works.

You can remove or sample data from a pandas DataFrame using indexing or the sample() method.

# Indexing
df_1000_rows = df[:1000] # get first 1000 rows of df

# Sample method
df_10_percent_sample = df.sample(frac=0.1) # get 10% random sample of df

[Programmer-RD-AI]

Does it differ for the experiments because if I run my experiments in the 10% of the data does it change for the whole dataset?

Thank you.

[mrdbourke]

It really depends on what kind of problem you're working with. Sometimes 1% of the data is enough to get started, sometimes you'll want to use 10%.

Either way, the important point is to sample the data in a way that retains the overall distribution of samples. To do this, I'm more of a fan of the sample() method because it samples a DataFrame randomly, in turn, hopefully preserving the distribution of samples throughout.

The experiments you run on 10% of the data will influence how you scale things up to 100% of the data.

What you're trying to do is figure out things which don't work so when you do scale up, you're confident in what you're trying will lead to something.

[Programmer-RD-AI]

OK Thanks you so much