Add pandas comparison

[tk3369]

This PR adds a new section to compare with Python pandas.

Couple things:

1. I have fixed some other typos in the dplyr section as well.
2. I have formatted the markdown table so it's easier on eyes when editing. If desired, I can apply formatting to dplyr and stata sections as well (I'm using a vscode extension so it's literally "1-click").
3. I have kept the same use cases as copied from the dplyr section; however, I'm uncertain if this is the "best" list. Maybe we should run the list by more R/Python/Stata users and get some feedback.

Disclaimer: I'm not really proficient with pandas so if anyone can suggest better ways to write the sample code, please feel free to suggest below.

[bkamins]

Thank you for this PR

I have fixed some other typos in the dplyr section as well.

Thank you for spotting them.

I have formatted the markdown table so it's easier on eyes when editing. If desired, I can apply formatting to dplyr and stata sections as well (I'm using a vscode extension so it's literally "1-click").

Sure. Also maybe add information how to create this example mini-table for dplyr and Stata also for consistency.

I have kept the same use cases as copied from the dplyr section; however, I'm uncertain if this is the "best" list. Maybe we should run the list by more R/Python/Stata users and get some feedback.

I would try to finalize this PR, and then ask on #data in Slack if people have suggestions.

[nalimilan]

Thanks. While you're at it, can you fix two issues with the previous PR? These are 1) add a link to this page in the sidebar, and 2) check that in the generated HTML docs tables are rendered correctly (by running julia docs/make.jl).

[bkamins]

Both are already fixed on master. We only need to update the title to include pandas

[tk3369]

@bkamins Thanks for the feedback. I have revamped it further with a whole new subsection for indexing and joins.

[tk3369]

The latest commit includes

• Update to the left nav menu as Comparison with Python/R/Stata
• Keeping only one way to do things in the DataFrames.jl column in the Accessing Data section
• Removed the note about 1-based index
• Removed the note about the off-by-1 issue due to the additional id column. I think that might make it more confusing.
• Updated pandas sample code when calculating mean - using df['x'].mean() rather than df.mean().x

[tk3369]

Summary of last commit:

• Added examples/notes about handling missing data
• Added note about 1-based indexing
• Fixed indentation problem

P.S. I haven't been focusing on dplyr/stats sections. Once the pandas part is good, we can sync those up. I don't have access to Stata though so I will need some help by then.

[bkamins]

I will squash the PR to a single commit when merging.

[bkamins]

Please let me know when you are done with simplifying this PR and then I will do a review.

[tk3369]

Yes, I'm done now.

[bkamins]

Looks good except for some minor comments that I think should not be problematic.

[bkamins]

@KrainskiL - please run these examples and comment if you see something that confuses you as a user.

[KrainskiL]

I've got 3 comments:

1. Initially I was confused that I received different results by running

df.iloc[1, 1]	df[2, 2]

Of course this is due to additional id column in Julia's DataFrame, but still I was expecting to get the same number - maybe id should be moved to the end?

2. In following snippet:

df.loc[:, 'x']	df[:, :x]

df[:, :x] is returning an Array. I think it's worth mentioning that df[:,[:x]] will return a DataFrame, which is often desirable.

3. Maybe mention begin in the following part?

A special keyword end can be used to indicate the last index.

[tk3369]

@KrainskiL Thanks for the suggestions.

df[:, :x] is returning an Array. I think it's worth mentioning that df[:,[:x]] will return a DataFrame, which is often desirable.

Is it really? In my use cases, I never select a single column and still want a data frame as a result.

[bkamins]

I think we can skip comment 2 and 3 by @KrainskiL, but comment 1. should be fixed I think as indeed it is confusing (either by moving :id column to the end or by using a column name instead of column number, which is a recommended practice in general anyway)

[bkamins]

Thank you for working on it.

Let us get the ball rolling and merge this PR.

If we find anything to be fixed let us just open a PR (and also - as discussed - please open atomic PRs for the "debatable" things)

[tk3369]

@bkamins I have a second thought about the first two forms, which mutate the existing data frame. Pandas returns a copy and so it works better with transform. Shall we remove them? We could also mention in the notes about the mutation forms. For that matter, we could also add transform!.

[bkamins]

You mean that df.assign(z1 = df['z'] +1) copies df. I would be surprised if this is so (but maybe it does). But if it does not do a copy then we should use transform! not transform indeed. Just please open a PR fixing it. Thank you!

[tk3369]

Well, pandas does make a copy. Same behavior for rename and sort_values. I'd say that's unintuitive, and this is the time to appreciate Julia's convention of using !.

>>> df
   grp  x  y    z
a    1  6  4  3.0
b    2  5  5  4.0
c    1  4  6  5.0
d    2  3  7  6.0
e    1  2  8  7.0
f    2  1  9  NaN
>>> df.assign(z1 = df['z'] + 1)
   grp  x  y    z   z1
a    1  6  4  3.0  4.0
b    2  5  5  4.0  5.0
c    1  4  6  5.0  6.0
d    2  3  7  6.0  7.0
e    1  2  8  7.0  8.0
f    2  1  9  NaN  NaN
>>> df
   grp  x  y    z
a    1  6  4  3.0
b    2  5  5  4.0
c    1  4  6  5.0
d    2  3  7  6.0
e    1  2  8  7.0
f    2  1  9  NaN

[bkamins]

Yes - and I have checked that it is a copy indeed (so there is no column aliasing - as what you show does not check for this). Can you then please - as usual 😄 - add a fix to the examples in a separate PR?