from Hacker News

What would it take to recreate dplyr in Python? (2020)

by bshanks on 1/15/22, 7:11 PM with 59 comments

by sweezyjeezy on 1/17/22, 1:13 PM
Always been interested to know why pandas implemented index the way it did. I generally find myself doing .reset_index on everything by default because it's just one less thing to think about, but it's clear that pandas devs are very fond of it based on the API. Where it still feels weird is when e.g. groupby/pivot by default return everything with a custom index, when I've given no indication that it needs to be treated differently to a column, but then e.g. merge doesn't do this? It's also written out by default in .to_csv, like just... why? Not useful for any csv that is to be used outside pandas. God help you if you end up needing to use a multi-index for something - deeply unpleasant.
Was this just a high level (possibly misguided) paradigm that the pandas devs fell in love with - or is there a good, performance related reason to embed it so deeply in the API?
by peatmoss on 1/17/22, 3:34 PM
Dplyr (and its spin-off dbplyr) is to me a fantastic practical example of the power of lispy metaprogramming. While R gets knocked a lot for not being a "real" programming language, or being weird, or just generally being different than e.g. Python, I don't know that I've seen a cooler example of high-level programming language ideas expressed as cleanly.
Ditto the rest of the tidyverse.
by dunefox on 1/17/22, 5:23 PM
As a comparison to another language: for Julia there is DataFrames.jl: https://dataframes.juliadata.org/stable/
Comparison to dplyr, ...: https://dataframes.juliadata.org/stable/man/comparisons/#Com...
Comparison to Pandas: https://dataframes.juliadata.org/stable/man/comparisons/#Com...
by mgradowski on 1/17/22, 1:19 PM
DuckDB and Polars are my bets in the Python data-wrangling space. I grew tired of Pandas' weird-ass API.
by closed on 1/17/22, 4:07 PM
Author here--happy to answer questions :)
Siuba has come a long way since I wrote this, and now can optimize for fast grouped operations!:
* https://github.com/machow/siuba
* https://siuba.readthedocs.io/en/latest/developer/pandas-grou...
by psimm on 1/17/22, 10:29 PM
This article compares dplyr syntax with pandas, siuba, polars, ibis and duckdb: https://simmering.dev/blog/dataframes/
As other have said, escaping pandas is hard. Many visualization and data manipulation, validation and analysis libraries expect pandas input.
Siuba is really cool in that it offers a convenient syntax on top of pandas (and SQL databases) without requiring its own data format.
by lysecret on 1/17/22, 1:04 PM
Hey regarding groupby operations in pandas. I have been moving more and more (for too complicated groupbys) to running them something like:
```
   out_rec = []
   for id, group in data_frame.groupby("id"):
      ladidida....
      result = f(group)
      out_rec.append(result)
```
in my experience it isn't much slower than a groupby.apply.
by armanboyaci on 1/17/22, 7:23 PM
I am distracted with the example provided in the post. I am pretty sure that most pandas users will just put course_ids on the columns. I mean the shape of the user_courses dataframe is not suitable for the task.
```
  (user_courses
     .set_index(["student_id", 
                 "course_id"])
     .unstack()
     .apply(lambda x: x+1))
```
by tpoacher on 1/17/22, 10:35 PM
Meanwhile, every time I use an external library (including pandas) I still think numpy does everything you need, and does it well, and people just haven't bothered to learn it properly and keep reinventing the wheel.
(no disrespect to to the package in the article or OP who I know is active in this thread. just a general motif that I keep coming across in python).
by usermi on 1/17/22, 2:57 PM
There is a project called Datar (https://github.com/pwwang/datar), which mimics dplyr in Python.
by mint2 on 1/16/22, 4:29 PM
That looks neat and was very interesting. Pandas is sometimes a dark art to know how to do something fast enough.
Port the functionality of the R package but try to keep it python. Run flake8.
by bobolito on 1/17/22, 2:19 PM
Check out Tidypolars