from Hacker News

Metaprogramming in Python

by manjana on 9/16/22, 10:30 AM with 58 comments

by samwillis on 9/17/22, 3:07 PM
This oft-repeated quote is always relevant on Python metaclass posts:
“[Metaclasses] are deeper magic than 99% of users should ever worry about. If you wonder whether you need them, you don’t (the people who actually need them know with certainty that they need them, and don’t need an explanation about why).”
Tim Peters, Inventor of the timsort algorithm and prolific Python contributor
https://www.oreilly.com/library/view/fluent-python/978149194...
https://en.m.wikipedia.org/wiki/Tim_Peters_(software_enginee...
I would then also concur with the other comment that if you “know” you need metaclasses, 99% of the time actually you only need __subclass_init__.
A lot of online literature about Python meta programming misses out __subclass_init__ as it was only added to Python 3.6 in 2015 via PEP 487.
https://peps.python.org/pep-0487/
by narush on 9/17/22, 3:33 PM
This is a cool overview, and I certainly learned new things about the Python language from it. Thanks for posting!
We do lots of Python metaprogramming at Mito [1], but generally avoid all of this fancy Python fluff to get it done. Specifically, we avoid metaclasses, invisible decorators, etc. Instead, we take a much simpler approach of having Python code that literally populates as template .py file, and then writes it to the correct location in our codebase.
As a concrete example: we’re a spreadsheet, so we let our users transform data in a bunch of different ways - adding a column, writing a formula, deleting some rows. Anytime I want to add a new transform (say, encoding a column), I tell the metaprogramming package “python -m metaprogramming step —name “Encoding A Column”. It will ask me some questions about the parameters and types of those parameters, and then write most of the 4-6 boilerplate Python and Typescript files I need automatically! You can see it here [2].
This is still metaprogramming (it’s certainly code that writes code). But the code you end up with at the end of the day is very simple Python code that is extremely easy to understand / maintain long-term.
I’ll pass on the fancy stuff for now. Thanks though!
[1] https://trymito.io
[2] https://github.com/mito-ds/monorepo/blob/dev/mitosheet/dev/c...
by kkirsche on 9/17/22, 10:14 AM
Thanks for sharing! Instead of using meta classes for subclass registration though, please just use __subclass_init__ instead. It’ll often be simpler unless you have a complex use case.
https://docs.python.org/3/reference/datamodel.html#object.__...
by wheelerof4te on 9/17/22, 12:28 PM
It is shocking how much vodoo magic there is in a programming language marketed as "easy to learn and use".
And with the recent addition of new syntax, I would hardly call Python easy.
by rroot on 9/17/22, 1:09 PM
Python is an amazing language. It's basics are very easy to learn and I'd like to think it doesn't feel like a incomprehensible sorcery to the beginner.
But even for that, it's not a toy language as clearly evident by this example.
by pdonis on 9/17/22, 3:23 PM
The article's reference to "classobj" is wrong for Python 3 (that was the type of old-style classes in Python 2). In Python 3:
```
    >>> class SomeClass:
    ...     pass
    ...
    >>> type(SomeClass)
    <class 'type'>
```
by ngcc_hk on 9/17/22, 12:55 PM
The reason why you have to reinvent lisp when you go deep enough is that if you are not, it is getting too hard and too confusing.
by filipew on 9/17/22, 11:22 AM
Small typo at the start of the code examples.
We define the instance as: someobject = SomeClass()
but then we refer to it as someobj and some_object
by carapace on 9/17/22, 3:11 PM
This is awesome! Now, never ever do it.
Python is powerful and flexible enough that you don't need metaprogramming.
(I mean this quite literally: I sincerely doubt that there any code in Python using metaclasses etc., that wouldn't be more clear and maintainable if rewritten in "plain old" Python without them.)
(With the caveat that I'm not including "art" projects, I'm talking about working production code.)
(In case it's not clear, this is one of those "Prove Me Wrong" scenarios... If you think you have a counter-example to my claim, please don't keep it to yourself, "Shout it out so the whole theatre can hear you!")
by ok123456 on 9/17/22, 3:53 PM
It's interesting how differently two similar dynlangs, Python and Ruby, decided to offer meta-programming. Ruby decided to offer a very simple way of evaluating code at construction, but python has you define custom metaclasses. One sees it as a routine way of extending the language, and the other treats it as a 'black-art'.
Metaprogramming done wrong, no matter the mechanism, is very painful. So, I'm not sure which way is better.
by behnamoh on 9/17/22, 2:06 PM
It’s interesting that meta programming, one of the main selling points of lisps, is actually pretty common and possible in other languages as well. Lisp’s “code = data” achieves meta programming in a straightforward way, but this link shows that it’s not necessary. In fact, Python’s way might even be better because the language sorta “gets outta the way” because it’s really darn simple.
by xenophonf on 9/17/22, 3:01 PM
Stuff like this is how I created a very quick-and-dirty SaltStack module for AD Federation Services:
https://github.com/irtnog/salt-states/blob/production/_modul...
The SaltStack module starts out empty except for a function that runs at module load time. The initialization function queries PowerShell for all AD FS-related cmdlets and creates Python wrapper functions for them. It even copies the cmdlet's help to the function's docstring.
by stuckinhell on 9/17/22, 4:49 PM
As someone who just dabbles in python for small scripts, this is fascinating to me.
You can use some metaprogramming to create very clean interface points in python! I always wondered how django did so much with very clean readable implementations for end users.
by siaw23 on 9/17/22, 11:43 AM
very interesting, bookmarking for later. thanks.