by jaxxstorm on 9/4/23, 3:28 PM with 162 comments
by eduction on 9/4/23, 5:29 PM
Is SQL a “waste of time?” Regular expressions, HTML, Makefiles, CSS (and CSS selectors aka jquery selectors)?
It’s the bad ones that are a waste of time. The ones still called “dsl” instead of just “language, “format,” or “syntax.”
by chubot on 9/4/23, 5:58 PM
Survey of Config Languages - https://github.com/oilshell/oil/wiki/Survey-of-Config-Langua...
It puts a bunch of languages in 5 cateogires.
Terraform is at least not a YAML DSL, a very common pattern which I don't understand the appeal of. I guess the main appeal is that you can copy and paste, and pretend it's data in simple cases.
But even Terraform suffers from the problem of "external" DSLs -- you end up needing a general purpose language. And yes this pattern has repeated itself so many times.
Awk and Make both grew into general purpose languages, and in Make's case it's particularly bad.
Most SQL dialects have also grown a pretty bad stored procedure language.
So I do think making internal DSLs in general purpose languages is more promising. But they have to be designed for it. There have been many attempts to use Python as a DSL/config language, but it's missing a few things for that, and gets awkward pretty quickly.
Ruby does a bit better, but it's not perfect either. It's fairly big and heavy if you just want a config file.
by marcosdumay on 9/4/23, 5:25 PM
Yes, platform specification languages are a mess. Making them without the "language" part keeps all of the mess (the language is absolutely not the source of it). There are people trying to fix this, although I'm not sure if I'd place Pulumi on that group.
And DSLs are still a great software architecture technique with all kinds of applications.
by joshmarinacci on 9/4/23, 5:37 PM
They always start simple and easy, but they always eventually grow so complex that using a real language would be preferable.
I’ve never seen a system remain at the same level of complexity (or much less shrink) over a multi year timeline.
by fishnchips on 9/4/23, 8:07 PM
I think the redeeming factor of Terraform is not the language itself, though I really like the fact that it's not a programming language. But I think the redeeming factor is the tool that interprets this language, and the fact that this tool can be taught new tricks - one of which is the CDKTF (not a fan, I think we can do better than that).
Here's an analogy. Most folks I know are at least skeptical about Java. The same folks would then praise Kotlin, Scala or Clojure as "good" JVM languages. The relative success of these languages owes much to the Java ecosystem and the JVM interop.
That's the reason I find Terraform worth saving. It's the JVM and the Java ecosystem of our cloud world. We can still build great things on it, whether you like HCL or not.
by waffletower on 9/4/23, 5:13 PM
by OJFord on 9/4/23, 4:57 PM
by denton-scratch on 9/4/23, 5:49 PM
VB did that to me; the lesson I learned is to avoid[0] languages with just a single implementation. As a consequence, I'm not very interested in languages like Ruby, Go and Rust. In the old days, every language had a train of compilers/interpreters following it. Pascal, COBOL, FORTRAN, even ALGOL. I believe that in those days, building a compiler for an existing language was a pretty standard hobby project. Everyone and his dog had built one, and manufacturers bundled their own ones with the hardware.
[0] Learning is hard; I jumped from VB to PHP.
by spion on 9/4/23, 5:58 PM
TypeScript is a great fit to model the unwieldy yaml / json schemas with precision, getting less in the way compared to most languages with sum types. Deno could be a great player in this space that avoids the complexity of npm, especially with its sandbox to restrict side-effects, networking and external commands.
CDKs come with their own multi-language compiler (jsii-rosetta) which results with a ~300MB npm install. Even when all you wanted was to use simple typescript functions to output some JSON and/or yaml.
by cogman10 on 9/4/23, 5:36 PM
This dance has played out with the likes of the Javascript community. Just write a js script, no actually we need a pipeline so let's use grunt. Actually we hate grunt so let's use gulp, actually gulp is just wrong lets get declarative with webpack. Actually that's too inflexible and weedy let's just use a framework (like react-scripts) which just runs our code and gets out of the way.
After 10+ years of programming, it seems like we are on an endless loop of "let the framework do everything" to "let's enable maximum flexibility with a full programming language". (see ant in the java world for an even earlier version of this). It seems like the push and pull is this constant problem that "describing the problem is hard, and messing up that description makes everything more complex to maintain". The lack of flexibility feels nice because there's less to mess up, but then when something really complex arises you usually end up doing something 10x more nasty than you would with an ultimate flexibility framework. On the flip side, super flexible frameworks lend themselves to being completely unique butterflies that require a heavy up front research to understand (as, you don't know what that foo method is doing and why it does it). They can 1x the complex situation but it feels like the tendency is to want to overcomplicate simple problems by reinventing a framework for your company. (My company is dealing with this and jenkins pipelines)
by gumby on 9/4/23, 5:33 PM
People often talk about Lisp as a language in which people specify DSLs, but that acronym was something I only encountered outside the Lisp community. The way it always felt to me was that I "simply" designed a set of datastructures and operations around the abstractions and metaphors of the problem domain and then manipulated them to solve my problem. Isn't that the essence of programming in any language?
by phendrenad2 on 9/4/23, 5:17 PM
by OhMeadhbh on 9/4/23, 5:25 PM
by fnordpiglet on 9/4/23, 5:33 PM
I tend to agree on that point. I think CDK and related approaches are much easier. I also find just writing SDK based scripts faster and more reliable. I find very little advantage in things like Terraform, despite my driving a few megacorps to adopt it. (Listen, in my defense, I couldn’t sell the idea of just scripting things, and generally they were repurposing admins who had no programming experience and had a mental block against learning a real language - but somehow a configuration file like language was ok)
However I’ve written a ton of DSLs in my life. They’re always useful, because I write them when I find it cumbersome to not. The broad statement about DSL is of course absurd. However at one point in my life I built a system that made it easy to write DSLs in a monorepo, and it was a huge mistake. Everyone wrote dsls everywhere and you constantly encountered code in some weird language that you had to reverse engineer. So, there’s a DSL entropy principle to be aware of - DSLs are useful if they are bounded. But like anything too much of a good thing ain’t
by raffraffraff on 9/4/23, 6:51 PM
Maybe DSLs are a waste of Lee's time because he knows they'll be "dead languages" when the next major evolution of systems takes place. But meh, I'll learn the next DSL, and maybe even hook hiera up to it too.
by dunk010 on 9/4/23, 7:04 PM
> The whole nasty "configuration" problem becomes incredibly more convenient in the Lisp world. No more stanza files, apache-config, .properties files, XML configuration files, Makefiles — all those lame, crappy, half-language creatures that you wish were executable, or at least loaded directly into your program without specialized processing. I know, I know — everyone raves about the power of separating your code and your data. That's because they're using languages that simply can't do a good job of representing data as code. But it's what you really want, or all the creepy half-languages wouldn't all evolve towards being Turing-complete, would they?
Via: https://sites.google.com/site/steveyegge2/the-emacs-problem
by tremon on 9/4/23, 6:41 PM
If you want to represent a data structure, PLEASE pretty please just use an existing data format (json, yaml, toml, xml). Those structures can be parsed and validated by generic tools, and you won't need to write your own domain-specific parsers and syntax highlighters to deal with a problem of your own making.
And if you want to allow for runtime evaluation of constructs, PLEASE pretty please just use a programming language that has friendly syntax (lua or python comes to mind). Don't get cute and try to invent your own language, just provide a good API on top.
by jesseryoung on 9/4/23, 6:43 PM
As somebody who identifies as a "programmer" and often has to manage infrastructure I would absolutely love for a serious IaS library to pop up in my favorite language. I would switch to it immediately. However unless we start hammering home that if you are regularly writing in these DSLs that you are actually a programmer and there's nothing wrong with that people will still flock to these tools.
by jamesblonde on 9/4/23, 6:00 PM
In this case, DSLs may have a short-term wow factor, that you don't have to build an maintain feature pipelines, but you're always playing catchup when the latest feature engineering technique is not available in your DSL. And then, you have your developers careers to think of - do they want to put a DSL for feature engineering on their resume?
by intrasight on 9/4/23, 4:52 PM
Perhaps if no relational database existed before today, the standard grammar would just be javascript - but I think not.
by kwhitefoot on 9/4/23, 6:53 PM
As for being left with expertise in a language that no one uses any more, welcome to the club. My first language was Algol-60, the next was Leasco Basic, after that I wrote some Fortran IV, then whatever dialect of Dartmouth Basic was on the Exeter Uni time sharing system, then Z80 assembler, then 6502 assembler, followed by Turbo Pascal in its various versions (I skipped all the 'standard Pascals), more Fortran (but I forget which version) then VB4, 5, 6, then VB.Net then C#. SQL was involved for quite a lot of the time too. Then I retired and now I play with Nim.
Were all of those a waste of time too? I certainly have no use for the specifics of Fortran, Algol, Dartmouth Basic, VB.Net, any assembly language, or even C# now, but they were all useful at the time and I created things that were important and valuable for the companies in which I worked.
by stevenally on 9/4/23, 5:21 PM
by aranchelk on 9/4/23, 6:54 PM
The syntax shouldn’t take you terribly long unless it was badly made.
In my experience virtually any project whether you use a standalone DSL, an embedded DSL, framework, library, or writing something yourself, you will have to understand the underlying domain to be useful.
The model of execution can be simple or possibly tricky. There may be a small benefit to having an embedded DSL or framework over a standalone DSL as information learned could be transferable, but model of execution could still be more complicated and different than the underlying language in the other case. I’m thinking of things like Chef, Angular, and Parsec.
That leaves the vocabulary which you’ll either need to learn or develop.
You’ll take all of the domain knowledge, which as already stated, is the important bit. If you work in the same domain, even if it’s with a very different language, model, etc., you’ll have a great insight into what’s going on.
I feel like the opinions expressed in the article really just show a lack of experience of what happens after you’re done working with a piece of tech.
by skywhopper on 9/4/23, 6:28 PM
His examples of confusing Terraform are not particularly interesting without some counterexample showing how his preferred CDK-style alternative is better (spoiler: it isn’t any less messy).
In fact, CDK/Pulumi-style programming is actually just an imperative-style wrapper that generates the underlying declarative IaC code. You might even call it a reverse DSL. But adding one more layer of abstraction away from what’s actually happening isn’t actually useful. It just adds more seams and more opportunities for leaky abstractions. Using a generic programming language to wrap your declarative infrastructure actually encourages more layers of abstraction. Not a good idea.
Could Terraform’s language be improved? Yes. But replacing it with JavaScript or Python is not the answer.
by zamalek on 9/4/23, 6:20 PM
I mean, sure, you don't get that automagic in-between later reuse - but I've rarely come across a project that does the magic incantations to get that to work anyway. Rather delegate the rarely changing stuff to a base image, probably updating in CI.
by olvy0 on 9/4/23, 5:59 PM
Since then the situation hasn't gotten any better, but sort of stayed the same. Two people on our team have been solving bugs in this DSL for the last few months, rather minor ones. Tomorrow in fact I have a meeting with one of them about adding a sort of limited pass-by-reference semantics into the language (which currently doesn't allow pass-by-reference). I'm very torn about it, as I feel this is a waste of time, and we should ditch this DSL. Users complain rightly that it isn't a "real" programming language. Our "standard library" is very very lacking, for example there's no I/O.
We just added FFI 2 months ago. My team lead who was supposed to do this years ago didn't, and in fact effectively stopped developing it 2 years ago, feeling overwhelmed, and ended up leaving a couple of months ago.
What this language or rather ecosystem DOES do well is having an internal build system that is completely transparent to the users (unless there are bugs). Well, I designed this build system so I'm partial to it.
I had actually written the C# API which I wrote about at the end of [0], that was supposed to be a replacement of this DSL with a "real" language. However I got burned out a couple of months ago after my team lead left, and halted development. There was also a long series of heated discussions with some of my users, who wanted me to remove some of the (rather simple) guardrails I made, and give them the ability to build complex hierarchies above this API, an API which I designed for simplicity. I warned them those hierarchies would end up creating DLL hell and would make it necessary to build devops tooling and maybe even infrastructure as code [1], which IMO is silly for most of the internal projects, which contain at most 5-6 people. From 2 different organizational groups, but still. My users aren't developers per se, and have a more or less support role.
I'm still torn as to what the correct direction here. Long term our small team (only 2 full time developers and 3 part time) cannot support 2 half-assed non-standard DSLs, plus that C# API for our many internal users. Something gotta give.
[0] https://news.ycombinator.com/item?id=31205265
[1] There, I tied this response to the OP!
by pgib on 9/4/23, 5:35 PM
by pyrolistical on 9/4/23, 5:50 PM
Tools like terraform offer simple modules to make your devops life easy.
But simple looking modules abstract a complex thing by hiding details. Details which are critical once your infrastructure reaches a critical complexity threshold.
At this point the tool is a burden. It is actively getting in your way to understand and get at the the actual infrastructure you are maintaining.
This is why imo alternative like CDK got popular. It doesn’t try to over abstract for you. It just gives you an api you write custom code over. The complexity of your code goes up with the intrinsic complex of the infrastructure.
by jauntywundrkind on 9/4/23, 6:13 PM
By compare every dsl requires learning some new language. The languages seem to grow and grow. I know plenty of people with a little bit of Terraform experience, but almost no one with extensive knowledge. It's great that surfing along the top works as well as it does, and having such a confined language keeps a normativeness, but in general I so strongly feel it's just better to use available known well supported tools.
by computerfriend on 9/5/23, 8:04 AM
True, but it's not an either/or situation. Maybe it's not optimal, but a lot of people do both. I consider myself a Python and Terraform expert and I'd say I reached that stage in Terraform in less than 10% of the time it took for Python.
by TheGRS on 9/4/23, 5:46 PM
The example given is pretty good at demonstrating how Terraform itself has some unique problems that they themselves created: using counts for enabling/disabling resources, using locals as proxies for data comprehension, writing long-ass resource descriptions, often with built-in functions and ternaries to get it to do what you want in various scenarios. Trying to be DRY in Terraform can often be a chore. I think Terraform's MO is actually that some duplicate code isn't the worst thing, so you end up with piles of duplicate lines that are extremely difficult to read, and where writing a function would be super helpful. We had an engineer do a bunch of Terragrunt work for a project and I found that code nigh-incomprehensible.
So, I get it as well, I get where they're coming from.
I also think working with DSLs isn't a huge waste of time, since many of the things we work on are abstractions of abstractions. It seems like everything I'm doing today is probably going to go away in due time, but I know how to program, read docs, and pivot quickly. I do feel like as long as you're gaining an understanding of the underlying elements, your time spent in a DSL (assuming you're getting paid or you end up using it for several years), is still well spent.
They mention Puppet, and I remember getting a job offer once because I was able to read the Puppet documentation and write out a little script that could do the ask in Puppet, even though I had never used it in a professional capacity before. It was easy to pick up and I would've happily worked in it if I accepted the offer. The knowledge I gain around deploying to AWS or other cloud providers is often transferrable in that manner too.
The clouds themselves are a sort of DSL often, but I can't say that my time spent with Azure was a waste just because I'm working in AWS now. No, we delivered stuff on that platform, I learned its pros and cons, I gained some wider understanding of what cloud platforms could do, and I got paid along the way :)
by turtleyacht on 9/4/23, 4:44 PM
budget: $23000
pricing: cheapest
cpu:
preferred: xeon
cores: 8
mem: 64gb
disk:
preferred: ssd
by SonOfLilit on 9/4/23, 6:33 PM
However, this is not at all an argument against DSLs. Using a "real" language to specify the cloud infrastructure, you'd still want your API to be declarative and ergonomic, i.e. you'd want a DSL.
by AtlasBarfed on 9/4/23, 6:26 PM
A lot of DSLs are a wall to understanding what each "figure of speech" actually does. You need to navigate a parser before you get to the underlying execution of the language
by ec109685 on 9/4/23, 5:39 PM
Really, if OpenTF happens, you’ll be back to square one?
“ When the maintainers of OpenTF inevitably decide that they want to add enhancements that require language changes, you as the end user end up in a situation where you’re effectively back to square 1, learning (potentially) a new DSL.”
by mvaliente2001 on 9/4/23, 6:28 PM
by _Algernon_ on 9/4/23, 6:30 PM
Define your abbreviations, people, especially when its not the first hit when googling it. It's not that hard and makes your writing much more accessible.
by yegle on 9/4/23, 11:58 PM
[for x in x.addresses: x if length(regexall(".*[:].*", x))>0]by thrashh on 9/4/23, 6:38 PM
Throwing a full programming language in place of a DSL means now you deal with everyone and their mom inventing their own way of doing things.
by julienfr112 on 9/4/23, 6:52 PM
by specialist on 9/4/23, 5:28 PM
I have no dog in this fight. But I do have opinions about DSLs.
The declarative vs imperative, or some hybrid, debate is ongoing with build systems.
James Duncan Davidson, author of Ant build system for Java, famously wrote a post-mortem. Mostly wrt to choice of XML for Ant's syntax and semantics. TLDR: Wheels fall off a declarative model once imperative logic is added.
FWIW, Here's how I think about declarative DSLs. Such as a scenegraph.
You have an exquisite mental model for a scene. It's probably hierarchical.
You attempt to capture that mental model in code. The representation is probably a grove data structure. aka Directed Acyclic Graph where nodes can have key/value pairs for metadata.
You need to serialize that grove. So you pick a suitable syntax. Hopefully something that looks like VRML-97. Basically s-expressions (JSON without the syntactic vinegar), with some way to represent object prototyping (define/use), and don't forget identifiers ("id" field). Bonus points if you include path expressions in your syntax.
That (human readable) serialized grove is the declarative DSL.
In an future perfect world, your mental model, runtime representation, serialization format, AND the real world will be isomorphic.
Not isomorphic? Of course they're isomorphic. Why even mention it?
Funny you should ask.
In the Java world, dom4j for representing XML documents, and most of its progeny, is not isomorphic. Only JDOM2's class hierarchy design actually matches the serialization format. (As of the last time I checked, a few years ago.)
dom4j's mismatch, leaky abstraction, poor design, what have you, begat untold additional labor and heartache. Which is all completely mooted by JDOM2's correct design.
Ensuring correct isomorphism avoids errors in the same way that strong typing and garbage collection do.
To wrap this up...
In my experience, isomorphism is the exception.
I have no idea if Terraform is a good declarative DSL. But I do know that it should be isomorphic with the runtime representation AND the real world. And that should be settled before weaving in imperative functionality.