by floriangosse on 2/27/24, 11:15 PM with 253 comments
by SOLAR_FIELDS on 2/28/24, 4:17 AM
Testcontainers is the library that convinced me that shelling out to docker as an abstraction via bash calls embedded in a library is a bad idea. Not because containerization as an abstraction is a bad idea. Rather it’s that having a library that custom shell calls to the docker CLI as part of its core functionality creates problems and complexity as soon as one introduces other containerized workflows. The library has the nasty habit of assuming it’s running on a host machine and nothing else docker related is running, and footguns itself with limitations accordingly. This makes it not much better than some non dockerized library in most cases and oftentimes much much worse.
by dm03514 on 2/28/24, 12:20 AM
Pretty much every project I create now has testcontainers for integration testing :)
I setup CI so it lints, builds, unit tests then integration tests (using testcontainers)
https://github.com/turbolytics/latte/blob/main/.github/workf...
Their language bindings provide nice helper functions for common database operations (like generating a connection uri from a container user)
https://github.com/turbolytics/latte/blob/main/internal/sour...
I use them in $day job use them in side projects use them everywhere :)
by simonw on 2/28/24, 12:50 AM
I find integration tests that exercise actual databases/Elasticsearch/Redis/Varnish etc to be massively more valuable than traditional unit tests. In the past I've gone to pretty deep lengths to do things like spin up a new Elasticsearch index for the duration of a test suite and spin it down again at the end.
It looks like Testcontainers does all of that work for me.
My testing strategy is to have as much of my application's functionality covered by proper end-to-end integration-style tests as possible - think tests that simulate an incoming HTTP request and then run assertions against the response (and increasingly Playwright-powered browser automation tests for anything with heavy JavaScript).
I'll use unit tests sparingly, just for the bits of my code that have very clear input/output pairs that afford unit testing.
I only use mocks for things that I don't have any chance of controlling - calls to external APIs for example, where I can't control if the API provider will be flaky or not.
by redact207 on 2/27/24, 11:50 PM
> Creating reliable and fully-initialized service dependencies using raw Docker commands or using Docker Compose requires good knowledge of Docker internals and how to best run specific technologies in a container
This sounds like a <your programming language> abstraction over docker-compose, which lets you define your docker environment without learning the syntax of docker-compose itself. But then
> port conflicts, containers not being fully initialized or ready for interactions when the tests start, etc.
means you'd still need a good understanding of docker networking, dependencies, healthchecks to know if your test environment is ready to be used.
Am I missing something? Is this basically change what's starting your docker test containers?
by et1337 on 2/28/24, 3:56 AM
- on a Mac
- on a Linux VM
- in a Docker container on a Linux VM, with a Docker socket mounted
The networking for each of these is completely different. I had to make some opinionated choices to get code that could run in all cases. And running inside Docker prevented the test from being able to mount arbitrary files into the test containers, which turns out to be a requirement often. I ended up writing code to build a new image for each container, using ADD to inject files.
I also wanted all the tests to run in parallel and spit out readable logs from every container (properly associated with the correct test).
Not sure if any of these things have changed in testcontainers since I last looked, but these are the things I ran into. It took maybe a month of off and on tweaking, contrary to some people here claiming it can be done in an hour. As always, the devil is in the details.
edit: I did end up stealing ryuk. That thing can’t really be improved upon.
by ants_everywhere on 2/28/24, 12:58 AM
Wait what? They think you don't need unit tests because you can run integration tests with containers?
It's trivial to set up a docker container with one of your dependencies, but starting containers is painful and slow.
by fesc on 2/28/24, 11:29 AM
Testcontainers is awesome and all the hate it gets here is undeserved.
Custom shell scripts definitely can't compete.
For example one feature those don't have is "Ryuk": A container that testcontainers starts which monitors the lifetime of the parent application and stops all containers when the parent process exits.
It allows the application to define dependencies for development, testing, CI itself without needing to run some command to bring up docker compose beforehand manually.
One cool usecase for us is also having a ephemeral database container that is started in a Gradle build to generate jOOQ code from tables defined in a Liquibase schema.by domano on 2/28/24, 9:25 AM
Especially if there are complex dependencies between required containers it seems to be pretty weak in comparison. But i also only used it like 5 years ago, so maybe things are significantly better now.
by senorrib on 2/28/24, 12:09 AM
by nonethewiser on 2/28/24, 1:09 AM
Its pretty much required when you want to setup/teardown in between tests though. This just usually isnt the case for me.
by aranw on 2/28/24, 12:03 PM
by simonw on 2/28/24, 12:41 AM
That wheel file is only 2.9KB, so I grabbed a copy to see how it works. I've put the contents in a Gist here: https://gist.github.com/simonw/c53f80a525d573533a730f5f28858...
It's pretty neat - it depends on testcontainers-core, sqlalchemy and psycopg2-binary and then defines a PostgresContainer class which fires up a "postgres:latest" container and provides a helper function for getting the right connection URL.
by jake_morrison on 2/28/24, 4:59 AM
See https://github.com/cogini/phoenix_container_example for a full example. This blog post describes it in detail: https://www.cogini.com/blog/breaking-up-the-monolith-buildin...
by mellutussa on 2/28/24, 9:48 AM
Things in the software world are very trendy. If this starts a trend of making people think that they're writing unit tests when they are writing integrations tests, we are fucked.
If I need to change code that you wrote I need a lightning fast way to figure out that I haven't broken your code according to the tests that you wrote. That's unit tests.
My changes might break the whole system. That's integration tests. I just to run that once and then I can go back to unit tests while I fix the mess I've made.
by srid on 2/28/24, 8:14 PM
https://github.com/juspay/services-flake
We actually do this in Nammayatri, an OSS project providing "Uber" for autos in India.
https://github.com/nammayatri/nammayatri
There is a services-flake module allowing you to spin the entire nammayatri stack (including postgres, redis, etc.) using a flake app. Similarly, there's one for running load test, which is also run in Jenkins CI.
by alifaziz on 2/28/24, 12:16 AM
by paxys on 2/28/24, 1:36 AM
by avensec on 2/28/24, 4:34 AM
by badoongi on 2/28/24, 4:36 AM
by nym3r0s on 2/29/24, 9:35 AM
With TestContainers - I've perceived that running integration tests / a single test repeatedly locally is extremely slow as the containers are shut down when the java process is killed. This approach allows for this while also allowing to keep it consistent - example, just mount the migrations folder in the start volume of your DB container and you have a like-for-like schema of your prod DB ready for integration tests.
I've found the https://github.com/avast/gradle-docker-compose-plugin/ very useful for this.
by xyst on 2/28/24, 12:39 PM
One small note: test run time will probably increase. If a person has an outdated computer, I suspect they will have a hard time running the IT suite. Especially if it’s a complicated system with more than one dependency.
by nsteel on 2/28/24, 9:05 AM
Except where everyone is saying that's too slow and instead they have a long-lived instance which they manually teardown each time. That's even what the examples do (some, at least, I didn't check them all).
If you've already bought into the container world then why not embrace a few more. For everyone else, not sure there's much point in extra complexity (they call it simplicity) or bloat.
by nslindtner on 2/28/24, 3:18 AM
Why not keep this information in code .. often the developers are ending up doing those task anyway. (not recommended .. but seen it so many times)
Link: Microsoft aspire (https://learn.microsoft.com/en-us/dotnet/aspire/get-started/...)
by pylua on 2/28/24, 2:56 AM
by iamkoch on 2/28/24, 7:23 AM
Go has a lot of in-memory versions of things for tests, which run so much quicker than leaning on docker. Similarly, I found C# has in-memory versions of deps you can lean on.
I really feel that test containers, although solving a problem, often introduces others for no great benefit
by deathanatos on 2/28/24, 2:54 AM
That's an integration test. These are integration tests. You're literally testing multiple units (e.g., Redis, and the thing using Redis) to see if they're integrating.
Why do we even have words.
These are valuable in their own right. They're just complicated & often incredibly slow compared to a unit test. Which is why I prefer mocks, too: they're speedy. You just have to get the mock right … and that can be tricky, particularly since some APIs are just woefully underdocumented, or the documentation is just full of lies. But the mocks I've written in the past steadily improve over time. Learn to stop worrying, and love each for what they are.
(Our CI system actually used to pretty much directly support this pattern. Then we moved to Github Actions. GHA has "service containers", but unfortunately the feature is too basic to address real-world use cases: it assumes a container image can just … boot! … and only talk to the code via the network. Real world use cases often require serialized steps between the test & the dependencies, e.g., to create or init database dirs, set up certs, etc.)
by jackcviers3 on 2/28/24, 2:26 PM
Testcontainers does have a docker compose integration [1].
by febed on 2/28/24, 4:45 AM
by omeid2 on 2/28/24, 9:19 AM
by mrklol on 2/29/24, 2:04 PM
by supahfly_remix on 2/28/24, 12:17 PM
by jackcviers3 on 2/28/24, 2:44 PM
by bloopernova on 2/28/24, 1:14 AM
by leonardXu on 2/28/24, 4:12 AM
by puradawid on 3/1/24, 7:32 PM
by whalesalad on 2/28/24, 4:04 PM
by asciii on 2/28/24, 4:17 AM
by paulv on 2/28/24, 2:50 AM
by joeevans1000 on 2/28/24, 5:38 AM
by anton-107 on 2/28/24, 2:31 PM
by circusfly on 2/28/24, 4:11 AM
by Claudiusseibt on 2/28/24, 9:06 AM
by globular-toast on 2/28/24, 3:02 PM
I use a very similar thing via pytest-docker: https://github.com/avast/pytest-docker The only difference seems to be you declare your containers via a docker-compose file which I prefer because it's a standard thing you can use elsewhere.
by jillesvangurp on 2/28/24, 6:38 AM
I don't like layering abstractions on top of abstractions that were fine to begin with. Docker-compose is pretty much perfect for the job. An added complexity is that the before/after semantics of the test suite in things like JUnit are a bit handwavy and hard to control. Unlike testng, there's no @BeforeSuite (which is really what you want). The @BeforeAll that junit has is actually too late in the process to be messing around with docker. And more importantly, if I'm developing, I don't want my docker containers to be wasting time restarting in between tests. That's 20-30 seconds I don't want to add on top of the already lengthy runtime of compiling/building, firing up Spring and letting it do it's thing before my test runs in about 1-2 seconds.
All this is trivially solved by doing docker stuff at the right time: before your test process starts.
So, I do that using good old docker compose and a simple gradle plugin that calls it before our tests run and then again to shut it down right after. If it's already running (it simply probes the port) it skips the startup and shut down sequence and just leaves it running. It's not perfect but it's very simple. I have docker-compose up most of my working day. Sometimes for days on end. My tests don't have to wait for it to come up because it's already up. On CI (github actions), gradle starts docker compose, waits for it to come up, runs the tests, and then shuts it down.
This has another big advantage that the process of running a standalone development server for manual testing, running our integration tests, and running our production server are very similar. Exactly the same actually; the only difference configuration and some light bootstrapping logic (schema creation). Configuration basically involves telling our server the hosts and ports of all the stuff it needs to run. Which in our case is postgres, redis, and elasticsearch.
Editing the setup is easy; just edit the docker compose and modify some properties. Works with jvm based stuff and it's equally easy to replicate with other stuff.
There are a few more tricks I use to keep things fast. I have ~300 integration tests that use db, redis, and elasticsearch. They run concurrently in under 1 minute on my mac. I cannot emphesize how important fast integration tests are as a key enabler for developer productivity. Enabling this sort of thing requires some planning but it pays off hugely.
I wrote up a detailed article on how to do this some years ago. https://www.jillesvangurp.com/blog/2016-05-25-functional-tes...
That's still what I do a few projects and companies later.
by mrAssHat on 2/28/24, 12:45 AM
by m3kw9 on 2/28/24, 1:16 AM
by sigmonsays on 2/28/24, 1:14 AM
Running integration tests are significantly more complicated to write and take longer to run.
There is also race conditions present that you need to account for programmatically.. Such as waiting for a db to come up and schema to be applied. Or waiting for a specific event to occur in the daemon.
That being said, this looks like a decent start. One thing that seems to be missing is the ability to tail logs and assert specific marks in the logs. Often you need to do an operation and wait until you see an event.
by friedrich_zip on 2/28/24, 11:14 AM