from Hacker News

Ask HN: Best Architecture Patterns for Integrations?

by Syntaf on 6/17/24, 2:44 PM with 5 comments

Hey all, anyone here have personal experiences or resources to share on how companies have successfully approached designing an architecture to handle N integrations syncing down into one application?

More often than not I've seen these systems designed around an inheritance structure which frankly I don't agree with -- given enough time integrations will diverge enough that the once "shared" functionality no longer applies and you end up with a mess of inherited methods and overridden function chains.

I'm not _really_ sure what the alternative to OOO + inheritance is in designing an integration system though, Events? CQRS? Etc?

  • by amacalac on 6/17/24, 3:06 PM

    You'll likely need to consider if the data egress or ingress is realtime, or asynchronous – it could well be a mixture of all of the above.

    If the "one application" is yours, and you require strict adherence to a particular data format, you'll be in a better position. You would design a contract to require all integrations to work against – a good example here would be something like Zapier, which defines API contracts that applications must adhere to, in order to work.

    If you're looking to integrate with many platforms with differing domain models, and you want to store them in the one application, you'll need something a bit more free-flowing, and you'll require a translator on your platform. You should absolutely version something like this from the beginning, since integrations can and do change their APIs at the drop of a hat. You'll also want to be able to temporarily pause syncing data from individual integrations, since any API changes, or downtime on 3rd party platforms may very well impact your own uptime.

  • by octo-andrero on 6/17/24, 7:58 PM

    I would suggest you to split the question into multiple:

    1/ What should be communication pattern you need: synchronous calls via API, message brokers (you configure routing rules inside message broker) or stream of events.

    2/ How you should structure the code that processes your data once you've received it somehow. This is the point where it's possible to start thinking about OOO and design patterns. Regarding your concern, looks like you mean premature abstraction. You may have N different code packages that do the integration and have nothing in common. This approach works well if you suspect that your integrations are going to evolve independently from each other. If you could provide some more details on what you're going to build I can help.

  • by ossm1db on 6/17/24, 8:08 PM

    I am investigating this issue on my own as well, and data-oriented programming looks to be the most promising: https://www.manning.com/books/data-oriented-programming

  • by quintes on 6/17/24, 7:43 PM

    From an architecture perspective you need to understand the sources, the kinds of data they provide and the frequency. Then you need to decide if you sync, async or batch, how and when.

    Then you look at what standards/protocols they use and architect for that

  • by spdustin on 6/18/24, 1:08 AM

    Meltano was my go-to for ingesting data, and dbt for transforming as needed to (ultimately) materialized views for the “core” consumer.