by mprast on 2/21/24, 6:49 PM with 38 comments
by jerf on 2/21/24, 8:45 PM
That's a good question. But you need to start from the fact that people have actually tried really, really hard, rather than nobody having tried at all, and it in fact hasn't worked despite rather substantial investment. When you start from there, you may obtain some useful understanding. Perhaps even useful understanding that will lead to solving some of the problem. I've got no problem with people jousting with windmills, I just think that anyone who wants to take such a task on really ought to research previous jousting attempts first if they want to succeed (as opposed to just doing it for fun).
But if you start from the blatently false presumption that you're the first person to ever dream of data that can just flow between systems, well, yeah looking out into the world you're going to be confused at what you see.
We have in fact tried the "correct" way, and it turned out not to be correct. Whereas what we're doing now, however annoying it definitely is, does in fact at least partially work.
by Animats on 2/21/24, 8:25 PM
The original idea for "electronic data interchange"[1] was to have a standard format for inter-company invoices, purchase orders, bills of lading, customs clearance, etc. So you only need N implementations, not N^2. This got going in the 1970s. So the formats are ancient. This is a flight ticket availability request in the international EDIFACT format.
UNA:+.? '
UNB+IATB:1+6XPPC:ZZ+LHPPC:ZZ+940101:0950+1'
UNH+1+PAORES:93:1:IA'
MSG+1:45'
IFT+3+XYZCOMPANY AVAILABILITY'
ERC+A7V:1:AMD'
IFT+3+NO MORE FLIGHTS'
ODI'
TVL+240493:1000::1220+FRA+JFK+DL+400+C'
PDI++C:3+Y::3+F::1'
APD+74C:0:::6++++++6X'
TVL+240493:1740::2030+JFK+MIA+DL+081+C'
PDI++C:4'
APD+EM2:0:1630::6+++++++DA'
UNT+13+1'
UNZ+1+1'
Anyone involved in modern systems for this sort of thing?
[1] https://en.wikipedia.org/wiki/Electronic_data_interchange
by bradleybuda on 2/21/24, 8:44 PM
If you're old enough to remember what happens when you make a repeated analog copy of something (fax machines, copy machines, VHS cassettes, etc) you know what happens with this kind of data flow topology. We advise our customers to do the /exact/ opposite of this - follow as few links as possible for any piece of data and move towards hub-and-spoke, ideally with the hub as a highly configurable, low loss node - in my world, this is a data warehouse or data lake, but it might also mean an iPaaS or similar.
by zer00eyz on 2/21/24, 8:08 PM
I am dumbfounded.
With the current state of the market why in the name of all that is holy would you depend on one vendor to get to another?
I dont need to be at the whim of one vendor to get to the next one. What happens when the vendor in-between goes out of business or wants to present their own service or...
I have no words for how bad this idea is.
by daltonlp on 2/22/24, 6:12 AM
1) It offers both superior control and convenience.
2) It is simple to comprehend and cheaper to operate.
3) Systems are connected to each other with drawn lines.
4) The architecture "Decouples" systems to make them simpler, not more complex.
5) Data structures map cleanly to each other, with no loss.
6) Data structures are 1-dimensional collections of "fields".
7) The architecture is not used in production. No organizations depend on it economically.
.
On the other hand, a practical architecture can be distinguished by the following characteristics:
1) It is used in production. People rely on it daily.
2) It is expensive, and needs constant human attention to balance control and convenience.
3) It has no optimal state - only varying states of failure.
4) Its systems have myriad ways to connect. Some are incompatible with others.
5) Data structures are messy and arbitrary. Some are provably incorrect.
6) Unicode in particular is a problem.
7) It is not comprehensible by any single person.
.
With apologies to Admiral Rickover :)
by weego on 2/21/24, 7:59 PM
I highly doubt theres a solution to anything in there.
by axelthegerman on 2/22/24, 3:24 AM
Data migration is more than just "a function" that we can "just apply". Maybe this would work if all integrations were written in compatible languages with some sort of common SDK/API.
There is a reason open standards exist, to achieve interoperability. They have drawbacks like slower innovation cycles due to being standardized and now hard to change - hence all our $bigcorp don't give a flying fart and instead keep building up their walls which often don't even provide the ability to implement point to point integrations
by ivan_gammel on 2/21/24, 8:35 PM
by lijok on 2/21/24, 9:46 PM
by BrandoElFollito on 2/21/24, 9:37 PM
I have the feeling that this is the article of someone who has a vision under the shower and shares it with the wrong-doers to bring them wisdom.
Now, I have not read this specific article so it may be revolutionary for once.
by kkfx on 2/22/24, 9:22 AM
The original model was an OS-framework where an application was just a bit of code added to the common live codebase.
The web 2.0 try to reimplement such model without admitting the commercial horror we are in. Hence it can only fails despite all the progresses. Web n.x can't solve the underlying issue. The solution we have is simply accept that software as a product and as a service can't exists in a proper IT sound world. Data can be sold, iron can be sold, not more.
Data can be XML, EDI, json, yaml, ... as long as we have the base we can have easy enough integration because agreeing with a real world standard is not something doable up front while having the flexible base, the ease user-level/high-level development of classic systems do solve much integration issue. Try Emacs and you'll understand.
by apimade on 2/21/24, 8:35 PM
I can already see the rejections:
Debugging and Support: Rather than a centralised support system with levels, to support this you'd need a technical user familiar with the integration, or have the field mappings so well-documented and available a level 2 support rep could figure it out. The latter will never happen.
Brittle: This introduces a lot of risk to the organisation, as understanding conditions of failure and what the repercussions are in the event one of those services stops working becomes.. Complicated. There's a reason authN/authZ providers have opted for hub & spoke model, and that's because you need a highly specialised, often under-resourced team to look after it. Sound familiar? If you've worked in integration, it will.
Development and Resourcing: Like any proprietary product, this is another thing to hire for, and support. The uplift required for someone to understand this concept, and run with it for the use-case you've offered which is adopting it across the entire organisation requires.. Executive-level buy-in. To put it plainly, adopting this would introduce too much risk to the business in its current form.
I'd recommend taking a smaller bite with your approach. You're one person. Take a niche, and build something people want for that - people don't want technology for the sake of technology anymore. That money's dried up, and if you're trying to form a business off this concept - you'll need deep pockets to develop it.
Here's one that could be relatively successful: Take SIEM tooling, for example. For small and medium-sized businesses, operating a SIEM can be a massive undertaking. Usually the first thing an organisation wants to understand when they look to do this is: what audit logs do we have, and how can we look at audit logs for a user or employee across a range of services. Every company operating in a regulated environment, or company wanting to ensure they understand what an employee did during their last 2 weeks of offboarding would be interested in this.
If you built a product and demo purely to extract user/employee audit logs from services and expose them in this interface - I'd be lining up to try it.
by hadlock on 2/21/24, 8:43 PM
I've had to integrate new functionality with disasters like this and it is always a huge pain to support. Especially when the one guy maintaining the zendesk application leaves the company and we're like, "what is the purpose of this zendesk app in the django monolith doing?" and six months after we update it to support the new thing we added, we find out that django app's functionality was superceeded by xyz a year ago.
If I found out the mission critical biz ops side of the company was run like this I would not accept the offer.
by Almondsetat on 2/21/24, 8:22 PM