Once we have established whether the ESB makes sense in our integration problem space (Part 1), and once we have established whether we want to approach the delivery as a fully distributed, fully centralised or hybrid ESB deployment (Part 2), then we have the relatively trivial task of making our minds up if we need to buy something, embrace opensource or simply ‘refactor’ what we already have.
This article presents my personal experiences as an Enterprise Architect looking at integration architecture within a very large (LARGE) scale SOA transformation programme. As such my views are influenced by that specific context and I emphasise that certain conclusions I draw here are not entirely relevant for smaller scale or application-centric integration work.
So the landscape I face consisted of thousands of legacy applications, hundreds or emerging strategic applications, 14 logical groupings of related applications, 160 strategic services offered by the 14 platforms, average of 10 operations per service, a mandate to introduce common-information-model, a mandate to exploit WS*/SOAP conventions. Add to this the organisational model in which suddenly, organisational divisions were aligned to platforms and as a result engineering work commenced in parallel across the enterprise to provide and consume strategic services based on predefined contract. So the task of identifying an integration architecture which would support such a heavily distributed implementation model was quite a challenge, and clearly laid waste to our traditional domain-hub centric appoach which effectively centralised the integration problems and pushed them through a finite pipeline based on the limited, specialised resource of the hub teams.
Issues ranging from ownership, funding, scheduling, delivery capacity, scalability, runtime capacity, and so on effectively put nails in the coffin of our existing mechanism, and it soon became apparent that our only chance of facilitating such a wide ranging integration storm was full distribution of the infrastructure and therefore the ownership of the problem-space into the active delivery programmes. The bus concept was formed…based on a pragmatic reaction to a whole range of environmental factors – NOT – vendor promises.
Next was the question of ‘do we need to buy new stuff’ or do we just ‘adopt a mindset’. Clearly our massive intellectual and fiscal investment in all manner of middleware to that point provide a baseline componentry which could be refactored. There was absolutely no case to justify further investment in emerging pure-play options, especially when one considered that we would not use the majority of the ‘potential’ elements at that stage in our SOA transformation. As such we needed a consistent messaging backbone which could be introduced as an infrastructure backplane to connect the emerging endpoints. We had MOM software all over the place, and there were no real technical problems with it, we just used it badly and inconsistently. Refactoring and introducing more rigour into the exploitation of this component would instantly enable us to leverage all our MOM experience to that point whilst forming our ESB connectivity backplane.
Next came the service container component, in other words the unit of integration logic we would look to deploy at the endpoints, effectively connecting the MOM backplane to the logical platform offering service. We examined our current portfolio, found we had a suitable container for that layer of our ESB, again something we had a lot of confidence in using albeit supporting different patterns to those which we were about to depend on, but overall some confidence is better than no confidence. So we reused an existing component, I would refer to it as our lightest integraton broker with efficient integration with our MOM layer.
At that point we stopped worrying about orchestration, registry, monitoring, management, complex-event-processing, and all the other bells-n-whistles we could add later. The pitch of ESB vendors actually reinforced this view in that the ESB was inherently so modular that you can add and remove both infrastructure service or business service providers at any point :-).
So we refactored existing infrastructure. We combined a simplified infrastructure blueprint with a core set of service design patterns, along with the newly formed protocol and document standards driven out of the strategic architecture. That was enough to get the integration backbone embedded into the delivery programmes in such a way as to simplify the model of how they would ‘plug’ into the backplane.
Did we suffer for not buying a pure-play?
Nope. We had the basic componentry on the books already, it was the clarity of vision which mattered in the early stages of our decentralised approach. We had that.
Would we have achieved more with adoption of a pure-play on day 1?
Nope. We would still have only used the same base-layers we are using today. We’re still breaking into the higher layers of the ESB food-chain anyhow. A vendor stack would have been gathering dust for 18 months.
There were implementation problems…more later…