This Week’s Micro-Obsession

July + Sunshine + Arctic Wind =>

Geek + Wifi + (Geek Fuel == Books) + Garden =

Green Spiky Mental Energy

Having spent some months working in an application centric continuous integration and test-oriented developement project I am now wondering how I ever survived without it? The mental loading you have to bear if you manage the ‘virtual’ CI in your head is a pretty heavy price and detracts from the creativity at the sharp end as a result of mental-resource-contenton. Having experienced the CI model for the first time, I’m a complete convert and have been amazed how simple it is to create a very effective CI process out of public-domain tools…

Now exploring if the same automation can actually be exploted to the same extent, with the same degree of certainty being introduced into large scale integration projects through a synergy of SOA and Contract-First design philosophy, Cross Domain Test Driven Development where I write tests against your contract, Continuous Integration meaning explicit verification of tests against contracts and implementations of contracts. This takes the concept I’ve been hearing about - that associated with ‘integration of components’ witin an application or system, and expands it out to sit across numerous application / system engineering projects to provide an overarchiing, verifiable build or ‘integraiton’ process triggered by modifications to key aspects of what defines the inter-domain contract.

Interesting challenge…

More later.

Powered by Qumana

Is ‘Mandatory Community’ Still a Community?

In the public domain people form real communities out of true, intrinsic personal interest and gain (or at least that is the motivation initially). As such the cost of entering those web2.0 communities - ‘purchasing’ the infrastructure to gain access the the web (i.e. a pc and a pipe) and conforming to the globally agreed web protocols and community behaviours, is trivial when compared to the benefit we attribute to being part of a community and gaining the knowledge, support, respect, participation, belonging, etc.

The phenomenal growth we see in specific aspects of the web community, especially those formed by knowledge based or more proactive open-source engineering communities has changed the world in a sense that a new, viral, problem solving engine has been created to harness the grey-matter of an ever increasing resource-pool.

It’s amazing to think that such a powerful entity has been formed on a purely volutary basis by people sharing little more than a common interest. Looking to this kind of community model as a means of increasing innovation and collabration within the workplace is a natural aspiration, and a valid aspiration, but I’m wondering if the essence and the inherent success of the open community model does actually translate effectively into the closed community model (I refer to that which exists between employees within a large organisation).

When we look at the motivations of individuals in the workplace I am guessing that the vast majority see their labour as a means of generating income and as such tune their labour inputs to a level which is comfortable in respect of the income generated. Within that population we then have a subset, and I would argue a small subset, who generally aspire to going the extra mile and see their labour as an opportunity to take things to the next level - using that as a means of justifying increased effort with the goal of longer term reward. Regardless of the numbers/ratios, my point here is that the population within this kind of community is less interested in the value of the community model than those who choose to opt-in to an open community such as those we see in the public domain.

If we therefore make participation in the community a ‘mandatory’ obligation, with an expectation of quantifiable value, over and above meeting the daily functional goals against which we are measured and rewarded, I fear we are likely to alienate more than inspire. This does not mean I have no faith in the community model, I believe I am someone who aspires to more than ‘breaking-even’, but more of an interest in terms of how much we can expect to gain from the imposition of community on a mixed popultion of employees, who at best may form pockets of shared interest. I guess a large part of this is the business-as-usual model by which we stay in business. Add to that the associated issues of corporate regulatory policy and intellectual property rights which further isolate the enterprise community from the public domain and again we’re further limiting the viral expansion and value generated from this kind of model.

Funny that probably the most effective shared-interest community I can see in my enterprise today is the Union, which is not entirely focusing on the same goals as that of an innovation-centric community like apache now is it?

"In human communities, intent, belief, resources, preferences, needs, risks, and a number of other conditions may be present and common, affecting the identity of the participants and their degree of cohesiveness."

Powered by Qumana

Enterprise Integration: SOA “re-use” is a wolf ?

Is our focus on achieving Service re-use actually harming the work we’re doing in the creation of the SOA? Are we so focused on the re-use model that we’re damaging the implementation of the SOA blueprint? Let me try explain why I believe this is the case.

Re-use is a measure of what? In some cases it means how much common code is duplicated and leveraged in new software developments. In other cases it means the more literal measure of concurrent exploitation of a shared resource. In our SOA governance structure, re-use is definitely analagous to the latter case - the one-size-fits-all coarse grained, generic service that enables me to service all variants of any particular requirement through a single, common service interface. Breathe………..!

That’s great for the provider who can now hide behind the ‘re-usable’ facade, pointing knowingly at the SOA governance literature every time you try to have the conversation that his service requires a PhD in data-modelling and xml cryptography to use it?

“But it’s re-usable” comes the reply as you weep, holding out your outstretched arms, cradling the tangled reams of xml-embossed printer-paper representing the only ticket into the service you need to use to avoid being branded a heretic. “Are you challenging the SOA governance policy? Let me just make that call to the department of SOA Enforcement…err I mean the Chief Architect…what’s your name again?” comes the prompt follow-up as the hand reaches for the phone…

I’m now convinced that re-use is a proverbial wolf in sheep’s clothing. We must look more closely at the cost of creating and operating these ‘jack-of-all-trades’ services, not just the fact that we can create them. If it transpires that we’re incurring more cost (both financial and operational) at both the provider-side and the consumer-side, by virtue of aspiring to ‘re-use through generalisation’ then we have truly lost the plot. Could this be part of the reason that SOA ROI is such a difficult subject to discuss and often results in a “year-n payback” kind of response?

I also think there’s an analogy to make here. Take one of our local government services which are considered part of the service fabric of our community. If the services are made too generalised and therefore spawn highly complex forms and a high level of complex dialogue in face-to-face scenarios, who is that helping? Yes we can say that we’ve consolidated ‘n’ simpler services into a single generalised service and saved on office infrastructure and so forth. But if we then increase the cost of processing requests for that generalised service both in terms of ’steps’ to get from the generalised input to the specific action (therefore requiring larger offices in which to house the longer queues) and also in terms of now having to cater for the increased and excessive fall-out volumes based on the fact that it’s just so damn complex to fill the forms in (therefore requiring even larger offices to in which to house the even longer queues)….then we really missed the point about re-use.

It makes complete sense to look to generalise and re-use in the SOA design-space, such that we can converge similar designs to a common reference model and avoid the unconstrained artistry of technicians with deadlines. In terms of service contracts and interface specifications through,  translating that design-time re-use to a wire-exposed endpoint seems like we’re stopping short of the ultimate goal of accelerating re-use by empowering consumers to bond more efficiently with that service.

Powered by Qumana

Enterprise Integration: Don’t Look Down !

Can SOA truly be successful if service consumers have to be technology consumers? The service layer is supposed to insulate us from the technical complexities and dependencies of the enabling technology, but I see more and more the technology being the centre of attention.

The promise of Web Serivces whilst standardising on the logical notion of integration,  has embroiled us in a complexity relating not to the ‘act’ of exchanging documents, but instead relating to the diversity within the Enterprise, the various technologies and tools used across a widely disparate ecosystem, and debating the finer details of which interpretations of which standards we want to use.

More significantly the large deployed base of messaging software, and the service endpoints exposed to MQ or JMS endpoints are left out of the handle-cranking associated with the synchronous style of endpoint. As such - a SOA layering ‘consistency’ across such a diverse ecosystem is a myth in my experience. We’re still struggling to find the SOA ‘blue-touch-paper’ despite all of the top-down justification and policy.

I believe that until we push the technology further down towards the network such that it becomes irrelevant to the service consumer, and raise the service interaction higher up in terms of decoupling the ‘interaction’ from the ‘technology’ we are going to struggle to not only justify the benefit of service orientation, but more significantly we’ll continue to struggle to justify the inevitable rework and technical implications of that service orientation in mandating conformance to brittle and transient technical standards.

I’m going to explore an approach to doing this - by encapsulating middleware facilities as RESTian resources, and then looking at the bindings between WSDL generated stubs and these infrastructure resources…effectively removing technologies (apart from the obvious RESTian implications) from the invocation of a web-service. Various header indicators can flex QoS expectations in the service invocation (i.e. synch or asynch, timeouts, exception sinks etc) but that has no relationship to any given protocol or infrastructure type. Furthermore, the existence of such a set of ‘resource primitives’ would enable direct interaction where WSDL-based integration does not yet exist…where I resolve, send, receive and validate though direct interaction with RESTian services from any style of client-side application.

This is motivated purely by the belief that, much like the chap in the picture, our focus is on the endpoint and not what lies beneath…

Enterprise Integration: The Target Architecture

Refactoring architectural roadmaps. Where to start after time in the wilderness? Surveying the scale of the landscape, evolving mass of vendors, old and new initiatives and legacy sprawl. Getting that cold feeling in the pit of the stomach, like the one you get when the ‘flight’ option is removed from the ‘fight or flight’ juncture…

So how do you actually make headway and add real value to the enterprise?

My revised target architecture is summarised in the following image. We need keep the technology partners trolls under the bridges, and make sure our diverse users know where the bridges are, how wide they are, how much it costs to build one, what it costs to keep a bridge safe,  and make them forget about the noises down below…

Phase 2 will entail sealing the foundations in concrete we mixed ourselves, and re-routing the rivers. Therefore no space for ‘noise’ from the space beneath…happy times.

I’m starting to see the light…

Powered by Qumana

Enterprise Integration : QoS is What not How…

Looking at reasons why we adopt certain technical approaches to enterprise systems integration I’m increasingly concerned by a ‘how’ approach, using proprietary, expensive technology to create relatively simple styles of interaction effectively eclipsing the ‘what’. This ‘how’ approach appears to be rooted in technical policy and commercial structures rooted in the past, during which times proprietary/vendor solutions had more credibility than home-grown options. However looking at the ongoing commoditisation of application integration, the dominance of the web, and all manner of open-source technical options, it strikes me that we need to review our position, and attempt to find ways of selectively breaking out of the vicious circle of ‘licence renewal’ and ‘false economy’ in favour of a more blended approach.

Systems integration is dominated by TLA’s with a heavy vendor influence, and as such it’s easy to get lost in the cloud of complexity associated with MOM, JMS, MQ, XA, SSL, PKI, REST, HTTP, SYNCH, ASYNCH, PERSISTENT, QOS, etc. It’s no surprise, therefore that systems integration shoulders the burden of the difficult stuff getting brushed under the ‘proverbial carpet’ as opposed to the aspiration of the ‘intelligent application-aware network’. As that carpet gets ‘lumpier’, we throw more TLA’s into the mix and add more and more intelligence into the network, resulting in more complexit and a vicious circle. It’s no surprise that we’ve struggled to gain the confidence of the enterprise to the extent where we are in a position to move to commoditise what has become a very complex array of sticking plasters upon sticking plasters.

Shifting the focus to ‘the web’ and my interactions across that global ‘unreliable’, diverse, evolving network.  I note initially that I have a natural understanding of what I want to happen each time I select one or my many application’s. My mail client gives me the ability to create asynchronous 1:1 or 1:n distribution flows, with the ability of conveying large payloads and attachments. My instant messenger client allows me to enage in synchronous 1:1 or 1:n interactions, my feed-reader will sink event streams at a frequency I define. My blog client let’s me cache up a range of 1:n broadcast documents which are periodically published to a hosting platform. My twitter client lets me post informal event snippets. The list goes on.

The key point here is that I care not how any of my selected applications undertake my requested interaction, and in all honesty (even though I’ve implemented countless protocols in my time) I don’t have the time to care because if they are working ok, then I understand the QoS I expect, and if I don’t get that QoS I vote with my feet. We sometimes hear terms like Jabber, XMPP, SMTP, POP3, HTTP, HTTPS, IRC, etc. in assocation with the applications we use, but I would argue that only a tiny proportion of users of Thunderbird/Outlook actually have any clue about the implication of those SMTP settings at a technical level.

So to my conclusion. Systems integrators (and adopters of such technology in the enterprise) evolved from a time and a market-place differentiated by the ‘how’ factor. Subsequent up-selling on that original platform has created a momentum of expectation and desire in the technology consumer space, by constantly mapping the ‘how’ factor to the ‘what’ factor, eclipsing the emergence of a more commoditised alternative gaining a foothold. This is a highly effective point of attack when coupled with a reminder of achieving ROI on the last n-years of similar investment right ?!

The web, by contrast has simplifed and commoditised the same kind of end product, and evolved a user community purely focused on the ‘what’, without a care for the ‘how’ factor. As such it is relatively simple to decouple a user of a web-application from any underlying technical infrastructure, so long as the QoS and the ‘what’ factor is maintainted. Would I even know if my Thunderbird/Outlook mail client began using a competely different store-n-forward protocol? I don’t believe I would.

I believe we need to bring this learning into the enterprise space, and detatch our ‘how’ users from the underlying detail and coaching them to become ‘what’ users, such that we in the integration layer can get to work commoditising the technical fabric in such a way that we do gain the necessary ’specialisation’ from key vendors, but in the main, we regain control/choice of what it takes to actually pass an ‘xml document’ from source to destination, across a trusted network we control, and guarantee it gets there in one piece.

We seem to have missed this…as it doesn’t cost me six 0’s to hook into, and run an effective real-time business over the web now does it?

Powered by Qumana

REST: Logical, Cosmetic or Fundamental

In my experience the successful application of the RESTful architectural style to any given software integration opportunity, requires an appreciation of the levels of RESTfulness gained from the relative levels of effort expended. I will attempt to explain my interpretation of these and identify where I see positive an negative implications:

Logical REST

The approach where the interactions between agents are ‘designed’ as resource-oriented, based on a finite set of well-defined methods analagous to the RFC2616/HTTP ‘verbs’, or more generally a CRUD scheme. This kind of approach achieves common-ground in the design-space, but is not followed-through with a strict adherence to the RFC2616/HTTP protocol as a framework for implementation, possibly using a proprietary xDBC stack, language-specific binary protocol or heavy SOAP stack. As such we see a logical resource-oriented perspective but with none of the true benefits truly embracing the web as a foundation.

Advantages

• Achieve uniformity in resource/interface design.
• Establish a common inter-domain design philosophy.

Disadvantages

• Not addressing the heterogeneous middleware friction.
• Stopping short of realising the full value of REST.
• Little or no departure from traditional CRUD data-access.

Cosmetic REST

This is possibly the most common category where a Logical REST design is implemented within a RFC2616/HTTP framework, exposing concrete resource addresses on well-defined http URI’s in a web-compatible manner. However, the cosmetic dimension relates to the fact that these RESTful endpoints are thin, ‘cosmetic’ wrappers over traditional RPC oriented service endpoints, effectively acting as proxies. As such Cosmetic REST is of benefit to clients who have less intellectual and technical friction in consuming a service, but do little to thin-out the general integration sprawl within the domain of the service-provider. As such the addition of public RESTful to private RESTless bridging increases the integration burden of the provider.

Advantages

• Creation of real RESTful service endpoints removing technical friction from clients when consuming service.
• Unhook clients from internal integration dependencies by providing a uniform, intuituve interface.

Disadvantages

• Increases the integration burden of the provider.
• New facade on existing brittle integration points.
• No increase in productivity gained from supporting simplified exposure.

Fundamental REST

This approach is a full application of the architectural style in a deeper sense than the cosmetic use of the HTTP protocol in exposing interfaces. Resource oriented design of a system is a significant departure from simply exposing functionality or information services from a complex machine using some arbitrary exposure/protocol dialect. Instead resource orientation inverts the ‘functional’ exposure of traditional RPC styles of integration by facilitating an information-first (not functionality first) interaction between a client and a system. In simple terms your interaction with a true RESTful system is through the manipulation of these resources, where creation, updating or deletion of resources or resource-atributes has explicit (or implicit) behaioural contracts which will conclude within a stateless atomic context. This completely decouples a client from the internal workings, dialects, processes, etc of a traditional approach to opening out the functions within a ‘RESTless’ system not based on the resource orientated architectural style.

Advantages

• Cleaner relationship between RESTful exposure and implementation of underlying system.
• Reduces the ‘thickness’ of interation logic sitting between the network addressable endpoint and the implementation.
• Integrate clients with your information and not your process and/or functional granularity.

Disadvantages

• Despite the RESTful or resource-oriented architetural style being synonymous with the web, it is still perceived as a ‘new approach’ in the context of internal enterprise business systems.
• A more significant mind-shift is required before this appreciated
• Still little hard evidence of the positive implications of resource orientation over traditional approachs  -  it just ‘feels’ like the right thing to do.

In my experience to date, for anything other than a green-field web-application which may go directly to a Fundamental REST approach, the larger integration challenges associated with service enablement of a more complex enterprise or enterprise system will ultimately start life, as we have, in a Cosmetic RESTful approach - most likely resulting from the combination unmovable silo’s of historical investment, as well as the limited impact of a Cosmetic or Facade based approach when breaking new ground.  In such cases the cost-effectiveness of REST and resource-orientation do become hamstrung, purely because to exploit REST you have to build more layers of mediation initially…until the confidence in REST increases to a point at which true, or more Fundamental REST can be driven from the edge of the enterprise, down into the silos.

The Mean Machine on tour…

The Mean Machine on tour…

Originally uploaded by Wertster

We entered our first beach-touch-rugby tournament last weekend in Bridlington. Took some time to adapt to the tag-rules variant but made it into the semi-finals before running out of legs. Very respectable and a good day had by all :-).

We adopted the team name ‘The Leeds Winoes’ as a subtle play on ‘The Leeds Rhinos’ super league team. That is now our official brand…

More action here at the Leeds Winoes photo log.

Bring Back Tight Coupling and P2P Integration !?

So we aspire to loose-coupling, re-usable interfaces model abstraction as a means of implementing our SOA. Why? Well we’re told that the alternative is bad ! That alternative is unconstrained tactical wiring between applications, with the resulting unsustainable wiring being the essence of bad practice. I do agree to a point about the unconstrained integration being a bad thing, but there’s also some marketing greyness I need to dispel.

Point-to-Point or tactical integration is the term used to describe the creation of an application integration solution between 2 components, where the aspiration, the design, and the solution is only concerned with that specific requirement at that point in time. Shock horror - who would do such a thing?! Well there’s plenty of reasons for why this kind of approach may be suitable in some scenarios - in fact this IS the most popular approach to integration right!

However the subtle difference between the archetypal P2P interface and a reusable service is in how the design is approached - bear in mind P2P interactions still exist via re-usable services too. Has the interface been based on open standards in the infrastructure layer, has the interface been abstracted at a functional and information level to support additional dimensions (i.e. products, customer types etc) over time? Whether we use web-service technology or not we can still create re-usable services in the application integration landscape.

Now at the other end of the food chain we have our new-friend the coarse-grained, heavily abstracted, reusable Business Services driven out of the mainstream SOA approach to rewiring the Enterprise. Here we have, from the outside looking in, a single exposure for a complex array of related functions (i.e. multi dimensional product ordering), based on WSDL/SOAP/XSD/XML/WS* standards. This kind of approach is the current fashion, and is purported to simplify integration. Wrong!

What we do find is that the new, extensible interface simply creates a thin but strategic veil over the previous P2P interfaces, and effectively causes 2 areas of complex integration. Firstly - behind the new exposure, the service provider has to manage the mediation of an inbound request across his underlying domain models. Secondly the consumers of this newly published service have to deal with their own client-side mediation to enable their localised dialects to be transformed into a shape which can traverse the wire and be accepted by the remote service provider - or at least by the new strategic facade.

My point here is that SOA and the inherent style of wrapping functionality introduce integration challenges in their own right! So it’s not all rosy in the SOA garden, and this is where I’m seeing opportunity for a hybrid approach….and (appologies for the heresy, I’ll burn in hell if I’m wrong) a resurgence of P2P runtime integration based around a well managed reusable service design process.

Eh!? Have I unwittingly turned to the dark-side?

What I mean is P2P is OK if the cost of change is minimal - and if we minimise the client specific aspect we can reduce this cost to a point where it’s comparable to that of the alternative of exposing the common model to the wire. In traditional approaches, cost of change is high because the entire design of the solution was hardwired to one specific purpose. Introduce a requirement to flex that solution and we have to rip and replace. However if the P2P ‘design’ is managed correctly and involves the creation of mappings between a common model and the provider domain models, then in addition to exposing that generic interface to the wire, we have a facility to enable the consumers of the service to declaratively derive their own native transformations, which can cut out a transformation step in the runtime.

If we use a toolset such as Progress DXSI, for capturing the Service Provider mappings into the Common model, and then capturing the Consumer mappings into the Common model, then we can relatively simply derive transformations between the Consumer dialect and the Provider dialect. Any changes to the provider, or the Common model would simply require a re-generation of the transformation code that would then execute on the client. This sounds sort of logical…unless my logic has become skewed somehow

So this hybrid approach simply blends the best of a fully decoupled SOA approach with the runtime efficiencies of a tightly coupled P2P approach, based on the fact that the design-framework is declarative and can reduce the cost of change so as to mitigate the risk of P2P solutions.

I’m going to explore this in more detail, but I’m confident there’s a way of getting the best from both worlds…unless the SOA police catch me first…

Architect, Developer or Mineral?

Working within the IT community of a large Telco I am fortunate enough to work with a range of specialists across a range of IT related disciplines. I use the term specialist intentionally, as one of the biggest culture shocks I encountered when joining the corporate entity from a start-up and freelancing background, was the degree of specialisation afforded IT professionals within the protective enclave of the corporate firewall. Terms like ‘I only deliver software I don’t support it’, ‘I am a solution designer, not a developer’ and ‘I don’t do customer’ were sadly common and although strategic programmes have re-aligned focus on what matters (i.e. customers) there is still a great degree of specialisation by choice rather then specialisation by demand as in the open-market.

So why is this important. Well I constantly hear and experience (mostly) light-hearted jousting between Developers and Enterprise Architects (such as I). The fundamental basis for this banter is the fact that the Developers believe the Architects are ‘wannabe developers’, seemingly detatched from the build process because they are too busy sculpting new ivory towers in which to postulate the meaning of next months’ ’strategic strategy for planning’. Developers meanwhile, (I’m led to believe) keep the business running by cutting code in their sleep, being part of some niche music scene, knowing why Ruby is far better then Perl, and have discovered cool t-shirt and bag shops where the doors are closed to anyone with MS Powerpoint installed on their laptop !

Having recently made the transition from a pure Architecture to a pure(ish) Developer role to get a closer appreciation for skills I had stopped using some years ago, I now consider myself able to understand a little more clearly, this perceived void between Developers and Architects. It is simply a matter of specialisation through necessity.

Enterprise Architects have a role in which they are rewarded for looking toward the horizon, and dove-tailing business and IT activities to make optimal use of available resources whilst mindful of all the constraints in the delivery engines. The number of bases one must cover (speaking from painful experience) if one is to be an effective Enterprise Architect, does require a similar degree of intensity as that of a developer with head-in-code, albeit detail of a different nature. It is unlikely that Enterprise Architects, however technically minded, can find time to drop beneath a system-level perspective.

Developers have a role in which they are rewarded for effective delivery of solutions involved in a current or near-term horizon, by ‘going vertical’ into the detail with a narrower ‘enteprise’ scope but becoming a domain expert within a specific field. The level of focus required for a developer to become fully integrated with the evolving solution, leaves little ‘free memory’ or ‘cpu cycles’ for engaging in the Enterprise planning activities keeping the EA guys so busy.

Moving between these roles does take a period of mental refactoring of one’s head-space from a horizontal to a vertically arrangement, so it’s not the sort of thing one can do on demand (unless you are one of the lucky ones with unbounded mental scalability which I aint!). So I am emphasising the specialisation thing again, where we specialise on either 1) What we get paid to, or 2) What we enjoy. So to round off this ramble…

An Enterprise without Architects is a ship without a rudder…

An Enterprise without Developers is a ship without an engine…

 

So let’s start appreciating each other…