Big Data – Who will find the patterns of value?

May 8, 2012

Big Data has great promise – no doubt that we are on the brink of discovering innovative ways to make a lot of money due to the ability to consume and analyze a greater volume, velocity, and variety of data than ever before.

The question, however, is who will make these innovative discoveries?

Can the innovation be made by a kid in a garage, a college kid in his dorm room, or a talented IT professional dreaming of a start-up in his free time?

On the one hand, the answer is NO – the college kid in the dorm will NOT be changing the world via big data the way Mark Zuckerberg did with social networking. Why? The reason quite simply is because “Big Data” requires DATA and a lot of it and in potentially many different forms.

Data is not free

In fact, there are billion dollar businesses that are essentially data companies – they sell raw data (and analytics on it). I’ve even worked at one in my time. OK – Some of that raw data is free but much of the data is not. The sheer volume of data is prohibitively expensive to acquire for a college kid to start to experiment with. Additionally, some big data use cases revolve around social data that sites like Facebook can use to make money (data is why Facebook has such a high valuation). Did you think Facebook gives away their data for free? If the college kid can’t even get his hands on the data then the game is over before it even starts.

On the bright side, however, cloud computing infrastructure as a service and pay-as-you-go pricing provides affordable infrastructure for the college kid. Powerful infrastructure is also a pre-requisite for Big Data – but it’s not a barrier for the college kid.

It goes back – again – to access to data.

The majority of the big data discoveries will come from big established companies with the resources to acquire it.

Agree or disagree? Actually, proving me wrong would make me happy.


Application Portfolio Management: Maximizing Value

March 11, 2012

Application Portfolio Management (APM) is about maximizing the value of the applications running your business.  The goal is to achieve beneficial changes that improve the balance of value to supported cost and risk.

Without an APM practice, the unmanaged portfolio of applications grow stale. This may be in the form of higher operating costs to “keep the lights on” for the application, weak service level and support, or obsolete technology – to name a few. The key here then is to methodically and pragmatically assess all or a subset of the applications top-down in the business in order to devise strategies and define projects and request funds.

We begin the process by assessing key factors of each application in order to categorize each application. Ideally the scope is the entire application inventory but it may be segmented based on current strategic business initiatives with a process, risk, cost, or value perspective. There are various methods to categorize the applications. A popular format is based on the 2X2, 4 quadrant matrix. An example is the “TIME” (Tolerate, Invest, Migrate, Eliminate) method as defined by Gartner.

T – Tolerate applications that provide good enough business value but may have an old architecture or not be well integrated. Without APM, this is probably the largest category.

I – Invest in strategic or newer applications in order to increase value and use.

M – Migrate technologies that have high business value but have issues the with people, data, or technologies that support it. The technology may be unsupported or people may be on the verge of retiring with a declining pool of replacement skills.

E – Eliminate applications that no longer provide sufficient business value.

Once each application has been categorized then define a strategy and high level set of actions, and create a business case and project charter. Also, add these newly defined strategies to the overall future-state enterprise architecture plans and roadmaps.

Over time, the application portfolio balance should shift to a greater percentage of “invest” applications with a lower percentage of tolerate and eliminate applications. This will result in a greater proportion of IT spending to business value and growth based projects  – and that’s the goal.

Top 3 Drivers for an IAM Business Case and 8 Presentation Tips

March 4, 2012

In this post, we will discuss the top 3 Drivers for an Identity and Access Management (IAM) Business Case and 8 Presentation Tips.

Who: As always, consider your audience – who will be most interested and in what driver. At a minimum, include the following teams and see the benefits through their eyes:

1) IT Operations Management 2) Security and Legal teams 3) Business (revenue focused) Managers

What (the drivers/benefits):

1. Efficiency – The ability to do more, faster and with less effort. Examples include automating access removal when someone leaves a company, reduction of helpdesk calls from automation of password resets, SLA improvement, and quicker consolidation of infrastructure.

Primary audience: IT Operations Management

2. Effectiveness – Doing the right things and doing them well. Examples include more accurate reports, savings from reduced regulatory fines from inaccurate reports, better general consistency and automation of reports, better customer and auditor perception.

Primary Audience: Security and Legal Teams

3. Agility – Change faster with less effort. Examples include the reduction of effort to form business partnerships (and thus encouraging more partnerships), reduce the time to integrate a newly acquired company, and improved customer service.

Primary Audience: Business (revenue focused) Managers

Socializing and Presentation Tips

1. Emphasize non-quantifiable benefits over ROI calculations. The reason is because ROI calculations are based on assumptions that can often be easily challenged, derailing the entire business case if successful. Only emphasize ROI if you are comfortable sitting in front of the CFO for 20 minutes going through the detailed assumptions and calculations. It’s a safer bet to stick with non-quantifiable benefits. If you must include an ROI, be sure to include others in the assumptions and calculations.

2. “Road test” the business case presentation in one on one or smaller meetings in order to get feedback and improve your message.

3. In the presentation, spend more of your time on the expected benefits as opposed to the why, how and technical jargon, which often only detracts focus from the main drivers.

4. The overall format should include at a minimum one slide for the following: problem statement, who was involved in the business case analysis and objectives, proposed solution, expected benefits, high level plan/options and costs, and an Appendix (assumptions and calculations).

5. Present in a professional, conversational, and competent manner

6. Know the material well (more than what is just on the slides)

7. Formally present for 20 minutes and then answer questions and have a conversation for another 20 minutes (often the most important piece)

8. Speak with conviction and above all else, honesty.

Why SOA is the Foundation for Innovation – Part II

February 13, 2011

In this previous post (Why SOA is the Foundation for Innovation – Part I), an explanation was given for why the principle of Loose Coupling in a SOA is fundamental to the promotion and fostering of innovation in a system. In this post, we will discuss fundamental patterns that implement the key principle of Loose Coupling.

Loose Coupling Defined

First off, Coupling can be defined as the relationship between things. If there is a very strong (tight) relationship between two things, then when one thing changes the other thing must change or the system will break. In a complex, tightly coupled system with 100’s or 1000’s of “things”, the effort or cost to change is very high because a change to one thing requires changes to many other things, directly or indirectly, coupled to it – otherwise the entire system will break. Clearly, this paradigm is strongly opposed to ease of change and innovation!

On the other hand, Loose Coupling advocates minimizing the dependencies between things. Additionally, the dependencies that DO exist between things are put under a microscope and micro-managed to prevent the proliferation of future dependencies. Therefore, agility (or the ability to quickly change) is increased due to the reduced impact that a change to one component has on all the other components in a system. This also allows individual things in a system to evolve independently which fosters the spirit of innovation.

Loose Coupling Patterns

Now that we understand that Loose Coupling, a key principle of SOA, is fundamental to fostering innovation, let’s turn to a couple of fundamental SOA design patterns that, if followed, make Loose Coupling “real”. In a SOA, the goal is to promote loose coupling in 2 main areas:

1) Between Services – The provider service is able to perform its functions without any help from other services.

2) Within Services – The interface is decoupled from technology and implementation details and only exposes the functional spirit of the domain.

The following pattern summaries help to realize loose coupling in a SOA:

Transformation – For example, a service which wraps a legacy system, may utilize an intermediary component that performs a data format transformation of CSV (common in legacy environments) to XML, before the legacy processing begins. This transformation component decouples the interface from the implementation. Transformations tend to reduce performance yet they are unavoidable in most environments.

Intermediate Routing – dynamically determines a message path based on factors such as the message contents or service utilization (load balancing). This may be achieved through intermediary agents that read messages and dynamically determines their path, calling the appropriate service. This decouples the connection between services.

Asynchronous Queuing – An intermediary queue receives a service request, message ABC, and forwards them to the provider, when it’s ready. The provider may not be ready because it is already busy processing a previous request. The queue will determine when the provider is ready. Additionally, this enables the consumer to go on processing other requests until the provider has finished processing message ABC (in a request/response message pattern) – all enabled by a queue, decoupling the consumer and provider.

Event-Driven – enables consumers to be automatically notified of provider events. An intermediary event agent manages the consumer subscriptions and provider event notifications, decoupling the communication between services.

These above patterns, commonly found in an Enterprise Service Bus, are some of the more important ones that can be used to realize a loosely coupled SOA that serves as the foundation for innovation!

Clarifying the Architect Role: Logical vs. Physical

January 30, 2011

The field of [adjective] architecture is still quite immature when compared to more established fields such as Accounting or Dentistry. This makes it difficult to observe and define consistent architecture roles across organizations and therefore is an issue that immensely affects an architect’s career. There needs to be broader acceptance of common terms.

In order to better understand an architect’s role, one should first fully understand the difference between logical and physical. For example, timesheets (or logging the number of hours worked by an employee) existed before computers. Over the years a timesheet may have been physically implemented differently – on pen and paper, a punch card, or a computer application. From a logical architectural view, the timesheet has NOT changed. From a physical architectural view, the timesheet has significantly changed.  When architecting large solutions, the business is generally more concerned with the logical view.

One easy to understand metaphor: Logical = Music Notes and Physical = Orchestra.

Architectural role definition can be more easily understood in terms of logical and physical.

The Solution architect tends to work from a logical view and the Technical architect tends to work from a physical view – all for a particular program or line of business. Both Solution and Technical architects work across all or a subset of architectural domains (business, application, data, and infrastructure). In large organizations or programs (e.g. $100M+), the solution and technical architect roles may be performed by separate individuals.

Furthermore, both the logical and physical view responsibilities may be combined to individuals performing Business, Application, Data, and Infrastructure architect roles. The Business, Application, Data, and Infrastructure (or Technical as defined in TOGAF) architect works at both the logical and physical views for their respective architectural domains. For example, an application architect is responsible for both the logical timesheet and the physical timesheet implementation such as vendor ABC timesheet system.  In this case, the Solution Architect is more in the role of coordination, as opposed to architecture definition, across the architectural domains.

One common pattern is the Solution Architect defines the logical application architecture while the Data Architect defines the logical data architecture for a given solution. In other words, the Solution architect may “outsource” pieces of the logical solution to be fulfilled by other, more specialized architects.  The actual working relationships amongst architects depend on a myriad of factors such as business industry, business size and program size.

From a modeling perspective, the Enterprise Architect tends to work at the logical view for the purpose of shaping and choosing business strategies and investments. Additionally, the Enterprise Architect performs governance (manages business change) and manages enterprise debt to remove or shorten the gap between business strategy and execution. An Enterprise Technical Architect tends to work at the physical view (e.g. choosing an Enterprise wide technology such as an Enterprise Service Bus technology that all future solutions will use).

As stated previously, the same individual may wear different architectural hats depending on the size of the business or program. As size increases so does number of specialized architectural roles.

Why SOA is the Foundation for Innovation – Part I

January 17, 2011

How does service-orientation, an architectural style, help a business to innovate and grow? 

To many, SOA is just a technology concept – something of minor importance to a product manager or “the business” to actually grow the business. This could not be further from the truth! Let me explain…

Let us begin by exploring the “Adaptive Cycle” model pioneered by Lance Gunderson and C.S. Holling. In this model, the various lifecycle phases (reorganization, exploitation, release, and conservation) of a system are outlined. Note, that in the model, a system can be anything such as a natural ecological system (e.g. a forest), a human system, or an automated information system such as a CRM.

There are three major factors in the model (crudely summarized below):

  1. Capital – the value or profit of a system
  2. Resilience – the ability to renew into something of greater value (i.e. innovate) – not just a return to normal.
  3. Connectedness – the interdependencies and complexity of the elements in a system

A key point: As the green part of the model shows, innovation builds upon innovation in the exploitation phase and therefore the capital (or value/profit) increases. The connectedness, however, also increases and as a result the resilience (or ability to innovate) decreases. In other words, a critical mass of yesterday’s and today’s innovations will slow down tomorrow’s innovation. Eventually, a system becomes so “connected” that all resiliency is lost, bringing innovation to a halt, and therefore the system must be released (or destroyed) and re-reorganized.

We return to the original question – How does SOA support innovation? In the context of the Adaptive Cycle model, an effective SOA enables HIGH capital (value) and resilience (innovation) and LOW connectedness. The slope of the green line becomes positivley steeper and the length longer. In other words, there are LONGER cycles of growth and more frequent innovations due to the low “connectedness” of the elements in the system.

But what does a low level of “connectedness” really mean? It means things are loosely connected – or as a Technical Architect would say – loosely coupled” – and this is a key foundational principle of SOA.

Stay tuned for part II – as we define specific characteristics of a “loosely coupled” system.