Framework dimensions

Our evaluation criteria have been segmented into four central categories:

1. 1. Solution design: This area focuses on dimensions that shape the broad design patterns that make up our solution.

2. 2. Solution development: This topic addresses what it will take to construct the solution.

3. 3. Solution operations: Here we highlight factors that influence how the solution will be maintained after it has been built.

1. 4. Organizational considerations: These are facets of the solution that take enterprise standards and organization direction into account.

Each category contains a set of criteria along with a description of what those criteria help identify.

Solution design aspects

In this section, we look at characteristics relevant to the overarching design of a solution. Once again, this list isn't exhaustive but it should provide you with a framework for thinking and steer you towards a particular implementation technology.

These data integration considerations touch upon a variety of needs in data sharing scenarios including how you receive data, how much of it you process, and what quality assurances are necessary.

Data integration considerations
Software criteria	Description
Supports high volumes of data	Can a heavy throughput of data reliably flow through the software? While this characteristic is often dependent on other criteria mentioned below (for example: load balancing, latency), it should be a known aspect of any software solution. "High volume" may also be a subjective description. In some organizations, a high volume of data is thousands of records per day, whereas other organizations expect thousands of records per minute.
Handles large individual data sets	The raw size of a data set can greatly influence which software solution you select. Some products are tuned to process small data blocks: while others hungrily tackle megabytes, gigabytes, and terabytes of data all at once. If a particular software package is a perfect match for a solution except for its inability to process thousands of records at once, then maybe we can revisit the data processing requirements and bite off smaller chunks of data.
Offers guaranteed, at-least-once delivery	Reliable delivery is frequently a requirement of messaging solutions, but sometimes, this is a nice-to-have instead of must-have. If a particular piece of data does not reach a destination, can it be sent again from the source? For instance, few request/response query operations require guaranteed delivery since transmission failures can be instantly retried. Asynchronous operations are more directly associated with guaranteed delivery. Also, guaranteeing once-only delivery is something that has to be explicitly designed for. While some tools do support this, this is also where we can architect an idempotent interface where we can send data over and over again, without negatively impacting the data context.
Able to access a wide variety of data repositories	A thorough analysis of the data requirements will surface the range of data sources that our solution needs to leverage. Our chosen software platform should be able to easily interact with the full spectrum of data sources that make up our solution architecture.
Able to access a wide variety of target system APIs	Much like the preceding criteria, we need to uncover which systems we need to interact with and determine whether a given software product is capable of consuming the interface offered by the dependent system. This could be a web service interface, native interface, or a protocol interface such as MSMQ.
Works with batches of data	We looked at large data sets and this criterion is an extension of that. Regardless of the batch size, we need to know if a particular software platform can easily unpack a collection of data records and process them individually. Sometimes, records are batched out of convenience to reduce network traffic and sometimes they are batched because the records are all part of a related transaction. Ideally, our software platform can treat a batch of data differently depending on this distinction.
Accepts real-time data input	The timeliness of data processing is a foundational aspect of most solutions. If we are building an application that demands a real-time (that is, not batch) interface, then we have clear choices as to which type of product to leverage.
Offers data quality features (de-duplication, format standardization)	When sharing data between systems, we may need to apply a series of data quality rules that cleanse and improve the integrity of the data. This could involve removing incomplete data, eliminating duplicate records, enriching the data with information from external sources, and standardizing data field formats such as phone number.

Security is often one of those things that we cannot compromise during application design. We may have to comply not only with organizational standards, but those of regulatory bodies. Here are some criteria that touch on user profiles, data in transit, and data at rest.

Security considerations
Software criteria	Description
Enables enterprise-wide and cross-organization users to access the application	It is important to know if the product can support enterprise directories for authentication and authorization so that we can establish reusable security groups and wide access.
Includes Single Sign-On capabilities	If your solution demands access to systems and repositories that span security domains, then having an SSO product available will be a lifesaver. A strong SSO product will enable you to securely map credentials from one domain to another and seamlessly access cross-domain resources.
Offers range of authentication mechanisms to invoke operations	This applies to items at an API level or even access to the application itself. If we need to be able to authenticate users against a default Windows domain, or also need alternate ways to prove identity (for example: certificates, HTTP basic authentication) then we have to choose a product with this capability baked in.
Provides authorization controls to invoke operations	Ideally, there is at least a coarse set of knobs that we can fiddle with in a software package, which allow us to restrict capabilities by the role of the user. Some software packages may allow granular access only through customization.
Provides authentication and authorization options for securing administrative aspects of application	Depending on the scope of your solution, you may need to provision administrative functions to distinct sets of users. For instance, you might have one set of users who can administer the entire application, while others can only modify specific settings. If your solution demands this, you will need to consider this capability in the available software packages.
Has compartmentalized components with independent security boundaries	One way to reduce the attack surface of a solution is to partition not only the application tiers, but also the modules within a given tier. A component that requires elevated permissions could be isolated on one server while other components running with least privilege can execute on additional servers.
Enables secure storage of configuration or reference values	When we extract volatile data values from code and store them in external configuration repositories, we make our code easier to maintain. However, these pieces of data may contain sensitive information such as passwords and connection strings. Whether embedded in the code or stored in a configuration repository, this data should be encrypted or isolated from prying eyes.

The project's need for exception handling can often be evaluated late in a delivery cycle, which is unfortunate. This is one of those areas to ask our clients about early on and get an understanding of the business need when system, data, or logical errors arise. Based on the expected types and volume of errors, we may lean one way or another on which product to leverage.

Error handling considerations
Software criteria	Description
Failures within the system are captured in predictable way	Even the best of us write code that fails. What we hope to have is a software platform that enables a graceful handling of exceptions occurring in native components, custom components or infrastructure. If the product relies on a database to operate, what happens if the database temporarily goes offline? Or, what if you deploy custom code to the application and an uncaught exception flows up to the application? Knowing how a product handles failure goes a long way in understanding how to build for exceptions and prepare future solution administrators.
There are limited single points of failure	We all hope to build solutions that do not have any one component that can bring the entire system down. When we look at underlying platform products, we need to understand where things can go wrong and which components can or cannot survive a failure.
Failures in dependent systems are handled consistently	When we built systems that rely on other systems, we must prepare for the event when those dependent systems become unavailable. How does a particular software product resolve downstream failures?
Includes facilities to monitor the system and configure alerts	If a product does not offer a management dashboard itself, hopefully it should minimally provide an instrumentation layer that enterprise monitoring tools can tap into so as to actively track the health of the application.

There are always a few uncategorized design aspects that touch upon how a solution can be designed to be maintainable and best leverage existing enterprise investments. Here we have a few items that address loose coupling of components, how operations can execute, and establishing transactions across system boundaries.

General design considerations
Software criteria	Description
Includes a modular set of components with a clear separation of concerns	Software products are at their best when they have clearly defined modules that work both independently and seamlessly with each other. When looking at your solution requirements, you may see aspects that are a perfect match for components of one product while other aspects are ideal for another. If those products are built well, then leveraging the best of each should be possible. Having modular components also means that changes can be made to one component without being forced to deploy or test all of the other ones.
Has functional flexibility and can be built for change	Sometimes a solution satisfies a very fixed need with static business rules and firm interfaces. However, that scenario appears to be an exception rather than a rule. If our solution requirements outline a very dynamic business space where interfaces may undergo change or business logic is susceptible to update, then we need a software product that can accommodate such demands.
Leverages asynchronous model for processing	Asynchronous processing allows us to execute operations without forcing the initiator to wait for a response. This is valuable for long-running processes, activities that can be handled at a later time, or broadcasting data to an unknown number of interested parties.
Is capable of enlisting both local and distributed transactions	If our solution requirements call for us to synchronize the update to multiple repositories then we have to make sure our software is capable of participating in transactions that potentially span application boundaries.

Solution delivery aspects

The big picture solution aspects cannot be the sole factor in choosing a particular implementation technology. We must also seriously consider how well the technology aids in the rapid and successful implementation of our architecture blueprint. A product that looks perfect from a design perspective may introduce an unnecessary burden on the implementation team.

First off, we should look at our development resources and consider if the technology at hand is something that existing .NET or SQL Server developers can quickly adopt, and what sort of physical environments are needed to perform development. While BizTalk Server is a mature product, it can still be difficult to find top-notch talent in the open market. For new products like Windows Server AppFabric and Windows Azure, there is an obvious gap in the marketplace until these offerings become more commonplace and skills can be developed. Consider whether you need, product expertise or have the internal skill set available to grow the expertise in house.

Resource considerations
Software criteria	Description
Skilled developers can be acquired for this technology	This criterion relates to both in-house developers and contract developers. Do we have the resources within the organization and are they even available to work on this solution? If the answer to either question is no, then how easily can we get external expertise?
If a new technology for the organization, the skill can be picked up by existing developers	Many of the products in Microsoft's application platform have similar development paradigms. Someone with expertise in the .NET Framework could quickly understand the development process for products like Windows Communication Foundation or StreamInsight.
Solution components can be run on a standard developer workstation	Most software can be installed on a typical developer computer but we should know early on if we require centralized server software, virtual machines, or 64-bit hardware.

Once we evaluate the compatibility of products with our resource demands, we can look at how well a technology helps us actually build the solution we want. This includes topics such as the richness of the development toolset, maturity of the community ecosystem, and the existence of solid test and automation capabilities.

Construction considerations
Software criteria	Description
Robust set of tools / IDEs available to construct the solution	New technologies typically have development and administrative tooling that is fairly basic. It seems that the priority of the software vendor is on the underlying component maturity and tooling is not a primary concern on the initial release. That said, the proposed solution may not require significant coding and thus advanced tooling is ideal, but not required.
Rich ecosystem of plug-ins, community code, tutorials and blogs to help developers	When building solutions on a given product we always hope to follow best practices and leverage lessons learned by others. This is where established, mature technologies have advantages over newer, less investigated ones.
Written in an expressive language that accomplishes tasks in limited lines of code	Ideally, developers do not have to spend a majority of time writing excessive lines of code to complete simple tasks.
Integrates with a variety of source control systems	A software solution can be comprised of code, configuration files, scripts, images, and a host of other artifacts. We should understand how to collect all of the solution artifacts and centrally manage them in a durable source control system.
Allows developers to build and execute thorough unit tests	The cost of testing and bug fixes goes up as a project progresses towards completion. A software product should enable straightforward unit testing of each component.
Can be set up to run in an automated build environment	A solution may be made up of a number of software packages and components, so automating the regular solution built during construction can free up resources to focus on more strategic tasks.

Solution operation aspects

Even after we have satisfied our design and implementation needs, we absolutely must consider the operational aspects of the proposed solution. Although the project delivery team inevitably moves on to other work after a successful deployment, the actual solution may remain in a production state for years on end. If we have a grand architecture that was constructed cleanly, but is an absolute nightmare to maintain, then we have not delivered a successful project. In fact, many of these operational concerns actually directly affect our original solution design. These factors, which are often gathered through the non-functional requirements, have a noticeable effect on the architecture of the system.

Performance considerations address various topics ranging from application business process performance to data volume and latency.

Performance considerations
Software criteria	Description
Key Performance Indicators (KPIs) can be captured and monitored	KPIs could relate to the business capabilities built into the application or KPIs could refer to the performance of the application itself. If the business client wants to monitor the efficiencies of their processes, then we will want to choose a product that lets us easily capture and modify key business metrics.
Can produce sub-second latency for both simple and complex request/reply operations	Latency requirements will factor into the overall design of the solution, but this also relates to the operations of the solution. Can performance be tuned in the production infrastructure?
Predictable behavior during both standard and non-standard volumes of data	Many integration solutions have to deal with spikes in data processing load at both regular and unexpected intervals. If this is possible in your environment, then you want to make sure that the software can gracefully handle floods of data without crashing.

The availability needs of the client have direct impacts on which product we should choose. How mission-critical is the application? Can we afford for the system to be down for a significant amount of time? What is the consequence if we lose some data when recovering the application? Honest answers to these questions, which typically mean fighting the urge to over-inflate the importance of a given application, will help us direct appropriate attention to availability attributes.

Availability considerations
Software criteria	Description
Natively includes load balancing capabilities	You may not need the software to contain its own load balancing mechanism if you have existing infrastructure to distribute work among machines. However, if you are dealing with a high volume environment with many long running processes, you may benefit from a technology that efficiently leverages the available resources across software nodes.
Can systematically fail over to other active servers	This also is a factor in solution design. While it may be quite useful to leverage a software platform that automatically switches execution to additional nodes when a given node fails, we may also want to define a stateless design. If we limit the state that each node must maintain, then we limit points of failure and can embrace automatic node switching.
Includes data backup routines	We may have to back up application data persisted in the software or back up the artifacts and metadata that comprise the software solution.
Support zero message loss in the product or through storage mirroring	If a software product stores application data (even while in transit), then there may be a business requirement to avoid any data loss in the event of system failure. Accomplishing this can be challenging, so we need to determine the real need and see if the software platform can accommodate this.

The day-in-the-life maintenance of an application is not the most exciting thing to mull over during project planning, but paying attention to this aspect is the greatest gift you can give to a system administrator. What are some of the general things you can do to make the maintenance of this application as straightforward as possible? We must consider the tools we provide, the way we have separated our components, and the means for making incremental changes to the application over time.

General operation considerations
Software criteria	Description
Rich set of support tools and interfaces	Strong administrative tools could be graphical in nature or through a well-defined programmatic interface. We may not want to teach administrators a brand new tool, but rather leverage existing skill sets or enterprise configuration tools. This would factor in to our product choice.
Clear strategy for versioning system components	If a product is built with a clear separation of concerns, it will be easier to make isolated changes. That said, if a solution is expected to undergo regular changes then we have to fully grasp the ways to consistently deploy new versions.
Defined extensibility points	Extensibility can be built both into the software itself and into the system built on top of the software.
Built-in instrumentation and tracing	Many organizations have existing application monitoring tools and it is important to find out if a particular software package can feed its data and system events into such tools.

Organizational aspects

You would think that after you took your project's design, development, and operations into account you have done proper due-diligence prior to architecting a solution. However, a good solutions architect always keeps an eye on organization strategy to make sure that what they are proposing for an isolated solution is in line with the broad vision of the company.

Here are a few things to consider when switching perspective from a project-centric viewpoint to an enterprise one.

Organization considerations
Software criteria	Description
Is sufficient for both temporary solutions and long-lived solutions	Sometimes we build solutions that are meant to temporarily solve a given problem. Maybe the organization is planning a massive system upgrade but needs an intermediate solution to a particular pain point. Conversely, we may be designing a solution that is expected to remain in operation for 4 to 6 years. If we look at the planned lifespan of the solution, this can help us decide which product offers the lowest total cost of ownership over that duration.
Includes support from Microsoft for solutions implemented with product	Product support is a critical component of enterprise systems. When you build a solution on top of a packaged application, you often get more vendor support than when you build a solution on a base framework.
Leverages existing software investments within the organization	These existing investments could be in employees or other software packages. Does the product use an underlying database technology already deployed at the organization? Or, is it an additional module of a product already in heavy use?
Limited impact on budget as introduction of this technology can be built upon existing environments	This relates to the prior criteria. The cost of software that underlies a solution is frequently a factor in product selection and ideally we can share existing infrastructure.
Complies with "buy vs. build" strategy of the organization	If you have an expert staff of developers on site and frequently find yourself customizing packaged products, then your organization may prefer building solutions vs. restricting themselves to packaged products. On the other hand, if an organization prefers to fit their needs into the capabilities of package applications so as to reduce ownership cost and accelerate development, then a heavier evaluation weighting should go to products with fixed boundaries and limited customization options.
Matches the risk tolerance of the organization	Some companies love being early adopters of technology and getting the chance to take advantage of the latest products and capabilities. For such companies, the risks of deploying new technologies are outweighed by the business benefits those technologies offer. However, other companies have a "service pack 1" mentality where only mature products are introduced into the organization landscape.
Provides sufficient speed to market for new solutions	We cannot make a blanket statement that building solutions with "Product X" is faster than building with "Product Y." This all depends on the solution. That said, we want to evaluate our candidate software choices by looking at which software allows us to build (and change!) a given solution as quickly as possible.