Subscribe by RSS
Enterprise software, both legacy and new, handles upgrading customized software very poorly. Some software allows customizations, but forget about upgrading to a newer version of the base product without redoing all of your customizations. Some vendors take the approach that you shouldn't customize at all and, instead, use the software as it comes out of the box to avoid the travails of upgrading.
At Dovetail, we asked the question, “What if we could design our base product so that it was both easy to customize and easy to upgrade? Why had no one thought of this before? Or better yet, why hadn't someone done it?” It is a tough problem to tackle, that’s why. In other words, we saw this problem as an opportunity for us to solve and make the world a little better. We fundamentally believe that organizations should not accept that they should spend large sums of money to keep up to date with the latest technology or that they should just live with outdated technology.
When we started designing the new Dovetail CRM product, one of the primary goals we had at the time was to make it so that the software would always be easy to upgrade, even if it were heavily customized. We noticed that a lot of enterprise software (not just legacy software, but software that is being developed currently all around the world) handles customization poorly.
Thinking through the issues involved, we came up with a set of principles and practices that we thought would guide us through the process. These principles and practices fit into two categories:
- Quantity: Minimizing the amount of customizations that will be affected by upgrades of the base product and
- Quality: Minimizing the pain of upgrading certain customizations that will be broken by the base product upgrade.
We realize that there is no way to completely mitigate all upgrade pain. However, we can get rid of the vast majority of it and make the little pain that’s left easy to deal with. In the next section, we’ll discuss the principles and practices we’ve identified (and followed) for “upgrade-safe customizations” in an enterprise software product.
Principles
First Principle: Small Surface Area
How can I keep my customization code from breaking when I upgrade the underlying product?
The simple answer is: You can’t -- at least, not entirely. You can only minimize potential breakages. The first principle, “Small Surface Area,” deals with this problem. That is, keep the “surface area” (or “site” or “scope”) of customization small. By this, we mean that the place where custom code and baseline code intersect should be as small as possible while still being able to accomplish the customization. This does not mean keeping customization size or scope small, but keeping the points where custom code intersects baseline code small. You should be able to make as big a customization as you want, but this principle says that the interaction between that customization and the baseline product should have as small of a surface area as possible. The goal is to allow the greatest possible flexibility of customization while minimizing the contact between customization code and baseline code. By keeping intersections small, the baseline code doesn’t have to change as much and customization code touches fewer points in the baseline code, thus resulting in fewer forward compatibility issues.
We found that in order to satisfy the needs of this principle, the baseline code itself would have to provide as many extension points as possible, and to keep those extension points as small as possible while still being useful. We accomplished this in our product through the following means:
Internal Architecture Extension Points
The internal design of our code and the APIs it exposes for our internal use and for external developers is very compositional and not monolithic. The term compositional means that the software is the sum of many small parts each working together (sort of like using Lego® blocks to build a big statue). The term monolithic means that the software is more like a sculpture carved from a solid chunk of marble. Both are means to an end, but one means (compositional) allows for greater flexibility once the end (enterprise software) is achieved. Due to our use of compositional architecture, many of the smaller aspects of our software can be extended or even replaced entirely.
A concrete example of this is our ability to allow custom code for numbering schemes. Every time a new Case is created in our system, a new, unique identifier is created for it (i.e. Case 1000). These identifiers are configurable by the user to a certain extent, but if the user has a complicated numbering scheme (i.e. Case 20071231-frank-smith-arizona), you can write your own .NET code and have full control over identifier generation. You can replace the one “Lego® block” that controls Case identifier generation with your own custom-developed block. The first principle is satisfied in this scenario because the “surface area” of change is limited to a small interface between baseline code and custom code: identifier generation.
Event Architecture Extension Points
Our system generates events when certain interesting activities happen in the course of using the system that can be “hooked” by custom code to perform some action in response to that event. This response could include calling back into the system to affect some change thus triggering even more events. Consider this concrete example, you could write a custom event handler that responds to the “Case Created” event to automatically dispatch that Case to a specific queue for processing by agents. In this manner, there is no direct interaction between baseline code and custom code (more like sending letters in the mail [indirect] versus an in-person spoken conversation with someone [direct]). This indirect communication allows for high degrees of change on both the baseline code side and custom code side such that compatibility issues are minimized.
User Interface Architecture Extension Points
Our system allows custom code in the user interface (on the client-side, running in the user’s browser) to change or enhance behavior of the system. This is done through our extensive JavaScript client-side facilities and the facilities of the browser itself. Custom JavaScript script files can be loaded on the following levels: system, area, page-by-page, and even partial-page. We also allow the loading of custom Cascading Style Sheet (CSS) and display format customizations to tweak the look and feel of the application. These custom JavaScript and CSS files are loaded on top of baseline JavaScript and CSS files to extend or override them. In this way, custom code builds on top of baseline code versus interacting with it directly. There is still potential for compatibility breakage, but not as severe as direct interaction between baseline and custom code. Custom code progressively enhances baseline code in a distinct and separate way, thus allowing both baseline code and custom code to diverge slightly in future versions but still remain functional. We’ll talk about “progressive enhancement” later in this post.
Second Principle: Separate Customization Code and Baseline Code
How can I be sure that when I upgrade my baseline product that my customizations won’t mysteriously disappear?
The second principle, “Separate Customization Code and Baseline Code,” answers this question. Customization code should always be kept separate from baseline code so that the baseline code can be upgraded or patched independently of the customization code. By keeping the two separate, it’s always clear which code you’re working with so that there is no confusion. This also ensures that the baseline code remains 100% pure and in a known state so that there are no surprises when the baseline is upgraded. For example, many systems will mingle baseline and customization code. They allow you to directly change or customize the baseline code, which allows for great short-term flexibility, but hampers future options. By mingling the two types of code, we create a problem in the future when we need to upgrade the baseline code because now the customization code is co-mingled with the baseline.
It is important that custom code be physically separate from baseline code – that is, in a completely different folder on the disk. Consider Figure 1, which shows the folder structure of customizations for Dovetail CRM:
Figure 1: Example folder hierarchy for customization code
As you can see, all customer overrides are kept in the Customer -> Overrides folder -- separate, safe, and manageable. When it comes time to upgrade the baseline product, we can safely lift these customizations out, set them aside, upgrade the product, and then replace the customizations with high confidence that we haven’t created an unfortunate situation of customizations being lost due to co-mingling with baseline code.
Third Principle: Automate Acceptance Testing
How can I be sure that when I upgrade my baseline product that my customizations will still work?
The third principle, “Automate Acceptance Testing,” deals with this question. To recap a little, we’ve established how to customize with upgrades in mind (first principle), where to keep customizations in light of upgrades (second principle), now we have to deal with the problem of upgrades to the baseline product breaking our customizations. Unfortunately, there is no way to future-proof customizations. The product will change out from underneath you. There is no way to avoid this either from the baseline developer’s side or from the customization developer’s side. Instead, we can try to avoid this fact from negatively affecting our customizations and we can try to “pin down” our changes so that, after a simulated upgrade in our testing environment, we can detect with surety, which customizations will break or are no longer needed.
At Dovetail, we use a lot of automated acceptance testing using Web testing frameworks like Selenium RC. We script out tests that can verify whether a particular feature is working correctly (as we have defined “correct”). We can and do run these tests often since they are fully automated and it doesn’t take that long. When things break or changes to one area of the system break something in another area of the application, we can detect that automatically.
In essence, we are “pinning down” features so they don’t blow away in the breeze of change. As the system changes, we can be certain through our tests that either these features will remain functional or we will be alerted to their breakage. With this power, we can confidently change the system and know that we are not breaking things or are fully aware of the things we are breaking so that we can fix them.
We recommend that customizations be tested just as baseline functionality is tested and be treated with the same respect and importance. This treatment of functionality testing helps achieve confidence that, after having upgraded the baseline, our customizations will still work or at least we produce a definite list of the customizations that do not work so that we can address them systematically.
Practices
First Practice: Use source control
One key practice we use here at Dovetail both for our code and for customization code/professional services is source control. Our favorite source control provider is Subversion and we highly recommend it, especially when collaborating with customers.
Source control is important because it:
- Provides a history of changes to the code over time (also allowing you to “roll back” to a previous version)
- Allows multiple contributors to the code through merging many changes together versus having to pass documents around to individuals and waiting for each person to add their changes in series
- Provides a de facto backup of the code, so the code is not lost. Source control also prevents the loss of one developer’s computer necessarily resulting in the loss of the code since the source control server is an independent storage location for the code.
- Provides a central repository, which represents the canonical form of the code from which all people can be sure that this is the latest version of the code. It removes doubt and confusion as to where the code went and prevents code loss when developers move on to other positions or companies. The code is no longer associated with the individual, but with the server thus providing greater accountability and visibility to the organization.
By using source control, we eliminate a number of unfortunate and avoidable situations that can happen while developing code or maintaining code throughout the lifecycle of an enterprise application.
Second Practice: Use progressive enhancement as much as possible
Another practice we have found very key for making customizations “upgrade-safe” (not to mention for the development of the baseline product itself) has been to use “progressive enhancement” as much as possible. “Progressive enhancement” has a more formal definition that deals with web browser compatibility. We’re not referring to that definition. We’re speaking in a more general sense that customizations should build upon the foundation of the baseline product. Customizations should act upon the features in the baseline product (disabling them, overriding them, or enhancing them) but should not remove or replace them.
A concrete example of this is a UI element that appears in the baseline product but needs to be removed via customization. Rather than copying the entire baseline web page and removing that element, we can inject some JavaScript code that will hide that UI element. Either way the user will not see the element. In this manner, we have accomplished more than just the stated requirement to hide the UI element. We have accomplished the unstated requirement, which is to make sure that the stated requirement stays that way after successive upgrades of the baseline product. Both solutions accomplish the stated goal, but only the “inject some JavaScript code” solution accomplishes the unstated goal because it does not copy any of the baseline code. Copying baseline code is an anti-pattern that will most certainly result in conflicts during upgrades. The baseline code will likely change from version to version, so in light of the second principle above, we should always keep customization code separate from baseline code. This includes negative customizations such as removing functionality from the baseline.
By layering on small units of functionality, we can still have the same level of freedom of customization without compromising our ability to upgrade. We will still be able to upgrade the baseline product while keeping the customizations intact to the maximum extent possible.
Summary
The problems of keeping customizations when upgrading a base enterprise software product are largely avoidable. Using the principles and practices described above, Dovetail has created an enterprise software product that is both upgradeable and customizeable. This has given us and our customers a lot of freedom to focus on solving current business problems rather than re-solving past business problems.
Post new comment