Category Archives: Development

How In-Memory Database Objects Affect Database Design: Hybrid Code

In my first attempts at building my code, I strictly went with either native or on-disk code. I specifically wrote the on-disk code to only use features that worked in-memory. This lead to one majorly silly bit of code, used to create system assigned key values. How would I create a customer number that was unique. We can’t use the Max(value) + 1 approach because it will be very hideous with MVCC isolation levels, since 100 connections might see the same value, leading to lots of duplication. You…(read more)

Cloud Computing Architecture Patterns: Don’t Focus on the Client

Normally I try to put topics in the positive in other words “Do this” not “Don’t do that”. Sometimes its clearer to focus on what *not* to do. Popular development processes often start with screen mockups, or user input descriptions. In a scale-out pattern like Cloud Computing on Windows Azure, that’s the wrong place to start.

Start with the Data

   Instead, I recommend that you start with the data that a process requires. That data might be temporary or persisted, but starting with the data and its requirements helps to define not only the storage engine you need but also drives everything from security to the integrity of the application. For instance, assume the requirements show that the user must enter their phone number, and that this datum is used in a contact management system further down the application chain. For that datum, you can determine what data type you need (U.S. only or International?) the security requirements, whether it needs ACID compliance, how it will be searched, indexed and so on. From one small data point you can extrapolate out your options for storing and processing the data. Here’s the interesting part, which begins to break the patterns that we’ve used for decades: all of the data doesn’t have the same requirements. The phone number might be best suited for a list, or an element, or a string, with either BASE or ACID requirements, based on how it is used. That means we don’t have to dump everything into XML, an RDBMS, a NoSQL engine, or a flat file exclusively. In fact, one record might use all of those depending on the use-case requirements.

Next Is Data Management

 With the data defined, we can move on to how to store the data. Again, the requirements now dictate whether we need a full relational calculus or set-based operations, or we can choose another method based on the requirements for the data. And breaking another pattern its OK to store in more than once, in more than one location. We do this all the time for reporting systems and Business Intelligence systems, so this is a pattern we need to think about even for OLTP data.

Move to Data Transport

How does the data get around? We can use a connection-based method, sending the data along a transport to the storage engine, but in some cases we may want to use a cache, a queue, the Service Bus, or Complex Event Processing.

Finally, Data Processing

Most RDBMS engines, NoSQL, and certainly Big Data engines not only store data, but can process and manipulate it as well. Its doubtful that you’ll calculate that phone number right? Well, if you’re the phone company, you most certainly will. And so we see that even once we’ve chosen the data type, storage and engine, the same element can have different computing requirements based on how it is used.

Sure, We Need A Front-End At Some Point

Not all data is entered by human hands in fact most data isn’t. We don’t really need a Graphical User Interface (GUI) we need some way for a GUI to get data into and out of the systems listed earlier.  

But when we do need to allow users to enter or examine data, that should be left to the GUI that best fits the device the user has. Ever tried to use an application designed for a web browser on a phone? Or one designed for a tablet on a phone? Its usually quite painful. The siren song of “We’ll just write one interface for all devices” is strong, and has beguiled many an unsuspecting architect. But they just don’t work out.  

Instead, focus on the data, its transport and processing. Create API calls or a message system that allows for resilient transport to the device or interface, and let it do what it does best.


Microsoft Architecture Journal:
Patterns and Practices:
Windows Azure iOS, Android, Windows 8 Mobile Devices SDK:
Windows Azure Facebook SDK:

One more reason to to understand query plans, not directly performance related

One of the things that separates a good programmer from a great one is a firm understanding about what is going on inside the computer. For some programming languages, it is very obvious what is going on inside the computer because you are working at a very low level. For example, if you are a C/C++ programmer writing an OS, you will know a lot about the hardware as you will interact with it directly. As a .NET programmer you are more encapsulated from the hardware experience, making use of the .NET…(read more)