Category: Work

In my work I often deal with, or I help others deal with, significant volumes of data, with the goal of identifying trends or correlations or to validate a hypothesis.

Too often, there is a tendency to let the automated software find a result, and then trust that result as a mathematical fact and the solution to the problem, which is neither well defined nor sufficiently bounded. In the wrong hands, sophisticated analysis tools lead to the wrong conclusions. It’s like teaching a kid how to use a calculator before teaching the 4 basic math operations. Or teaching a teenager the formula for standard deviation without explaining the principles behind a normal distribution.

When a computer produces a result that the human driving that computer, as the owner of the data, does not fully understand, the result is probably junk for the problem the human is trying to solve.

The corollary is that the correct result should be understood by the human that owns the data and posed the problem to be solved by the computer. Fundamental stuff worth remembering.

After a brief summer vacation, the stress caused by external expectations at work is a little more difficult to handle. We’ve all been through periods of intense external pressure at work, and despite efforts to deliver according to expectations, the results are not entirely satisfactory for the recipients.

A recent difficulty involves a parser for an industry standard format, where the objective is to enable a key differentiator in our commercial product. The parser meets the requirements of most users, but fails to satisfy those who planned to extend the use cases beyond the original product intentions.

Another situation concerns customization of the product, using the built-in tools to provide the automation for special use cases, creating an application with high added value. The initial requirements are met, but users always desire new functionality as their experience grows and they face new challenges.

This type of situation is often encountered for those providing products and services with high added value, and can lead to a never ending list of remaining tasks and a growing set of deliverables. My advice is:

– Set and agree on the correct expectations early in the project, and remind everyone of this agreement as often as needed.

– Create the appropriate environment and business model to cope with an increased specification or list of deliverables.

Creating a win-win scenario between the user or buyer, and the supplier, requires both of the above conditions, and will avoid frustrations on both sides as is typically seen with complex projects. The difficulty in implementing such a situation often comes from the involvement of multiple persons, with individual goals, and the end result is a loop that is not closed.

Enough procrastination, time for action!

All new software projects go through a phase where performance, defined as the number of tasks and the volume of data handled, divided by the time required to achieve the desired result, is much less than originally planned. When this performance shortfall impacts overall project success, solutions must be found, and it’s not always easy to retrofit them.

A common software programming approach is to build complex multi-purpose objects, and give those objects many properties and methods. This leads to significant overhead, creating performance issues when dealing with huge data sources and/or situations where the methods are too generic and not optimized.

A complementary approach is to create numerous reusable building blocks, each with limited functionality, and then sequence many of these together to create an application, commonly known as a workflow. This leads to significant inter-process data transfer, which is typically limited to one way transfers. There is significant overhead associated with managing each block, and performance decreases significantly as the workflow becomes more complex with a large number of blocks.

The data structures used can also represent a significant usability and/or performance constraint, such as choosing flat data representations when hierarchy exists, or sequentially accessed data files when random access is required, or when data compression schemes interfere with optimal data access (such as column compression when row access is required from tables).

The highest performance can only be achieved with software that enables the use case in the most direct way, with the least number of unused options and the smallest number of building blocks. In complex environments, this can lead to a large number of use case specific software, and a balance has to be found with other techniques.

My advice is to carefully align the software approach with the expected use cases, and to accept fundamental changes to correct early wrong assumptions, instead of implementing workarounds which inevitably lead to less performance. And when ultimate performance is required, specific development can circumvent existing methods and objects to “get the job done fast”.

There are some ideas in life that summarize the approach we each have in everyday situations.

For work, where we provide software to analyze and solve complex industrial problems, my approach is: simple, reusable and adaptable.

Simple: this leads to the fastest solution, least time spent on specification, and less rework when the requirements change.

Reusable: software solutions are assemblies of individual building blocks, and with reusable blocks new solutions can be competed faster.

Adaptable: the details of the requirement are never fully understood until the solution is in use in real situations, so it is essential to adapt deliverables in the field to achieve success.

More than a philosophy, these concepts are part of my daily activity.