Why you Should keep your Developers away from your Data Integration

First of all, some definitions. We can approach Data Integration (DI) in two basic ways. You could ask your developers to write some specific software (code) that links your data to where you want it, and modifies it along the way. This is a Code-Based approach. Alternatively, we can use a tool that has a dashboard upon which we define what we want to happen. The tool then does the work for you. This is a Tool-Based approach.

The key arguments for the Code-Based approach are that the end solution will be:

– Quicker: The code is only used for the task in-hand and not to support a dashboard or auditing .etc
– More flexible: The code is bespoke, it can be tailored to accurately meet your needs

Both of these arguments are relevant, but that doesn’t mean that you should use a Code-Based approach. Here are 5 reasons why you should keep your developers away:

1. Speed is not everything

Unless you are continuously streaming large quantities of data for a real-time application, speed is unlikely to be the key constraint. You are far more likely to come up against other hurdles before your DI tool keels over.

2. Specification is REALLY difficult

Most IT projects fail because of the specification, not the technology. This isn’t because the team wasn’t trying, it’s because writing a good ICT specification is an art form in itself. So yes, coding will give you exactly what you ask for, but are you asking the right questions? A tool-based approach assumes that you won’t set the perfect specification, giving you the ability to adapt and correct easily and readily later down the line.

3. You will lose your developers, and with them, the logic of your code

If your developers are young and good at what they do, they will move jobs every 2-3 years on average. This means that their code logic, that you rely on, will walk out the door in 2-3 years. Another developer will be able to unpick the thinking, but this is a hard and laborious job. On the other hand DI tools use an interface that shows you in a non-technical way what is happening, in what order, and with what dependencies. This means that you have a far greater number of people who can understand the logic and manage it going forward.

4. Development takes time to arrange and manage

Your environment is changing quickly and the rate of change is increasing. Therefore you need a way to manage your data integrations and adjust to these changes at a speed that means you’re not the slowest element. Good development takes time to arrange, specify, manage and test. DI tools are designed to enable fast and flexible changes, their interfaces will nearly always enable you to make a change faster than coding will.

5. Standardisation is key to making the whole environment work

Your environment will work best when your team is presented with tasks that require a consistent set of skills. It means they’ll know how to approach a problem immediately without having to look it up. It also means that risks and task queues are avoided as skills are not “siloed”. One team member can take-over another’s work in a seamless manner. Coding prevents this, Tool-Based approaches encourage it.

At miso, we have a team of coders and we create bespoke code for a huge variety of purposes. We see the value of code, but acknowledge that there is a time and place for it. We also see the value in using a Tool-Based approach for all our day-to-day data management.

It’s not really a matter of either a Code-Based approach or a Tool-Based approach. It’s more a matter of using code for tasks where you see long-term stability, and using tools for where you anticipate change and need flexibility.