Technology & Beyond: Uncover Business Insights in “Dark Data” Using Data Integration

If you were paying attention during the recent total solar eclipse, then you probably experienced an interesting, albeit brief, shift in perspective. Maybe you noticed a soft, eerie twilight or unusual shadows during the peak. Or even a noticeable temperature drop. Even though the experience passed by quickly, you’ll remember those high-quality minutes for your life. Especially if you were in the zone of totality.

It’s thought-provoking to think about the things that had to line up for the optimal perspective, starting with the path of the moon across the sun, to your physical latitude during the eclipse to the weather conditions to whether or not you were able to get to a good vantage point and so on.

Getting an optimal perspective on “dark data”—the valuable data that is piling up in systems and databases that you never really see or use—also requires careful alignment of a lot of different factors. There are the technical choices around what data integration and dashboarding or data visualization solution to choose and why. What data you actually need to get to and why. And how to ensure that data you are using is complete/accurate and uncorrupted. Read this blog that provides guidance on why it is important for manufacturers to have the right data and data visualization strategy to manage operations.

Gaining perspective on a data integration project

If you’ve never been through a data integration project, you might be struggling with where to even begin. And if you have, you may have questions as to why you weren’t able to get the results or insights you anticipated.

Either way, we plan to spend some time exploring the key ins and outs of a data integration project over the course of a few blog posts. We’ll start with some solution fundamentals in this post, and then move into important questions and issues around:

• Properly understanding the data you have and how it is used…hint…not everyone is using it the same way and have different definitions and calculations
• Insight failures – how discovery data integration processes often go wrong and how to avoid common mistakes
• Understanding what data, resources and processes are needed to support the business so it can run itself

High-level IT considerations

At a high level, technology decisions related to an integration project are relatively straightforward. Your organization needs to get to data from disparate sources and there are different ways to do that. Some of the most common include:

• Using file transfer protocol (FTP) to share data with your dashboard tool
• Using middleware or a middle tier to set up a data integration layer that connects systems and databases to a central dashboard
• Relying on a cloud-based solution to capture data for analysis via a dashboard

Deciding what approach to use and why usually comes down to weighing key considerations about the complexity of your IT environment and bigger-picture business goals, risk posture, and maintenance and budget considerations. For smaller organizations looking toward their first data integration project, or even larger organizations trying to quickly make headway against key business goals, an FTP approach can be ideal because the setup is relatively quick and cost efficient and the maintenance challenges are minimal.

Of course, setting up a data integration layer is more complicated. It requires additional technology investments, a longer setup time and ongoing maintenance considerations. The setup of a cloud-based solution is usually much faster than that of an integration layer, although it raises a different set of security questions.

In all cases, what’s most important is to ensure that the business needs related to the types and amounts of data you need to see, how quickly and so on, drive the technical choices around the approach that will support the best results. Then you can talk to solution providers about things like how they transfer and protect your data and weigh all of the considerations accordingly.

Getting to the data that matters

Dploy Solutions, is a flexible business process management solution that gives you options when it comes to integrating your data. Whether you want to start with a secure file transfer protocol (SFTP) approach or are considering a data integration layer or the cloud, Dploy allows you to choose the data integration and sharing approach that makes the most sense for your business and technical needs and visualize it in intuitive dashboards. We also have operations management and process experts who can help ensure that you identifying the data and data sources you need to achieve business goals. And the data discovery process, as we will see in an upcoming post, is vital to project success and harder than it may sound.

In the meantime, if you have questions about your data integration options in general, or Dploy Solutions specifically, don’t hesitate to call (+1.919.471.5535).