How do you centralize, store, analyze, and visualize the enormous amount of data your organization is collecting?
Generally, there are two options:
Let’s clarify this up front: neither option is inherently bad (although one will take a lot more time than the other). For either option, there is always a use-case that makes sense. So which one should you choose?
Here’s a few questions that can help you decide:
The simple truth is that while both data warehouses and end-to-end business intelligence systems can be effective ways to wrangle all of your company’s data, each option has different initial setup and configuration processes.
In this blog post, we’ll outline the set up required for a BI solution that requires a custom data warehouse, and one that doesn’t.
If you’re going to tackle a custom data warehouse, it’s just going to be a single layer in your business intelligence technology stack. You’re also going to need an ETL system and a system for data analysis (at the minimum).
The chief challenge in setting up a workable data warehouse is ensuring that all the systems and processes that the data warehouse depends on are set up and managed correctly. You have two options when it comes to setting up your data warehouse. You can either host it on your own servers, or you can host it through a cloud service.
Some businesses prefer to host everything on their own servers for a few reasons:
That being said, maintaining your custom data warehouse on your own servers means that you are fully responsible for maintenance and optimization—there is no other line of support to fall back on (self-hosted business intelligence solutions are also called on-premise business intelligence).
This is one of the most popular choices for setting up a custom data warehouse, especially now when there are more cloud-hosting services than there were even a few years ago. Essentially, you pay on a subscription basis to rent server space. You can keep costs down, and rely on the support of the cloud service to protect your data warehouse.
Examples of these services include Amazon Redshift and Snowflake.
In either case, in a custom data warehouse-based approach you will face significant hardware, software, and human resource costs. At minimum, you should expect the building process alone to take you six months. This doesn’t include the time you’ll have to spend planning and designing your data warehouse, or the time allotted to training your team to use and maintain a custom data warehouse.
The design and development of your data warehouse are critical to its success, and more than 60% of data warehouse projects ultimately fail due to poor design.
Whether you build your entire data warehouse using in-house talent, or bring in expert consultants—this is recommended due simply to the depth of your investment in the data warehouse’s success—you will need to ultimately staff at least four ongoing positions:
Each staffing position will require its own management processes, although there can be some crossover. For example, data analysts work in the ETL layer cleaning and prepping data, as well as in the dashboard layer building data visualizations.
But one of the most significant differences between staffing your team for an end-to-end business intelligence system and for a custom data warehouse is the role of IT. Generally speaking, your IT team will need to support your data warehouse at each layer.
Generally, an end-to-end business intelligence system is faster to set up because it:
The trade-off is the loss of customization capability. While you have total control over your entire business intelligence flow with a custom build, you’re going to be sacrificing some of that with an end-to-end system.
(Fortunately, unless you’re a huge, enterprise-level business this isn’t going to be an issue).
Some end-to-end platforms (not all) still provide options for self-hosting. Because the software is already built out, installing it on your servers takes much less time than a custom data warehousing solution would, but it still takes longer than a SaaS or cloud service set up would.
You are still required to protect and update your system yourself.
Setting up an end-to-end solution on the cloud is the fastest way to get your business intelligence system up and running. It’s also the most secure (you’ll have the service’s dedicated team of developers constantly updating and protecting your system).
In general, the benefit of most end-to-end business intelligence platforms is that they have been built for technical and non-technical users. Additionally, because each install will be similar, most services have put out an extensive amount of documentation that can guide you through the process of building metrics and creating dashboards.
In terms of human resources, a BI implementation is also simpler and less expensive. You’ll need just one data analyst. That’s it. No massive IT teams or outsourced firms. Just one data analyst who knows about transforms, metrics, and datasets and who can manage those tasks in the system. And some BI platforms also offer templates and preconfigured workflows to help you get more accustomed to advanced BI functions.
Whereas the latter can require up to a year for full implementation, end-to-end BI solutions can take anywhere from two days to two months to implement. All the heavy-lifting configuration and training elements vanish.
Setting up a custom data warehouse requires a significant amount of time before you even start seeing data insights. Custom data warehouse implementation also requires a team of people to manage it. But for a business that needs the capabilities of a custom system, the initial implementation process is worth the pay off in the long run.
Alternatively, you can start seeing data insights in under a week with an end-to-end BI solution that doesn’t require setting up separate, custom systems.
The bottom line is this: you have options.
There are alternative BI solutions that require much less time to set up than a data warehouse, while simultaneously being more budget-friendly. Unless yours is one of those rare instances where only a data warehouse will fit your use-case, we believe that the set-up isn’t worth the expense of the data warehouse.
Curious about the capabilities of an end-to-end BI solution? Demo Grow’s platform for yourself.