How do you centralize, store, analyze, and visualize the enormous amount of data your organization is collecting?
Generally, there are two options:
- A traditional data warehouse. Most likely custom-built, often hosted on your own servers. This stores your data and works in conjunction with other tools used to transform data and derive insights.
- A cloud-based, end-to-end business intelligence (BI) solution. This includes everything (ETL tools, storage, and analytics) in one system (Tableau, Power BI, Grow, Domo).
A custom data warehouse vs. end-to-end BI SaaS
Let’s clarify this up front: neither option is inherently bad (although one will take a lot more time than the other). For either option, there is always a use-case that makes sense. So which one should you choose?
Here’s a few questions that can help you decide:
- Do you need to start seeing insights immediately? (If yes, then you need an end-to-end BI system. If you have something in place that works well enough for you now that you can wait a year, then you can wait for a custom data warehouse).
- Do you have the budget, the people, and the hardware on hand to manage a data warehouse, or do you need to plan for those things? (If yes, then you’re ready for a custom data warehouse. If no, then you may want an end-to-end platform).
The simple truth is that while both data warehouses and end-to-end business intelligence systems can be effective ways to wrangle all of your company’s data, each option has different initial setup and configuration processes.
In this blog post, we’ll outline the set up required for a BI solution that requires a custom data warehouse, and one that doesn’t.
Getting a data warehouse up and running
If you’re going to tackle a custom data warehouse, it’s just going to be a single layer in your business intelligence technology stack. You’re also going to need an ETL system and a system for data analysis (at the minimum).
The chief challenge in setting up a workable data warehouse is ensuring that all the systems and processes that the data warehouse depends on are set up and managed correctly. You have two options when it comes to setting up your data warehouse. You can either host it on your own servers, or you can host it through a cloud service.
If you host your data warehouse on your own servers
Some businesses prefer to host everything on their own servers for a few reasons:
- If a business has other large (or custom) reporting systems that they need to communicate large volumes of data at a time, it’s easier to do so when everything is on the same server.
- They have more control over the server—they don’t have to share server space with anyone else.
- Users can access the system only if they’re on the server, making it more secure.
That being said, maintaining your custom data warehouse on your own servers means that you are fully responsible for maintenance and optimization—there is no other line of support to fall back on (self-hosted business intelligence solutions are also called on-premise business intelligence).
If you host your data warehouse on the cloud
This is one of the most popular choices for setting up a custom data warehouse, especially now when there are more cloud-hosting services than there were even a few years ago. Essentially, you pay on a subscription basis to rent server space. You can keep costs down, and rely on the support of the cloud service to protect your data warehouse.
Examples of these services include Amazon Redshift and Snowflake.
If you choose either self-hosted or cloud-hosted custom data warehousing
The development
In either case, in a custom data warehouse-based approach you will face significant hardware, software, and human resource costs. At minimum, you should expect the building process alone to take you six months. This doesn’t include the time you’ll have to spend planning and designing your data warehouse, or the time allotted to training your team to use and maintain a custom data warehouse.
The design and development of your data warehouse are critical to its success, and more than 60% of data warehouse projects ultimately fail due to poor design.
The team
Whether you build your entire data warehouse using in-house talent, or bring in expert consultants—this is recommended due simply to the depth of your investment in the data warehouse’s success—you will need to ultimately staff at least four ongoing positions:
- An information systems manager
- A backend developer
- IT support
- Data analyst
Each staffing position will require its own management processes, although there can be some crossover. For example, data analysts work in the ETL layer cleaning and prepping data, as well as in the dashboard layer building data visualizations.
But one of the most significant differences between staffing your team for an end-to-end business intelligence system and for a custom data warehouse is the role of IT. Generally speaking, your IT team will need to support your data warehouse at each layer.
Getting your end-to-end BI solution up and running
Generally, an end-to-end business intelligence system is faster to set up because it:
- Includes data warehousing, ETL functions, data visualization tools, and analytics. You don’t have to set these up separately.
- Can be available on-premises (hosted on your own servers) or on the cloud.
- Requires less design and development from your IT team—most of that work is already done for you.
The trade-off is the loss of customization capability. While you have total control over your entire business intelligence flow with a custom build, you’re going to be sacrificing some of that with an end-to-end system.
(Fortunately, unless you’re a huge, enterprise-level business this isn’t going to be an issue).
If you host on-premises
Some end-to-end platforms (not all) still provide options for self-hosting. Because the software is already built out, installing it on your servers takes much less time than a custom data warehousing solution would, but it still takes longer than a SaaS or cloud service set up would.
You are still required to protect and update your system yourself.
If you host on the cloud
Setting up an end-to-end solution on the cloud is the fastest way to get your business intelligence system up and running. It’s also the most secure (you’ll have the service’s dedicated team of developers constantly updating and protecting your system).
Training your team
In general, the benefit of most end-to-end business intelligence platforms is that they have been built for technical and non-technical users. Additionally, because each install will be similar, most services have put out an extensive amount of documentation that can guide you through the process of building metrics and creating dashboards.
In terms of human resources, a BI implementation is also simpler and less expensive. You’ll need just one data analyst. That’s it. No massive IT teams or outsourced firms. Just one data analyst who knows about transforms, metrics, and datasets and who can manage those tasks in the system. And some BI platforms also offer templates and preconfigured workflows to help you get more accustomed to advanced BI functions.
Whereas the latter can require up to a year for full implementation, end-to-end BI solutions can take anywhere from two days to two months to implement. All the heavy-lifting configuration and training elements vanish.
Rolling out your business intelligence strategy
Setting up a custom data warehouse requires a significant amount of time before you even start seeing data insights. Custom data warehouse implementation also requires a team of people to manage it. But for a business that needs the capabilities of a custom system, the initial implementation process is worth the pay off in the long run.
Alternatively, you can start seeing data insights in under a week with an end-to-end BI solution that doesn’t require setting up separate, custom systems.
The bottom line is this: you have options.
There are alternative BI solutions that require much less time to set up than a data warehouse, while simultaneously being more budget-friendly. Unless yours is one of those rare instances where only a data warehouse will fit your use-case, we believe that the set-up isn’t worth the expense of the data warehouse.
Curious about the capabilities of an end-to-end BI solution? Demo Grow’s platform for yourself.
