Modern data warehouse fueling the digital transformation
Article By : Joey Baruch, Alvarez & Marsal
Here is a sneak peek into the five stages of digital transformation: collection, consolidation and sharing, analytics, automation and culture.
Digital transformation is the name of the game in 2021. It’s everywhere! However, there’s another, lesser-known trend that, if used properly, can supercharge that transformation: the rise of the modern data warehouse (DW).
Two decades of tech disruption, along with a new generation of consumers and a global pandemic, have dismantled the old way of doing things for “old guard” companies. As these companies race to achieve digital transformation for securing competitive advantage, understanding how to leverage the underlying technology will be critical for their present and future success.
Contrast to the world before
Once upon a time, having a working relationship with an organization’s data came at considerable cost. Multiple engineers were needed to maintain costly legacy big data ecosystems such as Hadoop. Several others were required to build and maintain data pipelines. Business analysts were at the mercy of all of them. This dependency significantly slowed the delivery of statistical models, decision automation, and even ad-hoc queries to answer simple business questions.
Luckily, so much has changed. Cloud computing significantly reduced the cost of data storage through economies of scale and iterative innovation. Clever engineering in modern data warehouses separated storage from compute, dramatically reducing costs and enabling software-as-a-service (SaaS) and cloud-based models for data warehousing solutions.
Teams of engineers are no longer needed to extract and transform data for analysts to use. Thanks to innovations in the modern DW, analytics teams can now load all the data in raw form instead of transforming or truncating original data for a specific purpose. These innovations empower analysts to discover, clean and utilize the data as they see fit and on their schedule.
This trend recently took another step forward with the rise of dbt (data build tool), an open-source project by dbt-labs, previously Fishtown Analytics. dbt opens the door for age-old software engineering practices to enter data analytics. This dynamic further empowers the shift of the analysts’ workload from ad-hoc reactive query responses to building production-grade data transformations and delivering continuous value to key stakeholders.
Road to digital transformation
Digital transformation can be broken into five stages: collection, consolidation and sharing, analytics, automation, and culture. Each stage adds another layer of value to the company. Different companies will have different starting points; still, the “digital promised land” is now within greater reach, thanks to recent DW innovations, which have lowered costs and barriers to entry.
Stage one of digital transformation journey begins with data collection. For example, does the marketing team collect data about how customers interact with the website and ad campaign performance? What are people saying about the company and how are they engaging with content?
Identifying data for collection should be a critical strategic consideration rather than mindless data dumping. Whatever data is selected to collect, modern data ingestion practices will significantly reduce time, money, and necessary resources required. Thanks to open-source projects like Singer and companies like Stitch, Fivetran, and Airbyte, an analyst on the marketing team can ingest website usage and ad-clicks data to the DW through a beautifully simple user interface.
Stage two is the consolidation, aggregation, and sharing of data across systems and business functions. It’s often easiest to consolidate collected data from multiple sources in one place and make it accessible to those who need it. In most cases, different teams will need a subset of the company-wide data collected strategically. Once data is loaded to a modern DW, the consolidation and sharing step is as easy as granting access to the DW. It’s easy to picture product usage data from the product team and payment data from the finance team continuously streaming into the DW, where marketing has direct access.
DW access creates a trove of opportunities for analytics and is linked to stage three which focuses on empowering data-driven decision making. Because the underlying data is sourced from all the relevant functions, it provides a comprehensive picture and mitigates against a myopic, limited view. Imagine data that supports the marketing team’s decision to reallocate budget after seeing correlations between the ad campaigns, social media chatter, product usage data, and new customer payments.
Stage four involves decision automation that can streamline operations, reduce costs and human error, and allow leaders to focus on what really matters. By leveraging analytics along with heuristics and minimal human intervention, the marketing team can now focus on higher-level strategy while leaving the grunt work to an automated system.
A word of caution. In situations with real-time or low-latency needs, generally less than five minutes, stream processing technologies should be considered.
Stage five centers on the mindset transformation that must accompany digital transformation. Success and failure must be redefined to sustain innovation. Building the right culture and incentives that empower teams to try new things is crucial to digital transformation. While a DW can significantly lower the cost of asking and answering further questions, the cultural transformation must happen across the whole organization and not just with the data-wranglers.
This article was originally published on EDN.
Joey Baruch is chief technology officer (CTO) at the Data Intelligence Gateway (DIG) operations of Alvarez & Marsal.