Transforming data infrastructure on an enterprise level

data transformation

Think-it’s journey started with BMW Group’s work to stay at the forefront of the auto industry’s digital transformation. 

The Think-it team was aware of BMW Groups desire to transform its data infrastructure to a single platform solution agile enough to support both internal data business needs but also to support the various emerging customer use cases.  Additionally, they ultimately sought to deliver real-time access to data such as information on speed, tire status, engine status, and more. In addition, they wanted to integrate analytics and machine learning into the data infrastructure to accelerate the development of new, innovative services. On top of that, there would arise future needs to be able to perform sovereign data exchange with external companies. So above all, the data infrastructure being built would have to provide the governance required to ensure privacy and security.

Our goal

Build the right data architecture for end-to-end data-driven use cases delivering a smart and holistic experience for BMW Group and its customers.

Unlock driver data to help improve driving experience and safety

Automate the ingestion of new data into the Cloud Data Hub

Make data engineering more scalable and efficient by facilitating access to data

Provide intelligent services on top of driver data to customers and dealers

The journey

2019
2020
2021
2022
2023
2024
Shaping driver behavior with data
Scale BMW Group's cloud platform utilization on an enterprise level
Provide intelligent services on top of driver data to customers and dealers
Share sovereign data from BMW Group externally
Create a single source of truth to manage data sharing

The challenges

OUR FIRST STEP

Shaping driver behavior with data

Think-it first contributed to BMW Group’s value chain by producing driver-centric added value and unlocking new product initiatives. Our first use-case was supporting AWS cloud adoption within the BMW Group by building an end-to-end data-driven use case delivering an intelligent personal driver assistant that served as a boilerplate for other use cases moving forward.

Our PoC was a data architecture that reliably ingested data and extracted insights to analyze driver behavior, delivered in a mobile-first UI. This could lead to increasing road safety, developing usage-based pricing, and offering tailored consumer services.

The system followed a 100% cloud native architecture optimized for large scale networks and big data volumes. We ensured scalability through an extensive use of serverless architecture coupled with automated deployment and delivery with a combination of CodePipeline, Terraform, and CloudFormation. Additionally, the system was responsible for performing data gathering, analysis, modeling, and visualization using state of the art machine learning on Sagemaker.

The collaboration was a success and met all its objectives. As BMW Group's cloud adoption increased, the partnership was deepened in order to further scale the utilization of the BMW Cloud Data Hub.

BMW Mobile

Source: BMW Group

BMW Mobile

 

NEXT STEP

Scale BMW Group's cloud platform utilization on an enterprise level

The BMW Group’s cloud adoption and creation of the Cloud Data Hub (CDH) on Amazon Web Service (AWS) was a huge step forward from the previous data lake, which posed organizational bottlenecks that slowed down innovation. With massive amounts of data available, The BMW group needed to make all that data available through one single platform.

To better manage this data, the notion of “data providers” and “data consumers” was introduced to increase both the autonomy and agility of its software engineering teams. Data providers ingest and transform data. Data consumers could then leverage data for their use cases. Both providers and consumers could use these services in their own accounts, and only share well-defined interfaces that could be controlled by a central API, helping prevent bottlenecks. 

How can the Think-it team help manage the digital transformation into the cloud on an enterprise level, allowing more data providers and consumers to effectively onboard on the CDH and leverage it?  We automated the onboarding and cloud configuration for new data consumers within the CDH to allow users without cloud expertise to leverage the platform and ensure scaling to enterprise level.

The Think-it team had the goal of implementing a seamless and automated onboarding process for data consumers on the cloud infrastructure. This meant: automate the bootstrapping and setup of new cloud accounts and resources, automate setup of tools (EMR) for big data processing and analysis, enable data access and consumption through a simple and intuitive UI, and infrastructured deployment via Terraform & CDK.

BMW whiteboard planning

Early whiteboard planning

BMW team

Presenting our work during the BMW Cloud Days 2019

GOAL: Automate the creation of data consumer (data analyst) stacks seamlessly integrated within the CDH portal in a cross-account AWS based architecture:

  • Robust backend using AWS-CDK for generating CloudFormation templates that can be deployed to consumer accounts
  • CloudFormation templates created using “Cloud Resources API” to define the structure of the analyst stack and guarantee its ability to pull harmonized data
  • Data engineering is available to CDH consumers within a single step inside their notebooks
  • Cost-effective solution that relies on the ability to:
    • Merge capabilities of data consumers into a single resource (EMR cluster)
    • Estimate costs when creating a new stack
    • Show billing insights for existing stacks
    • Auto-suspend and auto-termination capabilities for idle resources\

The CDH main components were to Explore, Query, Ingest, and Analyze data. 

BMW CDH Analyze

CDH Analyze

BMW High level architecture

High level architecture

BMW Cloud Data Hub Architecture

Cloud Data Hub Architecture (Source: BMW Group)

The Data Portal front end application served as a data explorer to enhance the productivity of data analysts, data scientists, and engineers by offering a “popularity index” based on data usage patterns as well as  clearly displaying data resources.

As Think-it played its part with other contributors, BMW Group saw more integration of users and a thriving ecosystem of participants in the CDH. This opened opportunities to start building data-driven use-cases (while still contributing to the evolution of the CDH) on top of the data architecture. And one particularly innovative and unique use case in the competitive environment emerged around Digital Tyre Diagnosis to provide the driver with comprehensive information about the condition of the vehicle's tyres.

BMW Data portal

Screenshot of data portal. Source: BMW Group

STEP 3

Provide intelligent services on top of driver data to customers and dealers

Now that we’ve unleashed the data within BMW group, how can we build unique services on top of that data?

The BMW Group had the goal to provide drivers with early insights about their tire conditions. As part of the enhanced My BMW App experience, they wanted to offer customers more comprehensive information about aspects such as pressure loss or tire wear — and connect them with the right maintenance support at the right time.

The BMW team needed the right external support to showcase the power of a cloud-based infrastructure to fuel data-driven use cases that deliver superior user experiences — and build a robust and reliable system that can handle the roll-out across multiple geographies.

The Think-it team set out to build a Proof of Concept for scalable back-end cloud infrastructure for a tyre inflation pressure loss detection system - resulting in an end-to-end solution notifying customers of tyre conditions such as pressure loss and tire wear and connecting them with the right opportunity to buy their next set of wheels at the right time.

GOAL: Build a data-driven PoC for predictive tyre diagnosis and maintenance.

BMW Driver app

Connected driver app

BMW Driver app

 

We used the right architecture to provide data through the API to two core stakeholder groups, including:

  • Real-time, secure and flexible access to insights about tyre data, like pressure and wear
  • Implementation of technical building blocks on AWS cloud
  • Automation of the CI/CD and IaC with continuous monitoring and optimization
  • Distribution of relevant insights with end users via React native mobile-first UI

Due to using a cloud-based algorithm, any tyre inflation pressure losses could now be detected much earlier than by the vehicle's tyre pressure monitoring system.

This is enabled by a diagnostic function implemented with Artificial Intelligence (AI) in the BMW back end. Any loss of pressure is indicated to the driver by the My BMW App along with recommended action. Statistical information and AI methods could also be used to make predictions about the wear of the vehicle tyres, so that a forecast could be made about the remaining service life of the tyres until the recommended minimum tread depth is reached.

BMW High level architecture

High level architecture

BMW components architecture

AWS components level architecture

We collaborated with BMW to build the Proof-of-Concept providing intelligent services on top of their tyre business — improving customer experience through the My BMW app by notifying them ahead of time about tyre maintenance opportunities.

This approach boosted customer insights and satisfaction — while also empowering BMW’s technical teams with a powerful and efficient platform to easily retrieve, explore, and analyze relevant data in order to explore new innovation potential on top of the PoC.

BMW Table

As part of our deliverables, we created libraries and modules for other developer teams at BMW to use for their infrastructure. We also hosted Terraform training sessions for other developer teams at BMW to support them appropriately and use our solutions.

This PoC  has since been integrated as an aspect of the My BMW App with the press release at the time claiming that the digital tyre diagnosis by artificial intelligence was unique in the competitive environment and available for the first time in automotive engineering.

Throughout this project, the Think-it team additionally continued its work on the ever-evolving CDH. The next big step in this evolution would be the ability to exchange data sovereignly with other companies; as a first day member of Gaia-X, the European initiative for establishing sovereign data spaces, as well as a member of Catena-X, The BMW Group is an early mover in sovereign data spaces–something that would be ever-more important with regulation changes in the future.

BMW

Photo from press release. Source: BMW Group

STEP 4

Share sovereign data from BMW Group externally

BMW Group is a pioneer when it comes to sovereign data exchange and one of the main drivers behind the Catena-X data ecosystem as well as the Mobility Data Space.  The Mobility Data Space (MDS) is a European data space that launched in January 2022 with 80 datasets provided by around 200 companies that seek to explore first use cases in the mobility sector offering “access to real-time traffic data and sensitive mobility data beyond their secure exchange” linking data platforms to each other.

With Catena-X, the automotive industry of the future uses a trustworthy, collaborative, open and secure data ecosystem. All players are networked in end-to-end value chains, in which all partners are on an equal ground, have sovereign control over their data and no lock-in effects occur, which provides a sustainable solution for the digitalization of supply chains, especially for medium-sized and small companies, and supports the cooperation and collaboration of market participants and competitors.

BMW Diagram

Source: Catena-X

In order to bring Catena-X and the idea of sovereign data exchange to life, Think-it was tasked with the challenge to bring data space connectors to a production-grade quality so they could be leveraged by BMW Group and other participants to connect and exchange data with one another.

Concretely, Think-it focused on two challenges:

  • Enable sovereign data exchange at the BMW Group by integrating and operationalizing data space connectors within the existing environment
  • Scale sovereign data exchange on an industry level by enhancing the functionality and scalability of the Eclipse Dataspace Connector
BMW Diagram

High-level AWS example set-up

STEP 5

Create a single source of truth to manage data sharing

The BMW Group wanted to give business users insights into available data and conditions as well as create data pipelines and contract offers for data being exchanged with external organizations. They wanted a centralized place to enable non-technical users to exchange data with external organizations. There was no alternative tool available, and the current solution was human-driven manual work.

Think-it conducted research with stakeholders across 6 different departments within BMW Group to fully understand the business needs. We built a powerful Data Portal plug-in that creates a unified view of all data assets. Furthermore, as new use cases are created, this plugin is automatically deployed inside their familiar environment.

BMW governance UI

Business users inside BMW Group now have a single source of truth to manage data that is being provided and consumed without needing to access another new tool. The dashboard is aggregated across multiple legal entities with specified roles and accountabilities with metrics displayed in multiple levels of granularity.

Concretely, Think-it enabled:

  • A single source of truth across BMW Group.
  • An overview of data that is provided and consumed.
  • Aggregate multiple legal entities (e.g., BMW Holding, BMW A.G., etc.)
  • Foresight into contract agreements closure, so that legal and dealers can follow-up with new negotiations.
  • Defining resources in multiple EDC Connector instances, and technical and business operators to be able to see the status of their connectors.

BMW governance UI architecture

Enable sovereign data exchange at the BMW Group

After diving deep into the domain of data spaces we took ownership of setting up a data space connector (DSC) and subsequently integrating and operationalizing it within BMW Group’s existing cloud environment to production-level. This included:

  • Adding new functionalities such as usage policies and sovereignty 
  • Building pipelines to handle building, configuring and deploying the dataspace connector for different environments
  • Advising the internal data engineering team on how to create and share resources via the connector
  • Conducting design sprints and expert interviews to design and implement a seamless UI
  • Integrating the connector within BMW’s Data Portal to allow users to exchange data with external companies using their familiar user interfaces and data catalogues.
  • Becoming the subject matter experts within the organization on data space connectors and how to use them.
  • Providing documentation and trainings for internal capacity building and knowledge management

Scale sovereign data exchange on an industry level

Beyond enablement of the BMW Group as an organization, Think-it was also appointed on behalf of the Group to make contributions to the Eclipse Dataspace Connector open source effort as one of the core committers since inception of the project.

The Eclipse Dataspace Connector (EDC) has established itself as the current leading Data Space Connector in Europe. The goal of the EDC initiative is to create an efficient and functional data transfer component that implements the protocols of the IDS standard and pursues compatibility with the requirements of the GAIA-X project.

Think-it continues to be one of the most prolific contributors to the open source connector technology

Eclipse GitHub Commits

As one of the core committers and most prolific contributors, we own central areas of the EDC and are involved in defining the functionalities to enhance the usability of the EDC. We also drive decision-making and implementation of central elements and functionalities of the connector and work with the rest of the core committer group to brainstorm, identify and prioritize needs to inform and realize the product roadmap.

We’re the only SDK providers for Java, Node.js, and HTTP, with the Node.js SDK being used by the official Eclipse Data Dashboard

Since the inception of the open source project, Think-it’s team has been playing a major role in driving technological innovation forward and building the functionalities and extensions needed for data spaces such as Catena-X, Mobility Data Space, Market 4.0 and others to reach maturity and operate at scale.

Interested in how Think-it can drive your data innovation?

Reach out