In this video blog post I covered the serving layer step of building your Modern Data Warehouse in Azure. There are certainly some decisions to be made around how you want to structure your schema as you get it ready for presentation with whatever your business intelligence tool of choice, for this example I used Power BI, so I discuss some of the areas you should focus on:
What is your schema type? Snowflake or Star, or something else?
Where should you serve up the data? SQL Server, Synapse, ADLS, Databricks, or Something Else?
What are your Service level agreements for the business? What are your data processing times?
Can you save cost by using an option that’s less compute heavy?
Pragmatic Works is considered to be experts in the Microsoft Data Platform, both on-premises and in Azure. That being said, we often get asked many questions like, how can a certain technology benefit my company? One technology we are asked about a lot is Azure Databricks. This was released over a year ago in preview in the Azure portal and we’re starting to see some massive adoption by many companies, but not everyone is ready to delve into data science and deep analytics, so they haven’t had much exposure to what Databricks is and what it can do for their business.
There are some barriers preventing organizations from adopting data
science and machine learning which can be applied to solve many common
business challenges. Collaboration between data scientists, data
engineers, business analysts who are working with data (structured and
unstructured) from a multitude of sources is an example of one of those
In addition, there’s a complexity involved when you try to do things with these massive volumes of data. Then add in some cultural aspects, having multiple teams and using consultants, and with all these factors, how do you get that one common theme and common platform where everybody can work and be on the same page? Azure Databricks is one answer.
Here’s an overview of 3 common use cases that we’re beginning to see and how they can benefit your organization:
1. Recommendation Engines – Recommendation
Engines are becoming an integral part of applications and software
products as mobile apps and other advances in technology continue to
change the way users choose and utilize information. Most likely when
you’re shopping on any major retail site, they are going to make
recommendations to related products based on the products you’ve
selected or that you’re looking at.
2. Churn Analysis – Commonly known as
customer attrition; basically, it’s when we lose customers. Using
Databricks, there are ways to find out what some of the warning signs
are behind that. Think about it, if you get ways to correlate the data
that leads to a customer leaving your company, then you know that you
have a better chance to possibly save that customer.
And we all know that keeping a customer and giving them the service they need or the product they want is significantly less costly than having to acquire new customers.
3. Intrusion Detection – This is needed to
monitor networks or systems and activities for malicious activity or
policy violations and produce electronic reports to some kind of
dashboard or management station or wherever that is captured.
With the combination of streaming and batch technologies tightly
integrated with Databricks and the Azure Data Platform, we are getting
access to more real-time and static data correlations that are helping
to make faster decisions and try to avoid some of these intrusions.
Once we get triggered that there is a problem, we can shut if off very quickly or use automation options to do that as well.
Today I wanted to highlight some of the ways that you can utilize
Databricks to help your organization. If you have questions or would
like to break down some of these barriers to adopting machine learning
and data science for your business, we can help.
We are using all the Azure technologies and talking about them with
our customer all the time, as well as deploying real world workload