Overview of Azure Databricks

I’d like to tell you about Azure Databricks. If you don’t know what that is, Azure Databricks provides an end-to-end, managed Apache Spark platform optimized for the cloud. It’s a fast, easy and collaborative analytics platform designed to help bridge the gap between data scientists, data engineers and business decision-makers using the power of Databricks on Azure.

Azure Databricks uses Microsoft Azure Active Directory as its security infrastructure and it’s optimized for ease of use, as well as ease of deployment within Azure. It features optimized connectors to Azure storage platforms (e.g. Data Lake and Blob Storage) for the fastest possible data access, and one-click management directly from the Azure console.

Some key features are:

Auto-scaling – This feature makes scaling much quicker and allows you to scale up or down as you need.

Auto-terminator – Helps you control the costs of your compute time, as well as assist you in preventing cost overruns (a concern for many cloud users).

Notebook Platform – The Notebook Platform supports standard languages (SQL, Python and R for example) and it builds a whole discussion environment around those platforms, enhancing collaboration amongst teams.

Here are some simple steps to get you started:

  • First, you’re going to prepare your data by ingesting it from your Azure storage platform, which has native support with Azure Databricks.
  • Next, you’re going to do any kind of transformation you need on your ingested data and store it in a Data Warehouse.
  • From here, you’ll want to start to perform analytics on your data. These platforms are built for lots of data and you’ll have the capability to explore large data sets in real time, as well as the ability to explore very quickly.
  • Lastly, you’re going to display the data. Databricks has native support for tools like Power BI to build your dashboards and analytics models.

So, Azure Databricks provides an end-to-end data solution. You can quickly spin up a cluster or do advanced analytics with this powerful platform. And with it, you can create and monitor robust pipelines that will help you dig deep and better understand your data, allowing you to make better business decisions.

3 Reasons Why You Should Move Your Business to the Cloud

Cyber security is on everyone’s mind these days and it can be a challenge for many organizations. If this sounds like you and you haven’t moved to the cloud, it’s something you should think about. I’d like to tell you why you should move your business to the cloud and why it could be more secure there.

1.  When you’re in the cloud business, having a secure cloud drives more business. That’s why cloud companies are willing to invest more to hire the best and brightest. So, the top security people in the world are going to the top cloud companies in the world.

2.  When moving to the cloud, typically, the customer only has to focus on one aspect of security because the rest is already taken care of, so by default, secure. You’d have to intentionally unlock something to make yourself less secure.

3.  Regulatory and certification requirements are more easily satisfied. With a foundation in place that’s already secure and certified, it allows you to focus on your app or infrastructure or whatever requirements you need to satisfy those regulatory compliance issues.

So, make this your year to move to the cloud and take some of the cyber security challenges off your mind.

Overview of Power BI Embedded

Everyone is familiar with Power BI Desktop, Cloud and On-Prem. But not as many are familiar with Power BI Embedded. So, what is it? Power BI embedded allows your company to embed the dashboards and reports in your in-house developed applications, and you only need one Power BI account to be able to have a Power BI embedded environment.

This Azure service is separate from Power BI Premium or Pro and is built for compute, rather than per user, as with other Power BI iterations. The design is to focus on your applications and your customer, instead of the management and maintenance of things.

You have options when setting up your Azure tenant. You can use your existing tenant ID, create a new application for the tenant or a tenant for a specific customer. There are 3 straightforward steps to get you up and running:

1.  Set up your Azure Power BI Embedded environment within Azure. Then set up your tenets, user requirements and workspaces.

2.  Then you’re going to embed your content by going to your backend and set up your application and connect to Azure through the REST API that Azure provides. This is all secure and encrypted traffic going over SSL. If you’re using the authentication when you’re displaying your reports and dashboards, then you’re doing this through your backend application authentication system, rather than the Azure application authentication system.

3.  Lastly, you’re going to release your reports and dashboards to production. You’ll need to decide what compute requirements you need and then set up your tiered pricing, pick your plan and you’re ready to go.

Why Your Infrastructure Belongs in the Cloud

You haven’t moved to the cloud yet? In this Azure Every Day installment, I’d like to tell you the top 5 reasons why you may want to move your infrastructure to the cloud.

1. Cost – Many people can take advantage of operational cost savings by not having to invest in a bunch of hardware that sits unused. In the cloud, you only pay for what you use.

2.  Business Continuity – With the cloud, you have better, more guaranteed up-time without having to worry about in-house appliances or certain infrastructures or servers. You also get easier administration. The cloud locations in Azure are set up so you can easily maintain and migrate your systems. And there’s no need for a second data center, giving you high availability, as well as more cost savings.

3.  Agility – You don’t have to spend money having something running all the time. It’s easy to spin up and spin down as you need it. You also have the ability to scale at an exponential rate. You can start small, but quickly build in traffic or performance capabilities or whatever you need.

4.  Management and Maintenance – You can drastically reduce the time needed to maintain and manage your environment, as well as have one central area for monitoring and maintaining your systems. You’ll save time wasted on running back ups and maintaining servers.

5.  Improved Security – Cloud providers have it in their best interest to be secure. There are over 300,000 open security jobs in the US alone. Where do you think those people want to work when there’s top quality companies paying top dollar? You guessed it – cloud companies.

Most Important Components of Azure Data Factory

Are you new to Azure and not know what Azure Data Factory is? Azure Data Factory is Microsoft’s cloud version of an ETL or ELT tool that helps you get your data from one place to another and to transform it. Today, I’d like to tell you about the high-level components within Azure Data Factory. These components pull together a data factory that helps your data flow from its source and have an ultimate end-product for consumption.

  • Pipeline – A pipeline is a logical grouping of activities that performs a grouping of work. An example of an activity may be: you’re copying on-premise data from one data source to the cloud (Azure Data Lake for instance), you then want to run it through an HDI Hadoop cluster for further processing and analysis and put it into a reporting area. The components will be contained inside the pipeline and would be chained together to create a sequence of events, depending upon your specific requirement.
  • Linked Service – This is very similar to the concept of a connection string in SQL Server, where you’re saying what is the source and destination of your data.
  • Trigger – A trigger is a unit of processing that determines when a pipeline needs to be run. These can be scheduled or set off (triggered) by a different event.
  • Parameter – Essentially, the information you can store inside a pipeline that will pass in an argument when you need to fill in what that dataset or linked service is.
  • Control Flow – The control flow in a data factory is what’s orchestrating how the pipeline is going to be sequenced. This includes activities you’ll be performing with those pipelines, such as sequencing, branching and looping.

What is Internet of Things (IoT) and Why It Matters to IT

The Internet of Things (IoT) has become a growing topic both inside and outside of the workplace. It has the ability to change how we live and how we work. Many are already on board with IoT, and global IoT revenues are projected to reach over a trillion dollars by 2020. If you’re not there yet, I’d like to talk today about what IoT is and how it’s being used.

Internet of Things, or IoT, is defined as a device that used to be a stand-alone device, but is now connected to the internet. Consumer based devices include Google Home, Alexa, Smart Watches, Fitbits and home thermostats. These products are already changing the way the owners of these devices live.

From a business standpoint, with the help of services like Azure IoT Hub, this gets much bigger, with a much larger impact on how people work. Large engine monitoring devices for trains and planes, for example, have millions of components that are being monitored all the time, therefore, showing real-time statistics about what is happening on those devices.

Chevron is using Microsoft Azure IoT Hub in the backend as they build out their IoT infrastructure for monitoring their oil, and deep well, drilling devices. John Deer is mounting IoT devices on their equipment that tracks things such as where, how far apart or how deep seeds are being planted, so farmers can get planting information right from these devices.

Yes, IoT is a big deal and it’s helping manufacturing and other companies, as well as consumers alike, by expanding the capabilities of things that are, or can be, connected to the internet.