Super hot off the press- the Early Release of my book Mastering Azure Analytics: Architecting in the cloud with Azure Data Lake, HDInsight and Spark is now available! You can get the first two chapters right away and new chapters are right around corner. Here's what I cover: Microsoft Azure has over 20 platform-as-a-service (PaaS) offerings that can act in support of a big data analytics solution. So which one is right for your project? This practical book helps you understand the breadth of Azure services by organizing them into a reference framework you can use when crafting your own... Continue reading

If you are getting an exception when running an Azure Data Lakes Analytics U-SQL query against a CSV source file, you may get an inscrutable error like the following: ERROR VertexFailedFast. Vertex failure triggered quick job abort. Vertex failed: SV1_Extract[0][0] with error: Vertex user code error. DESCRIPTION Vertex failed with a fail-fast error RESOLUTION DETAILS Vertex SV1_Extract[0][0].v2 {43B20D9E-E63F-48AF-8E9A-FFFAE288FCB8} failed Error: Vertex user code error exitcode=CsExitCode_StillActive Errorsnippet=An error occurred while processing adl://adlmvp.azuredatalakestore.net/Zoiner/ExtentAligned/On_Time_On_Time_Performance_2014_1_OneColumn.csv In my experience so far, the root causes you can check for that trigger this error include: The columns in your U-SQL query don't line up with the number... Continue reading

I commonly get the question, "what's the difference between Event Hubs and IoT Hubs"? Both are basically positioned at the edge of a cloud analytics solution, and after all are responsible for storing ingested telemetry as messages or events. While you can get very deep answering this question, there are three high-level fundamental differences that can help you in making the right selection. #1 Messaging Directions Event Hubs provides what I like to call a "multi-consumer" queue that defers state management responsibility (e.g., progress reading thru the queue) to the consumer. This type of queue is great for ingesting huge... Continue reading

Last year, Microsoft released the Visio template you can use to create really cool looking 3D perspective illustrations of Azure topologies. These templates are visually interesting and also super helpful in getting high-level points across to an introductory audience. To that aim, I felt there was a illustration missing in the official blueprints provided by Microsoft and so I created my own for IoT. It explores some of the options for creating a lambda architecture on Azure that can be used to process the telemetry produced by IoT devices. You can download a PDF of my template here: Download IoT... Continue reading

If you're migrating from HDInsight on Windows to HDInsight on Linux, you are probably upgrading to Spark 1.6 from Spark 1.3 and shifting from Zeppelin to Jupyter. This turns out to have some pretty fundamental changes. If you, like me, get cross-eyed trying navigate the Scala docs, I'll be creating a few posts about key changes with examples of the new syntax. In this post, we'll examine how to save a DataFrame to permanent table, that could be queryable via Hive or via external tools. In the previous version of Spark, you could have simply called saveAsTable against a DataFrame... Continue reading

If you're migrating from HDInsight on Windows to HDInsight on Linux, you are probably upgrading to Spark 1.6 from Spark 1.3 and shifting from Zeppelin to Jupyter. This turns out to have some pretty fundamental changes. If you, like me, get cross-eyed trying navigate the Scala docs, I'll be creating a few posts about key changes with examples of the new syntax. In this post, we'll begin at the beginning- loading a CSV text file. In the previous version of HDI you would have loaded a text file using the following syntax: val textLines = sparkContext.textFile("wasb:///subfolder/myfile.csv") In Spark 1.6 the... Continue reading

When most folks think of applying Azure Search, they are thinking about the traditional text search scenario such as find documents that contain the text "run" and it will match (due to its support for Natural Language Processing and linguistic stemming) documents that contain "run" and "running". While it's true Azure search does a great job supporting full text search, I encourage thinking about its application with a broader lens- as the external index to another data store. For example, Azure Table Storage has long been bemoaned as not having support for secondary indexes (it only supports a single composite... Continue reading

If you're coding the bleeding edge, you may find yourself in situation where the version of the App Services SDK you want has not been officially released yet, but it's there on GitHub. To give you an example, I needed to get at the still burgeoning support for EventProcessorHost in Web Jobs via the Microsoft.Azure.WebJobs.ServiceBus assembly. How do you get these packages into your solution? The process to getting at the "nightly" builds turned out to be surprisingly simple. Within Visual Studio, right click on your solution and choose Manage NuGet Packages for Solution. In the top, right corner of... Continue reading

While it might be obvious to some, I still frequently find clients confused about the type of messaging Azure Notification Hubs provides. In this post, I thought I would tackle that by clarifying what it does not provide. First of all, like the title suggests, Azure Notification Hubs does not itself provide SMS text messaging to mobile devices. What it does provide is in-app, push notifications. In other words, your recipients need to have your mobile app installed on their device before they can receive notifications. There is often confusion on this with because the SMS and push notifications are... Continue reading

For those of you familiar with launching HDInsight with Spark clusters on Windows (which was the only option), you may be surprised to find that this is no longer an option. Here is what you used to see: With the latest update linux is the new black, and its your only option for running Spark with HDInsight. To be clear, various Hadoop options on Windows are still available with HDInsight, just not Spark. This change comes with it some other ramifications if you are coming from the HDInsight + Spark on Windows. The most notable is the notebook (sorry, I... Continue reading

A near real-time message processing topology I often see considered is to use Event Hubs as the point of message ingest, followed by Stream Analytics to do some form of filtering or transformation of the message and ultimately delivering the message to a Service Bus Topic for consumption by consumers who hold a subscription on that topic. However, there is a gotcha here. Most solutions using Service Bus Topics build Subscriptions using SQL Filters against the message properties to decide if the message is relevant to the Subscription (for example, it might indicate high priority messages for which the consumer... Continue reading

In 2015 Microsoft acquired Revolution Analytics, and today this influence can be seen in the rapidly expanding set of options for running and hosting R based machine learning models. Each option appears fully integrated into an existing product or service, so at times can be difficult to get a lay of the land in order to make the right choice. In this post, we provide that overview. Azure Machine Learning Azure Machine Learning provides a PaaS experience for designing, training and operationalizing machine learning models. It can run R scripts (and supports pulling in dependencies). The key benefit is it's... Continue reading

Azure Functions abstract away a lot of the plumbing required to write cloud-hosted code that reacts to an event (e.g. an HTTP request, a message in a Queue, etc.). Because Event Hubs are different type of queue, I was intrigued to do a little deeper dive into using Azure Functions as a consumer-- particularly since on this surface this seems like a great fit. First, ensure you have your Event Hub setup and that you have something pushing messages to it (I used messages from chat client, but that's a story for another day). Second, setup your Azure Function App... Continue reading

Continuing with our theme of working with your data sourced from Google Analytics, in this post we look at how to get your Google Analytics data loaded into R so that you can use it to drive machine learning and prediction. What we want to show is how you could predict the user’s browser given his or her region by building a simple One Rule model that consumes the the nthminute, region and browser dimensions, using hits by minute as the metric. First a quick note, on why we chose this classifier and data set. We felt this provides a... Continue reading

In this post we consider how you can get data out from Google Analytics by examining the 5 Things you probably didn’t know about the Embed API. This article will highlight the interesting aspects from the sample dashboard, which you can access by going here. For the basics of getting your OAuth2 credentials going and a quick start on the Embed API I highly recommend you visit the Embed API Demos Site first. If you want to download the sample ASP.NET project, you can get it from the GitHub repo here. The sample shows you how you can allow users... Continue reading

Most people just getting started with Google Analytics have the perception that the flow of data in Google Analytics looks like this (in other words tracking beacons in a website, reports viewed from the GA Web Interface): When in reality, the real capabilities for loading data into and extracting data from Google Analytics are significantly more comprehensive and could be summarized from a stratospheric level like this: In short, the data-in story revolves around inputting tracking beacon data (as might be collected from a website, an app or even a device) and loading custom data (typically used to enrich the... Continue reading

Google Universal Analytics has enabled you to "tag" URL's with marketing campaign details for a while now. This enables you to craft a URL you can share in an email, on social media or in a blog that provides you additional context on how users arrived at your site, beyond just the referrer. The challenge is the typical format is quite long, for example: http://yourcloudbuddy.com/?utm_source=rocks&utm_medium=banner&utm_campaign=dem1401 Obviously, for twitter this length of URL is a non-starter (it's over 50% of your 140 character limit). The typical solution is to then run this URL through a URL shortner. This is both cumbersome... Continue reading

If you read my post about using video for a background or hero in your website design, you may have been wondering how to track that. Most web developers are accustom to using Google Analytics for tracking page views, but did you know it is also a great tool for tracking video related events? Since Universal Analytics launched from Beta earlier this year, it's an even more compelling time to be asking "what else can I track?" with Google's Universal Analytics. So, let's think about the video scenario for a moment. Video Analytics enables you to answer questions like How... Continue reading

In my previous post I introduced a practical scenario where one might leverage Azure Media Services as part of the design aesthetic in building a modern website. By being practical, the intent is to show you don't need have massive video requirements in order to benefit. By understanding the pricing, however, you can see understand why I say it is for the "Little Guys". Let's price out our background video scenario, for the all-up costs, per month. I'll be using the published retail rates here, so if you have some form of commitment plan (and I'm glad you do) you... Continue reading

Not everyone needs a full-blown content management system (CMS) or intends to deliver a library of video content to rival that of Netflix or Amazon Instant Video. Sometimes, you just need some video on your website. However, when looking at Azure Media Services, many tend to react that its only intended for this largeer use case scenarios. In fact, the documentation for Azure Media Services is in part to blame for this, since it tends to immediately jump into using the programmatic API's to ingest, encode and deliver your content-- as if everyone always wanted to do that. I propose... Continue reading

Neo4j is a graph database, which places at least as much (and often more) emphasis on the relationship between data elements as the properties and values associated with each datum. Each data element is referred to as a node, that can have a collection of key-value pairs associated with it. Relationships, also referred to as edges, have a collection of key-value pairs. If you're familiar with relational database concepts, then conceptually a graph database tacks on queryable key-value pairs to your foreign key constraints, so you can query not just by the values of fields in a row, but by... Continue reading

Many of the databases in this arc are quickly created from virtual machine VHD images from the VM Depot. This site, currently still in preview, is created by a subsidiary of Microsoft-- Microsoft Open Technologies, but the images themselves are, largely, provided by 3rd parties. You can navigate directly to the VM Depot by going to http://vmdepot.msopentech.com/. This provides you with a search interface into the VM catalog, where you can get a good description of the image and the publisher. This site also provides you with the deployment script you can tweak to run using the Azure Command Line... Continue reading

In the spirit of the excellent book Seven Databases in Seven Weeks, I will be kicking off a new arc on this blog that looks at different relational, non-relational and graph databases and what I learned getting started with them when running on Windows Azure, and generally leaving you ready to run the samples in the book. You can expect some examples of their use and the tools I found useful when using each of them. I might even throw in a surprise database at the end! Continue reading

The holy grail of load testing your website just may well be have virtual users navigating your site just like your real users are, except where you control the number of users. The reality of this, especially for a large scale site, is that instead of saying "holy grail" you might be saying "holy expensive!". In this multipost ar we try to paint the right perspective on load testing simulating real user patterns (as opposed to load testing invidual requests as we showed previously) as well as how to get the most bang for your buck. For our customers over... Continue reading

I just launched a new website that let's you explore how to apply the latest incarnataion of Google Analytics- Universal Analytics (beta). The website provides you with a sample Single Page Application highlighting common interactions you might actually want to track by showing you them live and then letting you drill into the tracking implementation and best practices by means of code snippets and video. This site will continue to get new and interesting Universal Analytics features, so check back periodically to learn new things! For those curious, the site is built in AngularJS and derived from Google's Phone Catalog... Continue reading