Follow us on @SQLServer and Microsoft SQL Server on Facebook to stay-up-to date on the latest news and announcements about Microsoft’s participation at PASS Summit 2014
Register to attend PASS Summit 2014
Image may be NSFW.Clik here to view.

Follow us on @SQLServer and Microsoft SQL Server on Facebook to stay-up-to date on the latest news and announcements about Microsoft’s participation at PASS Summit 2014
Register to attend PASS Summit 2014
Image may be NSFW.We are happy to announce that the ODL 6.6.0 is released and available on nuget along with the source code oncodeplex (please read the git history for the v6.6.0 code info and all previous version). Detailed release notes are listed below.
Add Instance Annotation on ODataProperty and ODataComplexValue
Call to Action
You and your team are highly welcomed to try out this new version if you are interested in the new features and fixes above. For any feature request, issue or idea please feel free to reach out to us atodatafeedback@microsoft.com.
Image may be NSFW.La richesse de la plateforme décisionnelle de Microsoft (MSBI) permet de répondre à l’ensemble des besoins d’analyse, de reporting, et de tableaux de bord formulés par les directions métiers.
Certains facteurs tels que la maturité et l’autonomie des utilisateurs, la variété des données, permettent d’innover dans la mise en œuvre des solutions décisionnelles
Nous allons à travers cet article décrire les composantes de la plateforme MBSI, et illustrer par des exemples réels de mise en œuvre.
...(read more)Image may be NSFW.This post provides walkthroughs that discuss the use of SQL Server Database Projects (.sqlproj files) with Team Foundation Build in Team Foundation Server 2013. The scenarios addressed are:
This walkthrough covers the process of configuring a scheduled build using Team Foundation Build in TFS 2013 and Visual Studio 2013.
Once Team Foundation Build is configured to build your database project, Team Foundation Build can also execute your database unit tests after every build. By default, Team Foundation Build will attempt to execute unit test projects it finds in the build output, but some additional configuration is necessary to get database unit tests working correctly. This walkthrough covers the process of configuring unit test execution using Team Foundation Build in TFS 2013 and Visual Studio 2013.
Image may be NSFW. Clik here to view. ![]() |
The Team Foundation Server Administration Console’s Build Configuration tab displays which Windows user account the Build Service executes under. |
Shared Sub New()
Environment.SetEnvironmentVariable("VisualStudioVersion", "12.0")
End Sub
static SqlDatabaseSetup()
{
Environment.SetEnvironmentVariable("VisualStudioVersion", "12.0");
}
%ProgramFiles%\Microsoft SQL Server\120\Tools\Binn\SqlLocalDB.exe
%ProgramFiles%\Microsoft SQL Server\110\Tools\Binn\SqlLocalDB.exe
"%ProgramFiles%\Microsoft SQL Server\120\Tools\Binn\SqlLocalDB.exe" create ProjectsV12 -s
$(TF_BUILD_BINARIESDIRECTORY)\SetupLocalDB.cmd
$(TF_BUILD_BINARIESDIRECTORY)\ContosoApp\SetupLocalDB.cmd
Publishing a database project will update the schema of a target database to match the schema defined in the database project. In some application lifecycle management processes, it is desirable to automatically publish a database project as part of a daily or continuous integration build. Automatic publishing can be accomplished in multiple ways using Team Foundation Build. This document describes prerequisites applicable to all scenarios, and several common methods for achieving automatic publishing (in ascending order of difficulty).
Team Foundation Build will need to connect to an instance of SQL Server in order to deploy your database project. Decide what credentials Team Foundation Build will use to connect to SQL Server. If you already have a login that uses SQL Server authentication and has sufficient permissions to deploy the database project, then you can re-use those credentials. Documentation on the minimum permissions required to create or deploy a database project can be found at http://msdn.microsoft.com/en-us/library/jj889462(v=vs.103).aspx#DatabaseCreationAndDeploymentPermissions. Note that the permissions required depend on the options selected and features used in the database project. For example, a database project that deploys a SQLCLR assembly to the target SQL Server requires a login that is a member of sysadmin role.
Otherwise, choose whether to create a SQL Server authentication login or a Windows authentication login for the Windows account under which Team Foundation Build executes, and create a login by following the instructions at http://technet.microsoft.com/en-us/library/aa337562.aspx
Notes:
Image may be NSFW. Clik here to view. ![]() |
The Team Foundation Server Administration Console’s Build Configuration tab displays which Windows user account the Build Service executes under. |
For any deployment option described in this section, you will need to specify the target database for deployment. You can either tell the system where to deploy the new database schema by using a Publish Profile (a publish.xml file) or, in some cases, you can use individual parameters to specify the connection information. A Publish Profile is recommended because it normally simplifies re-use. If you want to use a Publish Profile (a publish.xml file), follow these steps to create one:
The default build processes in Team Foundation Build use a component named MSBuild to compile SQL Server Database Projects. MSBuild has an option that, when enabled, tells MSBuild to publish the SQL Server Database Project after a successful build.
The approach outlined below will configure Team Foundation Build to always pass arguments to MSBuild that tell it to publish the database project. This approach is straightforward, but it is not appropriate in some situations:
If this option is appropriate for your build, follow these steps to pass the appropriate arguments to MSBuild.
/t:Build;Publish /p:SqlPublishProfilePath=your_file_name.publish.xml
Notes:
Image may be NSFW. Clik here to view. ![]() |
The Build process parameters grid in the Build Definition tab contains the MSBuild arguments field under Build > Advanced. In this case, a file called myserverprofile.publish.xml contains the connection information for the target database. |
SQL Server Data Tools includes a command-line utility named SqlPackage.exe that can be used to publish a database, and TFS 2013’s default build process includes a Post-build script field that can be used to execute SqlPackage.exe.
The approach outlined below will configure Team Foundation Build to call SqlPackage.exe after building. This approach is straightforward, but it is not appropriate in some situations:
If this option is appropriate for your build, follow these steps to call SqlPackage.exe after a build.
C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\Extensions\Microsoft\SQLDB\DAC\120\SqlPackage.exe
/a:Publish /pr:$(TF_BUILD_BINARIESDIRECTORY)\your_file_name.publish.xml /sf:$(TF_BUILD_BINARIESDIRECTORY)\your_dacpac_name.dacpac
/a:Publish /pr:$(TF_BUILD_BINARIESDIRECTORY)\YourProject\your_file_name.publish.xml /sf:$(TF_BUILD_BINARIESDIRECTORY)\YourProject\your_dacpac_name.dacpac
/a:Publish /tcs:"Data Source=YourSQLServer;Integrated Security=true;Initial Catalog=YourTargetDatabase;Pooling=false" /sf:$(TF_BUILD_BINARIESDIRECTORY)\your_dacpac_name.dacpac
/a:Publish /tcs:"Data Source=YourSQLServer;Integrated Security=true;Initial Catalog=YourTargetDatabase;Pooling=false" /sf:$(TF_BUILD_BINARIESDIRECTORY)\YourProject\your_dacpac_name.dacpac
Note: Additional SqlPackage.exe arguments are available. See documentation here: http://msdn.microsoft.com/en-us/library/hh550080(v=vs.103).aspx
Image may be NSFW. Clik here to view. ![]() |
The Build process parameters grid contains the Post-build script path and arguments fields under Build > Advanced. In this case, a file called myserverprofile.publish.xml contains the connection information for the target database. |
SQL Server Data Tools includes a command-line utility named SqlPackage.exe that can be used to publish a database, and TFS 2013’s default build process includes a Post-build script field that can be used to execute a batch file.
The approach outlined below will configure Team Foundation Build to call a batch file after building, and the batch file will in turn call SqlPackage.exe. This approach is not appropriate in some situations:
If this option is appropriate for your build, follow these steps:
C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\Extensions\Microsoft\SQLDB\DAC\120\SqlPackage.exe
"C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\Extensions\Microsoft\SQLDB\DAC\120\SqlPackage.exe" /a:Publish /pr:%TF_BUILD_BINARIESDIRECTORY%\your_file_name.publish.xml /sf:%TF_BUILD_BINARIESDIRECTORY%\your_dacpac_name.dacpac
"C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\Extensions\Microsoft\SQLDB\DAC\120\SqlPackage.exe" /a:Publish /pr:%TF_BUILD_BINARIESDIRECTORY%\YourProject\your_file_name.publish.xml /sf:%TF_BUILD_BINARIESDIRECTORY%\YourProject\your_dacpac_name.dacpac
"C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\Extensions\Microsoft\SQLDB\DAC\120\SqlPackage.exe" /a:Publish /tcs:"Data Source=YourSQLServer;Integrated Security=true;Initial Catalog=YourTargetDatabase;Pooling=false" /sf:%TF_BUILD_BINARIESDIRECTORY%\your_dacpac_name.dacpac
"C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\Extensions\Microsoft\SQLDB\DAC\120\SqlPackage.exe" /a:Publish /tcs:"Data Source=YourSQLServer;Integrated Security=true;Initial Catalog=YourTargetDatabase;Pooling=false" /sf:%TF_BUILD_BINARIESDIRECTORY%\YourProject\your_dacpac_name.dacpac
$(TF_BUILD_BINARIESDIRECTORY)\your_name_here.cmd
$(TF_BUILD_BINARIESDIRECTORY)\YourProject\your_name_here.cmd
Note: Additional SqlPackage.exe arguments are available. See documentation here: http://msdn.microsoft.com/en-us/library/hh550080(v=vs.103).aspx
Image may be NSFW. Clik here to view. ![]() |
The Build process parameters grid contains the Post-build script path under Build > Advanced. In this case, a file called your_name_here.cmd calls SqlPackage.exe in order to publish the SQL Server Database Project. |
In Team Foundation Build, build processes are Windows Workflow (XAML) files. The Windows Workflow file used for your build can be customized to call SqlPackage.exe. This option provides the greatest flexibility at the cost of greater complexity.
C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\Extensions\Microsoft\SQLDB\DAC\120\SqlPackage.exe
"C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\Extensions\Microsoft\SQLDB\DAC\120\SqlPackage.exe"
$(TF_BUILD_BINARIESDIRECTORY)\Release\YourProject\your_dacpac_name.dacpac
Copying file from "obj\Release\DBProjectUnitTests.dll" to "C:\Builds\2\SomeTeamProject\SampleBuildDefinition\bin\Release\DBProject\DBProjectUnitTests.dll".
"/a:Publish /pr:$(TF_BUILD_BINARIESDIRECTORY)\your_file_name.publish.xml /sf: $(TF_BUILD_BINARIESDIRECTORY)\your_dacpac_name.dacpac"
"/a:Publish /tcs:""Data Source=YourSQLServer;Integrated Security=true;Initial Catalog=YourTargetDatabase;Pooling=false"" /sf:$(TF_BUILD_BINARIESDIRECTORY)\your_dacpac_name.dacpac"
Deployment contributors are custom .NET classes that can modify or interact with the publishing of a database schema. The SQL Server Database Project deployment engine will search for deployment contributors in the same directory as the deployment code. This section details approaches to using deployment contributors with Team Foundation Build.
Some deployment contributors almost never change after having been written. In this case, manually placing the deployment contributor assemblies into the Program Files directory on the build server is a good option.
C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\Extensions\Microsoft\SQLDB\DAC\120\SqlPackage.exe
Suppose that you have a solution that contains both a SQL Server Database Project and a separate project that contains a deployment contributor, and you want Team Foundation Build to compile both, and then to publish the database project using the freshly-built deployment contributor. In this case it is probably not possible for Team Foundation Build to copy the deployment contributor into the directory where deployment contributors normally reside, because most organizations do not allow Team Foundation Build to execute under an account with Administrator permissions.
A workaround for this scenario is to copy both the deployment contributor and the SqlPackage.exe code into a temporary directory, and then have Team Foundation Build call that copy of SqlPackage.exe. This works because the code that performs SQL Server Database Project deployment loads extensions from its own directory – the directory in which it is executing.
The steps below will walk you through creating a batch file to execute SqlPackage.exe from a temporary directory.
C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\Extensions\Microsoft\SQLDB\DAC\120\SqlPackage.exe
md %TF_BUILD_BUILDDIRECTORY%\%TF_BUILD_BUILDNUMBER%\DEPLOY
xcopy /s /h /r /y /i %TF_BUILD_BINARIESDIRECTORY% %TF_BUILD_BUILDDIRECTORY%\%TF_BUILD_BUILDNUMBER%\DEPLOY
xcopy /s /h /r /y /i " C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\Extensions\Microsoft\SQLDB\DAC\120" %TF_BUILD_BUILDDIRECTORY%\%TF_BUILD_BUILDNUMBER%\DEPLOY
%TF_BUILD_BUILDDIRECTORY%\%TF_BUILD_BUILDNUMBER%\DEPLOY\sqlpackage.exe /a:Script /pr:%TF_BUILD_BINARIESDIRECTORY%\your_file_name.publish.xml /sf:%TF_BUILD_BINARIESDIRECTORY%\your_dacpac_name.dacpac /OutputPath:%TF_BUILD_DROPLOCATION%\DeploymentScript.sql /p:AdditionalDeploymentContributors=your_contributor_id /p:AdditionalDeploymentContributorArguments=your_contributor_arg=value
%TF_BUILD_BUILDDIRECTORY%\%TF_BUILD_BUILDNUMBER%\DEPLOY\sqlpackage.exe /a:Publish /pr:%TF_BUILD_BINARIESDIRECTORY%\your_file_name.publish.xml /sf:%TF_BUILD_BINARIESDIRECTORY%\your_dacpac_name.dacpac /p:AdditionalDeploymentContributors=your_contributor_id /p:AdditionalDeploymentContributorArguments=your_contributor_arg=value
rmdir /S /Q %TF_BUILD_BUILDDIRECTORY%\%TF_BUILD_BUILDNUMBER%
/pr:%TF_BUILD_BINARIESDIRECTORY%\YourProject\your_file_name.publish.xml /sf:%TF_BUILD_BINARIESDIRECTORY%\YourProject\your_dacpac_name.dacpac
$(TF_BUILD_BINARIESDIRECTORY)\your_name_here.cmd
$(TF_BUILD_BINARIESDIRECTORY)\YourProject\your_name_here.cmd
Note: Additional SqlPackage.exe arguments are available. See documentation here: http://msdn.microsoft.com/en-us/library/hh550080(v=vs.103).aspx
Microsoft provides a hosted Team Foundation Build service as part of Visual Studio Online. In general, configuration in Visual Studio Online is the same as an on-premise Team Foundation Server, but the following special considerations apply when using the Visual Studio Online build service:
C:\Program Files\Microsoft SQL Server\110\DAC\bin
Today we are releasing Visual Studio 2013 Update 3. You can read the release announcements by Soma and Brian Harry. This update includes the Entity Framework 6.1.1 (EF6.1.1) runtime and tooling.
EF6.1.1 was previously released out-of-band about 6 weeks ago, so you may have already installed the update to Entity Framework. If not, it will be automatically installed when you install Visual Studio 2013 Update 3.
You can download Visual Studio 2013 Update 3.
Note that installing the update will not affect the version of the Entity Framework runtime included in existing projects. We recommend using NuGet to update to the latest version of the runtime. For detailed information on how to upgrade, see Updating a Package in the NuGet documentation.
You can see a list of the fixes/changes included in EF6.1.1 on our CodePlex site.
In particular, we’d like to call out the following two fixes to issues that a number of people have encountered:
You’ve heard from John Platt a few times on this site, including his introductory post on What is Machine Learning? and this piece where he talks about Twenty Years of Machine Learning at Microsoft. Larry Larsen from Channel 9 recently got the opportunity to have an in-depth conversation with him. See what John has to say about AI, Machine Learning and his focus on Deep Neural Networks.
You can catch the full interview with John here, and here's a short clip of him demystifying Deep Learning. The full article is posted to the “Inside Microsoft Research” blog here.
ML Blog Team
Image may be NSFW.This post contains the slide deck from a recent presentation on how to integrate Visual Studio Database Projects with Visual Studio Team Foundation Server. We have also recently published an accompanying blog with extended content, details and scenarios: http://blogs.msdn.com/b/ssdt/archive/2014/07/24/sql-server-database-projects-and-team-foundation-build.aspx.
(Note: A link to the complete slide deck can be found at the bottom of the post)
Image may be NSFW.
Clik here to view.
Image may be NSFW.
Clik here to view.
Image may be NSFW.
Clik here to view.
Image may be NSFW.
Clik here to view.
SQL Server tooling inside Visual Studio is a broad topic covering plenty of great functionality. In this presentation we will focus on integrating database projects with TFS and using Database Unit Testing. There are plenty of existing presentations covering the other parts of the SQL tooling experience. |
Image may be NSFW.
Clik here to view.
To understand the benefits of using project-based development it helps to examine common development practices and explain the benefits and drawbacks of each one. Let’s start with simple, connected based development experience using SSMS with database changes occurring directly against the database. In this case the database is the source of truth and periodic backups are used to restore the DB if there’s a mistake / an error occurs. This is popular due to the speed of development (make a change and it is instantly applied), but has a number of drawbacks. For instance it’s hard to understand what changes have occurred, why they were applied, or even what the overall structure is. One developer might have removed the View or Table a stored procedure calls but this won’t be noticed until you call into it. There’s also a lack of separation between development and deployment, which impacts the ability to migrate the DB to other servers among other things. So this has some benefits and lots of drawbacks. |
Image may be NSFW.
Clik here to view.
Moving to a more advanced, and likely more common scenario in many development environments, a lot of developers continue to use connected development in SSMS but with the use of a set of scripts that represent the database creation and modification steps needed to get it into its current state. In this example, it’s common to define the initial state (v1) of the DB, then have a set of ALTER statements that move that to v2, another for v3, etc. This supports developing against one DB and then deploying changes to a staging server before deployment. It’s a definite step up from the more basic version of connected development and many teams use this today. It does have drawbacks too:
These drawbacks are always going to be there when using pure connected development. It’s a reason we feel there are strong benefits to using an offline, project-based development pattern just like in other languages. |
Image may be NSFW.
Clik here to view.
Here’s what a simple project-based development scenario looks like in Visual Studio, suitable for 3-5 developers. The previous set of incremental deployment scripts on a file share somewhere are replaced with only using pure Data Definition Language (DDL) describing how a table / view etc. should look. This has a big immediate benefit since you can see exactly how a given table should look for any version of your database. The changes (why they changed, what the change was) are stored by use of source control – Git in this example. That has built-in functionality for showing what changed, and commit comments explain why. You may be wondering how this works moving from v1 -> v2 -> v3? Before you needed incremental scripts, how does that work here? The answer is that the DacFx deployment technology used by our SQL tooling automatically generates those scripts for you. When developing and testing locally you’ll be making lots of small changes which are automatically applied (try hitting F5 on a database project to see your latest changes pushed to a local development DB, for instance). When publishing full changes to a staging server, you can choose to script out those changes for validation by a DBA or directly publish the changes using Visual Studio / our SqlPackage command line tools. In addition, now that the entire DB structure is in a project you get build-time validation to detect issues such as the broken stored procedure we would’ve missed in the connected world. This is really invaluable as a tool and can significantly improve long term maintainability. The big benefits here are therefore that versioning becomes much less of a concern, you get lots of extra validation (build time, using the local deployment loop, and via validation of the scripts generated for full publish), and your change information is documented and stored in a consistent manner. |
Image may be NSFW.
Clik here to view.
Getting to a more fully-fledged development pipeline you may wish to separate out the Developer and DBA roles. Here we go through an example of that, the only real change being that the developer outputs a Dacpac (a single file representing the expected database state, usable to deploy across Visual Studio, SSMS and our SqlPackage command line tooling) or a set of pre-build scripts build against the expected target server state, then hands these off to a DBA for deployment to the staging server. This is a common scenario in many companies where devs are not given access to staging / production servers. The tooling handles this well and readily supports scenarios such as this. |
Image may be NSFW.
Clik here to view.
Let’s show how all this works in Visual Studio, moving from the connected world to a project-based system with source control integration and some basic unit tests to validate your functions or procedures work as expected. |
Image may be NSFW.
Clik here to view.
Image may be NSFW.
Clik here to view.
Image may be NSFW.
Clik here to view.
Image may be NSFW.
Clik here to view.
Image may be NSFW.
Clik here to view.
Image may be NSFW.
Clik here to view.
Image may be NSFW.
Clik here to view.
Image may be NSFW.
Clik here to view.
Image may be NSFW.
Clik here to view.
Image may be NSFW.
Clik here to view.
Image may be NSFW.
Clik here to view.
Image may be NSFW.
Clik here to view.
Image may be NSFW.
Clik here to view.
Image may be NSFW.
Clik here to view.
What we’ve shown so far can work great for teams of 3-5 developers. However using a fileshare makes things harder than it needs to be, and doesn’t get the benefits that come with a full featured tool such as TFS. TFS isn’t just source control, it also has automated build support using a build agent, with team-level validation, integration with your standard development processes (resolve work items on checkin, roll back changes if they break the build / break unit tests). Here we see a full development process, based on a developer checking kicking off a continuous integration build on the TFS server. This adds another layer of testing and validation, and only if it succeeds will it output a drop to the build share. Hence there’s a known quality bar to what gets output there so DBAs can have confidence when deploying to a staging server. |
Image may be NSFW.
Clik here to view.
Image may be NSFW.
Clik here to view.
Image may be NSFW.
Clik here to view.
Image may be NSFW.
Clik here to view.
Image may be NSFW.
Clik here to view.
Image may be NSFW.
Clik here to view.
Image may be NSFW.
Clik here to view.
Image may be NSFW.
Clik here to view.
Image may be NSFW.
Clik here to view.
Image may be NSFW.
Clik here to view.
Image may be NSFW.
Clik here to view.
Image may be NSFW.
Clik here to view.
Image may be NSFW.
Clik here to view.
Image may be NSFW.
Clik here to view.
Image may be NSFW.
Clik here to view.
Image may be NSFW.
Clik here to view.
Image may be NSFW.
Clik here to view.
Image may be NSFW.
Clik here to view.
Image may be NSFW.
Clik here to view.
Image may be NSFW.
Clik here to view.
Image may be NSFW.
Clik here to view.
Image may be NSFW.
Clik here to view.
Image may be NSFW.
Clik here to view.
Image may be NSFW.
Clik here to view.
Not much needs to be said about Visual Studio Online. It works the same as TFS on-prem, and in many ways works with less work than an on-prem installation. It’s great and something we encourage you to explore. Note that support for our latest SQL Tooling update, including SQL Server 2014 support, is expected in late August 2014. In the future we expect the time between when SQL Server tooling updates are released and the time these are applied to Visual Studio Online build servers to be much shorter, so you can be sure that you’re testing using the most up to date code. |
Image may be NSFW.
Clik here to view.
Image may be NSFW.
Clik here to view.
Image may be NSFW.
Clik here to view.
Image may be NSFW.
Clik here to view.
Image may be NSFW.
Clik here to view.
I'm very excited to share the hard work some of my peers have been doing - Auditing in Azure SQL Database. After focusing on a bunch of behind the scenes security improvements, it's good to get something customer facing. Check out http://azure.microsoft.com/en-us/documentation/articles/sql-database-auditing-get-started/ and this video - http://channel9.msdn.com/Shows/Data-Exposed/Auditing-in-Azure-SQL-Database. It's a great start to what I hope are many customer facing features.
Image may be NSFW.This blog post is co-authored by Jamie Shotton, Antonio Criminisi and Sebastian Nowozin of Microsoft Research, Cambridge, UK.
Computer vision, the field of building computer algorithms to automatically understand the contents of images, grew out of AI and cognitive neuroscience around the 1960s. “Solving” vision was famously set as a summer project at MIT in 1966, but it quickly became apparent that it might take a little longer! The general image understanding task remains elusive 50 years later, but the field is thriving. Dramatic progress has been made, and vision algorithms have started to reach a broad audience, with particular commercial successes including interactive segmentation (available as the “Remove Background” feature in Microsoft Office), image search, face detection and alignment, and human motion capture for Kinect. Almost certainly the main reason for this recent surge of progress has been the rapid uptake of machine learning (ML) over the last 15 or 20 years.
This first post in a two-part series will explore some of the challenges of computer vision and touch on the powerful ML technique of decision forests for pixel-wise classification.
Imagine trying to answer the following image classification question: “Is there a car present in this image?” To a computer, an image is just a grid of red, green and blue pixels where each color channel is typically represented with a number between 0 and 255. These numbers will change radically depending not only on whether the object is present or not, but also on nuisance factors such as camera viewpoint, lighting conditions, the background, and object pose. Furthermore one has to deal with the changes in appearance within the category of cars. For example, the car could be a station wagon, a pickup, or a coupe, and each of these will result in a very different grid of pixels.
Supervised ML thankfully offers an alternative to naively attempting to hand-code for these myriad possibilities. By collecting a training dataset of images and hand-labelling each image appropriately, we can use our favorite ML algorithm to work out which patterns of pixels are relevant to our recognition task, and which are nuisance factors. We hope to learn to generalize to new, previously unseen test examples of the objects we care about, while learning invariance to the nuisance factors. Considerable progress has been made, both in the development of new learning algorithms for vision, and in dataset collection and labeling.
Images contain detail at many levels. As mentioned earlier, we can ask a question of the whole image such as whether a particular object category (e.g. a car) is present. But we could instead try to solve a somewhat harder problem that has become known as “semantic image segmentation”: delineating all the objects in the scene. Here’s an example segmentation on a street scene:
Image may be NSFW.
Clik here to view.
In photographs you could imagine this being used to help selectively edit your photos, or even synthesize entirely new photographs; we’ll see a few more applications in just a minute.
Solving semantic segmentation can be approached in many ways, but one powerful building block is pixel-wise classification: training a classifier to predict a distribution over object categories (e.g. car, road, tree, wall etc.) at every pixel. This task poses some computational problems for ML. In particular, images contain a large number of pixels (e.g. the Nokia 1020 smartphone can capture at 41 million pixels per image). This means that we potentially have multiple-million-times more training and test examples than we had in the whole-image classification task.
The scale of this problem led us to investigate one particularly efficient classification model, decision forests (also known as random forests or randomized decision forests). A decision forest is a collection of separately-trained decision trees:
Image may be NSFW.
Clik here to view.
Each tree has a root node, multiple internal “split” nodes, and multiple terminal “leaf” nodes. Test time classification starts at the root node, and computes some binary “split function” of the data, which could be as simple as “is this pixel redder than one of its neighbors?” Depending on that binary decision, it will branch either left or right, look up the next split function, and repeat. When a leaf node is finally reached, a stored prediction – typically a histogram over the category labels – is output. (Also see Chris Burges’ excellent recent post on boosted variants of decision trees for search ranking.)
The beauty of decision trees lies in their test-time efficiency: while there can be exponentially many possible paths from the root to leaves, any individual test pixel will only pass down just one path. Furthermore, the split functions computation is conditional on what has come previously: i.e. the classifier hopefully asks just the right question depending on what the answers to the previous questions have been. This is exactly the same trick as in the game of “twenty questions”: while you’re only allowed to ask a small number of questions, you can quickly hone in on the right answer by adapting what question you ask next depending on what the previous answers were.
Armed with this technique, we’ve had considerable success in tackling such diverse problems as semantic segmentation in photographs, segmentation of street scenes, segmentation of the human anatomy in 3D medical scans, camera relocalization, and segmenting the parts of the body in Kinect depth images. For Kinect, the test-time efficiency of decision forests was crucial: we had an incredibly tight computational budget, but the conditional computation paired with the ability to parallelize across pixels on the Xbox GPU meant we were able to fit [1].
In the second part of this series, we’ll discuss the recent excitement around “deep learning” for image classification, and gaze into the crystal ball to see what might come next. In the meantime, if you wish to get started with ML in the cloud, do visit the Machine Learning Center.
Thanks for tuning in.
Jamie, Antonio and Sebastian
[1] Body part classification was only one stage in the full skeletal tracking pipeline put together by this fantastic team of engineers in Xbox.
Image may be NSFW.It has been a really long, really busy week, so not many links today. Looking forward to the weekend and a short week next week. I'll be on vacation next Friday, so no links next week.
Emulate touch on the desktop: Useful if you need to test touch stuff, but don't have access to a touch device.
HueWheel: A nice color wheel in HTML5 & JavaScript.
ElixirConf 2014 videos are available: Maybe I'll go next year.
Learn Clojure by painting: Code that paints.
Enjoy!
- Larry
Image may be NSFW.This blog post is authored by John Langford, Principal Researcher at Microsoft Research, New York City.
Vowpal Wabbit is an open source machine learning (ML) system sponsored by Microsoft. VW is the essence of speed in machine learning, able to learn from terafeature datasets with ease. Via parallel learning, it can exceed the throughput of any single machine network interface when doing linear learning, a first amongst learning algorithms.
The name has three references---the vorpal blade of Jabberwocky, the rabbit of Monty Python, and Elmer Fudd who hunted the wascally wabbit throughout my childhood.
VW sees use inside of Microsoft for ad relevance and other natural-language related tasks. Its external footprint is quite large with known applications across a broad spectrum of companies including Amazon, American Express, AOL, Baidu, eHarmony, Facebook, FTI Consulting, GraphLab, IBM, Twitter, Yahoo! and Yandex.
Why? Several tricks started with or are unique to VW:
VW supports online learning and optimization by default. Online learning is an old approach which is becoming much more common. Various alterations to standard stochastic gradient descent make the default rule more robust across many datasets, and progressive validation allows debugging learning applications in sub-linear time.
VW does Feature Hashing which allows learning from dramatically rawer representations, reducing the need to preprocess data, speeding execution, and sometimes even improving accuracy.
The conjunction of online learning and feature hashing imply the ability to learn from any amount of information via network streaming. This makes the system a reliable baseline tool.
VW has also been parallelized to be the most scalable public ML algorithm, as measured by the quantity of data effectively learned from, more information here.
VW has a reduction stack which allows the basic core learning algorithm to address many advanced problem types such as cost-sensitive multiclass classification. Some of these advanced problem types, such as for interactive learning exist only in VW.
There is more, of course, but the above gives you the general idea – VW has several advanced designs and technologies which make it particularly compelling for some applications.
In terms of deployment, VW runs as a library or a standalone daemon, but Microsoft Azure ML creates the possibility of cloud deployment. Imagine operationalizing a learned model for traffic from all over the world in seconds. Azure ML presently exposes the feature hashing capability inside of VW via a module of the same name.
What of the future? We hunt the Wascally Wabbit. Technology-wise, I intend to use VW to experiment with other advanced ML ideas:
Can ML be made radically easier to program: http://arxiv.org/pdf/1406.1837?
Is efficient exploration possible: http://arxiv.org/pdf/1402.0555?
Can we predict one of k things efficiently in logarithmic time: http://arxiv.org/pdf/1406.1822?
Good answers to these questions can change the scope of future ML applications wadically.
John
Follow my personal blog here.
After WCF Data Services Entity Framework Provider version 1.0.0-alpha2 was released, we got lots of valuable feedbacks from customers. Recently we summarized all the feedbacks and fixed the most critical bugs reported. Now we are happy to announce the release of WCF Data Services Entity Framework Provider 1.0.0-beta2 (Below using EF6 Provider beta2 for short).
1. Create a new project, an ASP.NET Empty Web Application in Visual Studio 2012 for example.
2. Install the EF6 Provider beta2 NuGet package. WCF Data Service 5.6.2 and Entity framework 6.0.1 will be installed as dependencies.
3. Create a data model with EF6
Follow instructions in this blog post to upgrade to EF6 and make sure you install EF 6.x DbContext Generator if using DB first development.
4. Add a new WCF Data Service 5.6.2. This item will be added into Visual Studio by installing WCF Data Service 5.6.2 tooling installer. (5.6.2 tooling installer will be published soon, currently you can add a WCF Data Service, and upgrade WCF Data Service dll to 5.6.2)
Image may be NSFW.
Clik here to view.
5. Replace the base type of the DataService that was generated by the item template with EntityFrameworkDataService. See the article What’s the difference between DataService and EntityFrameworkDataService for more details.
If you upgrade an existing service to EF6 provider beta2, you just need to upgrade EF provider Nuget package and make sure your data service inherits from EntityFrameworkDataService where T is your data source class.
Based on customers’ feedbacks of previous EF6 Provider, we got lots of bugs fixed in this version. Please refer to the release note for details. The top-voted issues “complex type” and “CurrentDataSource” are fixed as the highest priority.
1. Complex type can be serialized now
Complex type is supported in both DB First and Code First development. The following are examples of complex type scenario in each approach.
In DB First development, for example in DataBaseFirstSample, create a complex type “ShipInfo” in EF designer (Refer Complex Types in the EF Designer) to contain properties like “ShipName”, “ShipAddress”, etc. and class “ShipInfo” will be auto-created after saving.
Then complex property can be accessed with URL:
~/DataBaseFirstSampleService.svc/Orders(10248)/ComplexProperty/ShipCountry
In Code First development, for example in CodeFirstSample, define a class named “PostContent”:
Define property“content” based on “PostContent” in entity type“Post”:
Then complex property can be access with URL:
~/CodeFirstSampleService.svc/Posts(1)/Content/Title
2. CurrentDataSource on DataService class can be used to retrieve DBContext now
CurrentDataSource is used often in functions, and query or change interceptors. The following gives an example of using CurrentDataSource in function definition.
For example in DataBaseFirstSample service, define a function “GetOrdersCount” which accesses “Orders” through CurrentDataSource:
Then the function “GetOrdersCount” can be called with URL:
~/DataBaseFirstSampleService.svc/GetOrdersCount
Apologize that we do not have enough time to support Enum and Spatial types in this version, service would fail to access if Enum or spatial type exists in data model.
The EF6 provider source code has been released under Apache 2.0 license. You can download the code from our Git repository at https://git01.codeplex.com/odata, or view more about OData at https://odata.codeplex.com/ . We continues our commitment to open source, and we are happy to receive community contributions and feedbacks.
Image may be NSFW.Today we’re providing a third CTP of the next version of Visual Studio, to gather your early feedback. For more information on this release, see Visual Studio “14” CTPs and the Visual Studio Blog. This post covers the places that Entity Framework is included in the release and some limitations to be aware of when using it.
There is a bug in the version of the designer included in this CTP that will cause the following error to appear when trying to use the EF Tools.
Method not found: 'Void Microsoft.VisualStudio.PlatformUI.DpiHelper.LogicalToDeviceUnits(System.Windows.Forms.ImageList ByRef, Microsoft.VisualStudio.PlatformUI.ImageScalingModel)'.
To resolve this issue, you will need to install an updated version of the Entity Framework 6.1.1 tools for Visual Studio “14” CTP3.
The Entity Framework 6.1.1 runtime is included in a number of places in Visual Studio “14” CTP.
We recently blogged about our plans for Entity Framework 7. Visual Studio “14” CTP 3 includes an early preview of ASP.NET vNext, which in turn includes a very early build of EF7. The EF7 runtime is installed in new ASP vNext projects that are created.
As with previous CTPs of Visual Studio “14”, this build of EF7 implements basic functionality and there are a number of limitations with the features that are implemented. Please bear in mind that this preview is designed to give you an idea of what the experience will be like and you will quickly hit limitations if you deviate from the code from the default project template.
For more information on what’s included in this build of EF7, see the release notes page on our GitHub project.
We’re making good progress on the EF7 code base, but it is still in the early stages of development. If you want to experiment with a build we would recommend visiting our GitHub wiki for information on using nightly builds.
Image may be NSFW.This blog post is authored by Mario Garzia, Partner Data Sciences Architect, Technology and Research
Image may be NSFW.
Clik here to view.Data Science and “big data” have become 21st century buzzwords in the tech industry. Yet in many ways the term “Big Data” is relative to our ability to collect, store and process data. Big data challenges are not new, historically there have been several notable encounters with big data. One interesting example is the US census. The 1880 census took eight years to tabulate, and at the time estimates were that the 1890 census would take more than 10 years to complete as a result of population expansion. This was a Big Data problem at the time, and a man by the name of Herman Hollerith came to the rescue with an invention that tabulated the 1890 census in a single year and under budget with his company eventually going on to become IBM. Hollerith accomplished this by developing a new and efficient way to collect and store the increasing volumes of data (punch cards) and an electric tabulating machine that could read the punch cards and compute the needed results. There are other similarly interesting big data challenges that occurred both before and after Dr. Hollerith’s time. So is today’s Big Data challenge any different from past ones?
Image may be NSFW.
Clik here to view.Data volumes are growing at a rate that continues to challenge our ability to collect, store and process data leading to the development of new technologies. But now the variety of data and the speed at which we collect the data are also accelerating. These growing trends have no visible end in sight. In a 2011 report Ericsson estimated 50 Billion connected devices world wide by 2020, each of these generating their own data in addition to the data exhaust generated by the systems that will manage the collection and processing of the device data. Another big difference today that presents tremendous opportunity is the ability to collect data directly from each of our end customers to learn about their experience with a device or service to a degree never before possible. This allows us to imagine entirely new ways to help and delight customers with new products and services that were previously unimaginable, that better understand what they need now and predict what they might need next. To date high tech companies have been the leaders in this data space, where in some cases the data itself is the product such as Bing Search or social networks, but a great aspect of today’s world is that technology is facilitating the democratization of data and analytics to derive insights across the full spectrum of human endeavor. So not only Big Data leaders but also more traditional businesses and other institutions can now leverage big data to improve their services and delight their customers. We live at a fascinating point in time where the once unimaginable is now becoming possible guided by data and analytics.
Microsoft has a very rich tradition of using data to gain insights and drive product decisions going back many years, long before Data Sciences and Big Data became terms de rigueur. I joined Microsoft in 1997 and have seen first-hand how we have evolved and grown in the data space. One of the things that I have loved most about working here is the ability to surround myself with and learn from very talented and passionate people. This is a culture where learning, gaining new understandings and striving to be the best are very much engrained. Because of this, data has always played an important role at Microsoft but that role has evolved and expanded over the past decade. We have grown from focusing on having a deep understanding of the product being shipped to also developing a deep understanding of customer experiences with our products and services.
In 2000 I came to the Windows team to form the Reliability group. Right from the start Windows Reliability was a data driven effort. For example, by the time we shipped Windows server 2000, we already had approximately 100 years of reliability runtime data on internal Microsoft production servers. After Windows server 2000 shipped we expanded data collection to other enterprises by developing a Reliability service for which companies could sign up free of charge, use it to collect reliability data from their datacenter serves and upload the data to Microsoft. This data would then be automatically analyzed and the results made available to each company individually in a website containing availability and reliability results and trends segmented by server type and computing environment. In many cases this was the first time these companies had access to such detailed data on the reliability of their data centers. This data could then also be leveraged by Windows to gain insights into operating system (OS) reliability and failure modes, set release criteria for new versions of the OS, and prioritize and drive product fixes based on failure frequency and gravity. We also used the insights from this data to develop new OS features like diagnostic services. This data driven approach allowed us to make decisions for when the product was ready to ship based on actual production system runtime criteria. While deep and comprehensive, this data was focused on product quality and ship readiness. Today the Windows operating system, and indeed all our products and services are focused not only on product quality attributes but also on better understanding customer needs. There is a renewed and expanded emphasis on building a data driven culture at the company where service and product quality remain critical but just as critical is the deep understanding of customer satisfaction, engagement and wants. Insights derived from data are used across all Microsoft products and services to deliver new, powerful features and capabilities.
Being a data driven culture means that understanding the product and customer data is not just for Data Scientists but for all of Microsoft, everyone needs to be data aware and data driven at Microsoft. Big data is used for product and service experimentation, improvement, and also to deliver enhanced and customized services leveraging techniques such as Machine Learning. Bing and Bing Ads are completely data driven. There is also a very deep heritage in Machine Learning at Microsoft over the past 20 years, from its beginnings with Bayesian Networks and speech recognition research to products such as SQL Server Data Mining. We now give companies the ability to build machine learning models and easily deploy them to the cloud with Microsoft Azure ML.
An exciting aspect of being a data scientist at Microsoft is the unparalleled breadth of customer touch points we have from computers and tablets, to phones, devices, gaming, Search and a myriad of services allowing us to better understand customer wants and experiences and use those insights to impact their everyday life in new and meaningful ways. The Data Sciences disciplines are at the core of our data driven corporate strategy. At Microsoft we recognize this and have a full engineering career path for Data Scientists, Machine learning Scientists and Applied Scientists that can reach the most senior levels in the company. We have multiple data scientist groups throughout the company resulting in a very vibrant and growing community. I believe there is no better place than Microsoft for a Data Scientist to learn, grow, have fun and make an impact.
An important event that many Microsoft Data Scientists attend each year is the Knowledge Discovery and Data Mining conference that takes place in August; this year in New York City. This is a premiere conference for data sciences. I am very much looking forward to attending this year’s KDD conference, I have been attending for many years now. It is great to share in the energy and excitement, exchange ideas with colleagues and meet new people. I always come out of the conference totally charged by the new ideas and people I’ve met. Microsoft is a Gold Sponsor at KDD this year and we are very excited to be there. Please make sure to stop by our Microsoft exhibitor booth to view demos from our Data Scientists in the Azure Machine Learning team, Bing team, MSR and many others. I hope to meet some of you at the conference.
Mario
If you are interested in a career at Microsoft, check out our openings:
Data Science Jobs at Microsoft
Machine Learning Jobs at Microsoft
Image may be NSFW.This blog post is co-authored by Jamie Shotton, Antonio Criminisi and Sebastian Nowozin of Microsoft Research, Cambridge, UK.
In our last post, we introduced you to the field of computer vision and talked about a powerful approach, classifying pixels using decision forests, which has found broad application in medical imaging and Kinect. In this second post we will look at some of the recent excitement around deep neural networks and their successes in computer vision, followed by a look at what might be next for computer vision and machine learning.
The last few years have seen rapid progress in the quality and quantity of training datasets we have access to as vision researchers. The improvements are to a large extent due to the uptake of crowdsourcing which has allowed us to scale our datasets to millions of labelled images. One challenging dataset, ImageNet, contains millions of images labeled with image-level labels across tens of thousands of categories.
After a few years of slow progress in the community on the ImageNet dataset, Krizhevsky et al. rather rocked the field in 2012. They showed how general-purpose GPU computing paired with some seemingly subtle algorithmic changes could be used to train convolutional neural networks much deeper than before. The result was a remarkable step change in accuracy in image classification on the ImageNet 1000-category test. This also garnered a lot of attention in the popular press and even resulted in some very large start-up buyouts. Since then “deep learning” has become a very hot topic in computer vision, with recent papers extending the approach to object localization, face recognition, and human pose estimation.
While clearly very powerful, are deep convolutional networks the end of the road for computer vision? We’re sure they’ll continue to be popular and push the state of the art in the next few years but we believe there’s still another step change or two to come. We can only speculate as to what these changes will be, but we finish up by highlighting some of the opportunities as we see them.
Representations: These networks learn to predict a relatively simple representation of the image contents. There’s no deep understanding of where individual objects live in the image, how they relate to one another, or the role of particular objects in our lives (e.g. we couldn’t easily combine the cue that a person’s hair looks slightly glossy with the fact that they are holding a hair-dryer to get a more confident estimate that their hair is wet). New datasets such as Microsoft CoCo may help push this forward by providing very detailed labeling of individual object segmentations in “non-iconic” images – i.e. images where there’s more than one object present that are not front-and-center.
Efficiency: While the evaluation of a deep network on a test image can be performed relatively quickly though parallelization, neural networks don’t have the notion of conditional computation that we encountered in our last post: every test example ends up traversing every single node in the network to product its output. Furthermore, even with fast GPUs, training a network can take days or weeks which limits the ability to experiment rapidly.
Structure learning: Deep convolutional networks currently have a carefully hand-designed and rather rigid structure that has evolved over many years of research. Changing, say, the size of a particular layer or the number of layers can have undesirable consequences to the quality of the predictor. Beyond simply brute-force parameter sweeps to optimize the form of the network, we hope there is opportunity to really learn a much more flexible network structure directly from data.
Recently, we have been taking baby steps towards addressing particularly the latter two of these opportunities. We’re particularly excited by our recent work on decision jungles: ensembles of rooted decision DAGs. You can think of a decision DAG as a decision tree in which child nodes have been merged together so that nodes are allowed to have multiple parents. Compared to decision trees, we’ve shown that they can reduce memory consumption by an order of magnitude while also resulting in considerably improved generalization. A DAG also starts to look a lot like a neural network, but does have two important differences: firstly, the structure is learned jointly with the parameters of the model; and secondly, the DAG retains the idea from decision trees of efficient conditional computation: a single test example follows a single path through the DAG, rather than traversing all nodes as would be the case with a neural network. We’re actively investigating whether decision jungles, perhaps in conjunction with other forms of deep learning including stacking and entanglement, can offer an efficient alternative to deep neural networks.
If you’re interested in trying out decision jungles for your problem, the Gemini module in Azure ML will let you investigate further.
Overall, computer vision has a bright future thanks in no small part to machine learning. The rapid recent progress in vision has been fantastic, but we believe the future of computer vision research remains an exciting open book.
Jamie, Antonio and Sebastian
Image may be NSFW.I am pleased to announce previews of new NoSQL database and search services and the evolution of our Hadoop-based service. Available as previews today are Azure DocumentDB, a fully-managed transactional NoSQL document database-as-a-service, and Azure Search, which enables developers to easily add search capabilities to mobile and cloud applications. Generally available today, Azure HDInsight, our Hadoop-based solution for the cloud, now supports Apache HBase clusters.
With these new and updated services, we’re continuing to make it easier for customers to work with data of any type and size – using the tools, languages and frameworks they want to -- in a trusted cloud environment. From Microsoft products like Azure Machine Learning, Azure SQL Database and Azure HDInsight to data services from our partners, we’re committed to supporting the broadest data platform so our customers get data benefits, in the cloud, on their terms.
Preview of Azure DocumentDB
Applications today must support multiple devices, multiple platforms with rapid iterations from the same data source, and also deliver high-scale and reliable performance. NoSQL has emerged as the leading database technology to address these needs. According to Gartner inquiries, flexible data schemas and application development velocity are cited as primary factors influencing adoption. Secondary factors attracting enterprises are global replication capabilities, high performance and developer interest.*
However, while NoSQL technologies address some document database needs, we’ve been hearing feedback that customers want a way to bridge document database functionality with the transactional capabilities of relational databases. Azure DocumentDB is our answer to that feedback – it’s a NoSQL document database-as-a-service that provides the benefits of a NoSQL document database but also adds the query processing and transaction semantics common to relational database systems.
Built for the cloud, Azure DocumentDB natively supports JSON documents enabling easy object mapping and iteration of data models for application development. Azure DocumentDB offers programming libraries for several popular languages and platforms, including.Net, Node.js, JavaScript, and Python. We will be contributing the client libraries to the open source community, so they can incorporate improvements into the versions published on Azure.com.
One DocumentDB customer, Additive Labs, builds online services to help their customers move to the cloud. "DocumentDB is the NoSQL database I am expecting today,” said Additive Labs Founder Thomas Weiss. “The ease and power of SQL-like queries had me started in a matter of minutes. And the ability to augment the engine’s behavior with custom JavaScript makes it way easier to adapt to our customers’ new requirements.”
Preview of Azure Search
Search has become a natural way for users to interact with applications that manage volumes of data. However, managing search infrastructure at scale can be difficult and time consuming and often requires specialized skills and knowledge. Azure Search is a fully-managed search-as-a-service that customers can use to integrate complete search experiences into applications and connect search results to business objectives through fine-tuned, ranking profiles. Customers do not have to worry about the complexities of full-text search or deploying, maintaining or managing a search infrastructure.
With Azure Search developers can easily provision a search service, quickly create and tune one or more indexes, upload data to be indexed and start issuing searches. The service offers a simple API that’s usable from any platform or development environment and makes it easy to integrate search into new or existing applications. With Azure Search, developers can use the Azure portal or management APIs to increase or decrease capacity in terms of queries per second and document count as load changes, delivering a more cost effective solution for search scenarios.
Retail platform provider Xomni is already using Azure Search to help the company manage its cloud infrastructure. "We have the most technically advanced SaaS solution for delivering product catalogue data to the retail industry in the market today,” said Xomni CTO Daron Yondem. “Integrating Azure Search into our platform will help solidify our leadership as datasets and faceted search requirements evolve over time."
General availability of Apache HBase for HDInsight
In partnership with Hortonworks, we’ve invested in the Hadoop ecosystem through contributions across projects like Tez, Stinger and Hive. Azure HDInsight, our Hadoop-based service, is another outcome of that partnership.
Azure HDInsight combines the best of Hadoop open source technology with the elasticity and manageability that enterprises require. Today, we’re making generally available HBase as a managed cluster inside HDInsight. HBase clusters are configured to store data directly in Azure Blob storage. For example, customers can use HDInsight to analyze large datasets in Azure Blobs generated from highly-interactive websites or can use it to analyze sensor and telemetry data from millions of end points.
Microsoft data services
Azure data services provide unparalleled choice for businesses, data scientists, developers and IT pros with a variety of managed services from Microsoft and our partners that work together seamlessly and connect to our customers’ data platform investments-- from relational data to non-relational data, structured data to unstructured data, constant and evolving data models. I encourage you to try out our new and expanded Azure data services and let us know what you think.
*Gartner, Hype Cycle for Information Infrastructure, 2014, Mark Beyer and Roxane Edjlali, 06 August 2014
###
Image may be NSFW.We are excited to announce the release of a SQL Server AlwaysOn template in the Microsoft Azure Portal Gallery. This offering was announced in Scott Guthrie’s blog post along with several other exciting new features.
This template fully automates the configuration of a highly available SQL Server deployment on Azure Infrastructure Services using AlwaysOn Availability Groups.
AlwaysOn Availability Groups, released in SQL Server 2012 and enhanced in SQL Server 2014, guarantee high availability for mission-critical workloads. Last year we started supporting Availability Groups on Azure Infrastructure Services. The main components of such a configuration are two SQL Server replicas (a primary and a secondary), and a listener (DNS name). The replicas are configured for automatic failover, and each replica is contained on a distinct Virtual Machine. The listener is a DNS name that client applications can use in their connection string to connect to the current primary replica. The image below shows a depiction of this setup.
Image may be NSFW.
Clik here to view.
Other components required are a file share witness to guarantee quorum in the configuration to avoid “split brain” scenarios, and a domain controller to join all VMs to the same domain. Similar to the SQL replicas, there is a primary and secondary domain controller to prevent a single point of failure for the domain. The SQL replicas are deployed to an availability set to ensure they are in different Azure failure and upgrade domains. Likewise, the domain controller replicas are in their own availability set. The configuration is depicted in the image below.
Image may be NSFW.
Clik here to view.
Setting up the Availability Group configuration requires a long set of steps and a decent time commitment. In order to dramatically simplify this, we have released a SQL Server AlwaysOn template in the Azure Gallery. This template fully automates the configuration of a highly available SQL Server deployment on Azure Infrastructure Services using an Availability Group. Currently, this feature only supports SQL Server 2014 Enterprise images.
The SQL Server AlwaysOn Template, depicted below, is found in the Gallery under “Virtual Machines” and “Recommended”.
Image may be NSFW.
Clik here to view.
After selecting it, it will show a description of the configuration that will be created, and the option to specify some arguments. This is depicted in the picture below.
Image may be NSFW.
Clik here to view.
The only arguments required are a Resource Group (an identifier of the deployment) and administrator credentials. From that point on, all settings are optional and will be auto-generated based on these 3 inputs. The domain Sysadmin account, the local SQL Server accounts, and the SQL Server service account password will be auto-generated based on the credentials entered. The names for all resources being created will be based off of what was entered for Resource Group name. The SQL Server service account name and the domain name will be auto-generated but will not be based on the Resource Group name or credentials. If you wish to customize any of these arguments, simply go to the other configurations and change the values entered for any setting. One argument that you may want to change is the Listener name, which your applications will use to connect to SQL Server. By default, entirely new resources will be provisioned for you. You have the option to select an existing domain for the deployment. In future updates, there will be more options to add existing resources to your configuration.
After the template has executed, 5 Virtual Machines will be created under the resource group: 2 Standard A5 VMs for the SQL Server replicas, 2 Standard A1 VMs for the Domain Controller replicas, and 1 Basic A0 VMs for the file share witness. This is depicted below:
Image may be NSFW.
Clik here to view.
You can RDP to one of the SQL Server VMs to see the Availability Group configuration as depicted below:
Image may be NSFW.
Clik here to view.
Try out the SQL Server AlwaysOn Template today by going to the Azure portal: http://portal.azure.com/
Image may be NSFW.The release candidate NuGet packages for ASP.NET Web API OData 5.3 are now live on the NuGet gallery!
Download this release
You can install or update the release candidate NuGet packages for ASP.NET Web API OData 5.3 using the NuGet Package Manager Console, like this:
What’s in this release?
This release primarily includes great new features for Web API OData v4 as summarized below:
ASP.NET Web API OData 5.3 Release Candidate
Additional Documentation
Tutorials and other information about Web API OData are available from the ASP.NET web site (http://www.asp.net/web-api/overview/odata-support-in-aspnet-web-api).
Questions and feedback
You can submit questions related to this release on the ASP.NET Web API forums. Please submit any issues you encounter and feature suggestions for future releases on our CodePlex site.
Thanks and enjoy!
Image may be NSFW.