Tag Archives: Azure

WADLogsTable missing in Cloud Service project


This is observed in Visual Studio Cloud Service Template (Environment  VS 2015 Enterprise Update 3 and Azure SDK 2.9.6 with .NET 4.6). This could be observed in most of the other versions – probably in Azure SDK version 2.4 and above, but I stated my working environment as soon or later this issue will be resolved.

Quick read and the reason : The template has the Trace.TraceInformation as the logging code line, but the configuration is set to log Errors by default. So when you run the application, the service has nothing to log and it doesn’t create WADSLogsTable. By changing the code to Trace.Error or changing the configuration to log information/verbose would solve the issue.

Analysis

Mostly beginners bounce into this issue, and fair reason to get panic because when the just create the fresh Azure Cloud Service out of the box from the available template, it doesn’t work as expected.

Go to the Worker Role properties and you can change the application log level settings to log Information level logs.

default trace log setting - error

or, change the code to this

image

The application creates the table when and only the information need to be persisted, so the available template does not create the WADSLogsTable until you do either of the suggested changes.

Advertisements

Controlling access to your Azure resources using RBAC


Being part of a software services company, customers often ask the question how to restrict access to Azure resources. It is understandable that any organization wouldn’t prefer to give all the rights of the organizational Azure subscription to a person.

In the classic Azure model the only way to give access to Azure portal is, adding the user as a co-admin for the subscription. This gives all the permissions to that user within the subscription except managing the administrators.

But the new Role Based Access Control  (RBAC) helps to solve this problem. Using RBAC we can control the permission scope to either subscriptions, resource groups or to individual resources.

Permissions in the top level scope are automatically inherited to the level below – meaning subscription level users have the same permissions to the resource groups and the resource group level users have the same permission to the individual resources within the resource group.

RBAC has several roles – read more about different roles

Here I’ve explained the flow of adding a new user to a Azure resource group and how his/her experience in accessing Azure via portal. Assume the user doesn’t have any permission in the Azure, and he’s just a developer with a Gmail account.

First, a subscription admin logs in to the portal and add this user in the Azure Active Directory of the specific subscription.

1

Note at this point, developer1 does not have a Microsoft account. She clicks on the link in the email  she received. She will be directed to create a Microsoft account with the specified email address. (if there’s a Microsoft account already available this step will not be required)

2

After creating the Microsoft account (entering a new password and create the Microsoft account), she can login to the Azure portal using – https://portal.azure.com But within the portal this user cannot create any resources. In case if the user tries to create or try to perform any action she will get the below message. This is a very similar message to the old grey error box in classic portal, as the user exist in the Azure Active Directory but does not have a subscription, in this case does not have any resource.

3

Now let the admin assign a resource group for the user.  Assume you have a resource group DevelopmentRG and in the resource group IAM settings add the user (developer1) as a contributor.

4

Contributor is a predefined role in Azure which has the create/edit/delete permissions of the resources within the specified scope. In this case developer1 has those permissions within the resource group – DevelopmentRG.

5

After setting developer1 as a contributor, you can notice that the access type of the user is set to Assigned, because this is a an assigned permission. Also note that the subscription admins have the permission to the resource group as Inherited permission.

6

Now the developer1 logins to the portal and she will see the assigned resource group. Developer1 can perform actions within this resource group.

7

Also note that since, developer1 has only the specified resource group, she cannot create a new resource group or any permission outside the scope of the specific resource group.

8

RBAC provides more granular permissions with various roles required for the businesses, this helps the organizations to carefully delegate the permissions to the people without exposing the entire Azure subscription.

The feature to limit/set the quota for a resource group is in the request from the community.

Distributed Transactions in Azure SQL Databases–Azure App Service and EF


Are you handing more than one SQL database in Azure for your application ? Most of the times the answer would be YES. In dedicated database multi-tenant systems at least you have your customer information in the master database and dedicated application database for each customers. Some CRUD operations need to touch both the master and customer specific databases.

We need MSDTC (Microsoft Distributed Transaction Controller) for distributed transactions in on premise systems, but in Azure the SQL Databases has the elastic distributed transaction feature enabled and using .NET 4.6.1 we can use them via TransactionScope class from Systems.Transactions.

This link explains how this works, but I wanted to test this with EF and Azure App service as the Azure App service has the target platform option .NET 4.6 and not 4.6.1.

I created two logical Azure SQL servers in two different regions, and enabled the transaction communication link between them using PowerShell.

2016-08-27_18-13-09

Then I created a small Web API project using .NET 4.6.2 (which is higher than the required version) and tested the app from the local machine and things worked well. I deployed the same stuff and things worked fine in Azure as well.

Even the though the target platform is .NET 4.6 in the Azure App Service, when we deploy the .NET 4.6.1 and .NET 4.6.2 projects, the required assemblies in the respected platform version are referenced.

But my swagger endpoint behaved strange and didn’t output the results, no idea why and need to launch another investigation for that.

You can reference the test project from my Github

Conclusion – We can use the Distributed transactions in Azure SQL Database using EF and deploy your projects written in .NET 4.6.1/ 4.6.2 in the Azure App Service platform targeting .NET 4.6

Directory contains one or more applications that were added by a user or administrator.


This post summarizes and lists down the information you need to solve the specific error which occurs when you try to delete a AAD.

There are plenty of articles for this and I recommend to read the link below which explains very frequent error messages of AAD and the fixes.

https://support.microsoft.com/en-us/kb/2967860

Also read this blog for a detailed description.

http://alexmang.com/2015/06/directory-contains-one-or-more-applications-that-were-added-by-a-user-or-administrator-solution/

Some useful information

In order to manage the AAD using PowerShell we need to install two components.

  1. AAD Module for PowerShell
  2. Microsoft Online Services Sign-in Assistant

See the below article on how you can accomplish this.

https://onlinehelp.coveo.com/en/ces/7.0/administrator/installing_the_windows_azure_ad_module_for_windows_powershell.htm

Quick Steps

  1. Login to Azure Portal using your service administrator account.
  2. Make sure there are no users or other external applications. (If you find any of them delete them)
  3. Create a Global Admin user in the AAD you want to delete.
  4. In the PowerShell, login as the created Global Admin user
  5. Run the following scriptimage
  6. You will get error messages as mentioned in Alexe’s Blog but you can simply ignore them.
  7. Then go to the portal and delete the created Global Admin user.
  8. And finally delete the AAD.

Design an online forum on Azure Document DB


Couple of weeks back I posted an article on how to design an online forum application on top of Azure Table Storage. This post is about how to design the same application using Azure Document DB. Same as the previous article, I want to stress the point that the way we design an application and the thinking behind the design completely differ based on the NoSQL technology we select.

These are the basic requirements / functionalities of the application.

  • Forum members can post questions under categories.
  • Forum members can reply to posts.
  • Users have points based on their forum contribution.

Document type NoSQL databases are handy in storing data as documents, most of the modern document databases support JSON as the document storage format.

Also I assume that you have the understanding of Azure Document DB about indexing, consistency levels and how it is structured as databases, collections, documents and more.

Basic Idea

Based on the above requirements, if we design a single document it would look similar to this.

image

As you see we can have a single document structure to cover everything the application requires, but it has some drawbacks too.

Mainly user data is redundant and if we want to update the points of the user we have to go through all the documents and update it, we use other data operations like map reduce to perform these operations in a large scale document type implementations.

Design for Azure Document DB

It is recommended and straight forward to have a dedicated collection for each identified entitiy. Thinking on that base, we would require four main collections they are users, categories, posts and replies.

This design is easy, highly scalable but expensive, because Document DB databases are containers for the collections. Collections are the containers for the documents and also a single collection is the billing entity, meaning that if you create a database and two collections within that database in a tier priced $25 per month, then you will be billed $50 per month as you have two collections.

In this design will be having 4 collections.

User document

image

Category document

image

Posts document

image

Reply document

image

 

But this would not be the ideal design for the solution in terms of the best tradeoff between the cost and the solution design.

Because having a dedicated collection for the category is not necessary, we can simply have the category as an attribute in the posts. Having a dedicated collection for users might sound too much. I do not totally offend this – because, sometimes it is a good idea to have a dedicated collection for the users, especially when the number of users grow in large scale.

Also remember the design using the Azure table storage where we used bucketing strategies to partition the users, we can use the same strategy here if we have millions of users. We can put them in different collections rather than keeping them in one single collection.

But say that we have only few hundreds of users and few categories, then we do not want to have separate collection for each. So we need a mechanism to put them in the same collection and query them.

The idea is simple, again this is not the technology but it is the best decision we make on top of the technologies we use.

Have two documents with their Ids represent the entities or you can have an attribute called type which represents the document.

image

image

When you change the design like this, there is a significant change that you should do in your queries.

But again the idea here is to give you the possibilities how you can design the solution on top of Document DB.

Also thinking about the posts and replies, better practice is to keep the posts and replies in separate collections as designed earlier. Because not only that you can scale them individually but also it is not a best practice to have unbounded attribute in a document, meaning an attribute who’s values have theoretically no limits. Replies is an unbounded array, so we will have a dedicated collection for that.

Conclusion

This is the second series of the post in designing applications on Azure NoSQL offerings, however one of the main point I want to clarify is the design decisions we make vary based on the NoSQL technology we pick.

Design an online forum application on Azure Table Storage


NoSQL technologies provide solutions for issues that relational databases cannot provide. At the same time, designing an application on top of a NoSQL technology requires specific technology dependent design decisions and architecture.

This post addresses the issue and explains how to model a real world problem using Azure Table Storage. This is neither an introduction to Azure Table storage nor a code sample, but this post provides the thinking behind designing applications on Azure Table Storage.

Designing in Azure Table Storage

Azure Table Storage is a column store NoSQL data store, it has 4 types of querying practices.

  1. Point Query – Query based on Partition Key and Row Key, retrieves single entity.
  2. Range Query – Query based on Partition Key and range of Row Keys, retrieves multiple entities.
  3. Partition Scan – Partition Key is used but Row Key is not known / not used in the in the query, other non-key fields might be used.
  4. Table Scan – Partition Key is not used in the query, other key fields might be used.

Scenario

Think something similar to StackOverflow or MSDN forum. (Be mindful that developing a forum in that scale requires lot more technologies and strategies other than NoSQL). But as a scenario let’s assume we’re going to build a small scale forum with the following features.

    • Forum members can post questions under categories.
    • Forum members can reply to posts.
    • Users have points based on their forum contribution.

Design

In modeling our application in the Azure Table Storage, we need to identify the tables first. Users, Posts, Replies and Categories are the main tables.

Categories table can have single partition or may be two partitions – Active and Archived.

1

Row Key has been used to store the category name, in the entity class CategoryName has IgnoreProperty attribute, which makes it virtual and there will not be a physical column called CategoryName in the table. Since category name is the Row Key under a partition there won’t be duplicates in category names within the partition.

image

Keep the fixed Partition Keys as enums, this avoids mistakes (mostly typing mistakes in dealing with strings) in defining Partition Keys.

image

A simple query (partition scan) to retrieve all Active categories.

image

 

Users table has a special design, email address and password are used as credentials. So email address should be unique across the entire Users table regardless of the Partition Key – Row Key combination. So are we going to design the Users table in a single partition with email being the Row Key ?

This is possible but it is not a good design practice, dumping millions of user records under single partition.

The strategy is simple bucketing, I define 6 partitions for the Users table with Partition Key simply being a single number, like 1 to 6. And allocate email addresses based on their first letter.

Consider, that any email address starting from ‘a’ to ‘d’ go to partition 1, email addresses starting from ‘e’ to ‘h’ go to partition 2 like shown in the table below. This achieves both the uniqueness of the email address across the table and gives the partition scalability.

aa

A simple method like below would decide the Partition Key.

image

 

Posts table would be a straight forward design with Partition Key being the category name and PostId (GUID) would be the Row Key. Posts of each category live in a separate parition.

image

Like the Category entity, Post entity class will link Partition Key and Row Key using two properties CategoryName and PostId respectively marked with the IgnoreProperty attribute. See the code snippet given below.

image

If you think, using category names as Partition Keys would outgrow the rows in a single partition since one category can have hundreds of thousands of rows, you can concatenate the category name along with the year and create partitions like Azure-2015, Azure-2016 or use any other possible variable.

But the point is, making sure that you can calculate the Partition Keys from a formula gives you the ability to limit your queries maximum to Partition Scans.

 

In this scenario, Replies table can take two highly possible designs.

First, there is no separate table for Replies, use the Posts table with an additional column called ParentId. Posts will have an empty ParentId and replies will have values for ParentId of the post they are made to. Replies also go to the same partition as Posts.

Second design is having a separate table for Replies – I would personally go for this design as we can have more detailed information specific to replies.

Partition Key would be the category name and Row Key would be the Reply ID. PostId would be another column. So in order to find the replies of a Post we would trigger a Partition Scan.

Conclusion

Designing an application on top of any NoSQL technology requires specific planning and architecture based on the domain and the selected NoSQL platform. The knowledge of the underlying NoSQL technology is very essential in order to make the design efficient.

For example, in the above design if we get a requirement to show the recent 20 posts in the home page, regardless of the category, this would definitely trigger a Table Scan and also we have to bring all the posts and sort it based on the TimeStamp property.

So a good decision would be having another temporary table to keep the top 20 posts, when a new post is added the Id of the post will be updated in that table and removing the last old one. We can use write behind strategies in the application to do this.

So make sure that you design the application for the technology in a correct and efficient way.

The biggest misleading point I always here in the industry is, NoSQL development is easy and takes less time. Those two arguments are subjective and also you need to compare it with some other technology, commonly they do the comparison with relational database technologies. But in my experience I don’t see any significant time savings in using a NoSQL technology. But there are other benefits for sure.

The remote server returned an error: (412) The append position condition specified was not met – Azure Append Blob issue


Azure Blob storage got a new addition recently, that is Append Blob. Based on the Microsoft documentations this is an ideal blob storage service for frequently modified data, which makes it a suitable candidate for the logging.

It is a natural tendency that we see AppendText method and we go for it for simple text based logging and you end up with this exception – The remote server returned an error : (412) The append position condition specified was not met.

This blogger has addressed the issue and provided a workaround in his post.

If you’re looking for a solution, then above link has it. If you want to read about the reason for this issue continue reading.

Investigation

I started the investigation directly by digging into the source code of the Azure Storage SDK in GitHub

In the documentation it is mentioned that AppendText method can be used only in the single write scenarios.

See this section CloudAppendBlob Methods

It is very explicit that in the SDK it has been mentioned that AppendText should be used only in the single writer scenarios, check that in line number 1739 of this file

CloudAppendBlob methods that cannot be used in concurrent access scenarios

  • AppendFromByteArray
  • AppendFromFile
  • AppendFromStream
  • AppendText

Also the Async counter parts of the above methods also cannot be used in concurrent access.

The only method that can be used in concurrent scenarios is AppendBlock / AppendBlockAsync

So Why AppendBlock is special ?

Investigating the source in GitHub it is clear that the call chain goes like this.

AppendText calls AppendFromByteArray, AppendFromByteArray calls AppendFromStream, AppendFromStream calls the internal method UploadFromStreamHelper 

UploadFromStreamHelper cannot handle concurrent scenarios.

Continuing the investigation…..

Still we need to investigate how AppendBlock can handle the concurrent requests. We pass a System.IO.Stream object to AppendBlock method and the AppendBlock method calls  WriteToSync on the passed Stream object.

Do we have a WriteToSync method in the System.IO.Stream ? No.

The Storage SDK has a implementation of WriteToSync as an extenstion method. See this file – line number 65

It is clear that WriteToAsync  has a synchronized call to the blob using the ManualResetEvent, so that is why AppendBlock could handle the concurrent access, but remember this is a blocking call.