BUG: VMM R2 allows a duplicate VLAN in a Network Site

While testing network configuration in Virtual Machine Manager 2012 R2 with Rollup 3, I stumbled upon the following rather peculiar bug. In the following sections I will guide you through the troubleshooting process and confirmation of the bug. If you don’t have enough time to read or reproduce everything after the validation section,you can always head over to the conclusion at the end of this blog. And of course if you stumbled upon this blog because you experienced the duplicate VLAN error and just want the quick fix then head over to the ‘Solution’ section.

Validation in VMM

When you try to add an existing VLAN without subnet information into the networksite via the VMM console (insert row), the console accepts the input but it doesn’t process the change since its duplicate. This can be verified by the absence of a job and the change isn’t visible if you open up the pane again. But if you change an existing VLAN number which is already present in the site to a duplicate one, the console produces the following error: “The network site has one or more errors: Row contains a duplicated Vlan.“. 

With these two confirmations we can conclude that it should not be possible to add a duplicate VLAN in VMM. The validation check in the console doesn’t allow you to make the change and subsequently pass that change on to the underlying VMM PowerShell cmdlet which in turn processes the change. One might assume that the cmdlet has the same validation and also doesn’t allow you to add duplicate VLAN’s?!  This can  be answered with YES and NO.  I know, it’s that kind of a little and nasty bug :)  Let’s go down to the nitty and gritty details.

 Adding a duplicate VLAN via PS

First we start from scratch by creating a network site combined with a new subnetvlan with VlanID id 1.

So far so good. Now if we read out all the VLAN’s in the network site we should only see VLAN 1. If you want to add the same VLAN via the cmdlet, the VLAN can’t be added with a hash entry as an addition to the existing configuration like most Exchange cmdlets accept. First you have to read out the existing values,store it to a variable, add your change and pass on the variable to the ‘-SubnetVlan’ parameter of the ‘set-SCLogicalNetworkDefinition’ cmdlet. To do this we have to create an empty allsubnetvlan array and add the existing subnetvlans information straight from the ‘Network Site’ which we already have from the ‘networksite variable in the code above and add a new ‘SubnetVlan’ to it with the ‘new-scsubnetvlan’ cmdlet. This will result in two subnetvlans entries with the same VlanID in one array.

Both of them have the same object/typename information which can be verified with the command and ouput below. The ‘get-member’ cmdlet outputs the two items with only one TypeName and all it’s corresponding properties. All the values are also exactly the same and verified with the ‘compare-object’ cmdlet along with the ‘-IncludeEqual’ switch so we know it picks up all the values and report them back as equal.

Now it’s time to apply the change and set the allsubnetvlan array with the two subnetvlans with a VlanID of 1 to the Network Site and see what happens J

Tada, it process the change and returns the Network Site information after the change. You can clearly see the two 1’s, the VMM console also shows the two subnet vlans with the same Vlan ID.

 

Solution

When you try to add any VLAN or remove any other VLAN besides the duplicate one from the current configuration the ‘set-SCLogicalNetworkDefinition’ cmdlet gives an error. It even gives the error if you try to set the current configuration again without any change or change any other attribute in the ‘Network Site’ configuration which involves setting the SubnetVlan information again. In the following example we will read out the current configuration and try to set it again without altering the variable. Now it validates the configuration correctly, gives an error and sees the duplicate VLAN in the configuration.

To solve the incorrect configuration you have to remove the duplicate VLAN by identifying the VLAN and filter it out. We will use the ‘group-object’ for this and filter out anything with a count of two (duplicate), keep the orginal one, remove the duplicate and add the rest of the VLAN’s to the ‘allSubnetVlan’ array and set the ‘SubnetVlan’ parameter again with the filtered out ‘SubnetVlans’.

We can try to add a duplicate VLAN using one other way where won’t use the ‘SubnetVlans’ information directly from the current configuation but instead retrieve the current configuration information in a foreach and supply it to a new ‘new-SCSubnetVLan’ object and add that to the ‘allSubnetVlan’ array. After the foreach we add our duplicate VLAN to the array also with the ‘new-SCSubnetvlan’ cmdlet.

This produces the same duplicate VLAN error as above and confirms that there is a validation check but not a very good one!

Troubleshooting

At this point I got a little frustrated, what on earth is going on here. Let’s try some things to narrow it down and hopefully get closer to the real cause. First lets see what kind of object is stored under ‘SubnetVLans’ and what type it is.

As it turns out the SubnetVlans object in the ‘Network Site’ configuration is a .NET ‘System.Collections.Generic.List’ with the ‘Microsoft.Systemcenter.VirtualMachineManager.SubnetVlan’ as accepted systemtype. Now let’s add a ‘SCSubnetVLAN’ object to the variable ‘allSubnetVlan’ where we stored the ‘SubnetVlans’ from the ‘Network Site’ and see what happens.

Interesting…. When you add an element (item) with the operator  ‘+=’  to the object, the variable which is a ‘system.collections.generic.list’ converts itself to an array. Apparently PowerShell automatically picks up all the items in the generic list and adds them to a whole new array. One might aspect that the ‘SCSubnetVLAN’ would be added to the ‘system.collection.generic.list’ or at least it would add ‘system.collection.generic.list’ and the ‘SCSubnetVLAN’ to an array with a total of two items. If we try to do the latter one by manually creating the array ourselves and apply that to the ‘Network Site’ we receive the error below.

The ‘–SubnetVLan’ parameter requires an object type of ‘SubnetVlan’. Basically it expects one or a collection of ‘SubnetVlan’s’ which can be supplied with an array or a ‘generic list’ only containing these SubnetVlan objects and nothing else. With this error and acquired knowledge we can rule out a problem combining these type of lists. Let’s get back at square one and see if we can narrow things down by picking up the ‘Microsoft.SystemCenter.VirtualMachineManager.SubnetVlan’ objects in the array and combine them. In the following example we use the ‘SubnetVlan’ object from the current ‘SubnetVlans’ configuration twice and set it again.

And what if we use our own newly created SCSubnetVLan object twice?

Same error, and now if we combine the two items like we did in the beginning of the article. One item from the current configuration and one item from our newly created ‘SCSubnetVlan’.

Success! There is one thing left to rule out literally everything and that’s creating the ‘generic list’ our self and see if there is something wrong with this list when you place an ‘Microsoft.SystemCenter.VirtualMachineManager.SubnetVlan’ object in it, retrieve it and set it to an variable again.

There it is again, nothing wrong with the list or any conversion from it, ruled that one out and now we really hit the dead end in our troubleshooting.

By verifying the objects with ‘get-member’ ,’gettype() and comparing the values from both objects there apparently is still a undetectable difference between the ‘Microsoft.SystemCenter.VirtualMachineManager.SubnetVlan’ object we retrieved from the configuration and our newly created ‘Microsoft.SystemCenter.VirtualMachineManager.SubnetVlan’ object from the ‘New-SCSubnetVlan’ cmdlet.

Conclusion

It’s possible to add a duplicate VLAN without subnet information with the ‘set-SCLogicalNetworkDefinition’ if you supply one item (element) from the existing configuration combined with one item from the new configuration which is generated by the ‘new-SCSubnetVlan’ cmdlet. We can acknowledge correct validation when the duplicate VLAN objects are retrieved from the configuration or when the duplicate VLAN objects are newly created with the cmdlet. Basically the cmdlet has an improper validation check which is triggered when there is a unexplainable difference between two ‘Microsoft.SystemCenter.VirtualMachineManager.Subnet Vlan’ objects. Unfortunately, despite all the effort dissecting this bug and comparing the difference between the two objects producing it,  it is not possible to exactly pinpoint the underlying root cause.

The next step would be reading both variables out of memory with some kind of debugger and execute a raw compare against both of them if that kind of tooling were available. Or of course opening up a support case with Microsoft and redirect them to this blog to speed things up but I could only imagine you would do that if you experience a similar issue which has far more impact. Therefore this bug can be titled as a minor one since you can workaround the issue by implementing your own validation or recreate every ‘SubnetVlan’ from the current configuration with the ‘new-SCSubnetvlan’ cmdlet. And you have to wonder how likely is it that your script will add the same VLAN twice. Still it was a fun time getting to know this bug and gain more knowledge about the VMM objects and cmdlets.

Feel free to comment on this post and make me happy with some new insight or even better the solution for this peculiar and if I might say nasty little bug.