NLB - Hyper-V Virtual Machines

This article begins, post VM creation and OS installation/Patching process.

Assumptions:

• Guest Operating System is Windows 2008 R2
• HyperV Host is running Windows 2008 R2
• Each VM guest has two network adapters presented to it.
• NLB Cluster will be in (UNICAST) Mode

Configuration Steps:

1. Configure Network Adapters
2. Configure Network Load Balancing
3. Validate Configuration

Scenario: Desired NLB Configuration

The image below illustrates the desired NLB topology/configuration for this article. 

Note:  This particular scenario is used for an Intranet, so the NLB and CorpNet networks are on the same network.  If we desired external access to the NLB cluster, the NLB and CorpNet would most likely be on seperate networks.

Configure Network Adapters

Note:  When running Unicast, the NIC that has NLB enabled will have its’ MAC Address overwritten (Each Node will share the MAC address assigned to the NLB Cluster).  Because of this there will be additional configuration required within HyperV for the assigned NLB network interfaces. 

1.  Install on the Guest VM Windows 2008, and install the latest patches and drivers.

2. Create DNS A-Record for the Cluster Name.  My cluster name is NLBCluster.contoso.com

3. On each Guest, in Windows Rename the network connections accordingly.  I have mine as such:

4.  On each Guest VM, in Windows, Configure each Network Adapter based on your network address scheme.  The required configuration information for each Network adapter is as follows:

• NLB NIC: IP Address and Subnet Only Configured
  •        Metric:  20
  •        WINS:  Disable Netbios

IMPORTANT:  DO NOT MANUALLY select the “network load balancing (NLB) connection”!! This will happen automatically during the NLB Cluster creation process.  If its manually selected the NLB Cluster creation will not pick up this network interface as an option.  This image is merely to serve as a reference to remove all connections accept NLB and TCP.

• Corp Network NIC: Corporate network configuration (IP, GW, DNS). 
  •        Metric:    1
• Binding Order
  • Corp Network
  • NLB
  • Remote Access Connections
  • 
  • 

My Server NIC configuration:

Server1

  Server2

5. Once the NICs are configured... Run an IPCONFIG /all and note the IP and MAC Address assignment for both Network Interfaces.

Note: Once the cluster is created, the designated cluster Network Interface will receive a new MAC Address, and NLB Property will be Enabled on the Network Interface (NLB) 

6.  IMPORTANT!!!     By default Windows 2008 has IP  Forwarding disabled.  In order for this configuration with TWO NLB NICs to work and Not Define a Gateway to adhere to best practices, you have to enable IP forwarding on the NLB NIC so that requests sent to it are forwarded to the other; otherwise backend communication will not function.  Below is the simple command line to run on each NLB Node.
Note:  Be sure to open the command prompt as “Run as Administrator”, also if you are running IPV6 make sure to substitute the ipv4 with ipv6.

netsh interface ipv4 set int “<NLB NIC NAME>” forwarding=enabled

My Scenario:    netsh interface ipv4 set int “NLB” forwarding=enabled

7.  Next proceed with creating and configuring the new NLB Cluster.

Create and Configure Network Load Balancing Cluster

This process will detail the process for creating and configuring the NLB Cluster.

1.  Log into one of the Guest Virtual Machines (NLB Nodes). In my case Server1 (First node in the cluster)

2.  Right Click Network Load Balancing Manager, and select “RUN AS ADMINSTRATOR”.

3. In the console tree, right-click Network Load Balancing Clusters, and then click New Cluster.

4. In the Host box, type the computer name (Server1.contoso.com) of the guest virtual machine that will be the first host of the cluster, and then click Connect. (Host Name, not cluster name)

5.  In the Interfaces available for configuring a new cluster list, select a network adapter for the NLB cluster, and then click Next.

6.  Under interfaces select NLB Network Interface.

Note:  If your NLB Network interface does not show up in the list, verify that the “Network Load Balancing” property is un checked within the network interface properties.  NLB will not pick up the network interface if that checkbox is already enabled.

7.  On the New Cluster: Cluster IP Addresses screen, define the virtual IP address. In my case its the IP Address assigned to

NLBCluster.contoso.com  A-Record (192.168.1.130) for the NLB cluster.

Important:  Note the MAC address in the Network Address property, and then click Next. You will use this MAC address later.

8.  Under Cluster operation mode, click Unicast, and then click Next.

9. Complete the rest of the NLB configuration, and then shut down the virtual machine.

10.  At this point you have a single node within the NLB Cluster, and the MAC Address has been changed.  To verify, run an IPCONFIG /ALL, and check to see if the NLB Network Connection changed from what was previously noted, to a new MAC Address.  In my case, mine changed from (Previous MAC: 00:15:5D:CC:16:5E to 02:BF:0A:01:0A:A0).

11.  If you didn't do it in step 7, be sure to note the new MAC Address, and shutdown the Node (Server1).

12.  Once the Guest (Server1) is shutdown, open HyperV Manager and Select the Server1 > Settings > Network Adapter (The one assigned to NLB).

13.  Under MAC Address, set the MAC Address to STATIC, and enter the MAC Address assigned to the NLB Network Interface (Note earlier).

15.  Next select “Enable spoofing of MAC addresses”, and select apply.

Add the second VM to the NLB Cluster

1. Continuing from step 15, startup the VM (Server1) and log into Windows.

2.  Open Network Load Balancing Manager (RUN AS ADMINISTRATOR).

3.  Right click the NLB cluster, and then click Add Host To Cluster.

4.  Specify the name of the new host (Server2), and then click Connect. The network adapters that are available for the host are listed at the bottom of the dialog box.

5.  Click the network adapter that you want to use for Network Load Balancing (In my case NLB), and then click Next.

6.  The IP address that is configured on this network adapter is the dedicated IP address (192.168.1.132) for this host.

7.  Complete the rest of the NLB configuration, and then shut down the virtual machine (Server2).

8.  Once the Guest (Server2) is shutdown, open HyperV Manager and Select the Server1 > Settings > Network Adapter (The one assigned to NLB).

9.  Under MAC Address, set the MAC Address to STATIC, and enter the MAC Address assigned to the NLB Network Interface (Note earlier).

10.  Next select “Enable spoofing of MAC addresses”, and select apply.

11. Next, Start the virtual machine (Server2).

12. Open Network Load Balancing Manager.

13.  Verify that the NLB cluster is up as well as both nodes.

14.  At this point you now have a two-node NLB Cluster, each node sharing the same MAC address on the NLB network interface.

Windows 2008 and NETBIOS browsing

 

It has been reported that NETBIOS is on its way out; however as most have found out, it is still a critical part of most Windows Environments.  This article will explain issues that may surface during the implementation/upgrade/migration to Windows 2008 server environments.

When installing a new Windows 2008 server or upgrading an existing server to Windows 2008, the Computer Browser service is set to disabled by default.

Issue

If you upgrade the Domain Controller that has been assigned the PDC Emulator FSMO role over transfer that role to a Windows 2008 DC, you will see the domain wide NETBIOS network browse list shrink and remote subnet machines will disappear from the list.  Eventually you may only see computers from the local subnet in the network browse list.  If you have subnets with only one server and it is Windows 2008, then you may also see inconsistent local subnet browse lists due to clients taking a master browser role and they may be rebooted, turned off, etc.

Solution

To resolve this problem, you can either set the Computer Browser service to Automatic on the DC holding the PDC role, or move the PDC role to another DC that has the Computer Browser service started.  You will need File and Printer sharing On in the Network and Sharing Center, otherwise the Computer Browser service will fail to start since the required ports will not be open.  In a multiple subnet environment, make sure WINS is configured properly so that you have the proper NetBIOS name resolution.  After making the corrections, the computers holding the master browser roles will begin to populate the browse list for the entire network.

Additional Checks;

Registry and TCP/IP configurations on the DCs for Browser parameters.

• TCP/IP Configuration:   Verify NetBIOS over TCP/IP is ENABLED
• Registry parameters: 
• HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Browser\Parameters\IsDomainMaster = TRUE
• HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Browser\Parameters\MaintainServerList = AUTO

Note:  You should reboot the PDC after making the change.

Troubleshooting

Tools to help troubleshoot the NETBIOS Browsing Issues:

• BROWSTAT.exe (Resource Kit)
• NBTSTAT (Native to the Windows system)

 

Command	Usage	Checks	Steps
net view	view the browse list available to the client machines and the servers hosting WINS	If a list of machines are presented, validate the which subnet the machines are located using “Ping”.  Does the subnet match that of the machine running the command?  Document a machines from different subnets to see what is displayed for each.  This will help in the troubleshooting process.	Some steps I use from a test machine: IPConfig /all Verify WINS Servers IPConfig /FlushDNS Flush DNS resolver cache NBTSTAT –n Check Local Names table NBTSTAT -RR Release and Refresh Registration net view Check browser list
NBTSTAT –n	Local Name Table	check status column for “Conflicts”.  If conflicts exist, you may have to clean up WINS.  Either remove the problem records or rebuild WINS DB (Easier to rebuild). Note:  If status column lists Registered or Registering you can release and refresh the registration.  NBTSTAT –RR Then check the NBTSTAT –n results.	Rebuild WINS DB> 1.  Stop WINS Service 2. Backup WINS DB to alternate location. 3.  Delete WINS directory contents 4.  Start WINS service, and the db will rebuild. 5.  Create necessary replication partners, but DONT pull from other possibly infected WINS DBs, unless they are cleaned prior.
browstat dn	To get the transports bound to the interface(s)	Use the correct interface number (ex. 1) for the next command.  Browstat getpdc 1 <Netbios domainName>
browstat GETPDC 1 contoso	Get the machine with PDC role	Run this check from machines that cannot see the browser list properly, as well as the WINS servers themselves.
browstat GETMASTER 1 contoso	Get the master browser

WINS REFERENCE MATERIAL:

• Microsoft Enterprise Networking Team:  NETBIOS Browsing across subnets may fail after upgrading to Windows Server 2008
• WINS Checklist: http://technet.microsoft.com/en-us/library/cc787830(WS.10).aspx
• WINS Troubleshooting: http://technet.microsoft.com/en-us/library/cc736478(WS.10).aspx
• WINS Best Practices: http://technet.microsoft.com/en-us/library/cc784459(WS.10).aspx
• WINS How To: http://technet.microsoft.com/en-us/library/cc787831(WS.10).aspx
• WINS Concepts:  http://technet.microsoft.com/en-us/library/cc739208(WS.10).aspx

FIM 2010 – Password Reset failing

 

ISSUE:

Ok… So you have brought up your Forefront Identity Management 2010 environment, configured policies, got password reset working and life is good.  Then down the road you of course make performance, configuration changes or tweaks to the environment, MAs, etc…  One day you initiate (or user) a password reset via a workstation “reset password” link, pass through the gate questions without any problems, enter the new password and submit.  Then to your surprise are presented with a not so intuitive error.  Wait… this worked before WTH is going on.

You do some digging, check for events on the server running the password reset and discover a slue of the following events under “Forefront Identity Manager” event node.

EventID: 3
Source: Microsoft.ResourceManagement
Details:   "PWReset activity could not connect to the directory"

After banging my head on the concrete, I recalled changes made to the environment and when, then matched them up to the last time Password Reset worked.  I recalled a change made to the AD MA in order to work around the Excessive CPU Utilization on the Synchronization Server, which was to set the ADMA to “Run in a separate process..”.   If you set the AD MA to run in a separate process, password reset fails.

SOLUTION: 

Make sure the AD MA is NOT enabled to “Run in a separate process”, and then restart the Forefront Identity Manager Synchronization Server Service (miisserver.exe).  Try another password reset and BAM, it works.

So what fixed one issue apparently damaged/disrupted another.  Until the fix for FIM 2010 is available, determine what is more important to you… a pegged processor on the Synchronization Server or Password Reset working.  I’ll leave that up to you.

FIM 2010: Approval email “This request cannot be approved or rejected…”

 

Issue:

Within a Distribution Group an approval workflow and email is generated, when you open Outlook as the approver you receive the error message below and the approve/reject buttons are dimmed 

"This request cannot be approved or rejected for the following reason(s): The sender (FIM Service Component User) is not an authorized sender of approval requests. Contact your system administrator for assistance."

Cause:

The FIM service account email address entered during the FIM Extensions and Add-ins install/configure process was not correct.  In my case when this was encountered, I entered the email address correctly for the FIMService account; however Exchange created a different PrimarySMTP address for the account, and set other as a secondary.  This caused the error.

Solution:

On the machine with the FIM Extensions and Add-ins installed, perform a “Change” on the installation.

Program and Features > Forefront Identity Manager Add-ins and Extensions > Change

During the change, enter the correct email address assigned to the FIM Service account.

Hopefully you discover this before deploying company wide.  If not, you can always make the change via Group Policy, or other Desktop management solution.

MA Attributes not listed in FIM Sync Rule

 

Ran into an issue with FIM 2010 (RTM), where custom or added attributes to Management Agents were not showing up in the list of available attributes when creating/editing a Synchronization Rule.  The event on the synchronization server is listed below:

Log Name:      Application
Source:        FIMSynchronizationService
Date:          11/29/2009 11:26:25 AM
Event ID:      6331
Task Category: MA Extension
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      fimserver.mydomain.local
Description:
A update on the configuration of a MA or MV failed to replicate to a target connector directory that is capable  of storing MA/MV configurations.  As a result, the MA/MV configuration data in this connector directory is not up to date.  Please correct the condition that causes the error, and triggers a resync by updating the password information of the target MA.
Additional information:
Error Code: 0x80230709
Error Message: (The extension operation aborted due to an internal error in FIM Synchronization Service.)
Operation: Update MV
Name of the MA to replicate: 

I exhausted all effort into try and get the attributes to show in the synchronization rule(s), and I even attempted to repair/reconfigure the FIM service and FIM Sync service installation.  

Combing through the forums, I noticed a few others experienced the same issue.  Two possible solutions existed… 1 was to repair/reconfigure the install, and the other was to completely uninstall, reinstall and rebuild.   Sorry to say that the first option did not work, but the completely uninstalling and reinstalling did work.

Now, I never got down to the true underlying reason for this hickup, so hopefully you dont experience this after investing a ton of time into building sync rules, etc.  During the reinstall I chose to create a new FIM Database, so I am not sure if restoring the existing FIM Database would surface the same issue.   All I know is that restoring the MA Configurations was ok.

The images below reference attributes present in the MA, Metaverse, etc. and they dont exist in the Synchronization Rule attribute list

.Metaverse (Person Object Attributes)

Export Attribute Flow: FIMMA

Schema Management: Attribute / Bindings

Filter Permissions

Synchronization Rule:
Destination (No mDBUseDefaults)

Synchronization Rule:

Source (No mDBUseDefaults)

Event Logs on Synchronization Server:

Log Name:      Application
Source:        FIMSynchronizationService
Date:          11/29/2009 11:26:25 AM
Event ID:      6331
Task Category: MA Extension
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      fimserver.mydomain.local
Description:
A update on the configuration of a MA or MV failed to replicate to a target connector directory that is capable  of storing MA/MV configurations.  As a result, the MA/MV configuration data in this connector directory is not up to date.  Please correct the condition that causes the error, and triggers a resync by updating the password information of the target MA.
Additional information:
Error Code: 0x80230709
Error Message: (The extension operation aborted due to an internal error in FIM Synchronization Service.)
Operation: Update MV
Name of the MA to replicate:

Exchange 2010 – 3rd party Certificate status “The certificate status could not be determined because the revocation check failed”

 

After installing the 3rd Party certificate for Exchange 2010, the certificate status of the installed certificate listed:

“The certificate status could not be determined because the revocation check failed”

Cause:

See this when Proxy is used!

Exchange 2010 uses WinHTTP to determine the validity of a certificate.  WinHTTP uses the Web Proxy Auto-Discovery Protocol (WPAD), so its possible that WinHTTP is not configured with all or any of the proxy settings listed in Internet Explorer.  To determine which settings are being used by the Exchange 2010 server, you can execute a NETSH command validate and set accordingly.  

To Check WinHTTP Proxy settings:

netsh winhttp show proxy

Resolution:

Set the correct proxy settings for winhttp:

Login to the Command shell with escalated permissions “Run as Administrator”

netsh winhttp set proxy proxy-server="http=myproxy:80;https=sproxy:80" bypass-list= "*.contoso.com"

Note:  Replace myproxy and sproxy with the name or IP of your own proxy server, and be sure to specify ports.  The bypass section is optional.

Following execution, open Exchange Server 2010 Management Console, and refresh the Server Configuration>Exchange Certificates.   The certificate should now have a status of “The certificate is valid for Exchange Server Usage”

FIM 2010 – Error during Password Reset Registration

 

After performing the Password Reset deployment tasks per http://technet.microsoft.com/en-us/library/ee534892(WS.10).aspx#reset_pswd_us_pswd_reset_portal  you log into a machine and initiate the password registration sequence.  Following the completion of the Gate questions, you receive and error stating:

“An error was encountered. Please call helpdesk or your system administrator”

Cause:

The reason for the error is that the FIM Service account does not have “READ” permissions on the “Forefront Identity Manager” certificate installed on the FIMService Server. 

Resolution:

To resolve the issue, grant the FIMService account READ permissions to the certificate designated for Forefront Identity Manager.  Assigning the permission via the certificate manager console > Manager Private Key.. process may fail with an “Access "Denied” error when initiated.  In order modify the permissions, you will have to initiate the security permissions change by running the remote process in the system account.  This can be performed by downloading PSEXEC on to the FIMService server and executing the command sequence in the order shown below:

On the server hosting FIMService, download and install psexec and execute the following procedure. In that order

psexec.exe -s -d -i cmd.exe
mmc.exe
add Cert snap-in -> local machine -> computer account
Personal store --> right click the cert --> all tasks -->manage private key
grant FIMService service account read permission.

Following the permissions change, the Password Reset Registration process should work (No reboot required).

There you go!!