Friday, August 1, 2008

Exchange Back Pressure

After setting up an Exhange 2007 test environment on a Virtual Machine, I noticed an interesting issue with the routing of mail. Actually no mail routed at all, and the user connected whether it be via Outlook or OWA could not send any emails. No warnings, error messages or NDRs were evident to the user and the emails would simply remain within their "DRAFTS" folder.

During my investigation on the Exchange Mailbox server. I noticed a warning within the application log for an

EventID 1009 - warning event with the Source and Category Labeled as "MSExchangeMailSubmission". The description of the event was as follows "The Microsoft Exchange Mail Submission Service is currently unable to contact any Hub Transport Server in the local Active Directory Site. The servers may be too busy to accept new connections at this time".

After discover of the warning on the Mailbox Server I investigated the Hub Transport application log and discovered a warning EventID 15002, Source MSExchangeTransport, Category: ResourceManager. Although a warning this targeted the source of the problem, "Resources". More indepth look at the warning displayed the following information:

The resource pressure is constant at High. Statistics: Queue database and disk space ("C:\Program Files\Microsoft\Exchange Server\TransportRoles\data\Queue\mail.que") = 67% [High] [Normal=62% MediumHigh=64% High=66%] Queue database logging disk space ("C:\Program Files\Microsoft\Exchange Server\TransportRoles\data\Queue\") = 67% [Normal] [Normal=91% MediumHigh=93% High=95%] Version buckets = 1 [Normal] [Normal=40 MediumHigh=60 High=100] Private bytes = 19% [Normal] [Normal=71% MediumHigh=73% High=75%] Physical memory load = 83% [limit is 94% to start dehydrating messages.] Inbound mail submission from other Hub Transport servers, the Internet, the Pickup directory, the Replay directory, and the Mailbox server, if it is on a Hub Transport server, has stopped.Loading of e-mail from the queuing database, if available, continues.

Granted servers were not down and services were up and running; however mail was still not flowing. So I decided to do some research and I found a technet article that references this thing called "
Back Pressure" for Exchange 2007. Basically Back Pressure is a resource monitoring feature built into Exchange 2007 Hub and Edge Transport server roles, which affects message delivery depending upon the current state of system resources available.

On a Hub or Edge Server the following three levels of system resource utilization are used:
  • Normal The resource is not overused. The server accepts new connections and messages.
  • Medium The resource is slightly overused. Back pressure is applied to the server in a limited manner. Mail from senders in the authoritative domain can flow. However, the server rejects new connections and messages from other sources.
  • High The resource is severely overused. Full back pressure is applied. All message flow stops, and the server rejects all new connections and messages.

All configuration options for back pressure are available in the EdgeTransport.exe.config application configuration file that is located in the C:\Program Files\Microsoft\Exchange Server\Bin directory. The EdgeTransport.exe.config file is an XML application configuration file that is associated with the EdgeTransport.exe file. EdgeTransport.exe and MSExchangeTransport.exe are the executable files that are used by the Microsoft Exchange Transport service. This service runs on every Hub Transport server or Edge Transport server. Changes that are saved to the EdgeTransport.exe.config file are applied after the Microsoft Exchange Transport service is restarted.

So in order to get my messages flowing referenced the warning EventID 15002 description and modified the "High" level values so that the calculation of my available resources would not exceed this value. Granted, you can increase the available resources (i.e. diskspace) by adding resources, however in my case this was a VM so I decided to go risky route and increase the EdgeTransport.exe.config file values to prove the concept.

In my case I modified the following line entry "PercentDatabaseDiskSpaceUsedHighThreshold" and increased it from 80 to 95.

Once I modified, saved the file and restarted the Transport service, messages were flowing out of the client "Draft" folders. Then I noticed that mail flow seized again, so I resorted to creating another drive/volume for messaging QUEUE and Logs then changed the Queue and Logs location. Of which can ALSO be performed by modifying the EdgeTransport.exe.config file. How to Change the Messaging Queues can be referenced
HERE. Once I allocated enough space for the messaging queues, messages were flowing and Exchange was happy.

Back Pressure
Changing Messaging Queues