Failed to store data in the Data Warehouse – SCOM Reports – Exchange Microsoft.Exchange.15.MailboxStatsSubscription

Recently when we tried to generate the top mailbox statistics report with the below option available from SCOM reports we weren’t able to generate them.

SCOMd

It was giving an empty report without any values.

Along with that few report data’s only for Exchange Servers like database IO reads/write  while trying too were empty with no values.

Upon looking into the operations manager log saw the below event ID.

Log Name:      Operations Manager
Source:        Health Service Modules
Date:          20.04.2017 09:36:58
Event ID:      31551
Task Category: Data Warehouse
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      SCOM1.exchangequery.com
Description:
Failed to store data in the Data Warehouse. The operation will be retried.
Exception ‘InvalidOperationException’: The given value of type String from the data source cannot be converted to type nvarchar of the specified target column.
One or more workflows were affected by this.
Workflow name: Microsoft.Exchange.15.MailboxStatsSubscription.Rule
Instance name: SCOM1.exchangequery.com
Instance ID: {466DF86F-CC39-046A-932D-00660D652716}
Management group: ExchangeQueryBy the above error we can see that this mailbox statistics subscription  rule has some problem and hence the reports were not generated.

Below 2 rules are required to be enabled to generate this report:

1) Exchange 2013: Mailbox Statistics Subscription.
2) Exchange 2013: Mailbox Statistics Collection.

SCOMd2

So by looking into the above event we can see that the SCOM is having trouble in writing the data into this target tables in the data-warehouse from the stage table.First the generated alerts are written on the operational stage table database by the SCOM. Then the operational database will insert these bulk datas into its Target DataWareHouse. It uses the option SQL bulk Insert because of the amount of data that it needs to insert from its stage table and needs to take this process.

During this process of bulk insert it will compare the value of the data that needs to be inserted with its default allowed values (NVARCHAR values for each tables). So if any of the alert titles have the values more than its default allowed limit then we will run into this problem.

This value can be seen in active stage under the columns in the operational manager database – Tables – Exchange2013mailboxstatsstaging- columns

Here we can see the nvarchar values for each properties of the mailbox which will be used to generate the mailbox statistics report from the scom 2012

SCOMd1

So here if any of  these nvarchar values which is required to generate the report value have exceeded the allowed limit then it will fail inserting the data into the datawarehouse. For example the default length of the allowed limit for Mailbox_EmailAddress is 1024.

Lets say if there is one system mailbox which has multiple smtp addresses added in them which exceeds this character limit then the  entire mailbox stats report will fail.

The SCOM requires in data type Nvarchar for Exchange because to support the unicode type for multi languages mainly. More details on SQL data types can be read here.

In our case we had a service account mailbox which had multiple SMTP addresses added on them and that exceed the allowed limit.

If any one run into the issue here is the simple command to identify the mailbox which has Email addresses of more than 1024 characters.

get-mailbox | where-object { $_.EmailAddresses.ProxyAddressString.ToCharArray().Length -ge 1024 } | foreach-object {write-host “$_”}

Once we find that mailbox we can remove that additional SMTP addresses and make the value less than 1024. After this the reports will be generating without any issues.

Another solution : ( Not Recommended)

Extend the nvarchar field values on the stage table as well as  target table (Exchange2013.MailboxProperties_) in DataWareHouse which will allow the data to get processed and generate the reports even if it has a large amount of data.

Its better not to change the default values as it might go as unsupported model , rather modifying  the mailbox and reducing the character limit which will keep everything in place without any customization.

Thanks & Regards
Sathish Veerapandian

Start-DatabaseAvailabilityGroup – Error: The network path was not found

During a DR activation the Activation went fine. But when trying to restore the main site after the DR tests are complete were getting the below error

Below was the Current state in the DR site before the restoration to main site :

Version of Exchange – Exchange 2016 CU3 with no coexistence

1) Main site was in stopped state for DAG and All main site exchange
servers were in Stopped mailbox servers list.
2) DR site was activated for DAG and only DR site exchange servers were in started mailbox servers and operational servers list.
3) All the DR copies were mounted , and users were connected.

After the DR tests were completed and trying to start the main site with below command was getting the below error :

Start-DatabaseAvailabilityGroup -ActiveDirectorySite  “MainSite”

A server-side database availability group administrative operation failed. Error The operation failed. CreateCluster errors may result from incorrectly configured static addresses. Error: An error occurred while attempting a cluster operation. Error: Cluster API ‘”AddClusterNode() (MaxPercentage=12) failed with 0x35. Error: The network path was not found”‘

Had a look at the DAG tasks logs and was getting the same above message :

Error: A server-side database availability group administrative operation failed. Error The operation failed. Create Cluster errors may result from incorrectly configured static addresses. Error: An error occurred while attempting a cluster operation. Error: Cluster API failed: “AddClusterNode() (MaxPercentage=12) failed with 0x35. Error: The network path was not found”.

Additionally was getting this message in the DAG Task logs

WriteError! Exception = Microsoft.Exchange.Management.Tasks.FailedToStartNodeException: Start-DatabaseAvailabilityGroup failed to start server

Solution :

Followed the below blog steps and it worked :

https://amagsmb.wordpress.com/2015/09/16/problem-adding-a-second-server-to-dag-error-cluster-api-addclusternode-maxpercentage12/

The Remote Registry Service should have the Startup type set to Automatic and be started.
An SP, Windows update or RU installation will put the service in a disabled state and it might be in that state after the update. In my case the main site servers OS were patched last week and post the patches these services might have gone to disabled state. While stopping  and evicting these nodes on activating the DR site there were no issues this was strange and went smooth without any issues.

The real issue happened  only when we tried to activate the main site on re adding these servers back to the DAG group.

Reason:

Some Exchange EMS/Power Shell functions, such as managing diagnostics logging requires the remote registry service to be enabled. So the Exchange required this service on the  remote servers to add them on the node. If this service is not started then the servers will not join in the DAG.

Thanks & Regards
Sathish Veerapandian

IMAP connection error – UID corruption detected

Recently in one of the IMAP application were getting complaints on accessing the emails via IMAP.

So enabled the imap logging to see the results

Set-IMAPSettings -Server “MBXservname” -ProtocolLogEnabled $true

After going through the lots we were getting the below error message UidCorruptionDetected

imap

Reason for this error:
Don’t access a mailbox with outlook web access while Outlook has an open connection to that mailbox using the IMAP client protocol .
If you do leave Outlook with IMAP access to a mailbox and access that mailbox from an alternate client, you might have some UID errors to accept when you get back to Outlook.
Example when an application requires either POP/IMAP connection to retrive the emails from Exchange better to choose any one of the connection type and not to access from multiple locations with different protocols.

Also another reason is if the IMAP account is configured on a application and if the application is configured to receive thousands of emails  daily, each and every time when the connection is established the  client app will try to access the entire set of emails. This will make this user account to exceed all the IMAP connection types and will make this mailbox logical UID corruption. So if we have any application accessing using IMAP connection then we need to make sure that an automation job from the application is configured on the client  side to purge the older emails which will solve the problem.

Solution :

Since its mailbox corruption , repairing the IMAPID corruption type will solve this problem.

New-MailboxRepairRequest -Mailbox “mailboxname” -CorruptionType ImapID

Thanks & Regards
Sathish Veerapandian

Integrate Exchange 2016 in Cisco Unity Connection Manager for Voice Mail feature

The Cisco unified messaging feature provides voice mail feature for Cisco Unity Connection client users. When this single inbox feature is enabled on Unity Connection, to the user mailbox in Unity Connection and then the mails are replicated to the user mailbox on Exchange the mails are replicated to the user mailbox on Exchange.

With Cisco Unified Messaging service users can access their Voice Mail in 2 ways :

Access their Voicemail from the Exchange inbox and listen them via computer speaker.
Directly from the phone interface.

Though there are multiple other options like text to speech, integrating with calendar options  that can be done, but this topic focuses only on the voice mail part considering the article will become too lengthy if other components are explained.

Cisco Unified Messaging service supports the Following Exchange servers:

  1. Exchange on premise versions Exchange 2007, Exchange 2010, Exchange 2013 & Exchange 2016.
  2. Office 365.

Single Inbox :
Single inbox is the Feature Name in Cisco Unity Connection Manager
This supports the Synchronization of voice messages between the CUCM and Exchange/Office 365.Before we configure on Exchange the Single inbox feature in unity connection needs to be configured.

How it establishes connection with Exchange :

Unified Messaging Service is the component in CUCM which defines the connection and establishes the communication between exchange and CUCM for enabling this feature and delivering the voice mail to end user mailboxes.

Below is the High Level Architecture of how it works:

CiscoUnityHLA.png

Components involved in them are :

Unity Connection Publisher.
Unity Connection Subscriber.
ExpressWay-E.
ExpressWay-C.
Microsoft Exchange.
Active Directory.
DNS records.
SSL Certificate for EWS/AutoDiscover.

Prerequisites  from the Exchange Side :

The Unified Messaging service can connect to Exchange server in 2 ways:

1) We can select a specific Exchange server to communicate.

If we select a specific Exchange server, Unity Connection sometimes detects when we move mailboxes from one Exchange server to another, and automatically access the Exchange mailbox in new database and server. In scenarios ,When Unity Connection cannot detect the new mailbox, then we must manually update unified messaging services or unified messaging accounts. So its better not to go with this option.
2) We can make unity connection to search for Exchange Servers.

If we allow Unity Connection to search for Exchange servers automatically , then we  need to give permissions from the Exchange servers through RBAC for the unity service account.

  1. One Unified Messaging  Active Directory account needs to be created. This AD account  will be configured on the Cisco Unified Messaging service to perform this operation.
  2. A dedicated RBAC role ApplicationImpersonation role needs to be created and assigned only to this account.
  3.  Cisco Unified Messaging service uses Autodiscover and EWS protocol for this service to work. So all the end user clients needs to have access to this protocol
  4. Exchange server SSL certificate for EWS and Autodiscover needs to be installed on  Cisco Unified Messaging service if require SSL is enabled for these 2 protocols on exchange server.

Network Requirements:
1)The Outlook clients & Cisco Jabber Clients should have perfect connection to EWS and Autodiscover which will be present by default.
2)The EWS connection between Unity connection and Exchange should be present.
3)The Autodiscover connection between Unity connection and Exchange should be present.
The default Unity Connection configuration settings is sufficient for a maximum of 2000 users and 80 milliseconds of round-trip latency between Unity Connection and Exchange servers.For more than 2000 users and/or more than 80 milliseconds of latency, we can change the default configuration as per cisco guide.

4)Unity Connection should be  configured to use DNS,  its recommend to configure Unity Connection to use the same DNS environment in which the Active Directory/exchange environment is publishing its records. If Split DNS configuration is used then all the required entries for Unity Connection needs to be configured on both the places.

Configuring Unified Messaging Services Account:

  1. This account will be used for unified messaging services to make it look generic.
  2. Do not create a mailbox for this domain user account. There are known histories where the unified messaging services not functioning properly with the mailbox.
  3. Do not add this account to any administrator group.
  4. This account must be enabled , with complex password and password never expire.

Create RBAC role for the dedicated mailbox account:

Run the below command in Exchange management shell for the created service account to get permission on exchange for searching the mailboxes.
New-ManagementRoleAssignment -Name rolename -Role:ApplicationImpersonation -User useraccount

Confirm Exchange 2016 Authentication and SSL Settings :

This part is very much required because Unity manager looks for EWS and Autodiscover for the service to work.

Authentication can be checked by logging into IIS manager in Exchange 2016 – Expand Sites – click Autodiscover and see the authentication and SSL settings

Cisco5

Check the same for EWS also.

Note: Unity Manager supports NTLM or Basic Authentication.
Depending upon the authentication setting we have on Exchange Autodiscover and EWS the same authentication must be used on the Unity manager.

If require SSL is enabled on both the protocols EWS and Autodiscover , then we need download SSL certificates from the Exchange server and install them on the Unity Connection server.

Below tasks needs to be completed in CUCM:

1) Create a new Unified Messaging Services in Unity Connection from Cisco Unity Connection Administration.
2) Upload the Exchange certificates to the Unity Connection server.
3) Enabled Unified Messaging for users in Cisco Unity Connection through Cisco Unity Connection Administration.
4) Users with Unity Connection mailboxes needs to have licenses assigned for this component through Cico Prime License Manager.

Rest detailed procedures on the above steps can be seen from the Cisco Articles for CUCM.

Thanks & Regards
Sathish Veerapandian

POP3 Error Msg=UserConnectionLimitReached

Recently in one of the Exchange 2013 environment POP3 clients started getting the problems in downloading the emails from the server.

The strange issue was users were unable to download the emails intermittently and it was not permanent for POP3 accounts.
When this issue occurs the POP3 accounts will stall for a while and later after some time it would start collecting the emails from the server without any issues.

This really looked strange and inorder to troubleshoot further enabled the POP3  protocol logging by the below command

Set-POPSettings -Server “CAS01” -ProtocolLogEnabled $true

After a while looked into the POP3 logging and strange to see the below message

ADFS1

This issue is happening, because the POP3 connections are sending more requests to the server.

This  is the main reason for the application to intermittently drop the connection.

When the connection limit  per user exceeds the default allowed limit, the connection will be forcibly closed by the mail server. And then this connection reset will happen after 4 minutes after which the client can reestablish the connection , download the emails until it reaches the threshold limit of per user.

The default value for the single user is 16

Can be seen from EAC – Servers – Edit – POP3 

Also can be seen by  running Get-POPSettings | fl

ADFS1

So the POP3 Throttling policy allow the counter reset after 24000 milliseconds . So when the user connection limit exceeds the default value he wouldn’t be able to connect till the next counter reset happens.

Solution:

So the POP Connection limit can be increased by running the below command

Set-POPSettings -MaxConnectionsPerUser  “connectionvalue”

Its important to note that both the POP services POP& backend needs to be restarted after this change to take effect,so we can go ahead and run the below command for the restart of the services.

Get-Service *POP* | Restart-Service

Additional Info:

The POP3 throttling policy value can be seen by running the below command:

Get-ThrottlingPolicy -Identity Default* | fl POP*

POPMaxConCurrency – The PopMaxConcurrency parameter specifies how many concurrent connections a POP user can have against an Exchange server at one time. A connection is held from the moment a request is received until a response is sent in its entirety to the requestor.
POpMaxBurst-  The PopMaxBurst parameter specifies the amount of time that a user can consume an elevated amount of resources before being throttled.
POPRechargeRate – The PopRechargeRate parameter specifies the rate at which the user budget is charged back
POPCutoffBalance – The PopCutoffBalance parameter specifies the resource consumption limits for a user before that user is completely blocked from performing operations on a specific component.

There were Get-WorkloadPolicy IMAP,POP commandlets present before Exchange 2013 CU6 , but later now these commandlets have been removed post CU6 and replaced with Set-SettingsOverride but strictly this Set-SettingsOverride command should be used only under the supervision of Microsoft Support professional.

These values also can be modified based on the requirement , just in case if we have any applications which requires these values to be modified as per the requirement.

Thanks & Regards
Sathish Veerapandian

Skype for Business Meeting error – Application Sharing Server has reached moderate capacity

This might occur when multiple meetings with high number of participants is initiated the Skype for Business pool is unable to process this request.

We can also see the following logs recorded in the FE servers also:

Log Name: SFB Server
Source: LS ApplicationSharing Conferencing Server
Event ID: 32032
Task Category: (1304)
Level: Warning
Description:
Internal Application Sharing Server health monitoring has detected that Application Sharing Server is running at a moderate capacity.

Application Sharing Server Health State: Loaded
Cause: Application Sharing Server has reach moderate capacity.

This application sharing parameter is controlled by MaxBandwidthPerAppSharingServiceMb

What is MaxBandwidthPerAppSharingServiceMb parameter used for ?

This value Indicates the maximum amount of bandwidth (in megabytes) set aside(allocated) for the Application Sharing Conferencing service.
MaxBandwidthPerAppSharingServiceMb can be set to any integer value between 50 and 100000, inclusive. The default value is 375 megabytes.

This setting is applied and restricted to share the bandwidth within the pool at the org level and not in the server level.
Changing this value cannot be scoped only on specific servers.

Solution for this problem:

Run Get-CsConferencingConfiguration
Look for the value MaxBandwidthPerAppSharingServiceMb.

The default value is 375 MB . Increase them to some extra value and see the results.

The value can be increased by running the below command:

Set-CsConferencingConfiguration -Identity conferencingconfigurationname -MaxBandwidthPerAppSharingServiceMb “MBSizeValue”

But the strange thing is really hard to identify this actual consumed value from the default 375MB which should be sufficient for a conferencing type for the Application sharing like a powerpoint in the presentation using Remote Desktop Protocol (RDP) protocol.

Whenever a meeting is schedule in Skype for Business server will create an entry in the Conferencing database but it will not reserve any memory in prior for the meeting which is going to occur in future. It will use the built in load balancing logic which will dynamically/instantly allocate conferencing resources on the FE servers based on the load on all FE servers and number of meeting that is occurring at the moment.

So here the value is exceeding because the pool is already running at its peak threshold level of its allocated capacity.May be if more meeting is initiated with more than 300 participants the Pool might run out of resource and this issue might occur.

The Conferencing CDR tables has the below records:

Conferences table in Skype for Business Server 2015:
Stores information about all conferences that were archived or whose details were recorded, including ConferenceURI, and start and end time.
ConferenceSessionDetails table in Skype for Business Server 2015:
Stores information about every SIP-based conference session, including start and end time, user ID, response code, and diagnostic ID for each session.
FocusJoinsAndLeaves table in Skype for Business Server 2015:
Stores information about conference joins and leaves, including users’ role and client version.
McuJoinsAndLeaves table in Skype for Business Server 2015:
Stores information about the A/V Conferencing Servers that are involved in a conference and the user join and leave times.

If the  organization requires meetings with greater than 250 participants, we need to plan for a dedicated pool to support the load

We can use the capacity planning conferencing for meeting – https://technet.microsoft.com/EN-US/library/mt297718.aspx

Thanks & Regards
Sathish Veerapandian

Exchange log the real client IPs in the IIS hit logs for SNAT load Balancing

In most of the cases we would like to know the Email client authentication attempts from external sources along with their source IPs.

It can be in below scenarios:

1) Frequent account lockouts happening for an email user where we would like to know the source host causing the account lockout.
2) Security team would like to collect the logs with the real ip for any future investigation for a compromised account.

In most of the cases exchange services are published through load balancer and servers are behind the load balancers. When Exchange is load balanced at layer 7, it will become non-transparent. Due to this the the actual client source IP address is replaced by the load balancer’s own IP address, and therefore ONLY this address will be recorded in the IIS logs.
As a result of this the Microsoft IIS client logs in the Exchange  for each client connections will have the assigned load balanced IP recorded rather than the actual source IP.

For example if the exchange services are published via SNAT  through a load balancer like KEMP, F5 etc.., the IIS logs  cannot get the real source ip. Because in a SNAT, the destination IP address is maintained but the  actual source IP address is changed.

Example of SNAT :

SNAT

When a packet passes through a NAT device Either source or destination IP address is changed/modified according to the type of NAT it is using. However the information about these changes made to packets are maintained in NAT device’s connection table

There is an option in the most of load balancers like KEMP , F5 to create an X-Forwarded header and enable them.

Once done The X-Forwarded-For header option when enabled will capture the source address of the client and append it in the header.

After this we need to add an extra value in the advanced logging module on all exchange servers to enable to log this real IP on the IIS logs.

Enable Advanced Logging on all Exchange 2016 Servers perform the below:

The first task is to deploy the Custom Logging role service. If we do not deploy this role service, we may receive an error  “Feature not supported” error when trying to edit the custom log definition.

To enable the Custom Logging role service in Windows server   2012  R2 & 2016 :
1. Open Server Manager.
2. Click Add Roles and Features.
3. In the Add Roles and Features wizard navigate to Custom Logging Role which  is under the Web Server > Web Server > Health and Diagnostics category.
4. On the Confirmation page, click Install.

Now Open IIS Manager- Select Logging

Untitled.png

 

Select Fields

Untitled1

 

Create a new custom field-

Field Name – we can give any name so that it will reflect on the logs as new column

Source Type – Request Header

Source – X-FORWARDED-FOR

Untitled2
Perform an IIS reset after this.Now we will start seeing the IP address of the client PC’s in our IIS logs rather than the IP of the load balancer.

 

%d bloggers like this: