Integrate Cisco TelePresence Management Suite Extension for Exchange with Exchange 2016

This article explains on integrating Cisco Telepresence Suite with Exchange Server 2016. Before that lets have a brief on these components.

Cisco Telepresence Management Suite (Core Component of Video Collaboration):

This component in the Cisco IPT infrastructure provides the on-premises video collaboration.By this component we would be able to configure, Deploy, manage ,schedule , analyze and track the telepresence utilization  within an organization.

Cisco TMS helps in the following:

1) Helps Admins in the daily operations, configuration and maintainence of the telepresence network.
2) Helps consumers to use these telepresence network according to their customization.Like telepresence deployment as a service Example : Setting up meeting rooms of multi-monitors, multi-microphones and multi-channel speaker systems which gives stunning real like audio,video experience.
3) Helps in monitoring the Telepresence utilization and analyzing them.

What is Cisco TelePresence Management Suite Extension for Microsoft Exchange ?

Cisco TelePresence Management Suite Extension for Microsoft Exchange (Cisco TMSXE) is an extension for Cisco TelePresence Management Suite that enables videoconference scheduling via Microsoft Outlook, and replicates Cisco TMS conferences to Outlook room calendars.

Cisco TelePresence Management Suite Extension for Microsoft Exchange (Cisco TMSXE) is one of their extension for Cisco TelePresence Management Suite.

How it helps us in Scheduling the Meeting :

1)By having this it enables the option to Video Conferencing Scheduling via Microsoft Outlook.
2)Replicates Cisco TMS conferences settings to Outlook Room Calendars.
3)Makes end users to book Audio/Video conferences based on the Meeting room Availability from Outlook.

Cisco TMSXE Installation:

This Cisco TMSXE server runs on Windows server Cisco TMSXE component will be installed on this server along with booking service option chosen.
It similarly uses the IIS as web server. Enable https on the Default Website after the installation.

All the other configurations in Cisco components required for this integration like integrating with CUCM , CMS must be configured on the Cisco TMSXE and Cisco TMS server. There are more configurations on the TMS and TMSXE componenets which needs to be performed before integrating with Exchange Server.

In a small deployment the Cisco TMS and its extensions can be co-located on the same server.
In large scale deployments Cisco TMSXE extensions is seperate and remote SQL instance is required. And seperate Cisco TMS and Cisco TMSPE are always co-resident.

DNS Requirements:

The Cisco TMSXE server must be present on the same server VLAN where we have AD,Exchange servers.
The communication will be authenticated using the Cisco TMSXE Exchange service user account.

EWS and Autodiscover must be reachable from the TMS and TMSXE server for them to function.

Licensing:

Each telepresence endpoints to be booked through Cisco TMSXE must be licensed for general Cisco TMS usage.

In our case from Exchange perspective only the Meeting rooms where we need telepresence to be enabled must have the license.

Supported Exchange Server Versions:

  1. Office 365 ( Active Directory Federation Services and the Windows Azure Active Directory Sync tool are required)
  2. Exchange Server 2016 CU1  (latest CU’s preferred)
  3. Exchange Server 2013 SP1  (latest CU’s preferred)
  4. Exchange Server 2010 Sp3  (Latest Roll-ups preferred)
  5. Exchange Server 2007   (Latest Roll-ups preferred)

Exchange Requirements:

  1. TMSXE purely depends upon Exchange  AutoDiscover and EWS components to show the configured resource mailboxes availability
  2. Room Mailboxes added to Cisco TMSXE must have below configurations
  3. a)Delete the subject
    b)Add the organizer’s name to the subject
    c)Remove the private flag on an accepted meeting

    3.Cisco TMSXE Service Account with Mailbox is required.This service account will be used in Cisco TMS to connect to Exchange, Cisco TMSXE and Cisco TMS.

Enable impersonation for the service user in Exchange to prevent throttling issues.

To enable impersonation run the below command:
New-ManagementRoleAssignment –Name:impersonationAssignmentName – Role:ApplicationImpersonation –User:[ServiceUser]

Certificate Requirements:

Https is the default communication protocol for communicating with Cisco TMS and with Exchange Web Services.

Certificate can be issued from a Trusted CA , since this is only server to server communcation between the Exchange CAS services (EWS/AutoDiscover) and TMSXE services no public SSL is required.

So the TMSXE server certificate issued from Trusted CA should have the below:

  1. Should have the host name of the TMSXE server.
  2. Should have the host name of the Exchange servers for the EWS and Autodiscover services in secure communication.

To verify that we have certificates that are valid and working:
1. Launch Internet Explorer on the Cisco TMSXE server.
2. Enter the URL for the Exchange CAS and verify that the URL field turns green.
3. Enter the URL for the Cisco TMS server and verify that the URL field turns green.

Below will be the Work Flow :

Cisco TMS

  1. End User Books a meeting through Outlook addin.TP.png
  2. Exchange Checks the resource Mailboxes availability and books the meeting and sends initial confirmation.
  3. Cisco TMSXE communicates with Exchange and passes them on to Cisco TMS.
  4. Cisco TMS checks system and WebEx availability and attempt to book routing resources for the telepresence.

Additional Tips:

  1. The Cisco TMS is dependent only on resource calendars which are configured for this Telepresence feature.
  2. Cisco TMSXE does not have permissions to modify the calendars of personal mailboxes.
  3. All the other configurations  required for this integration must be configured on the Cisco TMSXE and Cisco TMS server.

Thanks & Regards
Sathish Veerapandian

Exchange 2016 CU rollup readiness check fails – MSCORSVW(3404) has open files

During an Exchange CU update we were getting the below message

NGEN

Prior to this all the  Exchange servers were fully patched including  the latest .net assemblies since it was CU5 upgrade.

If we look into the task manager we can see this process running and consuming large CPU resources. This is a .net related process that does the compilation job based on the priorities it is having high priority assemblies  and low priority assemblies.

What is MSCORSVW.exe?

The .Net framework has technology  called Native Image Generator Technology (NGEN) which will speed up the process for .net apps which will run only on a periodic basis purely to improve the performance of that machine

This process MSCORSVW.exe is used by NGEN  to improve the startup performance of .NET apps. So probably after an windows update especially .net patch if we have we can see this process running only at that time and consuming more CPU.

Solution for this problem:

  1. Solution 1: We can wait for a while for this .net compilation job to complete probably 5 or 10 minutes time. Once completed if we rerun the setup  things will  go fine.
  2. Solution 2: By default, NGEN only uses one CPU core for this operation . There is an option to make this work done quickly by making it to use up to 6 cores when we require them. By doing this it will complete its compilation job quickly.

Open CMD in elevated mode and run this command from this path

c:\Windows\Microsoft.NET\Framework\v4.0.30319\ngen.exe executeQueuedItems

Untitlesd

Running the above will  Execute queued compilation jobs with extra CPU cores and make it faster.Now wait for the process to precompile all the assemblies, after a couple of minutes it will be completed.

There will be ngen log as well generated in the same location where we executed this command which we can have a look at after the job completes.

References:

https://msdn.microsoft.com/en-us/library/6t9t5wcf(v=vs.110).aspx
https://blogs.msdn.microsoft.com/dotnet/2013/08/06/wondering-why-mscorsvw-exe-has-high-cpu-usage-you-can-speed-it-up/

Thanks & Regards
Sathish Veerapandian

Failed to store data in the Data Warehouse – SCOM Reports – Exchange Microsoft.Exchange.15.MailboxStatsSubscription

Recently when we tried to generate the top mailbox statistics report with the below option available from SCOM reports we weren’t able to generate them.

SCOMd

It was giving an empty report without any values.

Along with that few report data’s only for Exchange Servers like database IO reads/write  while trying too were empty with no values.

Upon looking into the operations manager log saw the below event ID.

Log Name:      Operations Manager
Source:        Health Service Modules
Date:          20.04.2017 09:36:58
Event ID:      31551
Task Category: Data Warehouse
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      SCOM1.exchangequery.com
Description:
Failed to store data in the Data Warehouse. The operation will be retried.
Exception ‘InvalidOperationException’: The given value of type String from the data source cannot be converted to type nvarchar of the specified target column.
One or more workflows were affected by this.
Workflow name: Microsoft.Exchange.15.MailboxStatsSubscription.Rule
Instance name: SCOM1.exchangequery.com
Instance ID: {466DF86F-CC39-046A-932D-00660D652716}
Management group: ExchangeQueryBy the above error we can see that this mailbox statistics subscription  rule has some problem and hence the reports were not generated.

Below 2 rules are required to be enabled to generate this report:

1) Exchange 2013: Mailbox Statistics Subscription.
2) Exchange 2013: Mailbox Statistics Collection.

SCOMd2

So by looking into the above event we can see that the SCOM is having trouble in writing the data into this target tables in the data-warehouse from the stage table.First the generated alerts are written on the operational stage table database by the SCOM. Then the operational database will insert these bulk datas into its Target DataWareHouse. It uses the option SQL bulk Insert because of the amount of data that it needs to insert from its stage table and needs to take this process.

During this process of bulk insert it will compare the value of the data that needs to be inserted with its default allowed values (NVARCHAR values for each tables). So if any of the alert titles have the values more than its default allowed limit then we will run into this problem.

This value can be seen in active stage under the columns in the operational manager database – Tables – Exchange2013mailboxstatsstaging- columns

Here we can see the nvarchar values for each properties of the mailbox which will be used to generate the mailbox statistics report from the scom 2012

SCOMd1

So here if any of  these nvarchar values which is required to generate the report value have exceeded the allowed limit then it will fail inserting the data into the datawarehouse. For example the default length of the allowed limit for Mailbox_EmailAddress is 1024.

Lets say if there is one system mailbox which has multiple smtp addresses added in them which exceeds this character limit then the  entire mailbox stats report will fail.

The SCOM requires in data type Nvarchar for Exchange because to support the unicode type for multi languages mainly. More details on SQL data types can be read here.

In our case we had a service account mailbox which had multiple SMTP addresses added on them and that exceed the allowed limit.

If any one run into the issue here is the simple command to identify the mailbox which has Email addresses of more than 1024 characters.

get-mailbox | where-object { $_.EmailAddresses.ProxyAddressString.ToCharArray().Length -ge 1024 } | foreach-object {write-host “$_”}

Once we find that mailbox we can remove that additional SMTP addresses and make the value less than 1024. After this the reports will be generating without any issues.

Another solution : ( Not Recommended)

Extend the nvarchar field values on the stage table as well as  target table (Exchange2013.MailboxProperties_) in DataWareHouse which will allow the data to get processed and generate the reports even if it has a large amount of data.

Its better not to change the default values as it might go as unsupported model , rather modifying  the mailbox and reducing the character limit which will keep everything in place without any customization.

Thanks & Regards
Sathish Veerapandian

Start-DatabaseAvailabilityGroup – Error: The network path was not found

During a DR activation the Activation went fine. But when trying to restore the main site after the DR tests are complete were getting the below error

Below was the Current state in the DR site before the restoration to main site :

Version of Exchange – Exchange 2016 CU3 with no coexistence

1) Main site was in stopped state for DAG and All main site exchange
servers were in Stopped mailbox servers list.
2) DR site was activated for DAG and only DR site exchange servers were in started mailbox servers and operational servers list.
3) All the DR copies were mounted , and users were connected.

After the DR tests were completed and trying to start the main site with below command was getting the below error :

Start-DatabaseAvailabilityGroup -ActiveDirectorySite  “MainSite”

A server-side database availability group administrative operation failed. Error The operation failed. CreateCluster errors may result from incorrectly configured static addresses. Error: An error occurred while attempting a cluster operation. Error: Cluster API ‘”AddClusterNode() (MaxPercentage=12) failed with 0x35. Error: The network path was not found”‘

Had a look at the DAG tasks logs and was getting the same above message :

Error: A server-side database availability group administrative operation failed. Error The operation failed. Create Cluster errors may result from incorrectly configured static addresses. Error: An error occurred while attempting a cluster operation. Error: Cluster API failed: “AddClusterNode() (MaxPercentage=12) failed with 0x35. Error: The network path was not found”.

Additionally was getting this message in the DAG Task logs

WriteError! Exception = Microsoft.Exchange.Management.Tasks.FailedToStartNodeException: Start-DatabaseAvailabilityGroup failed to start server

Solution :

Followed the below blog steps and it worked :

https://amagsmb.wordpress.com/2015/09/16/problem-adding-a-second-server-to-dag-error-cluster-api-addclusternode-maxpercentage12/

The Remote Registry Service should have the Startup type set to Automatic and be started.
An SP, Windows update or RU installation will put the service in a disabled state and it might be in that state after the update. In my case the main site servers OS were patched last week and post the patches these services might have gone to disabled state. While stopping  and evicting these nodes on activating the DR site there were no issues this was strange and went smooth without any issues.

The real issue happened  only when we tried to activate the main site on re adding these servers back to the DAG group.

Reason:

Some Exchange EMS/Power Shell functions, such as managing diagnostics logging requires the remote registry service to be enabled. So the Exchange required this service on the  remote servers to add them on the node. If this service is not started then the servers will not join in the DAG.

Thanks & Regards
Sathish Veerapandian

IMAP connection error – UID corruption detected

Recently in one of the IMAP application were getting complaints on accessing the emails via IMAP.

So enabled the imap logging to see the results

Set-IMAPSettings -Server “MBXservname” -ProtocolLogEnabled $true

After going through the lots we were getting the below error message UidCorruptionDetected

imap

Reason for this error:
Don’t access a mailbox with outlook web access while Outlook has an open connection to that mailbox using the IMAP client protocol .
If you do leave Outlook with IMAP access to a mailbox and access that mailbox from an alternate client, you might have some UID errors to accept when you get back to Outlook.
Example when an application requires either POP/IMAP connection to retrive the emails from Exchange better to choose any one of the connection type and not to access from multiple locations with different protocols.

Also another reason is if the IMAP account is configured on a application and if the application is configured to receive thousands of emails  daily, each and every time when the connection is established the  client app will try to access the entire set of emails. This will make this user account to exceed all the IMAP connection types and will make this mailbox logical UID corruption. So if we have any application accessing using IMAP connection then we need to make sure that an automation job from the application is configured on the client  side to purge the older emails which will solve the problem.

Solution :

Since its mailbox corruption , repairing the IMAPID corruption type will solve this problem.

New-MailboxRepairRequest -Mailbox “mailboxname” -CorruptionType ImapID

Thanks & Regards
Sathish Veerapandian

Integrate Exchange 2016 in Cisco Unity Connection Manager for Voice Mail feature

The Cisco unified messaging feature provides voice mail feature for Cisco Unity Connection client users. When this single inbox feature is enabled on Unity Connection, to the user mailbox in Unity Connection and then the mails are replicated to the user mailbox on Exchange the mails are replicated to the user mailbox on Exchange.

With Cisco Unified Messaging service users can access their Voice Mail in 2 ways :

Access their Voicemail from the Exchange inbox and listen them via computer speaker.
Directly from the phone interface.

Though there are multiple other options like text to speech, integrating with calendar options  that can be done, but this topic focuses only on the voice mail part considering the article will become too lengthy if other components are explained.

Cisco Unified Messaging service supports the Following Exchange servers:

  1. Exchange on premise versions Exchange 2007, Exchange 2010, Exchange 2013 & Exchange 2016.
  2. Office 365.

Single Inbox :
Single inbox is the Feature Name in Cisco Unity Connection Manager
This supports the Synchronization of voice messages between the CUCM and Exchange/Office 365.Before we configure on Exchange the Single inbox feature in unity connection needs to be configured.

How it establishes connection with Exchange :

Unified Messaging Service is the component in CUCM which defines the connection and establishes the communication between exchange and CUCM for enabling this feature and delivering the voice mail to end user mailboxes.

Below is the High Level Architecture of how it works:

CiscoUnityHLA.png

Components involved in them are :

Unity Connection Publisher.
Unity Connection Subscriber.
ExpressWay-E.
ExpressWay-C.
Microsoft Exchange.
Active Directory.
DNS records.
SSL Certificate for EWS/AutoDiscover.

Prerequisites  from the Exchange Side :

The Unified Messaging service can connect to Exchange server in 2 ways:

1) We can select a specific Exchange server to communicate.

If we select a specific Exchange server, Unity Connection sometimes detects when we move mailboxes from one Exchange server to another, and automatically access the Exchange mailbox in new database and server. In scenarios ,When Unity Connection cannot detect the new mailbox, then we must manually update unified messaging services or unified messaging accounts. So its better not to go with this option.
2) We can make unity connection to search for Exchange Servers.

If we allow Unity Connection to search for Exchange servers automatically , then we  need to give permissions from the Exchange servers through RBAC for the unity service account.

  1. One Unified Messaging  Active Directory account needs to be created. This AD account  will be configured on the Cisco Unified Messaging service to perform this operation.
  2. A dedicated RBAC role ApplicationImpersonation role needs to be created and assigned only to this account.
  3.  Cisco Unified Messaging service uses Autodiscover and EWS protocol for this service to work. So all the end user clients needs to have access to this protocol
  4. Exchange server SSL certificate for EWS and Autodiscover needs to be installed on  Cisco Unified Messaging service if require SSL is enabled for these 2 protocols on exchange server.

Network Requirements:
1)The Outlook clients & Cisco Jabber Clients should have perfect connection to EWS and Autodiscover which will be present by default.
2)The EWS connection between Unity connection and Exchange should be present.
3)The Autodiscover connection between Unity connection and Exchange should be present.
The default Unity Connection configuration settings is sufficient for a maximum of 2000 users and 80 milliseconds of round-trip latency between Unity Connection and Exchange servers.For more than 2000 users and/or more than 80 milliseconds of latency, we can change the default configuration as per cisco guide.

4)Unity Connection should be  configured to use DNS,  its recommend to configure Unity Connection to use the same DNS environment in which the Active Directory/exchange environment is publishing its records. If Split DNS configuration is used then all the required entries for Unity Connection needs to be configured on both the places.

Configuring Unified Messaging Services Account:

  1. This account will be used for unified messaging services to make it look generic.
  2. Do not create a mailbox for this domain user account. There are known histories where the unified messaging services not functioning properly with the mailbox.
  3. Do not add this account to any administrator group.
  4. This account must be enabled , with complex password and password never expire.

Create RBAC role for the dedicated mailbox account:

Run the below command in Exchange management shell for the created service account to get permission on exchange for searching the mailboxes.
New-ManagementRoleAssignment -Name rolename -Role:ApplicationImpersonation -User useraccount

Confirm Exchange 2016 Authentication and SSL Settings :

This part is very much required because Unity manager looks for EWS and Autodiscover for the service to work.

Authentication can be checked by logging into IIS manager in Exchange 2016 – Expand Sites – click Autodiscover and see the authentication and SSL settings

Cisco5

Check the same for EWS also.

Note: Unity Manager supports NTLM or Basic Authentication.
Depending upon the authentication setting we have on Exchange Autodiscover and EWS the same authentication must be used on the Unity manager.

If require SSL is enabled on both the protocols EWS and Autodiscover , then we need download SSL certificates from the Exchange server and install them on the Unity Connection server.

Below tasks needs to be completed in CUCM:

1) Create a new Unified Messaging Services in Unity Connection from Cisco Unity Connection Administration.
2) Upload the Exchange certificates to the Unity Connection server.
3) Enabled Unified Messaging for users in Cisco Unity Connection through Cisco Unity Connection Administration.
4) Users with Unity Connection mailboxes needs to have licenses assigned for this component through Cico Prime License Manager.

Rest detailed procedures on the above steps can be seen from the Cisco Articles for CUCM.

Thanks & Regards
Sathish Veerapandian

POP3 Error Msg=UserConnectionLimitReached

Recently in one of the Exchange 2013 environment POP3 clients started getting the problems in downloading the emails from the server.

The strange issue was users were unable to download the emails intermittently and it was not permanent for POP3 accounts.
When this issue occurs the POP3 accounts will stall for a while and later after some time it would start collecting the emails from the server without any issues.

This really looked strange and inorder to troubleshoot further enabled the POP3  protocol logging by the below command

Set-POPSettings -Server “CAS01” -ProtocolLogEnabled $true

After a while looked into the POP3 logging and strange to see the below message

ADFS1

This issue is happening, because the POP3 connections are sending more requests to the server.

This  is the main reason for the application to intermittently drop the connection.

When the connection limit  per user exceeds the default allowed limit, the connection will be forcibly closed by the mail server. And then this connection reset will happen after 4 minutes after which the client can reestablish the connection , download the emails until it reaches the threshold limit of per user.

The default value for the single user is 16

Can be seen from EAC – Servers – Edit – POP3 

Also can be seen by  running Get-POPSettings | fl

ADFS1

So the POP3 Throttling policy allow the counter reset after 24000 milliseconds . So when the user connection limit exceeds the default value he wouldn’t be able to connect till the next counter reset happens.

Solution:

So the POP Connection limit can be increased by running the below command

Set-POPSettings -MaxConnectionsPerUser  “connectionvalue”

Its important to note that both the POP services POP& backend needs to be restarted after this change to take effect,so we can go ahead and run the below command for the restart of the services.

Get-Service *POP* | Restart-Service

Additional Info:

The POP3 throttling policy value can be seen by running the below command:

Get-ThrottlingPolicy -Identity Default* | fl POP*

POPMaxConCurrency – The PopMaxConcurrency parameter specifies how many concurrent connections a POP user can have against an Exchange server at one time. A connection is held from the moment a request is received until a response is sent in its entirety to the requestor.
POpMaxBurst-  The PopMaxBurst parameter specifies the amount of time that a user can consume an elevated amount of resources before being throttled.
POPRechargeRate – The PopRechargeRate parameter specifies the rate at which the user budget is charged back
POPCutoffBalance – The PopCutoffBalance parameter specifies the resource consumption limits for a user before that user is completely blocked from performing operations on a specific component.

There were Get-WorkloadPolicy IMAP,POP commandlets present before Exchange 2013 CU6 , but later now these commandlets have been removed post CU6 and replaced with Set-SettingsOverride but strictly this Set-SettingsOverride command should be used only under the supervision of Microsoft Support professional.

These values also can be modified based on the requirement , just in case if we have any applications which requires these values to be modified as per the requirement.

Thanks & Regards
Sathish Veerapandian

%d bloggers like this: