Category Archives: Exchange 2016

POP3 Error Msg=UserConnectionLimitReached

Recently in one of the Exchange 2013 environment POP3 clients started getting the problems in downloading the emails from the server.

The strange issue was users were unable to download the emails intermittently and it was not permanent for POP3 accounts.
When this issue occurs the POP3 accounts will stall for a while and later after some time it would start collecting the emails from the server without any issues.

This really looked strange and inorder to troubleshoot further enabled the POP3  protocol logging by the below command

Set-POPSettings -Server “CAS01” -ProtocolLogEnabled $true

After a while looked into the POP3 logging and strange to see the below message

ADFS1

This issue is happening, because the POP3 connections are sending more requests to the server.

This  is the main reason for the application to intermittently drop the connection.

When the connection limit  per user exceeds the default allowed limit, the connection will be forcibly closed by the mail server. And then this connection reset will happen after 4 minutes after which the client can reestablish the connection , download the emails until it reaches the threshold limit of per user.

The default value for the single user is 16

Can be seen from EAC – Servers – Edit – POP3 

Also can be seen by  running Get-POPSettings | fl

ADFS1

So the POP3 Throttling policy allow the counter reset after 24000 milliseconds . So when the user connection limit exceeds the default value he wouldn’t be able to connect till the next counter reset happens.

Solution:

So the POP Connection limit can be increased by running the below command

Set-POPSettings -MaxConnectionsPerUser  “connectionvalue”

Its important to note that both the POP services POP& backend needs to be restarted after this change to take effect,so we can go ahead and run the below command for the restart of the services.

Get-Service *POP* | Restart-Service

Additional Info:

The POP3 throttling policy value can be seen by running the below command:

Get-ThrottlingPolicy -Identity Default* | fl POP*

POPMaxConCurrency – The PopMaxConcurrency parameter specifies how many concurrent connections a POP user can have against an Exchange server at one time. A connection is held from the moment a request is received until a response is sent in its entirety to the requestor.
POpMaxBurst-  The PopMaxBurst parameter specifies the amount of time that a user can consume an elevated amount of resources before being throttled.
POPRechargeRate – The PopRechargeRate parameter specifies the rate at which the user budget is charged back
POPCutoffBalance – The PopCutoffBalance parameter specifies the resource consumption limits for a user before that user is completely blocked from performing operations on a specific component.

There were Get-WorkloadPolicy IMAP,POP commandlets present before Exchange 2013 CU6 , but later now these commandlets have been removed post CU6 and replaced with Set-SettingsOverride but strictly this Set-SettingsOverride command should be used only under the supervision of Microsoft Support professional.

These values also can be modified based on the requirement , just in case if we have any applications which requires these values to be modified as per the requirement.

Thanks & Regards
Sathish Veerapandian

Exchange log the real client IPs in the IIS hit logs for SNAT load Balancing

In most of the cases we would like to know the Email client authentication attempts from external sources along with their source IPs.

It can be in below scenarios:

1) Frequent account lockouts happening for an email user where we would like to know the source host causing the account lockout.
2) Security team would like to collect the logs with the real ip for any future investigation for a compromised account.

In most of the cases exchange services are published through load balancer and servers are behind the load balancers. When Exchange is load balanced at layer 7, it will become non-transparent. Due to this the the actual client source IP address is replaced by the load balancer’s own IP address, and therefore ONLY this address will be recorded in the IIS logs.
As a result of this the Microsoft IIS client logs in the Exchange  for each client connections will have the assigned load balanced IP recorded rather than the actual source IP.

For example if the exchange services are published via SNAT  through a load balancer like KEMP, F5 etc.., the IIS logs  cannot get the real source ip. Because in a SNAT, the destination IP address is maintained but the  actual source IP address is changed.

Example of SNAT :

SNAT

When a packet passes through a NAT device Either source or destination IP address is changed/modified according to the type of NAT it is using. However the information about these changes made to packets are maintained in NAT device’s connection table

There is an option in the most of load balancers like KEMP , F5 to create an X-Forwarded header and enable them.

Once done The X-Forwarded-For header option when enabled will capture the source address of the client and append it in the header.

After this we need to add an extra value in the advanced logging module on all exchange servers to enable to log this real IP on the IIS logs.

Enable Advanced Logging on all Exchange 2016 Servers perform the below:

The first task is to deploy the Custom Logging role service. If we do not deploy this role service, we may receive an error  “Feature not supported” error when trying to edit the custom log definition.

To enable the Custom Logging role service in Windows server   2012  R2 & 2016 :
1. Open Server Manager.
2. Click Add Roles and Features.
3. In the Add Roles and Features wizard navigate to Custom Logging Role which  is under the Web Server > Web Server > Health and Diagnostics category.
4. On the Confirmation page, click Install.

Now Open IIS Manager- Select Logging

Untitled.png

 

Select Fields

Untitled1

 

Create a new custom field-

Field Name – we can give any name so that it will reflect on the logs as new column

Source Type – Request Header

Source – X-FORWARDED-FOR

Untitled2
Perform an IIS reset after this.Now we will start seeing the IP address of the client PC’s in our IIS logs rather than the IP of the load balancer.

 

Compliance Search in Exchange 2016

Till Exchange 2013 we were using the Search-Mailbox to delete any suspicious spam emails circulated in the organization.

From Exchange 2016 there is a new component New-ComplianceSearch introducted for performing this action.
In exchange 2016, New-ComplianceSearch cmdlet was introduced to search and delete messages. There are no limits for the number of mailboxes in a single search when using New-ComplianceSearch. If you use Search-Mailbox, you can only search a maximum of 10,000 mailboxes in a single search.

Still the Search-Mailbox is applicable and working for Exchange 2016 servers as well.

Example to create compliance search:
New-ComplianceSearch -Name “New Phishing Message” -ExchangeLocation “All”

NCS

Allowed parameters are few of them but we require these two at-least for better search:

ContentMatchQuery – The ContentMatchQuery parameter specifies a content search filter and uses the KQL – keyword query language syntax

Example :

New-ComplianceSearch -Name “Remove Phishing Message” -ExchangeLocation “All” -ContentMatchQuery “‘virus’ AND ‘your account closure'”

ExchangeLocation – This parameter specifies the location to look for the search

Accepted values are:
Specific Mailbox can be mentioned.
A distribution group can be mentioned.
All – When we specify all it looks for All mailboxes.

Force – After specifying this parameter only the command executed . Not sure why this was the case.

Also there is an  option  to modify the created one by using Set-ComplianceSearch cmdlet

IMPNote:
When a new compliance search is created a shadow in-place ediscovery search will be created in In-Place eDiscovery & Hold page in the EAC like below.

NCS1
But the status will not be started and we can see this by running Get-MailboxSearch as well.

Microsoft recommends to delete this autocreated shadow In-Place eDiscovery search.
Instead run the Microsoft provided script in New-ComplianceSearch page that will convert an existing compliance search to an In-Place eDiscovery search

So when we run Get-ComplianceSearch we need to see the Compliances that we created

But When we run  Get-MailboxSearch We should not see any shadow in-placediscovery which was created f0r them.

In short below will be the procedure:

  1. Create a new compliance search.
  2. Remove the shadow in-placediscovery created for the new compliance search.
  3. Run the script provided in step 3 in this technet article – Compliance Search
  4. Start the In-Place eDiscovery search – Start-MailboxSearch
  5. Create an In-Place Hold
  6. Copy the search results
  7. Export the search results
  8. Use New-ComplianceSearchAction -SearchName “Remove Phishing Message” -Purge -PurgeType SoftDelete and delete the message

Tips:

When we run the compliance search ps1 script provided by microsoft we should enter the value of the new compliance we created as below

NCS3

While creating the inplace hold better to enter the values of all the available fields

NCS5

Once the search completed there is an option to preview the search results through delegated admin account.

 

After that the data can be exported as PST.

NCS9

Post that the New-ComplianceSearchAction command should be used to remove the emails.

Note:

  1. New-ComplianceSearch limits to deleting 10 emails per mailbox at once on a single command, though there is no limits on number of mailboxes to search.
  2. Search-Mailbox limits to deleting 10000 emails per mailbox on at once on a single command.
  3. New-MailboxSearch will be depreciated soon on future updates most likely , since this command will no longer be available on Office 365 from July 2017 as per technet source.

Thanks & Regards
Sathish Veerapandian
MVP -Office Servers & Services

Expanding the Disks on Exchange Databases

This article outlines few tips of extending the storage of exchange servers where it hosts the database and log files.

For physical Server:

Add new disks in the RAID hard-drive bay and use array management utility to install the new disks to  the existing Raid.

Then expand Raid size (using Raid utility) for these disk which needs to be expanded. After this in the Raid config utility we see the extra space.Most of the SAN systems has the option to dynamically extend the disk space allocated for the servers.

So we can check the below thing using the storage management utility that we have based on the type of RAID and storage we have (eg. netapp)

• Checking initial status of the existing drives to make sure they are healthy.
• Inserting new Hard Drives in the available slots in the hard drive bay.
• Check the Status of the new drives in the storage array management utility.
• Initialize the new disks and make it available.
Then use Disk-part and extend the disk in the windows on the Exchange hosts.

Example for extending the presented disk :
Open command prompt, type: Diskpart.exe
In DISKPART prompt, type: Select Volume 1 (Selects the volume.)
In DISKPART prompt, type: Extend Size=50000 (If you do not set a size, in example like 50GB set it will use  all of the presented size).
In DISKPART prompt, type: Exit.

Using Disk Part does not affect the system accessing the data and can be done anytime.

For VMWare:

Expand the volume size of Exchange database partition from the vSphere client.
After this the additional space will be reflected immediately on the Exchange servers on diskmgmt.msc.

Expansion of the Exchange database or log drives  in VMWare is seamless however to be safe its always recommended to have a good backup in place before making this change.

Extend the database partition on the VMWare.
Extend the Presented disk in disk management.
If the disks are assigned for VM make sure they are thick provisioned.

Most of the hosted LUNs (eg like from netapp and others) can be grown and shrunk without a single problem on the application side and other vendors are the same.

Using Disk Part does not affect the system accessing the data and can be done anytime.

For hyper v :

Switch-over all databases to one server.
Shutdown the server.
In Hyper-V, increase the disk size of all database disks.
Start the server.
After this we need to Expand the Disk in the disk manager before you move the databases back.
Move the databases back to the activate on preferred node.
Repeat for the remaining servers.

Additional tips:

1) If the primary Mailbox database is increasing its better to have a de-duplication archival solution in place which will manage the storage increase efficiently.
2)Make sure all the new presented exchange drives are MBR formatted.
3)If we are extending the disks for DAG then we need to extend the disks for all DAG members hosting the copies.
4)In larger deployments where we host multiple copies in DAG its always better to have the database disks aligned in the Mount Points only.
5)Dynamic expansion of the VHDx files are supported. Older method of Dynamically expanding VHD’s not supported.
6)Always use the file system as REFS for Exchange 2016 only for Exchange DB’s & logs. Use NTFS for Exchange binaries.
7)Microsoft recommends to use the partition structure as GPT since GPT Is a newer standard  supporting up to 128 partitions in windows and is gradually replacing MBR. MBR type partitions are still supported. MBR only works with disks up to 2 TB in size
8)Better to have a healthy backup before starting these procedures.
9)For VMWare partition expansion ensure that these VM’s are not in snapshots before extending the VMDK files.
10)Better to perform this operation on a low I\O operations period on the array.
For DAG members better to expand the disks one by one on their copies see the results and then proceed.

Thanks & Regards
Sathish Veerapandian
MVP – Office Servers & Services

OWA Error – There are too many active sessions connected to this mailbox

Recently one of the shared mailbox which resides on Exchange 2016 while trying to access from web mail the users were getting the below error.

This was a shared mailbox accessed by multiple team members.

mm

This issue happened for only one mailbox and it was fine for rest of the users.

Looked into the IIS logs for the affected mailbox and there were multiple connections coming from different sources.

IIS logs location can be found on below location
C:\inetpub\logs\logfiles\W3SVC1

Further looked  into the Event Viewer and found the event id 9646 with the below message for source MSExchangeIS
Client Type OWA Exceeded the maximum objects of 16 per session
So looked into the default connection OWA limit of the mailbox to see default values

The Default value can be seen by running the below command

Get-ThrottlingPolicy

See the values of RcaMaxConcurrency and OwaMaxConcurrency for Global Throttling Policy and the Default Throttling Policy

What is RcaMaxConcurrency ?

The RcaMaxConcurrency is a parameter which controls how many Simultaneous parallel connections an RPC Client Access user can establish against an Exchange server at same time.

These connections are considered when the server receives the request from the user until the connection is closed(Eg: The connection is considered as terminated only when the User closes the browser,goes offline,sign outs)
If users attempt to make more concurrent requests than their policy allows, the new connection attempt fails. However, the existing connections remain valid.

A valid value is an integer from 0 to unlimited. The default value is 40.

What is OwaMaxConcurrency ?

The OwaMaxConcurrency is a  parameter specifies how many concurrent connections an Outlook on the web user can have against an Exchange server at one time. A connection is held from the moment a request is received until a response is sent in its entirety to the requester. If users attempt to make more concurrent requests than their policy allows, the new connection attempt fails. However, the existing connections remain valid.

The OwaMaxConcurrency parameter has a valid range from 0 through unlimited . The default value is 20. To indicate that the number of concurrent connections should be unthrottled (no limit), this value should be set to $null.

Solution:
Create a new policy with some more values for RcaMaxConcurrency and OwaMaxConcurrency and then assign some or all users to that rather than changing the default policy

Create a new Throttling Policy
New-ThrottlingPolicy -Name HighUsage -OwaMaxConcurrency 50 -RcaMaxConcurrency 100

Apply this policy only to the affected users
Set-Mailbox -Identity tonysmith -ThrottlingPolicy HighUsage

There is one more method which will override the default throttling policy which can be applied on the registry but this will be applicable for all mailboxes :

Locate and then click the following key in the registry:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\MSExchangeIS\ParametersSystem
On the Edit menu, point to New, and then click DWORD Value.
Type Maximum Allowed Service Sessions Per User, and then press ENTER.
On the Edit menu, click Modify.
Type the decimal value that specifies the number of sessions that you want to use, and then click OK.
Exit Registry Editor.

Since this will be applicable for all mailboxes better to avoid this registry entry.

Note:
For the above behavior as a first step its always better to reach the affected end user , verify from how many devices and PC he has connected, Try to disable and re-enable the owa feature for a while and see the results. If still we keep getting the event id 9646 for the affected user then we can create a throttling policy and assign the user to the policy.

Thanks & Regards
Sathish Veerapandian
MVP – Office Servers & Services 

Frequent Popups in Outlook -The Microsoft Exchange Administrator has made a change that requires you quit and restart Outlook

This error message can  frequently appear for users after the mailbox migration from Exchange 2010 to 2013 or 2016 .

The actual cache is that this error will be coming up only for few users and it appears to be perfectly fine for rest of the users.The thing is that the Outlook will appear to be working fine , users will be able to send/receive emails except for this annoying message keeps prompting the users very often.

On Further Analysis identified that this occurs only for users who have  multiple delegated accounts mapped  under Outlook.The User mailbox resides  on different database and the mapped Delegated accounts resides on different databases.

The delegated account is not fully established the connection to the new Mailbox Databases after the migration due to some reason and the users delegated mailbox table did not receive the delegate permissions accounts information. We can further look  a deep analysis on the mailbox tables on the affected user by using MFCMAPI  and looking into ACL tables but then that will consume a lot of time.

Mostly the below two solutions will  fix this issue:

1)Recreate the Outlook profile which will reestablish the connectivity to the new databases for the delegated accounts and update the mailbox table for this user.
2)Moving the mailbox to a different database which will reset the mailbox table receive folder values , update the ACL tables for delegate accounts and solve the issue.

But still not sure what is causing this issue
Also there is one more possibility which might cause this issue
The msExchHomePublicMDB attribute on Exchange 2016 databases should not have the legacy public folder object(Exchange 2010).

If we find this value in Exchange 2016 databases we can go ahead and remove them ,Since there are no more OAB end points  that depends on PF’s and no more Outlook clients that require PF’s in Exchange 2013,2016 Environment.

Inorder to remove them perform the below:

Open ADSIEDIT.MSC – Configuration Container – Navigate to Configuration Container – Expand Services – Microsoft Exchange – Domain – Administrative Group – Exchange Admininstrative Group – Databases – Right click on the databases seen on the righ pane and choose properties – Look for msExchHomePublicMDB and if it has any values clear them. Make sure to clear this values for all the other databases we have.

$_109.jpg

Very IMP note:

This above troubleshooting is applicable only for users migrated from Exchange 2007/2010 to 2013/2016 and not for the below  scenarios in any cases.

1) Issue occurs after the mailbox was moved to a new Exchange site or forest with same Exchange versions Exchange 2010.
3) Issue occurs after Changes were made to the public folder databases in Exchange 2010.
4) Issue occurs after Changes were made to the Exchange server endpoint.
5) Lync wasn’t restarted after the mailbox was moved or after the Exchange server endpoint was changed.
6) You’re running an older version of the Outlook client.
7) The service re-balances mailboxes on databases at various sites.

Thanks & Regards
Sathish Veerapandian
MVP  – Office Servers & Services

Customize Meeting responses to HTML tag in Exchange 2016

By default when a meeting room response is received the end user receives a plain message that says your request was accepted.

This response  is ok for the internal users since they are aware of where the meeting room is located.
But when a external person or vendor is invited for the meeting it makes really difficult for that person to find the office and meeting room location.

This blog focuses on adding the meeting room location for the meeting room response in html,so that the external users can find the location of the office and the meeting room easily.

If we require to add only the additional response with basic plain text we can use the below command and add the required text message.

Set-CalendarProcessing -Identity “phoenix” -AddAdditionalResponse:$true  -AdditionalResponse:”Welcome to Phoenix Meeting Room”

But the above command will not help us in adding any html tags and company logos for the meeting response.

In order to add the custom HTML tag we can perform the below steps:

Adding html tags in meeting response is possible by accessing that resource mailbox via ECP through delegated admin account for that resource mailbox.

https://yourdomain.com/ecp/phoenix@exchangequery.com

After opening the resource mailbox via ECP navigate to settings

meeting2

After that enable the tick add additional text and add the required html tag.

Adding the direct link here will not run the HTML and show the actual links in the meeting response.The big change here from Exchange 2010 version is that we need to add the actual html code as shown in the below example.

meeting3

Just playing around with the simple html and adding the required values will suffice this requirement.

Also we can refer a background image company logo uploaded in the sharepoint sites to these meeting responses which will give a better look.

In below case have added only the office location so that the users can drive in easily and reach for the meeting and the company logo  fetched from SharePoint sites for better look with the below HTML tag.

<DIV><FONT size=2 face =Tahoma>For the office location, <A href="https://enter yourgooglemapslocationhere">Click here</A>
Address:
ExchangeQuery.
Jumeriah lake Towers
Opposite to Downtown
<div ><img src="https://exchangequery.sharepoint.com/Shared%20Documents/%24_109.jpg"></img></a></div>
</FONT></DIV>

After adding the above html  users get the meeting room location and the company logo at the bottom in their meeting response like below example.

meeting4

Make sure to use the supported  image formatting as per the below tech net source

http://technet.microsoft.com/en-us/library/bb124352.aspx#Images

Hope this helps

Thanks & Regards
Sathish Veerapandian
MVP – Office Server and Services

%d bloggers like this: