I was recently contacted by a customer who had experienced severe memory leaks on their NAV Server instances. The NAV server instances consumed more than 90% of the available memory on the server and had to be restarted twice daily to prevent the server being overloaded.
After some initial questions about what had changed I discovered that the service instance that was leaking memory was being used as part of a process that scans the boxes in the being packed in the warehouse and prints the invoices.
- The box scanning application calls a NAV Codeunit via a Web Service.
- The Codeunit posts the Warehouse Shipments.
- The Codeunit then prints the Sales Shipment report on the server.
At peak times the NAV Server would be printing 800-1000 reports per hour. Knowing that there have been memory issues with reporting assemblies, I decided to use Performance Monitor to monitor the number of assemblies being loaded by the NAV Server.
Here is the report view of the Performance Monitor log for the NAV Server instance:
The number of Current Assemblies (13000+) confirmed my suspicions. NAV Server normally has 100-200 assemblies loaded. The Total Number of AppDomains suggested that NAV Server was creating a separate App Domain when running each report and the assemblies generated for the report were not being unloaded after the AppDomain was unloaded.
As explained in the documentation about App Domains and Assemblies , when an assembly is loaded in an App Domain it cannot be unloaded. I figured that the issue was something to do with the way the assemblies are loaded for the reports.
We asked Torben from the NAV development team if he could help us with the issue and he came back with a great answer:
- Assemblies are generated for each NAV Report object with RDLC that is run on the NAV Server.
- If the RDLC contains shared variables (this is the case in the Sales Shipment report) then a unique assembly is generated for each instance of the report object. This is to prevent the value of the Shared variables being copied from one report instance to the next.
- NAV Server is configured by default to run the reports in the current App Domain to improve report performance (speed).
- The design decision was made to prioritise speed over memory consumption for NAV solutions where reports are run on the client.
In the NAV server the behaviour is controlled by the application configuration setting NetFx40_LegacySecurityPolicy This setting determines if the report is to be loaded in a separate AppDomain. The default setting is True. If you add the setting <NetFx40_LegacySecurityPolicy enabled=”false“> to the Microsoft.Dynmaics.Nav.Server.exe.config file then the NAV Server will to unload the App Domain and the report assembly when the report is completed.
In this case it was better to conserve memory and take the slight performance hit when running the reports. The customer changed the setting and saw an immediate change in the behaviour. Memory consumption was no longer increasing during the day as the memory used for the reports was being released from the server process. Here is the Performance Monitor log report showing that after the change the AppDomains are still being used but the Report assemblies are being unloaded when the reports are completed.
An alternative solution, if you don’t want to affect all reports is to set the EnableExternalAssemblies Report property to Yes for any report that will be run frequently on the server. This will force NAV Server to run that report in a separate App Domain.
Thanks again to Torben for his help in resolving the issue.