Recently I had this issue where Service Fabric Application upgrade fails to deploy as expected after changing the instance count in cloud.xml. Here is what I tried and error received.

problem:-

1. Create a stateless project with latest Azure Service fabric sdk 5.5
2. Deploy first with Stateless1_InstanceCount set to –1  (default)
3. Now set Stateless1_InstanceCount to say 2 from cloud.xml and redeploy with upgrade option checked

While publishing this upgrade from visual studio, I saw a property value expected to be “true” but no clue at initial glance.

Visual studio error:-

1>—— Build started: Project: Application3, Configuration: Debug x64 ——
2>—— Publish started: Project: Application3, Configuration: Debug x64 ——
2>Started executing script ‘GetApplicationExistence’.
2>Finished executing script ‘GetApplicationExistence’.
2>Time elapsed: 00:00:01.5800095
——– Package started: Project: Application3, Configuration: Debug x64 ——
Application3 -> D:Cases_CodeApplication3Application3pkgDebug
——– Package: Project: Application3 succeeded, Time elapsed: 00:00:00.7978341 ——–
2>Started executing script ‘Deploy-FabricApplication.ps1’.
2>. ‘D:Cases_CodeApplication3Application3ScriptsDeploy-FabricApplication.ps1’ -ApplicationPackagePath ‘D:Cases_CodeApplication3Application3pkgDebug’ -PublishProfileFile ‘D:Cases_CodeApplication3Application3PublishProfilesCloud.xml’ -DeployOnly:$false -ApplicationParameter:@{} -UnregisterUnusedApplicationVersionsAfterUpgrade $false -OverrideUpgradeBehavior ‘None’ -OverwriteBehavior ‘SameAppTypeAndVersion’ -SkipPackageValidation:$false -ErrorAction Stop
2>Copying application package to image store…
2>Copy application package succeeded
2>Registering application type…
2>Register application type succeeded
2>Start upgrading application…
2>Unregister application type ‘@{FabricNamespace=fabric:; ApplicationTypeName=Application3Type; ApplicationTypeVersion=1.1.0}.ApplicationTypeName’ and version ‘@{FabricNamespace=fabric:; ApplicationTypeName=Application3Type; ApplicationTypeVersion=1.1.0}.ApplicationTypeVersion’ …
2>Unregister application type started (query application types for status).
2>Start-ServiceFabricApplicationUpgrade : Default service descriptions can not be modified as part of upgrade.
2>Modified default service: fabric:/Application3/Stateless1. To allow it, set EnableDefaultServicesUpgrade to true.
2>At C:Program FilesMicrosoft SDKsService
2>FabricToolsPSModuleServiceFabricSDKPublish-UpgradedServiceFabricApplication.ps1:248 char:13
2>+             Start-ServiceFabricApplicationUpgrade @UpgradeParameters
2>+             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2>    + CategoryInfo          : InvalidOperation: (Microsoft.Servi…usterConnection:ClusterConnection) [Start-Servi
2>   ceFabricApplicationUpgrade], FabricException
2>    + FullyQualifiedErrorId : UpgradeApplicationErrorId,Microsoft.ServiceFabric.Powershell.StartApplicationUpgrade
2>
2>Finished executing script ‘Deploy-FabricApplication.ps1’.
2>Time elapsed: 00:00:22.5520036
2>The PowerShell script failed to execute.
========== Build: 1 succeeded, 0 failed, 1 up-to-date, 0 skipped ==========
========== Publish: 0 succeeded, 1 failed, 0 skipped ==========

Upon searching in our internal discussion forum, I noticed this property needs an update from resources.azure.com or through PS.

By default, we would be having this property set to “-1” in cloud.xml or application manifest xml. The value “-1” is default and it deploys to all available nodes. At situation, we  may need to reduce the instance count, so if that this is the case follow any of the option.

Option # 1 ( Update through resources.azure.com portal )

1) From the error message it is clear that, sf cluster expects a property “EnableDefaultServicesUpgrade” to be set it true to proceed this upgrade.

2) This link talks about adding sf cluster settings from resources.azure.com portal – https://docs.microsoft.com/en-us/azure/service-fabric/service-fabric-cluster-fabric-settings  ( refer the steps at the top of the page).

3) Update your cluster settings as below and wait for atleast 30-40 mins depends on the number of nodes etc.

4) After this PUT command, you would see a small banner message saying upgrading cluster in Portal.azure.com > sf cluster overview page blade.

5) Wait till the upgrade banner goes away so that you can run the GET command from resources.azure.com to confirm this value is reflecting or not.

Option#2: ( update through  PS )

You can use the below PS to update this value.

$ClusterName= “ eg. abc.westus.cloudapp.azure.com:19000”

$Certthumprint = “xxxxxx5a813118ef9cf523a4df13d”

Connect-serviceFabricCluster -ConnectionEndpoint $ClusterName -KeepAliveIntervalInSec 10 `

-X509Credential `

-ServerCertThumbprint $Certthumprint  `

-FindType FindByThumbprint `

-FindValue $Certthumprint `

-StoreLocation CurrentUser `

-StoreName My

Update-ServiceFabricService -Stateless fabric:/KeyPair.WebService/KeyPairAPI -InstanceCount 2

https://docs.microsoft.com/en-us/powershell/module/servicefabric/update-servicefabricservice?view=azureservicefabricps 

Final step:-

After the settings update, now go back to Visual Studio (2017) and try publishing app upgrade. At this point, we should see application getting deployed without any error.

You can confirm this by checking the number of node where this app is deployed. From the service fabric explorer (SFX) portal, you could see our application deployed just in 2 nodes instead all the available nodes. 

I had 3 node cluster where I set the instance count to 2 to see the application reduction.

Note:- The only caveat here is, we won’t be seeing the SFX portal manifest having this latest instance count value reflected. It would still show “-1” which you can ignore.