You can store the application setting for an Azure Web App at two places: in the web.config and in the Azure Portal (under Configuration). Not being aware of the second option, I used Kudu to look up the web.config file to find out that the appSettings were wrong. So I looked up the release pipeline to see how environment variables were used. In the Deploy Azure App Service task, you can specify the AppSettings under Application and Configuration Settings.
I didn’t specify the Configuration Settings, so I thought I would be able to change the web.config after doing so. Unfortunately that didn’t work. Then I found the following article on docs.microsoft.com that pointed me in the right direction: variable substitution. You can change AppSettings and Connection Strings in the web.config by enabling XML variable substitution in the release pipeline task.