Add configurable content name normalization with runtime Settings overrides

docs/content-naming-provider.md

@@ -33,6 +33,160 @@ We offer the following predefined naming providers. You may choose either one of
 
 Duplicated 'ReplacementChar' characters are replaced by a single character (so after the conversion the name may contain less characters than the display name). This conversion is very simple but there is a chance of non-unique name creation. The original file extension will be kept.
 
+### Optional name normalization
+
+The following independent `sensenet:contentNaming` switches all default to `false`.
+They affect `CharReplacementContentNamingProvider` only.
+
+| Setting | Effect |
+| --- | --- |
+| `ApplyCustomReplacements` | Applies the configured literal replacement pairs in array order. |
+| `RemoveDiacritics` | Removes decomposable Unicode diacritics using canonical decomposition. |
+| `Transliterate` | Uses AnyAscii 0.3.3 to transliterate Unicode text to ASCII. |
+| `Lowercase` | Converts letters using invariant casing. |
+
+The pipeline is custom replacements, diacritic removal, transliteration, lowercase,
+then the existing invalid-character replacement. Each step can be enabled separately.
+When all four switches are absent or `false`, name generation remains unchanged.
+Transliteration itself removes many diacritics. To preserve accents, disable both
+`RemoveDiacritics` and `Transliterate` and avoid custom replacements that remove them.
+A Settings veto disables its step; other enabled steps can still have overlapping effects.
+
+The Docker webapps include prepared replacement pairs in their base `appsettings.json`
+files. All four switches are explicitly `false`; the prepared list has no effect until
+`ApplyCustomReplacements` is enabled. These are configuration values, not built-in
+fallbacks in the provider or the configuration reader:
+
+```json
+{
+  "sensenet": {
+    "contentNaming": {
+      "RemoveDiacritics": false,
+      "Lowercase": false,
+      "Transliterate": false,
+      "ApplyCustomReplacements": false,
+      "ReplacementSources": [
+        "ä", "Ä", "ö", "Ö", "ü", "Ü", "ß", "ẞ",
+        "æ", "Æ", "œ", "Œ", "ø", "Ø", "ł", "Ł",
+        "đ", "Đ", "ð", "Ð", "þ", "Þ", "ħ", "Ħ",
+        "ı", "ij", "IJ", "ŋ", "Ŋ", "ŧ", "Ŧ", "ſ", "ŀ", "Ŀ",
+        "&", " "
+      ],
+      "ReplacementTargets": [
+        "ae", "AE", "oe", "OE", "ue", "UE", "ss", "SS",
+        "ae", "AE", "oe", "OE", "o", "O", "l", "L",
+        "d", "D", "d", "D", "th", "TH", "h", "H",
+        "i", "ij", "IJ", "n", "N", "t", "T", "s", "l", "L",
+        "and", "-"
+      ]
+    }
+  }
+}
+```
+
+The list covers common additional Latin letters used in European languages and
+includes German-style `ä -> ae`, `ö -> oe`, `ü -> ue`, `ß -> ss`, with uppercase
+counterparts, plus `& -> and` and spaces to hyphens. It is a starting point to
+customize for the repository's languages, not automatic language detection.
+Every configured pair applies to every input when custom replacement is enabled.
+
+- For Hungarian text, enable `RemoveDiacritics` and leave
+  `ApplyCustomReplacements` disabled. No custom Hungarian accent pairs are needed.
+- For German text, enable `ApplyCustomReplacements`. `RemoveDiacritics` can also
+  be enabled for other decomposable accents: custom replacement runs first, so
+  the German `ae`/`oe`/`ue` results are preserved.
+- For mixed-language content, the operator chooses the switches and customizes
+  the list. For example, the prepared `ö -> oe` and `ü -> ue` pairs also change
+  Hungarian `tükör` to `tuekoer` when custom replacement is enabled.
+
+With only `ApplyCustomReplacements` enabled, `Fähre & Straße schön kühl`
+becomes `Faehre-and-Strasse-schoen-kuehl`. The uppercase input `ÄÖÜẞ` becomes
+`AEOEUESS` with the prepared pairs.
+
+With `RemoveDiacritics` and `Lowercase` enabled and custom replacement disabled,
+`Árvíztűrő tükörfúrógép` becomes `arvizturo tukorfurogep`. Spaces remain under the
+default invalid-character pattern; the existing `InvalidNameCharsPattern` and
+`ReplacementChar` can be configured to replace whitespace with hyphens separately.
+If custom replacement is enabled as well, the prepared German pairs instead
+produce `arvizturo-tuekoerfurogep`.
+
+`RemoveDiacritics` alone maps `ä/ö/ü` to `a/o/u`; it does not change `ß` to `ss`.
+Other scripts can be handled by the separate `Transliterate` option.
+The original stored `DisplayName` is preserved.
+
+The replacement arrays are paired by index. Entries are case-sensitive literal
+strings, not regular expressions, and can contain multiple characters. An empty
+target string deletes a match. Sources must be non-empty strings and both arrays
+must have the same length; invalid pairs throw when custom replacement is enabled.
+If no replacement entries are loaded, the custom step does nothing.
+Use string values, not JSON null values.
+Pairs run in order and may cascade: `a -> b`, then `b -> c`, turns `ab` into `cc`.
+Custom replacement runs first, so `ä -> ae` can override the usual `ä -> a`.
+
+For Docker, the four switches are environment variables such as
+`sensenet__contentNaming__Transliterate=true`. Array elements use consecutive,
+zero-based indexes, for example `sensenet__contentNaming__ReplacementSources__0=ä`
+and `sensenet__contentNaming__ReplacementTargets__0=ae`. Do not leave gaps: array
+loading stops at the first missing index. Configuration providers merge array
+entries by index: a shorter array in a later provider does not clear remaining
+entries from the base configuration. Edit the complete paired list in the base
+`appsettings.json` when removing prepared pairs. The host package/image must contain
+these features; changing host configuration or replacement arrays requires a restart.
+
+### Runtime Settings overrides
+
+Operators can disable any configured step without restarting the repository by
+saving the global `/Root/System/Settings/ContentNaming.settings` content. Use the
+existing Settings content editing/management APIs and permissions. For example,
+this disables only transliteration:
+
+```json
+{
+  "Transliterate": false
+}
+```
+
+Use the same four switch names in the settings file. Settings are checked during
+name generation on the running repository and use the existing Settings cache
+invalidation. The rules are:
+
+| Host config | Settings value | Effective behavior |
+| --- | --- | --- |
+| `false` | Any value or absent | Disabled |
+| `true` | Absent or `true` | Enabled |
+| `true` | `false` | Disabled |
+
+Settings cannot enable a feature disabled by host configuration. Removing the
+override restores the configured behavior. Replacement arrays and the existing
+`InvalidNameCharsPattern`/`ReplacementChar` are not overridden by these settings.
+This is a global override; the naming API does not receive a content context path.
+
+### Preserved behavior and Unicode scope
+
+The stored `DisplayName`, input parameters, original extension handling, and name
+validation are unchanged. An extension retained from `originalName` is appended
+unchanged, even with normalization enabled (`NEW.TXT` with original name `old.TXT`
+becomes `new.TXT` with `Lowercase`). Changing naming configuration or Settings does
+not rename existing content; this is the standard content naming behavior.
+
+The existing `InvalidNameCharsPattern` and `ReplacementChar` still control invalid
+character and whitespace replacement. Custom replacements can replace a space
+with `-` without changing the global validation pattern. Changing the existing
+invalid-character pattern separately still affects repository-wide Name/Path validation.
+
+`RemoveDiacritics` removes combining marks in all scripts after canonical
+decomposition. It leaves `æ`, `ø`, `ß` and non-Latin letters that have no relevant
+decomposition. `Transliterate` additionally handles such Latin letters (`Æ -> Ae`,
+`Œ -> Oe`, `ß -> ss`) and other writing systems and symbols. AnyAscii works without
+language context; its output is not guaranteed to match language-specific spelling.
+It can expand symbols, introduce punctuation, or remove unsupported characters.
+The existing invalid-character replacement runs afterward. Transliteration does
+not lowercase ASCII letters; the separate `Lowercase` option controls that.
+
+Different input names can produce the same result or an empty name. Existing
+duplicate-name handling and validation apply. AnyAscii is distributed under the
+ISC license; its notice is included in the package and copied to output/publish.
+
 ## Underscore5FContentNamingProvider
 
 Contains a customized *GenerateNameFromDisplayName* method that encodes the display name. The encoding works with the standard UrlEncode .Net method, but the percent sign ('%') will be replaced with an underscore ('_') character. The names that are generated from unique names are guaranteed to remain unique. After the conversion, the name may contain more characters than the input name. The original file extension will be kept. Invalid characters are configurable in the **sensenet/contentNaming** section in the web.config.

docs/content-naming.md

@@ -122,6 +122,12 @@ Pattern must start with '[' and end with ']'. -->
 ```
 - **InvalidNameCharsPattern**: a regular expression that defines the invalid characters a Content Name may not contain. These invalid characters are automatically encoded (or replaced) during converting a display name to a name, depending on the configured [ContentNamingProvider](content-naming-provider.md). If the Name is not autogenerated (for example for Files when DisplayName Field Control is not visible in the Content View) and the user inputs a name that contains invalid characters, an error message will be displayed upon trying to save the Content. The regular expression may contain characterset with a negating clause ([^...]) thus defining allowed characters instead of invalid ones, or define invalid characters simply ([...]). In both cases the regular expression MUST start with '[' and end with ']'.
 - **ReplacementChar**: defines the character used to replace invalid characters.
+- **RemoveDiacritics**: optional diacritic removal for generated names, `false` by default. See [optional name normalization](content-naming-provider.md#optional-name-normalization) for configuration and behavior.
+- **Lowercase**: optional invariant lowercasing of generated names, `false` by default. See [optional name normalization](content-naming-provider.md#optional-name-normalization).
+- **Transliterate**: optional AnyAscii Unicode-to-ASCII transliteration, `false` by default.
+- **ApplyCustomReplacements**: optional ordered literal replacements, `false` by default. `ReplacementSources` and `ReplacementTargets` are paired string arrays.
+
+All four normalization switches can be disabled at runtime through global `ContentNaming.settings`; Settings cannot enable a switch disabled in host config. See [runtime Settings overrides](content-naming-provider.md#runtime-settings-overrides).
 
 > Note that changing InvalidNameCharsPattern will affect path validation logic in the whole Content Repository. It is therefore desired to change validation messages when changing invalid characters pattern, see [Invalid names and error messages](#Invalid-names-and-error-messages) below.
 

src/ContentRepository/ContentNamingProvider.cs

@@ -1,10 +1,12 @@
-using SenseNet.ApplicationModel;
+using AnyAscii;
+using SenseNet.ApplicationModel;
 using SenseNet.ContentRepository.Schema;
 using SenseNet.ContentRepository.Storage;
 using SenseNet.ContentRepository.Storage.Data;
 using SenseNet.Diagnostics;
 using System;
 using System.Collections.Generic;
+using System.Globalization;
 using System.IO;
 using System.Linq;
 using System.Text;
@@ -380,13 +382,61 @@ public override string GenerateNameFromDisplayName(string originalName, string d
             // convert the displayname to name: if extensions are equal, only filename is to be transformed. extension is added at the end
             var nameToConvert = origext == newext ? displayNameWithoutExtension : displayName;
 
+            if (IsEnabled(nameof(ContentNaming.ApplyCustomReplacements), ContentNaming.ApplyCustomReplacements))
+                nameToConvert = ApplyCustomReplacements(nameToConvert);
+
+            if (IsEnabled(nameof(ContentNaming.RemoveDiacritics), ContentNaming.RemoveDiacritics))
+                nameToConvert = RemoveDiacritics(nameToConvert);
+
+            if (IsEnabled(nameof(ContentNaming.Transliterate), ContentNaming.Transliterate))
+                nameToConvert = nameToConvert.Transliterate();
+
+            if (IsEnabled(nameof(ContentNaming.Lowercase), ContentNaming.Lowercase))
+                nameToConvert = nameToConvert.ToLowerInvariant();
+
             string newName = RemoveInvalidCharacters(nameToConvert);
 
             // attach original extension
             newName = string.Concat(newName, origext);
 
             return newName.TrimEnd('.', replacementChar);
         }
+        private static bool IsEnabled(string optionName, bool configuredValue)
+        {
+            // Repository settings can veto a configured feature, never enable a disabled one.
+            return configuredValue && (!Repository.Started() ||
+                Settings.GetValue("ContentNaming", optionName, defaultValue: true));
+        }
+
+        private static string ApplyCustomReplacements(string name)
+        {
+            var sources = ContentNaming.ReplacementSources;
+            var targets = ContentNaming.ReplacementTargets;
+            if (sources.Length != targets.Length || sources.Any(string.IsNullOrEmpty) || targets.Any(t => t == null))
+                throw new InvalidOperationException("ContentNaming ReplacementSources and ReplacementTargets must have " +
+                    "the same length, with non-empty sources and non-null targets.");
+
+            for (var i = 0; i < sources.Length; i++)
+                name = name.Replace(sources[i], targets[i], StringComparison.Ordinal);
+
+            return name;
+        }
+
+        private static string RemoveDiacritics(string name)
+        {
+            var result = new StringBuilder(name.Length);
+            foreach (var rune in name.Normalize(NormalizationForm.FormD).EnumerateRunes())
+            {
+                var category = Rune.GetUnicodeCategory(rune);
+                if (category != UnicodeCategory.NonSpacingMark &&
+                    category != UnicodeCategory.SpacingCombiningMark &&
+                    category != UnicodeCategory.EnclosingMark)
+                    result.Append(rune.ToString());
+            }
+
+            return result.ToString().Normalize(NormalizationForm.FormC);
+        }
+
         private string RemoveInvalidCharacters(string name)
         {
             name = Regex.Replace(name, ContentNaming.InvalidNameCharsPattern, replacementString);

src/ContentRepository/SenseNet.ContentRepository.csproj

@@ -36,6 +36,7 @@
 	</ItemGroup>
 
 	<ItemGroup>
+		<PackageReference Include="AnyAscii" Version="0.3.3" />
 		<PackageReference Include="GoogleAuthenticator" Version="3.2.0" />
 		<PackageReference Include="Microsoft.AspNetCore.Http.Abstractions" Version="2.3.0" />
 		<PackageReference Include="Microsoft.CSharp" Version="4.7.0" />
@@ -56,4 +57,10 @@
 		<ProjectReference Include="..\Storage\SenseNet.Storage.csproj" />
 	</ItemGroup>
 
+	<ItemGroup>
+		<Content Include="ThirdPartyNotices/AnyAscii.txt" CopyToOutputDirectory="PreserveNewest"
+		         CopyToPublishDirectory="PreserveNewest" Pack="true"
+		         PackagePath="contentFiles/any/any/ThirdPartyNotices/" PackageCopyToOutput="true" />
+	</ItemGroup>
+
 </Project>

src/ContentRepository/ThirdPartyNotices/AnyAscii.txt

@@ -0,0 +1,17 @@
+AnyAscii 0.3.3 - https://github.com/anyascii/anyascii
+
+ISC License
+
+Copyright (c) 2020-2025, Hunter WB <hunterwb.com>
+
+Permission to use, copy, modify, and/or distribute this software for any
+purpose with or without fee is hereby granted, provided that the above
+copyright notice and this permission notice appear in all copies.
+
+THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.

src/Storage/Configuration/ContentNaming.cs

@@ -1,5 +1,6 @@
 using System;
 using System.Linq;
+using System.Collections.Generic;
 
 namespace SenseNet.Configuration
 {
@@ -18,5 +19,55 @@ public class ContentNaming : SnConfig
 
         public static char ReplacementChar { get; internal set; } = GetString(SECTIONNAME,
             "ReplacementChar", "-").FirstOrDefault();
+
+        /// <summary>
+        /// Removes decomposable Unicode diacritics when generating a name from a display name.
+        /// Default: false.
+        /// </summary>
+        public static bool RemoveDiacritics { get; internal set; } =
+            GetValue<bool>(SECTIONNAME, nameof(RemoveDiacritics), false);
+
+        /// <summary>
+        /// Converts generated names to lowercase using invariant casing.
+        /// The original extension is preserved. Default: false.
+        /// </summary>
+        public static bool Lowercase { get; internal set; } =
+            GetValue<bool>(SECTIONNAME, nameof(Lowercase), false);
+
+        /// <summary>
+        /// Transliterates generated names to ASCII using AnyAscii. Default: false.
+        /// </summary>
+        public static bool Transliterate { get; internal set; } =
+            GetValue<bool>(SECTIONNAME, nameof(Transliterate), false);
+
+        /// <summary>
+        /// Applies the ordered ReplacementSources/ReplacementTargets pairs to generated names. Default: false.
+        /// </summary>
+        public static bool ApplyCustomReplacements { get; internal set; } =
+            GetValue<bool>(SECTIONNAME, nameof(ApplyCustomReplacements), false);
+
+        /// <summary>
+        /// Literal, case-sensitive source strings for custom replacements, in application order.
+        /// </summary>
+        public static string[] ReplacementSources { get; internal set; } =
+            GetReplacementArray(nameof(ReplacementSources));
+
+        /// <summary>
+        /// Replacement strings paired by index with ReplacementSources. Empty strings remove matches.
+        /// </summary>
+        public static string[] ReplacementTargets { get; internal set; } =
+            GetReplacementArray(nameof(ReplacementTargets));
+
+        private static string[] GetReplacementArray(string key)
+        {
+            var values = new List<string>();
+            for (var index = 0; ; index++)
+            {
+                var value = GetString(SECTIONNAME, key + ":" + index);
+                if (value == null)
+                    return values.ToArray();
+                values.Add(value);
+            }
+        }
     }
 }