internals: Support characters beyond the first unicode plane (WIP)

We used 16-bit variables to store the 'character code' everywhere but
this won't let us represent anything beyond U+FFFF.

This patch changes those variables to a custom type that can be 32 or 16
bits depending on the build, and adjusts numerous internal APIs and
datastructures to match.  This includes:

 * utf8decode() and friends
 * on-screen keyboard
 * font manipulation, caching, rendering, and generation
 * VFAT code parses and generates utf16 dirents
 * WIN32 simulator reads and writes utf16 filenames

Note that this patch doesn't _enable_ >16bit unicode support; a followup
patch will turn that on for appropriate targets.

Known bugs:

  * Native players in 32-bit unicode mode generate mangled filename
    entries if they include UTF16 surrogate codepoints.  Root cause
    is unclear, and may reside in core dircache code.

Needs testing on:

 * windows simulator (16bit+32bit)

Change-Id: I193a00fe2a11a4181ddc82df2d71be52bf00b6e6
This commit is contained in:
Solomon Peachy 2024-12-17 08:55:21 -05:00
parent 94712b34d4
commit d05c59f35b
44 changed files with 480 additions and 335 deletions

View file

@ -206,8 +206,8 @@ static int prompt_filename(char *buf, size_t bufsz)
{
#define KBD_LAYOUT "abcdefghijklmnop\nqrstuvwxyz |()[]\n1234567890 /._-+\n\n" \
"\nABCDEFGHIJKLMNOP\nQRSTUVWXYZ |()[]\n1234567890 /._-+"
unsigned short kbd[sizeof(KBD_LAYOUT) + 10];
unsigned short *kbd_p = kbd;
ucschar_t kbd[sizeof(KBD_LAYOUT) + 10];
ucschar_t *kbd_p = kbd;
if (!kbd_create_layout(KBD_LAYOUT, kbd, sizeof(kbd)))
kbd_p = NULL;
@ -1002,7 +1002,7 @@ next_line:
{
bufleft = bufsz - (pctx - filenamebuf);
ctx = -1;
int ctx_x_flag_count = (LAST_CONTEXT_PLACEHOLDER
int ctx_x_flag_count = (LAST_CONTEXT_PLACEHOLDER
* ARRAYLEN(context_flags));
for (int i=0;i < ctx_x_flag_count ;i++)
@ -2058,7 +2058,7 @@ static void synclist_set(int id, int selected_item, int items, int sel_size)
}
else if (menu_id == MENU_ID(M_SETKEYS))
{
keyset.view_columns = printcell_set_columns(&lists, NULL,
keyset.view_columns = printcell_set_columns(&lists, NULL,
ACTVIEW_HEADER, Icon_Rockbox);
printcell_enable(true);
int curcol = printcell_get_column_selected();