Ultimate64/II BASIC tokenizer

[radius75]

The changes introduced here are an evolution and continuation of my comments in #450. Detailed technical descriptions of specific modifications are also included in the commit messages.

Description

This PR significantly improves the user experience (UX) and accessibility of the Ultimate64/II BASIC tokenizer. These changes make utilizing the full PETSCII range intuitive through a modern visual interface, a flexible tagging system, and a modular configuration architecture.

Key Enhancements:

1. Interactive PETSCII Table & Tooltips:
   • Integrated a clickable table (0x00-0xFF) directly into index.html.
   • Added tooltips providing character descriptions and physical C64 key combinations.
   • Clicking an item in the table allows for instant insertion of characters or sequences (e.g., {delete}) into the editor.
2. Advanced Tagging System:
   • Users can now input special characters in two ways: by typing manually (e.g., {white}, {sh a}) or by selecting from the provided table.
   • This system mimics modern cross-development standards found in tools like CBM prg Studio.
3. Modular Configuration System:
   • CBMprgStudio.js (New Default): A new mapping set based on CBM prg Studio is now enabled by default, providing the familiar {} syntax.
   • Legacy Support: The original internal mapping in index.html remains untouched, allowing maintainers to decide on its future development.
   • SPECIAL.js (Control File): A configuration file that lets users specify which .js mapping to load. Leaving it empty reverts the tokenizer to the legacy internal mapping.
4. Customizable Editor Content:
   • External .js files now allow users to define the default text (instructions or example BASIC code) that appears in the editor window upon startup.
5. Abbreviations:
   • Introduced BASIC command abbreviations, matching the original C64 shorthand syntax.
6. Improved Line Addressing & 0x00 Logic:

• **Refined the logic for calculating 'Next Line' pointers to ensure stability during direct memory injection for multi-line BASIC programs. This fix ensures that parsing for the current line stops upon encountering a 0x00 byte, while the pointer to the next line is correctly calculated based on the line's actual memory offset.
• **Note for consideration: The tokenizer perhaps should continue parsing the line to its full extent even if a 0x00 byte is encountered; the correct end of the line should be determined by calculating the next line's address via link pointers. While the standard C64 editor might not create such lines, they can be manually constructed via POKE or memory manipulation to create non-standard BASIC structures. Supporting this would increase the tokenizer's flexibility for advanced users.
  .

Note on AI-assisted development & Future Refinements

• AI Assistance: Parts of this code were modified with the assistance of AI. A thorough review of the changes (especially the memory addressing and pointer logic) is recommended before merging.
• Work in Progress (WIP): Please note that tooltips and custom clickable character sets may not be 100% final. They may require future corrections if any tokenization inconsistencies or content errors are discovered during wider use.

How to test

1. Manual Entry: Type tags like {delete} or {white} directly into the editor and verify correct tokenization.
2. Interactive Table: Select and click characters from the provided table to insert them into the code and verify the output.
3. Tooltips: Hover over table elements to check the accuracy of C64 key descriptions and shortcuts.
4. Configuration & Refreshing: Change the mapping set in SPECIAL.js, then refresh the page (F5 and Ctrl+F5). Verify that the new table content, tooltips, and editor startup text load correctly.
5. Stability: Confirm that multi-line BASIC programs have correct link pointers in memory after injection.

Sample test code in CBMprgStudio format:

!- CBM prg Studio
!- --------------
!-
!- userport i/o control
!-
!- clear screen, set uppercase and disable shift + c=
!- for default: set pb0-pb7 as inputs, disable outputs activity, set temp as 0
0 print"{clear}{uppercase}{switch off}":poke56579,0:poke56577,0:te=0
!- decimal to binary precalculation
5 dimb$(255),c$(15)
10 fora=0to15:readc$(a):next
15 a=0:forh=0to15:forl=0to15
20 b$(a)=c$(h)+c$(l):a=a+1:nextl,h
25 data "0000","0001","0010","0011","0100","0101","0110","0111"
30 data "1000","1001","1010","1011","1100","1101","1110","1111"
!- screen
35 print"{clear}";
!-
50 print "          {white}userport i/o control          ";
51 print "                                        ";
52 print "               pb7>{gray}B{white}76543210{gray}B{white}<pb0       ";
53 print "{gray}{sh asterisk*12}CCCCCCC{sh +}{sh asterisk*8}{sh +}CCCCCCCCCCC";
54 print "{white}port current status{gray}B{white}--------{gray}B {white}---  56577";
55 print "{gray}{sh asterisk*12}CCCCCCC{cm w}{light gray}^^^^^^^^{gray}{cm q}CCCCCCCCCCC";
56 print "      {white}entered value{gray}B{white}--------{gray}B {white}---  56577";
57 print "{gray}{sh asterisk*13}CCCCCC{cm w}{light gray}^^^^^^^^{gray}{cm q}CCCCCCCCCCC";
58 print " {white}set port as in/out{gray}B{white}--------{gray}B {white}---  56579";
59 print "{gray}{sh asterisk*12}CCCCCCC{cm e}{sh asterisk*8}{cm e}CCCCCCCCCCC";
60 print "                                        ";
61 print "{white}set line as in-{reverse on}0{reverse off} or out-{reverse on}1{reverse off}, press {reverse on}1{reverse off}-{reverse on}8{reverse off} key";
62 print "   change output state, press {reverse on}q{reverse off}-{reverse on}i{reverse off} key   ";
63 print "{light gray}if the pb line is set as an output then ";
64 print " it has a higher short-circuit current. ";
65 print " do not connect external out to pb out. ";
66 print "                                        ";
67 print "        {black}g {red}+                   {black}g         ";
68 print "        n {red}5               {pink}9 9 {black}n         ";
69 print "        d {red}v               {pink}v v {black}d         ";
70 print "       {white}{cm a}{reverse on}{cm o}{reverse off}{cm r}{reverse on}{cm o}{reverse off}C{reverse on}{cm o}{reverse off}C{reverse on}{cm o}{reverse off}C{reverse on}{cm o}{reverse off}C{reverse on}{cm o}{reverse off}C{reverse on}{cm o}{reverse off}C{reverse on}{cm o}{reverse off}C{reverse on}{cm o}{reverse off}C{reverse on}{cm o}{reverse off}{cm r}{reverse on}{cm o}{reverse off}C{reverse on}{cm o}{reverse off}{cm s}        ";
71 print "       {cm z}{reverse on}{cm u}{reverse off}{cm e}{reverse on}{cm u}{reverse off}C{reverse on}{cm u}{reverse off}C{reverse on}{cm u}{reverse off}C{reverse on}{cm u}{reverse off}C{reverse on}{cm u}{reverse off}C{reverse on}{cm u}{reverse off}C{reverse on}{cm u}{reverse off}C{reverse on}{cm u}{reverse off}C{reverse on}{cm u}{reverse off}{cm e}{reverse on}{cm u}{reverse off}C{reverse on}{cm u}{reverse off}{cm x}        ";
72 print "        {black}g   {green}p p p p p p p p   {black}g         ";
73 print "        n   {green}b b b b b b b b   {black}n         ";
74 print "        d   {green}0 1 2 3 4 5 6 7   {black}d";

!- main loop
165 geta$:ifa$<>""thena=val(a$)-1:a2=2^a:gosub200
170 se=peek(56579):cu=peek(56577)
175 print"{home}{white}{down}{down}{down}{down}";
180 print,,b$(cu),"{left}";cu;"{left}  "
185 print,,b$(te),"{left}";te;"{left}  "
190 print,,b$(se),"{left}";se;"{left}  "
195 goto165
200 ifa<0ora>7then215
205 if(seanda2)=a2thenpoke56579,se-a2:return
210 poke56579,seora2:return
215 ifa$="q"thena2=2^0:goto260
220 ifa$="w"thena2=2^1:goto260
225 ifa$="e"thena2=2^2:goto260
230 ifa$="r"thena2=2^3:goto260
235 ifa$="t"thena2=2^4:goto260
240 ifa$="y"thena2=2^5:goto260
245 ifa$="u"thena2=2^6:goto260
250 ifa$="i"thena2=2^7:goto260
255 return
260 if(teanda2)=a2thente=te-a2:poke56577,te:return
265 te=teora2:poke56577,te:return
!- end main loop
270 rem radius75, 2024

Closes #450

[radius75]

1. SPECIAL.js / tokenizer tables

The description about falling back to the old built-in token mapping is outdated.

special.js specifies the external file containing the tokenizer tables. The old built-in TOKENS / SPECIAL mapping was intentionally removed, as it was much less complete than the token definitions used by CBMprgStudio.

If the tokenizer tables cannot be loaded correctly, there is no fallback to the old mapping. Instead, the editor displays the following message:

Data loading failed: please try reloading the page (Ctrl+F5).
Check for syntax errors in /Flash/html/special.js,
or your custom profile if modified.

I will update the description/comment so that it reflects this behavior.

---

2. Uppercase / case sensitivity

The removal of the automatic map(line => line.trim().toLowerCase()) of the tokenizer input is intentional.

The previous tokenizer converted typed/pasted text to lowercase. This made it impossible to distinguish between a character entered normally and the corresponding character entered with Shift. This distinction is important for entering C64/PETSCII characters that depend on Shift.

The current implementation preserves the original case of the input and follows the approach used by CBMprgStudio.

The existing uppercase checkbox from master is still present. It only changes how the text is displayed in the editor; it does not modify the underlying input used by the tokenizer. Therefore Shift and non-Shift characters remain distinguishable even when the checkbox is enabled.

When the checkbox is disabled, the original case is displayed again — characters entered with Shift remain uppercase and those entered without Shift remain lowercase.

So this is an intentional change to the tokenizer input handling.

[chrisgleissner]

Thanks for the explanations @radius75 .

As the change was peer reviewed, enriched with tests, and suggestions by Gideon about file naming were onboarded, I will now remove the 'Draft' flag, as it is ready for a wider review.

@GideonZ What are your thoughts about this PR?

[radius75]

I decided to bring back trim().toLowerCase() as an optional, profile-dependent setting.

This may be useful for someone creating a profile for pasting BASIC code that is already written in uppercase and does not use shifted characters.

Personally, I currently handle such code by converting it to lowercase in Notepad++. With this option, the tokenizer can now handle that conversion itself.

The cbm_prg_studio.js profile keeps NORMALIZE_TO_LOWERCASE = false, since it relies on case-sensitive abbreviations and character mappings.

The commit description and the comments in the code document how this option works and when it should be used.

[radius75]

The examples from the Commodore 64 Games Book 1 collection are particularly useful for testing this change because the programs were transcribed for use with CBMprgStudio.

The same applies to the other .bas programs in the repository, which provide additional real-world BASIC source prepared for use with CBMprgStudio.

https://github.com/TypeInBooks/books/tree/master/commodore/c64/gamesbook1

[chrisgleissner]

@radius75 Sounds great. Thanks for also making sure all tests pass on CI.

[radius75]

Thanks for also making sure all tests pass on CI.

@chrisgleissner
What kind of test supervision are you referring to?

I see that the test expects

("all", *browser_lib.BROWSERS)

but gets ("all",) + browser_lib.BROWSERS

Can this be improved? Because the tokenizer works as is.

--
Only now, after reviewing tokenizer_test.py, do I fully understand how the tests work. I realize that my recent changes require updating the tests to cover the new behavior, including case handling, abbreviations, and normalization

[chrisgleissner]

Hi @radius75 ,

I was referring to this broken build: https://github.com/GideonZ/1541ultimate/actions/runs/34208712219/job/102004311447#step:5:12

On a related note, in future, we will run both unit and e2e tests automatically on CI as part of each PR build. For the time being, it is the responsibility of anyone who raises a PR to run the unit tests (make) and E2E tests (run-tests <host>) against a locally built and flashed firmware.

This ensures that merging in PRs does not regress functionality.

Btw, thanks a lot for having spotted the Firefox regression last week. Thanks to you I created a Selenium-based E2E test for the index page. The change went live with the 3.15 release yesterday.

Best wishes
Christian

[radius75]

Hi Christian,

I don't have a local firmware build environment on Windows 7. Instead, I install the latest generated master build and manually replace index.html, special.js, and cbm_prg_studio.js on the device.

This allows me to test the tokenizer functionality on real hardware, but I understand it doesn't replace the full local build and test workflow.

[radius75]

Hi Christian,

Following our discussion about test coverage, here are two reference cases for the current tokenizer normalization behavior. They include the BASIC source, expected HEX output, and screenshots from the PC emulator and real C64.

These should help when extending the E2E tests to cover both normalization modes.

Test 1: CBMprgStudio v4.10.0 profile compatibility (NORMALIZE_TO_LOWERCASE = false)

Verify that BASIC source using uppercase/lowercase keywords and supported abbreviations produces the same tokenized bytes as CBMprgStudio. The resulting HEX output from CBMprgStudio and Ultimate must be identical.

10 PRINT"AB":GOTO20
20 print"ab":goto30
30 ?"Ab":gO40
40 rem Ab"Ab"
50 data Ab,"Ab"
60 readA$,b$
70 printA$,b$

!- >C:0801  0f 08 0a 00  99 22 c1 c2  22 3a 89 32  30 00 1d 08   
!- >C:0811  14 00 99 22  41 42 22 3a  89 33 30 00  2b 08 1e 00   
!- >C:0821  99 22 c1 42  22 3a 89 34  30 00 38 08  28 00 8f 20   
!- >C:0831  41 42 22 c1  42 22 00 46  08 32 00 83  20 41 42 2c   
!- >C:0841  22 c1 42 22  00 51 08 3c  00 87 41 24  2c 42 24 00   
!- >C:0851  5c 08 46 00  99 41 24 2c  42 24 00 00  00

Screenshot from the PC emulator showing the resulting program and its BASIC listing. The HEX output is compared byte for byte with CBMprgStudio.

---

Test 2: Legacy full-source normalization (NORMALIZE_TO_LOWERCASE = true)

Verify that the entire BASIC source is converted to lowercase before tokenization, including quoted strings and text following REM and DATA, preserving the behavior of the previous tokenizer version.

10 PRINT"AB":GOTO20
20 print"ab":goto30
30 PrInT"Ab":gOtO40
40 rEm Ab"Ab"
50 Data Ab,"Ab"
60 reaDA$,b$
70 printA$,b$

0F 08 0A 00 99 22 41 42 22 3A 89 32 30 00 1D 08 
14 00 99 22 41 42 22 3A 89 33 30 00 2B 08 1E 00 
99 22 41 42 22 3A 89 34 30 00 38 08 28 00 8F 20 
41 42 22 41 42 22 00 46 08 32 00 83 20 41 42 2C 
22 41 42 22 00 51 08 3C 00 87 41 24 2C 42 24 00 
5C 08 46 00 99 41 24 2C 42 24 00 00 00

Photo of the program running on a real C64, showing the resulting BASIC listing after full-source lowercase normalization.

[radius75]

The latest fix corrects : handling in DATA statements, allowing tokenization to resume after the colon. It also prevents $83 (DATA) and $8F (REM) bytes inside quoted strings from changing the tokenizer state.

Tested against CBMprgStudio — the generated BASIC bytes are identical.
The test can be performed with both NORMALIZE_TO_LOWERCASE = false and NORMALIZE_TO_LOWERCASE = true.

10 data 1,2 :print "after data"
20 rem test :print "after rem"
30 print "{131}":print "after 131"
40 print "{143}":print "after 143"

!- >C:0801  1b 08 0a 00  83 20 31 2c  32 20 3a 99  20 22 41 46 
!- >C:0811  54 45 52 20  44 41 54 41  22 00 39 08  14 00 8f 20 
!- >C:0821  54 45 53 54  20 3a 50 52  49 4e 54 20  22 41 46 54 
!- >C:0831  45 52 20 52  45 4d 22 00  51 08 1e 00  99 20 22 83 
!- >C:0841  22 3a 99 20  22 41 46 54  45 52 20 31  33 31 22 00
!- >C:0851  69 08 28 00  99 20 22 8f  22 3a 99 20  22 41 46 54 
!- >C:0861  45 52 20 31  34 33 22 00  00 00

[radius75]

Tests the BASIC line number limit introduced in the latest commit. Line numbers up to 65535 are accepted, while 65536 and 70000 are rejected to prevent 16-bit truncation and match CBMprgStudio behavior.

0 print "0"
1 print "1"
65535 print "65535"
65536 print "65536"
70000 print "70000"

[radius75]

Regarding the latest commit limiting the number of special character repetitions, I tested the following program:

10 print "{space*999}"
20 print "{space*1000}"
30 print "{space*900}{space*99}{space*1}"
40 print "{999 space}"
50 print "{1000 space}"
60 print "{900 space}{99 space}{1 space}"

Each line tests a different case:

10 — a single CBMprgStudio-style repetition at the safety limit (999).
20 — a single CBMprgStudio-style repetition exceeding the limit (1000).
30 — multiple CBMprgStudio-style repetitions that individually stay within the limit, but exceed it when combined (900 + 99 + 1 = 1000).
40 — a single magazine-style repetition at the safety limit (999).
50 — a single magazine-style repetition exceeding the limit (1000).
60 — multiple magazine-style repetitions that individually stay within the limit, but exceed it when combined (900 + 99 + 1 = 1000).

This verifies both supported repetition syntaxes as well as the cumulative repetition-count check.

The safety limit prevents the browser from attempting to expand extremely large repetition counts, which could consume excessive memory and cause the page or browser tab to become unresponsive.

--
dd47dfc only adds a safety limit to special character repetition expansion. It is not the BASIC line length limit — the final tokenized line length is still checked separately and limited to 256 bytes.

[radius75]

Hi @chrisgleissner

I've done quite a lot of compatibility testing of the Ultimate tokenizer,
comparing its output and behavior with the CBM prg Studio tokenizer and
with a real C64. So far, these tests have been successful and I haven't
found any significant compatibility problems that would require changes
to the current tokenizer.

As an additional source of real-world test programs, I've been using the
listings from the two Commodore 64 Games Book volumes:

https://archive.org/details/commodore-64-games-book_202304/mode/1up

https://archive.org/details/commodore-64-games-book-2_202409/mode/2up

Transcribed versions of these and many other C64 book listings can also
be found here:

https://github.com/TypeInBooks/books/tree/master/commodore/c64

One thing worth noting is that the descriptions/instructions for some
games do not always match what the programs actually do. However, in the
cases I've checked, these discrepancies were already present in the
original books and are not tokenizer or transcription errors.
The code in the GitHub repository is about eight years old and was most
likely created using a CBM prg Studio 3.1x-era version. This should also
be kept in mind when using it as a compatibility reference, since the
current CBM prg Studio version is 4.10 and its tokenizer may have changed
in various ways since then.

One more question regarding the PR: should I now keep this branch updated
against master instead of test-merge?

[chrisgleissner]

Hi @radius75 ,

excellent work!

About the target branch:

• The test-merge branch has been retired and should no longer be used.
• All PRs should target master.
• We will continue to cut releases from master for fixing 3.15 bugs, e.g. 3.15a (already cut), 3.15b, etc.
• Once all major issues on 3.15 have been fixed, in particular regression issues, we will repurpose master to accept larger work (new features etc.) in preparation for release 3.16.
• We may at that point create a releases/3.15 branch for fixing 3.15 issues that cannot wait for 3.16. The need for that is likely to be small if we continue to focus on 3.15 fixes for now.

This is as per the Github Flow model aka trunk based development: Only a single master which is always release-ready.

The above has been discussed with @GideonZ (except for the releases/3.15 part), but of course it is ultimately his call.

Thanks
Christian