Optional feature request: Ulticrunch

[RichardTND]

I have an idea which I would love to see implemented in a future firmware - but of course that is optional. After reading about a built in machine code monitor/assembler coming into a future firmware. I think it would be great to have something that could accompany that option. "Ulticrunch". Although I can't code one of those myself.

The idea is simply a built-in compression tool (cruncher) based on Exomizer that can compress full machine code programs from a PRG file directly onto the Ultimate 64. The first option would be to setup the parameters for example:

Loadname: (Prg to crunch)
Savename: (Prg to save out)
JMP $: (Hexadecimal jump address)
$001 $: (001 status ($36, $37, $38, etc.)
SEI/CLI: (Set or Clear)
Flashing border on decrunch: (On/Off)

Ulticrunch should then take seconds to compress the program into and executable that can be runnable from BASIC and generate a new program file.

[chrisgleissner]

Thanks for this idea, Richard. Much appreciated.

The parameter list is essentially the Exomizer sfx options in a form that would actually work from a menu, and the pairing with the monitor makes sense.

A few things would need working out, so I ran some numbers.

Four crunchers, eight real programs, up to a full 64K image.

Exomizer came out best on ratio, which is no surprise given what it does. The difficulty is on the compression side rather than the decrunching side. It needs around 68 MB of static tables and another 100 MB or so while it searches, and the Ultimate has an 8 MB heap to work with. Making it fit would mean rewriting how it stores its match data rather than porting it across, which is a much larger job than it first looks.

So the question became what else gets close. ZX0 came nearer than I expected. Using salvador it is within 2.7% of Exomizer on the worst file in the set and around 2% on most, and on two already-packed files it came in slightly smaller. Working set is 5.7 MB, so that one does fit. The decruncher is 121 bytes against Exomizer's 234 to 319, which recovers some of the difference on the finished file.

Program	Original	Exomizer	ZX0
Hover Bovver	38,869	8,820	9,043
CCGMS 2020	22,934	13,082	13,390
Elite	57,097	31,884	32,571

TSCrunch was 18 to 26% behind, so it is not really in the running.

@RichardTND, the question that decides this: for TND releases, is ZX0 close enough, or is Exomizer specifically the point?

@GideonZ, the firmware size cost is around 50 KB on the RISC-V builds and 60 KB on the U64. There is room on every machine except the U2 classic, which has 3.2 KB of application space left, so that one would miss out. Whether that is worth the budget, and whether a feature that skips a machine is acceptable, is your call.

Still unmeasured: crunch time on the real hardware. It is a fraction of a second on a PC, but the Ultimate runs a 50 MHz softcore with no hardware multiply, so tens of seconds is plausible. Worth measuring before going further.

Full write-up and raw numbers available if useful.

[mrdudz]

Not sure if its a good idea to put everyones pet tools into the firmware (and why exomizer, it hasnt been the packer of choice for a while)

[chrisgleissner]

@mrdudz It may be a good idea to await further votes on this issue. I agree that pet tools should not go into the firmware, especially if there is only a very small base of pet lovers for this particular tool combined with a very big impact on firmware size, as in this case.

It would be interesting to get more thoughts and ideas on this topic. And idealy upvotes by anyone who is interested.

[RichardTND]

I have used ZX0 crunching before using Dali V0.3.5 on my own productions. It is only an optional feature request which just sprung into my head over the weekend. I think it would be idea developers who actually program on their Ultimate 64s, although occasionally when I code stuff in Turbo Assembler/Turbo Macro pro. I usually use a native C64 compression tool in order to save me swapping and switching on my desktop.

[chrisgleissner]

Thanks for the additional context @RichardTND . If we were to add a cruncher based on ZX0, would it still be useful to you? It seems more achievable than Exomizer based on my preliminary analysis above.

[RichardTND]

Yes, that would be useful thanks.