zlib port

[rafael2k]

After scratching my head not being able to have gzip in ELKS, I decided to take a look in zlib.

So, long story short, zlib is already compatible with OW and 16bit intel. Here is an initial port of zlib (current version) (with minimal changes) to run on ELKS:
https://github.com/rafael2k/zlib

Currently I'm building two example apps, minimal implementations of gzip and "untgz"

[rafael2k]

[ghaerr]

Interesting! I'm interested to learn how big gzip or untgz is when you get them running. Our current compress is 12k.

[ghaerr]

Wow, you've got them running already! Will gzip unzip any size file, or is it limited to how much free memory someone might have?

[rafael2k]

[rafael2k]

Wow, you've got them running already! Will gzip unzip any size file, or is it limited to how much free memory someone might have?

Any file size, as it operates with fixed buffers as I understand.

[rafael2k]

ls -l untgz gzip 
-rwxrwxr-x 1 rafael2k rafael2k 81228 Jan 23 01:13 gzip
-rwxrwxr-x 1 rafael2k rafael2k 83534 Jan 23 01:13 untgz

[rafael2k]

Again, the key was to add ELKS specific memory allocation functions.

[rafael2k]

Interesting! I'm interested to learn how big gzip or untgz is when you get them running. Our current compress is 12k.

There is definitely some bloat there. But this one I think we can send the support upstream!
: )

[ghaerr]

It would be interesting to try using ELKS' tar to create a tar file of a complete image of /, then see if it can be compressed and then uncompressed.

-rwxrwxr-x 1 rafael2k rafael2k 81228 Jan 23 01:13 gzip
-rwxrwxr-x 1 rafael2k rafael2k 83534 Jan 23 01:13 untgz

Huge, not something that can easily hit the floppy distribution. Also, my executable compression technology won't compress non-a.out executables yet. Nonetheless very cool, should someone need to uncompress something on ELKS.

Again, the key was to add ELKS specific memory allocation functions.

I see. Was large model required? I am a bit surprised at the file size and am wondering if that's mostly text or data.

[ghaerr]

try using ELKS' tar to create a tar file

Our ELKS tar is pretty untested, and possibly not compatible with host tar file format, FYI.

[rafael2k]

It would be interesting to try using ELKS' tar to create a tar file of a complete image of /, then see if it can be compressed and then uncompressed.

It is exploding right now. I'll investigate.

-rwxrwxr-x 1 rafael2k rafael2k 81228 Jan 23 01:13 gzip
-rwxrwxr-x 1 rafael2k rafael2k 83534 Jan 23 01:13 untgz

Huge, not something that can easily hit the floppy distribution. Also, my executable compression technology won't compress non-a.out executables yet. Nonetheless very cool, should someone need to uncompress something on ELKS.

Again, the key was to add ELKS specific memory allocation functions.

I see. Was large model required? I am a bit surprised at the file size and am wondering if that's mostly text or data.

large model, yes, only mm officially supported. I'm still stabilizing for the best memory tunables zlib have.

there is lots of data. I forgot how to see the sizes of each section of the executable.
some tables and so on.

[rafael2k]

ps: the idea is that there will be lots of symlinks to the same application, as "one-do-all" approach works well with zlib, as most file formats (gzip, zip) it supports are just some simple headers + the library itself.

[ghaerr]

I forgot how to see the sizes of each section of the executable.

objdump86 -s gzip untgz

the idea is that there will be lots of symlinks to the same application

I see, that would be good. I do notice that gzip and untgz are 2k different in size though, so they're being built differently.

Another issue with chess.c -> chess.c.gz is that the latter file can't be created on FAT, since we don't support runtime creation of long filenames.

[rafael2k]

-rwxrwxr-x 1 rafael2k rafael2k 81228 Jan 23 01:13 gzip
-rwxrwxr-x 1 rafael2k rafael2k 83534 Jan 23 01:13 untgz

Huge, not something that can easily hit the floppy distribution. Also, my executable compression technology won't compress non-a.out executables yet. Nonetheless very cool, should someone need to uncompress something on ELKS.

Indeed. The price to pay for having a full blown compression library. zlib is pretty cool!

[ghaerr]

The price to pay for having a full blown compression library.

How long is the compress vs decompress taking?

This should also be able to be accomplished in almost the same way using compress and compress -d on ELKS, with a much smaller executable. It uses the older 12-bit compression format, so compression is likely not as great, but executable is only 12k and may be just as fast, I don't know.

[rafael2k]

objdump86 -s gzip untgz

I thinks this does not work with OS2 binaries

[rafael2k]

The price to pay for having a full blown compression library.

How long is the compress vs decompress taking?

This should also be able to be accomplished in almost the same way using compress and compress -d on ELKS, with a much smaller executable. It uses the older 12-bit compression format, so compression is likely not as great, but executable is only 12k and may be just as fast, I don't know.

compress is better, definitely. For the common native use - keep zlib for just when needed.

But I'll need zlib for PNG too, in the image viewer I plan to implement.

ps: my "bare metal" is not a good reference, so lemme try in a more accurate emulator apart of qemu. I bet it is slower than compress.

[ghaerr]

I thinks this does not work with OS2 binaries

Oops. Try using wdis, the OWC dumper. It's output is large, but look at the "Segment Table" section and the len and alloc values in hex show the sizes. Len means size in the executable and alloc means the size in RAM (different for data/bss).

compress is better, definitely

Why? Is it faster and smaller?

[ghaerr]

But I'll need zlib for PNG too, in the image viewer I plan to implement.

Sounds like I need to think about getting the Nano-X client library compilable via OpenWatcom. That would allow you to write a viewer that would decode the file in the client, then pass the data to the nano-X server for display. Multiple PNG image files could be displayed in windows, which would be pretty cool. (Providing we don't run out of memory)!

Of course, this can be done after you get a VGA version of the PNG viewer operational. We'll probably want to look at some speed issues as well, but after its working.

[rafael2k]

But I'll need zlib for PNG too, in the image viewer I plan to implement.

Sounds like I need to think about getting the Nano-X client library compilable via OpenWatcom. That would allow you to write a viewer that would decode the file in the client, then pass the data to the nano-X server for display. Multiple PNG image files could be displayed in windows, which would be pretty cool. (Providing we don't run out of memory)!

Some Felix the Cat pngs will not make us run out of memory!
: )

Of course, this can be done after you get a VGA version of the PNG viewer operational. We'll probably want to look at some speed issues as well, but after its working.

I like this plan! Definitely I'll do the VGA one first, for the sake of learning the details.

About wdis, it is complaining about missing ".o"

Concerning Nano-X, is there already any window manager? I'm not sure why the mouse is not working on qemu here.

ps: wdis gives me:

gzip.o: No such file or directory

[ghaerr]

About wdis, it is complaining about missing ".o"

Have you tried ".obj", the normal Watcom extension?

Concerning Nano-X, is there already any window manager?

Yes, but its not been integrated into our source yet.

I'm not sure why the mouse is not working on qemu here.

Uncomment the following line in qemu.sh and make sure you're running single user (init 1):

SERIAL="-chardev msmouse,id=chardev1 -device isa-serial,chardev=chardev1,id=serial1"

If running on real hardware, you've got to make sure you've used stty to set the baud rate for the /dev/ttyS0 serial port and you're running a Microsoft mouse. If not, then nanox/drivers/mou_ser.c needs to be edited to reset MOUSE_TYPE.

[ghaerr]

Try using wdis, the OWC dumper.

Oops again! I meant try wdump, the OWC EXE file dumper. wdis is only used for OWC .obj object files (OMF format).

[rafael2k]

for gzip:

==============================================================================
seg  fileoff  len  alloc prior priv flag
==== ======== ==== ====  ====  ==== ====
0001 000000E4 E9AC E9AC  0000  0003 0D00
  CODE|FIXED|NOSHARE|LOADONCALL|EXECREAD|RELOCS
0002 0000EB22 50D8 81C0  0000  0003 0D01
  DATA|FIXED|NOSHARE|LOADONCALL|READWRITE|RELOCS

[ghaerr]

From your wdump output it looks like segment 1 (code) is E9AC (59,820) bytes, which doesn't necessarily require large model; segment 2 is 50D8+81C0= 53,912 bytes, also less than 64k.

If the program uses fixed buffers and no malloc, this would (somewhat barely) fit in small model. But frankly I don't think the size of the program would be reduced by enough to matter very much.

[rafael2k]

I'm still tuning zlib configuration (just changed to -DMAX_WBITS=11 -DMAX_MEM_LEVEL=3, as the dos realmode version), but I'd like to ensure compatibility with any gzip created file, and it seems MAX_WBITS remove us the ability to process data with WBITS > 11 (up to 15).

[ghaerr]

it seems MAX_WBITS remove us the ability to process data with WBITS > 11 (up to 15).

I suppose not surprising, our compress is also limited to 12 bits IIRC, so we use host compress -b 12 ... to produce the compress man pages.

This may be able to be worked around by using the OWC huge model, but support for that is totally untested at the moment. The huge model allows for a 32-bit segment:offset pointer to access more than 64k bytes, at the cost of doing special arithmetic generated by the compiler for pointer operations and increments, etc.

[rafael2k]

Just realized our size_t parameter for malloc is 16 bit! I was getting strange errors, just to realize with high window size malloc input was overflowing

[rafael2k]

Just realized our size_t parameter for malloc is 16 bit! I was getting strange errors, just to realize with high window size malloc input was overflowing

I think I'll keep with 12 bits on gzip too then.

[rafael2k]

it seems MAX_WBITS remove us the ability to process data with WBITS > 11 (up to 15).

I suppose not surprising, our compress is also limited to 12 bits IIRC, so we use host compress -b 12 ... to produce the compress man pages.

This may be able to be worked around by using the OWC huge model, but support for that is totally untested at the moment. The huge model allows for a 32-bit segment:offset pointer to access more than 64k bytes, at the cost of doing special arithmetic generated by the compiler for pointer operations and increments, etc.

zlib definitelly is a good place to test huge model, but I don't think it is necessary right now. zlib works fine with 12 bits.

[toncho11]

@rafael2k Maybe you should release binaries now (as you did for the image viewer)

ELKS 9.0 will be an exciting release it seems with:

• toolchain
• image viewer
• gzip,untgz
• chess

on the MINIX and FAT HDD images.

[rafael2k]

I'm not sure what happened at last ELKS commits, but now I get an error running gzip:

[ghaerr]

now I get an error running gzip:

Geez, looks like I may have broken something, trying to fix a complicated kernel address validation issue when running OS/2 binaries. At the moment, I'm not coming up with any ideas why this is affecting your gzip.

For an immediate workaround, put a "return 0" at the top of elks/arch/i86/mm/user.c:

int verfy_area(void *p, size_t len)
{
    int i;
    segoff_t offset;

return 0; // <--- ADD THIS LINE FOR NOW

    /* Kernel tasks can always access user process boundaries */
    if (kernel_ds == current->t_regs.ds)
        return 0;

In order to debug this, I may have to play with your gzip. Which repo is that in?

[rafael2k]

In order to debug this, I may have to play with your gzip. Which repo is that in?

https://github.com/rafael2k/zlib
make -f elks/Makefile.elks
: )

[rafael2k]

now I get an error running gzip:

Geez, looks like I may have broken something, trying to fix a complicated kernel address validation issue when running OS/2 binaries. At the moment, I'm not coming up with any ideas why this is affecting your gzip.

For an immediate workaround, put a "return 0" at the top of elks/arch/i86/mm/user.c:

int verfy_area(void *p, size_t len)
{
    int i;
    segoff_t offset;

return 0; // <--- ADD THIS LINE FOR NOW

    /* Kernel tasks can always access user process boundaries */
    if (kernel_ds == current->t_regs.ds)
        return 0;

In order to debug this, I may have to play with your gzip. Which repo is that in?

With this things gets back to normal.
Uff! So many new goodies in the tree!

[rafael2k]

Did a release 0.1 of zlib for ELKS:
https://github.com/rafael2k/zlib/releases/tag/0.1

[ghaerr]

With this things gets back to normal.
Uff! So many new goodies in the tree!

Did you build a new OWC libc.lib after updating the kernel? There were some big changes to the Watcom C library having to do with setting the DS register to a far memory location then calling read/write etc. The kernel change which may not be working tries to check the passed DS register with all of the application's far memory segments (including fmemalloc and _fmalloc segments). If none match, then "FAULT" is printed out for now, and the verification fails, and the kernel aborts the program (gzip in this case).

If you did rebuild the OWC libc.lib, then we might want to printk the DS register value to try to debug what is happening.

[rafael2k]

I did recompile and confirmed other apps from image viewer (OS/2 exe) work fine. Can it be a zlib issue discovered by this check? We can check if the toolchain keeps running fine.

[ghaerr]

Can it be a zlib issue discovered by this check?

If only gzip is having the problem, this could very well be a bug in your zlib or gzip. The toolchain and all other programs are running fine. I'll add more debug code in the kernel for the "FAULT" issue and you can continue testing.

We can check if the toolchain keeps running fine.

The above "hack" using return 0 disables, not enables, protection. So it should not be included if at all possible. The toolchain doesn't need the hack as I found the protection problem in the first place using the toolchain and then fixed it.

To reiterate: the issue is that when a system call is made to the kernel in large model OWC, the DS register is changed to the far segment address of the "char *" system call argument. The "return 0" disables checking that pointer for being proper, that is, for being memory owned by the process. So it appears that gzip is possibly passing an invalid pointer to the kernel, in which case the kernel prints "FAULT" and sends SIGSEGV to the process. We are disabling that for the time being for your program only. One way to debug this is to run gzip with "strace" set in bootopts then determine which system call is failing (and remove the hack). You might try that to get a better understanding of how this all works and also using ELKS strace.

[rafael2k]

Can it be a zlib issue discovered by this check?

If only gzip is having the problem, this could very well be a bug in your zlib or gzip. The toolchain and all other programs are running fine. I'll add more debug code in the kernel for the "FAULT" issue and you can continue testing.

Perfect.

We can check if the toolchain keeps running fine.

The above "hack" using return 0 disables, not enables, protection. So it should not be included if at all possible. The toolchain doesn't need the hack as I found the protection problem in the first place using the toolchain and then fixed it.

To reiterate: the issue is that when a system call is made to the kernel in large model OWC, the DS register is changed to the far segment address of the "char *" system call argument. The "return 0" disables checking that pointer for being proper, that is, for being memory owned by the process. So it appears that gzip is possibly passing an invalid pointer to the kernel, in which case the kernel prints "FAULT" and sends SIGSEGV to the process. We are disabling that for the time being for your program only. One way to debug this is to run gzip with "strace" set in bootopts then determine which system call is failing (and remove the hack). You might try that to get a better understanding of how this all works and also using ELKS strace.

I understood the check which was triggering the fault. I'll debug with strace (I should just add "strace" to bootopts, right). I tried to change the size of WBITS and reduced memory usage a lot, and still the same error.

[rafael2k]

I uncommented "strace" and "FTRACE=1" from /bootopts, but I don't see no difference.
Now I'm getting:

ps: I had to recompile the kernel - forgot about.

So many system calls I can not roll the screen up to see what went wrong, but there is no clear system call before the error.

The error is at this non suspicious function call - I can printf before it, but can not from inside it.
https://github.com/rafael2k/zlib/blob/601fbb3218226c3109c576cfa45713e557a8273b/test/minigzip.c#L388

[ghaerr]

So many system calls I can not roll the screen up to see what went wrong, but there is no clear system call before the error.

If you set the console to serial using console=ttyS0,19200 in /bootopts, then the debug output will display to the terminal app on your host, making it far easier to read!

The strace output displays everything, including it's own write calls, so the error is above, just prior to the "FAULT" display:
"write(4, 0x0, 8192)". This is gzip trying to write 8K bytes to offset 0 in what is likely a fmemalloc'd memory region (DS is not shown).

The error is at this non suspicious function call - I can printf before it, but can not from inside it.

Yes, that's the same function call displayed as write(..., 8192) above, I believe.

The bug is that I had an off-by-one error and the very last byte 8192 was incorrectly disallowed.

Thank you for your testing on this, the problem should be fixed in #2220.

[rafael2k]

So many system calls I can not roll the screen up to see what went wrong, but there is no clear system call before the error.

If you set the console to serial using console=ttyS0,19200 in /bootopts, then the debug output will display to the terminal app on your host, making it far easier to read!

Cool. Now I can use it.
: )

The strace output displays everything, including it's own write calls, so the error is above, just prior to the "FAULT" display: "write(4, 0x0, 8192)". This is gzip trying to write 8K bytes to offset 0 in what is likely a fmemalloc'd memory region (DS is not shown).

The error is at this non suspicious function call - I can printf before it, but can not from inside it.

Yes, that's the same function call displayed as write(..., 8192) above, I believe.

The bug is that I had an off-by-one error and the very last byte 8192 was incorrectly disallowed.

Thank you for your testing on this, the problem should be fixed in #2220.

Yay! All working again.