v80.c: v80 assembler in c89 #13
Conversation
|
@Kroc please feel free to scribble any feedback or suggestions all over this PR, it's far from ready to merge at the moment! |
|
|
Pasting my question's and @Kroc's answers here for easy reference:
When v80 encounters a string, it simply writes the bytes to the code-segment one by one so the string is never stored anywhere whole -- with one exception: the file-name of an include
It's an error -- when v80 encounters “errors.txt” contains all possible errors in v80 and an explanation of what causes them so it’s a good source of detail on parsing behaviour
yes v80 is limited to a 16-bit number internally for everything. Considering that v80 can only output bytes or words to the code-segment, 32-bit results don't actually have a practical use! Note that v80 allows underflow but errors on overflow! This is so that the negate operator can work because numbers like
For memory and parsing-simplification reasons, expressions are limited to one line; the entire parser is line-orientated to allow for parsing a file larger than memory allows. v80 is 335KB of code which obviously doesn't fit into 64 KB of RAM :P But you have to understand that v80 is purposefully limited to fit into 8-bit hardware and that a C89 version shouldn't be assembling code that can't be assembled on real 8-bit hardware otherwise that defeats the point!
v80 is not trying to be an ideal assembler; it's trying to be minimal so that it can support many systems. Things like context-free grammars, macros etc. are features for a better, more language-orientated assembler (hopefully written in v80) -- v80 exists to bootstrap 8-bit software on 8-bit machines instead of relying on PC-only toolchains. Ergo, it has no goal to be anything more than a brutally simple assembler that acts as the bedrock of a broader range of 8-bit software. If an 8-bit computer can't modify and assemble it's own software then it might as well be proprietary. An 8-bit computer that can only run software that has to be compiled on a PC is not a real computer and v80 aims to break that cycle by allowing code on a PC to also assemble on 8-bit hardware. |
|
@Kroc 'nother question about local labels (possibly leading to reducing heap usage quite a bit):
|
|
Local labels are simply appended to the last non-local label defined forming a complete label-name. It was done this way for ease of implementation, but I would like to add anonymous labels in the future or change the way local labels are implemented so that they don't take up so much heap space. |
|
Sort of. I wondered whether you want to be able to rely on, eg: And if that's not an explicit goal, I think there's some low hanging fruit in heap size savings with segregating local labels into a short-lived table that gets reset at every non-local label boundary. (and allowing local labels a full 31 characters since there's no longer any need to prepend the non-local label) |
|
The heap in v80 cannot deallocate anything, ever! If a label gets added, it cannot ever be removed, because once something else gets added to the heap (like a deferred expression, a new constant), the heap cannot shrink without deleting something else important. The space cannot be reused because that creates a fragmentation problem that would take hundreds of bytes of code to work with. The heap is append only. Hope is not lost however; we could have label records include a sub-label linked list on the end of it so that only the local labels names are stored attached to the parent label by a linked list. The downside to this would be greater complexity and code size in label searches. |
- need a line-based parser to watch v80.v80, so instead of reading the next token from the input stream on demand, we buffer the next line - redid the GRAMMAR to support a line-based parser - factored out a better memory management API and built a getdelim and getline work-alike implementation with it - the tokenizer now sets a start pointer into the buffered line, and a token length - reworked the error messages to match errors.txt docs more closely -- can't resist including the current token in the error message for ease of use - added support for nested .i, along with input file stack management - added support for .a, along with a placeholder output stream - redid most of the low-level string functions for consistency and robustness - lost constants and labels support -- they need a do-over with the line-based parser
gvvaughan
force-pushed
the
gvvaughan/v80-c-bootstrap
branch
from
66b46d8
to
625420c
Compare
|
@Kroc Heap limitations make sense. For Largely rewrote |
|
Hmm.. just occurred to me that you could have local symbols in their own linked list, and as long as each entry is the same size (32bytes for the label name, and 4 bytes for the next entry pointer) and a zero length name marking the end of the list when searching, then there's no need to deallocate anything. When a new non-local label is encountered, we can error out for unresolved local label references, and then put a 0x0 tombstone at the head of the list. New local labels would then overwrite the entries from the local label list in place starting at the head (making sure that if the next entry was allocated, it get's a 0x0 tombstone) and reusing following entries until they are all used up, and then additional local labels get pushed onto the head of the list as before. The size of the local labels list would only ever be 36bytes * largest-number-of-locals-in-a-single-scope. Surely much better for very large programs, which are the ones most likely to overflow the heap? |
|
What remains:
Did I miss anything? |
- fix a few little compilation failures when copiling with strict c89 mode only
|
Thinking about it, what I'm trying to get at is that changes to v80's design in Z80 code can take weeks, even months -- it took six months of meticulous crafting instruction-by-instruction and I'm not the fastest developer already. Given that the assembler is now self-assembling, I don't want to break it without careful consideration, and rewriting what already works is equally time consuming, so there had to be clear net wins. This brings me on to instructions; I hadn't thought far enough ahead about a C version (I didn't actually think anybody would take up the offer), but the C version should reuse the instruction table binary so that this work isn't duplicated for every ISA -- v80 is unique in that support for different CPU instruction sets requires minimal code changes. The instruction set is encoded as a binary tree (see "isa_z80.v80") with a small amount of CPU-specific code to handle parameters ("v80_z80.v80"). However, I'm in the process of rewriting this table (see branch "v2") logic to both greatly simplify the instruction tables (see "is2_6502.v80" in branch "v2" for just how much simpler) and hopefully save more bytes, so you'll want to hold off of parsing instructions for the moment. |
|
Oh, I didn't mean to imply you should change the algorithm, but I think it's definitely worth throwing an error when attempting to jump into the middle of a local label from another scope so that some space optimizations are still on the table in case you want to do that one day 😁 In the unlikely event that the C version catches up, I might bug you for some specs for the v2 tables then. I secretly want to add support for my fantasy vm ISA after all! |
- added a line-wise tokenizer; keeping track of buffers and token start and end offsets by hand was too finicky - minimally tested
- define UINT_MAX if compiler/headers don't have it - set new global skipcol to UINT_MAX - add indent field to Include struct - new parse_condition sets skipcol if condition expression fails - parse_file sets files->indent from the column of the first token as each line is tokenized, not parsing any new lines until the first one with an indent no more than skipcol, when skipcol is reset to UINT_MAX - moved the line-too-long diagnostic to tokenize_line - when tokenize_line reaches a comment, return what was already tokenized, potentially avoiding line-too-long failures for comments - new diagnostic when a string token is found where a (non-byte-)expression is expected - expect a (non-byte-)expression after any keyword except .b, and also after a condition and when setting a constant
- exit with usage message for bad command line arguments - new xfopen helper to open a FILE* or exit with a diagnostic - do file extension substitution on input path to make an output filename if none was given, or fallback to v.out if there was no extension match in the table - keep all opened File objects on a stack and ensure they are all closed before exit - adjust grammar and implementation to allow multiple keywords on a single input line - snprintf is a C99 addition, carefully use sprintf instead - fix a variable declaration after a statement (C99 feature)
- adjust parser to work in two passes - for parsing pass 1, don't emit bytes - for parsing pass 2, don't set label addresses - reset include stack and pc value before each pass - elide __attribute__ annotations when __GNUC__ is not defined - simplify extreplace a little - fix a bug with closing files from the include stack - fix a bug with ERR_BADVALUE being too eager in .b and .w args - fix a bug with double for loop in .b argument parsing
|
I should add tests to flush out bugs in another PR, and I don't have any code to read the opcode tables yet - but the parser handles What's the usual way of building an assembler that does opcode lookup in the tables? And do you have a spec for v2 tables I can implement? |
- found some code that looks like `$ $ + 1 _label` in the cpm v80 assembly files... changed the parser to support that as setting PC to the result of an expression (followed by a local label) - fixed a bug in keyword parsing, where we should return a token that can't be parsed as part of the keyword arguments so the caller can try a different leg of the recursive descent - don't attempt to close the standard streams
- diagnose number overflow at any point in evaluation of an expression - can't close and reopen stdin, so remove '-' sentinel from command line - use separate len and num fields in Token struct so that we always (for the duration of working with a specific line anyway) have the token text, even when there's a number value in the token now - use a single T_COND, storing the condition type (= - ! +) in the newly available num field - simplify parse_condition and parse_line accordingly - new simpler err_fatal_token replaces both err_fatal_token_str and err_fatal_token_value - simplify callers with new token_new_number and token_new_string - simplify tokenize_line - remove unused functions stack_zstreq, token_type and token_value
|
Sorry for the slow response, I'm rather busy at home whilst my son is off school over summer. The process of parsing the instruction tables is covered by Lines 1234 to 1377 in cebb049 The "build.bat" script does some testing by building samples of the entire Z80/6502 instruction set and comparing against the same produced with WLA-DX maybe this would be a starting point? I haven't examined the PR enough to know what the build requirements of your C version are and if/how this would work as part of the current, rather crude, system. I use a batch file only so that v80 can be built out-of-the-box without having to install any dependencies or deal with high up-front demands like requiring knowledge of Docker -- remember that whatever is required to build v80 is itself a dependency of the 8-bit software at the end of the pipeline and the goal is to get away from gigabytes of constantly evolving build infrastructure :P |
|
No apologies necessary. I'm setting off on a 2-3 week road trip tomorrow, so any free time I would have had for coding will probably be spent on driving instead. Absolutely no hurry on anything from my perspective. Build requirements for v80.c are a c89 C-compiler toolchain and a libc with support for stdio I was looking at your I haven't tried building anything with WLA-DX or runcpm yet, so that's probably a good thing for me to get going to decide how to proceed, but I'd also like to write specific tests to exercise the tokenizer and parser in I'm still not clear on how to assemble the And finally (for now ;-) ) -- I was thinking it might be easier to share the instruction opcode to binary mappings between v80.c and v80 proper if we define the instruction set separately somewhere that v80.c can load directly into a hash table, and I also provide some code to generate the lookup table sources (for v80 sources) rather than you hand coding them. That will let you tune the format for speed/space efficiency without the work of hand coding the tables too. WDYT? |
gvvaughan
changed the title
- with _POSIX_C_SOURCE=1, use all local function implementations - add preprocessor guards to use library functions as available - unroll single use of TOKEN_TYPES x-macro - defer to standard ctype functions and use them as available - split xstrtou into two, and use standard strtoul library function if available - replace uses of zstrncpy and non-standard zstrlcpy with standard strlcat and strlcpy when available, or interface compatible local implementations otherwise (note: it can take some coaxing with feature macros to get declarations out of the standard headers!) - remove some newly unused functions - add some section comments
- provide fallback dirname() function in case libgen.h is missing - new global zincludedir - save a copy of the directory of infile argument to zincludedir (or "." if argv[1] has no directory component) - adjust parse_keyword_include and helpers to search zincludedir
- improve the option parser in v80.c, add a new `-i` option that
preloads the symbol table with the named ISA
- use a hash table for the symbol table instead of a linked list
- new .m keyword support. `.m instruction body tokens` stores
`instruction` as a key in the symbol table with the rest of the
tokenized line as its value
- new v1/tbl_6502.v80 defines the 6502 ISA using .m
- new v1/tbl_z80.v80 defines the Z80 ISA using .m
- when the parser encounters a (.m defined) instruction, it switches
to parsing the associated macro body, usually injecting the bytes
from the body into codesegment, evaluating expressions as necessary
to calculate those bytes: Except for the following tokens
+ .b - consume a byte from the assembly source, evaluating an
expression if necessary, and write the result to the codesegment
+ .w - consume a word from the assembly source, evaluating an
expression if necessary, and write the resulting two bytes in
little-endian order to the codesegment
+ .r - consume a word from the assembly source, evaluating an
expression if necessary, treating that as a destination address,
and write a single byte to the codesegment as a relative offset
to that destination address
|
Had a couple of unexpected evenings to finish the code! This implements the instruction tables for v80.c, as well as loading and parsing. It produces sensible looking (but untested) cpm_z80.com binary from the assembly sources, so can now serve as a bootstrap mechanism. I need to write some code to generate the QQ: v80.c is becoming hard to navigate at this size when editing it, but also having everything in a single file makes it easier to compile. I'm tempted to pull the polyfills (for missing libc APIs) and maybe some of the data structures (linked lists, hash tables, perhaps the tokenizer) into individual pseudo-headers. That would mean adding |
Thank you for hard work! Yes, you should split the code where you are essentially "patching" the base C-functionality; I fully expect that additional replacement functions may be needed for certain combinations of operating system and compiler -- C89 compatibility was very variable in compilers even late into the 90s! Such monkey-patching and non-portable considerations shouldn't factor into the code of v80 itself so that others may have an easier time fixing for their choice of compiler/OS. |
- add a more robust option processing loop - support --version - support -h, --help with some basic option help - rename --isa to --include
- polyfill/ a new directory with replacements for likely candidates for missing system headers and apis. Note: it's not a replacement for the system library, only fallbacks for apis used by this project - error.c: error handling - file.h, file.c: file handling - stack.c: simple generic stack datatype - hash.c: simple hash table datatype with buckets made with stack.c - symtab.c: symbols and a symbol table for them made with hash.c - token.h, token.c: token data type, and a line at a time tokenizer - parser.c: recursive descent parser for v80 assembly using token.c - main.c: command line processing, and driver for feeding the parser - Makefile: simple rules for making versions of v80 from the above to check that it works with almost nothing from libc using c89, and also using c99 with optimized libc functions instead of polyfills. - README.md: A little about how to build and use it all.
|
Okay, all done @Kroc! If I compile main.c from bootstrap to make a And then use that to make a It produces identical bytes after recompiling itself with And also identical bytes to recompiling sources with your most recent v80.com release: Incidentally the byte encodings for the If you like and merge this PR, I'll be happy to work on generating the Also, feel free to let me know if you have any suggestions for changes or improvements to what is already here. |
I had seen this and fixed it, but maybe that was only on the v2 branch :/ I can't remember things straight. My son will be back to school next week and I'll focus on integrating your C version then. I think we should merge it in the current state to a separate branch; are you able to update the PR to use a different branch (or this something I need to do?) |
|
Cool! I can definitely do it if you tell me what branch you'd like me to retarget to. I think you might also be able to do it with the |
started work on #4