-
Notifications
You must be signed in to change notification settings - Fork 1
/
new_language.html
269 lines (269 loc) · 10.6 KB
/
new_language.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta name="generator" content="Asciidoctor 2.0.23">
<title>Adding support for new target language</title>
<link rel="stylesheet" href="styles/bootstrap.min.css">
<link rel="stylesheet" href="styles/bootstrap-theme.min.css">
<link rel="stylesheet" href="styles/main.css">
<link rel="stylesheet" href="styles/pygments-default.css">
<link rel="stylesheet" href="./styles/colony.css">
<link rel="stylesheet" href="styles/asciidoctor-tabs.css">
</head>
<body class="article">
<nav class="navbar navbar-inverse navbar-fixed-top" id="main-navbar">
<div class="container">
<div class="navbar-header">
<button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#main-navbar-collapse" aria-expanded="false">
<span class="sr-only">Toggle navigation</span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<span class="navbar-brand">Kaitai Struct</span>
</div>
<div class="collapse navbar-collapse" id="main-navbar-collapse">
<ul class="nav navbar-nav">
<li class=""><a href="//kaitai.io/#what-is-it">What is it?</a></li>
<li class=""><a href="//kaitai.io/#quick-start">Quick Start</a></li>
<li class=""><a href="//kaitai.io/#download">Download</a></li>
<li class=""><a href="//kaitai.io/news/">News</a></li>
<li class=""><a href="//formats.kaitai.io/">Format Gallery</a></li>
</ul>
<ul class="nav navbar-nav navbar-right">
<li><a href="https://ide.kaitai.io/">Try it — Web IDE</a></li>
<li class="active"><a href="index.html">Documentation</a></li>
</ul>
</div>
</div>
</nav>
<div id="header">
<h1>Adding support for new target language</h1>
<div id="toc" class="toc">
<div id="toctitle">Table of Contents</div>
<ul class="sectlevel1">
<li><a href="#_get_acquainted_with_ks_as_a_user_at_least_on_the_intermediate_level">1. Get acquainted with KS as a user at least on the intermediate level</a></li>
<li><a href="#_plan_how_to_map_ks_concept_to_concepts_of_new_target_language">2. Plan how to map KS concept to concepts of new target language</a>
<ul class="sectlevel2">
<li><a href="#_structures">2.1. Structures</a></li>
<li><a href="#_streams">2.2. Streams</a></li>
<li><a href="#_primitive_and_complex_type_mapping">2.3. Primitive and complex type mapping</a></li>
<li><a href="#_coding_standards_naming_practices_and_other_traditions_in_target_language">2.4. Coding standards, naming practices and other traditions in target language</a></li>
</ul>
</li>
<li><a href="#_implement_runtime_library">3. Implement runtime library</a></li>
<li><a href="#_manually_compile_hello_world_ksy">4. Manually compile <code>hello_world.ksy</code></a></li>
<li><a href="#_start_testing">5. Start testing</a></li>
<li><a href="#_port_spec_for_helloworld">6. Port spec for HelloWorld</a></li>
</ul>
</div>
</div>
<div id="content">
<div id="preamble">
<div class="sectionbody">
<div class="paragraph">
<p>A general overview of what is needed to add support for new target language in reference KS compiler.</p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_get_acquainted_with_ks_as_a_user_at_least_on_the_intermediate_level">1. Get acquainted with KS as a user at least on the intermediate level</h2>
<div class="sectionbody">
<div class="paragraph">
<p>Things to check out and familiarize oneself with:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>general .ksy structure, metadata</p>
</li>
<li>
<p>sequences and instances</p>
</li>
<li>
<p>primitive types</p>
</li>
<li>
<p>repetitions</p>
</li>
<li>
<p>user-defined types</p>
</li>
<li>
<p>enums</p>
</li>
<li>
<p>processing algorithms</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Clone <a href="https://github.com/kaitai-io/kaitai_struct/">central KS repo</a>, learn <a href="developers.html">how to build a compiler from sources</a>, install prerequisites, try to build the compiler and run the tests. Take a look at our <a href="//ci.kaitai.io/">CI</a>.</p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_plan_how_to_map_ks_concept_to_concepts_of_new_target_language">2. Plan how to map KS concept to concepts of new target language</h2>
<div class="sectionbody">
<div class="paragraph">
<p>The best way to get some inspiration in how it’s done is probably take one or two of existing target languages and experiment a little, seeing which KS concepts result in what generated code.</p>
</div>
<div class="sect2">
<h3 id="_structures">2.1. Structures</h3>
<div class="paragraph">
<p>The core element that KS deals with is a "type". Basically, everything starts with a "type", i.e. a "file" is a "type". "Type" can include:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>a collection of fields parsed sequentially (<code>seq</code>)</p>
</li>
<li>
<p>a number of fields parsed randomly or calculated (<code>instances</code>)</p>
</li>
<li>
<p>definition of nested types (<code>types</code>)</p>
</li>
<li>
<p>lists of integer constants (<code>enums</code>)</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Most usually, a concept of "type" maps to a concept of "class" or "object" in target language, but it can be a standalone data structure with a couple of generated methods to work with it, or probably something completely different.</p>
</div>
</div>
<div class="sect2">
<h3 id="_streams">2.2. Streams</h3>
<div class="paragraph">
<p>Struct has to parse the data from somewhere. Decide on which target language’s abstraction of stream is best (probably it’s good to support reading both from disk files, if they’re available, and from arbitrary byte buffers). One would probably need to implement a KaitaiStream-style wrapper for these stream(s), as it might lack some useful functions, like bit-level reading, read-until-the-terminator, etc.</p>
</div>
</div>
<div class="sect2">
<h3 id="_primitive_and_complex_type_mapping">2.3. Primitive and complex type mapping</h3>
<div class="paragraph">
<p>KS deals with a few primitive and complex types:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>integer numbers</p>
</li>
<li>
<p>floating point numbers</p>
</li>
<li>
<p>byte arrays</p>
</li>
<li>
<p>strings</p>
</li>
<li>
<p>"generic" arrays of everything</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Decide upon mapping of all these types into native types of target language. Pay extra attention to:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>signed vs unsigned support</p>
</li>
<li>
<p>if target language has any platform-dependent types</p>
</li>
<li>
<p>encoding support for strings</p>
</li>
<li>
<p>substitutions / workarounds for everything that target language does not support</p>
</li>
<li>
<p>nulls / undefined state of variables - these could be particularly useful to implement lazy instances parsing; if a language does not support such state of variables, we’ll have to introduce extra "flag" variables to store information</p>
</li>
</ul>
</div>
</div>
<div class="sect2">
<h3 id="_coding_standards_naming_practices_and_other_traditions_in_target_language">2.4. Coding standards, naming practices and other traditions in target language</h3>
<div class="paragraph">
<p>Carefully check out different coding practices related to target language. If there is an official or semi-official standard or recommendation, try to follow it. Things to pay particular attention to:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>naming standards (i.e. UpperCamelCase, lowerCamelCase, under_score_case, something else) for various parts of generated code (names of source files, classes, methods, properties, etc)</p>
</li>
<li>
<p>code formatting style and guidelines (indent size & practices)</p>
</li>
<li>
<p>docstrings layout and general documentation standards</p>
</li>
<li>
<p>private/protected/public access restriction traditions</p>
</li>
</ul>
</div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_implement_runtime_library">3. Implement runtime library</h2>
<div class="sectionbody">
<div class="paragraph">
<p>Keeping it close to <a href="stream_api.html">our standard API</a> is heavily recommended, unless there’s some good reason to do it differently. In particular, try to stick to proposed method names, it will make life much easier.</p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_manually_compile_hello_world_ksy">4. Manually compile <code>hello_world.ksy</code></h2>
<div class="sectionbody">
<div class="paragraph">
<p>Take one simplest test <a href="https://github.com/kaitai-io/kaitai_struct_tests/blob/master/formats/hello_world.ksy">hello_world.ksy</a> and translate it manually into source code in a new target language, as it would have been compiled by KS compiler. Inspiration can be drawn from any other already supported language.</p>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_start_testing">5. Start testing</h2>
<div class="sectionbody">
<div class="paragraph">
<p>Choose a testing framework that target language would use. Ideally, we should be able to run it with installing little extra dependencies in our Travis CI configuration. At the bare minimum, we’ll need a command-line based test runner that will output some report file we can aggregate in our CI later (JUnit XML reports are usually good candidates).</p>
</div>
<div class="paragraph">
<p>Create testing infrastructure in <a href="https://github.com/kaitai-io/kaitai_struct_tests">tests</a> repo: usually it boils down to:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>spec/$LANG</code> - a project or just a bunch of files with test specs; it’s heavily recommended to do 1 test .ksy format = 1 test case = 1 file, if possible</p>
</li>
<li>
<p><code>run-$LANG</code> - a single script to launch test runner with all the tests</p>
</li>
</ul>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_port_spec_for_helloworld">6. Port spec for HelloWorld</h2>
<div class="sectionbody">
<div class="paragraph">
<p>Port spec for HelloWorld test to new target language and make it work with manually compiled <code>hello_world.ksy</code>.</p>
</div>
</div>
</div>
</div>
<div id="footer">
© 2015–2024 Kaitai Project
</div>
<script src="js/jquery-1.12.3.min.js"></script>
<script src="js/bootstrap.min.js"></script>
<script src="js/asciidoctor-tabs.js"></script>
</body>
</html>