forked from christianparpart/x0
-
Notifications
You must be signed in to change notification settings - Fork 0
/
HACKING
292 lines (222 loc) · 10.2 KB
/
HACKING
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
1.) REQUEST PROCESSING:
1.1) Request Processing Cycle:
Incoming requests are processed in multiple smaller sets and in most of them you may hook into to do or override
certain actions, such as translating the request URI into a local physical path, generating content based on
request URI or other certain criteria, or simply logging the request.
All hooks are defined in the server class, which you can connect to.
Each connect() returns a connection object which is strongly adviced to keep and store it in your plugin instance
to ensure automatic disconnection.
1.1.1) connection_open
Once a new client connects to the server, the connection_open hook is invoked and the pointer to the connection
object is passed.
1.1.2) pre_process
Before a request is being parsed, the pre_process hook is invoked with the request in question.
At this point, the request line and headers have been fully parsed but not yet interpreted, though,
you're the first to catch them.
1.1.3) resolve_document_root
Right after pre processing the new request, the document root needs to be resolved, usually by interpreting
the Host request-header.
If no handler has filled the request's document_root property it will be set to "/dev/null" by default to
prevent breaking out of jail.
1.1.4) resolve_entity
As we have now the document root for this request, we need to map the request URI to a local physical path,
usually by concating document_root and the URI's path part together.
Use this hook to customize the behaviour.
1.1.5) generate_content
This is by far the most interesting hook for most of the plugin developers, as you can now generate
the actual response by modifying response headers and writing to the body.
Remember, that once something is written to the body, no header modification may be done anymore as
they've been already serialized.
Usually we have multiple handlers registered and we need to iterate sequentially through most of them.
Which is why an iterator is being passed to the hook to call when do you don't feel responsible
for this request. But if you *do* feel responsible - and are done handling this request! - you need to invoke
the iterator's done() member method to notify the x0 core about.
1.1.6) post_process
The post_process hook is being invoked right before the response status line and headers are to be serialized.
You should not modify the headers anymore unless you know what you're doing. Usually you can use this
for diagnostics and logging.
The Content-Length header is not always set at this point, though, loggers interested in the bytes transferred
should better connect to the request_done hook.
1.1.7) request_done
Once the response has been *fully* written to the client and is considered *done*, the request_done hook is
being invoked and can do some cleanup or logging - just like the standard accesslog plugin hooks in here
to log all requests.
Although, the bytes transferred (and Content-Length) is definitely known at this point.
Now, if this was a keep-alive connection, the connection's request parser is being invoked to resume()
parsing, otherwise, the connection is being closed.
1.1.8) connection_close
If either the client aborts abnormally or the connection is to be closed server-side intentionally,
this hook is being invoked, and meant to be the very last point right before connection destruction.
==============================================================================================================
DESIGN GOALS
------------------------------------------------------------------------------
- modular, clean, self explanary, and well documented API
- avoid synchronous I/O as much as possible.
- use sendfile()/splice() where possible
- scale excellently over multiple CPUs/cores by supporting a threaded model (1 thread per CPU core)
- caching: when caching of local files/stat's, use inotify API to invalidate those caches.
- access control: support libwrap, if possible
(see external links at http://en.wikipedia.org/wiki/Libwrap)
RESPONSE FLOW
------------------------------------------------------------------------------
composite_source(response) {
buffer_source(status+header),
filter_source(body) {,
compress_filter(gzip),
composite_source(body) {
file_source(/path/to/file.html)
}
}
}
CLASSES
------------------------------------------------------------------------------
config holds configuration settings being read from file
connection abstracts an HTTP connection (no matter how many requests it is passing/pipelining)
connection_manager maintains the set of available connections
header request/response header API
listener HTTP TCP/IP listener API
plugin base class for x0 plugins
request parsed HTTP request object
request_parser HTTP request parser
response HTTP response object
server x0 server API
SUPPORT CLASSES
------------------------------------------------------------------------------
composite_buffer complex buffer creation and transmission class
function generic functor API
handler generic handler API (e.g. used for content generator hook)
ternary_search generic ternary search trie class
value_property basic value property
read_property read-only property
write_property write-only property
propety read/write property
THREADING MODEL:
------------------------------------------------------------------------------
- main thread *ONLY* performing I/O dispatching
- client threads processing the client requests and actions
- (main thread and client threads communicate through queues)
- avoid lockings as much as possible
REUQEST PROCESS FLOW (EBNF style):
------------------------------------------------------------------------------
http_connection_acceptor ::=
connection_open
(
pre_process
resolve_document_root
resolve_entity
generate_content
request_done
post_process
)*
connection_close
ASYNC APPROACH:
------------------------------------------------------------------------------
async_acceptor ::=
connection_open
pre_process
pre_process ::=
resolve_document_root
resolve_entity
generate_content[done_handler]
generate_content ::=
done_handler
done_handler ::=
request_done
post_process
( CLOSE | pre_process )
------------------------------------------------------------------------------
.
connect__>(request__>generator____________________>posthandler___)*>close
connect__>(request__>generator__>posthandler__)*>close
listener.accept:
connection_manager.start
connection.start
connection.handle_read
handle_request(request, new response);
...
response.done()
response.done:
response.flush(done2) -- send remaining data
response.done2:
delete request
delete response
SOME REQUIREMENTS:
------------------------------------------------------------------------------
* the server will not fail when an incoming connection immediately closes,
that is, serves no requests.
* on header connection=close we serve exactly up to this very request.
* on header connection=keep-alive we continue to serve another request.
* each connection must serve 0 to unlimited number of requests.
* each request must map to exactly one response object (and vice versa)
* request and response objects contain a reference to their connection
* a connection holds no references to their requests being parsed or responses being served,
except for the reference to the currently parsed request.
* once a request is fully parsed, we invoke the request handler which MAY return immediately
and process generate the response later (asynchronousely).
* a response is considered completed when response.done() is invoked.
* response.done() initiates post handlers and resuming the parsing of further
requests within this connection.
EVENT-TO-MODULE RELATION:
------------------------------------------------------------------------------
connection_open: ssl
pre_process:
resolve_document_root: vhost
resolve_entity: userdir, indexfile
generate_content: dirlisting, sendfile, cgi
request_done: accesslog
post_process:
connection_close: ssl
---------------------------------------------------------------
server
-> create listener (server.handle_accept)
handle_accept
-> create connection (handle_read)
-> connection.start
response.flush()
-> connection.close
-> connection.resume
connection.start
-> socket.async_read_some(handle_read)
connection.handle_read
-> parse_request_partial
-> handle_request
-> close
handle_read(request, response)
-> response.flush
listener
connection
request
response
---------------------------------------------------------------
(LINUX) SYSTEM CALLS OF INTEREST;
- timerfd, for handling timers through file descriptors
- eventfd, helpful for user driven event notification / main-loop interruption (instead of using pipes e.g.)
- signalfd, for receiving signal events via file descriptor instead of sigaction/... stuff
- inotify, file system event notification.
- fsnotify, potential successor of inotify and dnotify
- sendfile, most delicious when used fully async (for input and output)
- socketpair, not quite interesting, but worth a note
- posix_fadvise
- readahead
- everything around aio/libaio
composite_buffer - buffer object, creation and data access
composite_buffer_writer - synchronous writer
composite_buffer_async_writer - asynchronous writer
simple_writer - writes chunks via writev, and files via read+write
sendfile_writer - writes files via sendfile
aio_sendfile_writer - writes files via aio+sendfile
mmap_writer - writes files via mmap+writev
---------------------------------------------------------------
REQUEST HANDLERS / CONTENT GENERATORS:
They hook into server's content_genreator property by invoking server.generate_content.connect(...);
Do not forget to store the result of connect() (of type connection).
void my_request_handler(x0::request_handler::invokation_iterator next, x0::request& in, x0::response& out)
{
if (in.path() != "/")
return next();
out.status = x0::response::ok;
out.header("Content-Type", "text/plain");
out.write(std::make_shared<buffer_source>("Hello, World\n"), [&](const asio::error_code& ec, int nwritten) {
next.done();
});
}