Shell-escape support #708
Shell-escape support #708
Conversation
Codecov Report
@@ Coverage Diff @@
## master #708 +/- ##
==========================================
+ Coverage 46.79% 46.84% +0.05%
==========================================
Files 142 141 -1
Lines 58874 58943 +69
==========================================
+ Hits 27551 27614 +63
- Misses 31323 31329 +6
Continue to review full report at Codecov.
|
|
Thank you! I think we can get going with this straightforward implementation. Would you be able to try adding a few test cases to the test suite? The other thing that I would like to do is to integrate this feature into the new Tectonic.toml framework that I am prototyping in the "V2" CLI. If you'd like to look into implementing that, I can give some pointers, but that can also be a separate PR. |
|
I just tried this with Example\documentclass{article}
\usepackage{minted}
\begin{document}
\begin{minted}{c}
int main() {
printf("hello, world!");
return 0;
}
\end{minted}
\end{document}And the following command: And got the following output: OutputNo idea why, but figured you would like to know. Thanks for all the hard work! |
|
@LiHRaM Thanks. In As for By my read, this should fail on stock XeTeX as well. But the test in |
|
I've added some changes to make OutputIt seems like tempfile.tex\documentclass{article}
\newwrite\tempfile
\begin{document}
\immediate\openout\tempfile=lists.tex
\immediate\write\tempfile{this is interesting}
\immediate\write\tempfile{}
\immediate\write\tempfile{this too}
\immediate\closeout\tempfile
\input{lists}
\end{document}XeTeX creates |
|
That's a good observation. I just tested with the |
|
Hmmm, yeah. I think we'll have to add special hooks to write tempfiles to disk before any shell-escape commands run, because the intermediates are only written at the end of a compilation run, not in the middle when shell-escape tools need them. |
|
@ralismark FYI I'm about to merge #716 and follow up with more work to rearrange the Tectonic codebase into a more modular structure. As it stands I don't think it will break your PR, but there may be some changes that will require you to rebase your work. Apologies for the inconvenience, but I think I have a window to make a lot of progress on this front and I want to push ahead as rapidly as I can while that's the case. |
ralismark
force-pushed
the
shell-escape
branch
from
2e9eead
to
0c16253
Compare
|
Is there any chance to be merged? I really need |
|
Already posted in the Issue but for everybody with some deadlines, here you go: Since i cannot wait until the PR is merged (got some deadlines over here) and tectonic is by far the smoothest LaTeX compiler, i created a small helper script to use minted with tectonic with a tutorial until it is built in. https://breuer.dev/tutorial/tectonic-minted Just comment on the post if something is not working properly and i will try to fix it :) |
|
Sorry about the massive time gap, I've rebased this branch to master now. I've added a I've tested this with the Edit: It also seems like I can't see the codecov details? https://app.codecov.io/gh/tectonic-typesetting/tectonic/compare/708 is giving me |
|
Thank you @ralismark for your patience and putting up with the hassles of trying to hit a moving target with all of my API changes! I really appreciate it, and I'm sure that the number of people that have been asking for this feature for a long time do as well. The current build failure is a small Clippy complaint that's easy to fix; for reference, you can run the Clippy check with: I took a look at adding support for this feature into the "V2" CLI and discovered an issue that I think we'll need to address. Right now, the On the implementation side, I'm a little uncomfortable with the Unfortunately I don't see a super-easy solution. In the V2 model we shouldn't be modifying in the I apologize if this is a bit unclear — it's late. Maybe I'll get some inspiration in the morning. |
|
Yeah passing the necessary things to the core bridge seems like a big hassle. I agree with having \write18 and temporary files be put in the output directory, which means that we'd need to somehow pass For Also, could there be problems with commands that normally try to read from files in the same directory as the input file? As an additional note, regular XeTeX doesn't work for either of the examples in the above comments if you set an output directory -- it complains about trying to read from missing files -- but that should resolved by adding an extra filesystem layer. |
|
I thought about this further, and requiring the working dir to be passed to runsystem might be better. Even though xetex currently doesn't know about the output dir, we can pass it in via |
|
I just wanted to check in and say that I haven't forgotten about this — I had a major work deadline that took up all of my time until last week. Part of that effort happened to involve using Tectonic in ways very relevant to this PR so it's definitely been on my mind! There are still some aspects of the design here that I want to think about carefully, since these details are going to matter a lot for the overall coherence of the program design. |
This reverts commit d2e3269.
Instead, use the new IoSetup implementation of IoProvider. This helps us avoid some additional borrows that limit our flexibility. We do this with some aggressive use of local macros to de-boilerplate the implementation.
BREAKING: replace the IoEventBackend trait with a DriverHooks trait that is similar but has some important differences. In particular, DriverHooks is a single reference that provides combined access to the functionality that was previously split between the IoProvider and the IoEventBackend. This unification allows operations that previously wouldn't have been possible because they'd have required simultaneous access to both of these. Philosophically, the DriverHooks trait better captures its purpose: gathering all of the callbacks needed to help the C/C++ engines interface with the outside world. IoProvider functionality is *most*, but not all of this. Furthermore, in the DriverHooks paradigm, a lot of the events in IoEventBackend become superfluous. The only ones that can't be captured with a custom IoProvider implementation are the notifications for when input or output handles are closed.
Port the main crate to use the new DriverHooks interface. The main difference here is that IoSetup and IoEvents are merged into a single BridgeState struct, which becomes a "wholly owned subsidiary" of the ProcessingSession type. The core insight here is that what we need to do is collect all of the session functionality that has to be accessed by the C/C++ engines -- the DriverHooks implementation, essentially -- in one struct, which is a subset of the overall ProcessingSession state. When a C/C++ engine is running, it maintains a mutable reference to that *subset* of state, but it makes our life a lot easier if we can isolate that bit of the state from the overall ProcessingSession struct. Now, this subset must include the ProcessingSession's whole I/O stack, so it's a non-trivial subset, but overall the boundaries of the DriverHooks trait help us isolate what has to go in that subset and what can stay separate.
BREAKING: Here we change shell-escape to be implemented by invoking a new driver hook. This addition on its own need not be a breaking change, and indeed the DriverHooks trait is designed to allow this to be the case. However, since we've already accumulated some breaks, we go ahead and breaks the related C/C++ API and also the unrelated API of the file-closed event APIs, which really ought to take the status backend as a parameter in the same way that virtually all of the other I/O APIs do.
This removes the new API because it will no longer be needed in the updated implementation of shell-escape. This would be a breaking change, but the new API has never been released.
This goes back to the older boolean shell-escape API, which is all we need with the iterated architecture. We also track the C/C++ API change that was just made in bridge_core. Also we turn off diagnostic printouts when shell-escape is enabled, since the Tectonic driver can give much nicer diagnostics about what's happening there. This would be a breaking change, but the previous API was never released.
Gather up the previous API revisions and use them to provide a new shell-escape implementation. This one happens inside the driver's BridgeState struct, which has access to the detailed driver I/O and so doesn't need to muck around with the internals of the MemoryIo manually.
|
It is done.
|
This is a simple implementation of shell escape that runs the command in a shell. Resolves #38.
I haven't looked closely at how XeTeX handles shell escapes (e.g. how it handles quotes), so there may be differences with how some commands are interpreted. I've only tested this with a few basic commands and they seems to work.