Add Support for Vimwiki Syntax

[sk8ingdom]

The syntax is defined here: https://code.google.com/p/vimwiki/wiki/Syntax
It's very close to the Google wiki syntax defined here: https://code.google.com/p/support/wiki/WikiSyntax

Is this possible? Are there other vimwiki users that want to take advantage of pandoc??

Thanks!

[pixelpax]

I also want this! Seems close to existing options!

[ycpei]

I want this too. It would be helpful for me to use Jekyll with vimwiki. I'm also interested in implementing it if help is needed.

[ycpei]

I'm going to start working on this. Has there been any progress on it here or somewhere else?

[LarsEKrueger]

In the mean time, here's a kludge: https://github.com/LarsEKrueger/pandoc-vimwiki

It's a pandoc filter to support some of the extra markup.

[ycpei]

I have written a Vimwiki reader. It is a preliminary version, and I have listed the progress at the end of this post. I also wrote a simple test. The files are /test/vimwiki-reader.{wiki,html}.

I'm rather new to Haskell so any feedbacks are welcome :)

I couldn't find a dev branch so the PR is submitted to master. Please let me know what I can do to get it merged. Thanks.

progress:

• block parsers:
  • header
  • hrule
  • comment
  • blockquote -- currently only accepting four spaces rather than a mix of spaces and tabs
  • preformatted -- need to implement preformatted with attributes
  • displaymath - only $$ $$ delimiter, as there does not seem to be support in AST for general environments like equation, align etc?
  • bulletlist / orderedlist -- currently not calculating mixed tabs and spaces indentations for nesting levels.
    • orderedlist with 1., i., a) etc identification.
    • todo lists -- see https://github.com/LarsEKrueger/pandoc-vimwiki
  • table
    • centered table -- there does not seem to be a table builder in Pandoc.Builder that accepts attributes
    • colspan and rowspan -- pandoc limitation, see issue Support for table column spans, table attributes in AST #1024
  • paragraph
  • definition list
• inline parsers:
  • bareURL
  • strong
  • emph
  • strikeout
  • code
  • link
    • with thumbnails -- pandoc limitation? can't find builder of link with thumbnails
  • image -- yet to implement images with attributes - same as in preformatted
  • inline math
  • tag
  • sub- and super-scripts
• misc:
  • TODO: mark
  • placeholders: %title and %date -> metadata, %template -> template

[ycpei]

To implement everything I have the following questions:

1. Indentation. The parsers for blockquotes, bullet lists and ordered lists all require finding out the indentations at the beginning of the lines. For example
   \t may be width 5 or 3 depending on the tabwidth. Is there a function in the Pandoc module that can output the tabwidth?
   The current implementation simply uses the length function (in bullet/ordered lists) or assume the indentation is all spaces (in blockquotes) to determine the indentation.
2. Name-value pair attributes. The parsers for preformatted texts and images are implemented without taking attributes.
   I thought this was probably dealt in other parsers and didn't want to reinvent the wheels:
   Is there a function in the Pandoc module that can convert attributes written in html to Attr?
   Like for example, converting the string "class = "python" style = "width:150px"" to ("", ["python"], [("width", "150px")])?
3. Maths. display math with environments like equation or align do not seem to work. When converting to html the string in display math is automatically enclosed in $$ delimiter. Any solutions to that?
4. Table attributes. Is there a way to parse tables with attributes?
5. Is there a way to parse links with thumbnails, that is, pictures that are also links? I can't find a function in Text.Pandoc.Builder that does that.

[jgm]

+++ Yuchen Pei [May 28 17 18:30 ]:

To implement everything I have the following questions: 1. Indentation. The parsers for blockquotes, bullet lists and ordered lists all require finding out the indentations at the beginning of the lines. For example \t may be width 5 or 3 depending on the tabwidth. Is there a function in the Pandoc module that can output the tabwidth?

readerTabStop opts (where opts :: ReaderOptions).

The current implementation simply uses the length function (in bullet/ordered lists) or assume the indentation is all spaces (in blockquotes) to determine the indentation.

See indentWith in Text.Pandoc.Parsing. This will parse a certain number of spaces indentation, taking into account tabs. Is that what you're looking for? If instead you want to parse a bunch of spaces, then determine the indent level, you may need to write your own function. You could use tabFilter from Text.Pandoc.Shared to convert to spaces, then use length, but that's inefficient (since you're constructing a new string solely to count its length). Better to write a function (perhaps guided by tabFilter) that just outputs a number.

2. Name-value pair attributes. The parsers for preformatted texts and images are implemented without taking attributes. I thought this was probably dealt in other parsers and didn't want to reinvent the wheels: Is there a function in the Pandoc module that can convert attributes written in html to [1]Attr? Like for example, converting the string "class = "python" style = "width:150px"" to ("", ["python"], [("width", "150px")])?

There is attributes in the Markdown reader, but it's a bit different, since it's specialized to pandoc Markdown attribute specifiers. You could probably reuse some code from that but you'd have to write your own. In the HTML reader the attribute parsing is all handled by the TagSoup HTML tokenizer, but I don't think you can run that just on attributes. (If you have a whole HTML tag, then TagSoup might be some help.)

3. Maths. display math with environments like equation or align do not seem to work. When converting to html the string in display math is automatically enclosed in $$ delimiter. Any solutions to that?

Two options. (1) you can parse them as raw LaTeX blocks. This will work fine if you're targeting LaTeX, but it won't work for other formats. (2) you can parse them as displayMath, and change the environment name. E.g. changed align to aligned, which works inside $$..$$. It's not exactly the same, because you lose the numbering, but it's as good as we can currently do. This is what we do in the LaTeX reader: inlineEnvironments :: PandocMonad m => M.Map String (LP m Inlines) inlineEnvironments = M.fromList [ ("displaymath", mathEnvWith id Nothing "displaymath") , ("math", math <$> mathEnv "math") , ("equation", mathEnvWith id Nothing "equation") , ("equation*", mathEnvWith id Nothing "equation*") , ("gather", mathEnvWith id (Just "gathered") "gather") , ("gather*", mathEnvWith id (Just "gathered") "gather*") , ("multline", mathEnvWith id (Just "gathered") "multline") , ("multline*", mathEnvWith id (Just "gathered") "multline*") , ("eqnarray", mathEnvWith id (Just "aligned") "eqnarray") , ("eqnarray*", mathEnvWith id (Just "aligned") "eqnarray*") , ("align", mathEnvWith id (Just "aligned") "align") , ("align*", mathEnvWith id (Just "aligned") "align*") , ("alignat", mathEnvWith id (Just "aligned") "alignat") , ("alignat*", mathEnvWith id (Just "aligned") "alignat*") ]

4. Table attributes. Is there a way to parse table with attributes?

Not yet. See issue #1024; we plan to add attributes when we add colspan/rowspan, hopefully for pandoc 2.0. For now I suggest including the whole table in a Div with the attributes.

[ycpei]

Thanks for the answers.

I haven't found a convenient way to test when I make changes to the code.

Before making the pull request I simply added the following code to the Vimwiki.hs:

import Control.Monad.Except (throwError)
import Text.Pandoc.Class (PandocIO, runIO)
import Text.Pandoc.Parsing (readWithM, stateOptions, ParserState, ParserT)
import Data.Default -- def is there
import Text.Pandoc.Options (ReaderOptions)
import Control.Monad.Except (throwError)
import Text.Parsec.String (Parser)
import Text.Pandoc.Error (PandocError)
import Text.Parsec.Error (ParseError)
import Text.Parsec (parse)

runParser :: VwParser PandocIO a -> String -> PandocIO a
runParser p s = do
  res <- readWithM p def{ stateOptions = def :: ReaderOptions } s
  case res of
       Left e -> throwError e
       Right result -> return result

testP :: VwParser PandocIO a -> String -> IO (Either PandocError a)
testP p s = runIO $ runParser p (s ++ "\n")

simpleParse :: Parser a -> String -> Either ParseError a
simpleParse p s = parse p "" (s ++ "\n")

so that I can ghci Vimwiki.hs and use testP and simpleParse to quickly test all kinds of parsers, including the auxiliary ones whenever I make small changes to the code.

But I suppose this portion of code shouldn't stay when I commit and update the PR.

So I was thinking of putting it in a separate file, say TestParser.hs, in the same dir (Readers) as the Vimwiki.hs, adding TestParser.hs in the .gitignore, and loading both files together.

However, I can't find a way to load both files with ghci.

I tried ghci Vimwiki.hs TestParser.hs, and got an error saying "can't find the VwParser used in TestParser.hs", which is strange because VwParser is defined in Vimwiki.hs.

I also tried ghci Vimwiki.hs, and in the repl, do :load TestParser.hs, but I got the same error.

Any solutions how I can solve this problem and quickly test my edits in the code?

[LarsEKrueger]

Option 1: Add a unit test. The existing test framework is pretty cool. Check the test/command folder for examples.
Option 2: Try ghci -isrc -idist src/whatever/Vimwiki.hs from the top-level folder. You might need to cabal install missing packages unless you've build pandoc already.

[jgm]

The issue is that you want to interactively test non-exported functions. You can't do that by loading another file that imports your module, since it will only see the exported functions. One approach is to temporarily remove the list of exported functions after your module declaration; this will result in everything being exported. (Obviously, this should only be temporary.) Another approach would be to temporarily add the test functions to the VimWiki reader module, and remove them before the final version.

[ycpei]

@LarsEKrueger @jgm: thanks.

I have fixed all the problems mentioned earlier and implemented more parsers - here's the progress

• block parsers:
  • header
  • hrule
  • comment
  • blockquote
  • preformatted
  • displaymath
  • bulletlist / orderedlist
    • orderedlist with non-# markers, e.g. 1), I), a)...
    • todo lists -- see https://github.com/LarsEKrueger/pandoc-vimwiki
  • table
    • centered table -- used div
    • [O] colspan and rowspan -- pandoc limitation, see issue Support for table column spans, table attributes in AST #1024
  • paragraph
  • definition list
• inline parsers:
  • bareURL
  • strong
  • emph
  • strikeout
  • code
  • link
  • image
  • inline math
  • tag
  • sub- and super-scripts
• misc:
  • TODO: mark
  • placeholders: %title and %date -> metadata, %template -> template

It still needs a bit of cleaning up, but functionality-wise except table colspan, rowspan and placeholders, it can satisfy all my needs from a vimwiki reader, so unless there is demand, I am not planning to implement ordered lists with non-# markers, todo lists, definition lists, subscripts or superscripts.

Concerning placeholders, I wonder how I can implement it. For example, in Vimwiki syntax %title indicates the title of the document, e.g.

%title hello

should parse as changing the title metadata to hello.

After inspecting the Markdown parser, I wrote the following function:

titlePH :: PandocMonad m => VwParser m ()
titlePH = try $ do
  string "%title" >> spaceChar
  title <- (trimInlines . mconcat <$> (manyTill inline newline))
  let meta' = return $ B.setMeta "title" title nullMeta :: F Meta
  updateState $ \st -> st { stateMeta' = stateMeta' st <> meta' }

and added mempty <$ titlePH to the list of block parsers used by the main parser parseVimwiki.

But running parseVimwiki on "%title hello" still gives empty metadata (testP is the interactive parser tester I mentioned before):

*Text.Pandoc.Readers.Vimwiki> testP parseVimwiki  "%title hello"
Right (Pandoc (Meta {unMeta = fromList []}) [])

Why?

[jgm]

+++ Yuchen Pei [Jun 09 17 06:04 ]:

It still needs a bit of cleaning up, but functionality-wise except table colspan, rowspan and placeholders, it can satisfy all my needs from a vimwiki reader, so unless there is demand, I am not planning to implement ordered lists with non-# markers, todo lists, definition lists, subscripts or superscripts.

I'd like it if these other features could be supported. Even if you don't need them, others who use pandoc to convert from vimwiki might. When they find that these things aren't supported, they'll submit a bug report. Unless there's some reason why something can't be supported, we should aim for full support in all the readers. Otherwise it ends up being a maintenance burden later.

Concerning placeholders, I wonder how I can implement it. For example, in Vimwiki syntax %title indicates the title of the document, e.g. %title hello should parse as changing the title metadata to hello. After inspecting the Markdown parser, I wrote the following function: titlePH :: PandocMonad m => VwParser m () titlePH = try $ do string "%title" >> spaceChar title <- (trimInlines . mconcat <$> (manyTill inline newline)) let meta' = return $ B.setMeta "title" title nullMeta :: F Meta updateState $ \st -> st { stateMeta' = stateMeta' st <> meta' } and added mempty <$ titlePH to the list of block parsers used by the main parser parseVimwiki. But running parseVimwiki on "%title hello" still gives empty metadata (testP is the interactive parser tester I mentioned before): *Text.Pandoc.Readers.Vimwiki> testP parseVimwiki "%title hello" Right (Pandoc (Meta {unMeta = fromList []}) []) Why?

Well (not having looked at the details), I'd guess this is because you're updating the state, but never retrieving the metadata from state at the end. Compare the end of parseMarkdown, where the metadata is retrieved from state just before the final Pandoc value is returned. (Of course, the details are a bit different, because of the use of the "F" Monad in that reader. But I hope this gives you the clue you need.)

[ycpei]

@jgm OK thanks. I implemented the non-# markers for ordered lists and the date / title metadata placeholder parsers. Still work in progress though.

[jgm]

Great, let me know when you think it's ready to go.

[ycpei]

Any hint on implementing the following?

template placeholder:

%template template_name

See the vimwiki doc. It applies an html template when converting to html. An example template is the default one here.

nohtml placeholder:

%nohtml

Also can be found in the vimwiki doc near the template placeholder. It forbids conversion to html.

[ycpei]

I realised Pandoc has its own --template option, so I have changed the code to ignore the %template placeholder. In any case, the vimwiki templates and the pandoc templates have different syntax so I think it is better to just use the pandoc template option.

I also realised that I mistook codeblock as <pre> when converted to html, but in fact it is <pre><code>. How do I get <pre> without <code>? Is it rawblock? If so, what is the format field, and how do I pass attributes to the rawblock?

[ycpei]

I assume rawblock is raw html so I have left <pre> to be <pre><code>. One can use the following css to prevent the style of <code> affecting <pre><code>:

*:not(pre) > code {
/*style for code...*/
}

I think this may be ready to merge now. Let me know if there are any bugs or other problems.

[jgm]

There's no pandoc element that gets rendered to HTML with just pre, not pre/code. If there's a vimwiki element that means "pay attention to line breaks and leading spaces, but still parse inline syntax like links," then a pandoc LineBlock might be the right target. You should avoid using rawBlock as much as you can, because a raw block will only render properly in one output format. +++ Yuchen Pei [Jun 15 17 13:02 ]:

…

I realised Pandoc has its own --template option, so I have changed the code to ignore the %template placeholder. In any case, the vimwiki templates and the pandoc templates have different syntax so I think it is better to just use the pandoc template option. I also realised that I mistook codeblock as <pre> when converted to html, but in fact it is <pre><code>. How do I get <pre>? Is it rawblock? If so, what is the format field, and how do I pass attributes to the rawblock? — You are receiving this because you were mentioned. Reply to this email directly, [1]view it on GitHub, or [2]mute the thread. References 1. #863 (comment) 2. https://github.com/notifications/unsubscribe-auth/AAAL5C1TAt7x46TdF204lJ7PW3oJnqotks5sEY27gaJpZM4ArXxq

[ycpei]

@jgm OK thanks. I don't think there is a vimwiki element that behaves like LineBlock. I think codeblock with css (see my previous comment) may be a good solution to get preformatted texts in vimwiki, so my current implementation is simply codeblock. Another reason for doing so is that the MediaWiki reader also seems to be using codeblock for pre.

[jgm]

@ycpei's vimwiki reader has now been merged into master.