Switches Stack Overview to asciidoctor builds

[lcawl]

This PR updates the Stack Overview (https://www.elastic.co/guide/en/elastic-stack-overview/master/index.html) such that it's built via Asciidoctor.

Related to #606

[nik9000]

This one renders has some problems:

1. It is missing snippet json files which should be fixed by (Asciidoctor: Support for "open in widget" #627)
2. The edit me links are broken:

https://github.com/elastic/elasticsearch/edit/6.3/x-pack/docs/en/security/authentication/saml-guide.asciidoc

from asciidoc vs:

https://github.com/elastic/stack-docs/edit/6.3/docs/en/stack/../../../../elasticsearch/x-pack/docs/en/security/authentication/saml-guide.asciidoc

from asciidoctor. I'll have to look into those Edit Me links.....

[lcawl]

The CCR content also hits the broken beta tags:

[nik9000]

The edit me links are broken:

I opened #661

[nik9000]

Now that beta and the edit me links are fixed I reran the build locally and I see these file changes. The Modified files are probably ok, but the removed snippets are because Asciidoctor doesn't yet support snippet extraction. We'll need #627 before we can move on this.

[lcawl]

Now that #627 is merged, I get different errors when I try to build this book:

INFO:build_docs:asciidoctor: WARNING: ../../../../elasticsearch/docs/reference/ccr/getting-started.asciidoc: line 148: can't read image at any of ["/doc/elasticsearch/docs/images/icons/callouts/17.png", "/doc/kibana/docs/images/icons/callouts/17.png", "/docs_build/resources/asciidoc-8.6.8/images/icons/callouts/17.png"

I can get rid that message by removing the callouts in https://github.com/elastic/elasticsearch/blob/master/docs/reference/ccr/getting-started.asciidoc on line 134, but they don't seem incorrect to me so I think this is a bug.

INFO:build_docs:asciidoctor: WARNING: security/troubleshooting.asciidoc: line 76: can't read image at any of ["/doc/elasticsearch/docs/images/icons/callouts/17.png", "/doc/kibana/docs/images/icons/callouts/17.png"...

I can get rid of that message by removing the callouts in https://github.com/elastic/stack-docs/blob/master/docs/en/stack/troubleshooting.asciidoc on line 74, but ditto above re this being a bug.

INFO:build_docs:Error executing: asciidoctor -v --trace -r /docs_build/resources/asciidoctor/lib/extensions.rb -b docbook45 -d book ...
INFO:build_docs:/var/lib/gems/2.3.0/gems/asciidoctor-1.5.8/lib/asciidoctor/abstract_block.rb:226:in next_adjacent_block': asciidoctor: FAILED: /doc/stack-docs/docs/en/stack/index.asciidoc: Failed to load AsciiDoc document - undefined method +' for nil:NilClass (NoMethodError)
...

Is this last one related to asciidoctor/asciidoctor#3133?

[nik9000]

Is this last one related to asciidoctor/asciidoctor#3133?

It is, but I have a solution for that in #680

[nik9000]

I can get rid of that message by removing the callouts in https://github.com/elastic/stack-docs/blob/master/docs/en/stack/troubleshooting.asciidoc on line 74, but ditto above re this being a bug.

Yeah. There is something up here. I'll dig into it once I've got #680 in.

[lcawl]

The documentation build errors seem to be solved by #721

[lcawl]

The latest test shows a similar error that I reported in #756. See #760

[nik9000]

The latest test shows a similar error that I reported in #756. See #760

I think the asciidoctor approved fix is to add

[subs=attributes]

above it. If this is pretty rare I think we should probably just do that all of the places we use this.

[lcawl]

The latest issue I'm seeing now is related to code blocks deep in ordered lists:

(where the left is the expected behaviour and the right is the asciidoctor behaviour).
The source looks fine to me in
https://github.com/elastic/stack-docs/blob/7.x/docs/en/stack/security/troubleshooting.asciidoc

[nik9000]

I think this is more of the same "lists are pickier" issue we've hit before. When I do this it seems to work in both AsciiDoc and Asciidoctor:

diff --git a/docs/en/stack/security/troubleshooting.asciidoc b/docs/en/stack/security/troubleshooting.asciidoc
index d12c1f5..97203a8 100644
--- a/docs/en/stack/security/troubleshooting.asciidoc
+++ b/docs/en/stack/security/troubleshooting.asciidoc
@@ -426,36 +426,34 @@ these issues.
 
 . *Symptoms:*
 +
---
 Authentication in {kib} fails and the following error is printed in the {es} 
 logs:
-
++
 ....
 Cannot find any matching realm for [SamlPrepareAuthenticationRequest{realmName=null,
 assertionConsumerServiceURL=https://my.kibana.url/api/security/v1/saml}]
 ....
-
++
 *Resolution:*
-
++
 {es}, {kib} and your Identity Provider need all have the same view on what the
 Assertion Consumer Service URL of the SAML Service Provider is.
-
++
 .. {es} discovers this via the `sp.acs` setting in your {es} SAML realm configuration
 .. {kib} constructs this value using the `server.host` and `server.port` in 
 `kibana.yml`. For instance:
 +
---
 [source, shell]
 -----------------------------------------------
 server.host: kibanaserver.org
 server.port: 3456
 -----------------------------------------------
-
++
 These settings would mean that {kib} would construct the Assertion Consumer
 Service URL as `https://kibanaserver.org:3456/api/secuirity/v1/saml`. However,
 if for example, {kib} is behind a reverse proxy and you have configured the
 following  `xpack.security.public.*` settings:
-
++
 [source, shell]
 -----------------------------------------------
 xpack.security.public:
@@ -463,33 +461,31 @@ xpack.security.public:
   hostname: kibana.proxy.com
   port: 8080
 -----------------------------------------------
-
++
 These settings would instruct {kib} to construct the Assertion Consumer Service
 URL as `https://kibana.proxy.com:8080/api/secuirity/v1/saml`
---
-
++
 .. The SAML Identity Provider is either explicitly configured by the IdP
 administrator or consumes the SAML metadata that are generated by {es} and as
 such contain the same value for the Assertion Consumer Service URL as the one
 that is configured in the the `sp.acs` setting in the {es} SAML realm
 configuration. 
-
++
 The error encountered here indicates that the Assertion Consumer Service URL
 that {kib} has constructed via one of the aforementioned ways
 (`https://my.kibana.url/api/security/v1/saml`) is not the one that {es} is
 configured with. Note that these two URLs are compared as case-sensitive strings
 and not as canonicalized URLs.
-
++
 Often, this can be resolved by changing the `sp.acs` URL in `elasticearch.yml`
 to match the value that {kib} has constructed. Note however, that the SAML IdP
 configuration needs to also be adjusted to reflect this change.
-
++
 Alternatively, if you think {kib} is using the wrong value for the Assertion
 Consumer Service URL, you will need to change the configuration in `kibana.yml`
 by adjusting either the `server.host` and `server.port` to change the URL {kib}
 listens to or the `xpack.security.public.*` settings to make {kib} aware about
 its correct public URL.
---
 
 . *Symptoms:*
 +

I admit that isn't pleasant, but it is something.

[nik9000]

This also seems to work and is shorter:

diff --git a/docs/en/stack/security/troubleshooting.asciidoc b/docs/en/stack/security/troubleshooting.asciidoc
index d12c1f5..82e7675 100644
--- a/docs/en/stack/security/troubleshooting.asciidoc
+++ b/docs/en/stack/security/troubleshooting.asciidoc
@@ -439,6 +439,7 @@ assertionConsumerServiceURL=https://my.kibana.url/api/security/v1/saml}]
 
 {es}, {kib} and your Identity Provider need all have the same view on what the
 Assertion Consumer Service URL of the SAML Service Provider is.
+--
 
 .. {es} discovers this via the `sp.acs` setting in your {es} SAML realm configuration
 .. {kib} constructs this value using the `server.host` and `server.port` in 
@@ -472,8 +473,9 @@ URL as `https://kibana.proxy.com:8080/api/secuirity/v1/saml`
 administrator or consumes the SAML metadata that are generated by {es} and as
 such contain the same value for the Assertion Consumer Service URL as the one
 that is configured in the the `sp.acs` setting in the {es} SAML realm
-configuration. 
-
+configuration.
++
+--
 The error encountered here indicates that the Assertion Consumer Service URL
 that {kib} has constructed via one of the aforementioned ways
 (`https://my.kibana.url/api/security/v1/saml`) is not the one that {es} is

[lcawl]

Thanks, @nik9000 I've implemented the fix in elastic/stack-docs#293

[nik9000]

Let me regenerate the diff for this one.

[nik9000]

There are many changes to the edit me links - they look like they mostly fix broken links actually. So 👍

This kind of thing is fairly common:

- AsciiDoc
+ Asciidoctor
@@ -601,17 +601,7 @@
            </div>
           </div>
           <p>
-           <span class="beta">
-            [
-            <span class="beta_title">
-             beta
-            </span>
-            ]
-            <span class="detail">
-             This functionality is in beta and is subject to change. The design and code is less mature than official GA features and is being provided as-is with no warranties. Beta features are not subject to the support SLA of official GA features.
-            </span>
-           </span>
-           In 6.4 and later, you can use Metricbeat to collect and ship data about Kibana, rather than routing it through Elasticsearch. For example:
+           beta::[] In 6.4 and later, you can use Metricbeat to collect and ship data about Kibana, rather than routing it through Elasticsearch. For example:
           </p>
           <div class="informalfigure">
            <div class="mediaobject">

This should be fixed by #799.

[lcawl]

Not sure if this was affected by #799, but I'm now seeing occurrences like this for https://www.elastic.co/guide/en/elastic-stack-overview/6.7/audit-index.html even after I try to escape the text:

[nik9000]

Not sure if this was affected by #799, but I'm now seeing occurrences like this for https://www.elastic.co/guide/en/elastic-stack-overview/6.7/audit-index.html even after I try to escape the text:

This happens because in Asciidoctor deprecated is a macro and those can't have newlines in them. Further confusing things is that we use a preprocessor to convert our syntax (deprecated[]) into asciidoctor's macro syntax (deprecated:[] or deprecated::[]) and that also only works when deprecated is on one line.

There is a solution! I'd forgotten about it until this came up:

diff --git a/x-pack/docs/en/security/auditing/output-index.asciidoc b/x-pack/docs/en/security/auditing/output-index.asciidoc
index 0d2769e40fb..e684ce2565f 100644
--- a/x-pack/docs/en/security/auditing/output-index.asciidoc
+++ b/x-pack/docs/en/security/auditing/output-index.asciidoc
@@ -1,8 +1,13 @@
 [role="xpack"]
 [[audit-index]]
 === Index audit output
+ifdef::asciidoctor[]
+deprecated::[6.7.0, "The index output type is deprecated and will be removed in 7.0. The sole output for the audit trail will be the <<audit-log-output, logfile>> type."]
+endif::asciidoctor[]
+ifndef::asciidoctor[]
 deprecated[6.7.0, The index output type is deprecated and will be removed in 7.0.
 The sole output for the audit trail will be the <<audit-log-output, logfile>> type.]
+endif::asciidoctor[]

ifdef and ifndef work, and asciidoctor always declares the asciidoctor attribute. Note: I've changed the deprecation into a block macro in asciidoctor because I think that was the original intent. You can make it an inline macro with deprecate:[, note the single :.

[lcawl]

I think rather than fixing with ifdefs, it's time to flip the switch on this book and fix for Asciidoctor thereafter. Here we go!