Documentation page migration (apache#19)

* Documentation initial setup * Adjusted TableOfContents * Replaced toc with shortcode * Added Pipeline development lifecycle pages * Added Programming Guide * Added IO pages * Refactored Codeblock Highlight * Refactor classwrapper in github_sample * Refactored paragraph * Added Transform catalog Python * fixup! Added Programming Guide * Added Documentation transforms section * Added documentation patterns section * Added Documentation runtime and resources section * Refactored language-switcher
robertwb · Apr 30, 2020 · a633fc9 · a633fc9
1 parent c4252ef
commit a633fc9
Show file tree

Hide file tree

Showing 122 changed files with 3,328 additions and 5,481 deletions.
diff --git a/website/www/site/assets/scss/_page-nav.sass b/website/www/site/assets/scss/_page-nav.sass
@@ -58,3 +58,5 @@
       margin-bottom: 0
       ul
         padding-left: 20
+        ul
+          display: none
diff --git a/website/www/site/config.toml b/website/www/site/config.toml
@@ -33,6 +33,10 @@ unsafe= true
 [markup.highlight]
 noClasses = false
 
+[markup]
+  [markup.tableOfContents]
+    endLevel = 4
+
 ## Configuration for BlackFriday markdown parser: https://github.com/russross/blackfriday
 [blackfriday]
 plainIDAnchors = true

diff --git a/website/www/site/content/en/blog/adding-data-sources-to-sql.md b/website/www/site/content/en/blog/adding-data-sources-to-sql.md
@@ -80,9 +80,7 @@ The `TableProvider` classes are under
 
 Our table provider looks like this:
 
-{{% classwrapper class="language-java" %}}
-
-```java
+{{< highlight java >}}
 @AutoService(TableProvider.class)
 public class GenerateSequenceTableProvider extends InMemoryMetaTableProvider {
 
@@ -96,9 +94,7 @@ public class GenerateSequenceTableProvider extends InMemoryMetaTableProvider {
     return new GenerateSequenceTable(table);
   }
 }
-```
-
-{{% /classwrapper %}}
+{{< /highlight >}}
 
 All it does is give a type to the table - and it implements the
 `buildBeamSqlTable` method, which simply returns a `BeamSqlTable` defined by
@@ -111,9 +107,7 @@ allow users to define the number of elements to be emitted per second. We will
 define a simple table that emits sequential integers in a streaming fashion.
 This looks like so:
 
-{{% classwrapper class="language-java" %}}
-
-```java
+{{< highlight java >}}
 class GenerateSequenceTable extends BaseBeamTable implements Serializable {
   public static final Schema TABLE_SCHEMA =
       Schema.of(Field.of("sequence", FieldType.INT64), Field.of("event_time", FieldType.DATETIME));
@@ -147,9 +141,7 @@ class GenerateSequenceTable extends BaseBeamTable implements Serializable {
     throw new UnsupportedOperationException("buildIOWriter unsupported!");
   }
 }
-```
-
-{{% /classwrapper %}}
+{{< /highlight >}}
 
 ## The real fun
 

diff --git a/website/www/site/content/en/blog/beam-kotlin.md b/website/www/site/content/en/blog/beam-kotlin.md
@@ -41,100 +41,68 @@ Here are few brief snippets of code that show how the Kotlin Samples compare to
 
 ### Java
 
-{{% classwrapper class="language-java" %}}
-
-```java
+{{< highlight java >}}
  String filename = String.format(
                     "%s-%s-of-%s%s",
                     filenamePrefixForWindow(intervalWindow),
                     shardNumber,
                     numShards,
                     outputFileHints.suggestedFilenameSuffix);
-```
-
-{{% /classwrapper %}}
+{{< /highlight >}}
 
 ### Kotlin
 
-{{% classwrapper class="language-java" %}}
-
-```java
+{{< highlight java >}}
  // String templating
  val filename = "$filenamePrefixForWindow(intervalWindow)-$shardNumber-of-$numShards${outputFileHints.suggestedFilenameSuffix)"  
-```
-
-{{% /classwrapper %}}
+{{< /highlight >}}
 
 ### Java 
 
-{{% classwrapper class="language-java" %}}
-
-```java
+{{< highlight java >}}
 public static class FormatAsTextFn extends SimpleFunction<KV<String, Long>, String> {
     @Override
     public String apply(KV<String, Long> input) {
         return input.getKey() + ": " + input.getValue();
     }
 }
-```
-
-{{% /classwrapper %}}
+{{< /highlight >}}
 
 ## Kotlin
 
-{{% classwrapper class="language-java" %}}
-
-```java
+{{< highlight java >}}
 public class FormatAsTextFn : SimpleFunction<KV<String, Long>, String>() {
     override fun apply(input: KV<String, Long>) = "${input.key} : ${input.value}"  //Single line functions
 }
-```
-
-{{% /classwrapper %}}
+{{< /highlight >}}
 
 ### Java
 
-{{% classwrapper class="language-java" %}}
-
-```java
+{{< highlight java >}}
 if(tableRow != null){
     formatAndInsert(tableRow);
 }
-```
-
-{{% /classwrapper %}}
+{{< /highlight >}}
 
 ### Kotlin
 
-{{% classwrapper class="language-java" %}}
-
-```java
+{{< highlight java >}}
 tableRow?.let{
     formatAndInsert(it)  // No need for null checks
 }
-```
-
-{{% /classwrapper %}}
+{{< /highlight >}}
 
 ### Java
 
-{{% classwrapper class="language-java" %}}
-
-```java
+{{< highlight java >}}
 String tableName = "testTable";
-```
-
-{{% /classwrapper %}}
+{{< /highlight >}}
 
 ### Kotlin
 
-{{% classwrapper class="language-java" %}}
-
-```java
+{{< highlight java >}}
 val tableName = "testTable"  // Type inferencing
-```
-
-{{% /classwrapper %}}
+{{< /highlight >}}
 
 ## Contributors Welcomed!
 

diff --git a/website/www/site/content/en/blog/looping-timers.md b/website/www/site/content/en/blog/looping-timers.md
@@ -172,9 +172,7 @@ So how do timers help? Well let's have a look at a new transform:
 
 Edit: Looping Timer State changed from Boolean to Long to allow for min value check.  
 
-{{% classwrapper class="language-java" %}}
-
-```java
+{{< highlight java >}}
 public static class LoopingStatefulTimer extends DoFn<KV<String, Integer>, KV<String, Integer>> {
 
     Instant stopTimerTime;
@@ -238,9 +236,7 @@ public static class LoopingStatefulTimer extends DoFn<KV<String, Integer>, KV<St
       }
     }
   }
-```
-
-{{% /classwrapper %}}
+{{< /highlight >}}
 
 There are two data values that the state API needs to keep:
 
@@ -279,9 +275,7 @@ In the @OnTimer block, the following occurs:
 
 And that's it, let's add our transform back into the pipeline:
 
-{{% classwrapper class="language-java" %}}
-
-```java
+{{< highlight java >}}
   // Apply a fixed window of duration 1 min and Sum the results
   p.apply(Create.timestamped(time_1, time_2, time_3)).apply(
     Window.<KV<String, Integer>>into(FixedWindows.<Integer>of(Duration.standardMinutes(1))))
@@ -300,9 +294,7 @@ And that's it, let's add our transform back into the pipeline:
 
      }
   }));
-```
-
-{{% /classwrapper %}}
+{{< /highlight >}}
 
 1. In the first part of the pipeline we create FixedWindows and reduce the value
    per key down to a single Sum.

diff --git a/website/www/site/content/en/blog/splittable-do-fn.md b/website/www/site/content/en/blog/splittable-do-fn.md
@@ -345,9 +345,7 @@ smaller restrictions, and a few others.
 The "Hello World" of SDF is a counter, which takes pairs *(x, N)* as input and
 produces pairs *(x, 0), (x, 1), …, (x, N-1)* as output.
 
-{{% classwrapper class="language-java" %}}
-
-```java
+{{< highlight java >}}
 class CountFn<T> extends DoFn<KV<T, Long>, KV<T, Long>> {
   @ProcessElement
   public void process(ProcessContext c, OffsetRangeTracker tracker) {
@@ -365,13 +363,9 @@ class CountFn<T> extends DoFn<KV<T, Long>, KV<T, Long>> {
 PCollection<KV<String, Long>> input = …;
 PCollection<KV<String, Long>> output = input.apply(
     ParDo.of(new CountFn<String>());
-```
-
-{{% /classwrapper %}}
+{{< /highlight >}}
 
-{{% classwrapper class="language-py" %}}
-
-```py
+{{< highlight py >}}
 class CountFn(DoFn):
   def process(element, tracker=DoFn.RestrictionTrackerParam)
     for i in xrange(*tracker.current_restriction()):
@@ -381,9 +375,7 @@ class CountFn(DoFn):
 
   def get_initial_restriction(element):
     return (0, element[1])
-```
-
-{{% /classwrapper %}}
+{{< /highlight >}}
 
 This short `DoFn` subsumes the functionality of
 [CountingSource](https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/CountingSource.java),
@@ -405,9 +397,7 @@ A slightly more complex example is the `ReadFn` considered above, which reads
 data from Avro files and illustrates the idea of *blocks*: we provide pseudocode
 to illustrate the approach.
 
-{{% classwrapper class="language-java" %}}
-
-```java
+{{< highlight java >}}
 class ReadFn extends DoFn<String, AvroRecord> {
   @ProcessElement
   void process(ProcessContext c, OffsetRangeTracker tracker) {
@@ -433,13 +423,9 @@ class ReadFn extends DoFn<String, AvroRecord> {
     return new OffsetRange(0, new File(filename).getSize());
   }
 }
-```
-
-{{% /classwrapper %}}
+{{< /highlight >}}
 
-{{% classwrapper class="language-py" %}}
-
-```py
+{{< highlight py >}}
 class AvroReader(DoFn):
   def process(filename, tracker=DoFn.RestrictionTrackerParam)
     with fileio.ChannelFactory.open(filename) as file:
@@ -459,9 +445,7 @@ class AvroReader(DoFn):
 
   def get_initial_restriction(self, filename):
     return (0, fileio.ChannelFactory.size_in_bytes(filename))
-```
-
-{{% /classwrapper %}}
+{{< /highlight >}}
 
 This hypothetical `DoFn` reads records from a single Avro file. Notably missing
 is the code for expanding a filepattern: it no longer needs to be part of this