SPIKE: Elm PWA - Basic Offline Support

[nelsonic]

How much use is a "keep track of all the things" App that doesn't work offline?

We hypothesize that an App meant to help people capture everything on their mind and get stuff done must work offline.

With this in mind, we need to do a Spike¹ to explore using Elm for a Progressive Web App (PWA).

We can only test "offline support" by having some sort of "online" functionality thankfully we already have the "online" counterpart in the form of the capture workflow in dwyl/app-mvp-phoenix

Objectives

• Build a super basic Elm App with a Phoenix back-end (for storing data) /elm-pwa-example
  we only need the "capture" functionality from Sprint Zero with the most basic textarea

  There's a new repo /elm-pwa-example but we will import all the code from dwyl/app-mvp-phoenix once the Google Auth PR has been merged so we have that as a starting point. The reason for having a separate "spike" repo is to "snapshot" the basic PWA functionality in isolation but not have to re-write any of the auth or capture code from scratch.

  • Saving the content of the textarea should:
    • Update the Model (fairly obvious)
    • Should save Model.items to localStorage so that the page can be refreshed without losing the content see: https://elmprogramming.com/saving-app-state.html
    • if the device is online save the item to the backend.

    We may need to write some JS to handle this if we can't do it in Elm.

• Turn phone to "airplane mode" ✈️ (wifi and cellular radios disabled)
• Should display an icon signal_wifi_off in UI to inform the user they're offline.
• Should still be able to view the existing item when offline.
• Should be able to create a new item while offline
• Items created offline should sync to backend when re-connected.
• Meet the criteria specified in Google's PWA Checklist: https://developers.google.com/web/progressive-web-apps/checklist

Todo

• Refresh your Elm Skills if you feel at all "rusty". e.g: by reading
  • Official Elm Guide: https://guide.elm-lang.org
  • Elm in Action: https://www.manning.com/books/elm-in-action
  • Practical Elm Book: https://korban.net/elm/book
  • Programming Elm by Jeremy Fairbank: https://pragprog.com/book/jfelm/programming-elm
  • If you learn anything new during your refresh, please share as comments in this thread and/or a pull request to github.com/dwyl/learn-elm anything we can do to bring up the collective level of Elm skills in the team is of major value to @dwyl 👍
• Create the Elm PWA to fulfil the objectives above.
• Deploy to Heroku so that we can run Lighthouse on it. developers.google.com/web/tools/lighthouse

The PWA does not have to use Elm UI, but it would be a good "bonus level" to attempt to use it.
See: dwyl/learn-elm#145

We have collected a few useful links on Elm PWAs dwyl/learn-elm#54
including a complete application: https://github.com/rl-king/elm-hnpwa
We should be able to get started quite quickly by combining the work done in our dwyl/app-mvp-phoenix
with this https://github.com/opvasger/elm-pwa by following this guide: dwyl/learn-elm#54 (comment)

Constraints

• The Spike does not need to have any fancy functionality, the purpose is just to test the idea.
• We don't want to spend more than 3 days on this, so pace yourself accordingly.
• Must include all steps taken so that someone else can re-produce the app
  i.e. we aren't interested in just the "end result" we want to know exactly how it was done.
  If you're in any doubt what step-by-step instructions look like, see: /todo-list-javascript-tutorial
  The point of a Spike¹ is to share the knowledge acquired with the team
  (and in our case the wider @dwyl community)

¹https://en.wikipedia.org/wiki/Spike_(software_development)

[SimonLab]

Add new Elm project to Phoenix

• Make sure you have Elm installed: https://guide.elm-lang.org/install/
• Create a new elm project in the assert folder
  cd assets & elm init

We should now have a elm.json file and a src folder in assets where our Elm code will be added

Compile Elm to js using Webpack

Phoenix is already using Webpack to create one javascript containing all the code imported in assets/js/app.js

• Learn about the Webpack concepts https://webpack.js.org/concepts/
  • entry: which file to start the bundling from (ie assets/js/app.js)
  • output: where to save the bundle (for phx priv/static/js/app.js)
  • loader: tool used to compile file.

There is a specific webpack loader for Elm: https://github.com/elm-community/elm-webpack-loader that we can use in our Phoenix project to compile and bundle the Elm application

• install the Elm loader. Make sure you are still on the assets folder as it is where the package.json is defined and install elm-webpack-loader with npm:

cd /assets
npm install --save-dev elm-webpack-loader

• add a new rule in /assets/webpack.config.js to use the elm loader to bundle the .elm files:

      {
        test: /\.elm$/,
        exclude: [/elm-stuff/, /node_modules/],
        use: {
          loader: 'elm-webpack-loader',
          options: {
            optimize: true
          }
        }

Create the Elm application

To test that Elm is working well with Phoenix and webpack we can create a simple application:

• Create a Main.elm file in `assets/src with the following code:

module Main exposing (main)
import Html exposing (text)

main = text "This is Elm text!"

• Import the Elm application in /assets/js/app.js:

// import Elm application
import { Elm } from "../src/Main.elm";

var app = Elm.Main.init({
  node: document.getElementById('elm-app')
})

• Create the div html node where the Elm application will be added, for example in lib/app_web/templates/layout/app.html.eex we can add in the body:

    <div id="elm-app"></div>
    <script type="text/javascript" src="<%= Routes.static_path(@conn, "/js/app.js") %>"></script

• You can now run the Phoenix application with mix phx.server and you should be able to see "This is Elm text!".

• You can add changes to the Elm code, Phoenix will automatically detect the modificaiton in app.js and reload the page

[SimonLab]

Now that we have mange to embed and Elm application with Phoenix the next step is to make the application more "usable".

• Can we embed a single page application, see https://guide.elm-lang.org/webapps/
  using the url example (https://guide.elm-lang.org/webapps/navigation.html) we can use Browser.application and continue to nest the application in the div. However the application will not only replace the div but in this case all the html page:
  
• Add Elm application only on /capture endpoint
• Can Elm application work with Phoenix session authentication?

[SimonLab]

So mixing "normal" Phoenix template and a SPA Elm is a bit more tedious to implement.
My first idea was to have a specific javascript file for the SPA Elm and to not bundle it in app.js. However it doesn't seems possible to create multiple output with webpack. We could manually create the spa js file with elm make however this doesn't feel the right way to do this and to not using webpack.

However if we are going to use Elm for most of the feature I think we could use it for all the front end including authentication. Phoenix will only now be used to create the API. I think this might be the way to go and try to avoid rendering Phoenix templates

[SimonLab]

At the end It seems possible to create multiple outputs file with Webpack: https://webpack.js.org/concepts/output/#multiple-entry-points using the [name] placeholder. So we should be able to create a specific js file for the elm application and add to to the specific endpoint and keep the rest of the application using Phoenix template. I still feel that having a more distinct structure might be easier to manage, for example we could have the Elixir Phoenix API on one application and the Elm SPA in another place (repo?). Having a distinct API from the frontend will also allow us to let developers create their own frontend/apps

[nelsonic]

Hi @SimonLab, are you following a particular blog post or tutorial for getting Phoenix and Elm to work together? e.g: https://blog.ispirata.com/get-started-with-elm-0-19-and-phoenix-1-4-291beebb350b
or https://teamgaslight.com/blog/a-starter-project-with-goodies-phoenix-1-dot-4-elm-0-dot-19-and-parcel If so, please share the links that you are following. Other people have attempted to tackle this challenge before, so we should be sharing links and describing where they work or don't work.

Our objective with this spike is just to test the creation of an Elm PWA that sends data to a Phoenix backend. That requires one API endpoint in the Phoenix App e.g: dwyl/mvp#53
We only the most basic of Elm app initialisation and compilation (covered in the links shared above).
As noted in dwyl/learn-elm#54 (comment) the create-elm-app utility creates a PWA out of the box.
So that's what we should be using unless there is a compelling reason not to use it. 💭

As for the eventual architecture of the App we are building.
We will be adopting exactly the same approach as every other SaaS App:

1. Server-side rendered home/landing page describing the benefits of the App e.g: https://trello.com
2. Server-side Authentication with Google and Email+Password e.g: https://trello.com/signup
3. Once the person has successfully authenticated boot into a Full Page SPA that is 100% Elm.
   Here we can render the index.html.eex on the server and include the Flags in the template,
   and the immediately boot into Elm (with Flags) on the Client.
4. All communication with the Phoenix Backend should be done over HTTPS using Http.request
   for example /api/new to create a new "capture" item.
5. Presence (see: SPIKE: Presence - automatically display who is online #255) will be added for viewing the availability of Team Mates.

The trickiest part of all this is getting the Phoenix Session from the Backend into the Elm App.
If you decide to work on that, please open a separate issue, it's way beyond the scope of this SPIKE.

If you have implementation questions that require clarification, please ask them before diving in to avoid wasting time. Please ensure you have read the Books mentioned above both Jeremy and Richard cover most of the ground necessary for this SPIKE and the ELM PWA thread dwyl/learn-elm#54 (comment) has much relevant detail too.

If you had read the Elm PWA thread on learn-elm you would not have needed to use elm init because create-elm-app creates all the files needed ... then it would just be a case of following one of the Phoenix 1.4 + Elm 0.19 tutorials for wiring up the WebPack config and 90% of the setup work would be done.

Please ask more questions and share more links as you are going. 👍

[SimonLab]

Hi @SimonLab, are you following a particular blog post or tutorial for getting Phoenix and Elm to work together?

So yes I've been looking at https://blog.ispirata.com/get-started-with-elm-0-19-and-phoenix-1-4-291beebb350b and also checked How I've added Elm to Club Soda. Since CS Phoenix is now using Webpack to bundle javascript code together. The blog post doesn't explain how webpack works with Elm so I've explained on the comment above the main concepts and how Elm is compiled and bundle to the main app.js file.

As noted in dwyl/learn-elm#54 (comment) the create-elm-app utility creates a PWA out of the box.
So that's what we should be using unless there is a compelling reason not to use it.

My goal by creating the Elm with elm init instead of using the npm package create-elm-app is to understand all the steps for creating a PWA. create-elm-app code is a great reference and it will help me to implement some PWA features but I prefer to keep the application as simple as possible at the begining. Once I feel I understand all the steps we can then use the boilerplate outside of this spike

The trickiest part of all this is getting the Phoenix Session from the Backend into the Elm App.
If you decide to work on that, please open a separate issue, it's way beyond the scope of this SPIKE.

I've also add a look at how to use sessions with Elm and I think https://github.com/rtfeldman/elm-spa-example is doing authentication. I'll create another issue to investigate/learn how to do this 👍

I've spent some time yesterday thinking on how to use Navigation. As we are going to use mostly Elm on frontend I think we will need to use Browser.application "Create an application that manages Url changes."
This type of Elm application are a bit more complicated to setup as compare to Browser.element (which is used in the blog ⬆️ and by create-elm-app).
However for the spike itsel as we only want to test the capture test with PWA we can have an Elm application nested and we don't have to worry about navigation for now. Once the PWA is working I can comeback to test Elm navigation. Hope this explain a bit more my thinking from yesterday.
Working on dwyl/mvp#55

[SimonLab]

A basic Elm application is now on https://github.com/dwyl/elm-pwa-example/tree/gh-pages and hosted with Github pages: https://dwyl.github.io/elm-pwa-example/

The Elm app allow a user to create a new capture. The data is saved with the API defined and hosted on https://dwylapp.herokuapp.com/

The service worker on the application allow us at the moment to have the capture page available offline. This can be tested on Chrome with the Application tab:

[nelsonic]

@SimonLab it would be really good to get this PR dwyl/mvp#52 finished so that we can close the Milestone: https://github.com/dwyl/app/milestone/4 💭
Maybe worth taking a look at this early next week after your move.

[SimonLab]

Working on this issue at the moment and I want to come back to a few points I've blocked on:

• The main blocker was that I assume that the Elm Browser.application function used to build a full Elm application with navigation couldn't be integrated to a Phoenix template: SPIKE: Elm PWA - Basic Offline Support #254 (comment). Since I think I've manage to add em Elm application on an html template so I'm going to double check this point

• Next the other point I want to check is how service worker and manifest.json work with Phoenix. I'll read this article https://www.botsquad.com/2018/03/07/phoenix-sw/ which might give the answer. It looks like the manifest file is added in the priv folder, however it is not clear what is the difference between assets and priv folder, need to read https://elixirforum.com/t/phoenix-directory-structure-when-using-mix-phx-new/20164/7

As mention on the comment above I've manage to create an Elm application which is also a PWA. The goal of this issue is to see if we can embed the elm application in Phoenix.

[SimonLab]

Concerning the first point, the Browser.application function:

This function creates the Elm web application however it will replace all the existing Phoenix endpoints and can't coexist with the other Phoenix views. This means that if we have a welcome page or a login page rendered with Phoenix these pages will be replaced by the Elm application which is not ideal.
We can use instead the Browser.element function which create only some part of an html page with Elm. However this function doesn't provide any navigation management tool and Phoenix will be still used for routing. In this case I think it is better to just use Phoenix templating and to avoid embedding Elm inside a Phoenix view.

Concerning creating a PWA with Phoenix, I have managed to register a service worker however the configuration of the manifest.json file is not obvious and I still haven't found a simple solution (I've tried to apply some suggestions from https://stackoverflow.com/questions/45534076/site-cannot-be-installed-no-matching-service-worker-detected)

However I have manage previously to create an Elm PWA application, see https://github.com/dwyl/elm-pwa-example without using Phoenix.

To resume, I don't think using Elm to create a full application combine with a few Phoenix pages is the right way to go. If we want later on to add PWA to the application my choice would be to only use Phoenix to create the API, then using Elm on its own to create the UI/UX and PWA features. The API can be used later on by other tools to create the UI, for example Flutter.