Doctest failures on 0.3.5

[domenkozar]

### Failure in /tmp/nix-build-http-api-data-0.3.5.drv-0/http-api-data-0.3.5/src/Web/Internal/FormUrlEncoded.hs:254: expression `fromEntriesByKey [("name",["Nick"]),("color",["red","blue"])]'
expected: fromList [("color","red"),("color","blue"),("name","Nick")]
 but got: fromList [("name","Nick"),("color","red"),("color","blue")]
### Failure in /tmp/nix-build-http-api-data-0.3.5.drv-0/http-api-data-0.3.5/src/Web/Internal/FormUrlEncoded.hs:405: expression `toEntriesByKey [("name", "Nick"), ("color", "red"), ("color", "white")] :: Either Text [(Text, [Text])]'
expected: Right [("color",["red","white"]),("name",["Nick"])]
 but got: Right [("name",["Nick"]),("color",["red","white"])]
### Failure in /tmp/nix-build-http-api-data-0.3.5.drv-0/http-api-data-0.3.5/src/Web/Internal/FormUrlEncoded.hs:487: expression `urlEncodeForm [("name", "Julian"), ("lastname", "Arni")]'
expected: "lastname=Arni&name=Julian"
 but got: "name=Julian&lastname=Arni"
### Failure in /tmp/nix-build-http-api-data-0.3.5.drv-0/http-api-data-0.3.5/src/Web/Internal/FormUrlEncoded.hs:523: expression `urlDecodeForm "name=Greg&lastname=Weber"'
expected: Right (fromList [("lastname","Weber"),("name","Greg")])
 but got: Right (fromList [("name","Greg"),("lastname","Weber")])
### Failure in /tmp/nix-build-http-api-data-0.3.5.drv-0/http-api-data-0.3.5/src/Web/Internal/FormUrlEncoded.hs:580: expression `urlEncodeAsForm Person {name = "Dennis", age = 22}'
expected: "age=22&name=Dennis"
 but got: "name=Dennis&age=22"
### Failure in /tmp/nix-build-http-api-data-0.3.5.drv-0/http-api-data-0.3.5/src/Web/Internal/FormUrlEncoded.hs:695: expression `urlEncodeAsForm Project { projectName = "http-api-data", projectSize = 172 }'
expected: "size=172&name=http-api-data"
 but got: "name=http-api-data&size=172"
Examples: 126  Tried: 116  Errors: 0  Failures: 6

These happened only on 32bit linux, but it could mean they're just transient.

[phadej]

That happens because order of pairs in json object is not stable. @fizruk, we could add e.g. aeson-pretty as a dep to doctests, then encoding will be stable

[fizruk]

But this is not because of aeson, this is because of HashMap.
We could introduce a helper to display fields alphabetically and use that.

[deep-42-thought]

This is still the case (for me) with 0.3.7.2 on archlinux32.

[fizruk]

Ok, how about this?

urlEncodeParams :: [(Text, [Text])] -> BSL.ByteString
urlEncodeParams = toLazyByteString . mconcat . intersperse (shortByteString "&") . map encodePair
  where
    escape = urlEncodeQuery . Text.encodeUtf8

    encodePair (k, "") = escape k
    encodePair (k, v) = escape k <> shortByteString "=" <> escape v

urlEncodeForm :: Form -> BSL.ByteString
urlEncodeForm = urlEncodeParams . toList

urlEncodeFormStable :: Form -> BSL.ByteString
urlEncodeFormStable = urlEncodeParams . sortOn fst . toList

We can use urlEncodeFormStable for all doctests then.

@phadej do you approve?

[phadej]

👍

[phadej]

I don't have 32bit machine to test on. If someone have, than please make a change, verify locally, and submit a patch. @fizruk already did the work, I just want to be sure!

Note: http-api-date isn't broken, it's only about fixing volatile tests on 32bit arch, so it's low priority for me

[deep-42-thought]

ok, I tried applying the following patch:

--- a/http-api-data-0.3.7.2/src/Web/Internal/FormUrlEncoded.hs	2018-01-30 07:53:35.305207735 +0000
+++ b/http-api-data-0.3.7.2/src/Web/Internal/FormUrlEncoded.hs	2018-01-30 07:52:48.429999993 +0000
@@ -507,14 +507,20 @@
 --
 -- >>> urlEncodeForm [("fullname", "Andres Löh")]
 -- "fullname=Andres%20L%C3%B6h"
-urlEncodeForm :: Form -> BSL.ByteString
-urlEncodeForm = toLazyByteString . mconcat . intersperse (shortByteString "&") . map encodePair . toList
+urlEncodeParams :: [(Text, [Text])] -> BSL.ByteString
+urlEncodeParams = toLazyByteString . mconcat . intersperse (shortByteString "&") . map encodePair
   where
     escape = urlEncodeQuery . Text.encodeUtf8
 
     encodePair (k, "") = escape k
     encodePair (k, v) = escape k <> shortByteString "=" <> escape v
 
+urlEncodeForm :: Form -> BSL.ByteString
+urlEncodeForm = urlEncodeParams . toList
+
+urlEncodeFormStable :: Form -> BSL.ByteString
+urlEncodeFormStable = urlEncodeParams . sortOn fst . toList
+
 
 -- | Decode an @application/x-www-form-urlencoded@ 'BSL.ByteString' to a 'Form'.
 --

but probably did something fatally wrong (sry, I'm new to haskell), because I get (both on 32 and 64 bit):

src/Web/Internal/FormUrlEncoded.hs:511:19: error:
    • Couldn't match type ‘[Text]’ with ‘Text’
      Expected type: [(Text, [Text])] -> BSL8.ByteString
        Actual type: [(Text, Text)] -> BSL8.ByteString
    • In the expression:
        toLazyByteString
          . mconcat . intersperse (shortByteString "&") . map encodePair
      In an equation for ‘urlEncodeParams’:
          urlEncodeParams
            = toLazyByteString
                . mconcat . intersperse (shortByteString "&") . map encodePair
            where
                escape = urlEncodeQuery . encodeUtf8
                encodePair (k, "") = escape k
                encodePair (k, v) = escape k <> shortByteString "=" <> escape v
    |
511 | urlEncodeParams = toLazyByteString . mconcat . intersperse (shortByteString "&") . map encodePair
    |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

src/Web/Internal/FormUrlEncoded.hs:519:35: error:
    • Couldn't match type ‘Text’ with ‘[Text]’
      Expected type: Form -> [(Text, [Text])]
        Actual type: Form -> [Item Form]
    • In the second argument of ‘(.)’, namely ‘toList’
      In the expression: urlEncodeParams . toList
      In an equation for ‘urlEncodeForm’:
          urlEncodeForm = urlEncodeParams . toList
    |
519 | urlEncodeForm = urlEncodeParams . toList
    |                                   ^^^^^^

src/Web/Internal/FormUrlEncoded.hs:522:41: error:
    • Variable not in scope:
        sortOn :: ((a0, b0) -> a0) -> [(Text, Text)] -> [(Text, [Text])]
    • Perhaps you want to add ‘sortOn’ to the import list
      in the import of ‘Data.List’
      (src/Web/Internal/FormUrlEncoded.hs:38:1-57).
    |
522 | urlEncodeFormStable = urlEncodeParams . sortOn fst . toList
    |                                         ^^^^^^

[fizruk]

@deep-42-thought ok, hang on, I'll send a PR shortly.