Using the stream outside of the framework

[raisercostin]

I have a lot of functions that process streams. One of them would be
InputStreamUtils.copyBigFile(in:InputStreamProvider,out:OutputStreamProvider)

In order to use the current scalaj api my understanding is that I should do something like this:

def copyBigFile(url:String,out:OutputStreamProvider) = 
    Http(url).execute(inStream=>InputStreamUtils.copyBigFile(InputStreamProvider(inStream),out) )

This doesn't compose well. I would prefer to be able to get an unmanaged InputStream out of the library to use it and close it by myself.

Im thinking of making a pull request with something like https://github.com/yongjunj/scalaj-http/commit/5ecf2946f4f091c5958abf6198c962924ed9c9e7 . Would that be a possible solution? Do you have other suggestions?

Thank you!

[hoffrocket]

Hi, sorry for late reply. Is this limitation preventing you from implementing some functionality? or is this a style preference?

[raisercostin]

Thanks for reply. I was working with methods from org.apache.commons.io.IOUtils (commons-io-2.4) library. A lot of them require an InputStream/Reader parameter. Probably is a style preference that must be addopted by any library that will depend on scalaj-http.
Maybe I should consider changing the interface to a functional style like yours.

More context:

• I was working on a library that already has a lot of methods written on top of InputStreams.
  https://github.com/raisercostin/jedi-io/blob/master/src/main/scala/org/raisercostin/jedi/InputLocation.scala
• The adapter to scalaj-http is here:
  https://github.com/raisercostin/jedi-io/blob/master/src/main/scala/org/raisercostin/jedi/UrlLocation.scala

[zorba128]

I also have similar problem. I think there are few issues here.

1. No built-in way to pass stream outside of "execute" method
   I'd like to use something like request.asStream; naive implementation of request.execute(is => is) will not work, as execute method already closes stream (see finally block in private Http.doConnection).
   For this to work, it would be enough just to have an option to leave input stream open after leaving exec(). Only potential problem is how errors will be handled - should error stream be left open or not. This is somehow connected with next point (it execute handled errors automatically, there would be no problem here).

2. No error handling before/inside of "execute" method
   So it turned out I have to do all my work inside closure passed to execute/exec method. But then:

• my parser has to deal with all the response codes, including non-200 ones
• for errors - as I cannot return desired result - exception has to be thrown
• but then this doesn't work well with Result.throwError - which is applied afterwards

I think problem is Request.execute() should not pass non-200 to parser. As it is meant to be one-liner, framework should handle as much logic as it can. And checking error status and throwing exception for all non-success codes seems better choice, than pushing it to parser which has no idea how to interpret contents (as it doesn't know error code).
One might use Result.throwError and Result.throwServerError, but these applied too late (already
after parser was invoked).

regards,
marcin

[hoffrocket]

Hi Marcin,
execute delegates to the exec method which you can use instead and it takes a (Int, Map[String, IndexedSeq[String]], InputStream) => T The first param is the resultCode and the second is the headers so you can decide how you want to handle the parsing for various conditions

There's also an asString method which takes the same function

[zorba128]

I'd say that from my perspective it would be just fine if there was a possibility to use library just to send http request, without any further processing. I think it should have something like Request.send() that would give me back plain HttpURLConnection. Now this is handled privately inside exec, and it required me to copy full request sending code in order to use library for chunked response processing.

My use of library in the end boils down to:
/** Custom wrapper over scalaj.http web request. Returns response body stream reader.
* Throws exception in case of request configuration issues, for connection error and
* for non-200 response codes.
*/
private def connect(request: HttpRequest): InputStreamReader = ???

which internally calls some code copied from library, such as:
def getResponseHeaders(conn: HttpURLConnection): Map[String, Seq[String]] = ???
def getResponseCharset(headers: Map[String, Seq[String]]): String = ???
and some other minor utils: getStatusLine, is2xx, throwOnNoSuccess, ...

You'll notice - everything is there, I just cannot use it easily :). Good example is HttpResponse class with really nice utilities that would be useful in callback from exec method. You'll even notice HttpResponse signature almost matches parser's signature.

marcin