Dancer & Catmandu

Catmandu integrates very well with Perl frameworks such as Dancer for easy and fun webapp development. In the text below we will use our Catmandu tools to create small demo application: a web search engine that can be fed with CSV files.

This second part of the tutorial assumes some familiarity with Catmandu. If you have not yet read part 1, now would be a good time to do so: Introduction.

Dancer provides an intuitive and minimalistic approach to create web applications. The software package can be easily installed if you have cpanminus on your server:

cpanm Dancer

Creating a web application is as easy as running the command line dancer command

dancer -a DemoApp

Change your working directory to DemoApp and boot up the build-in web server.

cd DemoApp
perl bin/app.pl

Now you should have a running server on http://0.0.0.0:3000.

In the following section we are going to create a web form to upload a CSV file to the webserver. Open your editor and load the file 'lib/DemoApp.pm' which contains all the controller code of your application.

package DemoApp;
use Dancer ':syntax';

our $VERSION = '0.1';

get '/' => sub {
    template 'index';
};

true;

We add a new route '/upload' to the application which will display the content of the views/upload.tt template using the following code snippet.

get '/upload' => sub {
   template 'upload';
};

For this route to work we need to add a new template file views/upload.tt:

<form action="/upload" method="post" enctype="multipart/form-data">
 <p> 
   <input type="file" name="file">
 </p>
 <button type="submit" class="btn btn-large">
   Upload
 </button>
</form>

Restart your webapplication (perl bin/app.pl) and view your web form at http://localhost:3000/upload.

When you click on the submit button a file of your choice will be uploaded via a HTTP POST to the webserver. The contents of this file can be easily displayed by adding the following code to the DemoApp.pm controller:

post '/upload' => sub {
   my $file = request->upload('file');

   $file->content;   
};

Restart your webapplication (perl bin/app.pl) and try it out. Your complete code should look now like this:

package DemoApp;
use Dancer ':syntax';

our $VERSION = '0.1';

get '/' => sub {
  template 'index';
};

get '/upload' => sub {
  template 'upload';
};

post '/upload' => sub {
  my $file = request->upload('file');

  $file->content;
};

true;

Now that we are able to upload a file to the webserver we can add Catmandu code to parse this data and send it to a Solr search engine. We first create a file catmandu.yml in the home directory of the DemoApp which contains the location of our Solr engine and the parameters for the Catmandu importer we are going to use to parse the data (CSV-files in our case)

store:
    default:
        package: "Solr"
        options:
            url: "http://localhost:8983/solr"

importer:
    default:
        package: CSV
        options:
            sep_char: ";"

Reading from a CSV importer and indexing into a Solr Store can be easily added to the DemoApp.pm controller by adding the following lines at the top of the DemoApp.pm module:

use Catmandu -load;
use Catmandu::Sane;

sub store {
    state $bag = Catmandu->store('default')->bag;
}

sub importer {
    my $file = $_[0];

    Catmandu->importer('default', file => $file);
}

It is now very easy to parse CSV data. We change our POST /upload route into:

post '/upload' => sub {
    my $file = request->upload('file');

    to_dumper( importer($file->tempname)->to_array );
};

If you restart the web server and upload a CSVCSV file you should see a Data::Dumper output of all the fields. What we did in the code was to send the path of the uploaded file ($file->tempname) to a CSV importer configured in the catmandu.yml file. Every importer has a to_array method to read all the data into a Perl array. This array is then converted to a Data::Dumper string by the to_dumper call.

Indexing all the fields in the CSV file into a Solr index is straight forward (read your Solr manual how to setup a Solr search engine with your own indexable fields (hint: (schema.xml)[files/schema.xml] file]). All we need is to send the importer to a Solr Catmandu Store with the add_many command and commit the results.

post '/upload' => sub {
    my $file = request->upload('file');

    store->delete_all;

    store->add_many( importer($file->tempname) );

    store->commit;

    redirect '/';
};

Add this stage you'll have a basic website where you can upload a CSV-file into a Solr search engine. Next we are going to add a /search route to query the Solr index.

We start by creating a GET route /search in DemoApp.pm and provide it with a template to display a search box

get '/search' => sub {
    template 'search';
}

Here is the views/search.tt template it will read:

<form method="GET" action="/results">
 <input type="text" name="q">
 <button type="submit" class="btn btn-large">
  Search
 </button>
</form>

Next we are going to provide a GET route /results which will query the Solr index and display all the results:

get '/results' => sub {
    my $q     = param "q";
    my $start = param "start" // 0;
    my $limit = 10;

    my $results = [];

    if ($q) {
       $results = store->search(query => $q, start => $start, limit => $limit)->to_array;
    }

    template 'results' , { results => $results };
};

In the first lines we read the query parameters from the request. In case we have a query string we execute a Solr query and read all the results into a Catmandu::Hits iterator which will be converted into an array. We use the views/results.tt template to display all the results:

<%FOREACH hit IN results%>
   <dl>
    <dt>Title</dt>
    <dd><% hit.title %></dd>
   </dl>
<%END%>

To run this code we need a little change in the Dancer configuration file config.yml to activate the Template Toolkit language for all template files:

template: "template_toolkit"

Restart the webserver and browse to http://0.0.0.0:3000/search to view the search box. When you execute the query *:* you should see a list of all your documents:

This concludes our very short introduction into Catmandu and Dancer. We hope we have provided you with enough material and inspiration for new and exciting projects ;). Please find in the included DemoApp.zip ZIP file an elaborate example of the code used in this tutorial.