Check the job model still suits our needs

[atruskie]

Since we previously designed the job model, the requirements have changed.

• Proposal: name change (so as not to cause confusion with resque jobs)
  • options: analysis_jobs, custom_jobs, user_jobs, analyses
• Check required fields:
  • Name
  • Standard record metadata (created, modified, & users, etc..)
  • Cache of customised config
  • Total duration of saved search results
  • Total number of audio recordings from saved search results
  • Total number of resque jobs produced from saved search results
  • Date started (not same as modified?)
  • Current status (creating, generating, running, completed, paused, failed)
  • Last status date
  • others?
• Relations:
  • users
  • saved_searches
  • projects (many:many, extra table needed, no API needed)
  • scripts

Note: no need for JobItems equivalent model/relation

[atruskie]

Proposed additional field: cached status/progress (and accompanying last updated field).

Maybe a postgres JSON field where we can store cached counts of each status (success, failed, running, queued).

Will make showing a list of custom jobs and their progressed (say on the custom jobs list page) much more efficient.

[cofiem]

Also check that API for create and status update matches spec in #193

[cofiem]

current db structure:

create_table "jobs", force: :cascade do |t|
    t.string   "name",            limit: 255, null: false
    t.string   "annotation_name", limit: 255
    t.text     "script_settings"
    t.integer  "dataset_id",                  null: false
    t.integer  "script_id",                   null: false
    t.integer  "creator_id",                  null: false
    t.integer  "updater_id"
    t.integer  "deleter_id"
    t.datetime "deleted_at"
    t.datetime "created_at",                  null: false
    t.datetime "updated_at",                  null: false
    t.text     "description"
  end

[cofiem]

@atruskie Does the saved_search specify whether new audio recordings that match the filter are included in the job?

As we discussed, including something like audio_recording.created_at < [date] is how the issue of new recordings will be addressed?

What if a user wants to use an existing saved_search, but change only the audio_recording.created_at < [date] value?

What is there are lots of similar queries with only this value modified?

Not specifying the created_at value may mean that the saved_search will in the future match audio_recordings in other projects not accounted for in the custom_jobs_projects / projects_saved_searches table?

[atruskie]

Like you said in comment, yeah, the matching new recordings trick will simply be a filter. Nothing explicit. Only requires saved search to filter created date to be less than a_date (again as you said).

RE: what if a user want to change that date? simple, we let them edit the date. Simpler yet, we don't even expose any of this functionality in the client until its needed. Good to know we have the architecture to support it though.

What is there are lots of similar queries with only this value modified?

• I don't understand.

Not specifying the created_at value may mean that the saved_search will in the future match audio_recordings in other projects not accounted for in the custom_jobs_projects / projects_saved_searches table?

This should not be an issue. The query is executed once when the job starts (NB: perhaps worth saving the results, say in csv in job folder on disk, if only for debugging). It is correct when run, not important otherwise. If a user wants more recordings included, a new job to gets executed, which will execute once when started, and be correct when started.

[atruskie]

@cofiem have we settled on a name?

I'm reviewing #196 and noticed the API endpoints are all /jobs - I still think the name is ambiguous.

[cofiem]

Decided to change name to analysis_jobs. Any issues with this?

[atruskie]

👍