Pridavanie novych entit z ITMS API v2 #23

eronisko · 2017-10-05T19:18:00Z

Pridavanie novych entit z ITMS API v2 (#22)

…Discrepancy

eronisko · 2017-10-09T13:36:08Z

@ebalgava pridal som nejake nove entity (nezrovnalost,pohladavkovyDoklad,projektovyUkazovatel) a updatol nejake starsie tabulky. Ak mas cas, bol by som zvedavy na tvoju spatnu vazbu, najma na 7fe113f#diff-f9ae8314082d8229b38da27dd6c2642d

eronisko · 2017-10-09T13:40:32Z

app/jobs/itms_job.rb

+    unit = Itms::AccountsReceivableDocument.find_by(itms_id: json['id'])
+    return unit if unit.present?
+
+    Itms::SyncAccountsReceivableDocumentJob.perform_now(json['id'], downloader: downloader)


@jsuchal Je toto OK? Ta filozofia je taka, ze objekt sa neulozi pokial nebudu ulozene vsetky jeho associated objekty. A kazdy class dostane svoj 'sync job', ktory bude (synchronne) spustat sync joby tych associated objektov.

Ano toto dava zmysel pokym tam neni cyklus.

eronisko · 2017-10-09T13:46:29Z

app/jobs/itms/sync_discrepancy_job.rb

+      d.itms_href = json['href']
+      d.itms_created_at = json['createdAt']
+      d.itms_updated_at = json['updatedAt']
+      d.save!


@jsuchal Tymto (skory zapis nekompletneho zaznamu v transakcii) by som sa vyhol deadlockom (ked Objekt A potrebuje Objekt B a Objekt B potrebuje Objekt A).

Takto mozem vytvorit vztahy v self-referential join tabulkach (napr. nezrovnalost.suvisiace_nezrovnalosti) predtym nez su tie objekty kompletne vyskladane.

Asi uplne nerozumiem ako to riesi ten deadlock, kedze transakciu nevidiet pokym nie je commit.

Hej, riesi to zrejme len deadlock v ramci jednej transakcie. Na paralelne transakcie budem musiet vymysliet nieco lepsie.

Applies to Itms::NrfcApplication, Itms::PaymentClaim, Itms::Project

eronisko · 2017-11-23T17:11:48Z

@jsuchal spravil by si review?

Funguje to uz tak ako ma (potahalo mi to cely dataset). Na zaciatku to stale robi deadlocky, ale ked uz je vacsina dat potahanych, ide to v pohode aj paralelne.

Nejakych par entit (cca 45) sa cez API nezobrazuju ako by mali, takze ich to nevie stiahnut. V ITMS uz na to maju bug reporty a robia na tom.

pavolzbell · 2017-11-27T16:14:15Z

app/jobs/itms/sync_activity_job.rb

+    json = downloader.get_json_from_href(itms_href)
+
+    ActiveRecord::Base.transaction do
+      pa = Itms::Activity.find_or_create_by!(itms_id: json['id'])


pa = Itms::Activity co znamena pa? (napr. toto mi je jasne ard = Itms::AccountsReceivableDocument)

pavolzbell · 2017-11-27T16:14:59Z

app/jobs/itms/sync_activity_type_job.rb

+    json = downloader.get_json_from_href(itms_href)
+
+    ActiveRecord::Base.transaction do
+      u = Itms::ActivityType.find_or_create_by!(itms_id: json['id'])


u = Itms::ActivityType, preco nie at = ...?

pavolzbell · 2017-11-27T16:16:43Z

app/jobs/itms/sync_all_operational_programs_job.rb

+class Itms::SyncAllOperationalProgramsJob < ItmsJob
+  def perform(downloader: ItmsJob::Downloader)
+    json_list = downloader.get_json_from_href('/v2/operacneProgramy')
+    json_list.map do |json|


preco map a nie each?

pavolzbell · 2017-11-27T16:22:51Z

app/jobs/itms/sync_all_nrfc_applications_job.rb

+  def perform(downloader: ItmsJob::Downloader)
+    json = downloader.get_json_from_href('/v2/zonfp/prijate')
+    hrefs = json.map { |item| item['href'] }
+    hrefs.each { |href| Itms::SyncNrfcApplicationJob.perform_later(href) }


sledujem tu istu podobnost medzi SyncAll* jobmi:

napr. Itms::SyncAllNrfcApplicationsJob

json = downloader.get_json_from_href('/v2/zonfp/prijate') hrefs = json.map { |item| item['href'] } hrefs.each { |href| Itms::SyncNrfcApplicationJob.perform_later(href) }

vs.

napr. Itms::SyncAllOperationalProgramsJob

json_list = downloader.get_json_from_href('/v2/operacneProgramy') json_list.map do |json| Itms::SyncOperationalProgramJob.perform_later(json['href']) end

nebolo by dobre tento zapis zjednotit?

som za prvu moznost, pripadne by som upravil na:

json = downloader.get_json_from_href('/v2/zonfp/prijate') json.each { |object| Itms::SyncNrfcApplicationJob.perform_later(object['href']) }

pavolzbell · 2017-11-27T16:24:36Z

app/jobs/itms/sync_all_priority_axes_job.rb

+    operational_programs_json = downloader.get_json_from_href('/v2/operacneProgramy')
+    operational_programs_ids = operational_programs_json.map { |json| json['id'] }
+
+    operational_programs_ids.map do |op_id|


preco map?

pavolzbell · 2017-11-27T16:24:53Z

app/jobs/itms/sync_all_priority_axes_job.rb

+
+    operational_programs_ids.map do |op_id|
+      priority_axes_json = downloader.get_json_from_href("/v2/operacneProgramy/#{op_id}/prioritneOsi")
+      priority_axes_json.map { |json| Itms::SyncPriorityAxisJob.perform_later(json['href']) }


opat, preco map?

pavolzbell · 2017-11-27T16:54:03Z

app/jobs/itms/sync_all_priority_axes_job.rb

@@ -0,0 +1,11 @@
+class Itms::SyncAllPriorityAxesJob < ItmsJob
+  def perform(downloader: ItmsJob::Downloader)
+    operational_programs_json = downloader.get_json_from_href('/v2/operacneProgramy')


Zvazil by som drzat sa jednotneho pomenovania premennych, niekde mame json['href'], inde mame item['href'], atd. Potom niekde mame operational_programs_json = downloader.get_json_from_href(...), inde mame json = downloader.get_json_from_href(...) atd.

Pre downloader.get_json_from_href(...) / ...['href'] by som zvolil json = ... / object['href'] alebo object = ... / pair['href'], pozri http://json.org

pavolzbell · 2017-11-27T16:56:08Z

app/jobs/itms/sync_announced_proposal_call_job.rb

+            typ_aktivity: activity_type
+        )
+      end
+    end.flatten


tato syntax mi nesedi end.flatten, pozri https://github.com/bbatsov/ruby-style-guide#single-line-blocks

pavolzbell · 2017-11-27T16:56:59Z

app/jobs/itms/sync_announced_proposal_call_job.rb

+        activity_type = find_or_create_activity_type_by_json(activity_type_json, downloader)
+
+        scope.find_or_create_by!(
+            konkretny_ciel: specific_goal,


podla mna, stacia 2 medzery. pozri https://github.com/bbatsov/ruby-style-guide#no-double-indent

* Prefer each() over map() * Fix double indents * Do not chain after multi-line blocks

eronisko added 3 commits October 5, 2017 19:42

First spec for Itms::Discrepancy sync job

4d83c8d

Add webmock

6a368c1

Sync job works for first-level attributes of Itms::Discrepancy

ee0791c

eronisko added in progress ITMS labels Oct 5, 2017

eronisko added 14 commits October 5, 2017 22:42

Fix Itms::Discrepancy sync job

1ba20c9

API change detection job

6965cc8

Add timestamps to Itms::Discrepancy table

7e306e5

Add codelist table Itms::Code

50c046d

Make Itms::Code columns optional as per schema

7144ee2

Add codelist attributes to Itms::Discrepancy

5599b7a

Itms::Subject v2 and its sync job

4590e72

Add subject-based attributes to Itms::Discrepancy

5d9316a

Update Itms::Specific goal and add its sync job

8e6c738

Add Itms::SpecificGoal to Itms::Discrepancy

769327b

Add Itms::AccountsReceivableDocument and its sync job

db9dbca

find_or_initialize_code_by_json -> find_or_create_code_by_json

da99c20

Add timestamps to Itms::AccountsReceivableDocument table

06c784f

Add Itms::ProjectIndicator and its sync job

7c14b1c

eronisko force-pushed the itms_new_entities branch from 7cbbd67 to 7c14b1c Compare October 9, 2017 09:26

eronisko added 4 commits October 9, 2017 14:47

Add related discrepancies and accounts receivable documents to Itms::…

a59f700

…Discrepancy

Add discrepancy types to Itms::Discrepancy

ee633ce

Handle null values for related objects in JSON

32023ec

Updated structure.sql

7fe113f

eronisko commented Oct 9, 2017

View reviewed changes

eronisko added 3 commits October 9, 2017 17:14

Add Itms::ActivityType and its sync job

51a8370

Updated Itms::PriorityAxis for APIv2

63a2cf9

Updated Itms::OperationalProgram for APIv2

343a423

eronisko added 9 commits November 22, 2017 15:57

Add missing newlines

75c03c1

Un-delete Itms::DetectApiChangeJob. Use formatted JSON in comparison

0ec6135

Updated swagger.json

249b66f

Define some of the find_or_create_XYZ_by_json methods dynamically

9b2c348

Add rake tasks.

21d1f31

Remove old (and redundant) 'itms.kody' table

cfb8fa2

Refactoring: use own downloader for ITMS

1bdfad8

Add Itms::PaymentClaimSubmittingSubject to Itms::PaymentClaim

c29d557

Fix ItmsJob::Downloader.href_exists?

9111e66

eronisko removed the in progress label Nov 23, 2017

eronisko added 2 commits November 23, 2017 17:40

queue_as :itms

5a2684a

Add ekosystem_stav column to show current status of an entity

0308513

Applies to Itms::NrfcApplication, Itms::PaymentClaim, Itms::Project

eronisko added 3 commits November 24, 2017 12:33

Fix Itms::SyncPaymentClaimJob to use ItmsJob::Downloader

923fe17

Install & configure Rollbar

c6838e3

Fix Itms::DetectApiChangeJob to inherit from ItmsJob

d79f92e

pavolzbell reviewed Nov 27, 2017

View reviewed changes

eronisko added 6 commits November 27, 2017 19:19

Fix block variable names

3c369b8

Unify SyncAll jobs

0922a7a

Code style fixes

0911ec5

* Prefer each() over map() * Fix double indents * Do not chain after multi-line blocks

map() -> each() fix.

b8e74f3

Updated swagger.json

edc187f

Pass class_name value as string to prevent unnecessary eager loading

58d8d5d

jsuchal approved these changes Dec 1, 2017

View reviewed changes

eronisko merged commit 1d059f6 into master Dec 1, 2017

eronisko deleted the itms_new_entities branch December 1, 2017 13:39

This was referenced Dec 7, 2017

ITMS: Chyba ZoP 🡒 Deklarovane Vydavky #13

Closed

Chyba itms.zonfp_zamietnute.hodnotitelia #15

Closed

Verejne obstaravanie -> obstaravatel nerozlisuje medzi subjektami a dodavatelmi #17

Closed

Niektore subjekty su v itms.subjekty viac krat #16

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pridavanie novych entit z ITMS API v2 #23

Pridavanie novych entit z ITMS API v2 #23

eronisko commented Oct 5, 2017

eronisko commented Oct 9, 2017

eronisko Oct 9, 2017

jsuchal Oct 17, 2017

eronisko Oct 9, 2017

jsuchal Oct 17, 2017

eronisko Oct 18, 2017

eronisko commented Nov 23, 2017

pavolzbell Nov 27, 2017

pavolzbell Nov 27, 2017

pavolzbell Nov 27, 2017

pavolzbell Nov 27, 2017

pavolzbell Nov 27, 2017

pavolzbell Nov 27, 2017

pavolzbell Nov 27, 2017

pavolzbell Nov 27, 2017

pavolzbell Nov 27, 2017

Pridavanie novych entit z ITMS API v2 #23

Pridavanie novych entit z ITMS API v2 #23

Conversation

eronisko commented Oct 5, 2017

eronisko commented Oct 9, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eronisko commented Nov 23, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment