Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: restore operation fails (with no transaction history) #531

Open
awb99 opened this issue May 5, 2022 · 5 comments
Open

[Bug]: restore operation fails (with no transaction history) #531

awb99 opened this issue May 5, 2022 · 5 comments
Labels
bug Something isn't working triage

Comments

@awb99
Copy link

awb99 commented May 5, 2022

What version of Datahike are you using?

0.4.1480

What version of Java are you using?

openjdk 17.0.1 2021-10-19

What operating system are you using?

linux, guix

What database EDN configuration are you using?

{:store {:backend :file
:path "data/datahike-db"}
:keep-history? false}

Describe the bug

I run export-db from a running server.
Then I start on another machine with a blank new datahike database, and add a schema transaction,
then I run import-db from the same dump that I created earlier.

On the restore there are this errors:

Cannot add  #datahike/Datom [36 :db/ident :fulfillment/delivery-date 538003659 true]  because of unique constraint:  #datahike/Datom [39 :db/ident :fulfillment/delivery-date 536870913 true] 

{:error :transact/unique, :attribute :db/ident, :datom #datahike/Datom [36 :db/ident :fulfillment/delivery-date 538003659 true]}

Error during transaction Cannot add #datahike/Datom [36 :db/ident :fulfillment/delivery-date 538003659 true] because of unique constraint: #datahike/Datom [39 :db/ident :fulfillment/delivery-date 536870913 true]

This is an extract from the dump:

#datahike/Datom [36 :db/cardinality :db.cardinality/one 538003659 true]
#datahike/Datom [36 :db/ident :fulfillment/delivery-date 538003659 true]
#datahike/Datom [36 :db/valueType :db.type/instant 538003659 true]
#datahike/Datom [39 :db/cardinality :db.cardinality/one 538003659 true]
#datahike/Datom [39 :db/ident :fulfillment/courier 538003659 true]
#datahike/Datom [39 :db/valueType :db.type/string 538003659 true]

There are a couple hundred lines like this in the beginning. I assume it is the schema specification

Is a restore from a dump expecting that I dont commit my schema first?

Is it possible to do a restore that does not import the schema?

The import works when I delete all the "schema lines" in the dump manually before importing.

What is the expected behaviour?

I have setup the datahike services so that it creates my schema when there is no currently existing database directory
automatically. So what would be nice is to do a db restore WITHOUT including the schema transactions.
This would make sense because if I do not keep a transaction log, and I have schema changes over time,
then the transactions from the dump would fail.

So what I want to do is make sure that when I import from a dump, I have certainity that my current schema is
satisfied from all imported transactions.

If the import with my current schema fails, then I could start an import with no schema, and possibly it could work.
But it will only work if there was no schema change, or if there was a schema change and all transactions in the
dump are according to the schema.

So to me it is essential to import a dump without the schema from the dump. Then I can run a preprocessing script
that makes non compliant transations comply, or simply remove them.

How can the behaviour be reproduced?

I can post something if it shoudl be required. Right now I cannot think of something that would be relevant.

@awb99 awb99 added bug Something isn't working triage labels May 5, 2022
@kordano
Copy link
Member

kordano commented May 6, 2022

Hey @awb99, could you check with version 0.4.1488 since that fixed the history migration (https://github.com/replikativ/datahike/releases/tag/0.4.1488)?

@awb99
Copy link
Author

awb99 commented May 8, 2022

I will. Is the db storage of 0.4.1488 identical to 0.4.1480 ?

BTW: But I had to be very careful in my dependency management. I use datahike, shadow cljs and tml dataset in my project. And they together are not easy to get working together. So might take me some time to change version.

@awb99
Copy link
Author

awb99 commented Jul 21, 2022

0.4.1480 is much better than prior versions. But it is not perfect.

If I do transact a schema to the database before importing, then I get schema constraint warnings that should
not be there.

ERROR [datahike.connector:85] - Error during transaction Cannot add #datahike/Datom [16 :db/ident :woo-order/id 540545603 true] because of unique constraint: #datahike/Datom [52 :db/ident :woo-order/id 536870913 true]

I assume this is because my schema that I commit already defines a higher transaction-id than the schema in the
datahike dump.

@awb99
Copy link
Author

awb99 commented Jul 21, 2022

This documents what went wrong when I import to a brand new db without commiting
a schema first.

In my production database I did add the field :shiphero-product/active and then I
started to import data to it. In the dump the schema appears AFTER the items are
added. Since I use a write-schema it could never have happened like that. So I
think the export somehow got the sort order messed up.

cat data/datahike-dump/eavt-dump -n |grep shiphero-product/active
 62652	#datahike/Datom [16875 :shiphero-product/active true 540548187 true]
 62659	#datahike/Datom [16876 :shiphero-product/active true 540548307 true]
 62666	#datahike/Datom [16877 :shiphero-product/active true 540548228 true]
 62673	#datahike/Datom [16878 :shiphero-product/active true 540548229 true]
 62680	#datahike/Datom [16879 :shiphero-product/active true 540548232 true]
 62687	#datahike/Datom [16880 :shiphero-product/active true 540548246 true]
 62694	#datahike/Datom [16881 :shiphero-product/active true 540548248 true]
 62701	#datahike/Datom [16882 :shiphero-product/active true 540548253 true]
 62708	#datahike/Datom [16883 :shiphero-product/active true 540548308 true]
 62715	#datahike/Datom [16884 :shiphero-product/active true 540548313 true]
 62722	#datahike/Datom [16885 :shiphero-product/active true 540548341 true]
 62852	#datahike/Datom [18305 :shiphero-product/active true 540548281 true]
 63092	#datahike/Datom [19688 :shiphero-product/active true 540548166 true]
 63199	#datahike/Datom [50237 :shiphero-product/active true 540548211 true]
 63206	#datahike/Datom [50238 :shiphero-product/active true 540548225 true]
 63213	#datahike/Datom [50239 :shiphero-product/active true 540548236 true]
 63220	#datahike/Datom [50240 :shiphero-product/active true 540548237 true]
 63227	#datahike/Datom [50241 :shiphero-product/active true 540548295 true]
 63845	#datahike/Datom [312201 :shiphero-product/active true 540548274 true]
 65763	#datahike/Datom [631293 :shiphero-product/active true 540548160 true]
 65770	#datahike/Datom [631294 :shiphero-product/active true 540548312 true]
 65777	#datahike/Datom [631295 :shiphero-product/active true 540548314 true]
 67219	#datahike/Datom [783544 :shiphero-product/active true 540548338 true]
124827	#datahike/Datom [802790 :shiphero-product/active true 540548168 true]
124834	#datahike/Datom [803460 :shiphero-product/active true 540548162 true]
124841	#datahike/Datom [803461 :shiphero-product/active true 540548164 true]
124848	#datahike/Datom [803462 :shiphero-product/active true 540548170 true]
124855	#datahike/Datom [803463 :shiphero-product/active true 540548172 true]
124862	#datahike/Datom [803464 :shiphero-product/active true 540548174 true]
124869	#datahike/Datom [803465 :shiphero-product/active true 540548176 true]
124876	#datahike/Datom [803466 :shiphero-product/active false 540548178 true]
124968	#datahike/Datom [857535 :shiphero-product/active true 540548188 true]
124975	#datahike/Datom [857536 :shiphero-product/active true 540548226 true]
124982	#datahike/Datom [857537 :shiphero-product/active true 540548227 true]
124989	#datahike/Datom [857538 :shiphero-product/active true 540548231 true]
124996	#datahike/Datom [857539 :shiphero-product/active true 540548239 true]
125003	#datahike/Datom [857540 :shiphero-product/active true 540548244 true]
125010	#datahike/Datom [857541 :shiphero-product/active true 540548252 true]
125017	#datahike/Datom [857542 :shiphero-product/active true 540548254 true]
125024	#datahike/Datom [857543 :shiphero-product/active true 540548256 true]
125902	#datahike/Datom [1010890 :shiphero-product/active true 540548157 true]
125908	#datahike/Datom [1010891 :shiphero-product/active true 540548158 true]
125914	#datahike/Datom [1010892 :shiphero-product/active false 540548161 true]
125920	#datahike/Datom [1010893 :shiphero-product/active false 540548163 true]
125926	#datahike/Datom [1010894 :shiphero-product/active false 540548165 true]
125932	#datahike/Datom [1010895 :shiphero-product/active false 540548169 true]
125938	#datahike/Datom [1010896 :shiphero-product/active false 540548171 true]
125944	#datahike/Datom [1010897 :shiphero-product/active false 540548173 true]
125950	#datahike/Datom [1010898 :shiphero-product/active false 540548175 true]
125956	#datahike/Datom [1010899 :shiphero-product/active false 540548177 true]
125962	#datahike/Datom [1010900 :shiphero-product/active true 540548179 true]
125969	#datahike/Datom [1010901 :shiphero-product/active true 540548189 true]
125976	#datahike/Datom [1010902 :shiphero-product/active true 540548190 true]
125983	#datahike/Datom [1010903 :shiphero-product/active true 540548191 true]
125990	#datahike/Datom [1010904 :shiphero-product/active true 540548192 true]
125997	#datahike/Datom [1010905 :shiphero-product/active true 540548193 true]
126004	#datahike/Datom [1010906 :shiphero-product/active true 540548195 true]
126011	#datahike/Datom [1010907 :shiphero-product/active true 540548196 true]
126018	#datahike/Datom [1010908 :shiphero-product/active true 540548197 true]
126025	#datahike/Datom [1010909 :shiphero-product/active true 540548198 true]
126032	#datahike/Datom [1010910 :shiphero-product/active true 540548199 true]
126039	#datahike/Datom [1010911 :shiphero-product/active true 540548200 true]
126046	#datahike/Datom [1010912 :shiphero-product/active true 540548203 true]
126053	#datahike/Datom [1010913 :shiphero-product/active true 540548205 true]
126060	#datahike/Datom [1010914 :shiphero-product/active true 540548206 true]
126067	#datahike/Datom [1010915 :shiphero-product/active true 540548207 true]
126074	#datahike/Datom [1010916 :shiphero-product/active true 540548208 true]
126081	#datahike/Datom [1010917 :shiphero-product/active true 540548209 true]
126088	#datahike/Datom [1010918 :shiphero-product/active true 540548210 true]
126095	#datahike/Datom [1010919 :shiphero-product/active true 540548214 true]
126102	#datahike/Datom [1010920 :shiphero-product/active true 540548215 true]
126109	#datahike/Datom [1010921 :shiphero-product/active true 540548216 true]
126116	#datahike/Datom [1010922 :shiphero-product/active true 540548218 true]
126123	#datahike/Datom [1010923 :shiphero-product/active true 540548220 true]
126130	#datahike/Datom [1010924 :shiphero-product/active true 540548221 true]
126137	#datahike/Datom [1010925 :shiphero-product/active true 540548222 true]
126144	#datahike/Datom [1010926 :shiphero-product/active true 540548223 true]
126151	#datahike/Datom [1010927 :shiphero-product/active true 540548224 true]
126158	#datahike/Datom [1010928 :shiphero-product/active true 540548230 true]
126165	#datahike/Datom [1010929 :shiphero-product/active true 540548233 true]
126172	#datahike/Datom [1010930 :shiphero-product/active true 540548234 true]
126179	#datahike/Datom [1010931 :shiphero-product/active true 540548235 true]
126186	#datahike/Datom [1010932 :shiphero-product/active true 540548238 true]
126193	#datahike/Datom [1010933 :shiphero-product/active true 540548240 true]
126200	#datahike/Datom [1010934 :shiphero-product/active true 540548241 true]
126207	#datahike/Datom [1010935 :shiphero-product/active true 540548242 true]
126214	#datahike/Datom [1010936 :shiphero-product/active true 540548243 true]
126221	#datahike/Datom [1010937 :shiphero-product/active true 540548245 true]
126228	#datahike/Datom [1010938 :shiphero-product/active true 540548247 true]
126235	#datahike/Datom [1010939 :shiphero-product/active true 540548250 true]
126246	#datahike/Datom [1010940 :shiphero-product/active true 540548255 true]
126253	#datahike/Datom [1010941 :shiphero-product/active true 540548259 true]
126260	#datahike/Datom [1010942 :shiphero-product/active true 540548260 true]
126267	#datahike/Datom [1010943 :shiphero-product/active true 540548262 true]
126274	#datahike/Datom [1010944 :shiphero-product/active true 540548264 true]
126281	#datahike/Datom [1010945 :shiphero-product/active true 540548265 true]
126288	#datahike/Datom [1010946 :shiphero-product/active true 540548266 true]
126295	#datahike/Datom [1010947 :shiphero-product/active true 540548267 true]
126302	#datahike/Datom [1010948 :shiphero-product/active true 540548268 true]
126309	#datahike/Datom [1010949 :shiphero-product/active true 540548271 true]
126316	#datahike/Datom [1010950 :shiphero-product/active true 540548275 true]
126323	#datahike/Datom [1010951 :shiphero-product/active true 540548276 true]
126330	#datahike/Datom [1010952 :shiphero-product/active true 540548277 true]
126337	#datahike/Datom [1010953 :shiphero-product/active true 540548278 true]
126344	#datahike/Datom [1010954 :shiphero-product/active true 540548279 true]
126351	#datahike/Datom [1010955 :shiphero-product/active true 540548280 true]
126358	#datahike/Datom [1010956 :shiphero-product/active true 540548282 true]
126365	#datahike/Datom [1010957 :shiphero-product/active true 540548283 true]
126372	#datahike/Datom [1010958 :shiphero-product/active true 540548284 true]
126379	#datahike/Datom [1010959 :shiphero-product/active true 540548285 true]
126386	#datahike/Datom [1010960 :shiphero-product/active true 540548287 true]
126393	#datahike/Datom [1010961 :shiphero-product/active true 540548288 true]
126399	#datahike/Datom [1010962 :shiphero-product/active true 540548289 true]
126406	#datahike/Datom [1010963 :shiphero-product/active true 540548290 true]
126413	#datahike/Datom [1010964 :shiphero-product/active true 540548291 true]
126420	#datahike/Datom [1010965 :shiphero-product/active true 540548292 true]
126427	#datahike/Datom [1010966 :shiphero-product/active true 540548293 true]
126434	#datahike/Datom [1010967 :shiphero-product/active true 540548294 true]
126441	#datahike/Datom [1010968 :shiphero-product/active true 540548296 true]
126448	#datahike/Datom [1010969 :shiphero-product/active true 540548297 true]
126455	#datahike/Datom [1010970 :shiphero-product/active true 540548298 true]
126462	#datahike/Datom [1010971 :shiphero-product/active true 540548299 true]
126469	#datahike/Datom [1010972 :shiphero-product/active true 540548300 true]
126476	#datahike/Datom [1010973 :shiphero-product/active true 540548301 true]
126483	#datahike/Datom [1010974 :shiphero-product/active true 540548302 true]
126489	#datahike/Datom [1010975 :shiphero-product/active true 540548304 true]
126496	#datahike/Datom [1010976 :shiphero-product/active true 540548305 true]
126502	#datahike/Datom [1010977 :shiphero-product/active true 540548306 true]
126508	#datahike/Datom [1010978 :shiphero-product/active true 540548319 true]
126515	#datahike/Datom [1010979 :shiphero-product/active true 540548351 true]
126522	#datahike/Datom [1010980 :shiphero-product/active true 540548352 true]
126529	#datahike/Datom [1010981 :shiphero-product/active true 540548354 true]
126536	#datahike/Datom [1010982 :shiphero-product/active true 540548355 true]
126543	#datahike/Datom [1010983 :shiphero-product/active true 540548356 true]
126617	#datahike/Datom [1012017 :shiphero-product/active false 540548167 true]
126623	#datahike/Datom [1012018 :shiphero-product/active true 540548249 true]
126630	#datahike/Datom [1012019 :shiphero-product/active true 540548353 true]
128589	#datahike/Datom [1245862 :shiphero-product/active true 540548257 true]
128596	#datahike/Datom [1245863 :shiphero-product/active true 540548269 true]
133169	#datahike/Datom [1980911 :shiphero-product/active true 540548159 true]
133176	#datahike/Datom [1980912 :shiphero-product/active true 540548180 true]
133183	#datahike/Datom [1980913 :shiphero-product/active true 540548194 true]
133189	#datahike/Datom [1980914 :shiphero-product/active true 540548202 true]
133196	#datahike/Datom [1980915 :shiphero-product/active true 540548204 true]
133203	#datahike/Datom [1980916 :shiphero-product/active true 540548212 true]
133210	#datahike/Datom [1980917 :shiphero-product/active true 540548213 true]
133217	#datahike/Datom [1980918 :shiphero-product/active true 540548219 true]
133224	#datahike/Datom [1980919 :shiphero-product/active true 540548251 true]
133231	#datahike/Datom [1980920 :shiphero-product/active true 540548258 true]
133238	#datahike/Datom [1980921 :shiphero-product/active true 540548261 true]
133245	#datahike/Datom [1980922 :shiphero-product/active true 540548263 true]
133252	#datahike/Datom [1980923 :shiphero-product/active true 540548309 true]
133259	#datahike/Datom [1980924 :shiphero-product/active true 540548310 true]
133266	#datahike/Datom [1981757 :shiphero-product/active true 540548270 true]
133273	#datahike/Datom [1981758 :shiphero-product/active true 540548272 true]
133280	#datahike/Datom [1981759 :shiphero-product/active true 540548273 true]
133287	#datahike/Datom [1981760 :shiphero-product/active true 540548286 true]
133294	#datahike/Datom [1981761 :shiphero-product/active true 540548315 true]
133300	#datahike/Datom [1981762 :shiphero-product/active true 540548316 true]
133306	#datahike/Datom [1981763 :shiphero-product/active true 540548317 true]
133312	#datahike/Datom [1981764 :shiphero-product/active true 540548318 true]
133318	#datahike/Datom [1981765 :shiphero-product/active true 540548320 true]
133324	#datahike/Datom [1981766 :shiphero-product/active true 540548321 true]
133331	#datahike/Datom [1981767 :shiphero-product/active true 540548322 true]
133338	#datahike/Datom [1981768 :shiphero-product/active true 540548323 true]
133344	#datahike/Datom [1981769 :shiphero-product/active true 540548324 true]
133350	#datahike/Datom [1981770 :shiphero-product/active true 540548325 true]
133356	#datahike/Datom [1981771 :shiphero-product/active true 540548326 true]
133362	#datahike/Datom [1981772 :shiphero-product/active true 540548327 true]
133368	#datahike/Datom [1981773 :shiphero-product/active true 540548328 true]
133374	#datahike/Datom [1981774 :shiphero-product/active true 540548329 true]
133380	#datahike/Datom [1981775 :shiphero-product/active true 540548330 true]
133386	#datahike/Datom [1981776 :shiphero-product/active true 540548331 true]
133392	#datahike/Datom [1981777 :shiphero-product/active true 540548332 true]
133398	#datahike/Datom [1982610 :shiphero-product/active true 540548181 true]
133404	#datahike/Datom [1982611 :shiphero-product/active true 540548182 true]
133410	#datahike/Datom [1982612 :shiphero-product/active true 540548183 true]
133416	#datahike/Datom [1982613 :shiphero-product/active true 540548184 true]
133422	#datahike/Datom [1982614 :shiphero-product/active true 540548185 true]
133428	#datahike/Datom [1982615 :shiphero-product/active true 540548186 true]
133434	#datahike/Datom [1982616 :shiphero-product/active true 540548201 true]
133440	#datahike/Datom [1982617 :shiphero-product/active true 540548217 true]
133447	#datahike/Datom [1982618 :shiphero-product/active true 540548311 true]
133454	#datahike/Datom [1982619 :shiphero-product/active true 540548333 true]
133460	#datahike/Datom [1982620 :shiphero-product/active true 540548334 true]
133466	#datahike/Datom [1982621 :shiphero-product/active true 540548335 true]
133472	#datahike/Datom [1982622 :shiphero-product/active true 540548336 true]
133478	#datahike/Datom [1982623 :shiphero-product/active true 540548337 true]
133484	#datahike/Datom [1982624 :shiphero-product/active true 540548339 true]
133490	#datahike/Datom [1982625 :shiphero-product/active true 540548340 true]
133496	#datahike/Datom [1982626 :shiphero-product/active true 540548342 true]
133502	#datahike/Datom [1982627 :shiphero-product/active true 540548343 true]
133509	#datahike/Datom [1982628 :shiphero-product/active true 540548345 true]
133515	#datahike/Datom [1982629 :shiphero-product/active true 540548346 true]
133522	#datahike/Datom [1982630 :shiphero-product/active true 540548347 true]
133529	#datahike/Datom [1982631 :shiphero-product/active true 540548348 true]
133535	#datahike/Datom [1982632 :shiphero-product/active true 540548349 true]
133541	#datahike/Datom [1982633 :shiphero-product/active true 540548350 true]
133547	#datahike/Datom [1982634 :shiphero-product/active true 540548357 true]
133714	#datahike/Datom [2000103 :shiphero-product/active true 540548303 true]
133721	#datahike/Datom [2000104 :shiphero-product/active true 540548344 true]
136397	#datahike/Datom [2102671 :db/ident :shiphero-product/active 540545603 true]

@awb99
Copy link
Author

awb99 commented Oct 24, 2022

I fixed the import issue with this trick: I remove
all schema definitions on importing the datums. Before
I import the datums, I add the entire schema.

This works because sometimes the database import
has schema definitions that occur AFTER transactions with
this schema occur.

The only non-generic thing in my bugfix is that I
have to give the schema tx db-ids that are BIGGER than the
highest db-id in the db. If I do not do that then it seems
that the schema will get ids starting from 1, but then
it will override the schema with transactions starting
also from db-id 1. My fix has the start of the db-ids
for the schema coded.

(defn schema? [[dbid f v tx-id b]]
  (case f
    :db/cardinality true
    :db/ident true
    :db/valueType true
    :db/unique true
    :db/index true
    false))

(comment
  (schema? [2021167 :db/cardinality :db.cardinality/one 540708805 true])
  (schema? [2021167 :db/valueType :db.type/number 540708805 true])
  (schema? [2018253 :lineitem/id "a4ae178f-513b-4f3c-b8a9-a1d496048241" 540343705 true])

  (remove schema? [[2018253 :lineitem/id "a4ae178f-513b-4f3c-b8a9-a1d496048241" 540343705 true]
                   [2021167 :db/cardinality :db.cardinality/one 540708805 true]])
   ;
  )

(defn assoc-schema-ids [schemas]
  (into []
        (map-indexed (fn [idx itm]
                ;[idx itm]
                       (assoc itm :db/id (+ 2500000 idx)))
                     schemas)))

(defn import-db
  "Import a flat-file of datoms at path into your database."
  [conn path]
  (doseq  [datoms (->> (line-seq (io/reader path))
                       (map read-string)
                         (remove schema?)
                       (partition 1000 1000 nil))]
    (api/transact conn (vec datoms))))



(defn db-import []
  (crbdb/transact (assoc-schema-ids schema))
  ;(crbdb/db-import)
  (info "transacting in blocks of 1000...")
  (import-db  crbdb/conn "data/datahike-dump/eavt-dump")
  )

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working triage
Projects
Status: No status
Development

No branches or pull requests

2 participants