Code

feat: import books from Wikisource#9674

Code

Merged

cdrini merged 161 commits intointernetarchive:masterinternetarchive/openlibrary:masterfrom pidgezero-one:9671/feat/add-wikisource-import-scriptpidgezero-one/openlibrary:9671/feat/add-wikisource-import-scriptCopy head branch name to clipboardDec 5, 2024

+830

Commits

Commits on Aug 1, 2024

• first draft

  pidgezero-one

  committed
  19d9874

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for f23baad
  pre-commit-ci[bot]

  committed
  f23baad

• linting

  pidgezero-one

  committed
  34667e7

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for 17575bf
  pre-commit-ci[bot]

  committed
  17575bf

• use a class for imports

  pidgezero-one

  committed
  dca89fb

• Merge branch '9671/feat/add-wikisource-import-script' of https://github.com/pidgezero-one/openlibrary into 9671/feat/add-wikisource-import-script

  pidgezero-one

  committed
  147784a

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for df423ca
  pre-commit-ci[bot]

  committed
  df423ca

• mypy fixes

  pidgezero-one

  committed
  604a5d8

• merge

  pidgezero-one

  committed
  34fe390

• more linting

  pidgezero-one

  committed
  a307eca

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for 1fe4e84
  pre-commit-ci[bot]

  committed
  1fe4e84

• is this deprecated too?

  pidgezero-one

  committed
  d7f4065

• Merge branch '9671/feat/add-wikisource-import-script' of https://github.com/pidgezero-one/openlibrary into 9671/feat/add-wikisource-import-script

  pidgezero-one

  committed
  bcdcdb1

• is this deprecated too?

  pidgezero-one

  committed
  c504f6c

• is this deprecated too?

  pidgezero-one

  committed
  a996e64

Commits on Aug 2, 2024

• improved data model

  pidgezero-one

  committed
  a3c299e

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for 0f2b113
  pre-commit-ci[bot]

  committed
  0f2b113

• reformat name formatter

  pidgezero-one

  committed
  e18ba52

• ruff fix

  pidgezero-one

  committed
  4bdf428

• improve infobox fetching

  pidgezero-one

  committed
  57388b4

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for 31b2a7f
  pre-commit-ci[bot]

  committed
  31b2a7f

• uncomment

  pidgezero-one

  committed
  9492403

• merge

  pidgezero-one

  committed
  dacef92

• remove unnecessary print

  pidgezero-one

  committed
  1186696

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for dd82901
  pre-commit-ci[bot]

  committed
  dd82901

• uncomment imports

  pidgezero-one

  committed
  47b57c8

• Merge branch '9671/feat/add-wikisource-import-script' of https://github.com/pidgezero-one/openlibrary into 9671/feat/add-wikisource-import-script

  pidgezero-one

  committed
  3c82121

• better template check:

  pidgezero-one

  committed
  5e58373

• publishers?

  pidgezero-one

  committed
  2c268b2

• fix array

  pidgezero-one

  committed
  be0d1a8

• unused import

  pidgezero-one

  committed
  d52c109

• different wiki markup strip

  pidgezero-one

  committed
  df683a1

• reduce image calls

  pidgezero-one

  committed
  8e7cb38

• unstash

  pidgezero-one

  committed
  66744ef

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for 5c51bcc
  pre-commit-ci[bot]

  committed
  5c51bcc

• fix newlines

  pidgezero-one

  committed
  b5e4319

• undo comments

  pidgezero-one

  committed
  34da18d

• logger name

  pidgezero-one

  committed
  76a1724

• fix array typing

  pidgezero-one

  committed
  2ce0e17

• more cleanup

  pidgezero-one

  committed
  3645e9e

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for 0e9650b
  pre-commit-ci[bot]

  committed
  0e9650b

Commits on Aug 6, 2024

• dry run outputs to a jsonl file in a gitignored folder

  pidgezero-one

  committed
  9663789

• Merge branch '9671/feat/add-wikisource-import-script' of https://github.com/pidgezero-one/openlibrary into 9671/feat/add-wikisource-import-script

  pidgezero-one

  committed
  8f0810d

• add this directory

  pidgezero-one

  committed
  2a3e617

• .

  pidgezero-one

  committed
  0268018

• Merge branch 'master' into 9671/feat/add-wikisource-import-script

  pidgezero-one

  authored
  99f3c93

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for 5390b54
  pre-commit-ci[bot]

  committed
  5390b54

• .

  pidgezero-one

  committed
  b06b60d

• Merge branch '9671/feat/add-wikisource-import-script' of https://github.com/pidgezero-one/openlibrary into 9671/feat/add-wikisource-import-script

  pidgezero-one

  committed
  d7561f8

• unicode

  pidgezero-one

  committed
  3e10c5d

• remove dry run flag

  pidgezero-one

  committed
  4bd8e54

Commits on Aug 13, 2024

• this produces around 500 records

  pidgezero-one

  committed
  e445276

• wikisource API gives better image results. this script now gets most metadata from wikidata and remaining metadata from wikisource, and retries any failing API requests

  pidgezero-one

  committed
  d633c62

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for 4053014
  pre-commit-ci[bot]

  committed
  4053014

• undo comments

  pidgezero-one

  committed
  e1bd982

• clearer comments

  pidgezero-one

  committed
  62d1798

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for 25d9243
  pre-commit-ci[bot]

  committed
  25d9243

• formatting

  pidgezero-one

  committed
  bbe242f

• formatting

  pidgezero-one

  committed
  15aa2b2

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for cae28f9
  pre-commit-ci[bot]

  committed
  cae28f9

• condense

  pidgezero-one

  committed
  cb2a959

• more cleanup

  pidgezero-one

  committed
  5cb9dc9

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for 5370e41
  pre-commit-ci[bot]

  committed
  5370e41

• more cleanup

  pidgezero-one

  committed
  b96a07d

• precommit

  pidgezero-one

  committed
  4d4d091

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for da00d4e
  pre-commit-ci[bot]

  committed
  da00d4e

• aaaaa

  pidgezero-one

  committed
  d116d6e

Commits on Aug 14, 2024

• more false positives, letter filter literally does not work for reasons unbeknownst to me

  pidgezero-one

  committed
  8d83830

• this is annoying

  pidgezero-one

  committed
  f7e61c0

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for 3018a5f
  pre-commit-ci[bot]

  committed
  3018a5f

• ruff

  pidgezero-one

  committed
  878051c

• ruff

  pidgezero-one

  committed
  9135ff5

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for 113e6a7
  pre-commit-ci[bot]

  committed
  113e6a7

• cmt

  pidgezero-one

  committed
  42c05d0

• Merge branch '9671/feat/add-wikisource-import-script' of https://github.com/pidgezero-one/openlibrary into 9671/feat/add-wikisource-import-script

  pidgezero-one

  committed
  e9fef40

Commits on Aug 15, 2024

• filters

  pidgezero-one

  committed
  5193267

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for 2075f4e
  pre-commit-ci[bot]

  committed
  2075f4e

Commits on Aug 16, 2024

• fix

  pidgezero-one

  committed
  dbac756

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for 916c3ae
  pre-commit-ci[bot]

  committed
  916c3ae

• ruff

  pidgezero-one

  committed
  d1683a5

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for 287bfe0
  pre-commit-ci[bot]

  committed
  287bfe0

• aint no way precommit thinks 'pleas' is a typo

  pidgezero-one

  committed
  4ad37de

• Merge branch '9671/feat/add-wikisource-import-script' of https://github.com/pidgezero-one/openlibrary into 9671/feat/add-wikisource-import-script

  pidgezero-one

  committed
  e8fb019

• comment clarity

  pidgezero-one

  committed
  d8452d8

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for 0d15c54
  pre-commit-ci[bot]

  committed
  0d15c54

• fix publishers

  pidgezero-one

  committed
  db5c4a9

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for 3c46c9a
  pre-commit-ci[bot]

  committed
  3c46c9a

• fix WS-side category filtering

  pidgezero-one

  committed
  31d84a7

• Merge branch '9671/feat/add-wikisource-import-script' of https://github.com/pidgezero-one/openlibrary into 9671/feat/add-wikisource-import-script

  pidgezero-one

  committed
  3dbe7b9

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for d6d303e
  pre-commit-ci[bot]

  committed
  d6d303e

• ruff

  pidgezero-one

  committed
  aa73b6b

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for a8fdcfb
  pre-commit-ci[bot]

  committed
  a8fdcfb

• clean up some re-request loops

  pidgezero-one

  committed
  916dabc

• clean up some re-request loops

  pidgezero-one

  committed
  bb3c30c

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for e28c8cd
  pre-commit-ci[bot]

  committed
  e28c8cd

Commits on Sep 29, 2024

• addresses most PR comments

  pidgezero-one

  committed
  f6f6c99

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for beaf68c
  pre-commit-ci[bot]

  committed
  beaf68c

• precommit

  pidgezero-one

  committed
  e6fe169

• Merge branch '9671/feat/add-wikisource-import-script' of https://github.com/pidgezero-one/openlibrary into 9671/feat/add-wikisource-import-script

  pidgezero-one

  committed
  408da50

• fetches more author info, not sure how to format it yet

  pidgezero-one

  committed
  e7f714b

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for 8f01b5b
  pre-commit-ci[bot]

  committed
  8f01b5b

• brackets in wrong placE

  pidgezero-one

  committed
  77b23d8

Commits on Oct 12, 2024

• Merge branch 'master' into 9671/feat/add-wikisource-import-script

  pidgezero-one

  committed
  2aca912

Commits on Oct 13, 2024

• format that works with /import/api

  pidgezero-one

  committed
  7bfc39b

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for 461b02a
  pre-commit-ci[bot]

  committed
  461b02a

• wip

  pidgezero-one

  committed
  147fab3

• Merge branch '9671/feat/add-wikisource-import-script' of https://github.com/pidgezero-one/openlibrary into 9671/feat/add-wikisource-import-script

  pidgezero-one

  committed
  b73b8d1

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for 92065ed
  pre-commit-ci[bot]

  committed
  92065ed

• ?

  pidgezero-one

  committed
  0750cb2

• Merge branch '9671/feat/add-wikisource-import-script' of https://github.com/pidgezero-one/openlibrary into 9671/feat/add-wikisource-import-script

  pidgezero-one

  committed
  5842e13

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for 83a00f8
  pre-commit-ci[bot]

  committed
  83a00f8

• precommit errors

  pidgezero-one

  committed
  bc31426

• Merge branch '9671/feat/add-wikisource-import-script' of https://github.com/pidgezero-one/openlibrary into 9671/feat/add-wikisource-import-script

  pidgezero-one

  committed
  2e99560

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for cb14b14
  pre-commit-ci[bot]

  committed
  cb14b14

• support author id matching

  pidgezero-one

  committed
  8d4dac0

• Merge branch '9671/feat/add-wikisource-import-script' of https://github.com/pidgezero-one/openlibrary into 9671/feat/add-wikisource-import-script

  pidgezero-one

  committed
  7d85a0f

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for f1b0edd
  pre-commit-ci[bot]

  committed
  f1b0edd

• can't get it to work

  pidgezero-one

  committed
  5b57217

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for 0bd52e8
  pre-commit-ci[bot]

  committed
  0bd52e8

• unnecessary changes

  pidgezero-one

  committed
  05f3644

• idk

  pidgezero-one

  committed
  12b96f4

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for 6a8234c
  pre-commit-ci[bot]

  committed
  6a8234c

Commits on Oct 14, 2024

• it works

  pidgezero-one

  committed
  b9c36e4

• Merge branch '9671/feat/add-wikisource-import-script' of https://github.com/pidgezero-one/openlibrary into 9671/feat/add-wikisource-import-script

  pidgezero-one

  committed
  2fd4569

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for 5a8ea7a
  pre-commit-ci[bot]

  committed
  5a8ea7a

• wip: support more identifiers

  pidgezero-one

  committed
  268f055

• Merge branch '9671/feat/add-wikisource-import-script' of https://github.com/pidgezero-one/openlibrary into 9671/feat/add-wikisource-import-script

  pidgezero-one

  committed
  60e5d00

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for b6b68f3
  pre-commit-ci[bot]

  committed
  b6b68f3

• fix remote ids

  pidgezero-one

  committed
  e02ae78

• fix remote ids

  pidgezero-one

  committed
  d7dd818

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for ee00b9a
  pre-commit-ci[bot]

  committed
  ee00b9a

• comment

  pidgezero-one

  committed
  0ecba61

• Merge branch '9671/feat/add-wikisource-import-script' of https://github.com/pidgezero-one/openlibrary into 9671/feat/add-wikisource-import-script

  pidgezero-one

  committed
  f952c4f

• fix wd condition

  pidgezero-one

  committed
  0e2c510

• remote_ids will never be empty in script

  pidgezero-one

  committed
  1265681

• attempt unit tests

  pidgezero-one

  committed
  216cb50

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for eb8d3d4
  pre-commit-ci[bot]

  committed
  eb8d3d4

• irrelevant

  pidgezero-one

  committed
  d298d39

• update comment

  pidgezero-one

  committed
  8c17a14

• unnecessary change

  pidgezero-one

  committed
  8917ae8

Commits on Oct 15, 2024

• Update openlibrary/components/AuthorIdentifiers.vue

  Show description for a174609
  

  pidgezero-one

  and

  Freso

  authored
  a174609

• Update openlibrary/components/AuthorIdentifiers.vue

  Show description for ad263aa
  

  pidgezero-one

  and

  Freso

  authored
  ad263aa

• Update openlibrary/components/AuthorIdentifiers.vue

  Show description for ab624f6
  

  pidgezero-one

  and

  Freso

  authored
  ab624f6

Commits on Oct 16, 2024

• suggested rename

  pidgezero-one

  committed
  320be8b

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for f07e5fb
  pre-commit-ci[bot]

  committed
  f07e5fb

• why did changing remote_ids to identifiers break tests?

  pidgezero-one

  committed
  3bfe2d9

• Merge branch '9671/feat/add-wikisource-import-script' of https://github.com/pidgezero-one/openlibrary into 9671/feat/add-wikisource-import-script

  pidgezero-one

  committed
  a0aaf40

• fix import key

  pidgezero-one

  committed
  d97921d

• identifiers

  pidgezero-one

  committed
  565853b

• identifiers

  pidgezero-one

  committed
  d84c5fd

Commits on Nov 27, 2024

• move this code

  pidgezero-one

  committed
  f0a1103

• /

  pidgezero-one

  committed
  8d3fe1f

• unused imports

  pidgezero-one

  committed
  eea9276

Commits on Nov 28, 2024

• this got moved out by accident

  pidgezero-one

  committed
  aca95e0

• wtf?

  pidgezero-one

  committed
  d3b4767

Commits on Dec 3, 2024

• add another exclusion

  pidgezero-one

  committed
  2545c12

• Merge branch 'master' into 9671/feat/add-wikisource-import-script

  pidgezero-one

  committed
  eb3cb40

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for 88648e0
  pre-commit-ci[bot]

  committed
  88648e0

• ruff fix

  pidgezero-one

  committed
  e04b80c

• just get birth and death dates for authors to use in import matching

  pidgezero-one

  committed
  c55261b

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for abb913c
  pre-commit-ci[bot]

  committed
  abb913c