sqlite3.IntegrityError: NOT NULL constraint failed: _source_info_.format_name

[yajo]

Describe the bug

Cannot perform a simple conversion from CSV with simplesqlite==1.5.0.

Expected behavior

Conversion works, just like with simplesqlite==1.4.0

To Reproduce

Using python 3.10:

pip install sqlitebiter
wget https://github.com/odoo/odoo/raw/35d5342/addons/crm/security/ir.model.access.csv
sqlitebiter --debug -v file ir.model.access.csv

Environments

Please complete the following information:

- uname: Linux
- Python version: CPython 3.9.18
- sqlitebiter version: 0.36.2
- platform version: Debian GNU/Linux 12 (bookworm)
- SimpleSQLite version: 1.5.0
- pytablereader version: 0.31.4

• Installation Method: pip

Additional context

Error logs with simplesqlite==1.5.0

DEBUG    | simplesqlite.core:connect:280 - connect to a SQLite database: path='out.sqlite', mode=w
DEBUG    | simplesqlite.core:create_table:1324 - CREATE TABLE IF NOT EXISTS '_source_info_' ("source_id" INTEGER NOT NULL, "dir_name" TEXT, "base_name" TEXT NOT NULL, "format_name" TEXT NOT NULL, "dst_table" TEXT NOT NULL, size INTEGER, mtime INTEGER)
DEBUG    | sqlitebiter.converter._file:convert:83 - converting 'ir.model.access.csv'
DEBUG    | pytablereader.factory._file:__init__:41 - detect encoding: file=ir.model.access.csv, encoding=ascii
DEBUG    | pytablereader.factory._file:create_from_path:75 - TableFileLoaderFactory.create_from_path: extension=csv, loader=csv
DEBUG    | pytablereader._logger._logger:logging_load:61 - loading file: format=csv, path=ir.model.access.csv, encoding=ascii
DEBUG    | sqlitebiter.converter._file:__convert:138 - loaded tabledata: table_name=ir.model.access, headers=[id, name, model_id:id, group_id:id, perm_read, perm_write, perm_create, perm_unlink], cols=8, rows=32
DEBUG    | tabledata.normalizer:normalize:61 - normalize: SQLiteTableDataSanitizer
DEBUG    | simplesqlite.core:connect:280 - connect to a SQLite database: path=':memory:', mode=w
DEBUG    | simplesqlite.core:__create_table_from_tabledata:1773 - __create_table_from_tabledata:
    tbldata=table_name=ir_model_access, headers=[id, name, model_id:id, group_id:id, perm_read, perm_write, perm_create, perm_unlink], cols=8, rows=32
DEBUG    | tabledata.normalizer:normalize:61 - normalize: SQLiteTableDataSanitizer
DEBUG    | dataproperty._extractor:to_dp_matrix:489 - max_workers=1, preprocessor=strip_str=None, replace_tabs_with_spaces=True, tab_length=2, line_break_handling=LineBreakHandling.NOP, line_break_repl= , escape_html_tag=False, escape_formula_injection=False
DEBUG    | dataproperty._extractor:to_column_dp_list:422 - converting to column dataproperty:
DEBUG    | dataproperty._extractor:to_column_dp_list:450 -   params:
DEBUG    | dataproperty._extractor:to_column_dp_list:450 -     headers=8
DEBUG    | dataproperty._extractor:to_column_dp_list:450 -     prev_col_count=None
DEBUG    | dataproperty._extractor:to_column_dp_list:450 -     matrix_formatting=MatrixFormatting.HEADER_ALIGNED
DEBUG    | dataproperty._extractor:to_column_dp_list:450 -     column_type_hints=()
DEBUG    | dataproperty._extractor:to_column_dp_list:452 -   results:
DEBUG    | dataproperty._extractor:to_column_dp_list:483 -     column=0, type=STRING, align=left, ascii_width=47
DEBUG    | dataproperty._extractor:to_column_dp_list:483 -     column=1, type=STRING, align=left, ascii_width=46
DEBUG    | dataproperty._extractor:to_column_dp_list:483 -     column=2, type=STRING, align=left, ascii_width=39
DEBUG    | dataproperty._extractor:to_column_dp_list:483 -     column=3, type=STRING, align=left, ascii_width=30
DEBUG    | dataproperty._extractor:to_column_dp_list:483 -     column=4, type=INTEGER, align=right, ascii_width=9, bit_len=1, int_digits=1, decimal_places=0
DEBUG    | dataproperty._extractor:to_column_dp_list:483 -     column=5, type=INTEGER, align=right, ascii_width=10, bit_len=1, int_digits=1, decimal_places=0
DEBUG    | dataproperty._extractor:to_column_dp_list:483 -     column=6, type=INTEGER, align=right, ascii_width=11, bit_len=1, int_digits=1, decimal_places=0
DEBUG    | dataproperty._extractor:to_column_dp_list:483 -     column=7, type=INTEGER, align=right, ascii_width=11, bit_len=1, int_digits=1, decimal_places=0
DEBUG    | simplesqlite.core:create_table:1324 - CREATE TABLE IF NOT EXISTS 'ir_model_access' (id TEXT, name TEXT, "model_id:id" TEXT, "group_id:id" TEXT, "perm_read" INTEGER, "perm_write" INTEGER, "perm_create" INTEGER, "perm_unlink" INTEGER)
DEBUG    | simplesqlite.core:insert_many:643 - insert 32 records into ir_model_access
DEBUG    | simplesqlite.core:commit:1633 - commit: path=':memory:'
DEBUG    | simplesqlite._func:append_table:62 - append table: src=:memory:.ir_model_access, dst=/out.sqlite.ir_model_access
DEBUG    | simplesqlite.core:__create_table_from_tabledata:1773 - __create_table_from_tabledata:
    tbldata=table_name=ir_model_access, headers=[id, name, model_id:id, group_id:id, perm_read, perm_write, perm_create, perm_unlink], cols=8, rows=32
DEBUG    | tabledata.normalizer:normalize:61 - normalize: SQLiteTableDataSanitizer
DEBUG    | dataproperty._extractor:to_dp_matrix:489 - max_workers=1, preprocessor=strip_str=None, replace_tabs_with_spaces=True, tab_length=2, line_break_handling=LineBreakHandling.NOP, line_break_repl= , escape_html_tag=False, escape_formula_injection=False
DEBUG    | dataproperty._extractor:to_column_dp_list:422 - converting to column dataproperty:
DEBUG    | dataproperty._extractor:to_column_dp_list:450 -   params:
DEBUG    | dataproperty._extractor:to_column_dp_list:450 -     headers=8
DEBUG    | dataproperty._extractor:to_column_dp_list:450 -     prev_col_count=None
DEBUG    | dataproperty._extractor:to_column_dp_list:450 -     matrix_formatting=MatrixFormatting.TRIM
DEBUG    | dataproperty._extractor:to_column_dp_list:450 -     column_type_hints=(String, String, String, String, Integer, Integer, Integer, Integer)
DEBUG    | dataproperty._extractor:to_column_dp_list:452 -   results:
DEBUG    | dataproperty._extractor:to_column_dp_list:483 -     column=0, type=STRING, align=left, ascii_width=47
DEBUG    | dataproperty._extractor:to_column_dp_list:483 -     column=1, type=STRING, align=left, ascii_width=46
DEBUG    | dataproperty._extractor:to_column_dp_list:483 -     column=2, type=STRING, align=left, ascii_width=39
DEBUG    | dataproperty._extractor:to_column_dp_list:483 -     column=3, type=STRING, align=left, ascii_width=30
DEBUG    | dataproperty._extractor:to_column_dp_list:483 -     column=4, type=INTEGER, align=right, ascii_width=9, bit_len=1, int_digits=1, decimal_places=0
DEBUG    | dataproperty._extractor:to_column_dp_list:483 -     column=5, type=INTEGER, align=right, ascii_width=10, bit_len=1, int_digits=1, decimal_places=0
DEBUG    | dataproperty._extractor:to_column_dp_list:483 -     column=6, type=INTEGER, align=right, ascii_width=11, bit_len=1, int_digits=1, decimal_places=0
DEBUG    | dataproperty._extractor:to_column_dp_list:483 -     column=7, type=INTEGER, align=right, ascii_width=11, bit_len=1, int_digits=1, decimal_places=0
DEBUG    | simplesqlite.core:create_table:1324 - CREATE TABLE IF NOT EXISTS 'ir_model_access' (id TEXT, name TEXT, "model_id:id" TEXT, "group_id:id" TEXT, "perm_read" INTEGER, "perm_write" INTEGER, "perm_create" INTEGER, "perm_unlink" INTEGER)
DEBUG    | simplesqlite.core:insert_many:643 - insert 32 records into ir_model_access
DEBUG    | simplesqlite.core:commit:1633 - commit: path='/out.sqlite'
INFO     | sqlitebiter._common:logging_success:40 - convert '/ir.model.access.csv' to 'ir_model_access (id, name, model_id:id, group_id:id, perm_read, perm_write, perm_create, perm_unlink)' table
DEBUG    | simplesqlite.core:close:1657 - close connection to a SQLite database: path=':memory:'
DEBUG    | simplesqlite.core:commit:1633 - commit: path=':memory:'
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/simplesqlite/core.py", line 349, in execute_query
    result = self.connection.execute(str(query))
sqlite3.IntegrityError: NOT NULL constraint failed: _source_info_.format_name

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/sqlitebiter", line 8, in <module>
    sys.exit(cmd())
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.9/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/sqlitebiter/__main__.py", line 368, in file
    converter.convert(file_path)
  File "/usr/local/lib/python3.9/site-packages/sqlitebiter/converter/_file.py", line 106, in convert
    self.__convert(fpath, source_info_record_base)
  File "/usr/local/lib/python3.9/site-packages/sqlitebiter/converter/_file.py", line 153, in __convert
    SourceInfo.insert(record)
  File "/usr/local/lib/python3.9/site-packages/simplesqlite/model.py", line 181, in insert
    cls.__connection.insert(cls.get_table_name(), record, list(record.keys()))
  File "/usr/local/lib/python3.9/site-packages/simplesqlite/core.py", line 603, in insert
    if self.execute_query(query, logging.getLogger().findCaller()) is None:
  File "/usr/local/lib/python3.9/site-packages/simplesqlite/core.py", line 355, in execute_query
    raise OperationalError(
simplesqlite.error.OperationalError

[thombashi]

@yajo
Thank you for your report.
I fixed the problem at SimpleSQLite 1.5.1.
Please upgrade SimpleSQLite package for a workaround.

[thombashi]

Fixed at sqlitebiter 0.36.3