Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add basic vector type #2866

Merged
merged 1 commit into from
Jul 24, 2024
Merged

Add basic vector type #2866

merged 1 commit into from
Jul 24, 2024

Conversation

dbussink
Copy link
Contributor

This adds the new MySQL 9.0 vector type. It can still be handled like a binary blob for now I think. Maybe it's worth in the future to directly decode / parse it into an array of floats here?

This adds the new MySQL 9.0 vector type. It can still be handled like a
binary blob for now I think. Maybe it's worth in the future to directly
decode / parse it into an array of floats here?

Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com>
@dbussink
Copy link
Contributor Author

See https://dev.mysql.com/doc/dev/mysql-server/latest/field__types_8h.html also for the constant value here.

Copy link

codecov bot commented Jul 18, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 88.18%. Comparing base (013922f) to head (9bd49bd).

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #2866   +/-   ##
=======================================
  Coverage   88.18%   88.18%           
=======================================
  Files          71       71           
  Lines       12874    12875    +1     
  Branches     1350     1351    +1     
=======================================
+ Hits        11353    11354    +1     
  Misses       1521     1521           
Flag Coverage Δ
compression-0 88.18% <100.00%> (+<0.01%) ⬆️
compression-1 88.18% <100.00%> (+<0.01%) ⬆️
tls-0 87.60% <100.00%> (+<0.01%) ⬆️
tls-1 87.94% <100.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@sidorares
Copy link
Owner

hey @dbussink do you have any pointers to documentation on how the value is transmitted over the wire? Or maybe you can help with some examples I can run and try to reverse engineer serialisation format.

Happy to merge without detailed deserealizing, we can tackle that later

@sidorares sidorares merged commit 3659488 into sidorares:master Jul 24, 2024
68 checks passed
@wellwelwel
Copy link
Sponsor Collaborator

@sidorares, can we include a symbolic commit to release it? Also, to include it in change logs.

Or if you prefer, we can wait until a next release to bring this update along, but it won't be included in the logs.

@dbussink
Copy link
Contributor Author

hey @dbussink do you have any pointers to documentation on how the value is transmitted over the wire? Or maybe you can help with some examples I can run and try to reverse engineer serialisation format.

See https://dev.mysql.com/doc/refman/9.0/en/vector.html and if you look in the MySQL CLI, it's a binary blob of data. It's always a multiple of 4 bytes, since it's up to N 32 bit floating point values all concatenated together in that single blob.

@dbussink dbussink deleted the add-vector-type branch July 24, 2024 11:24
@sidorares
Copy link
Owner

@wellwelwel sorry I should've updated commit to have feature: prefix. I'll try to add deserealisation soon and reference this PR

@sidorares
Copy link
Owner

hey @dbussink do you have any pointers to documentation on how the value is transmitted over the wire? Or maybe you can help with some examples I can run and try to reverse engineer serialisation format.

See https://dev.mysql.com/doc/refman/9.0/en/vector.html and if you look in the MySQL CLI, it's a binary blob of data. It's always a multiple of 4 bytes, since it's up to N 32 bit floating point values all concatenated together in that single blob.

So it might be as simple as doing packet.readFloat() until the end and pushing numbers to the array

@sidorares
Copy link
Owner

yes, readFloat should be enough

connection.execute(`SELECT TO_VECTOR("[1.05, -17.8, 32, 123.456]")`, (err, res) => {
  if (err) throw err;
  const buf = Object.values(res[0])[0];

  let offset = 0;
  while (offset < buf.length) {
    const v = buf.readFloatLE(offset);
    console.log(v);
    offset += 4;
  }

  connection.end();
});

prints

1.0499999523162842
-17.799999237060547
32
123.45600128173828

@sidorares
Copy link
Owner

@dbussink this + #2894 published as v3.11.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants