Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use plotly.js base64 API to store and pass typed arrays declared by numpy, pandas, etc. #4470

Merged
merged 128 commits into from
Oct 21, 2024
Merged
Show file tree
Hide file tree
Changes from 22 commits
Commits
Show all changes
128 commits
Select commit Hold shift + click to select a range
1f3e605
revisit validator
archmoj Oct 25, 2023
a627c9e
pass numpy conversions as typed array spec
archmoj Oct 25, 2023
c363f3e
adjust validators
archmoj Nov 2, 2023
61aa4ea
add b64 file for tests
archmoj Jan 9, 2024
730c0dd
adjust test_dataarray_validator.py
archmoj Jan 9, 2024
7c31dc1
adjust test_pandas_series_input.py
archmoj Jan 9, 2024
d4c3162
adjust test_xarray_input.py
archmoj Jan 9, 2024
d1b4706
adjust test_figure_factory.py
archmoj Jan 9, 2024
2b33199
adjust test_imshow.py
archmoj Jan 9, 2024
f2c8d66
adjust test_px.py
archmoj Jan 9, 2024
b4d23b5
adjust test_px_functions.py
archmoj Jan 9, 2024
03aa9e0
adjust test_px_input.py
archmoj Jan 9, 2024
4514e92
adjust test_px_wide.py
archmoj Jan 9, 2024
048455e
adjust test_trendline.py
archmoj Jan 9, 2024
520e9bf
adjust test_figure_factory.py
archmoj Jan 9, 2024
c48f3ce
adjust test_utils.py
archmoj Jan 9, 2024
dcb6b91
skip test_fast_track_finite_arrays
archmoj Nov 15, 2023
9b1cbfd
skip test_violin_fig on CI
archmoj Nov 15, 2023
bd7e9dc
skip few mocks in compare pandas v1 vs v2
archmoj Dec 21, 2023
124f12d
remove clean_float which is not necessary
archmoj Dec 21, 2023
82c8199
add examples using base64
archmoj Jan 9, 2024
f200767
Update packages/python/plotly/_plotly_utils/basevalidators.py
archmoj Jan 11, 2024
5fb9ee3
Merge remote-tracking branch 'origin/master' into pass-b64_dev
archmoj Feb 29, 2024
b31dc27
Merge branch 'master' into pass-b64
archmoj Mar 15, 2024
0323215
Merge branch 'master' into pass-b64
archmoj Mar 22, 2024
99ebbbc
Merge branch 'master' into pass-b64
archmoj Jul 8, 2024
7c7ef30
Merge branch 'master' into pass-b64
archmoj Jul 24, 2024
f5df6db
also check for dtype in is_typed_array_spec function
archmoj Jul 24, 2024
f332922
remove print
archmoj Jul 24, 2024
5947221
Add performance test for b64
marthacryan Jul 26, 2024
62d9aa7
Add tests for size
marthacryan Jul 26, 2024
e5c24fe
Add test for array_ok and b64 together in IntegerValidator:
marthacryan Jul 26, 2024
8430c52
Black
marthacryan Jul 26, 2024
baeedc9
Change the time difference to be larger between b64 and raw array
marthacryan Jul 30, 2024
4f63296
Add random seed
marthacryan Jul 30, 2024
a566543
Change numpy array to python list before comparison
marthacryan Jul 30, 2024
3d63fa2
Remove unnecessary casting to np array
marthacryan Jul 30, 2024
7fbb701
specify width and height and fix logic of time comparison
marthacryan Jul 30, 2024
6e53e51
Add hard-coded margins
marthacryan Jul 30, 2024
dd1aba8
Add uint8 and float32 tests
marthacryan Jul 30, 2024
b6f9d14
Update performance margin to be a little smaller
marthacryan Jul 30, 2024
555d960
Black
marthacryan Jul 30, 2024
b301d99
Fix size performance tests and add graph object tests
marthacryan Jul 31, 2024
4323c28
Remove print statements
marthacryan Jul 31, 2024
c1e6728
Add numpy as a requirement for core tests
marthacryan Jul 31, 2024
7822635
Black
marthacryan Jul 31, 2024
58d4844
update requirements for python 3.12
marthacryan Jul 31, 2024
43de1cb
Update packages/python/plotly/plotly/tests/test_core/test_graph_objs/…
marthacryan Aug 1, 2024
1105328
Update names
marthacryan Aug 1, 2024
65f0dad
Update variables used in tests
marthacryan Aug 1, 2024
da100bc
Lower threshold for passing
marthacryan Aug 1, 2024
554f5cb
Use different version of setuptools
marthacryan Aug 1, 2024
024f3c1
Black
marthacryan Aug 1, 2024
8a051a3
Update tests to remove conversion to base64 before passing numpy arrays
marthacryan Aug 1, 2024
ece3e3d
remove setuptools from requirements
marthacryan Aug 1, 2024
0675f5b
Add setup tools install before requirements
marthacryan Aug 2, 2024
2fc29f8
Remove pin on numpy version
marthacryan Aug 2, 2024
d8924c5
Try removing the setuptools from config
marthacryan Aug 2, 2024
e1f91cd
Update performance thresholds
marthacryan Aug 2, 2024
98f2541
Parametrize functions and lower performance thresholds
marthacryan Aug 2, 2024
3702686
Code format
marthacryan Aug 2, 2024
4932cdb
Remove px tests (duplicates)
marthacryan Aug 2, 2024
ddbc3f1
Remove px tests (duplicates)
marthacryan Aug 2, 2024
0b83ebd
Add back in max_value and parameterize the count
marthacryan Aug 2, 2024
dabbcb8
Remove numpy requirement after moving performance tests back to optional
marthacryan Aug 2, 2024
6c01e6a
Use scattergl instead of scatter
marthacryan Aug 2, 2024
a056d7e
Add verbose flag to debug ci
marthacryan Aug 2, 2024
6d82e48
Only run performance tests for debugging
marthacryan Aug 2, 2024
8823a8c
Try commenting out all but one test
marthacryan Aug 2, 2024
25f6ccf
Print pio.renderers to debug
marthacryan Aug 2, 2024
b931fd5
Debug
marthacryan Aug 2, 2024
addfd47
Try rendering as png
marthacryan Aug 2, 2024
ef51cce
Add back in other tests and update renderer default
marthacryan Aug 2, 2024
3142c62
Black
marthacryan Aug 2, 2024
72e9ad0
Update failing performance threshold
marthacryan Aug 2, 2024
2d4ad05
Update failing performance threshold
marthacryan Aug 2, 2024
9d3b50d
Update thresholds
marthacryan Aug 2, 2024
63335d2
Update thresholds
marthacryan Aug 2, 2024
644992f
Merge pull request #4695 from marthacryan/add-tests-b64
marthacryan Aug 2, 2024
0f68aff
Add validator test to basetraces validator
marthacryan Aug 5, 2024
2a81d2a
Add more validator tests
marthacryan Aug 5, 2024
1c0b48e
black
marthacryan Aug 5, 2024
e500e38
Add more base64 array_ok tests
marthacryan Aug 7, 2024
c2453d3
Add other int types to integer validator tests
marthacryan Aug 7, 2024
c8d35b4
Add more integer types to the validation
marthacryan Aug 7, 2024
8861451
black
marthacryan Aug 7, 2024
8afa4d8
remove unused imports
marthacryan Aug 12, 2024
69729db
Remove unnecessary usage of numpy dtypes to prevent throwing error
marthacryan Aug 12, 2024
c52e9f4
Merge pull request #4707 from plotly/validator_tests
marthacryan Aug 12, 2024
c6390f4
Add test for geojson not converting to b64
marthacryan Aug 23, 2024
a3940ee
Simplify tests
marthacryan Aug 26, 2024
259d509
Add tests for layers and range keys
marthacryan Aug 27, 2024
d201b58
Code format
marthacryan Aug 27, 2024
f018291
Update packages/python/plotly/plotly/tests/test_optional/test_graph_o…
marthacryan Aug 28, 2024
8d7edf6
Merge branch 'master' of github.com:plotly/plotly.py into pass-b64
marthacryan Aug 28, 2024
d59ceb0
Merge branch 'pass-b64' of github.com:plotly/plotly.py into add-skipp…
marthacryan Aug 28, 2024
40166bb
Potential fix to conversion bug
marthacryan Aug 28, 2024
17b531c
Refactor logic to be clearer
marthacryan Aug 30, 2024
2fa09e5
remove todo
marthacryan Aug 30, 2024
4452868
Black
marthacryan Aug 30, 2024
02fb3a4
Merge pull request #4727 from plotly/add-skipped-key-tests
marthacryan Aug 30, 2024
a8995b7
Merge branch 'master' of github.com:plotly/plotly.py into pass-b64
marthacryan Aug 30, 2024
12ff7f3
remove failing orca tests to prevent confusion while waiting on updat…
marthacryan Aug 30, 2024
ca4340b
Remove another part of config that we're removing to prevent CI failure
marthacryan Aug 30, 2024
dd9379a
Add base64 to the changelog
marthacryan Aug 30, 2024
9362db3
Remove examples that over-complicate the usage of base64 spec
marthacryan Sep 17, 2024
6364d4e
Remove base64 documentation
marthacryan Sep 17, 2024
61e9178
Convert base64 in validate_coerce_fig_to_dict instead of validate_coerce
marthacryan Oct 7, 2024
066564e
Update logic to be recursive
marthacryan Oct 10, 2024
fb036c7
Move conversion to to_dict function
marthacryan Oct 10, 2024
4b289e9
Fix import path
marthacryan Oct 10, 2024
aabfa6e
Consolidate import statemenets
marthacryan Oct 10, 2024
de2bcb5
Merge pull request #4784 from plotly/b64-before-render
archmoj Oct 15, 2024
0d0dad2
Revert changes to validator tests
marthacryan Oct 15, 2024
9c5d112
Revert changes to tests that check data field
marthacryan Oct 15, 2024
5e50d8c
Merge with master
marthacryan Oct 15, 2024
8fff9c5
Remove performance tests
marthacryan Oct 15, 2024
bd0a2d3
Update base64 tests to reflect new approach
marthacryan Oct 15, 2024
b3ed838
Fix failing tests
marthacryan Oct 16, 2024
7bd7993
Merge with master
marthacryan Oct 16, 2024
cd8e0be
Revert changes to to_json_plotly tests
marthacryan Oct 16, 2024
f60b122
revert changes to percy compare pandas
marthacryan Oct 17, 2024
9d6b0c7
Remove unused util
marthacryan Oct 17, 2024
960adb9
Revert changes to validators
marthacryan Oct 17, 2024
12ab42a
Revert changes to circleci config
marthacryan Oct 17, 2024
59fe206
Address review
marthacryan Oct 17, 2024
60c73a8
fix doctsring
marthacryan Oct 17, 2024
f481af7
update skipped keys to include layers
marthacryan Oct 17, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
198 changes: 198 additions & 0 deletions doc/python/b64.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,198 @@
---
marthacryan marked this conversation as resolved.
Show resolved Hide resolved
jupyter:
jupytext:
notebook_metadata_filter: all
text_representation:
extension: .md
format_name: markdown
format_version: '1.3'
jupytext_version: 1.15.2
kernelspec:
display_name: Python 3 (ipykernel)
language: python
name: python3
language_info:
codemirror_mode:
name: ipython
version: 3
file_extension: .py
mimetype: text/x-python
name: python
nbconvert_exporter: python
pygments_lexer: ipython3
version: 3.9.0
plotly:
description: How to format axes of 3d plots in Python with Plotly.
display_as: b64
LiamConnors marked this conversation as resolved.
Show resolved Hide resolved
language: python
layout: base
name: b64
order: 1
page_type: example_index
permalink: python/b64/
thumbnail: thumbnail/b64.png
---

LiamConnors marked this conversation as resolved.
Show resolved Hide resolved
### Simple example showing how arrays of numbers could be passed as base64 typed array objects to plotly.js

```python
import plotly.graph_objects as go

archmoj marked this conversation as resolved.
Show resolved Hide resolved
# x = [-200000 -100000 0 100000 200000]
x = {'dtype': 'int32', 'bdata': 'wPL8/2B5/v8AAAAAoIYBAEANAwA='}

# y = [0 1 2 3 4 5 6 7 8 9]
y = {'dtype': 'uint8', 'bdata': 'AAECAwQFBgcICQ=='}

# z = [
# [ 61 -295 -765 863 932]
# [-897 96 724 791 -993]
# [ -95 -796 -285 381 669]
# [ 985 -153 425 -40 136]
# [-856 955 -871 414 996]
# [ 966 607 -154 -251 -882]
# [-492 -116 414 426 305]
# [ 919 202 -505 300 -833]
# [ 278 -152 -643 -950 -86]
# [ 898 -532 608 -93 110]]
z = {
'dtype': 'int16',
'bdata': 'PQDZ/gP9XwOkA3/8YADUAhcDH/yh/+T84/59AZ0C2QNn/6kB2P+IAKj8uwOZ/J4B5APGA18CZv8F/478FP6M/54BqgExAZcDygAH/iwBv/wWAWj/ff1K/Kr/ggPs/WACo/9uAA==', 'shape': '10, 5'
}

fig = go.Figure(data=[go.Surface(
x=x,
y=y,
z=z
)])

fig.show()
```

### Example where base64 is applied to pass values as typed array objects to plotly.js

```python
import plotly.graph_objects as go
archmoj marked this conversation as resolved.
Show resolved Hide resolved
import numpy as np
from base64 import b64encode

def b64(arr) :
return {
'dtype': str(arr.dtype),
'bdata': b64encode(arr).decode('ascii')
}

np.random.seed(1)

N = 10000

x = np.random.randn(N)
y = np.random.randn(N).astype('float32')
z = np.random.randint(size=N, low=0, high=256, dtype='uint8')
c = np.random.randint(size=N, low=-10, high=10, dtype='int8')

fig = go.Figure(data=[go.Scatter3d(
x=b64(x),
y=b64(y),
z=b64(z),
marker=dict(color= b64(c)),
mode='markers',
opacity=0.2
)])

fig.show()
```

### Similar example where base64 is automatically applied to pass numpy arrays to plotly.js

```python
import plotly.graph_objects as go
import numpy as np

np.random.seed(1)

N = 10000

x = np.random.randn(N)
y = np.random.randn(N).astype('float32')
z = np.random.randint(size=N, low=0, high=256, dtype='uint8')
c = np.random.randint(size=N, low=-10, high=10, dtype='int8')

fig = go.Figure(data=[go.Scatter3d(
x=x,
y=y,
z=z,
marker=dict(color=c),
mode='markers',
opacity=0.2
)])

fig.show()
```


### Example where base64 is applied to pass 2 dimensional values as typed array objects to plotly.js using shape in the spec

```python
import plotly.graph_objects as go
import numpy as np
from base64 import b64encode

def b64(arr) :
return {
'dtype': str(arr.dtype),
'bdata': b64encode(arr).decode('ascii'),
'shape': None if arr.ndim == 1 else str(arr.shape)[1:-1]
}

np.random.seed(1)

M = 100
N = 200

x = np.arange(0, M, 1, 'int32')
y = np.arange(0, N, 1, 'uint8')
z = np.random.random([N, M])

fig = go.Figure(data=[go.Surface(
x=b64(x),
y=b64(y),
z=b64(z)
)])

fig.show()
```

### Similar example where base64 is automatically applied to pass multi-dimensional numpy arrays to plotly.js

```python
import plotly.graph_objects as go
import numpy as np
from base64 import b64encode

np.random.seed(1)

M = 100
N = 200

x = np.arange(0, M, 1, 'int32')
y = np.arange(0, N, 1, 'uint8')
z = np.random.random([N, M])

fig = go.Figure(data=[go.Surface(
x=x,
y=y,
z=z
)])

fig.show()
```


```python

```

```python

```
Loading