Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hoist Data Management code from XPCS and HEDM #938

Merged
merged 9 commits into from
Mar 18, 2024
Merged

Conversation

prjemian
Copy link
Contributor

@prjemian prjemian commented Mar 13, 2024

Note: Since we cannot test this code on GitHub, it is expected that the code coverage report from the GitHub workflow will decrease with these additions.

@prjemian prjemian added this to the 1.6.19 milestone Mar 13, 2024
@prjemian prjemian self-assigned this Mar 13, 2024
@prjemian prjemian marked this pull request as draft March 13, 2024 19:43
@prjemian
Copy link
Contributor Author

Workflow is failing at

>       from dm import ProcApiFactory
E       ModuleNotFoundError: No module named 'dm'

For Python 3.11, the workflow has installed

2024-03-14T21:51:40.3065013Z   aps-dm-api                     5.0.3         py37_1                   aps-anl-tag

which is for Python 3.7! Can't be found by Py3.11.

Because of this:

- aps-dm-api >=5 # linux-64 osx-64

@prjemian
Copy link
Contributor Author

Anaconda reports latest version is 8.0.2

@prjemian prjemian marked this pull request as ready for review March 14, 2024 23:16
@prjemian
Copy link
Contributor Author

@MDecarabas @rodolakis - This is ready for review


### upload

In addition to automatic file copying with a DAQ, DM provides for single file
Copy link

@sveseli sveseli Mar 15, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to clarify, you can "upload" the whole directories as well. The difference between the "file" and "directory" processing modes is that in the file mode files are processed (transfer, catalog, invoke workflow, etc) one by one, while in the directory mode there will be a single transfer of the whole directory, after which all files will be cataloged, etc. The advantage of the file mode is that you can keep track of progress and see/recover from errors easier, while the advantage of the directory mode is that it is more efficient, especially for large number of small files, where any new transfer connection is more expensive relative to the data transfer time.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

Copy link
Collaborator

@MDecarabas MDecarabas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@prjemian prjemian merged commit a165d24 into main Mar 18, 2024
8 of 13 checks passed
@prjemian prjemian deleted the 872-aps-dm-branch2 branch March 18, 2024 21:38
MDecarabas pushed a commit that referenced this pull request Apr 12, 2024
MDecarabas pushed a commit that referenced this pull request Apr 12, 2024
MDecarabas added a commit that referenced this pull request Apr 12, 2024
* Added edge_align plan for beam edge alignment

* Need to find a better way to check signal quality.

* Formatted using ruff

* small formatting changes

* removed unused stats import

* removed another unused import

* Added a check function that yields correct results

* intermediate commit saving progress

* finished try/except when bad signal is detected

* Fixed formatting

* Added scipy to environment.yaml & requirements.txt

* CI #909 switch to 'ruff'

* DOC #909 update release notes

* MNT #909 revise per 'ruff check'

* CI #909 update

* Bump davidslusser/actions_python_ruff from 1.0.0 to 1.0.1

Bumps [davidslusser/actions_python_ruff](https://github.com/davidslusser/actions_python_ruff) from 1.0.0 to 1.0.1.
- [Release notes](https://github.com/davidslusser/actions_python_ruff/releases)
- [Commits](davidslusser/actions_python_ruff@v1.0.0...v1.0.1)

---
updated-dependencies:
- dependency-name: davidslusser/actions_python_ruff
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* MNT #872 hoist from MPE devices

* TST #872 adjust for new hoist

* MNT #872

* MNT #872 refactor

* MNT #872 hoist utils

* DOC #912

* ENV #938 bump the minimum requirement

* DOC #932 update relase notes

* DOC #938 contribution from @sveseli

* MNT #914 setup.py --> pyproject.toml

* DOC #921 copyright year

* MNT #914

* Added edge_align function for beam edge alignment

* added scipy to pyproject.toml

* CI #909 switch to 'ruff'

* DOC badge

* DOC badge

* DOC #940 example scans with sscan record

* DOC #940 it's only a word ...

* DOC #940

* DOC #940 it's a demo, actually

* DOC #940

* DOC #940 convert the blocking call to st.wait()

* DOC #940 per review

* DOC #940 per review

* DOC #940 proofreading

* DOC #940 per review, remove the section with the polling loop

* DOC #940 new title

* DOC #940 more proofing

* DOC #940 clarify the default sscan record array size of 1,000

* DOC #940 refactor to polling loop. Show the recommended style in a details section.

* DOC #940

* Added put_complete=True to sensitivity value and unit for the SRS-570 Pre-amp.

* Added a test for the sensitivity_unit put_complete, and removed the put_complete from sensitivity_value.

* CI #955 move requirements file

* MNT #955 sphinx config

* DOC #955 re-arrange

* DOC #955 apply to lower-level pages

* DOC #955 add docs upload

* CI #955

* CI #955

* CI #955 wip

* CI #955

* CI #955

* DOC #917 release note

* DOC #957 merge overview into home page

* DOC #955

* DOC #955

* DOC #955

* DOC #955

* Added edge_align function for beam edge alignment

* Added edge_align function for beam edge alignment

* Added edge_align function for beam edge alignment

* DOC #909 update release notes

* DOC #932 update relase notes

* Added edge_align function for beam edge alignment

* CI #909 switch to 'ruff'

* DOC #955

* Fixed merge issue

* fixed unused import

* change for tests to pass

* Formatting changes

* Fixed repeated section

* Small formatting changes

* Small syntax change

* Added toolz & scipy to conf.py

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Codrea <ecodrea@nefarian.xray.aps.anl.gov>
Co-authored-by: Pete R Jemian <prjemian@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Pete R Jemian <prjemian@users.noreply.github.com>
Co-authored-by: Mark Wolfman <canismarko@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add support DM DAQ data management setup
3 participants