Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Format for running tests #8

Open
JimLewis opened this issue Aug 10, 2024 · 1 comment
Open

Format for running tests #8

JimLewis opened this issue Aug 10, 2024 · 1 comment

Comments

@JimLewis
Copy link

I noted in your pull request for OSVVM that you run tests by saying:

      - name: Test Rand
        run: orbit test --tb Demo_Rand --target gsim -- --std=08

Let me point out a couple of things:

  1. this is quite verbose for running just one thing.
  2. you have mixed your tool to run with a test case. You need to separate the simulator from the test case as this is awkward if you need to test against multiple simulators.
  3. running a set of tests often requires additional files that must be compiled before this.
  4. VHDL supports libraries. To keep different test harnesses separate, I recommend using libraries.
  5. OSVVM requires conditionals based on tool, tool version, variable settings.
  6. What do you do for test case artifact management - creating a simulator transcript, managing test output files? In particular some methodologies such as OSVVM produce yaml output files that get translated to HTML by the scripting.
  7. how do you manage two test cases producing test output files that have the same name?
  8. How do you support conditionals? OSVVM requires conditionals to support different tools, tool versions, and settings.
  9. How do you handle grouping tests into test suites?
  10. is building the design and/or running the test cases in any way hierarchical?

You can certainly solve some of this in YAML or other format, however, tests create output. Managing and post processing these may be methodology specific and likely require the methodologies runner.

Even building RTL designs is going to eventually require some sort of conditional support for variations of clock managers for the different FPGAs.

@JimLewis
Copy link
Author

Going further, for projects that use a variety of tools, the value that you can bring is:

  1. Identify a repository version
  2. Identify how to get it
  3. Identify how to build it.

This would help projects that bring different methodologies together that do not have the same build methodology.

For simple projects without their own build manager (ones just calling a single tool) it may be worthy of having a simple build process

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant