This repository has been archived by the owner on Nov 20, 2023. It is now read-only.
Random ArgumentOutOfRangeException #1280
Assignees
Labels
Comments
|
I just want to add some extra information. This is one of two projects where I am using Tye and Dapr together. The project above is a mono-repo, where the tye.yml file is in the ./src/ folder, in the root of all of the project folders. For example: The other project works every time with no issues. That project is a multi-repo setup, where the tye.yml file is in its own repo, and locally uses As an important side note, the first project where I have the problem used to use the |
|
@stephanjohnson Do any of the service/dapr logs show errors causing it to cycle (i.e. be continually restarted), or perhaps have dependencies on other services being up prior to that service being brought up? I suspect the error shown is more a symptom of an underlying issue; I can see one path that could result in that error being logged, but it's an error path which means the process expected to run isn't running successfully. (Fixing this bug may allow the real error to get logged.) For reference, it looks like there's an issue in |
|
@stephanjohnson The latest CI builds should fix the error handling; give it a try and see if you can get additional error messages. |
|
Hi @philliphoff,
Sometimes I have frequent restarts of the dapr sidecar processes. Below is the log from the Tye Dashboard. The logs below are from the latest CI build. I'll post a second post with the full verbose logs for this version below |
|
Hey @philliphoff, here is the debug output from the latest CI build. |
|
The latest CI build does not start my own code when I look at the Tye dashboard, it only starts the dapr sidecars, and they go into an infinite restart loop every time. Tye version The dapr sidecar restarts are visible from the Tye dashboard, and the command line. However it is not until you mentioned this
that I connected the ArgumentOutOfRange exception with the cycling dapr sidecars. What I mean is that sometimes I can see continuous restarts and Tye will log the results to the console and the dashboard. However when the ArgumentOutOfRange exception is thrown it happens before there is enough time see the output on the console or the Tye dashboard. Obviously it does log to the console since you spotted this behavior from the logs, but it does not do so in a way that catches my eye to differentiate from all the other information being output. |
|
@stephanjohnson Do you see these types of errors in other (i.e. non-Tye) contexts: When you initialize Dapr, are you initializing it with or without the default Redis and Zipkin stores (i.e. via |
|
I used I have noticed those errors, and it seems that tye fails to cleanup properly when it does throw the I have a cleanup process that I run when I see those errors appear frequently: This wipes out everything in docker so that tye has a fresh slate to start from. I also use tye to initialize redis and zipkin: Another thing that is interesting, since I have another project (the multi-repo) where I use tye with no issues at all, in this problematic project... is that I have to set my This project (the problematic one) has fewer files than the multi-repo, fewer projects, and also uses tye to initialize the redis and zipkin instances. I was working on the multi-repo project last week, which is why I am only responding this week to this issue. I did not have to restart tye once. It worked as expected every time. I even tried to restructure this project to have a similar directory structure to the multi-repo project to see if that wasn't part of the problem. It was not. |
|
@stephanjohnson Very curious. So you're running Tye completely within WSL? In both the problematic and non-problematic cases?
So you're saying that, even with at multi-repo arrangement, the problematic project is still problematic? |
|
Yes. 100% WSL. Both the mono-repo (problematic) project, and multi-repo project are on the same machine. I have just uninstalled Dapr, cleaned up Docker and reinitialized Dapr without the Do you maybe want the log from the working project? Would that be any help at all? Just as a side note: I like to keep the zipkin and redis managed by tye because then I can purge my docker between working on different projects so that each project gets a clean slate when I'm working on it. So I know that there is no possibility that I have some stale docker images from the working project interfering with the problematic project. |
Possibly; couldn't hurt, anyway. I'll need to do more experimentation within WSL, maybe. (Most of my testing is done on Mac and Windows.) It may just be that I don't have a sample project substantial enough to expose these sorts of issues. |
|
Just another point of interest: This morning I figured that maybe it is one of the projects causing the error, so I commented out the When I start tye then, it works 100%. I then uncommented the first project, and I get immediate failure. I am currently checking the dotnet version of the projects in the working vs non-working solutions. I know that I was on net 6 RC on both projects last year, and in November 2021 I upgraded the problematic project to net 6 RTM. I only started the code for the problematic project in November 2021, and I seeded the initial structure of the project from the working project. |
I just checked, they are all The logs for the working project are incoming... |
|
@philliphoff here are the logs from the WORKING multi-repo project. |
|
I noticed that the dapr side-car services restart once on this project each time I start tye, but it is not a continuous cycle as before. The only main difference here is that the folder structure is completely different. I also use the |
Not to make you do more work than you've done already, but is there a project which exhibits the issue that can be stripped down to the bare minimum and possibly shared? (If IP is still an issue, you are also welcome to share it to |
I can do this for you |
|
I am just posting the logs for two runs here over the next two comments. This one, empty tye.yml file: |
|
This one, uncommented a single project in the tye.yml file: I manually stopped this run because it did not throw an
|
|
@philliphoff I have sent you an email |
|
I think I figured it out. I am not sure, but I was playing around with including selective projects into the tye.yml file, and then looking deeper at the dapr side car logs. I get two issues:
This error is strange, because redis is running and I can connect to it normally.
So, I have multiple components that are all state stores, and the goal is that in production I will be using different state stores for each, but in development... they're all pointing to the local development redis. This is a major change from the working multi-repo project, where I am not using the state stores. I do have multiple pubsub components that all use redis, but not a single state store. My feeling is that this is a dapr issue, where using multiple state stores with unique names, but pointing to the same place is causing an issue with the side-car. And if I have to put my finger on it, specifically the I am testing this now, will revert with feedback. |
|
Ok so that was not the issue, I am getting the same |
|
@stephanjohnson So I'm seeing the same Is it correct that each individual actor state store in your solution is used only by individual services? If that's that the case, then you may just need to partition your Dapr component configuration files on a per-service basis. Earlier versions of Tye did not allow this (because you could only specify a single So, it'd look something like: extensions:
- name: dapr
# Default component path directory.
components-path: ./components/
services:
account-server:
# Components path just for account-server service
components-path: ./components/account/ |
|
Hi @philliphoff I will test this over the next two days and revert with feedback. Thank you for your assistance thus far. |
|
@philliphoff I have upgraded to the 0.11 release and since then I have not had this issue. I have purposefully stopped and started my solution a number of times, and it seems that the issue is no longer happening. I have also ensured that my dapr components are declared correctly (i.e. no duplicate actor state stores). |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Describe the bug
I randomly, though frequently, encounter the following error when running tye. It usually takes about 4 or 5 attempts to start tye before I get a fully running instance.
Sometimes, though much more seldom, the
--watchproperty on the run command will cause the same error when rebuilding the sources.This started happening more and more frequently since September after I updated to Dapr 1.4 and Tye 0.10.0.
I recently attempted to update tye to 0.11, but that caused way more errors with the Dapr sidecar services that I didn't have time, or energy, to diagnose.
I also attempted to downgrade to tye 0.09, but that did not solve the problem.
The problem became more and more frequent over time, and definitely seemed to become worse when I went from .net 6 RC to the .net 6 RTM. I have not added more projects to the tye.yml file, or changed anything on my dev machine that could have impacted this. When I attempted to play around with the versions of Tye, I also updated to the latest .net 6 RTM patch (101) and that seemed to have made things a little bit better.
To Reproduce
This happens when I run
tye run --watch --debugtye.ymlfile:Further technical details
0.10.0-alpha.21420.1+7cec0655cc474aee8d3c3a3d180678985209d439debug output
The text was updated successfully, but these errors were encountered: