Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

If the eureka client frequently sends register or renew requests, nodes in the eureka server cluster may be inconsistent #1509

Open
tkf0707 opened this issue Jul 25, 2023 · 1 comment

Comments

@tkf0707
Copy link

tkf0707 commented Jul 25, 2023

Scene description: An abnormal eureka client sends dozens of renew requests within 1s. As a result, the number of nodes in the eureka server cluster is inconsistent, and node information cannot be synchronized between the two servers.

Analyze:When nodes are synchronized in the eureka server cluster, a batchTaskDispatcher is created to package and send tasks in batches, and tasks are queued in the AcceptorExecutor.
  If the renewal frequency of an application instance(eureka.instance.lease-renewal-interval-in-seconds) is smaller than the synchronization execution frequency(MAX_BATCHING_DELAY_MS=500), and the Task of the application instance is at the head of the pendingTasks to be processed, the registry server fails to determine whether the difference between the task creation time and the current time is smaller than the synchronization execution time. As a result, the registry server does not put the renewal task in the batchWorkQueue to be executed.

AcceptorExecutor.java:      

        void assignBatchWork() {
            if (hasEnoughTasksForNextBatch()) {
                if (batchWorkRequests.tryAcquire(1)) {
                    long now = System.currentTimeMillis();
                    int len = Math.min(maxBatchingSize, processingOrder.size());
                    List<TaskHolder<ID, T>> holders = new ArrayList<>(len);
                    while (holders.size() < len && !processingOrder.isEmpty()) {
                        ID id = processingOrder.poll();
                        TaskHolder<ID, T> holder = pendingTasks.remove(id);
                        if (holder.getExpiryTime() > now) {
                            holders.add(holder);
                        } else {
                            expiredTasks++;
                        }
                    }
                    if (holders.isEmpty()) {
                        batchWorkRequests.release();
                    } else {
                        batchSizeMetric.record(holders.size(), TimeUnit.MILLISECONDS);
                        batchWorkQueue.add(holders);
                    }
                }
            }
        }

        private boolean hasEnoughTasksForNextBatch() {
            if (processingOrder.isEmpty()) {
                return false;
            }
            if (pendingTasks.size() >= maxBufferSize) {
                return true;
            }

            TaskHolder<ID, T> nextHolder = pendingTasks.get(processingOrder.peek());
            long delay = System.currentTimeMillis() - nextHolder.getSubmitTimestamp();
            return delay >= maxBatchingDelay;
        }

How should the server solve this problem to protect the synchronization between the server clusters?

@error0702
Copy link

error0702 commented Jul 25, 2023 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants