Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

removed recursion from Explain Action to avoid stackoverflow in some … #419

Merged
merged 1 commit into from
Jul 20, 2022
Merged

removed recursion from Explain Action to avoid stackoverflow in some … #419

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
80 changes: 21 additions & 59 deletions ...h/indexmanagement/indexstatemanagement/transport/action/explain/TransportExplainAction.kt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@ import org.opensearch.indexmanagement.indexstatemanagement.util.MetadataCheck
import org.opensearch.indexmanagement.indexstatemanagement.util.checkMetadata
import org.opensearch.indexmanagement.indexstatemanagement.util.managedIndexMetadataID
import org.opensearch.indexmanagement.opensearchapi.parseWithType
import org.opensearch.indexmanagement.opensearchapi.suspendUntil
import org.opensearch.indexmanagement.spi.indexstatemanagement.model.ISMIndexMetadata
import org.opensearch.indexmanagement.spi.indexstatemanagement.model.ManagedIndexMetaData
import org.opensearch.indexmanagement.util.SecurityUtils.Companion.buildUser
Expand Down Expand Up @@ -339,68 +340,29 @@ class TransportExplainAction @Inject constructor(
val filteredPolicies = mutableListOf<PolicyID?>()
val enabledStatus = mutableMapOf<String, Boolean>()
val filteredAppliedPolicies = mutableMapOf<String, Policy>()
filter(0, filteredIndices, filteredMetadata, filteredPolicies, enabledStatus, filteredAppliedPolicies)
}

@Suppress("LongParameterList")
private fun filter(
current: Int,
filteredIndices: MutableList<String>,
filteredMetadata: MutableList<ManagedIndexMetaData?>,
filteredPolicies: MutableList<PolicyID?>,
enabledStatus: MutableMap<String, Boolean>,
filteredAppliedPolicies: MutableMap<String, Policy>
) {
val request = ManagedIndexRequest().indices(indexNames[current])
client.execute(
ManagedIndexAction.INSTANCE,
request,
object : ActionListener<AcknowledgedResponse> {
override fun onResponse(response: AcknowledgedResponse) {
filteredIndices.add(indexNames[current])
filteredMetadata.add(indexMetadatas[current])
filteredPolicies.add(indexPolicyIDs[current])
enabledState[indexNames[current]]?.let { enabledStatus[indexNames[current]] = it }
appliedPolicies[indexNames[current]]?.let { filteredAppliedPolicies[indexNames[current]] = it }
if (current < indexNames.count() - 1) {
// do nothing - skip the index and go to next one
filter(current + 1, filteredIndices, filteredMetadata, filteredPolicies, enabledStatus, filteredAppliedPolicies)
} else {
sendResponse(
filteredIndices, filteredMetadata, filteredPolicies, enabledStatus,
totalManagedIndices, filteredAppliedPolicies
)
}
}

override fun onFailure(e: Exception) {
when (e is OpenSearchSecurityException) {
true -> {
totalManagedIndices -= 1
if (current < indexNames.count() - 1) {
// do nothing - skip the index and go to next one
filter(
current + 1,
filteredIndices,
filteredMetadata,
filteredPolicies,
enabledStatus,
filteredAppliedPolicies
)
} else {
sendResponse(
filteredIndices, filteredMetadata, filteredPolicies, enabledStatus,
totalManagedIndices, filteredAppliedPolicies
)
}
}
false -> {
actionListener.onFailure(e)
}
}
CoroutineScope(Dispatchers.IO).launch {
// filter out indicies for which user doesn't have manage index permissions
for (i in 0 until indexNames.count()) {
val request = ManagedIndexRequest().indices(indexNames[i])
try {
client.suspendUntil<NodeClient, AcknowledgedResponse> { execute(ManagedIndexAction.INSTANCE, request, it) }
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Although this will solve the stack overflow issue, I still see the multiple calls for TransportManagedIndexAction for every index as a potential problem and bottleneck for the clusters with large number of indices. It could still block the primary (transport) worker thread until all the transport action calls succeeds or fails for every index and could lead to starvation issue.

The non operational transport action that is used by ISM to check if the user has required index permissions to manage index in form of TransportManagedIndexAction seems to be a workaround due to lack of better ways for mapping user permission with indices.

I will prefer instrumenting some cleaner and direct ways of identifying index permissions for the user. This is outside the scope of this PR, but we should open an issue and follow up separately on the possible approaches.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Created #422 so that we don't miss out on this and look into this at priority.

cc: @qreshi @downsrob @petardz @praveensameneni

filteredIndices.add(indexNames[i])
filteredMetadata.add(indexMetadatas[i])
filteredPolicies.add(indexPolicyIDs[i])
enabledState[indexNames[i]]?.let { enabledStatus[indexNames[i]] = it }
appliedPolicies[indexNames[i]]?.let { filteredAppliedPolicies[indexNames[i]] = it }
} catch (e: OpenSearchSecurityException) {
totalManagedIndices -= 1
} catch (e: Exception) {
actionListener.onFailure(e)
}
}
)
sendResponse(
filteredIndices, filteredMetadata, filteredPolicies, enabledStatus,
totalManagedIndices, filteredAppliedPolicies
)
}
}

@Suppress("LongParameterList")
Expand Down