Improve memory usage of DnRoleMapper #34237
Assignees
Labels
:Security/Authentication
Logging in, Usernames/passwords, Realms (Native/LDAP/AD/SAML/PKI/etc)
Comments
jaymode
added
help wanted
|
Pinging @elastic/es-security |
jaymode
added a commit
to jaymode/elasticsearch
that referenced
this issue
Oct 2, 2018
The `DnRoleMapper` class is used to map distinguished names of groups and users to role names. This mapper builds in an internal map that maps from a `com.unboundid.ldap.sdk.DN` to a `Set<String>`. In cases where a lot of distinct DNs are mapped to roles, this can consume quite a bit of memory. The majority of the memory is consumed by the DN object. For example, a 94 character DN that has 9 relative DNs (RDN) will retain 4KB of memory, whereas the String itself consumes less than 250 bytes. In order to reduce memory usage, we can map from a normalized DN string to a List of roles. The normalized string is actually how the DN class determines equality with another DN and we can drop the overhead of needing to keep all of the other objects in memory. Additionally the use of a List provides memory savings as each HashSet is backed by a HashMap, which consumes a great deal more memory than an appropriately sized ArrayList. The uniqueness we get from a Set is maintained by first building a set when parsing the file and then converting to a list upon completion. Closes elastic#34237
jaymode
added a commit
that referenced
this issue
Oct 3, 2018
The `DnRoleMapper` class is used to map distinguished names of groups and users to role names. This mapper builds in an internal map that maps from a `com.unboundid.ldap.sdk.DN` to a `Set<String>`. In cases where a lot of distinct DNs are mapped to roles, this can consume quite a bit of memory. The majority of the memory is consumed by the DN object. For example, a 94 character DN that has 9 relative DNs (RDN) will retain 4KB of memory, whereas the String itself consumes less than 250 bytes. In order to reduce memory usage, we can map from a normalized DN string to a List of roles. The normalized string is actually how the DN class determines equality with another DN and we can drop the overhead of needing to keep all of the other objects in memory. Additionally the use of a List provides memory savings as each HashSet is backed by a HashMap, which consumes a great deal more memory than an appropriately sized ArrayList. The uniqueness we get from a Set is maintained by first building a set when parsing the file and then converting to a list upon completion. Closes #34237
jaymode
added a commit
that referenced
this issue
Oct 3, 2018
The `DnRoleMapper` class is used to map distinguished names of groups and users to role names. This mapper builds in an internal map that maps from a `com.unboundid.ldap.sdk.DN` to a `Set<String>`. In cases where a lot of distinct DNs are mapped to roles, this can consume quite a bit of memory. The majority of the memory is consumed by the DN object. For example, a 94 character DN that has 9 relative DNs (RDN) will retain 4KB of memory, whereas the String itself consumes less than 250 bytes. In order to reduce memory usage, we can map from a normalized DN string to a List of roles. The normalized string is actually how the DN class determines equality with another DN and we can drop the overhead of needing to keep all of the other objects in memory. Additionally the use of a List provides memory savings as each HashSet is backed by a HashMap, which consumes a great deal more memory than an appropriately sized ArrayList. The uniqueness we get from a Set is maintained by first building a set when parsing the file and then converting to a list upon completion. Closes #34237
kcm
pushed a commit
that referenced
this issue
Oct 30, 2018
The `DnRoleMapper` class is used to map distinguished names of groups and users to role names. This mapper builds in an internal map that maps from a `com.unboundid.ldap.sdk.DN` to a `Set<String>`. In cases where a lot of distinct DNs are mapped to roles, this can consume quite a bit of memory. The majority of the memory is consumed by the DN object. For example, a 94 character DN that has 9 relative DNs (RDN) will retain 4KB of memory, whereas the String itself consumes less than 250 bytes. In order to reduce memory usage, we can map from a normalized DN string to a List of roles. The normalized string is actually how the DN class determines equality with another DN and we can drop the overhead of needing to keep all of the other objects in memory. Additionally the use of a List provides memory savings as each HashSet is backed by a HashMap, which consumes a great deal more memory than an appropriately sized ArrayList. The uniqueness we get from a Set is maintained by first building a set when parsing the file and then converting to a list upon completion. Closes #34237
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
The
DnRoleMapperclass is used to map distinguished names of groups and users to role names. This mapper builds in an internal map that maps from acom.unboundid.ldap.sdk.DNto aSet<String>. In cases where a lot of distinct DNs are mapped to roles, this can consume quite a bit of memory; in one case a single DnRoleMapper was found to consume over 1GB of heap space.The majority of the memory is consumed by the DN object. A 94 character DN that has 9 relative DNs (RDN) will retain 4KB of memory. The majority of this memory comes from building objects for the relative dns and retaining those which includes their ASN1 representation.
We should look and see if we can optimize the memory usage of this class so that heap space can be better utilized.
The text was updated successfully, but these errors were encountered: