Get SharePoint List Items coming in an infinite loop with 100 items List

[gavdgavd]

Category

• Bug

Describe the bug

Goal: get the first 100 items from a SharePoint List containing more than 100 items
If the List contains 99 or less items, everything works fine. If the List contains 100 or more items, the routine enters an infinite loop and does not return control.

Steps to reproduce

01 - Create a SharePoint List (just an out-of-the-box List, with only a Title column) and add 99 items
02 - Create a .NET 6.0 Console Application
03 - Add the code to get a PnPContext (using an AD App Registration with Delegated rights)
04 - Add the code to get the Items from the List:

    IList myList = myContext.Web.Lists.GetByTitle(myListName, p => p.Title);

    int intCounter = 0;
    foreach (var listItem in myList.Items)
    {
        intCounter++;
    }
    Console.WriteLine(intCounter);

("myContext" is the earlier collected PnPContext, "myListName" is a string with the List name)

05 - Run the App
06 - RESULT: It gets "99" as output (as expected) (see Debug trace at the end of this message)
07 - Add one item to the List (it will contain now 100 items)
08 - Run the App
09 - RESULT: The routine enters an infinite loop at the "myList.Items" statement and does not return control (see Debug trace at the end of this message)

Expected behavior

If the SharePoint List contains 100 or more Items, expected to get the first 100 Items.

Environment details (development & target environment)

• SDK version: NuGet PnP.Core "Last stable" version 1.6.0. and NuGet PnP.Core.Auth "Last stable" version 1.6.0.
• OS: Windows 10 and Windows Server 2022, both fully patched
• SDK used in: Tested in a .NET Console App and in a .Net Windows Forms App
• Framework: .NET v6.0
• Browser(s):
• Tooling: Visual Studio 2022 Enterprise 64-bit fully patched
• Additional details: see Additional context

Additional context

---> Ouput running the application with LogLevel.Debug option and 99 Items in the List:

dbug: Microsoft.Extensions.Hosting.Internal.Host[1]
Hosting starting
info: Microsoft.Hosting.Lifetime[0]
Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
Hosting environment: Production
info: Microsoft.Hosting.Lifetime[0]
Content root path: C:\Projects\Test\Test_GetListItems\bin\Debug\net6.0
dbug: Microsoft.Extensions.Hosting.Internal.Host[2]
Hosting started
dbug: PnP.Core.Services.PnPContext[0]
GET https://domain.sharepoint.com/sites/O365Dev/_api/Web?$select=Id%2cUrl%2cRegionalSettings%2f*%2cRegionalSettings%2fDateFormat&$expand=RegionalSettings%2cRegionalSettings%2fTimeZone
info: System.Net.Http.HttpClient.SharePointRestClient.LogicalHandler[100]
Start processing HTTP request GET https://domain.sharepoint.com/sites/O365Dev/_api/Web?$select=Id%2cUrl%2cRegionalSettings%2f*%2cRegionalSettings%2fDateFormat&$expand=RegionalSettings%2cRegionalSettings%2fTimeZone
info: System.Net.Http.HttpClient.SharePointRestClient.ClientHandler[100]
Sending HTTP request GET https://domain.sharepoint.com/sites/O365Dev/_api/Web?$select=Id%2cUrl%2cRegionalSettings%2f*%2cRegionalSettings%2fDateFormat&$expand=RegionalSettings%2cRegionalSettings%2fTimeZone
info: System.Net.Http.HttpClient.SharePointRestClient.ClientHandler[101]
Received HTTP response headers after 282.0417ms - 200
info: System.Net.Http.HttpClient.SharePointRestClient.LogicalHandler[101]
End processing HTTP request after 318.9283ms - 200
dbug: PnP.Core.Services.PnPContext[0]
GET https://domain.sharepoint.com/sites/O365Dev/_api/Site?$select=Id%2cGroupId
info: System.Net.Http.HttpClient.SharePointRestClient.LogicalHandler[100]
Start processing HTTP request GET https://domain.sharepoint.com/sites/O365Dev/_api/Site?$select=Id%2cGroupId
info: System.Net.Http.HttpClient.SharePointRestClient.ClientHandler[100]
Sending HTTP request GET https://domain.sharepoint.com/sites/O365Dev/_api/Site?$select=Id%2cGroupId
info: System.Net.Http.HttpClient.SharePointRestClient.ClientHandler[101]
Received HTTP response headers after 198.8981ms - 200
info: System.Net.Http.HttpClient.SharePointRestClient.LogicalHandler[101]
End processing HTTP request after 214.0505ms - 200
dbug: PnP.Core.Services.PnPContext[0]
GET https://domain.sharepoint.com/sites/O365Dev/_api/web/lists?$select=Id%2cTitle&$filter=Title+eq+%27TestList%27&$top=1
info: System.Net.Http.HttpClient.SharePointRestClient.LogicalHandler[100]
Start processing HTTP request GET https://domain.sharepoint.com/sites/O365Dev/_api/web/lists?$select=Id%2cTitle&$filter=Title+eq+%27TestList%27&$top=1
info: System.Net.Http.HttpClient.SharePointRestClient.ClientHandler[100]
Sending HTTP request GET https://domain.sharepoint.com/sites/O365Dev/_api/web/lists?$select=Id%2cTitle&$filter=Title+eq+%27TestList%27&$top=1
info: System.Net.Http.HttpClient.SharePointRestClient.ClientHandler[101]
Received HTTP response headers after 132.7383ms - 200
info: System.Net.Http.HttpClient.SharePointRestClient.LogicalHandler[101]
End processing HTTP request after 143.7675ms - 200
dbug: PnP.Core.Services.PnPContext[0]
GET https://domain.sharepoint.com/sites/O365Dev/_api/web/lists(guid'5bd20879-3528-48e6-9a0d-79602a5d0ffa')/items?$top=100
info: System.Net.Http.HttpClient.SharePointRestClient.LogicalHandler[100]
Start processing HTTP request GET https://domain.sharepoint.com/sites/O365Dev/_api/web/lists(guid'5bd20879-3528-48e6-9a0d-79602a5d0ffa')/items?$top=100
info: System.Net.Http.HttpClient.SharePointRestClient.ClientHandler[100]
Sending HTTP request GET https://domain.sharepoint.com/sites/O365Dev/_api/web/lists(guid'5bd20879-3528-48e6-9a0d-79602a5d0ffa')/items?$top=100
info: System.Net.Http.HttpClient.SharePointRestClient.ClientHandler[101]
Received HTTP response headers after 161.4474ms - 200
info: System.Net.Http.HttpClient.SharePointRestClient.LogicalHandler[101]
End processing HTTP request after 175.6273ms - 200
dbug: PnP.Core.Services.PnPContext[0]
GET https://domain.sharepoint.com/sites/O365Dev/_api/Web/Lists(guid'5bd20879-3528-48e6-9a0d-79602a5d0ffa')?$select=Id%2cFields%2fInternalName%2cFields%2fFieldTypeKind%2cFields%2fTypeAsString%2cFields%2fTitle%2cFields%2fId&$expand=Fields
info: System.Net.Http.HttpClient.SharePointRestClient.LogicalHandler[100]
Start processing HTTP request GET https://domain.sharepoint.com/sites/O365Dev/_api/Web/Lists(guid'5bd20879-3528-48e6-9a0d-79602a5d0ffa')?$select=Id%2cFields%2fInternalName%2cFields%2fFieldTypeKind%2cFields%2fTypeAsString%2cFields%2fTitle%2cFields%2fId&$expand=Fields
info: System.Net.Http.HttpClient.SharePointRestClient.ClientHandler[100]
Sending HTTP request GET https://domain.sharepoint.com/sites/O365Dev/_api/Web/Lists(guid'5bd20879-3528-48e6-9a0d-79602a5d0ffa')?$select=Id%2cFields%2fInternalName%2cFields%2fFieldTypeKind%2cFields%2fTypeAsString%2cFields%2fTitle%2cFields%2fId&$expand=Fields
info: System.Net.Http.HttpClient.SharePointRestClient.ClientHandler[101]
Received HTTP response headers after 184.3365ms - 200
info: System.Net.Http.HttpClient.SharePointRestClient.LogicalHandler[101]
End processing HTTP request after 198.0333ms - 200
99 <-- As expected, correct output

---> Ouput running the application with LogLevel.Debug option and 100 Items in the List (begin of the output):

dbug: Microsoft.Extensions.Hosting.Internal.Host[1]
Hosting starting
info: Microsoft.Hosting.Lifetime[0]
Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
Hosting environment: Production
info: Microsoft.Hosting.Lifetime[0]
Content root path: C:\Projects\Test\Test_GetListItems\bin\Debug\net6.0
dbug: Microsoft.Extensions.Hosting.Internal.Host[2]
Hosting started
dbug: PnP.Core.Services.PnPContext[0]
GET https://domain.sharepoint.com/sites/O365Dev/_api/Web?$select=Id%2cUrl%2cRegionalSettings%2f*%2cRegionalSettings%2fDateFormat&$expand=RegionalSettings%2cRegionalSettings%2fTimeZone
info: System.Net.Http.HttpClient.SharePointRestClient.LogicalHandler[100]
Start processing HTTP request GET https://domain.sharepoint.com/sites/O365Dev/_api/Web?$select=Id%2cUrl%2cRegionalSettings%2f*%2cRegionalSettings%2fDateFormat&$expand=RegionalSettings%2cRegionalSettings%2fTimeZone
info: System.Net.Http.HttpClient.SharePointRestClient.ClientHandler[100]
Sending HTTP request GET https://domain.sharepoint.com/sites/O365Dev/_api/Web?$select=Id%2cUrl%2cRegionalSettings%2f*%2cRegionalSettings%2fDateFormat&$expand=RegionalSettings%2cRegionalSettings%2fTimeZone
info: System.Net.Http.HttpClient.SharePointRestClient.ClientHandler[101]
Received HTTP response headers after 260.983ms - 200
info: System.Net.Http.HttpClient.SharePointRestClient.LogicalHandler[101]
End processing HTTP request after 302.6084ms - 200
dbug: PnP.Core.Services.PnPContext[0]

---> Ouput running the application with LogLevel.Debug option and 100 Items in the List (Infinite loop). Note that the "skip" value increments with 100 each time:

dbug: PnP.Core.Services.PnPContext[0]
GET https://domain.sharepoint.com/sites/O365Dev/_api/web/lists(guid'5bd20879-3528-48e6-9a0d-79602a5d0ffa')/items?$top=100&$skip=14800
info: System.Net.Http.HttpClient.SharePointRestClient.LogicalHandler[100]
Start processing HTTP request GET https://domain.sharepoint.com/sites/O365Dev/_api/web/lists(guid'5bd20879-3528-48e6-9a0d-79602a5d0ffa')/items?$top=100&$skip=14800
info: System.Net.Http.HttpClient.SharePointRestClient.ClientHandler[100]
Sending HTTP request GET https://domain.sharepoint.com/sites/O365Dev/_api/web/lists(guid'5bd20879-3528-48e6-9a0d-79602a5d0ffa')/items?$top=100&$skip=14800
info: System.Net.Http.HttpClient.SharePointRestClient.ClientHandler[101]
Received HTTP response headers after 159.6204ms - 200
info: System.Net.Http.HttpClient.SharePointRestClient.LogicalHandler[101]
End processing HTTP request after 178.5766ms - 200
dbug: PnP.Core.Services.PnPContext[0]
GET https://domain.sharepoint.com/sites/O365Dev/_api/web/lists(guid'5bd20879-3528-48e6-9a0d-79602a5d0ffa')/items?$top=100&$skip=14900
info: System.Net.Http.HttpClient.SharePointRestClient.LogicalHandler[100]
Start processing HTTP request GET https://domain.sharepoint.com/sites/O365Dev/_api/web/lists(guid'5bd20879-3528-48e6-9a0d-79602a5d0ffa')/items?$top=100&$skip=14900
info: System.Net.Http.HttpClient.SharePointRestClient.ClientHandler[100]
Sending HTTP request GET https://domain.sharepoint.com/sites/O365Dev/_api/web/lists(guid'5bd20879-3528-48e6-9a0d-79602a5d0ffa')/items?$top=100&$skip=14900
info: System.Net.Http.HttpClient.SharePointRestClient.ClientHandler[101]
Received HTTP response headers after 192.0159ms - 200
info: System.Net.Http.HttpClient.SharePointRestClient.LogicalHandler[101]
End processing HTTP request after 203.3252ms - 200

Thanks for your contribution! Sharing is caring.

[jansenbe]

@gavdgavd : I've tested your sample and it works for me. I think there's a misunderstanding on how the loop works: foreach (var listItem in myList.Items) will iterate over all the pages, in my test I had a library with 2499 documents which results in 25 requests being sent to SharePoint (see below snippet from the log output).

That being said, when the list contains exactly 100 items there indeed is an infinite loop, which is a bug we'll fix. Thanks for testing PnP Core SDK and providing feedback.

PnP.Core.Services.PnPContext: Debug: GET https://bertonline.sharepoint.com/sites/prov-1/_api/web/lists(guid%2737726f50-b250-4f2c-b15b-768ef0421d62%27)/items?%24skiptoken=Paged%3dTRUE%26p_SortBehavior%3d0%26p_ID%3d2200&%24top=100
PnP.Core.Auth.OAuthAuthenticationProvider: Information: Retrieved access token for resource UsernamePasswordAuthenticationProvider with scopes 'https://bertonline.sharepoint.com/sites/prov-1' from , https://bertonline.sharepoint.com/.default
System.Net.Http.HttpClient.SharePointRestClient.LogicalHandler: Information: Start processing HTTP request GET https://bertonline.sharepoint.com/sites/prov-1/_api/web/lists(guid%2737726f50-b250-4f2c-b15b-768ef0421d62%27)/items?%24skiptoken=Paged%3dTRUE%26p_SortBehavior%3d0%26p_ID%3d2200&%24top=100
System.Net.Http.HttpClient.SharePointRestClient.ClientHandler: Information: Sending HTTP request GET https://bertonline.sharepoint.com/sites/prov-1/_api/web/lists(guid%2737726f50-b250-4f2c-b15b-768ef0421d62%27)/items?%24skiptoken=Paged%3dTRUE%26p_SortBehavior%3d0%26p_ID%3d2200&%24top=100
System.Net.Http.HttpClient.SharePointRestClient.ClientHandler: Information: Received HTTP response headers after 159.7408ms - 200
System.Net.Http.HttpClient.SharePointRestClient.LogicalHandler: Information: End processing HTTP request after 163.9837ms - 200
PnP.Core.Services.PnPContext: Debug: GET https://bertonline.sharepoint.com/sites/prov-1/_api/web/lists(guid%2737726f50-b250-4f2c-b15b-768ef0421d62%27)/items?%24skiptoken=Paged%3dTRUE%26p_SortBehavior%3d0%26p_ID%3d2300&%24top=100
PnP.Core.Auth.OAuthAuthenticationProvider: Information: Retrieved access token for resource UsernamePasswordAuthenticationProvider with scopes 'https://bertonline.sharepoint.com/sites/prov-1' from , https://bertonline.sharepoint.com/.default
System.Net.Http.HttpClient.SharePointRestClient.LogicalHandler: Information: Start processing HTTP request GET https://bertonline.sharepoint.com/sites/prov-1/_api/web/lists(guid%2737726f50-b250-4f2c-b15b-768ef0421d62%27)/items?%24skiptoken=Paged%3dTRUE%26p_SortBehavior%3d0%26p_ID%3d2300&%24top=100
System.Net.Http.HttpClient.SharePointRestClient.ClientHandler: Information: Sending HTTP request GET https://bertonline.sharepoint.com/sites/prov-1/_api/web/lists(guid%2737726f50-b250-4f2c-b15b-768ef0421d62%27)/items?%24skiptoken=Paged%3dTRUE%26p_SortBehavior%3d0%26p_ID%3d2300&%24top=100
System.Net.Http.HttpClient.SharePointRestClient.ClientHandler: Information: Received HTTP response headers after 165.0289ms - 200
System.Net.Http.HttpClient.SharePointRestClient.LogicalHandler: Information: End processing HTTP request after 171.1971ms - 200
PnP.Core.Services.PnPContext: Debug: GET https://bertonline.sharepoint.com/sites/prov-1/_api/web/lists(guid%2737726f50-b250-4f2c-b15b-768ef0421d62%27)/items?%24skiptoken=Paged%3dTRUE%26p_SortBehavior%3d0%26p_ID%3d2400&%24top=100
PnP.Core.Auth.OAuthAuthenticationProvider: Information: Retrieved access token for resource UsernamePasswordAuthenticationProvider with scopes 'https://bertonline.sharepoint.com/sites/prov-1' from , https://bertonline.sharepoint.com/.default
System.Net.Http.HttpClient.SharePointRestClient.LogicalHandler: Information: Start processing HTTP request GET https://bertonline.sharepoint.com/sites/prov-1/_api/web/lists(guid%2737726f50-b250-4f2c-b15b-768ef0421d62%27)/items?%24skiptoken=Paged%3dTRUE%26p_SortBehavior%3d0%26p_ID%3d2400&%24top=100
System.Net.Http.HttpClient.SharePointRestClient.ClientHandler: Information: Sending HTTP request GET https://bertonline.sharepoint.com/sites/prov-1/_api/web/lists(guid%2737726f50-b250-4f2c-b15b-768ef0421d62%27)/items?%24skiptoken=Paged%3dTRUE%26p_SortBehavior%3d0%26p_ID%3d2400&%24top=100

[gavdgavd]

Hi Bert.
Thanks for taking care about this. I can confirm that the issue happens only when there are exactly 100 items. Just tested with more items in a List, and it works as expected.
One again, thank you for your attention and for the very good work done with the PnP Core SDK.
Gustavo

[jansenbe]

@gavdgavd : a fix + extra test cases to cover the "list contains exactly one page size of items" have been added. This will be included in the next nightly build which you can test tomorrow. We plan to release version 1.7 before end of the month.

I'll close the issue now, but please re-open or create a new one in case you still have issues after testing with the next nightly (build 1.6.72 or higher).