Use and Access Site Lists

[s7clarke10]

Hi,

This tap looks like it can be very handy to obtain a SharePoint List, however I'm not quite sure how I could do this.

I can't access this URL at my workplace:
https://graph.microsoft.com/v1.0/groups

But I can access this to get the content of the SharePoint List.

https://graph.microsoft.com/v1.0/sites/{SiteID}/Lists/{ListID}/items/?expand=fields

This call will return the contents of the List which I will need to paginate through the items in it.

Having a look at the code I have a feeling that I need to add the following code to access sites.

streams.py

class SitesStream(MSGraphStream):
    name = "sites"
    path = "/sites"
    primary_keys = ["id"]
    odata_context = "sites"

tap.py

from tap_ms_graph.streams import (
    GroupMembersStream,
    GroupsStream,
    SubscribedSkusStream,
    SitesStream,
    UsersStream,
)

STREAM_TYPES = [
    GroupsStream,
    GroupMembersStream,
    SubscribedSkusStream,
    SitesStream,
    UsersStream,
]

However when I run the following Meltano command meltano select tap-ms-graph --list, I can't see any Sites available to select. I would like to add some enhancements to access SharePoint Lists but not sure if I am heading in the right direction or not.

Any advice would be much appreciated as I need to ingest several SharePoint Lists.
Thanks

[robby-rob-slalom]

Hey @s7clarke10,

I had a chance to look into this and the /sites endpoint appears to only work with application permissions and shows root sites across geographies (see here). Try /sites/root or /sites/{site_id} (or /sites/{site_id}/sites for subsites).

To get to the lists of a site, there would need to be a way to get the {site_id} to a stream endpoint /sites/{site_id}/lists.

Here is an example of adding two streams and chaining them together:

class SiteRootsStream(MSGraphStream):
    name = "siteRoots"
    path = "/sites/root"
    records_jsonpath = "$"
    primary_keys = ["id"]
    odata_context = "sites/$entity"
    child_context = {"id": "site_id"}

class SiteListsStream(MSGraphChildStream):
    parent_stream_type = SiteRootsStream
    name = "siteLists"
    path = "/sites/{site_id}/lists"
    primary_keys = ["site_id", "id"]
    odata_context = "sites/lists"

    parent_context_schema = {
        "site_id": {"type": "string"},
    }

The odata_context and other settings may need to be adjusted for the schema to work.

[s7clarke10]

Thank you very much for providing that insight into supporting sites and lists. I will look to see how I can progress with this new knowledge. Kind Regards Steve From: Robby Robinson ***@***.***> Sent: Thursday, June 29, 2023 6:55 AM To: Slalom-Consulting/tap-ms-graph ***@***.***> Cc: Steve Clarke ***@***.***>; Mention ***@***.***> Subject: Re: [Slalom-Consulting/tap-ms-graph] Use and Access Site Lists (Issue #17) Hey @s7clarke10 <https://github.com/s7clarke10> , I had a chance to look into this and the /sites endpoint appears to only work with application permissions and shows root sites across geographies (see here <https://learn.microsoft.com/en-us/graph/api/site-list?view=graph-rest-1.0&tabs=http> ). Try /sites/root or /sites/{site_id} (or /sites/{site_id}/sites for subsites). To get to the lists of a site, there would need to be a way to get the {site_id} to a stream endpoint /sites/{site_id}/lists. Here is an example of adding two streams and chaining them together: class SiteRootsStream(MSGraphStream): name = "siteRoots" path = "/sites/root" records_jsonpath = "$" primary_keys = ["id"] odata_context = "sites/$entity" child_context = {"id": "site_id"} class SiteListsStream(MSGraphChildStream): parent_stream_type = SiteRootsStream name = "siteLists" path = "/sites/{site_id}/lists" primary_keys = ["site_id", "id"] odata_context = "sites/lists" parent_context_schema = { "site_id": {"type": "string"}, } The odata_context and other settings may need to be adjusted for the schema to work. — Reply to this email directly, view it on GitHub <#17 (comment)> , or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUDU42WBCZWA5ZWA4UX565DXNR4ZFANCNFSM6AAAAAAYP4JVH4> . You are receiving this because you were mentioned. <https://github.com/notifications/beacon/AUDU42VWOMLBLDN4TK7RXMTXNR4ZFA5CNFSM6AAAAAAYP4JVH6WGG33NNVSW45C7OR4XAZNMJFZXG5LFINXW23LFNZ2KUY3PNVWWK3TUL5UWJTTACPYUI.gif> Message ID: ***@***.*** ***@***.***> >