Calling saspy-script from Enterprise Guide - project becomes unresponsive

[Erra3]

Hi!

I've been up and down the documentation of saspy for a couple of days now and I feel I might need some help on this one.
First let me give you some background on what I'm working with.

We have a setup where we run SAS from on Linux (RHEL).
All of our end users have access to SAS through clients like SAS Enterprise Guide or other windows clients.

On the linux server I have installed a lot of python modules, like saspy and pandas to enable some of our end users to work with SAS data in python instead of only using SAS-code.
The setup seems to work fine and I am able to create python scripts with functions that call saspy to get tables in and out of SAS. The scripts run fine when I call them directly from the command prompt in putty.
Right now I am using the default connection in sascfg_personal.py and I have pointed the variable to match our folder tree for SASHome/SASFoundation/.../... so that we point out the correct binary file (sas_u8).

Now to my problem:
As mentioned before our users access SAS through clients like EG.
What I would like to achieve is that they can submit some sort of rc/x/pipe command towards the python code that is stored on the server and get the data back into a sas library in EG. Everything seems to be in order except one little thing and that is the connection between my client session and saspy.

As soon as I submit an rc-command from EG towards the python script the session just hangs and becomes unresponsive. I've tracked this down to the connection string to saspy that I have in my python-script. The connection string looks like this:

sas = saspy.SASsession(cfgfile='/home/xxx/.local/lib/python3.8/site-packages/saspy/sascfg_personal.py')

The above code works from command line, but not when submitting the script from EG. As said above, the session just hangs and gives zero response back to what might be wrong.

One other thing to mention is that I'm using the same user on the server as in EG, hence it's probably not an access right problem.
The difference as I can tell is that EG connects via the Metadata Server and executes on the workspace server while the putty session is directly towards the workspace server.

I have a hunch that it could have something to do with the connection typ used, but I am not sure and have not been able to connect with any other connection setup either.

Could someone please advice on how to proceed or come with some clever suggestions?

Thanks!

[tomweber-sas]

Hey, I can help with this. There are two different things I would have input on. First is since your users are connecting to your SAS servers via EG, this means your deployment is a production deployment with workspace servers, metadata server, and configurations providing authentication and authorization. By having SASPy use STDIO to connect to the local Linux deployment, you're bypassing all of that deployment configuration. You're not connecting users to any workspace server, you're letting each just start independent SAS session on the Workspace server machine. The more correct way to connect is via the IOM access method, which uses metadata credential to connect to actual Workspace servers based upon metadata configuration. This is normally done by the clients themselves from their client machine. But, having the clients run scripts to call your server side python, is a choice; maybe you don't expect your users to have python on their own machines?
The second thing is what's happening w/ your scripts. Without seeing what's in these config files, I can only guess, but it looks like you expect each user to have python deployed in their own home directories on the server machine, and have a config file in there. Using STDIO, as you are, that seems unnecessary since it would need nothing user specific in it's configuration. Do all of the users have this config file in this path of their home dirs? The 'hang' you describe could be caused from saspy trying to prompt for some info if the config file has more than one configuration definition or something. Without seeing how your script works and how it would handle a failure in the Python script, I can't tell what's happening. If it's a case where saspy could be prompting, thus hanging the session, you could add prompt=False to the SASsession() call, and if prompting was causing it to sit there, it would produce a failure instead. I assume you've tried running your remote script with a known failure case to show it handles that and doesn't hang?

So, 1st question is why wouldn't the users use IOM to connect to the workspace servers from their own machines, same as using EG?
And, can we try out prompt=False, and also show that your script handles a failure cleanly too, to get a little more on what's going wrong with that part of what you have.

Thanks,
Tom

[Erra3]

Hi Tom,

Thanks for your reply!
I am aware that we are surpassing the metadata authorization by doing it the way we do, but it is also kind of intended. The user has already once authorized towards the metadata server once signing in to Enterprise guide and that is good enough for now.
As you say, what I am trying to achieve here is for the user to be able to spawn new SAS sessions using the built in STDIO connection. The intention of this is to later on be able to batch schedule the code they've built using DI Studio. The batch execution ofcourse takes place with some sort of admin account. Since this is the preferred work pattern we would like for it to be possible to execute python scripts from DI or SAS EG using some sort of server commands and still fetch data using SASPY. This is where we fail right now when the EG-session just becomes unresponsive once trying to establish an connection to saspy (using STDIO).
Since our users won't schedule on their own computers and not all of them are admins, I would like to avoid using the IOM access method with personal java setups and local installations of python on their computers. Everything should be available for them from SAS basically - in a dream world!

I've tested your suggestion with setting prompt='False' on both the connection string and also in the config file without any luck. The session still just "hangs" and becomes unresponsive.

Have you experienced anything like this when working with saspy yourself? From my point of view it feels very strange that the session just stops.

The python script we call looks like below and is right now just a dummy script to test out the concept:

`
#! /opt/rh/rh-python38/root/bin/python3.8

import sys

def saspython(workpath, datasetString):
import pandas
import saspy

dummy='Dummy'

sas = saspy.SASsession(cfgfile='/home/userXXX/.local/lib/python3.8/site-packages/saspy/sascfg_personal.py')
sas.saslib('worklib', path=workpath)

df = pandas.read_sas(workpath+datasetString+".sas7bdat")
city= ['Delhi']
df['city'] = city

print(df)

sas_dataset_object = sas.dataframe2sasdata(
df=df,
libref='worklib',
table='CITY'
)

return(dummy)

saspython(sys.argv[1],sys.argv[2])
`

In addition to that our code in SAS to call the script looks like below:

 data _null_; rc=system("/path/to/script/folder/test_saspython.py {workpathOnServer} {tableLocatedOnWorkpath}"); put rc; run; 

Please note that we have tested pipe and x commands in addition to the above mentioned rc-command.

If you have any suggestions I would be very thankful!

Thanks,
Erik

[tomweber-sas]

Hey, thanks for that, that helps. Yes, I didn't expect the prompt to be the problem, but couldn't tell without more info. So that's good. What you have seems like it could work for you. I suspect the hang has something to do with your script or the system() function from the data step. I mocked up something similar and it worked fine with this method. I wasn't able to get what you show as '{workpathOnServer} {tableLocatedOnWorkpath}` to resolve to anything. I tried both SAS macros and environment variables, but I only get the stings themselves.

Here's the script I tried:

#!/opt/tom/anaconda3/bin/python3
import sys


print(sys.argv)
x = input("hey")
print(x)

import saspy
sas = saspy.SASsession(cfgname='tdi')
print(sas)
df = sas.sd2df('cars','sashelp')
print(df.shape)
print(sas.endsas())

And when I use the system() call from SAS:

 17? data _null_; rc=system("/opt/tom/github/saspy/saspy/misc/nomain.py {PS1} {PWD}"); put rc; run;
['/opt/tom/github/saspy/saspy/misc/nomain.py', '{PS1}', '{PWD}']
hey there
 there
SAS Connection established. Subprocess id is 155411

Access Method         = STDIO
SAS Config name       = tdi
SAS Config file       = /opt/tom/github/saspy/saspy/sascfg_personal.py
WORK Path             = /usr/tmp/SAS_work55AD00025F13_tom64-7/
SAS Version           = 9.04.01M8P01182023
SASPy Version         = 5.4.4
Teach me SAS          = False
Batch                 = False
Results               = Pandas
SAS Session Encoding  = utf-8
Python Encoding value = utf_8
SAS process Pid value = 155411


(428, 15)
SAS Connection terminated. Subprocess id was 155411
None

0
NOTE: DATA statement used (Total process time):
      real time           7.76 seconds
      cpu time            0.01 seconds


 18?

Can you just run that python script locally without the hang? Can you comment out the SASPy parts and just have some prints so you can see if it's doing anythin g, or what it's getting as parms when calling it from SAS? Do you see it execute when called from SAS?

Tom

[Erra3]

Hi Tom,

Nice to hear that you could get it to run using a similar setup.
I have tested to run my script from EG and directly from the local server. When executing the script from the local server like this:

./test_saspython.py /opt/sas/Scripts/PythonTest/test_saspython.py /opt/sas/saswork/SAS_work7E5C0000AA2C_x15805azz.nextrel.tre.se/SAS_work6F1D0000AA2C_x15805azz.nextrel.tre.se/ table3

The workpath is the location of a previously created table in SASWORK and the second parameter is the name of that table.
When executing this localy with saspy active I get it to complete and create a new table with an additional column.

However if i run the same command via pipe-command from SASEG i get no return and the EG-program just keeps spinning forever.

If I comment out the connection string to saspy I get the script to run from SAS EG, but then of course it fails on "sas" not being defined. See return below:

Traceback (most recent call last): File "/opt/sas/Scripts/PythonTest/test_saspython.py", line 64, in <module> saspython(sys.argv[1],sys.argv[2]) File "/opt/sas/Scripts/PythonTest/test_saspython.py", line 18, in saspython sas.saslib('worklib', path=workpath) NameError: name 'sas' is not defined

I have also (with saspy connection commented out) printed the input variables to file when executing from SAS EG and it looks just fine.
They resolve to:

/opt/sas/saswork/SAS_work7E5C0000AA2C_x15805azz.nextrel.tre.se/SAS_work6F1D0000AA2C_x15805azz.nextrel.tre.se/ and table3

As said before, when i execute from SAS EG with saspy connection enabled the session just hangs. See image below.

And at the same time the log just states that it is "running"...

`1 The SAS System 21:14 Wednesday, November 29, 2023

1 ;';";*/;quit;run;
2 OPTIONS PAGENO=MIN;
3 %LET _CLIENTTASKLABEL='Script to call python with SAS table and get SAS table back';
4 %LET _CLIENTPROCESSFLOWNAME='Process Flow';
5 %LET _CLIENTPROJECTPATH='C:\Users\Karlen01\OneDrive - Hi3G Access AB\EG-projekt\ConnectToPythonScript_v2.egp';
6 %LET _CLIENTPROJECTPATHHOST='L10-5CG103B57V';
7 %LET _CLIENTPROJECTNAME='ConnectToPythonScript_v2.egp';
8 %LET _SASPROGRAMFILE='';
9 %LET _SASPROGRAMFILEHOST='';
10
11 ODS ALL CLOSE;
12 OPTIONS DEV=SVG;
13 GOPTIONS XPIXELS=0 YPIXELS=0;
14 %macro HTML5AccessibleGraphSupported;
15 %if %_SAS_VERCOMP_FV(9,4,4, 0,0,0) >= 0 %then ACCESSIBLE_GRAPH;
16 %mend;
17 FILENAME EGHTML TEMP;
18 ODS HTML5(ID=EGHTML) FILE=EGHTML
19 OPTIONS(BITMAP_MODE='INLINE')
20 %HTML5AccessibleGraphSupported
21 ENCODING='utf-8'
22 STYLE=HtmlBlue
23 NOGTITLE
24 NOGFOOTNOTE
25 GPATH=&sasworklocation
26 ;
NOTE: Writing HTML5(EGHTML) Body file: EGHTML
27
28 data table3;
29 var1=1;
30 var2=11;
31 var3=111;
32 run;

NOTE: Compression was disabled for data set WORK.TABLE3 because compression overhead would increase the size of the data set.
NOTE: The data set WORK.TABLE3 has 1 observations and 3 variables.
NOTE: DATA statement used (Total process time):
real time 0.00 seconds
cpu time 0.01 seconds

33
34 %put &SASWORKLOCATION;
"/opt/sas/saswork/SAS_work73F40000E751_x15805azz.nextrel.tre.se/SAS_work95F60000E751_x15805azz.nextrel.tre.se/"
35
36
36 ! data null;
37 rc=system("/opt/sas/Scripts/PythonTest/test_saspython.py
37 ! /opt/sas/saswork/SAS_work73F40000E751_x15805azz.nextrel.tre.se/SAS_work95F60000E751_x15805azz.nextrel.tre.se/ table3");
38 put rc;
39 run;`

Thanks
Erik

[tomweber-sas]

Ok, so that's more clear. I see your notation of {workpathOnServer} was simply representing the actual parameters you would be passing; so that one's solved.

Are you saying that when you have an syntax error in the python code, that it returns, in EG, with that failure? As well as if you remove the saspy code (without python syntax errors), then it works in EG also? The python script actually ran and did something on the server side.

And, I'm not really sure what you mean by the terms pipe-command and rc-command. I just see you're using the system function in the data step. Do you use those other terms to refer to that? Or is there something else that I'm missing? I'm curios of the case where you ran locally being
./test_saspython.py /opt/sas/Scripts/PythonTest/test_saspython.py {path} {table}
What's that first script, calling a second script? In my test there's only one script, which is the SASPy code. I call it directly from either the command line locally, or from SAS (thought it was SAS on my machine, not EG and not remotely on another system - I don't have that set up).

Can you see if the python process that should be started from EG is actually running or not, when you try that case? Is the SAS process saspy tries to start, actually running on that machine or not? While EG is hung; what's happening on the server side?

If it all works from EG except for when you do this, unless you try to use saspy in that script, then I can only think of permission issues where the userid starting the workspace server has issue with somethin gin the saspy config, like the file you're pointing to. But that would simply fail and return then. So, I'm really not sure what can be happening without looking at what's actually happening on that server side system when you try this.

Tom

[Erra3]

Hi,

What I'm trying to say is that if i take out saspy connection from the server side python script the script will run from EG with an error message that i have not initiated an saspy connection :) I can also run a plain python script without any reference to saspy what so ever and it completes just fine. Hence my conclusion is that the problem resides within the saspy connection, since that is the one that's causing the script to hang.

I made some tests with locating the processes server side and I am successful of doing so both when executing from EG and when executing straight from the SAS Application server (using a workspace server connection). Below you see two processes running the script. One is from SAS EG and one is started straight from the server.

I am feeling quite confident that this is some sort of issue with how saspy handles or sets up the connection when i come from EG, but I can't put my finger on what it is that goes wrong.

When I say x-commands and pipe I refer to different ways to call the server side from SAS EG or any other SAS client application.
https://documentation.sas.com/doc/en/pgmsascdc/9.4_3.5/hostunx/n1ceb0xedanuj3n19l3g73awk1wf.htm
https://documentation.sas.com/doc/en/pgmsascdc/9.4_3.5/hostunx/p08yb3p7dchum8n1poyexvfve3gq.htm
Basically I have tested to call the script from SAS EG using all of the above ways and also using rc as you saw previously.

Could it be an idea to use the IOM connection instead and see if that works better? I would really like to avoid installing a separate java just to try this, but if needed I will try it. I can't think of any possible way where the ssh connection would be the answer since i'm starting the script from the server where the sas binaray can be found.

If you find this issue interesting I could probably show you more in a live session or something like that, but perhaps that is to much to ask.

Anyhow - thanks a lot for your input on the issue!

Br.
Erik

[tomweber-sas]

Actually, I was gonna see about getting on a teams call to see this, next. Let's do that, as it will make this much quicker to figure out! I can set that up, or you can, either way! Some time tomorrow?

[Erra3]

Let's set it up then! I guess we're working out of quite different time zones, so I will try to adjust to your time. Stockholm where I'm based is currently in GMT+1 so perhaps the earliest time you have tomorrow (today)?

[tomweber-sas]

Ok, I'm EST, so +5 currently. I'm a bit swamped right now with other (not saspy) customer problems I'm involved with. Can we set this up for Monday? I bet we can figure it out looking at it together. I do this with customers a fair amount and it does make strange things like this much easier to figure out!

[Erra3]

Should I just send you an email and we can continue the meeting arrangements from there?

[tomweber-sas]

Yes, please, that will be great!

[Erra3]

Sent you an email :)

[tomweber-sas]

Got it! I'll send an invite soon

[Erra3]

Hi Tom!

I got the things running. I figured out that SAS ships with a private java which means I don't need to install any separate java.
Instead of using the default connection I am now using the iomlinux with the java parameter pointing towards /opt/sas/SASHome/SASPrivateJavaRuntimeEnvironment/9.4/jre/bin/java. When doing so I get the script to run, both when executing from the server directly and when executing it from SAS Enterprise Guide.

Still a bit uncertain if this is a good setup with using the private runtime, but perhaps you can enlighten me on that?

Thanks!

[tomweber-sas]

Actually, that's perfectly fine. The IOM access method uses the Java IOM Client to interact with an actual workspace server, so that's the more correct way to connect; you're actually getting a defined workspace server same as EG. So any predefined libraries will already be assigned and any other configuration will be in place. As opposed to just starting your own SAS session. And, any java is fine, it just need a basic JVM for that client. So the JRE SAS ships with is just fine!

That's Great!

[Erra3]

Thanks a lot for the help in the call today Tom!
I went ahead and made some more trouble shooting on my own and managed to recreate the problem with different users showing from different modules and methods.

What I did is that I run a python script from the server and append the values from these different methods.

`
user=getpass.getuser()
user2=os.path.expanduser('~')
user3=os.environ.get('USER')
user4=os.getlogin()

    f = open("/tmp/test.txt", "w")
    f.write(user or "Not found")

    f = open("/tmp/test.txt", "a")
    f.write(" "+user2 or " Not Found")

    f = open("/tmp/test.txt", "a")
    f.write(" "+user3 or " Not found")

    f = open("/tmp/test.txt", "a")
    f.write(" "+user4 or " Not found")

`

When running it straight from the server i get the expected user in the file from all methods- you remember the name right? :)

However whey i run it from SAS EG i get another user from the method os.getlogin(). Instead of the user I expect to appear I find it is printing user "sas".

Makes me think that it could have something to do with it, or what do you think?
Are you using os.getlogin() anywhere when validating the user?

[tomweber-sas]

I don't use that, no. For STDIO I don't know or care who the user is. For IOM, there needs to be metadata credentials, else you can't start a workspace server. So yes, I think this user issue is part of the problem, though I still am not sure why it would be hanging. Thus the diagnostics we were trying. Since we kelp getting permission problems, maybe try using a file in /tmp to get around that, so we can see if saspy is getting stderr or stdout back from SAS (cuz it seems to not be, else it shouldn't hang).

[Erra3]

Okay, so I redirected the file to /tmp and now the script runs fine from console, but just hangs when executing from EG (without any permission errors thrown). The file however gets zero content in it, so there is nothing to be found in the log.

[tomweber-sas]

There's something off between what user is running this and what SAS settings are being used when trying it that way. Just didn't get far enough to see what was really going on with it. What happens when you run the script locally, but from that user: 'sas' was it?
sudo su - sas
./run_the_scipt