@DetectivePikachu,
I never did hear of any solutions to this issue. I ended up rewriting my code to not use this method for loading a dataset. I am not sure why the code still works in a recipe, but not elsewhere.
It's been over a year - so I'm having a hard time remembering which new code I wrote to avoid this issue - but I generally read in datasets using the below code:
#Create SQL connection
qs_connect()
#Read in dataset using SQL connection
%%read_sql dataset -c QS_CONNECTION
SELECT *
FROM aep_dataset
LIMIT 5000000
Where:
'dataset' would be replaced by the name you want the loaded data to have within notebooks
* in the SELECT statement can be replaced by desired variables (if you don't want to include all dataset fields)
'aep_dataset' replaced with the dataset name from the Datasets section of AEP (this method is also nice because you don't need the long dataset id, you can just use the actual dataset name)
LIMIT 5000000 quantity can be replaced or removed, depending on dataset size
Running this code puts the loaded data into a pandas dataframe automatically.
Hopefully this helps in some way!