0

As the title states I'm looking for some help with running my python script in Power BI.

I currently have the following setup that I need to automate:

  1. My data is on an Azure DWH. I export the dataset to a CSV file.
  2. This CSV is loaded in Python and I'm basically transforming/enhancing the dataset here. (I'm applying text analytics from the Cognitive Services and K-means clustering). Then I export the dataset to CSV.
  3. Now I load the CSV in Power BI to do the visualisation.

Unfortunately we're not allowed to run Python scripts on the Azure DWH as this is resource intensive.

My plan is to connect directly to the DWH from Power BI to load in the data. Now I want to schedule the python script to run lets say every week or something. However, I don't know if this is possible. Has anyone got experience with this?

I'm currently doing some research on "Python script editor" and "Python script". Is it possible to write back data into the Azure DWH using the python script editor?

Looking forward to your reply and thanks in advance.

Kind regards,

0

I suggest you look at writing a custom ETL(Extract Transform and Load) Script.

  1. Connect to your DWH (using db connectors), fetch the data, do the processing and save the output file directly.
  2. Save the output data on a common folder source which PowerBI uses to refresh it's data at defined intervals of time. Existing files would be replaced with new ones. You could look at something like s3 or a common sftp in the same network to organize this better.
  3. If you can't use windows task scheduler or linux crontab to schedule the script on the azure dwh machine, just schedule it on a separate server if possible. Essentially the data processing steps remain the same, just that the processing happens on a separate server.

And yes its possible to write back data to azure using python. That's another approach, write back the transformed data into a separate table and point powerBI to that table.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.