SingleStore Notebooks
New
Running Notebooks from Another Notebook with Fusion SQL
Notebook
In this notebook, we demonstrate how to use Fusion SQL to run a notebook from another notebook, either within the same session (useful to avoid code duplication) or in a new session (useful for parallel execution).
Creating the Sample Notebook
In this section, we will create a sample notebook to be used in our examples.
Create the sample notebook in the local filesystem:
In [1]:
1import nbformat as nbf2 3nb = nbf.v4.new_notebook()4 5cell = nbf.v4.new_code_cell("""# This is a code cell6if 'sample_var' not in globals():7 sample_var = 'sample value'8print('Sample Notebook has been executed!')""")9 10cell.metadata = {11 "language": "python"12}13 14nb.cells.append(cell)15 16# Save the notebook to a file17with open('sample_notebook.ipynb', 'w') as f:18 nbf.write(nb, f)19 20print("Notebook 'sample_notebook.ipynb' created successfully in the local filesystem.")
Upload it to the Data Studio shared files for later download:
In [2]:
1import time2 3sample_notebook_name='Sample Notebook {}.ipynb'.format(int(time.time() * 1_000_000))4 5%sql UPLOAD SHARED FILE TO '{{ sample_notebook_name }}' FROM 'sample_notebook.ipynb';6 7print("Notebook '{}' has been created in the Data Studio shared files.".format(sample_notebook_name))
Running the Sample Notebook in the Current Session
In this example, we will run the previously created sample notebook within the current session. This approach is useful for avoiding code duplication in your notebooks, such as environment setup or reusable functions.
To run the notebook, we can use the %run_shared magic command. Let's run the notebook and confirm that the sample_var variable set in the sample notebook is accessible in the current session:
In [3]:
1if 'sample_var' in globals():2 del sample_var3 4%run_shared {{ sample_notebook_name }}5 6print("The value of 'sample_var' is '{}'.\n".format(sample_var))
Running the Sample Notebook in a New Session
Instead of running a notebook in the current session, we can run it in a new session — either synchronously or asynchronously — using jobs. This approach is useful for parallel execution or running code in a separate runtime environment.
Run two jobs in parallel and wait for their completion (each job will run on a separate session):
In [4]:
1job_ids = []2for x in range(2):3 print("Running job for {}...".format(x))4 job_res = %sql RUN JOB USING NOTEBOOK '{{ sample_notebook_name }}' WITH PARAMETERS {"sample_var": "{{x}}"}5 job_ids.append(job_res[0].JobID)6 7print(f'Waiting for jobs to complete... {job_ids}')8success = %sql WAIT ON JOBS {{ job_ids }} WITH TIMEOUT 60 MINUTES9 10print(f'All jobs completed with success: {bool(success[0].Success)}')
View the executions of the jobs we ran:
In [5]:
1for job_id in job_ids:2 execs = %sql SHOW JOB EXECUTIONS FOR {{ job_id }} from 1 to 13 print(execs)
The jobs can now be dropped:
In [6]:
1for id in job_ids:2 print(f"Dropping job '{id}'...")3 %sql DROP JOBS {{id}}
Cleanup
Delete the sample notebook from the Data Studio shared files:
In [7]:
1%%sql2DROP SHARED FILE '{{ sample_notebook_name }}'
Details
About this Template
Learn how to run Notebooks from another Notebook in SingleStoreDB Cloud using Fusion SQL.
This Notebook can be run in Standard and Enterprise deployments.
Tags
notebooksjobspythonfusionstarter
License
This Notebook has been released under the Apache 2.0 open source license.
See Notebook in action
Launch this notebook in SingleStore and start executing queries instantly.