Using Threads

As for other features of Python for Delphi (py4d), threading is similar as in the Python/C API. Please consult the Python/C API Reference Manual's section on Thread State and the Global Interpreter Lock as an introduction to threading in py4d.

This section will concentrate on Delphi-specific information that will allow you to use py4d to execute python code in your multi-threaded application. It assumes that you have read the API manual page linked above.

Handling the Global Interpreter Lock

As you hopefully would have read, the Python library has an interpreter that is protected by a global lock. Since the Python library doesn't have a way to manage our threads for us, we must manage them ourselves. There are two ways to do this:

1. Using TPythonThreads instead of TThreads.
2. Maintaining thread states and the global interpreter lock ourselves.

TPythonThreads

TPythonThread is a child of Delphi's TThread. It provides extra functionality that maintains the Python information that is necessary for a Delphi thread to use py4d.

Demo11 contains an example of using TPythonThreads in a multi-threaded application. A TPythonThread that contains a sorting algorithm is created by the main class, executes immediately and runs until completion, upon which the main class returns to a normal state again. The main class does not execute any Python code and maintains the global interpreter state for the three different python threads that run their sorting algorithms upon creation.

Note: TPythonThread acquires the global Python lock when the thread starts and releases it when it ends. This is fine if the thread is executing purely Python code because Python will periodically release the global lock to allow other threads to run. However if your thread spends much time in plain Pascal code then you need to be aware that it will block all Python threads. In this case read on for the second (more involved) solution to thread control in py4d.

Maintaining the Lock and Your Own Thread States (No TPythonThreads)

This method should only be used if you require running python code in your main thread and subsequent threads you create. By far the easiest method is to put your Python functionality within their own TPythonThreads and maintain them from your main thread. Sometimes this is not feasible in certain scenarios, for example:

• you require lots of calls from both main and other threads,
• you need to provide an extra library that uses python and don't want to wrap all of your code in TPythonThreads,* you are wrapping TPythonEngine to provide extra logic, etc.

As TPythonThreads took out the problem of having to handle Python's necessary information for each thread, we must solve that problem in our own code.

Firstly we need to keep track of the interpreter state using a global variable.

var
  g_PyInterpreterState: PPyInterpreterState;

Each of our threads must know its own PyThreadState, just as each TPythonThread does:

threadvar
  u_PyThreadState: PPyThreadState;

We also need to have one and only one call that initialises py4d's global TPythonEngine. This call must be done from the main thread when there is no doubt that only the main thread is executing code! (See void PyEval_InitThreads in the Python/C API threading section for why.)

procedure CreatePythonEngine;
var
  engine: TPythonEngine;
begin
  engine := TPythonEngine.Create(Application);
  // < Your loading and setting calls. >
  engine.Initialize;
  // Share the global interpreter state information.
  g_PyInterpreterState := engine.GetInterpreterState;
  // Set the existing thread state for this thread (i.e. the main thread).
  u_PyThreadState := engine.PyThreadState_Get;
  // Release main thread's lock on the interpreter so other threads can access it.
  PyEval_ReleaseThread(u_PyThreadState);
end;

Whilst the main thread will get its own thread state automatically from the engine when it's created, we still need each of our other threads to get their Python thread state. In each TThread you use, you will need to call a procedure to create a python thread state and then destroy it when you are done. Since you will be doing this in multiple places, make two procedures:

// This must be called during the startup of all threads than need Python,
// apart from the main thread, who gets one automatically during Engine
// creation.
procedure CreateThreadState;
begin
  // The engine and interpreter state information is shared globally.
  u_PyThreadState := PyThreadState_New(g_PyInterpreterState);
end;
 
procedure DestroyThreadState;
begin
  // Clear and delete the thread state.
  PyThreadState_Clear(u_PyThreadState);
  PyThreadState_Delete(u_PyThreadState);
  u_PyThreadState := nil;
end;

The best place to call these procedures is in your TThread's Execute procedure. Once this is done, each thread needs to acquire the global interpreter lock before any python code is executed (including the main thread). Combining this and the thread state create/destroy, your TThread's Execute procedure will look like this:

procedure TUpdateAlertsThread.Execute;
begin
  CreateThreadState;
  try
     PyEval_AcquireThread(u_PyThreadState);
     // < Your Python for Delphi calls. >
     PyEval_ReleaseThread(u_PyThreadState);
  finally
     DestroyThreadState;
  end;
end;

The code given has been very simple. You can make your code more modular by putting all the python thread-handling logic and library calls in one unit and simply ensuring that each library call aquires and releases the lock through AcquireThread/ReleaseThread. As long as you make sure that each TThread creates and destroys its python thread state before and after execution, every thing should work.