Modeling Nonlinear Stochastic Kinetic System and Stochastic Optimal Control of Microbial Bioconversion Process in Batch Culture *

In this paper, we analyze a stochastic model representing batch fermentation in the process of glycerol bio-dissimilation to 1,3-propanediol by klebsiella pneumoniae. The stochasticity in the model is introduced by parameter perturbation which is a standard technique in stochastic population modelling. Thus, based on the nonlinear deterministic dynamical system of glycerol bioconversion to 1,3-propanediol in batch culture, we present the stochastic version of the batch fermentation process driven by a five-dimensional Brownian motion and Lipschitz coefficients, which is suitable for the factual fermentation. Subsequently, we study the existence and uniqueness of solutions for the stochastic system as well as the boundedness and Markov property of solutions. Moveover a stochastic optimal control model is constructed and the sufficient and necessary conditions for optimality are proved via dynamic programming principle. Finally we present computer simulation for the stochastic system by using Stochastic Euler–Maruyama scheme. Compared with the results from the deterministic system, numerical results reveal the peculiar role of stochasticity in the dynamical responses of the batch culture.


Introduction
Over the past several years, 1,3-propanediol (1,3-PD) has attracted much attention in microbial production throughout the world because of its lower cost, higher production and no pollution [1,2].Many researches have been carried out including the quantitative description of the cell growth kinetics of multiple inhibitions, the metabolic overflow kinetics of substrate consumption and product formation [3][4][5], existence of equilibrium points and stability [6], transport mechanism [7] and impulsive control [8] for the models of the continuous cultures, feeding strategy of glycerol [9,10], optimal control [11,12] and multistage modeling [13,14] in fed-batch culture and so on.
Compared with continuous and fed-batch cultures, glycerol fermentation in batch culture can obtain the highest production concentration and molar yield 1,3-PD to glycerol [15].So nonlinear dynamical systems in this culture have been extensively considered in recent years [16,17].The typical cell growth in batch culture includes several growth phases, which are defined as the lag, exponential growth, decreased growth and death phases.Modelling, parameter identification and optimal control of the multi-stage dynamic system in batch culture are discussed in [18][19][20] However, the dynamics of the system are not deterministic, but intrinsically stochastic, and consideration of inherent stochasticity of microorganism is necessary to uncover the precise nature of stochastic differential equation governing the system dynamics [21].In this paper, the stochasticity in the model is introduced by parameter perturbation which is a standard technique in stochastic population modelling [22].The process is modeled by a stochastic ordinary differential system driven by five dimensional Brownian motion, which is time independent and suitable for the factual fermentation.Suitable conditions on the coefficients of the stochastic system are proposed to assure the existence and uniqueness of solution of the stochastic system.Furthermore, based on the theory of stochastic integration and stochastic differential equations, several important properties of the solution of the stochastic system are proved, including boundedness and Markov property.
Stochastic control is a subfield of control theory which deals with the existence of uncertainty.Stochastic control aims to design the optimal controller that performs the desired control task.In this paper, we study the stochastic optimal control problem of the stochastic system, the volumetric productivity of 1,3-PD and dilution rate are used as the optimization target and manipulated variable, respectively.Our main concern is to derive some tractable characteristics of the value function and optimal control.This article is intended to prove the sufficient and necessary conditions of optimal solution and that the optimal solution depends in a continuous way on the parameters (perturbations).Finally stochastic simulation is carried out under the Stochastic Euler-Maruyama scheme.Numerical examples confirm that the proposed stochastic system is more suitable to formulate the dynamics of batch culture.This paper is organized as follows.In Section 2, we present a nonlinear stochastic kinetic system of batch fermentation process.In Section 3, we prove the properties of the stochastic dynamic system as well as the existence and uniqueness of solutions to the stochastic dynamic system.Furthermore the boundedness and Markov property of solutions to the system are discussed.Section 4 gives the key results on the characterization of optimality.Numerical examples are provided to simulate the nonlinear stochastic dynamical system of batch culture in Section 5.In Section 6, we draw the conclusions and trace the direction for future works.
2 Modeling nonlinear stochastic kinetic system of batch culture Mass balances of biomass, substrate and products in batch microbial culture are written as follows (see [16]): where the specific growth rate of cells µ, specific consumption rate of substrate q 2 and specific formation rate of product q i are expressed by Eqs. ( 2)-( 5), respectively.
where x 1 (t), x 2 (t), x 3 (t), x 4 (t), x 5 (t) are the concentration of biomass, glycerol, 1,3-PD, acetic acid and ethanol at time t in reactor, respectively.x 0 ∈ R 5 + denotes the initial state.Under anaerobic conditions at 37 • C and pH = 7.0, µ m is the maximal specific growth rate of cells, and k s is Monod saturation constant.The critical concentrations of biomass, glycerol, 1,3-PD, acetic acid and ethanol for cell growth are x * 1 = 10 g/L, x * 2 = 2039 mmol/L, x * 3 = 939.5 mmol/L, x * 4 = 1026 mmol/L and x * 5 = 360.9mmol/L, respectively.T ∈ (0, +∞) is the terminal time of batch fermentation.As a result of fact, the following assumption can be made: Assumption 1. Medium is adequately intermixed.No medium is pumped inside and outside the bio-reactor in the process of batch fermentation.
In this paper, let I = [0, T ] be the time interval of batch fermentation, we choose a probability space (R 5 , B(R 5 ), P ), as well as a 5-dimensional vector Brownian motion W = {W t (•), F W t : t ∈ I} defined on the probability space, where the denotes the σ-algebra generated by {W s (•): 0 s t}, i.e., the smallest σ-algebra containing the family of {W s (•): 0 s t}.We also assume that this space is rich enough to accommodate a random vector ξ taking values in R 5 , independent of 3,4,5) denote the scalar stochastic process on biomass, glycerol, 1,3-PD, acetic acid and ethanol on I, respectively.The state variable corresponding to the stochastic process X is , 5} be the state domain of the stochastic process X = {X(t): t ∈ I}.
Equation ( 1) can be rewritten in the matrix form where A = (a ij ) 5×5 and from Eq. ( 1) we can see that Now let us stochastically perturb each parameter a ij as follows: where Ẇ j t is a Gaussian white noise, σ = (σ ij ) 5×5 satisfies the following condition: Thus, under Assumption 1, the course of the batch culture with uncertain perturbations can be formulated as the following nonlinear stochastic dynamical system: Here: (i) The drift vector is continuous on S 0 .
Here X ∈ R 5 and σ is the given R 5×5 diffusion matrix.
(iii) Let W = (W 1 , . . ., W 5 ) T ∈ R 5 , where W i = {W i t (•): t ∈ I} denotes one-dimensional Brownian motion defined on the given probability space (R 5 , B(R 5 ), P ) and adapted to F t in B(R 5 ) and they are independent of each other.
3 Properties of solutions to stochastic system Firstly, let's review some basic concepts about the stochastic dynamic system.Most of them can be found in [25].
for all open sets U ∈ R n (or, equivalently, for all Borel sets U ⊂ R n ).
Definition 3. Suppose that F (x) and σ(x) are given by Eqs. ( 7)-( 8).Then, stochastic process X is said to be a solution of the stochastic differential equation ( 6) on the probability space (Ω, F, P ) with respect to the Brownian motion W and initial condition ξ provided: (i) X(t) is adapted to the filtration F t of (5), i.e., X is a mapping: (iii) with probability one, the solution X of (6) and the Brownian motion W satisfy Definition 4. If X 1 (t) and X 2 (t) are arbitrary two solutions of ( 6) with respect to W with the same initial condition ξ, and Then a solution X(t) of ( 6) is called unique.
Let E = C(I, R 5 ) be the space of all continuous functions x defined on I with values in R 5 , equipped with the max norm topology x E = max t∈I x(t) .For where • denotes the Euclidean norm in R 5 and ∧ means min{max 0 t k x 1 (t) − x 2 (t) , 1}.It is well known that E is a complete separable metric space with respect to this metric ρ.
Proof.It is clear from the continuity of the function F (x) and σ(x) on I.
Theorem 2. For the vector-valued function F (x) and the matrix-valued function σ(x) defined in (7) and (8), there exist positive constants K and K such that, for t ∈ I, the following conditions hold: Proof.We begin with the uniform Lipschitz condition.By Quasi-differential Mean Value theorem, we now obtain where θ ∈ (0, 1) and JF (x 1 +θ(x 2 −x 1 )) denotes Jacobian of F at x 1 +θ(x 2 −x 1 ).Since F (x) is differentiable on S 0 and S 0 is a compact set, so Let M = max x∈S0 JF (x 1 + θ(x 2 − x 1 )) , we can rewrite the above inequality and by the definition of the function σ and let L = 25 max 1 i,j 5 {σ ij }, we have Thus it follows from ( 10) and ( 11) that Let K = M + L, then we have the following equality: Next, we will show the growth condition of the function F and σ.Note that, because of (8), we see that From Eqs. ( 2)-( 5), we can conclude that there exist positive constants letting C = max i∈I5 {|C i |}, referring to (12), we see that It is clear from the definition of the function σ Therefore, letting K = L + C, we can complete the proof by Based on the Theorem 5.2.1 in [25] and Theorem 2, we can prove the following theorem.
Theorem 3 (Existence and uniqueness).Given the vector-valued function F (x) and the matrix valued function σ(x) defined by (7) and (8), the system (6) has a unique solution X(t) satisfying the initial condition ξ on I.
According to the proof in Theorem 2, Theorem 7.1.2in [25] and Theorem 5.2 in [26], we can prove the following theorem.
Theorem 4 (Markov property and boundedness).Suppose Assumption 1 holds.The unique solution X(t) is a Markov process on the interval I whose initial probability distribution at t = 0 is the distribution of ξ and X(t) has continuous paths.Moreover where constant B depends only on K, σ and T .
Define the solution set of system (6) relative to U ad , i.e., The aim of the microbial fermentation in batch culture is to maximize the yield of the 1,3-PD, so we establish the stochastic optimal control model of the batch culture as follows: From the theory on continuous dependence of solutions on parameters, we know that X(s, u) is continuous on u, so J(u) is continuous on u ∈ U ad .Moreover U ad is a closed bounded convex subset of R 2 + .Hence we know the optimal control must exist by Theorem V.6.3 in [27], namely, exists u * ∈ U ad such that J(u * ) J(u) for all u ∈ U ad .
For any time s ∈ [0, τ ], define the value function and the operator L u X (s) takes the form Theorem 5. Assume that V (s, X) is a solution of the dynamic programming equation If u * is an admissible feedback control, then u * is optimal if and only if Let us replace s, X, v by t, X(t), u(t) = u(t, X(t)), s t τ .We apply Theorem V.5.1 in [27] with We have equality in (14) for u = u * .Therefore, V (s, X) = J(u * ) using Theorem V.5.2 in [27].Thus u * is optimal.
Necessity.Applying the principle of optimality in dynamic programming we get Multiplying h −1 on both sides of above formula and letting h → 0 + , we noticed that X is controlled by Itô differential equation ( 6), we can deduce by Itô differential formula On the other hand, assume the optimal control u * can be achieved on [s, s + h], then From ( 15) and ( 16), we get Thus the proof is completed.

Numerical simulation
To illustrate the stochastic nature of batch fermentation process sufficiently, a numerical example is given.In the example, we let σ ii = 0.02, σ ij = 0.0004, i = j, and use Monte Carlo method to generate five thousand random inputs, which consist of the infinitesimal increment of standard Brownian motion dW (t).Afterwards, we solve the proposed stochastic model using the following Stochastic Euler-Maruyama scheme and obtain five thousand solution paths of the model.Our numerical approximation to X(τ j ) will be denoted by X j .Stochastic Euler-Maruyama method [28]: where ∆t = T /L, for some positive integer L, X k , denotes the kth component of the X(t) and τ j = j∆t.ξ = (0.405 g/L, 441.37 mmol/L, 0, 0, 0) T , the components of which are the initial concentrations of biomass, substrate, 1,3-PD, acetic acid and ethanol, respectively.All the parameters of the stochastic system are given in Table 1.In this simulation, we let T = 6.5 h and L = 1000 in Eq. ( 17), it had been shown that EM has strong order of convergence γ = 1/2 [28], i.e., the error between the true solution and the numerical solution is a constant multiply ∆t 1/2 .Figures 1-5 shows the comparison of biomass, substrate and product concentrations between experimental and simulated results, where the points denote the experimental values, written as y(τ i ) = (y 1 (τ i ), y 2 (τ i ), y 3 (τ i ), y 4 (τ i ), y 5 (τ i )) T , i ∈ I 5 , and the real lines denote the computational curves EX k (t), k ∈ I 5 .We obtain the errors e 1 = 18.39%, e 2 = 24.19%,e 3 = 26.68%,e 4 = 67.25%,e 5 = 29.97%.The large errors e 4 and e 5 might due to the intermittent feeding of alkali into the reactor to maintain the pH value at 7. Comparing the errors in this paper with the reported results [16] and the stochastic nature of the bioprocess, we conclude that the stochastic system is more fit for modeling actual batch fermentation under investigation.

Conclusions
In this paper we have proposed a nonlinear stochastic kinetic system of batch culture.Then we proved the existence and uniqueness of solutions to the stochastic system and the stochastic optimal control of the nonlinear stochastic system.Our current tasks accommodate the stochastic modeling and some properties of the nonlinear stochastic system as well as the stochastic optimal control.In a future work, the objective of our efforts is to develop into the parameter estimation and numerical result of the stochastic optimal control problem for the stochastic system of batch culture.Further we will pursue the verification and validation of the proposed stochastic system and make detailed comparison between deterministic and stochastic models of batch culture.

Table 1 .
Parameters values of each reactant in the stochastic system.