Optimal Portfolios: Why and How?Vivek PalaniappanBlockedUnblockFollowFollowingFeb 3Portfolio optimization is a crucial part of managing risk and maximizing returns from a set of investments.

Be it for the fundamental investor, or the quantitative trader, portfolio optimization is imperative for a successful portfolio.

This article delves to explain how portfolio optimization works and how to use OptimalPortfolio library to implement it.

Mean-Variance OptimizationThe usual framework for optimizing portfolios was developed by Markowitz, and it is also known as the mean-variance optimization.

This is due to the fact that the mean and the variance of the portfolios are the ones that are considered in finding the portfolio that maximizes the returns or minimizes risk, both of which will be defined later.

The general methodology for the mean-variance optimization, as discussed by Attilio Meucci, is as follows:Determine market invariants.

Estimate distribution of market invariants.

Estimate the mean and covariance of the portfolio of assets.

Find optimal weights for given objective function.

Market invariants are quantities that do not evolve over time.

For stocks, the market invariants are the log returns.

For an exploration of the issue of market invariance, do take a look at the following article.

Market invariants for assets other than stocks are much harder to determine and acquire.

For example, for options, the market invariant is the difference in the rolling implied volatility of the option.

This additional computation required makes acquiring the data harder and it also poses an issue in step 3.

Once we have the data for the market invariants, we can perform statistical estimation to find out the distribution and its parameters that best suits the data we have.

This is done usually by maximum likelihood estimation or shrinkage methods.

The estimation of the distribution of the market invariants is where most of the work and research is focused on.

The issue with estimating the distribution is the amount of data available to fit the distribution to.

For example, if the amount of data we have is infinite, then a simple nonparametric estimation would be the best and would in fact give the exact population mean and covariance.

However, in real life cases, the amount of data we have available is much less than ideal.

Sometimes this might be hard to believe as we are always told that we generate more data than we can analyse and hence the amount of data should not be the limiting factor.

However, high quality data relevant to the market invariant that can be used is a small portion of the data generated.

Furthermore, the curse of dimensionality makes the data requirement for portfolios of several hundred assets exponentially high.

A way to get around this constraint is to develop novel methods of estimating the distribution of the market invariants.

The estimator of the distribution must be robust to small changes in data and must provide an accurate estimate of the moments of the population distribution given the sample data.

Once we have an estimate of the distribution of the market invariants, we need to find the mean and covariance of the portfolio of assets.

In the case of the a portfolio with just stocks, it is trivial as the distribution of the market invariants already give us the mean and covariance of the portfolio.

However, for cases with derivatives and fixed income securities, we need to find the inverse mapping from the market invariants to the portfolio value and hence find the distribution of that using the distribution of our market invariants.

In those cases, it is useful to expand the inverse mapping in terms of the generating function of the distribution of the market invariants and then find a suitable approximation to the degree of accuracy one requires.

Now all we have to do is find the optimal weights for the portfolio that maximizes some utility function that takes into account the estimates means and covariances.

There are a few schools of thought.

For mean-variance optimization, the Sharpe ratio, which is essentially the mean divided by the standard deviation, is maximized.

For my library, I implement a novel utility function that takes into account higher order moments such as skew and kurtosis as they give useful information about the portfolio behavior.

How to Use OptimalPortfolioNow that we know the structure of how portfolio optimization works, we can delve into how to actual use the library.

The library is built such that every step of the portfolio optimization process can be called independently and sequentially if required.

As such, they are split into different files, each with a class for the particular step.

The methods of the class are the different ways to achieve the desired result of the step.

Hence, in order to use the library you will have to call the various classes and methods according to your needs.

For example, given that you have your market invariant data, and would like to estimate the distribution of the market invariants, you can instantiate the MLE class from the moment_est.

py file.

The class itself is arranged as follows:class MLE: """ Provide methods to calculate maximum likelihood estimators (MLE) of mean, covariance and higher moments.

Currently implemented distributions: – Normal – Student-t Instance variables: – “invariants“ (market invariants data) – “dist“ (distribution choice) – “n“ (number of assets) – “mean“ (estimate of mean, initially None) – “cov“ (estimate of covariance, initially None) – “skew“ (estimate of skew, initially None) – “kurt“ (estimate of kurtosis, initially None) Public methods: – “norm_est“ (calculates the normally distributed maximum likelihood estimate of mean, covariance, skew and kurtosis) – “st_est“ (calculates the student-t distributed maximum likelihood estimate of mean, covariance, skew and kurtosis) """ def __init__(self, invariants, n, dist="normal"): """ :param invariants: sample data of market invariants :type invariants: pd.

Dataframe :param n: number of assets :type n: int :param dist: choice of distribution: "normal" :type dist: str """ self.

invariants = invariants self.

dist = dist self.

n = n self.

mean = None self.

cov = None self.

skew = None self.

kurt = None def norm_est(self): """ Calculates MLE estimate of mean, covariance, skew and kurtosis, assuming normal distribution :return: dataframes of mean, covariance, skew and kurtosis :rtype: pd.

Dataframe """ if self.

dist == "normal": self.

mean = 1/self.

n * np.

sum(self.

invariants) self.

cov = 1/self.

n * np.

dot((self.

invariants – self.

mean), np.

transpose(self.

invariants – self.

mean)) self.

skew = 0 self.

kurt = 0 return self.

mean, self.

cov, self.

skew, self.

kurt def st_est(self): """ Calculates MLE estimate of mean, covariance, skew and kurtosis, assuming student-t distribution :return: dataframe of mean, covariance, skew and kurtosis :rtype: pd.

Dataframe """ if self.

dist == "student-t": self.

mean, self.

cov = expectation_max(self.

invariants, max_iter=1000) self.

skew = 0 self.

kurt = 6Within the class, you can choose your distribution and call the method accordingly.

The output would be the mean, covariance, skew and kurtosis of the market invariant data you provide.

Another example would be if you wanted to optimize the allocation within a portfolio.

You would then call the OptimalAllocations class from the opt_allocations.

py file and then call the methods according to which optimization you wish to perform.

The class is made as such:class OptimalAllocations: def __init__(self, n, mean, cov, tickers, weight_bounds=(0, 1)): """ :param n: number of assets :type n: int :param mean: mean estimate of market invariants :type mean: pd.

Dataframe :param cov: covariance estimate of market invariants :type cov: pd.

Dataframe :param tickers: tickers of securities used :type tickers: list :param weight_bounds: bounds for portfolio weights.

:type weight_bounds: tuple """ self.

n = n self.

mean = mean self.

cov = cov self.

weight_bounds = (weight_bounds,)*self.

n self.

x0 = np.

array([1 / self.

n] * self.

n) self.

constraints = [{"type": "eq", "fun": lambda x: np.

sum(x) – 1}] self.

tickers = tickers self.

weights = None self.

skew = None self.

kurt = None def moment_optimisation(self, skew, kurt, delta1, delta2, delta3, delta4): """ Calculates the optimal portfolio weights for utility functions that uses mean, covariance, skew and kurtosis of market invariants.

:param skew: skew of market invariants :param kurt: kurtosis of market invariants :param delta1: coefficient of mean, (i.

e how much weight to give maximising mean) :param delta2: coefficient of covariance, (i.

e how much weight to give minimising covariance) :param delta3: coefficient of skew, (i.

e how much weight to give maximising skew) :param delta4: coefficient of kurtosis, (i.

e how much weight to give minimising kurtosis) :return: dictionary of tickers and weights """ self.

skew = skew self.

kurt = kurt args = (self.

mean, self.

cov, skew, kurt, delta1, delta2, delta3, delta4) result = minimize(utility_functions.

moment_utility, x0=self.

x0, args=args, method="SLSQP", bounds=self.

weight_bounds, constraints=self.

constraints) self.

weights = result["x"] return dict(zip(self.

tickers, self.

weights)) def sharpe_opt(self, risk_free_rate=0.

02): """ Maximise the Sharpe Ratio.

:param risk_free_rate: risk-free rate of borrowing/lending, defaults to 0.

02 :type risk_free_rate: float, optional :raises ValueError: if “risk_free_rate“ is non-numeric :return: asset weights for the Sharpe-maximising portfolio :rtype: dict """ args = (self.

mean, self.

cov, risk_free_rate) result = minimize(utility_functions.

sharpe, x0=self.

x0, args=args, method="SLSQP", bounds=self.

weight_bounds, constraints=self.

constraints) self.

weights = result["x"] return dict(zip(self.

tickers, self.

weights)) def portfolio_metrics(self, verbose=False, risk_free_rate=0.

02): """ After optimising, calculate (and optionally print) the return, volatility and Sharpe Ratio of the portfolio.

:param risk_free_rate: risk-free rate of borrowing/lending, defaults to 0.

02 :type risk_free_rate: float, optional :return: expected return, volatility, Sharpe ratio.

:rtype: (float, float, float) """ sigma = np.

sqrt(utility_functions.

volatility( self.

weights, self.

cov)) mu = self.

weights.

dot(self.

mean) sharpe = -utility_functions.

sharpe(self.

weights, self.

mean, self.

cov, risk_free_rate) print(f"Expected annual return: {100*mu}") print(f"Annual volatility: {100*sigma}") print(f"Sharpe Ratio: {sharpe}") return mu, sigma, sharpeConclusionPortfolio optimization is a difficult yet necessary process and the library OptimalPortfolio, I believe, makes the process clear and effective.

I have planned out the following roadmap to the development of the library:Calculating invariants for bonds and derivativesExponentially weighted skew and kurtosisUsing stable distributions for MLEOptimal choosing of shrinkage for Nonparametric+MLE shrinkageShrinkage for higher momentsCopula based CVaR optimisationMonte Carlo simulationsI hope that you have enjoyed this article and do check out OptimalPortfolio on GitHub.

.