機器學習驅(qū)動的基本面研究程序

1、等：學習驅(qū)動的基本面量化投資研究2019 年第 8 期附 錄正文未報告部分程序列表1.數(shù)據(jù)及代碼說明錯誤!未定義書簽。1)數(shù)據(jù)說明：12)代碼說明：12.代碼整理4a)MainFile.py4b)DataTransfrom.py9c)NWttest.py12d)ReturnSeriesTest.py14e)StrategyConstruct.py16f)FactorTest.py27g)DFN.py45h)RNNM.py48i)Ensembleall.py51j)transecfee.py52k)selectFactor.py541等：學習驅(qū)動的基本面量化投資研究2019 年第 8 期數(shù)據(jù)及代

2、碼說明1.1)數(shù)據(jù)說明：a)factor 文件夾中包含初始 96 項因子數(shù)據(jù)（由于雜志社對附件大小的要求，這里僅在factorselect文件夾內(nèi)展示了 16 項因子數(shù)據(jù)，完整的 96 項因子數(shù)據(jù)請通過數(shù)據(jù)庫自行或作者）b)factorselect 文件夾中包含篩選完成的 16 項因子數(shù)據(jù)c)returnseries 文件夾中包含 3/12/24/36滑動窗口下各算法構(gòu)建投資組合月度收益序列d)ff3/ff5 分別為 Fama-French3 因子數(shù)據(jù)，RF 為月度無風險利率數(shù)據(jù)，final_return 為股票月度數(shù)據(jù)e)ff30 為去掉市值最小的 30%股票后分別根據(jù)市值（size)和價格

3、比（EP）分組后構(gòu)建 MKT/SMB/VMG 3 因子數(shù)據(jù)f)factorEW/factorVW 分別為單因子檢驗 10-1組合（等權(quán)重/市值）月度序列2)代碼說明：a)MainFile.py主程序，運行該函數(shù)即可得到各主要結(jié)果b)DataTransfrom.py導 入 基 礎(chǔ) 數(shù) 據(jù) 并 進 行 預 處 理 ， 最 后 將 原 始 數(shù) 據(jù) 轉(zhuǎn) 換 成 每 個 截 面 一 個 stocknumfactornum 的Dataframe,列名為各因子名稱+stock'+'ret'，'stock'為股票代碼，ret為對應(yīng)截面股票月度c)NWttest.py定義

4、對某一序列是否異于 0 進行 Newey and West (1987) t 檢驗函數(shù)d)ReturnSeriesTest.py在獲取各個算法構(gòu)建組合月度序列后， 對各個序列與 OLS 回歸（benchmark）和 DFN（表現(xiàn)最好的深度算法）序列是否顯著差異進行 NW-T 檢驗1等：學習驅(qū)動的基本面量化投資研究2019 年第 8 期e)StrategyConstruct.pyi.投資組合構(gòu)建通用函數(shù)（output),ii.FC 和 ensemble 無內(nèi)置算法包，故單獨構(gòu)建 FC 和 ensemblennf)FactorTest.py因子檢驗補充結(jié)果，內(nèi)容包含：i.在去掉市值最小的 30%股

5、票后分別根據(jù)市值（size)和價格比（EP）分組后構(gòu)建 MKT/SMB/VMG 3 因子ii.單因子 10 分組 10-1/1-10組合因子調(diào)整iii.單因子 10 分組檢驗各組因子調(diào)整iv.各因子與size 因子雙變量分組檢驗結(jié)果v.各因子與 BM 因子雙變量分組檢驗結(jié)果vi.6.96 項因子 fama macbeth 回歸檢驗結(jié)果g)DFN.py深度前饋庫文件，包含深度前饋函數(shù)DFN()，需在 GPU 環(huán)境下運行h)RNNM.py循環(huán)神經(jīng)庫文件，包含如下內(nèi)容：i.基礎(chǔ)循環(huán)神經(jīng)單元：BaseRnn()ii.可用于訓練的循環(huán)神經(jīng)整體架構(gòu)：lstmmodule()，該架構(gòu)基于 BaseRnn()

6、構(gòu)建循環(huán)神經(jīng)。該文件需在 GPU 環(huán)境下運行i)Ensembleall.py集成深度學習模型庫文件，包含集成模型：Ensemblelr()。因其中集成深度學習模型，需在 GPU 環(huán)境下使用，如不集成深度學習模型，則無 GPU限制。j)transecfee.py計算不同成本下的投資組合績效變化， 包含函數(shù) transecfee() 以及showtransecfee()直接調(diào)用showtransecfee()即可2等：學習驅(qū)動的基本面量化投資研究2019 年第 8 期k) selectFactor.py用于篩選重要因子，包含：i.非循環(huán)神經(jīng)模型所用的 dropimportant()函數(shù)ii.循環(huán)神

7、經(jīng)模型所用的dropimportant2()函數(shù)iii.FC 方法的篩選函數(shù) FCselect()函數(shù)注：獲取結(jié)果需要運行的python 文件僅為：MainFile.py 和 FactorTest.py3等：學習驅(qū)動的基本面量化投資研究2019 年第 8 期2. 代碼整理a) MainFile.py1234567891011121314151617181920212223242526272829303132333435363738#!/usr/bin/env python# -*- coding: utf-8 -*-"""description:構(gòu)建學習驅(qū)動多因子

8、投資策略主函數(shù)1.首先輸入各算法參數(shù)（參數(shù)根據(jù)第一個滑動窗口網(wǎng)格調(diào)參確定 此處直接輸入）2.全樣本 3/12/24/36滑動窗口函數(shù)運行,最終直接輸出output'文件夾內(nèi)組合月度FF3/5-alpha,sharperatio并將序列保43.全樣本 3/12/24/36滑動窗口各個算法組合月度序列是否顯著差異NW-T 檢驗4.不同費率下的組合績效結(jié)果5.全樣本 12華東窗口下各個算法的特征篩選6.特征篩選后 16 項因子 12滑動窗口函數(shù)運行,最終輸出組合月度FF3/5-alpha,sharpe ratio 并將序列保output'文件夾內(nèi)注：深度學習算法要在使有GPU 的環(huán)境

9、下進行訓練"""from StrategyConstruct import FC, output, output2, comboutput, ensemblennfrom selectFactor import dropimportant, dropimportant2, FCselectfrom DataTransfrom import datatransfrom, datatransfrom2from xgboost.sklearn import XGBRegressorfrom sklearn.ensemble import GradientBoosting

10、Regressorfrom sklearn.linear_mimport LinearRegression,Lasso,ElasticNet,Ridgefrom sklearn.cross_decomposition import PLSRegressionfrom sklearn.neural_network import MLPRegressorfrom sklearn.svm import SVRimport DFNimport RNNMas rmimport Ensembleall as eaimport warningsfrom mxnet import gpuimport osfr

11、om transecfee import showtrasecfeefrom ReturnSeriesTest import returnseriestestwarnings.filterwarnings('ignore')#各個算法參數(shù)（根據(jù)第一個窗口網(wǎng)格調(diào)參確定）window=3,12,24,36PLS_params=2,2,1,1等：學習驅(qū)動的基本面量化投資研究2019 年第 8 期3940414243444546474849505152535455565758596061626364656667686970717273747576777879805lasso_param

12、s=1e-3,5e-4,0.01,0.01ridge_params=0.1,0.005,0.01,0.005elasticnet_params='alpha':0.01,1e-3,0.01,0.1,'l1_ratio':0.3,0.3,0.7,0.3SVR_params='kernel':'linear','linear','rbf','rbf','gamma':1e-3,1e-3,1e-3,1e-4,'C':0.01,0.001,0.01,1e-4G

13、BDT_params='learning_rate':0.1,0.1,0.1,0.1,'maxdepth':2,3,2,2,'n_estimators':100,100,100,100#XGBOOST 與GBDT 相同 此處共用ENANN_params = 'max_iter': 100, 100, 200, 300, 'p': 0.3, 0.5, 0.7, 0.5DFN_params = 'learning_rate':0.1, 0.1, 0.1, 0.001, 'batch':

14、300, 400, 300, 400LSTM_params = 'learning_rate':1e-4, 1e-5, 1e-4, 1e-6, 'depth': 2, 2, 1, 2,'hidden_number': 256*4RNN_params = 'learning_rate':0.1, 0.1, 0.1, 0.001, 'depth': 1, 1, 2, 1,'hidden_number': 256*4#*2.全樣本 3/12/24/36滑動窗口函數(shù)運行*#path = r'.Dat

15、aBasefactor'#96 項因子所在路徑factorname = x1:-4 for x in os.listdir(path)riskfree, timeseries, factor, timeseries2, index = datatransfrom(path)0,datatransfrom(path)1, datatransfrom(path)2, datatransfrom2(path)0,datatransfrom2(path)1for i in range(4):i= 0output(windowi,LinearRegression(),'OLS'+

16、str(windowi),riskfreei, timeseries)FC(windowi, riskfreei, timeseries, 96,'FC')output(windowi, PLSRegression(PLS_paramsi), 'PLS' + str(windowi), riskfreei,timeseries)output(windowi,Lasso(alpha=lasso_paramsi),'Lasso'+ str(windowi), riskfreei,timeseries)output(windowi,Ridge(alph

17、a=ridge_paramsi),'Ridge'+str(windowi),riskfreei,timeseries)output(windowi,ElasticNpha= elasticnet_params'alpha' i,l1_ratio=elasticnet_params'l1_ratio'i),'ElasticNet'+str(windowi),riskfreei, timeseries)output(windowi,SVR(kernel=SVR_params'kernel'i,gamma= SVR_pa

18、rams 'gamma'i,C=SVR_params 'C'i ),'SVR'+str(windowi),riskfreei, timeseries)output(windowi,GradientBoostingRegressor(n_estimators=GBDT_params'n_estimators'i,max_depth=GBDT_params'maxdepth'i,learning_rate=GBDT_params'learning_rate'i), 'GBDT' +str

19、(windowi),riskfreei, timeseries)output(windowi,XGBRegressor(n_estimators=GBDT_params'n_estimators'i,max_depth=GBDT_params'maxdepth'i等：學習驅(qū)動的基本面量化投資研究2019 年第 8 期818283848586878889909192939495969798991001011021031041051061071081091101111121131141151161171181191201211226, learning_rate=G

20、BDT_params'learning_rate'i), 'XGBOOST' + str(windowi), riskfreei,timeseries)output(windowi, ensemblenn(5,muse = MLPRegressor(solver = 'lbfgs',max_iter=ENANN_params'max_iter'i), pickpercent=ENANN_params'p'i), 'ENANN' +str(windowi), riskfreei, timeseries

21、)output(windowi, DFN.DFN(outputdim=1, neuralset=96, 50, 25, 10, 5, 2, ctx=gpu(0),epoch=10, batch_size=DFN_params'batch'i, lr=DFN_params'learning_rate'i), 'DFN' +str(windowi), riskfreei, timeseries)output2(windowi, rm.lstmmodule(96, LSTM_params'hidden_number'i,LSTM_par

22、ams'depth'i, 100, 3571, lr=LSTM_params'learning_rate'i), 'LSTM'+str(windowi) ,riskfreei, timeseries2)output2(windowi, rm.lstmmodule(96, RNN_params'hidden_number'i,RNN_params'depth'i, 100, 3571, lr=RNN_params'learning_rate'i, ntype='RNN'), '

23、RNN'+str(windowi), riskfreei, timeseries2)mlist = DFN.DFN(outputdim=1, neuralset=96, 50, 25, 10, 5, 2, ctx=gpu(0),epoch=10, batch_size=DFN_params'batch'i, lr=DFN_params'learning_rate'i),ensemblenn(5,muse = MLPRegressor(solver = 'lbfgs',max_iter=ENANN_params'max_iter&#

24、39;i), pickpercent=ENANN_params'p'i),XGBRegressor(n_estimators=GBDT_params'n_estimators'i,max_depth=GBDT_params'maxdepth'i, learning_rate=GBDT_params'learning_rate'i),GradientBoostingRegressor(n_estimators=GBDT_params'n_estimators'i,max_depth=GBDT_params'm

25、axdepth'i,learning_rate=GBDT_params'learning_rate'i),PLSRegression(PLS_paramsi),Ridge(alpha=ridge_paramsi),SVR(kernel=SVR_params'kernel'i,gamma= SVR_params 'gamma'i,C=SVR_params 'C'i)# PLS 一定要放在倒數(shù)第三個（PLS 輸出形式為list 故進行了進一步處理）nmolist = rm.lstmmodule(96, LSTM_params&

26、#39;hidden_number'i, LSTM_params'depth'i,100, 3571, lr=LSTM_params'learning_rate'i),rm.lstmmodule(96, RNN_params'hidden_number'i, RNN_params'depth'i,100, 3571, lr=RNN_params'learning_rate'i, ntype='RNN')# 循環(huán)神經(jīng)模型mname = 'DFN', 'En-ann

27、9;, 'xgboost', 'GBDT', 'lasso', 'Elasticnet', 'pls', 'Ridge','svm', 'LSTM', 'RNN'ensemblem= ea.Ensemblelr(mlist, nmolist, mname)comboutput(windowi,ensemblem, 'Ensemble'+str(windowi),riskfreei, timeseries2,index)#*3.各算法序列

28、差異NW-t 檢驗*#for i in window:returnseriestest(i)等：學習驅(qū)動的基本面量化投資研究2019 年第 8 期1231241251261271281291301311321331341351361371381391401411421431441451461471481491501511521531541551561571581591601611621631647#*4.不同費率情形*#showtrasecfee(0.005)showtrasecfee(0.0075)showtrasecfee(0.01)#*5.全樣本 12特征篩選過程*#i = 1#選取 1

29、2滑動窗口篩選因子dropimportant(windowi ,LinearRegression(), 'OLS'+str(windowi), factorname,timeseries,0.0201)FCselect(factorname, timeseries)dropimportant(windowi, PLSRegression(PLS_paramsi), 'PLS', factorname, timeseries,0.0230)dropimportant(windowi, Lasso(alpha=lasso_paramsi), 'Lasso&#

30、39;, factorname, timeseries,0.0208)dropimportant(windowi, Ridge(alpha=ridge_paramsi), 'Ridge', factorname, timeseries,0.0208)dropimportant(windowi, ElasticNpha= elasticnet_params'alpha' i,l1_ratio=elasticnet_params'l1_ratio'i), 'ElasticNet', factorname, timeseries, 0.

31、0212)dropimportant(windowi, SVR(kernel=SVR_params'kernel'i,gamma= SVR_params'gamma'i,C= SVR_params 'C'i ), 'SVR', factorname, timeseries, 0.0225)dropimportant(windowi,GradientBoostingRegressor(n_estimators=GBDT_params'n_estimators'i,max_depth=GBDT_params'm

32、axdepth'i,learning_rate=GBDT_params'learning_rate'i), 'GBDT', factorname,timeseries, 0.0268)dropimportant(windowi,XGBRegressor(n_estimators=GBDT_params'n_estimators'i,max_depth=GBDT_params'maxdepth'i, learning_rate=GBDT_params'learning_rate'i), 'XGBOOS

33、T',factorname, timeseries, 0.0273)dropimportant(windowi, ensemblenn(5,muse = MLPRegressor(solver = 'lbfgs',max_iter=ENANN_params'max_iter'i), pickpercent=ENANN_params'p'i), 'ENANN',factorname, timeseries, 0.0234)dropimportant(windowi, DFN.DFN(outputdim=1, neuralse

34、t=96, 50, 25, 10, 5, 2, ctx=gpu(0),epoch=10, batch_size=DFN_params'batch'i, lr=DFN_params'learning_rate'i), 'DFN',factorname, timeseries, 0.0278)dropimportant2(windowi, rm.lstmmodule(95, LSTM_params'hidden_number'i,LSTM_params'depth'i, 100, 3571, lr=LSTM_param

35、s'learning_rate'i), 'LSTM', factorname,timeseries2, 0.0257)dropimportant2(windowi, rm.lstmmodule(95, RNN_params'hidden_number'i,等：學習驅(qū)動的基本面量化投資研究2019 年第 8 期1651661671681691701711721731741751761771781791801811821831841851861871881891901911921931941951961971981992002012022032042

36、052068RNN_params'depth'i, 100, 3571, lr=RNN_params'learning_rate'i, ntype='RNN'), 'RNN',factorname, timeseries2, 0.0210)#*6.特征篩選后 16 項因子 12滑動窗口函數(shù)運行*#path = r'.DataBasefactorselect'#經(jīng)過篩選后因子集合所在路徑riskfree,timeseries,factor,timeseries2=datatransfrom(path)0,datatr

37、ansfrom(path)1,datatransfrom(path)2,datatransfrom2(path,after=True)0i=1 #選取 12滑動窗口測試篩選后因子集合績效表現(xiàn)output(windowi,LinearRegression(),'OLS'+str(windowi),riskfreei, timeseries)FC(windowi, riskfreei, timeseries, 11,'FC')output(windowi, PLSRegression(PLS_paramsi), 'PLS' + str(windowi

38、), riskfreei,timeseries)output(windowi,Lasso(alpha=lasso_paramsi),'Lasso'+ str(windowi), riskfreei,timeseries)output(windowi,Ridge(alpha=ridge_paramsi),'Ridge'+str(windowi),riskfreei,timeseries)output(windowi,ElasticNpha= elasticnet_params'alpha' i,l1_ratio=elasticnet_params&

39、#39;l1_ratio'i),'ElasticNet'+str(windowi),riskfreei, timeseries)output(windowi,SVR(kernel=SVR_params'kernel'i,gamma= SVR_params 'gamma'i,C=SVR_params 'C'i ),'SVR'+str(windowi),riskfreei, timeseries)output(windowi,GradientBoostingRegressor(n_estimators=GBDT

40、_params'n_estimators'i,max_depth=GBDT_params'maxdepth'i,learning_rate=GBDT_params'learning_rate'i), 'GBDT' +str(windowi),riskfreei, timeseries)output(windowi,XGBRegressor(n_estimators=GBDT_params'n_estimators'i,max_depth=GBDT_params'maxdepth'i, learnin

41、g_rate=GBDT_params'learning_rate'i), 'XGBOOST' + str(windowi), riskfreei,timeseries)output(windowi, ensemblenn(5,muse = MLPRegressor(solver = 'lbfgs',max_iter=ENANN_params'max_iter'i), pickpercent=ENANN_params'p'i), 'ENANN' +str(windowi), riskfreei, ti

42、meseries)output(windowi, DFN.DFN(outputdim=1, neuralset=16, 50, 25, 10, 5, 2, ctx=gpu(0),epoch=10, batch_size=DFN_params'batch'i, lr=DFN_params'learning_rate'i), 'DFN' +str(windowi), riskfreei, timeseries)output2(windowi, rm.lstmmodule(11, LSTM_params'hidden_number'i,

43、LSTM_params'depth'i, 100, 3571, lr=LSTM_params'learning_rate'i), 'LSTM'+str(windowi) ,riskfreei, timeseries2)output2(windowi, rm.lstmmodule(11, RNN_params'hidden_number'i,RNN_params'depth'i, 100, 3571, lr=RNN_params'learning_rate'i, ntype='RNN'

44、), 'RNN'+str(windowi), riskfreei, timeseries2)等：學習驅(qū)動的基本面量化投資研究2019 年第 8 期b)DataTransfrom.py123456789101112131415161718192021222324252627282930313233343536373839409#!/usr/bin/env python# -*- coding: utf-8 -*-"""description:導入基礎(chǔ)數(shù)據(jù)并進行一些預處理 最后變成每個截面一個Dataframe,列名為各因子名稱+stock'+

45、'ret' index 代表單只股票"""import glob,osimport pandas as pdimport warnings#*1.導入因子數(shù)據(jù) 無風險利率 股票月度數(shù)據(jù)*#warnings.filterwarnings('ignore')def datatransfrom(datapath):path=datapathfile = glob.glob(os.path.join(path, "*.csv")k=for i in range(len(file):k.append(pd.read_csv

46、(filei)#股票月度ret=pd.read_csv('.DataBasefinal_return.csv')#無風險利率rf=pd.read_csv('.DataBaseRF.csv')rf3=rf.iloc3:-1,:rf12=rf.iloc12:-1,:rf24=rf.iloc24:-1,:rf36=rf.iloc36:-1,:riskfree = rf3, rf12, rf24, rf36#因子名稱factor=for i in range(len(file):factor.append(filei20:-4)factor.append('st

47、ock')#*對原始數(shù)據(jù)進行預處理 每個截面一個Dataframe,列名為96 因子名稱+stock'+'ret'index 代表單只股票*#timeseries=for i in range(len(ret.columns)-1):等：學習驅(qū)動的基本面量化投資研究2019 年第 8 期414243444546474849505152535455565758596061626364656667686970717273747576777879808182# 加入月度令月度不為null方便函數(shù)處理for i in range(len(timeseries2):10k

48、l=pd.concat(kj.iloc:,i+1 for j in range(len(file),axis=1)kl'stock' = ret.iloc:,0kl.columns = factorkl=kl.iloc:-2,:timeseries.append(kl)#刪除月度不的數(shù)據(jù)條for i in range(len(timeseries):timeseriesi'ret'=ret.iloc:,i+1timeseriesi'ret'=timeseriesi'ret'.fillna('null')timese

49、riesi=timeseriesitimeseriesi'ret'.isin('null')return riskfree,timeseries,factor# 為LSTMRNN 設(shè)計的數(shù)據(jù)函數(shù)def datatransfrom2(datapath, after=False):path=datapathfile = glob.glob(os.path.join(path, "*.csv")k=for i in range(len(file):k.append(pd.read_csv(filei)#股票月度ret=pd.read_csv(

50、9;.DataBasefinal_return.csv')#因子名稱factor=for i in range(len(file):factor.append(filei20:-4)factor.append('stock')#*對原始數(shù)據(jù)進行預處理 每個截面一個Dataframe,列名為96 因子名稱+stock'+'ret'index 代表單只股票*#timeseries2=index = for i in range(len(ret.columns)-1):kl=pd.concat(kj.iloc:,i+1 for j in range(l

51、en(file),axis=1)kl'stock' = ret.iloc:,0kl.columns = factorif after:# 保證篩選后因子個數(shù)為 3571 個kl = kl.iloc:, :else:kl = kl.iloc:-2,:timeseries2.append(kl)等：學習驅(qū)動的基本面量化投資研究2019 年第 8 期8384858611timeseries2i'ret' = ret.iloc:, i + 1timeseries2i'ret' = timeseries2i'ret'.fillna('

52、;null')index.append(timeseries2i'ret'.isin('null')return timeseries2, index等：學習驅(qū)動的基本面量化投資研究2019 年第 8 期c)NWttest.py12345678910111213141516171819202122232425262728293031323334353637383940414212# -*- coding:utf-8 -*-'''description:NW-t 檢驗所用包'''import numpy as

53、 npfrom collections import namedtuplefrom scipy.stats import distributionsdef _ttest_finish(df, t):''':param df:自由度:param t: t 值:return: 輸出 t 和對應(yīng)p 值'''prob = distributions.t.sf(np.abs(t), df) * 2 # use np.abs to get upper tailif t.ndim = 0:t = t()return t, probNWt_1sampleResu

54、lt = namedtuple('NWT_1sampResult', ('statistic', 'pvalue')def nwttest_1samp(a, popmean, axis=0,L=1):'''主函數(shù):param a: 數(shù)據(jù)列表:param popmean: 原假設(shè)值u0:param axis: 行還是列 默認行:param L: lag滯后多少 默認 1:return: 輸出 nw-t 和對應(yīng)p 值'''a = np.array(a)N = len(a)df = N-1e = a - np.mean(a)residuals = np.sum(e*2)Q = 0for i in range(L):w_l = 1 - (i+1)/(1+L)for j in range(1,N):Q += w_l*ej*ej-(i+1)S = residuals + 2*Q等：學習驅(qū)動的基本面量化投資研究2019 年第 8 期434445464748495013nw_var = S/Nd = np.m