版權說明:本文檔由用戶提供并上傳,收益歸屬內容提供方,若內容存在侵權,請進行舉報或認領
文檔簡介
1、VMware Integrated Openstack解決方案劉承罡Continued enhancements based on user demandInvestment Protection: VMware Integrated Openstack for Telco2Production ServicesvCloud Director APIOpenStack MaturationProofs of Concept, Test DeploymentsCommercial ScaleTelco GradeLive Trial ServicesOpen Container APIvClou
2、d NFV PlatformvRealizeTM OperationsVirtual SANNSXTMvSphereCloud Foundry APIvCloud APIVIO APIContainersvRealize Orchestrator 優(yōu)勢與主要用途3PlatformPlug-InsEcosystem 優(yōu)點快速集成VMware 解決方案到現(xiàn)有IT環(huán)境和流程中減少VMware 解決方案與第三方解決方案與系統(tǒng)的交互成本自動化云計算加速實現(xiàn)IT as a Service (ITaaS)主要用途為自動化: 云計算管理任務突發(fā)事件處理普通定時IT管理任務Workflow開發(fā)重用已有的內容Wo
3、rkflowsActions Resource 元素預制 “scriptable” 任務超過500 個可重用的actions和workflows 更多第三方組件plug-ins4客戶端vRealize Orchestrator 客戶端 用于開發(fā) workflows 的主要工具5版本更新CONFIDENTIAL6V0.1 根據(jù)VIO Deep Dive v11和VMworld 2014 VIO的資料整理完成 by 劉承罡V0.2 漢化部分內容,條理化一些章節(jié),增加deep dive 301的部分內容V0.3 根據(jù)blog內容更新一些VIO 2.0的重點featureAgenda1什么是VMwar
4、e Integrated Openstack(VIO)2VMware 的VIO技術架構3VMware VIO+NSX+vSphere物理部署4VMware VIO使用場景CONFIDENTIAL7OpenStack2014年度最受歡迎的開源項目最受歡迎的開源云項目最受歡迎的開源IaaS項目CONFIDENTIAL8OpenStack:提升應用開發(fā)與維護效率9適用于OpenStack的負載 主要是新應用,或者從公有云遷回到私有云的應用。常見應用舉例:水平擴展的SaaS/Web應用,移動應用后端開發(fā)與測試的一體化平臺科研、批處理任務:數(shù)據(jù)分析,編碼和模擬等。OpenStack是一個可以幫助企業(yè)快速
5、開發(fā)和維護新應用的框架,同時也是運行在虛擬基礎架構上的一組工具。應用開發(fā)與運維團隊備置代碼更新擴展或縮減腳本或代碼API callsOpenStack IaaS Cloud應用管理工具or虛擬計算、網(wǎng)絡與存儲?CONFIDENTIALHypervisor網(wǎng)絡虛擬化存儲管理基礎架構服務API + 驅動基礎架構消費工具虛擬化技術10云的運維與管理硬件資源云管理應用管理, 財務報告云運維監(jiān)視,診斷私有云環(huán)境下的OpenStackOpenStack需補充 底層的軟硬件基礎架構產品云運維與管理產品私有云Web門戶計算服務網(wǎng)絡服務存儲服務OpenStack APIs/ SDKs命令行工具OpenStack
6、框架OpenStack可提供開發(fā)者友好的云服務API廠商中立的基礎架構服務CONFIDENTIALVMware是OpenStack的主要貢獻者Source: Stackalytics for “integrated (i.e., core) OpenStack projects in OpenStack Icehouse ( ) 貢獻內容涵蓋所有7個核心項目: Nova, Neutron, Cinder, Glance, Keystone, Horizon, Ceilometer在最新的OpenStack Icehouse發(fā)行版中貢獻量位列第四21 專職開發(fā)員414 內容提交66,488 代碼
7、數(shù)量3,770 更新審閱CONFIDENTIALOpenStack面對的主要挑戰(zhàn)(1)-復雜度高12復雜度構建云監(jiān)控故障分析容量擴展或縮減升級或打補丁簡化CONFIDENTIALOpenStack面對的主要挑戰(zhàn)(2)-功能有待完善可擴展性?”降低總體擁有成本”可用性與可靠性?”“應該選擇什么樣的技術.” 有效的運維與升級?”. 基礎架構與應用的服務質量?” 故障分析與技術支持?”高級特性?安全性?”CONFIDENTIAL?Web PortalCLI ToolsCompute APINetwork APIStorage APIImage APIOpenStack API SDKs?NSXESX
8、 & vCentervCenter Datastores3rd-party / Virtual SANIaaS API服務與驅動開發(fā)者工具虛擬化技術OpenStack Framework 應用開發(fā)團隊應用管理與自動化硬件技術云基礎架構團隊基礎架構運維與管理vCOPs, Log Insight, ITBMvCAC14VMware云解決方案是OpenStack的有力補充App DirectorCONFIDENTIALVMware Integrated OpenStack成熟完善的云架構現(xiàn)存的虛擬化環(huán)境VMware Integrated OpenStack支持OpenStack的云管理CONFIDE
9、NTIALVMware開發(fā)的OpenStack發(fā)行版以OVA的形式發(fā)布的虛擬設備內部集成了用于安裝和管理OpenStack的工具松散集成產品OpenStack APIsOpenStack用戶的新選擇CONFIDENTIAL開源代碼、產品包或OpenStack發(fā)行版VMware組件廠商中立的開發(fā)者API緊密集成產品OpenStack APIsVMware Integrated OpenStack (VIO)Agenda1什么是VMware Integrated Openstack(VIO)2VMware 的VIO技術架構3VMware VIO+NSX+vSphere物理部署4VMware VIO
10、使用場景CONFIDENTIAL17VMware Integrated OpenStack整體架構和演進思路 Horizon(web portal)CLI Tools / SDKsvCAC / ITBMCost visibility, governance, etc. vCenter Nova(compute)Neutron(network)Cinder(block storage)Glance(images)vCenter Datastores3rd-party / Virtual SANHeat, Trove, Ceilometer, etc.(Additional higher laye
11、r services in future releases)Keystone(identity)Local DB & LDAPSSOLog Insight Log collection, O/S Content packNSXvCOPsOpenStack mgmt packs基于“OpenStack Icehouse”發(fā)行版VMware提供Openstack開源軟件打包、優(yōu)化、支持一體化服務體系充分利用VMware SDDC現(xiàn)有組件,和Openstack取長補短Included OpenStack Components:Integrated VMware Technologies:Curren
12、tFutureSwift(object store)Basic open sourcevSphere Web Client Install, Configure and TroubleshootEMC ViPRVIO 架構 VIO部署拓撲架構VIO 架構 使用VIO的SDDC邏輯拓撲VIO 架構 OpenStack Management ServerThe VIO OpenStack Management Server 保存整個VIO環(huán)境的配置信息, 并負責在安裝和升級中推送整個VIO配置信息到其他VIO所有的核心組件中.所有的VIO核心組件都會從一個模版VIO Template中創(chuàng)建生成.
13、VIO 架構 Load Balancers對上:VIO的部分服務, 比如Horizon Dashboard, compute-api, glance-api, keystone, cinder-api, neutron-api and nova-vncproxy 會通過 負載均衡模塊的public virtual IP提供給最終租戶.對下:負載均衡模塊對Openstack Controller、RabbitMQ and Memcached 提供服務VIO 架構 OpenStack Controllers控制節(jié)點包括:All OpenStack API servicesAll OpenStack
14、 schedulersOpenStack schedulers 服務被用來分發(fā)計算、網(wǎng)絡和存儲卷的服務請求. The following services are connected to the messaging backend (RabbitMQ) and can be scaled-out:nova-schedulernova-conductorcinder-schedulerneutron-serverOpenStack projects provide an API service for resource management. VIO 架構 OpenStack Controll
15、ersVIO Controllers 和VMware 后臺組件的交互:vCenter/vSAN/NSX Interactions和vCenter/NSX 負責交互的主要是4個服務:Nova (Compute Service), Glance (Image Service), Cinder (Volume Service), Neutron (Networking).OpenStack Glance - vSphere Interactions在Controller VM中運行.上傳用戶鏡像(vmdk, iso, ova)到Glance所在的Datastore中Glance所在的Datastor
16、e 可以是VMFS/NFS/vSAN Datastores.OpenStack Cinder - vSphere Interactions在Controller VM中運行.在Cinder數(shù)據(jù)平面的Data Store中創(chuàng)建Flat VMDK.OpenStack Neutron - vSphere Interactions支持兩種Neutron服務- NSX/VDS and runs in the controller and DHCP node VMs.NSX 模式和 NSX Manager交互VDS 模式和VC 交互VIO 架構 MemcachedVIO數(shù)據(jù)庫服務的前端緩層,用來緩存從數(shù)據(jù)
17、庫返回的小規(guī)模數(shù)據(jù)VIO中主要緩存的數(shù)據(jù)是Keystone tokensVIO 架構 RabbitMQRabbitMQ 消息中間件,A/A部署,主要負責整個VIO環(huán)境的異步消息傳遞OpenStack中的子組件之間的通訊,比如nova-scheduler 到 pute會使用該服務組件和組件之間的服務,比如Nova到Neutron的通訊,通過 REST API之間調用.For more details about the HA implementation of RabbitMQ, please click here.VIO 架構 Database數(shù)據(jù)庫節(jié)點,一共三個節(jié)點,Active/Stand
18、by架構,使用MariaDB.VIO 架構 Nova ComputeNova的 Compute Driver VM和加入到VIO的計算容量內的Cluster一對一配置. pute runs in this Compute VM (each Compute VM has 1:1 correspondence with a vCenter cluster).Compute VM 負責的功能包括:創(chuàng)建Instance前,負責從Glance DS中拷貝到Cluster所在的共享DS(用來緩存).基于拷貝過來的VMDK用來創(chuàng)建Instance.給Instance添加相應的虛擬接口給instances添加
19、Cinder服務創(chuàng)建的vmdk.VIO 如何與 vSphere and NSX交互Nova Compute Nova ServicesNSX ManagerNeutron PluginNeutron ServerHeatGlanceRabbitMQvCenterESXi-2vSphere PluginNSXVIOESXi-1NSXKeystoneCinderVMDK DriverVMDK DriverIntroduction NSX vSphere Neutron PluginVMwares OpenStack Team開發(fā), 后端和NSX-v集成.該插件隨VIO發(fā)行版自帶集成, 上傳到Ope
20、nstack的官方(repo is vmware-nsx in StackForge).最大支持2000VMs和250臺主機.Requires NSX vSphere 6.1.3 or later 支持 VLAN (Neutron Provider Networks) and VXLAN (Overlay Tenant Networks).支持 Distributed Routing, scalable DHCP services, NAT/no-NAT Centralized Routing and Distributed Firewall.Additional information in
21、 speakers notes.Web Network (Logical Switch)基本的Neutron工作流App-1App-2Web-1Web-2App Network (Logical Switch)Floating IP (NSX Edge Services Gateway-DNAT)Security Group (Distributed Firewall Security Group)IntranetInternetRouter (NSX Edge Services Gateway-SNAT)DHCP (NSX Edge Services Gateway)NSX PluginvD
22、S PluginProvider Networks leveraging VLANsAPI/Management Plane High AvailabilityDC-Wide Control Plane ScaleHighLimitedLayer 3/NAT High Availability & Scale-Full Neutron feature-setPrivate Logical Network Identifier Independent of VLANsHighly Available DHCP ServiceSecurity Groups Metadata Service Int
23、egration & SupportL3/NAT & Floating IP Support-Enterprise FeaturesMicro-segmentation with line-rate Stateful distributed firewallProvider-side security via Service InsertionIn-kernel distributed routingMany more.-vCenter Operations & Log Insight Content Packs-VMware VIO 兩種Neutron插件NSXv Neutron Plugi
24、n 支持拓撲多層應用、集中路由Intranet/InternetVM1VM2Network ANetwork BTenant Logical Router(NSX ESG)VM4Provider Logical RouterStatic RoutingNotes:租戶網(wǎng)絡可以是VLAN 或者VXLAN由NSX Edge實現(xiàn)租戶的核心路由.支持NAT (floating IPs) 或者 no-NAT 配置.可選的DHCP服務,DHCP服務會由一個共享的Edge設備提供(VIO根據(jù)IP地址是否重疊自動創(chuàng)建)每個Edge最大10個租戶網(wǎng)絡在Edge和邊界路由器之間只支持靜態(tài)路由(通過API/CLI配
25、置).Tenant SpaceDHCP Server(NSX ESG)VM5VM3Provider SpaceNSXv Neutron Plugin 支持拓撲多層應用、分布式路由Intranet/InternetVM1VM2Network ANetwork BTenant Router(NSX ESG)VM4Provider Logical RouterStatic RoutingNotes:租戶網(wǎng)絡只能是VXLAN通過NSX Distributed Logical Router創(chuàng)建分布式路由器通過Edge實現(xiàn)租戶核心路由最大支持999個租戶網(wǎng)絡VIO自動生成DLR和Edge之間的網(wǎng)絡,并自動
26、在Edge和DLR之間生成靜態(tài)路由從VIO的角度DLR和Edge會顯示為一個租戶路由器在Edge上支持NAT no-NAT可選的DHCP服務,DHCP服務會由一個共享的Edge設備提供(VIO根據(jù)IP地址是否重疊自動創(chuàng)建)Tenant SpaceDHCP Server(NSX ESG)Transit Logical SwitchDistributed Logical RouterVM3VM5Static RoutingProvider SpaceNSX vSphere Neutron Plugin NSX Edges PoolNSX Edges, whether DLRs or ESGs, r
27、equire a VM to operate. This means provisioning time could be significant when requesting these resources from Neutron.為了減少供給時間,VIO在初始化安裝的時候自動創(chuàng)建一個Edge PoolEdges are then selected by the Neutron plugin, based on function and purpose (centralized routing, distributed routing and/or DHCP server).這個池包含1
28、2個事先供給的NSX Edges. The naming format for the pre-provisioned Edges is backup-XXXXX, for example, backup-60e5f345-8adf. Currently, the number of Edges is not configurable.Every time a pre-provisioned Edge is selected, a new one is spun off in order to maintain the pool whole. The Edge is renamed dhcp-
29、XXXX when the ESG is being used for DHCP services, or NAME-XXXX if used for Tenant routing services (按照租戶名字來命名租戶的Edge)An enhancement in NSX 6.2 will allow for the NSX DLR to operate without the need of a control VM, but only when using static routing, which is the only option supported in Neutron at
30、 the moment. This will help with provisioning time as well as VM sprawl.Backup NSXESG -RouterVIO Prepopulated Edge PoolNSXESG - DHCPBackup NSXDistributed RoutersNSX vSphere Neutron Plugin DHCP ImplementationWhen a Tenant requests DHCP services for a network segment, an NSX ESG is picked from the poo
31、l and configured with static DHCP bindings. This NSX ESG is different from the Tenant Router ESG.Emphasis: DHCP bindings are statically configured (no DHCP Pools).If overlapping IPs are required, a new ESG will be selected; otherwise the same ESG will be used. IMPORTANT: The same ESG will be used ac
32、ross Tenants in order to optimize resource utilization and mitigate VM sprawl.VLAN-trunks and VXLAN-trunks (sub-interfaces) are leveraged in order to e the 10-vNIC limit of the ESG. In VIO 1.0, 200 sub-interfaces are supported per DHCP ESG (NSX dependency).VLAN/VXLAN TrunkDHCP Server(NSX ESG)DHCP Se
33、rvers(NSX ESG)Non-overlapping IPsOverlapping IPsTenant ATenant BTenant CTenant ATenant BTenant C/24/24/24/24/24/24Agenda1什么是VMware Integrated Openstack(VIO)2VMware 的VIO技術架構3VMware VIO+NSX+vSphere物理部署4VMware VIO使用場景CONFIDENTIAL37Compute ClustersInfrastructure Clusters (Edge, vCenter and NSX/VIO Manag
34、ement)IntranetInternetThe VIO 的安裝架構要求vSphere的環(huán)境必須按照計算機架、管理機架、Edge機架的拓撲架構來進行安裝部署Optional: 管理集群和Edge集群可以合并管理集群至少三臺主機. 計算和Edge機架必須安裝NSX VTEP(主機準備)并且處于一個Transport Zone中.VIO 1.0只支持一個Transport Zone.VIO 架構 物理部署拓撲存儲設計:vSphere Design Requirements for VIOVIO Controller 1Shared Datastore 1VIO 基礎架構管理組件,默認需要3個da
35、ta store實現(xiàn)冗余,data store也作為所有的A/A組件、A/S組件布置在這三個data store上,保障冗余和高可用,同時供給給這三臺主機的 data store也作為Glance Image Service計算集群和Edge集群需要配置共享存儲用來存放Instance和虛擬路由器(Edge)計算集群的data store作為cinder的數(shù)據(jù)平面,為instance存放持久的卷數(shù)據(jù)VIO Load Balancer 1VIO DB 1VIO DB 2VIO RabbitMQ 1VIO Memcache 1VIO Object Storage 1VIO Controller 2
36、VIO Load Balancer 2VIO DB 3VIO RabbitMQ 2VIO Memchache 2VIO Object Storage 2VIO Compute 1VIO Compute 2OpenStack Management ServerVIO Compute nManagement (VIO Infrastructure)Tenant VMTenant VMTenant VMTenant VMCompute/EdgeShared Datastore(s)Tenant Cinder VolumesRedundancy provided by vSphere HARedund
37、ancy provided in VIOGlance Image 1Glance Image 2Glance ImagesGlance Image nNSX ESGDLR CVMShared Datastore 2Shared Datastore 3VIO安裝準備CONFIDENTIAL40CONFIDENTIAL41VIO安裝準備-2VIO 部署CONFIDENTIAL42VIO界面CONFIDENTIAL43Troubleshooting vRealize Log InsightNSX Content Pack for Log Insight available now.OpenStack
38、 Content Pack for Log InsightTroubleshooting vRealize Operations ManagerNSX Management Pack for vROPS available now.OpenStack Management Pack for vROPS available now.Agenda1什么是VMware Integrated Openstack(VIO)2VMware 的VIO技術架構3VMware VIO+NSX+vSphere物理部署4VMware VIO使用場景CONFIDENTIAL46典型場景:在Horizon中創(chuàng)建一個租戶
39、的典型拓撲47租戶創(chuàng)建一個邏輯拓撲拓撲和需求如下VM1VM2Web TierDB TierTenant Router(NSX ESG)VM3Sample Logical TopologyDHCP Server(NSX ESG)Transit Logical SwitchDistributed Logical RouterStatic RoutingNotes:二層應用 (Web and DB).安全策略:All inbound TCP traffic to Web is allowed on external routable IP (Floating IP)Web to DB traffic
40、 allowed on TCP 3306 (MySQL)Use Distributed Routing for optimized East-West communications (CLI provisioning).Use Centralized Routing for North-South connectivity.云管理員:配置外部網(wǎng)絡(0of 6)Admin-only operation: Create an external VLAN-backed Provider Network (for floating IPs) in Horizon and the Neutron CLI
41、.neutron net-create EXTNET -router:external True -provider:network_type portgroup -provider:physical_network dvportgroup-108Note: Checking the “Shared” and “External” options will allow floating IPs to exist on this VLAN-backed network, given a subnet is created, a range is provisioned and the physi
42、cal network has been mapped to the corresponding VDS PortGroup (which can be obtained from the vCenter MOB).Note: The dvportgroup corresponds to the vSphere Port Group mapped to the external network on the Edge cluster. You can obtain this OID by browsing the vCenter Management Object Base (MOB): ht
43、tps:/mob 云管理員:配置外部網(wǎng)絡的subnet(0 of 6)To create a Floating IP Range (Allocation Pools), simply create a subnet for the Provider External Network, with DHCP disabled:租戶創(chuàng)建一個邏輯拓撲(1 of 6)51第一步創(chuàng)建兩個二層網(wǎng)絡,并分別創(chuàng)建兩個Instance加入這兩個二層網(wǎng)絡From Horizon, access the Network tab and create the Web and DB Networks, with diff
44、erent IP Subnets and DHCP Enabled.Launch the corresponding VMs from the Instances tab and connect them to the appropriate Network (Project Compute).Examine the Dashboard Topology Map (Project Network Topology).租戶創(chuàng)建一個邏輯拓撲(2 of 6)使用Neutron CLI創(chuàng)建分布式路由器. VIO會自動創(chuàng)建分布式路由器和分布式路由器上聯(lián)的租戶核心路由器 neutron router-cr
45、eate Tenant-Router distributed True在Horizon中,添加Web和DB兩個租戶網(wǎng)絡的接口到路由器上(后臺自動在Edge和DLR之間添加靜態(tài)路由)設置路由器的默認網(wǎng)關在外部網(wǎng)絡上.VIO會自動制定該Edge為核心路由并為路由上已經(jīng)增加的Instance添加SNAT The Topology Dashboard will display both NSX routers as a single Neutron Router (Project Network Routers).租戶創(chuàng)建一個邏輯拓撲(3 of 6)為Web Instance創(chuàng)建 Floating I
46、Ps to the Web Instances (Project Compute Associate Floating IP).租戶創(chuàng)建一個邏輯拓撲(4 of 6)創(chuàng)建2個安全組并編輯規(guī)則(Web and DB) (Project Compute Access and Security Security Groups Manage Rules).邊界Instance并添加安全組規(guī)則 (Project Compute Instances Edit Security Groups).租戶創(chuàng)建一個邏輯拓撲(5 of 6)在NSX UI上檢查租戶拓撲、確認配置租戶創(chuàng)建一個邏輯拓撲(6 of 6)在NS
47、X UI上檢查租戶拓撲、確認配置:DHCP/DFW.總結:VIO簡化企業(yè)環(huán)境中OpenStack的交付與運維自動化的OpenStack管理通過vCenter自動化OpenStack的常規(guī)管理任務vCOPs集成,用于OpenStack的監(jiān)控與故障分析Log Insight集成,用于診斷OpenStack的服務日志簡單快速的OpenStack部署以虛擬設備形式交付OpenStack通過vSphere Web Client簡化安裝使用現(xiàn)有的集群和數(shù)據(jù)存儲來配置OpenStackOpenStack部署自動配置為高可用與水平擴展架構企業(yè)級的OpenStack云健壯的基礎架構vSphere, NSX &
48、VSAN為OpenStack提供豐富的基礎架構功能OpenStack代碼強化與測試單一廠商提供技術服務VMware為以下內容提供技術支持OpenStack代碼底層基礎架構安裝和運維產品升級與補丁57CONFIDENTIALVIO Roadmap SummaryVIO 1.0 (2015.03 GA)Production Ready ArchitectureAutomated Deployment & ConfigurationAutomated Patching Nova with vCenter DriverGlance and Cinder with VMDK driversNSXv Neutron pluginL2/L3/ security groups/ floating IPsAuthentication w/ Active Directory Encrypted Passwords in configuration filesFeaturesIcehouse-basedNova, Neutron, Cinder, Glance, Horizon, Keystone, HeatOpenStack ReleaseLife-Cycle Mgmt. of OpenStack
溫馨提示
- 1. 本站所有資源如無特殊說明,都需要本地電腦安裝OFFICE2007和PDF閱讀器。圖紙軟件為CAD,CAXA,PROE,UG,SolidWorks等.壓縮文件請下載最新的WinRAR軟件解壓。
- 2. 本站的文檔不包含任何第三方提供的附件圖紙等,如果需要附件,請聯(lián)系上傳者。文件的所有權益歸上傳用戶所有。
- 3. 本站RAR壓縮包中若帶圖紙,網(wǎng)頁內容里面會有圖紙預覽,若沒有圖紙預覽就沒有圖紙。
- 4. 未經(jīng)權益所有人同意不得將文件中的內容挪作商業(yè)或盈利用途。
- 5. 人人文庫網(wǎng)僅提供信息存儲空間,僅對用戶上傳內容的表現(xiàn)方式做保護處理,對用戶上傳分享的文檔內容本身不做任何修改或編輯,并不能對任何下載內容負責。
- 6. 下載文件中如有侵權或不適當內容,請與我們聯(lián)系,我們立即糾正。
- 7. 本站不保證下載資源的準確性、安全性和完整性, 同時也不承擔用戶因使用這些下載資源對自己和他人造成任何形式的傷害或損失。
最新文檔
- 2025年度交通樞紐工程承包合同協(xié)議4篇
- 2025年湖北揚帆旅游發(fā)展有限公司招聘筆試參考題庫含答案解析
- 2025年浙江嘉興捷固五金有限公司招聘筆試參考題庫含答案解析
- 2025年廣東深圳市龍崗區(qū)融媒集團招聘筆試參考題庫含答案解析
- 2025年浙江溫州浙南糧食有限公司招聘筆試參考題庫含答案解析
- 二零二五年度棉花運輸安全風險評估與管理合同4篇
- 2025年高速公路養(yǎng)護工程勞務分包合同模板4篇
- 2025年度綠色環(huán)保獸藥銷售合作框架協(xié)議3篇
- 2025年度個人住宅建筑防水工程售后服務合同4篇
- 運城師范高等??茖W校《民族文化遺產保護》2023-2024學年第一學期期末試卷
- 北師大版小學三年級上冊數(shù)學第五單元《周長》測試卷(含答案)
- 國家安全責任制落實情況報告3篇
- DB45T 1950-2019 對葉百部生產技術規(guī)程
- 2024年度順豐快遞冷鏈物流服務合同3篇
- 六年級下冊【默寫表】(牛津上海版、深圳版)(漢譯英)
- 合同簽訂培訓
- 新修訂《保密法》知識考試題及答案
- 電工基礎知識培訓課程
- 鐵路基礎知識題庫單選題100道及答案解析
- 金融AI:顛覆與重塑-深化理解AI在金融行業(yè)的實踐與挑戰(zhàn)
- 住宅樓安全性檢測鑒定方案
評論
0/150
提交評論