Get up and running with OpenStack Swift, the free, open source solution for deploying high-performance object storage clusters at scale. In this practical guide, Joe Arnold, co-founder and CEO of SwiftStack, brings you up-to-speed on the basic concepts of object storage and walks you through what you need to know to plan, build, operate, and measure the performance of your own Swift storage system.

Object storage is essential today with the growth of web, mobile, and software-as-a-service (SaaS) applications. This book helps you through the process, with separate sections on application development, installation, administration, and troubleshooting.

Learn Swift’s concepts for organizing, distributing, and serving data
Explore basic and advanced features of the Swift RESTful API
Delve into Swift’s many client libraries, including useful Python features
Write middleware to customize and simplify your storage system
Understand requirements for planning a Swift deployment—including your specific use case
Learn options for coaxing the best performance from your cluster
Get best practices for daily operations, such as monitoring and planning capacity additions
Pick up techniques for testing and benchmarking your Swift cluster

Preface

xiii

Part I. Fundamentals and Architecture

1 The Evolution of Storage

(10)

Storage Needs for Today's Data

(2)

The Growth of Data: Exabytes, Hellabytes, and Beyond

(1)

Requirements for Storing Unstructured Data

(1)

No One-Size-Fits-All Storage System

(1)

Object Storage Compared with Other Storage Types

(1)

A New Storage Architecture: Software-Defined Storage

(1)

Software-Defined Storage Components

(2)

Benefits of Software-Defined Storage

(1)

Why OpenStack Swift?

(1)

Conclusion

(3)

2 Meet Swift

(8)

Meet SwiftStack

(4)

3 Swift's Data Model and Architecture

(20)

Swift Data Model

(3)

Swift Architecture

(3)

Server Processes

(2)

Consistency Processes

(1)

Locating the Data

(7)

Ring Basics: Hash Functions

(1)

Ring Basics: Consistent Hashing Ring

(3)

The Rings: Modified Consistent Hashing Ring

(2)

Distribution of Data

(1)

Creating and Updating the Rings

(2)

Creating or Updating Builder Files

(1)

Rebalancing the Rings

(1)

Inside the Rings

(1)

Conclusion

(2)

4 Swift Basics

(12)

Talking to the Cluster: The Swift API

(1)

Sending a Request

(2)

Storage URL

(1)

Authentication

(1)

HTTP Verbs

(1)

Authorization and Taking Action

(1)

Getting a Response

(1)

Communication Tools

(4)

Command-Line Interfaces

(2)

Custom Client Applications

(1)

Example Scenarios

(1)

Conclusion

(3)

Part II. Application Design with Swift

5 Overview of the Swift API

(16)

What Is an API, Anyway?

(1)

The CAP Theorem

(1)

Swift's Sweet Spot: High Availability, Redundancy, and Throughput

(1)

Swift API: Background

(5)

Review of the Hypertext Transfer Protocol (HTTP)

(2)

Representational State Transfer (REST)

(1)

Swift, HTTP, and REST

(1)

Using the Swift API

(6)

About Your Swift Cluster

(1)

Authentication

(1)

Retrieving Data

(2)

Storing Data

(1)

Deleting Data

(1)

Updating Metadata

(1)

Conclusion

(2)

6 Swift Client Libraries

(10)

Client Libraries

(1)

The Authentication Exchange

(1)

Storage Requests: Basic Usage

(3)

Client Libraries in Other Languages

(1)

Ruby

(1)

PHP

(1)

Java

(1)

Storage Requests: Advanced Usage

(1)

Additional Considerations When Using Python

(1)

Conclusion

(2)

7 Advanced API Features

(22)

Large Objects

(2)

Object Versioning

(1)

Object Expiration

(1)

Temporary URL Middleware (TempURL)

(1)

Form Post Middleware

(1)

Custom Metadata

(1)

PUTting and POSTing Metadata

(1)

Cross-Origin Resource Sharing (CORS)

(1)

Swift Cluster Info

(1)

Range Requests

(1)

Domain Remap Middleware

(1)

Static Web Hosting

(1)

Content-Type Header

(1)

Bulk Operations Middleware

(1)

Code Samples

(11)

Static Large Objects

(1)

Dynamic Large Objects

(1)

Object Versioning

(1)

TempURL (Time-Limited URLs)

(1)

Form Post

(1)

Cross-Origin Resource Sharing

(1)

Custom Metadata

(1)

Swift Cluster Info

(1)

Range Requests

(1)

Domain Remapping

(1)

Static Web Hosting

(1)

Content-Type

(1)

Bulk Upload

(1)

Bulk Delete

(1)

Conclusion

100

(1)

8 Developing Swift Middleware

101

(20)

Introduction to WSGI

101

(2)

Programming WSGI

103

(2)

Streaming and Making Modifications to Data

105

(1)

Configuring Middleware Through Paste

106

(3)

How to Write Swift Middleware

109

(1)

Inside Out

110

(1)

Some Simple Examples

111

(1)

Doing More in Middleware

112

(5)

A Look Back and a Look Forward

117

(1)

Conclusion

118

(3)

Part III. Installing Swift

9 Installing OpenStack Swift from Source

121

(26)

Downloading OpenStack Swift

121

(2)

Dependencies

122

(1)

Installing the Swift CLI (python-swiftclient)

122

(1)

Installing Swift

122

(1)

Copying in Swift Configuration Files

123

(1)

Configuring Swift

123

(14)

Adding Drives to Swift

123

(4)

Storage Policies

127

(2)

Creating the Ring Builder Files

129

(5)

Adding Devices to the Builder Files

134

(2)

Adding Drives

136

(1)

Building the Rings

136

(1)

Configuring Swift Logging

137

(1)

Creating the Log Configuration File

137

(1)

Restarting Rsyslog to Begin Swift Logging

138

(1)

Configuring a Proxy Server

138

(1)

Setting the Hash Path Prefix and Suffix

138

(1)

Starting the Proxy Server

139

(1)

Setting up TempAuth Authentication and Authorization with Swift

139

(3)

Starting memcached

139

(1)

Adding Users to proxy-server.conf

140

(1)

Starting the Servers and Restarting the Proxy

141

(1)

Account Authentication

141

(1)

Verifying Account Access

142

(1)

Creating a Container

143

(1)

Uploading an Object

144

(1)

Starting the Consistency Processes

145

(1)

Configuring rsync

145

(1)

Starting the Remaining Consistency Processes

146

(1)

Conclusion

146

(1)

10 Installing SwiftStack

147

(16)

SwiftStack Controller and Node Overview

147

(3)

SwiftStack Controller

147

(2)

SwiftStack Node

149

(1)

Creating a Swift Cluster Using SwiftStack

150

(9)

Creating a SwiftStack Controller User

151

(1)

Installing the SwiftStack Node Software

151

(1)

Claiming a New Node

152

(1)

Creating a Cluster

152

(1)

Ingesting a Node

153

(1)

Enabling a SwiftStack Node

154

(1)

Provisioning a SwiftStack Node

154

(1)

Adding Swift Users

155

(1)

SwiftStack Middleware

156

(1)

Deploying to Cluster

157

(1)

Creating a Container and Uploading an Object via Web Console

158

(1)

Conclusion

159

(4)

Part IV. Planning a Swift Deployment

11 Hardware for Swift

163

(12)

Node Hardware Specifications

164

(6)

CPU

164

(3)

RAM

167

(2)

Drives

169

(1)

Cluster Networking

170

(3)

Network Cards

171

(1)

Outward-Facing Network

172

(1)

Cluster-Facing Network

172

(1)

Replication Network

173

(1)

Out-of-Band Management

173

(1)

Other Networking Connections

173

(1)

Conclusion

173

(2)

12 Planning a Swift Deployment

175

(18)

Your Use Case

175

(2)

System Design

177

(9)

How Many Nodes?

177

(5)

Tiering Node Services

182

(1)

Defining Your Cluster Space

182

(2)

Node Naming Conventions

184

(1)

Authentication and Authorization

185

(1)

Networking

186

(4)

Outward-Facing Network

187

(1)

Cluster-Facing Network

188

(2)

Sample Deployments

190

(2)

Small Cluster: Several Nodes

191

(1)

Medium-Size Cluster: Multi-Rack

191

(1)

Large Cluster: Multi-Region

191

(1)

Conclusion

192

(1)

13 Authentication and Authorization

193

(26)

Authentication

193

(10)

How Authentication Works

194

(2)

Authentication Request

196

(3)

Authentication Handling

199

(2)

Authentication Response

201

(2)

Using the Auth Token in Storage Requests

203

(1)

Authorization

203

(6)

Authorization Examples

203

(1)

How Authorization Works

204

(3)

Storage Request Processing

207

(1)

Token Verification and Authorization Information Lookup

208

(1)

Authorization Callback and Response

208

(1)

Authorization and Access Levels

209

(1)

Account-Level Access Control

210

(2)

Read-Only Access

211

(1)

Read-Write Access

211

(1)

Admin Access

211

(1)

JSON for Account Access Control

211

(1)

Container-Level Access Control

212

(2)

Container ACL Examples

213

(1)

Swift Authentication Systems

214

(2)

Keystone

214

(1)

TempAuth

214

(1)

SWAuth

215

(1)

SwiftStack Authentication Systems

216

(1)

SwiftStack Auth

216

(1)

SwiftStack LDAP

216

(1)

SwiftStack Active Directory

217

(1)

Conclusion

217

(2)

14 Cluster Tuning and Optimization

219

(16)

Swift Settings

219

(8)

Workers

220

(1)

Chunk Size

221

(2)

Settings for Background Daemons

223

(4)

Externally Managed Settings

227

(1)

Swift Middleware

227

(6)

Middleware Pipeline

227

(1)

Essential Middleware

228

(3)

Most Useful Middleware

231

(1)

Other Middleware

231

(2)

The SwiftStack Approach

233

(1)

Conclusion

234

(1)

15 Operating a Swift Cluster

235

(30)

Operational Considerations

235

(2)

How Swift Distributes Data

236

(1)

Keeping Track of the Rings and Builder Files

236

(1)

Managing Capacity

237

(10)

What to Avoid

238

(1)

Adding Capacity

238

(1)

Existing Cluster: Initial Ring on Node

239

(3)

Adding Nodes

242

(5)

Removing Capacity

247

(4)

Removing Nodes

248

(1)

Removing Disks

249

(2)

Managing Capacity Additions with SwiftStack

251

(5)

Adding Capacity

252

(1)

Adding Drives

252

(1)

Adding a Node

253

(1)

Removing Capacity

254

(1)

Removing a Node

254

(1)

Removing a Disk

254

(2)

Monitoring Your Cluster

256

(4)

Swift-Specific Metrics: What to Look For

256

(1)

Monitoring and Logging Tools

257

(1)

SwiftStack Tools

258

(2)

Operating with SwiftStack

260

(5)

Part V. Debugging and Troubleshooting

16 Hardware Failures and Recovery

265

(8)

Handling a Failed Drive

265

(2)

Handling a Full Drive

267

(1)

Handling Sector or Partial Drive Failure (a.k.a. Bit Rot)

267

(1)

Handling Unreachable Nodes

268

(1)

Handling a Failed Node

269

(1)

Node Failure Case Study

270

(1)

Conclusion

271

(2)

17 Benchmarking

273

(26)

Evaluating Performance

275

(1)

Performance Metrics, Benchmarking, and Testing

275

(6)

Preparing Your Cluster for Benchmarking

276

(1)

Pitfalls and Mistakes to Avoid

277

(1)

Benchmarking Goals and Tools

278

(1)

Don't Get Greedy

279

(1)

Bottlenecks

280

(1)

Benchmarking with ssbench

281

(9)

Installing ssbench

281

(1)

A Basic ssbench Run

282

(1)

Defining Use Cases

283

(1)

How ssbench Works

284

(1)

Measuring Basic Performance

284

(1)

Taking ssbench Further

285

(1)

Defining the Scenario File

286

(3)

The ssbench-worker

289

(1)

Ways to Start ssbench-worker

289

(1)

Benchmarking with swift-bench

290

(8)

Preparation

291

(1)

How swift-bench Works

291

(1)

Number of Containers

292

(1)

Testing High Concurrency (-c, -b)

292

(1)

Testing Latency

293

(1)

Object Size (-s, -l)

293

(1)

Number of Objects (-n)

294

(1)

Number of GETs (-g)

294

(1)

Don't Delete Option (-x)

294

(1)

Creating a Configuration File

294

(1)

Sample swift-bench Run

295

(1)

Running a Distributed swift-bench

295

(1)

Sample swift-bench Configuration

295

(2)

Statistics Tools

297

(1)

Conclusion

298

(1)

A Swift Afterword

299

(4)

Index

303

Joe Arnold is co-founder and CEO of SwiftStack, a leading provider of object storage software. He built one of the first widely-used cloud platforms-as-a-service (PaaS) on top of Amazon Web Services at Engine Yard, managed the first public OpenStack launch of Swift after its release as an open source project. He has subsequently deployed multiple large-scale cloud storage systems. While at Engine Yard, he additionally oversaw the development of their Ruby on Rails deployment platform, AppCloud and managing the open source efforts of Rails 3, JRuby and Rubinius.

Object Storage with Swift [Mīkstie vāki]

Konts un iestatījumi

Meklēšana

Meklēt datubāzē

Refine By

Tēmas Grāmatas angļu valodā

Izvēlieties iepirkumu grozu