minio高可用搭建
文档地址:Install and Deploy MinIO — MinIO Object Storage for Linux
准备多台minio服务器
我这里准备了两台
192.168.40.139 192.168.40.140
关闭防火墙
(或者开放minio需要的端口)
systemctl stop firewalld.service
systemctl disable firewalld.service
搭建负载均衡器
负载均衡器应使用“最少连接数”算法将请求路由到 MinIO 部署,因为部署中的任何 MinIO 节点都可以接收、路由或处理客户端请求。
官方推荐了两个:nginx 和 HaProxy
我这边使用nginx,搭建过程可参考:Nginx:安装 – 秋风飒飒吹 – 博客园 (cnblogs.com)
配置如下:(/etc/nginx/nginx.conf)
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log notice;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
#tcp_nopush on;
keepalive_timeout 65;
upstream minioServer {
least_conn;
server 192.168.40.139:9000;
server 192.168.40.140:9000;
}
server {
listen 80;
server_name localhost;
location ^~ /file/ {
proxy_buffering off;
proxy_pass http://minioServer/;
}
}
}
准备连续的域名
MinIO 需要使用扩展表示法 {x…y} 表示创建服务器池时连续的一系列 MinIO 主机。因此,MinIO 需要使用按顺序编号的主机名来表示部署中的每个 minio 服务器进程。
在开始此过程之前创建必要的 DNS 主机名映射。例如,以下主机名将支持 2 节点分布式部署:
minio1.file.com
minio2.file.com
您可以使用扩展表示法 minio{1…2}.example.com 指定整个主机名范围。
两台机器分别设置hostname:
hostnamectl set-hostname minio1.file.com
hostnamectl set-hostname minio2.file.com
cat >> /etc/hosts << EOF
192.168.40.139 minio1.file.com
192.168.40.140 minio2.file.com
EOF
存储说明
MinIO 强烈建议使用带有 XFS 格式磁盘的直连 JBOD 阵列,以获得最佳性能。
确保部署中的所有节点使用相同类型(NVMe、SSD 或 HDD)的驱动器,具有相同的容量(例如 N TB)。
MinIO 不区分驱动器类型,也不会从混合存储类型中受益。
此外。MinIO 将每个磁盘使用的大小限制为部署中的最小驱动器。例如,如果部署有 15 个 10TB 磁盘和 1 个 1TB 磁盘,则 MinIO 将每个磁盘的容量限制为 1TB。
MinIO 需要使用扩展表示法 {x…y} 表示创建新部署时的顺序磁盘系列,其中部署中的所有节点都有一组相同的装载驱动器。MinIO 还要求物理磁盘的顺序在重新启动后保持不变,以便给定的装入点始终指向相同格式化的磁盘。因此,MinIO 强烈建议使用 /etc/fstab 或类似的基于文件的挂载配置,以确保驱动器顺序在重新启动后不会更改。
我每一台minio服务器准备了两块磁盘
在分布式环境中启动新的 MinIO 服务器时,存储设备不得具有现有数据。
启动 MinIO 服务器后,与数据的所有交互都必须通过 S3 API 完成。使用 MinIO 客户端、MinIO 控制台或其中一个 MinIO 软件开发工具包来处理存储桶和对象。
在每一个节点上安装minio
下载:
yum install -y wget
wget https://dl.min.io/server/minio/release/linux-amd64/minio
chmod +x minio
sudo mv minio /usr/local/bin/
新建systemd 服务文件:
vi /etc/systemd/system/minio.service
[Unit]
Description=MinIO
Documentation=https://min.io/docs/minio/linux/index.html
Wants=network-online.target
After=network-online.target
AssertFileIsExecutable=/usr/local/bin/minio
[Service]
WorkingDirectory=/usr/local
User=minio-user
Group=minio-user
ProtectProc=invisible
EnvironmentFile=-/etc/default/minio
ExecStartPre=/bin/bash -c "if [ -z \"${MINIO_VOLUMES}\" ]; then echo \"Variable MINIO_VOLUMES not set in /etc/default/minio\"; exit 1; fi"
ExecStart=/usr/local/bin/minio server $MINIO_OPTS $MINIO_VOLUMES
Restart=always
LimitNOFILE=65536
TasksMax=infinity
TimeoutStopSec=infinity
SendSIGKILL=no
[Install]
WantedBy=multi-user.target
创建用户和用户组:
groupadd -r minio-user
useradd -M -r -g minio-user minio-user
chown minio-user:minio-user /mnt/disk1 /mnt/disk2
创建minio配置文件:
minio配置文件全部参数:MinIO Server — MinIO Object Storage for Linux
vi /etc/default/minio
# Set the hosts and volumes MinIO uses at startup
# The command uses MinIO expansion notation {x...y} to denote a
# sequential series.
#
# The following example covers four MinIO hosts
# with 4 drives each at the specified hostname and drive locations.
# The command includes the port that each MinIO server listens on
# (default 9000)
MINIO_VOLUMES="http://minio{1...2}.file.com:9000/mnt/disk{1...2}/minio"
# Set all MinIO server options
#
# The following explicitly sets the MinIO Console listen address to
# port 9001 on all network interfaces. The default behavior is dynamic
# port selection.
MINIO_OPTS="--console-address :9001"
# Set the root username. This user has unrestricted permissions to
# perform S3 and administrative API operations on any resource in the
# deployment.
#
# Defer to your organizations requirements for superadmin user name.
MINIO_ROOT_USER=minioadmin
# Set the root password
#
# Use a long, random, unique string that meets your organizations
# requirements for passwords.
MINIO_ROOT_PASSWORD=minioadmin
# Set to the URL of the load balancer for the MinIO deployment
# This value *must* match across all MinIO servers. If you do
# not have a load balancer, set this value to to any *one* of the
# MinIO hosts in the deployment as a temporary measure.
#这里在实际生产过程种需要写负载均衡url
#我这里只是为了方便,写的本机
MINIO_SERVER_URL="http://192.168.40.139:9000"
运行minio服务:
sudo systemctl start minio.service
sudo systemctl status minio.service
journalctl -f -u minio.service
进入首页:MinIO Console
账号密码都是minioadmin
登录成功,进入首页,创建一个bucket,设置policy为public
上传图片:
测试访问:http://192.168.40.139/file/test-bucket/2.png
访问成功:
说明:
这里使用nginx 做的负载均衡, 当以 /file
开头的路径会代理到两台minio服务。
并且policy设置的是public,可以直接访问。
java api + springboot
配置类:MinioConfiguration.java
@Configuration
public class MinioConfiguration {
@Bean
public MinioClient minioClient(MinioProperties minioProperties) {
return MinioClient.builder()
.endpoint(minioProperties.getEndpoint())
.credentials(minioProperties.getAccessKey(), minioProperties.getSecretKey())
.build();
}
}
Properties类:MinioProperties
@Data
@ConfigurationProperties("minio")
public class MinioProperties {
private String endpoint;
private String accessKey;
private String secretKey;
}
配置文件 application.yml
minio:
endpoint: http://xxxxxxxxxxxxx
accessKey: xxxxxxxxxxxxxxxxxx
secretKey: xxxxxxxxxxxxxxxxxxx
minio的操作接口:MinioTemplate
public interface MinioTemplate {
//~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~桶操作~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~//
boolean makeBucket(String bucketName);
boolean makeBucket(String bucketName, boolean objectLock);
boolean setBucketPolicy(String bucketName, String policyJson);
/**
* 默认从类路径下 policy/policy.json 文件读取
* @author wen.jie
* @since 2022/11/10 17:59
*/
boolean setBucketPolicy(String bucketName);
boolean makeBucketAndSetPolicy(String bucketName);
boolean removeBucket(String bucketName);
boolean bucketExists(String bucketName);
/**
* 设置Retention
* 在设置Retention,该桶必须开启对象锁
* @author wen.jie
* @since 2022/11/14 10:43
* @param mode 保留模式 分为两种
* {@link RetentionMode#GOVERNANCE} 监管模式:除非用户具有特殊权限,否则用户不能覆盖或删除对象版本,或更改其锁定设置。
* {@link RetentionMode#COMPLIANCE} 合规性模式:任何用户都不能覆盖或删除受保护的对象版本, 在合规性模式下锁定对象后,
* 其保留模式便无法更改,其保留期限也不能缩短。
*/
void setObjectLockConfiguration(String bucket, RetentionMode mode, RetentionDuration retentionDuration);
ObjectLockConfiguration getObjectLockConfiguration(String bucket);
void deleteObjectLockConfiguration(String bucket);
void listenBucketNotification(String bucketName, String[] events, Consumer<NotificationRecords>consumer);
//~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~对象操作~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~//
ObjectWriteResponse putObject(InputStream inputStream, String bucket, String objectName);
ObjectWriteResponse putObject(InputStream inputStream, String bucket, String objectName, Map<String, String> userMetadata);
List<Item> listObjects(String bucket);
List<String> listObjectNames(String bucket);
/**
* @author wen.jie
* @since 2022/11/11 13:47
* @param path 路径前缀
*/
List<Item> listObjects(String bucket, String path);
List<String> listObjectNames(String bucket, String path);
List<Item> listObjects(String bucket, String path, int maxKeys, boolean recursive);
List<String> listObjectNames(String bucket, String path, int maxKeys, boolean recursive);
/**
* copy对象
* @author wen.jie
* @since 2022/11/11 14:04
*/
void copyObject(String sourceBucket, String sourceObject, String targetBucket, String targetObject);
/**
* @param targetFilePath 下载的文件目标目录
* @author wen.jie
* @since 2022/11/11 14:23
*/
void downloadObject(String bucket, String objectName, String targetFilePath);
/**
* @param targetFile 目标文件的路径
* @author wen.jie
* @since 2022/11/11 14:23
*/
void downloadObject(String bucket, String objectName, String targetFile, boolean overwrite);
/**
* @return 返回的inputStream需要自行关闭
* @author wen.jie
* @since 2022/11/11 15:40
*/
InputStream getObject(String bucket, String objectName);
void removeObject(String bucket, String objectName);
/**
* 删除指定版本对象
* @author wen.jie
* @since 2022/11/14 11:03
*/
void removeObject(String bucket, String objectName, String versionId);
void removeObjects(String bucket, List<String> objectNames);
/**
* 查询对象的信息
* @author wen.jie
* @since 2022/11/14 9:07
*/
StatObjectResponse statObject(String bucket, String objectName);
String getPresignedObjectUrl(String bucket, String objectName);
String getPresignedObjectUrl(String bucket, String objectName, int time, TimeUnit timeUnit, Map<String, String> reqParams);
//~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~对象的tag操作~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~//
void setObjectTags(String bucket, String objectName, Map<String, String> tags);
Map<String, String> getObjectTags(String bucket, String objectName);
void deleteObjectTags(String bucket, String objectName);
//~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~对象的LegalHold操作~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~//
/**
* LegalHold 没有相关的保留期,在被删除之前一直有效
* @author wen.jie
* @since 2022/11/14 11:15
*/
void enableObjectLegalHold(String bucket, String objectName, String versionId);
void disableObjectLegalHold(String bucket, String objectName, String versionId);
boolean isObjectLegalHoldEnabled(String bucket, String objectName, String versionId);
Retention getObjectRetention(String bucket, String objectName, String versionId);
}
实现类:
/**
* @author wen.jie
* @description MinioTemplate
* @since 2022/11/10 17:10
*/
@Slf4j
@Component
public class MinioTemplateImpl implements MinioTemplate {
private final MinioClient minioClient;
private final String END_SIGNAL = "/";
private final String BLANK_CONTENT = "";
@Override
public boolean makeBucket(String bucketName) {
return makeBucket(bucketName, false);
}
@Override
public boolean makeBucket(String bucketName, boolean objectLock) {
if (bucketExists(bucketName)) return true;
try {
minioClient.makeBucket(MakeBucketArgs.builder().bucket(bucketName).objectLock(objectLock).build());
return true;
} catch (Exception e) {
log.warn(e.getMessage());
return false;
}
}
@Override
public boolean setBucketPolicy(String bucketName, String policyJson) {
try {
SetBucketPolicyArgs args = SetBucketPolicyArgs.builder()
.bucket(bucketName).config(policyJson).build();
minioClient.setBucketPolicy(args);
return true;
} catch (Exception e) {
log.warn(e.getMessage());
return false;
}
}
@Override
public boolean setBucketPolicy(String bucketName) {
if (bucketExists(bucketName)) {
String policyJson;
try (InputStream resource = this.getClass().getClassLoader().getResourceAsStream("policy/policy.json")) {
policyJson = StreamUtils.copyToString(resource, Charset.defaultCharset());
return setBucketPolicy(bucketName, policyJson.replace("${bucketName}", bucketName));
} catch (IOException e) {
log.warn(e.getMessage());
}
}
return false;
}
@Override
public boolean makeBucketAndSetPolicy(String bucketName) {
return makeBucket(bucketName) && setBucketPolicy(bucketName);
}
@Override
public boolean removeBucket(String bucketName) {
try {
if (bucketExists(bucketName))
minioClient.removeBucket(RemoveBucketArgs.builder().bucket(bucketName).build());
return true;
} catch (Exception e) {
log.warn(e.getMessage());
return false;
}
}
@Override
public boolean bucketExists(String bucketName) {
try {
return minioClient.bucketExists(BucketExistsArgs.builder().bucket(bucketName).build());
} catch (Exception e) {
log.warn(e.getMessage());
return false;
}
}
@Override
public void setObjectLockConfiguration(String bucket, RetentionMode retentionMode, RetentionDuration RetentionDuration) {
try {
ObjectLockConfiguration config =
new ObjectLockConfiguration(retentionMode, RetentionDuration);
minioClient.setObjectLockConfiguration(
SetObjectLockConfigurationArgs.builder().bucket(bucket).config(config).build());
} catch (Exception e) {
log.warn(e.getMessage());
}
}
@Override
public ObjectLockConfiguration getObjectLockConfiguration(String bucket) {
try {
return minioClient.getObjectLockConfiguration(
GetObjectLockConfigurationArgs.builder().bucket(bucket).build());
} catch (Exception e) {
log.warn(e.getMessage());
}
return null;
}
@Override
public void deleteObjectLockConfiguration(String bucket) {
try {
minioClient.deleteObjectLockConfiguration(
DeleteObjectLockConfigurationArgs.builder().bucket(bucket).build());
} catch (Exception e) {
log.warn(e.getMessage());
}
}
@Override
public void listenBucketNotification(String bucketName, String[] events, Consumer<NotificationRecords> consumer) {
ListenBucketNotificationArgs notificationArgs = ListenBucketNotificationArgs.builder()
.bucket(bucketName).prefix(BLANK_CONTENT).suffix(BLANK_CONTENT).events(events).build();
try (CloseableIterator<Result<NotificationRecords>> ci = minioClient.listenBucketNotification(notificationArgs)) {
while (ci.hasNext()) {
NotificationRecords records = ci.next().get();
consumer.accept(records);
}
} catch (Exception e) {
log.warn(e.getMessage());
}
}
@Override
public ObjectWriteResponse putObject(InputStream inputStream, String bucket, String objectName) {
return putObject(inputStream, bucket, objectName, null);
}
@Override
public ObjectWriteResponse putObject(InputStream inputStream, String bucket, String objectName, Map<String, String> userMetadata) {
boolean path = objectName.endsWith(END_SIGNAL);
try {
inputStream = inputStream == null ? new ByteArrayInputStream(new byte[] {}) : inputStream;
long objectSize;
long partSize;
//判断路径
if (path) {
objectSize = 0;
partSize = -1;
} else {
int available = inputStream.available();
objectSize = available ==0 ? -1 : available;
partSize = available != -1 ? -1 : ObjectWriteArgs.MIN_MULTIPART_SIZE;
}
PutObjectArgs.Builder builder = PutObjectArgs.builder()
.bucket(bucket)
.object(objectName)
.stream(inputStream, objectSize, partSize)
.userMetadata(userMetadata)
.contentType(getContentType(objectName));
if (!path) builder.contentType(getContentType(objectName));
PutObjectArgs putObjectArgs = builder.build();
return minioClient.putObject(putObjectArgs);
} catch (Exception e) {
log.warn(e.getMessage());
} finally {
closeQuietly(inputStream);
}
return null;
}
@Override
public List<Item> listObjects(String bucket) {
return listObjects(bucket, BLANK_CONTENT);
}
@Override
public List<String> listObjectNames(String bucket) {
return listObjectNames(bucket, BLANK_CONTENT);
}
@Override
public List<Item> listObjects(String bucket, String path) {
return listObjects(bucket, path, 1000, true);
}
@Override
public List<String> listObjectNames(String bucket, String path) {
return listObjects(bucket, path).stream().map(Item::objectName)
.collect(Collectors.toList());
}
@Override
public List<Item> listObjects(String bucket, String path, int maxKeys, boolean recursive) {
ListObjectsArgs listObjectsArgs = ListObjectsArgs.builder()
.bucket(bucket)
.includeUserMetadata(true)
.prefix(path)
.maxKeys(maxKeys)
.recursive(recursive)
.build();
Iterable<Result<Item>> results = minioClient.listObjects(listObjectsArgs);
List<Item> list = new ArrayList<>();
try {
for (Result<Item> result : results) {
Item item = result.get();
list.add(item);
}
} catch (Exception e) {
log.warn(e.getMessage());
}
return list;
}
@Override
public List<String> listObjectNames(String bucket, String path, int maxKeys, boolean recursive) {
return listObjects(bucket, path, maxKeys, recursive).stream().map(Item::objectName)
.collect(Collectors.toList());
}
@Override
public void copyObject(String sourceBucket, String sourceObject, String targetBucket, String targetObject) {
try {
minioClient.copyObject(
CopyObjectArgs.builder()
.bucket(targetBucket)
.object(targetObject)
.source(
CopySource.builder()
.bucket(sourceBucket)
.object(sourceObject)
.build())
.build());
} catch (Exception e) {
log.warn(e.getMessage());
}
}
@Override
public void downloadObject(String bucket, String objectName, String targetFilePath) {
File file = new File(targetFilePath);
String path = file.getAbsolutePath();
if (!file.exists()) {
if (!file.mkdirs()) log.warn("file [{}] mkdirs failed", path);
}
File targetFile = new File(path, FilenameUtils.getName(objectName));
downloadObject(bucket, objectName, targetFile.getAbsolutePath(), true);
}
@Override
public void downloadObject(String bucket, String objectName, String targetFile, boolean overwrite) {
try {
minioClient.downloadObject(
DownloadObjectArgs.builder()
.bucket(bucket)
.object(objectName)
.filename(targetFile)
.overwrite(overwrite)
.build());
} catch (Exception e) {
log.warn(e.getMessage());
}
}
@Override
public InputStream getObject(String bucket, String objectName) {
try {
return minioClient.getObject(
GetObjectArgs.builder()
.bucket(bucket)
.object(objectName)
.build());
} catch (Exception e) {
log.warn(e.getMessage());
}
return null;
}
@Override
public void removeObject(String bucket, String objectName) {
removeObject(bucket, objectName, null);
}
@Override
public void removeObject(String bucket, String objectName, String versionId) {
try {
minioClient.removeObject(
RemoveObjectArgs.builder().bucket(bucket).versionId(versionId).object(objectName).build());
} catch (Exception e) {
log.warn(e.getMessage());
}
}
@Override
public void removeObjects(String bucket, List<String> objectNames) {
List<DeleteObject> objects =
objectNames.stream().map(DeleteObject::new).collect(Collectors.toList());
Iterable<Result<DeleteError>> results =
minioClient.removeObjects(
RemoveObjectsArgs.builder().bucket(bucket).objects(objects).build());
for (Result<DeleteError> result : results) {
try {
DeleteError error = result.get();
if (error != null)
log.error("Error in deleting object [{}] ; [{}]", error.objectName(), error.message());
} catch (Exception e) {
log.warn(e.getMessage());
}
}
}
@Override
public StatObjectResponse statObject(String bucket, String objectName) {
try {
return minioClient.statObject(
StatObjectArgs.builder().bucket(bucket).object(objectName).build());
} catch (Exception e) {
log.warn(e.getMessage());
}
return null;
}
@Override
public String getPresignedObjectUrl(String bucket, String objectName) {
return getPresignedObjectUrl(bucket, objectName, 7, TimeUnit.DAYS, null);
}
@Override
public String getPresignedObjectUrl(String bucket, String objectName, int time, TimeUnit timeUnit, Map<String, String> reqParams) {
String url =
null;
try {
url = minioClient.getPresignedObjectUrl(
GetPresignedObjectUrlArgs.builder()
.method(Method.GET)
.bucket(bucket)
.object(objectName)
.expiry(time, timeUnit)
.extraQueryParams(reqParams)
.build());
} catch (Exception e) {
log.warn(e.getMessage());
}
return url;
}
@Override
public void setObjectTags(String bucket, String objectName, Map<String, String> tags) {
try {
minioClient.setObjectTags(
SetObjectTagsArgs.builder().bucket(bucket).object(objectName).tags(tags).build());
} catch (Exception e) {
log.warn(e.getMessage());
}
}
@Override
public Map<String, String> getObjectTags(String bucket, String objectName) {
try {
return minioClient.getObjectTags(
GetObjectTagsArgs.builder().bucket(bucket).object(objectName).build()).get();
} catch (Exception e) {
log.warn(e.getMessage());
}
return Collections.emptyMap();
}
@Override
public void deleteObjectTags(String bucket, String objectName) {
try {
minioClient.deleteObjectTags(
DeleteObjectTagsArgs.builder().bucket(bucket).object(objectName).build());
} catch (Exception e) {
log.warn(e.getMessage());
}
}
@Override
public void enableObjectLegalHold(String bucket, String objectName, String versionId) {
try {
minioClient.enableObjectLegalHold(
EnableObjectLegalHoldArgs.builder()
.bucket(bucket)
.object(objectName)
.versionId(versionId)
.build());
} catch (Exception e) {
log.warn(e.getMessage());
}
}
@Override
public void disableObjectLegalHold(String bucket, String objectName, String versionId) {
try {
minioClient.disableObjectLegalHold(
DisableObjectLegalHoldArgs.builder()
.bucket(bucket)
.versionId(versionId)
.object(objectName)
.build());
} catch (Exception e) {
log.warn(e.getMessage());
}
}
@Override
public boolean isObjectLegalHoldEnabled(String bucket, String objectName, String versionId) {
try {
return minioClient.isObjectLegalHoldEnabled(
IsObjectLegalHoldEnabledArgs.builder()
.bucket(bucket)
.versionId(versionId)
.object(objectName)
.build());
} catch (Exception e) {
log.warn(e.getMessage());
}
return false;
}
@Override
public Retention getObjectRetention(String bucket, String objectName, String versionId) {
try {
return minioClient.getObjectRetention(
GetObjectRetentionArgs.builder()
.bucket(bucket)
.object(objectName)
.versionId(versionId)
.build());
} catch (Exception e) {
log.warn(e.getMessage());
}
return null;
}
private String getContentType(String objectName) {
Optional<MediaType> mediaTypeOpt = MediaTypeFactory.getMediaType(objectName);
return mediaTypeOpt.orElse(MediaType.APPLICATION_OCTET_STREAM).toString();
}
private void closeQuietly(Closeable closeable) {
if (closeable != null) {
try {
closeable.close();
} catch (IOException e) {
log.warn(e.getMessage());
}
}
}
public MinioTemplateImpl(MinioClient minioClient) {
this.minioClient = minioClient;
}
}
原文地址:http://www.cnblogs.com/wwjj4811/p/16888548.html