Tom Crasset
be2b389b81
add more logging for s3 signature ( #6652 )
go: build dev binaries / cleanup (push) Waiting to run
go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Blocked by required conditions
go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Blocked by required conditions
go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Blocked by required conditions
go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Blocked by required conditions
docker: build dev containers / build-dev-containers (push) Waiting to run
End to End / FUSE Mount (push) Waiting to run
go: build binary / Build (push) Waiting to run
Ceph S3 tests / Ceph S3 tests (push) Waiting to run
2025-03-21 07:37:15 -07:00
zuzuviewer
db759a1ad1
Fix s3 auth failed with X-Forwarded-Host and X-Forwarded-Port ( #6650 )
2025-03-20 20:47:07 -07:00
chrislu
cb33ee006e
skip headers when signing a request
...
go: build dev binaries / cleanup (push) Has been cancelled
docker: build dev containers / build-dev-containers (push) Has been cancelled
End to End / FUSE Mount (push) Has been cancelled
go: build binary / Build (push) Has been cancelled
Ceph S3 tests / Ceph S3 tests (push) Has been cancelled
go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Has been cancelled
go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Has been cancelled
go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Has been cancelled
go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Has been cancelled
fix https://github.com/seaweedfs/seaweedfs/issues/6576#issuecomment-2724577279
2025-03-18 08:35:53 -07:00
SmoothDenis
e60aa71890
fix: restore deletion audit of individual objects ( #6644 )
go: build dev binaries / cleanup (push) Waiting to run
go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Blocked by required conditions
go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Blocked by required conditions
go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Blocked by required conditions
go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Blocked by required conditions
docker: build dev containers / build-dev-containers (push) Waiting to run
End to End / FUSE Mount (push) Waiting to run
go: build binary / Build (push) Waiting to run
Ceph S3 tests / Ceph S3 tests (push) Waiting to run
2025-03-18 05:01:54 -07:00
NinaLua
271505717e
chore: make function comment match function name ( #6607 )
go: build dev binaries / cleanup (push) Has been cancelled
docker: build dev containers / build-dev-containers (push) Has been cancelled
End to End / FUSE Mount (push) Has been cancelled
go: build binary / Build (push) Has been cancelled
Ceph S3 tests / Ceph S3 tests (push) Has been cancelled
go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Has been cancelled
go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Has been cancelled
go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Has been cancelled
go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Has been cancelled
2025-03-07 00:45:47 -08:00
Tom Crasset
9604db2c93
implement s3 streaming-unsigned-payload-trailer ( #6539 )
...
* implement s3 streaming-unsigned-payload-trailer
* chore: remove print
2025-02-12 12:29:13 -08:00
Tom Crasset
9ca2b0b763
omit http and https ports when using X-Forwarded-Port ( #6527 )
go: build dev binaries / cleanup (push) Has been cancelled
docker: build dev containers / build-dev-containers (push) Has been cancelled
End to End / FUSE Mount (push) Has been cancelled
go: build binary / Build (push) Has been cancelled
Ceph S3 tests / Ceph S3 tests (push) Has been cancelled
go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Has been cancelled
go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Has been cancelled
go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Has been cancelled
go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Has been cancelled
2025-02-07 10:55:09 -08:00
Tom Crasset
a7b964af96
add s3 signature tests and prepare implementation of STREAMING-UNSIGNED-PAYLOAD-TRAILER ( #6525 )
...
* add tests for s3 signature
* add test for newSignV4ChunkedReader.Read()
* add glog import
2025-02-07 10:54:31 -08:00
zouyixiong
8eab76c5db
fix: record and delete bucket metrics after inactive ( #6523 )
...
Co-authored-by: XYZ <XYZ>
2025-02-07 10:26:39 -08:00
Tom Crasset
a250699225
use X-Forwarded-Host and X-Forwarded-Port to determine correct host for AWS signature ( #6514 )
2025-02-06 07:54:50 -08:00
chrislu
7fd42f5d06
Revert "Add content length to filer's requests ( #6489 )"
...
This reverts commit 9f42ce3f01
2025-02-03 09:24:14 -08:00
chrislu
42efade0dc
adjust fix
...
go: build dev binaries / cleanup (push) Waiting to run
go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Blocked by required conditions
go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Blocked by required conditions
go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Blocked by required conditions
go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Blocked by required conditions
docker: build dev containers / build-dev-containers (push) Waiting to run
End to End / FUSE Mount (push) Waiting to run
go: build binary / Build (push) Waiting to run
Ceph S3 tests / Ceph S3 tests (push) Waiting to run
fix https://github.com/seaweedfs/seaweedfs/issues/6497
2025-02-01 14:11:57 -08:00
Tom Crasset
551baa44b2
verify if bucket exists in GetBucketLocationHandler ( #6485 )
go: build dev binaries / cleanup (push) Waiting to run
go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Blocked by required conditions
go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Blocked by required conditions
go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Blocked by required conditions
go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Blocked by required conditions
docker: build dev containers / build-dev-containers (push) Waiting to run
End to End / FUSE Mount (push) Waiting to run
go: build binary / Build (push) Waiting to run
Ceph S3 tests / Ceph S3 tests (push) Waiting to run
2025-01-29 08:55:08 -08:00
Hadi Zamani
9f42ce3f01
Add content length to filer's requests ( #6489 )
2025-01-29 08:53:39 -08:00
Hadi Zamani
a2330f624b
Add metrics for uploaded and deleted s3 objects ( #6475 )
go: build dev binaries / cleanup (push) Has been cancelled
docker: build dev containers / build-dev-containers (push) Has been cancelled
End to End / FUSE Mount (push) Has been cancelled
go: build binary / Build (push) Has been cancelled
Ceph S3 tests / Ceph S3 tests (push) Has been cancelled
go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Has been cancelled
go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Has been cancelled
go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Has been cancelled
go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Has been cancelled
2025-01-25 21:55:06 -08:00
Tom Crasset
c5f21b2b01
fix S3 per-user-directory Policy ( #6443 )
...
go: build dev binaries / cleanup (push) Has been cancelled
docker: build dev containers / build-dev-containers (push) Has been cancelled
End to End / FUSE Mount (push) Has been cancelled
go: build binary / Build (push) Has been cancelled
Ceph S3 tests / Ceph S3 tests (push) Has been cancelled
go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Has been cancelled
go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Has been cancelled
go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Has been cancelled
go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Has been cancelled
* fix S3 per-user-directory Policy
* Delete docker/config.json
* add tests
* remove logs
* undo modifications of weed/shell/command_volume_balance.go
* remove modifications of docker-compose
* fix failing test
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2025-01-17 01:03:17 -08:00
Hadi Zamani
c7ae969c06
Add bucket's traffic metrics ( #6444 )
...
go: build dev binaries / cleanup (push) Waiting to run
go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Blocked by required conditions
go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Blocked by required conditions
go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Blocked by required conditions
go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Blocked by required conditions
docker: build dev containers / build-dev-containers (push) Waiting to run
End to End / FUSE Mount (push) Waiting to run
go: build binary / Build (push) Waiting to run
Ceph S3 tests / Ceph S3 tests (push) Waiting to run
helm: lint and test charts / lint-test (push) Has been cancelled
* Add bucket's traffic metrics
* Add bucket traffic to dashboards
* Fix bucket metrics help messages
* Fix variable names
2025-01-16 08:23:35 -08:00
Tom Crasset
aa299462f2
improve iam error handling ( #6446 )
...
* improve iam error handling
* Delete docker/test.py
2025-01-16 08:23:19 -08:00
ftong2020
2304d2b472
Bugfix s3 audit missing requester for PUT requests ( #6434 )
...
go: build dev binaries / cleanup (push) Waiting to run
go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Blocked by required conditions
go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Blocked by required conditions
go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Blocked by required conditions
go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Blocked by required conditions
docker: build dev containers / build-dev-containers (push) Waiting to run
End to End / FUSE Mount (push) Waiting to run
go: build binary / Build (push) Waiting to run
Ceph S3 tests / Ceph S3 tests (push) Waiting to run
fix s3 audit missing requster for PUT
2025-01-15 11:58:43 -08:00
chrislu
d7dfb11be9
fix compilation
2024-12-19 19:26:28 -08:00
chrislu
ec155022e7
"golang.org/x/exp/slices" => "slices" and go fmt
2024-12-19 19:25:06 -08:00
Warren Hodgkinson
a1a76ccb8c
Fix for DeleteMultipleObjectsHandler wrongly deleting parent folders ( #6380 )
...
What problem are we solving?
Fix : #6379
How are we solving the problem?
We check for the AllowEmptyFolders option prior to cascade
deleting parent folders in S3 DeleteMultipleObjectsHandler.
How is the PR tested?
We ran SeaweedFS in a Kubernetes Cluster with a joint Filer
and S3 server in one container, with leveldb2 as the filer storage,
and AllowEmptyFolders set to true.
When using the Distribution Registry as the S3 client, it calls the
DeleteMultipleObjectsHandler as part of the artifact upload process
(uploads to a temp location, then performs a copy and delete).
Without this fix, the deletion cascade deleted parent folder until
the entire contents of the bucket were gone.
With this fix, the existing content of the bucket remained, and the
newly uploaded content was added.
Checks
[ ] I have added unit tests if possible.
[ ] I will add related wiki document changes and link to this PR after merging.
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2024-12-19 18:00:08 -08:00
zouyixiong
9987a65e8a
fix: record and delete bucket metrics after inactive ( #6349 )
2024-12-12 20:34:02 -08:00
steve.wei
88fa64a01a
change comment for expect header ( #6289 )
...
* remove case of expect
* Set the default value of the Expect header for compatibility.
2024-11-26 08:35:41 -08:00
chrislu
6b2612690b
consistent error message if auth is not setup
...
fix https://github.com/seaweedfs/seaweedfs/issues/6262
2024-11-20 23:11:18 -08:00
han
65fb8fad99
Update filer_multipart.go ( #6188 )
...
fix the #6177 bug
2024-11-01 00:38:35 -07:00
chrislu
1c2b10303a
remove X-Forwarded-For
...
go: build dev binaries / cleanup (push) Has been cancelled
docker: build dev containers / build-dev-containers (push) Has been cancelled
End to End / FUSE Mount (push) Has been cancelled
go: build binary / Build (push) Has been cancelled
helm: lint and test charts / lint-test (push) Has been cancelled
Ceph S3 tests / Ceph S3 tests (push) Has been cancelled
go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Has been cancelled
go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Has been cancelled
go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Has been cancelled
go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Has been cancelled
fix https://github.com/seaweedfs/seaweedfs/issues/6136
2024-10-23 17:52:45 -07:00
chrislu
0df4a4298b
adjust comment
2024-10-21 22:36:25 -07:00
steve.wei
cfbe45c765
feat: add in-flight metric for s3/file/volume-server ( #6120 )
2024-10-14 12:10:05 -07:00
chrislu
6845e25318
set proxied request content length
go: build dev binaries / cleanup (push) Has been cancelled
docker: build dev containers / build-dev-containers (push) Has been cancelled
End to End / FUSE Mount (push) Has been cancelled
go: build binary / Build (push) Has been cancelled
Ceph S3 tests / Ceph S3 tests (push) Has been cancelled
go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Has been cancelled
go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Has been cancelled
go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Has been cancelled
go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Has been cancelled
2024-10-10 13:55:32 -07:00
Konstantin Lebedev
44b275879b
[s3] add {Get,Put,Delete}BucketTagging and PublicAccessBlock Handlers ( #6088 )
...
go: build dev binaries / cleanup (push) Has been cancelled
docker: build dev containers / build-dev-containers (push) Has been cancelled
End to End / FUSE Mount (push) Has been cancelled
go: build binary / Build (push) Has been cancelled
Ceph S3 tests / Ceph S3 tests (push) Has been cancelled
go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Has been cancelled
go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Has been cancelled
go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Has been cancelled
go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Has been cancelled
* add {Get,Put,Delete}BucketTagging Handlers
* s3 add skip bucket PublicAccessBlock handlers
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2024-10-04 10:59:14 -07:00
chrislu
a3a8f6217e
fix
go: build dev binaries / cleanup (push) Waiting to run
go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Blocked by required conditions
go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Blocked by required conditions
go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Blocked by required conditions
go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Blocked by required conditions
docker: build dev containers / build-dev-containers (push) Waiting to run
End to End / FUSE Mount (push) Waiting to run
go: build binary / Build (push) Waiting to run
Ceph S3 tests / Ceph S3 tests (push) Waiting to run
helm: lint and test charts / lint-test (push) Has been cancelled
2024-10-03 09:03:17 -07:00
Konstantin Lebedev
a3556acd98
[s3] add skip bucket encryption handlers ( #6091 )
...
s3 add skip bucket encryption handlers
2024-10-03 08:10:43 -07:00
Er2
5644bc8f01
s3api: Fix signature v4 with reverse proxy at sub-path ( #6092 )
2024-10-03 08:08:19 -07:00
chrislu
f747767aa1
support load balancer in front of s3
go: build dev binaries / cleanup (push) Waiting to run
go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Blocked by required conditions
go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Blocked by required conditions
go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Blocked by required conditions
go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Blocked by required conditions
docker: build dev containers / build-dev-containers (push) Waiting to run
End to End / FUSE Mount (push) Waiting to run
go: build binary / Build (push) Waiting to run
Ceph S3 tests / Ceph S3 tests (push) Waiting to run
2024-10-01 12:57:45 -07:00
steve.wei
ae8c6d95a6
fix: Prevent potential metadata change events from being lost. ( #6066 )
2024-09-26 08:31:57 -07:00
chrislu
50e22864cf
refactor
2024-09-17 23:10:41 -07:00
chrislu
43d9f68219
more consistent server header for s3 apis
2024-09-09 09:07:58 -07:00
Konstantin Lebedev
16984ef07a
[s3] reducing the number of metrics for 403 ( #5961 )
...
reducing the number of metrics
2024-09-04 11:06:54 -07:00
chrislu
6ce27affa4
also use /healthz for most consistent health check
2024-08-21 10:55:39 -07:00
Oleg Salionov
4cfc7d3b10
Dont try lazy decode content in proxyToFiler if no accept-encoding provided ( #5907 )
2024-08-18 21:51:17 -07:00
Kamran Sarwar
63c707f9c1
added access-control-expose-headers to put request ( #5876 )
2024-08-09 15:52:05 -07:00
chrislu
aed2a18223
get, set and add path conf
2024-07-20 11:27:11 -07:00
vadimartynov
86d92a42b4
Added tls for http clients ( #5766 )
...
* Added global http client
* Added Do func for global http client
* Changed the code to use the global http client
* Fix http client in volume uploader
* Fixed pkg name
* Fixed http util funcs
* Fixed http client for bench_filer_upload
* Fixed http client for stress_filer_upload
* Fixed http client for filer_server_handlers_proxy
* Fixed http client for command_fs_merge_volumes
* Fixed http client for command_fs_merge_volumes and command_volume_fsck
* Fixed http client for s3api_server
* Added init global client for main funcs
* Rename global_client to client
* Changed:
- fixed NewHttpClient;
- added CheckIsHttpsClientEnabled func
- updated security.toml in scaffold
* Reduce the visibility of some functions in the util/http/client pkg
* Added the loadSecurityConfig function
* Use util.LoadSecurityConfiguration() in NewHttpClient func
2024-07-16 23:14:09 -07:00
Konstantin Lebedev
9c440d472c
[s3] revert skip deletion error, since the error file was not found is already skipped on the side of the grpc function ( #5760 )
...
* revert skip deletion error, since the error file was not found is already skipped on the side of the grpc function
* fix response error
* fix test_lifecycle_get
* Revert "fix test_lifecycle_get"
This reverts commit 8f991bdcf9
2024-07-10 08:47:43 -07:00
chrislu
b8121e4b24
simplify
2024-07-08 11:10:41 -07:00
chrislu
f618121e63
Fix S3 deletion in deep folders, and names with empty spaces
...
fix https://github.com/seaweedfs/seaweedfs/issues/5748
2024-07-08 10:20:09 -07:00
Konstantin Lebedev
f77eee667d
add s3test for sql ( #5718 )
...
* add s3test for sql
* fix test test_bucket_listv2_delimiter_basic for s3
* fix action s3tests
* regen s3 api xsd
* rm minor s3 test test_bucket_listv2_fetchowner_defaultempty
* add docs
* without xmlns
2024-07-04 11:00:41 -07:00
Konstantin Lebedev
5ffacbb6ea
refactor all methods strings to const ( #5726 )
2024-07-01 01:00:39 -07:00
Lê Hoàng Nam
1f2dc54647
fix(s3api): make post api upload object default return 204 ( #5648 )
2024-06-04 06:18:47 -07:00
