Tom Crasset
c5f21b2b01
fix S3 per-user-directory Policy ( #6443 )
...
go: build dev binaries / cleanup (push) Has been cancelled
docker: build dev containers / build-dev-containers (push) Has been cancelled
End to End / FUSE Mount (push) Has been cancelled
go: build binary / Build (push) Has been cancelled
Ceph S3 tests / Ceph S3 tests (push) Has been cancelled
go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Has been cancelled
go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Has been cancelled
go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Has been cancelled
go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Has been cancelled
* fix S3 per-user-directory Policy
* Delete docker/config.json
* add tests
* remove logs
* undo modifications of weed/shell/command_volume_balance.go
* remove modifications of docker-compose
* fix failing test
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2025-01-17 01:03:17 -08:00
Lisandro Pin
eab2e0e112
ec.encode: Fix bug causing source volumes not being deleted after EC conversion. (#6447 )
...
This logic was originally part of `spreadEcShards()`, which got removed during
the unification effort with `ec.balance` (https://github.com/seaweedfs/seaweedfs/pull/6344 ),
accidentally breaking functionality in the process.
The commit restores the deletion code for EC'd volumes - with parallelization support.
2025-01-17 01:02:30 -08:00
Hadi Zamani
c7ae969c06
Add bucket's traffic metrics ( #6444 )
...
go: build dev binaries / cleanup (push) Waiting to run
go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Blocked by required conditions
go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Blocked by required conditions
go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Blocked by required conditions
go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Blocked by required conditions
docker: build dev containers / build-dev-containers (push) Waiting to run
End to End / FUSE Mount (push) Waiting to run
go: build binary / Build (push) Waiting to run
Ceph S3 tests / Ceph S3 tests (push) Waiting to run
helm: lint and test charts / lint-test (push) Has been cancelled
* Add bucket's traffic metrics
* Add bucket traffic to dashboards
* Fix bucket metrics help messages
* Fix variable names
2025-01-16 08:23:35 -08:00
Tom Crasset
aa299462f2
improve iam error handling ( #6446 )
...
* improve iam error handling
* Delete docker/test.py
2025-01-16 08:23:19 -08:00
ftong2020
2304d2b472
Bugfix s3 audit missing requester for PUT requests ( #6434 )
...
go: build dev binaries / cleanup (push) Waiting to run
go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Blocked by required conditions
go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Blocked by required conditions
go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Blocked by required conditions
go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Blocked by required conditions
docker: build dev containers / build-dev-containers (push) Waiting to run
End to End / FUSE Mount (push) Waiting to run
go: build binary / Build (push) Waiting to run
Ceph S3 tests / Ceph S3 tests (push) Waiting to run
fix s3 audit missing requster for PUT
2025-01-15 11:58:43 -08:00
dependabot[bot]
1609682ac2
chore(deps): bump github.com/getsentry/sentry-go from 0.30.0 to 0.31.1 ( #6437 )
...
go: build dev binaries / cleanup (push) Waiting to run
go: build dev binaries / build_dev_linux_windows (amd64, linux) (push) Blocked by required conditions
go: build dev binaries / build_dev_linux_windows (amd64, windows) (push) Blocked by required conditions
go: build dev binaries / build_dev_darwin (amd64, darwin) (push) Blocked by required conditions
go: build dev binaries / build_dev_darwin (arm64, darwin) (push) Blocked by required conditions
docker: build dev containers / build-dev-containers (push) Waiting to run
End to End / FUSE Mount (push) Waiting to run
go: build binary / Build (push) Waiting to run
Ceph S3 tests / Ceph S3 tests (push) Waiting to run
helm: lint and test charts / lint-test (push) Has been cancelled
* chore(deps): bump github.com/getsentry/sentry-go from 0.30.0 to 0.31.1
Bumps [github.com/getsentry/sentry-go](https://github.com/getsentry/sentry-go ) from 0.30.0 to 0.31.1.
- [Release notes](https://github.com/getsentry/sentry-go/releases )
- [Changelog](https://github.com/getsentry/sentry-go/blob/master/CHANGELOG.md )
- [Commits](https://github.com/getsentry/sentry-go/compare/v0.30.0...v0.31.1 )
---
updated-dependencies:
- dependency-name: github.com/getsentry/sentry-go
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* fix compilation
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: chrislu <chris.lu@gmail.com>
2025-01-13 12:50:33 -08:00
ftong2020
e7f2936dcc
fix force arg dropped during volume balance command ( #6432 )
2025-01-12 23:30:18 -08:00
dsd
da2a234b00
[weed] change -n to -force ( #6421 )
2025-01-08 09:57:18 -08:00
chrislu
39c1dde9df
3.81
2025-01-06 11:38:50 -08:00
Brad Murray
bc3640ee64
Update command_fs_merge_volumes.go ( #6406 )
2025-01-02 08:57:26 -08:00
Guang Jiong Lou
3b1ac77e1f
worm grace period and retention time support ( #6404 )
...
Signed-off-by: lou <alex1988@outlook.com>
2024-12-31 18:41:43 -08:00
Trim21
0594006c81
use same time format in volume volume UI and filter UI ( #6395 )
...
use same time format in volume server UI with filter UI
2024-12-26 19:23:29 -08:00
dsd
20cbc9e4eb
skip error while executing volume.fix.replication ( #6382 )
2024-12-20 07:36:13 -08:00
chrislu
d7dfb11be9
fix compilation
2024-12-19 19:26:28 -08:00
chrislu
de09a82a32
go fmt
2024-12-19 19:25:27 -08:00
chrislu
bfd59dd579
adjust import
2024-12-19 19:25:06 -08:00
chrislu
ec155022e7
"golang.org/x/exp/slices" => "slices" and go fmt
2024-12-19 19:25:06 -08:00
Warren Hodgkinson
a1a76ccb8c
Fix for DeleteMultipleObjectsHandler wrongly deleting parent folders ( #6380 )
...
What problem are we solving?
Fix : #6379
How are we solving the problem?
We check for the AllowEmptyFolders option prior to cascade
deleting parent folders in S3 DeleteMultipleObjectsHandler.
How is the PR tested?
We ran SeaweedFS in a Kubernetes Cluster with a joint Filer
and S3 server in one container, with leveldb2 as the filer storage,
and AllowEmptyFolders set to true.
When using the Distribution Registry as the S3 client, it calls the
DeleteMultipleObjectsHandler as part of the artifact upload process
(uploads to a temp location, then performs a copy and delete).
Without this fix, the deletion cascade deleted parent folder until
the entire contents of the bucket were gone.
With this fix, the existing content of the bucket remained, and the
newly uploaded content was added.
Checks
[ ] I have added unit tests if possible.
[ ] I will add related wiki document changes and link to this PR after merging.
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2024-12-19 18:00:08 -08:00
Lisandro Pin
4d91ec359b
Fix volume replica parallelization within ec.encode. ( #6377 )
...
See 826edd5d
2024-12-19 17:46:11 -08:00
Lisandro Pin
ba0707af64
Allow configuring the maximum number of concurrent tasks for EC parallelization. ( #6376 )
...
Follow-up to b0210df0
2024-12-18 13:26:26 -08:00
Lisandro Pin
44c48c929a
Parallelize volume replica operations within ec.encode. ( #6374 )
2024-12-18 11:59:48 -08:00
dsd
72af97162f
[shell] feat:stop vacuum immediately once volume.vacuum.disable was executed ( #6375 )
...
stop vacuum immediately once volume.vacuum.disable was executed
Co-authored-by: dsd <dsd2019@foxmail.com>
2024-12-18 11:56:40 -08:00
Lisandro Pin
9fbc4ea417
Rework shell.EcBalance()'s waitgroup code into a standalone type. ( #6373 )
...
Rework `shell.EcBalance()`'s waitgroup with errors code into a standalone type.
We'll re-use this for other EC jobs - for example, volume creation. Also fixes
potential concurrency issues when collecting error results.
2024-12-17 09:39:51 -08:00
zemul
e77e50886e
mount metacache add ttl ( #6360 )
...
* fix:mount deadlock
* fix
* feat: metaCache ttl
* Update weed/command/mount.go
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* fix InodeEntry
---------
Co-authored-by: zemul <zhouzemiao@ihuman.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2024-12-16 20:19:32 -08:00
coffeecloudgit
c531e2eb0e
Update disk_location_ec.go ( #6359 )
...
Add lock for l.ecVolumes
2024-12-15 22:05:47 -08:00
coffeecloudgit
7eb8257126
Update store.go ( #6358 )
...
Add lock for location.ecVolumes And single delete
2024-12-15 22:05:22 -08:00
Lisandro Pin
9b48ce0613
Parallelize EC shards balancing within racks ( #6354 )
...
Parallelize EC shards balancing within racks.
2024-12-15 13:36:23 -08:00
Lisandro Pin
926cfea3dc
Parallelize EC shards balancing across racks. ( #6352 )
2024-12-13 06:05:32 -08:00
Lisandro Pin
b81def5e5c
Parallelize EC balancing for racks. ( #6351 )
2024-12-13 05:33:53 -08:00
zouyixiong
d6f3e1970d
fix: filer may crash by bucketLastActiveTsNs concurrency access. ( #6350 )
2024-12-13 05:30:21 -08:00
Konstantin Lebedev
0a4b1909a2
[shell] only apply the balancing for writable volumes ( #6346 )
2024-12-13 01:10:00 -08:00
zouyixiong
9987a65e8a
fix: record and delete bucket metrics after inactive ( #6349 )
2024-12-12 20:34:02 -08:00
Lisandro Pin
b0210df081
Begin implementing EC balancing parallelization support. ( #6342 )
...
* Begin implementing EC balancing parallelization support.
Impacts both `ec.encode` and `ec.balance`,
* Nit: improve type naming.
* Make the goroutine workgroup handler for `EcBalance()` a bit smarter/error-proof.
* Nit: unify naming for `ecBalancer` wait group methods with the rest of the module.
* Fix concurrency bug.
* Fix whitespace after Gitlab automerge.
* Delete stray TODO.
2024-12-12 09:14:44 -08:00
Lisandro Pin
23ffbb083c
Limit EC re-balancing for ec.encode to relevant collections when a volume ID argument is provided. ( #6347 )
...
Limit EC re-balancing for `ec.encode` to relevant collections when a volume ID is provided.
2024-12-12 08:41:33 -08:00
Lisandro Pin
6320036c56
Delete legacy balancing code for ec.encode. ( #6344 )
2024-12-12 07:42:03 -08:00
Konstantin Lebedev
700b95304b
[shell] volume.list show only writable volumes ( #6338 )
...
* show only writable volumes
* fix import
2024-12-11 09:06:15 -08:00
Konstantin Lebedev
c37281735e
volume.list avoid output empty data center and rack and disk info ( #6341 )
2024-12-11 09:03:16 -08:00
Lisandro Pin
8c82c037b9
Unify the re-balancing logic for ec.encode with ec.balance. ( #6339 )
...
Among others, this enables recent changes related to topology aware
re-balancing at EC encoding time.
2024-12-10 13:30:13 -08:00
Konstantin Lebedev
ff1392f7f4
[shell] use constant for hdd of type ( #6337 )
...
use constant for hdd of type
2024-12-10 08:43:59 -08:00
Lisandro Pin
522a25790a
Remove average constraints when selecting nodes/racks to balance EC shards into. ( #6325 )
2024-12-06 09:00:06 -08:00
vadimartynov
914ff0e718
Replaced https.client.enabled with false in scaffold/security ( #6320 )
2024-12-05 10:39:28 -08:00
chrislu
f3dde99796
adjust error message
2024-12-05 09:33:50 -08:00
Lisandro Pin
34cdbdd279
Share common parameters for EC re-balancing functions under a single struct. ( #6319 )
...
TODO cleanup for https://github.com/seaweedfs/seaweedfs/discussions/6179 .
2024-12-05 09:00:46 -08:00
Lisandro Pin
edef485333
Account for replication placement settings when balancing EC shards within the same rack. ( #6317 )
...
* Account for replication placement settings when balancing EC shards within racks.
* Update help contents for `ec.balance`.
* Add a few more representative test cases for `pickEcNodeToBalanceShardsInto()`.
2024-12-04 10:47:51 -08:00
Lisandro Pin
351efa134d
Account for replication placement settings when balancing EC shards across racks. ( #6316 )
2024-12-04 09:00:55 -08:00
Konstantin Lebedev
b65eb2ec45
[security] reload whiteList on http seerver ( #6302 )
...
* reload whiteList
* white_list add to scaffold
2024-12-02 10:38:10 -08:00
Lisandro Pin
b2ba7d7408
Resolve replica placement for EC volumes from master server defaults. ( #6303 )
2024-12-02 08:44:07 -08:00
Lisandro Pin
9a741a61b1
Display details upon failures to re-balance EC shards racks. ( #6299 )
2024-11-28 08:42:41 -08:00
Lisandro Pin
559a1fd0f4
Improve EC shards rebalancing logic across nodes ( #6297 )
...
* Improve EC shards rebalancing logic across nodes.
- Favor target nodes with less preexisting shards, to ensure a fair distribution.
- Randomize selection when multiple possible target nodes are available.
- Add logic to account for replication settings when selecting target nodes (currently disabled).
* Fix minor test typo.
* Clarify internal error messages for `pickEcNodeToBalanceShardsInto()`.
2024-11-27 11:51:57 -08:00
steve.wei
88fa64a01a
change comment for expect header ( #6289 )
...
* remove case of expect
* Set the default value of the Expect header for compatibility.
2024-11-26 08:35:41 -08:00
