This is the first commit in a series of commits that aims at adding a primary balancer to Ceph and improving the current upmap balancer functionality. This first commit focuses on simplifying (refactoring) the code of `calc_pg_upmaps` so it is easier to change in the future. This PR keeps the existing functionality as-is and does not change anything but the code structure.
As part of the work is major refactoring of OSDMap::calc_pg_upmaps, the first thing is adding an --upmap-seed param to osdmaptool so test results can be compared without the random factor.
Other changes made:
- Divided sections of `OSDMap::calc_pg_upmaps` into their own separate functions
- Renamed tmp to tmp_osd_map
- Changed all the occurances of 'first' and 'second' in the function to more meaningful names.
Signed-off-by: Josh Salomon <josh.salomon@gmail.com>
(1) adding arrow/parquet to make(install is missing)
(2) s3select-operation contains 2 flows CSV and Parquet
(3) upon parquet-flow s3select processing engine is calling (via callback) to get-size and range-request, the range-requests are a-sync, thus the caller is waiting until notification.
(4) flow : execute --> s3select --(arrow layer)--> range-request --> GetObj::execute --> send_response_data --> notify-range-request --> (back-to) --> s3select
(5) on parquet flow the s3select is handling the response (using call-backs) because of aws-response-limitation (16mb)
add unique pointer (rgw_api); verify magic number for parquet objects; s3select module update
fix buffer-over-flow (copy range request)
change the range-request flow. now,it needs to use the callback parametrs (ofs & len) and not to use the element length
refactoring. seperate the CSV flow from the parquet flow, a phase before adding conditional build(depend on arrow package installation)
adding arrow/parquet installation to debian/control
align s3select repo with RGW (missing API"s, such as get_error_description)
undefined reference to arrow symbol
fix comment: using optional_yield by value
fix comments; remove future/promise
s3select: a leak fix
s3select: fixing result production
s3select,s3tests : parquet alignments
typo: git-remote --> git_remote
s3select: remove redundant comma(end of projections); bug fix in parquet flow upon aggregation queries
adding arrow/parquet
editorial. remove blank lines
s3select: merged with master(output serialization,presto alignments)
merging(not rebase) master functionlities into parquet branch
(*) a dedicated source-files for s3select operation.
(*) s3select-engine: fix leaks on parquet flows, enabling allocate csv_object and parquet_object on stack
(*) the csv_object and parquet object allocated on stack (no heap allocation)
move data-members from heap to stack allocation, refactoring, separate flows for CSV and parquet. s3select: bug fix
conditional build: upon arrow package is installed the parquet flow become visable, thus enables to process parquet object. in case the package is not installed only CSV is usable
remove redundant try/catch, s3select: fix compile warning
arrow-devel version should be higher than 4.0.0, where arrow::io::AsyncContext become depecrated
missing sudo; wrong url;move the rm -f arrow.list
replace codename with $(lsb_release -sc)
arrow version should be >= 4.0.0; iocontext not exists in namespace on lower versions
RGW points to s3select/master
s3select submodule
sudo --> $SUDO
Signed-off-by: gal salomon <gal.salomon@gmail.com>
When testing remember about `bluestore_max_blob_size` as it's
only 64 KB by default while the entire huge page-based pools
machinery targets far bigger scenrios (initially 4 MB!).
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
The idea here is to bring a pool of `mmap`-allocated,
constantly-sized buffers which would take precedence
over the 2 MB-aligned, THP-based mechanism. On first
attempt to acquire a 4 MB buffer, KernelDevice mmaps
`bdev_read_preallocated_huge_buffer_num` (default 128)
memory regions using the MAP_HUGETLB option. If this
fails, the entire process is aborted. Buffers, after
their life-times going over, are recycled with lock-
free queue shared across entire process.
Remember about allocating the appropriate number of
huge pages in the system! For instance:
```
echo 256 | sudo tee /proc/sys/vm/nr_hugepages
```
This commit bases on / cherry-picks with changes
897a4932bee5cba3641c18619cccd0ee945bfcf8.
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
crimson/os/seastore/../segment_manager: improve logs and validations
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Xuehan Xu <xuxuehan@360.cn>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
If wget fails (e.g. due to a certificate issue), it still creates
an empty file. Then this file is marked executable, ./"${SCRIPT}"
immediately returns 0 and run_xfstests_qemu.sh exits successfully
without running a single xfstest.
This started on Sep 30, 2021 with the expiration of Let's Encrypt
root certificate -- all qemu jobs with "test: qa/run_xfstests_qemu.sh"
just booted the VM for a couple of seconds and reported success.
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
This was a side-effect of consolidating the Zipper API, and resulted in
a large performance hit. Stats are only needed if they are requested,
so don't load them every time.
Signed-off-by: Daniel Gryniewicz <dang@redhat.com>
Update allocation file when we expand-device
Add the expended space to the allocator and then force an update to the allocation file
There is also a new standalone test case for expand
Fixes: https://tracker.ceph.com/issues/53699
Signed-off-by: Gabriel Benhanokh <gbenhano@redhat.com>
Addition of a new column in PG dump, OBJECTS_SCRUBBED, which keeps track of the number of objects scrubbed.
Signed-off-by: Aishwarya Mathuria <amathuri@redhat.com>
test/crimson: fix a race condition in SeastarRunner
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
Capturing the RocksDB version number in Telemetry would allow us to check that users are using the appropriate RocksDB version for their Ceph cluster. For instance, if a user is working in a Pacific cluster, but their RocksDB version is meant for Nautilus, that might be a problem.
It is strucured as "rocksdb_stats" --> "version" in anticipation of more stats that can will be added under "rocksdb_stats".
Signed-off-by: Laura Flores <lflores@redhat.com>
It is only necessary here to link the rocksdb include directory
since the mgr simply needs access to the rocksdb version numbers.
Signed-off-by: Laura Flores <lflores@redhat.com>
Co-authored-by: Kefu Chai <tchaikov@gmail.com>
Co-authored-by: Adam Kupczyk <akupczyk@redhat.com>
