From 18b39f1f34ed3d38ad29c58ca073634ebe024d5c Mon Sep 17 00:00:00 2001 From: velonica0 <47554626+velonica0@users.noreply.github.com> Date: Sun, 26 Jul 2026 07:31:52 +0800 Subject: [PATCH] RVV: switch scalable universal intrinsics from LMUL=2 to LMUL=1 (#29493) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * dnn:rvv: restore the fixed block size C0 == 8 #29304 made the net-wide block size follow the RVV vector width (defaultC0 = max(8, vlanes())), so the blocked memory layout of every tensor depended on the host's VLEN and on the LMUL of the universal intrinsics in core. The same model gets a different layout on a 128-, 256- or 1024-bit machine. Restore the fixed default of 8, which is the mainstream, well-tested setting on every platform. C0/K0 become compile-time constants again on RVV as they already were elsewhere, so the runtime block-size plumbing and the vlanes()-based scratch sizing are dropped, and getConvFunc_ no longer advertises block sizes the kernels cannot handle. The blocked kernels currently require C0 == vlanes(), which this alone does not satisfy; the next commit removes that requirement. * dnn:rvv: pin the vector length to the channel block The blocked kernels assumed one channel block is exactly one vector (C0 == vlanes()). That is not a property of the layout, it is a property of the hardware, so the kernels only worked where the two happened to coincide: with C0 fixed at 8 they break wherever vlanes() != 8 -- under-filling the block when the vector is narrower (silently leaving output channels unwritten) and overrunning it when the vector is wider (CV_Assert). The universal intrinsics cannot express this: v_float32 is always vlanes() wide and there is no way to make it span exactly C0 elements. So the RVV paths drop to native intrinsics and set the vector length explicitly: const size_t _vl = __riscv_vsetvl_e32m2(C0); // == 8 on every VLEN >= 128 LMUL=2 is required, not incidental: vlmax(e32,m2) = VLEN/32*2 >= 8 even at VLEN=128, the narrowest RVV configuration, so _vl == C0 on every RVV 1.0 core. One vector then spans precisely one channel block regardless of VLEN, and the C0-vs-vlanes asserts and VLEN-dependent branches are removed rather than extended. The existing int8 kernels already choose LMUL=2 for the same reason (conv2_int8_kernels.simd.hpp). This is a native per-kernel choice and is independent of the LMUL the universal intrinsics are built with; dnn no longer reads the vector width of intrin_rvv_scalable.hpp at all. conv, deconv, maxpool, avgpool and depthwise use native intrinsics. batch_norm stays portable and instead replicates the C0-wide coefficient pattern across the vector, which keeps full lane utilisation where a fixed vl would idle most of it. The x86/ARM paths are restored to their pre-#29304 form and are unchanged. Tested on SpaceMIT K3 (X100 VLEN=256, A100 VLEN=1024), opencv_test_dnn: no regressions against upstream on either core. * core:rvv: switch scalable universal intrinsics from LMUL=2 to LMUL=1 Move the RVV universal-intrinsic types (v_uint8..v_float64) from m2 (paired registers, 16 usable) to m1 (individual registers, 32 usable) per #29454, to relieve register pressure in the complex kernels that rely on universal intrinsics. Simple kernels that regress can take native RVV branches; the complex ones cannot, so the default should suit them. Only intrin_rvv_scalable.hpp changes. The widening/narrowing cascades and max_nlanes are adjusted, and helpers that spell an LMUL into the intrinsic name shift down one step (m2->m1, m1->mf2): v_popcount, v_lut (vector and f64 paths), the f64 dot-product reduction (#27003), v_matmul, and the byte-indexed v_lut added by #29250, whose __riscv_vluxei8_v_u8m2 no longer matches the m1 base type. VLEN=1024 needs no change: under m1, max_nlanes = CV_RVV_MAX_VLEN/SZ equals vlanes() at 1024, and CV_RVV_MAX_VLEN already defaults to 1024. Verified on a 1024-bit core. Tested on SpaceMIT K3, which exposes both a 256-bit (X100) and a 1024-bit (A100) core. opencv_test_core: 5605/5612, the 7 failures (filestorage_base64, globbing) non-SIMD and failing identically on the m2 baseline. Perf (x-factor = m2/m1, >1 = m1 faster), median >= 300us: core 0.966 -> 0.996 and imgproc 0.950 -> 1.038 as VLEN goes 256 -> 1024, i.e. m2's width advantage disappears once both are wide while m1's register headroom persists. m1 wins resize (1.50/1.41), Exp (1.28), gaussianBlur (1.23), norm (1.22), scharr (1.20); m2 still wins the light memory-streaming kernels addWeighted (0.76), compare (0.82) and dot (0.87), which are the natural candidates for native RVV branches. * Revert "dnn:rvv: pin the vector length to the channel block" This reverts commit 40587c6dd9cc6621ce54c6a2bf902ee2a5b04fd9. * dnn:rvv: disable CV_SIMD_SCALABLE in the new-engine conv kernel Under the m1 switch the RVV conv kernel breaks at VLEN=128: it assumes K0 == vlanes(), but m1 makes vlanes()=4 while the block stays C0=K0=8, so it computes only half of each output block. Temporarily disable the RVV (CV_SIMD_SCALABLE) branches so conv runs scalar on RVV, exactly as #29180 did for the other new-engine layers. The vectorized path will be restored in a follow-up. Only conv is affected: maxpool/avgpool/depthwise/deconv/batch_norm already handle C0 == 2*vlanes (the SSE/NEON case) and stay on their universal-intrinsic paths. * dnn:rvv: document the supported VLEN range and the v_setvlmax follow-up With the fixed C0=8 the RVV blocked kernels cover VLEN=128 and 256 (C0 >= vlanes); VLEN>=512 (C0 < vlanes) is not yet handled. Add TODO notes recording this and pointing at the planned fix: a portable v_setvlmax(n) universal intrinsic that caps the vector to the block (no-op on fixed-width ISAs, sets the vl/mask on RVV/SVE), with an optimized multi-pixel variant as a later step, possibly an RVV HAL for the new dnn engine. See the #29493 discussion. * dnn:rvv: disable CV_SIMD_SCALABLE in max/avg pooling and depthwise These three blocked kernels assert (C0 == nlanes || ...) at VLEN>=512, where the fixed C0=8 is narrower than the vector register — breaking whole networks (AlexNet/ResNet/YOLO) on wide RVV cores, though CI (VLEN=128) is unaffected. Disable their RVV (CV_SIMD_SCALABLE) paths so they run scalar on RVV at every VLEN, matching #29180's scope. Verified on a SpaceMIT K3: this removes the 92 VLEN=1024 assert failures, with no change at VLEN=128/256. deconv and batch_norm are left enabled: they degrade to scalar at VLEN>=512 (deconv's C0 % vlanes guard, batch_norm's vector-loop bound) without asserting, so they stay correct and keep their 128/256 vectorization. All of these re-vectorize uniformly via the v_setvlmax(n) follow-up. --- .../opencv2/core/hal/intrin_rvv_scalable.hpp | 902 +++++++++--------- modules/dnn/src/layers/avgpool_layer.cpp | 11 +- .../cpu_kernels/conv2_depthwise.simd.hpp | 27 +- .../layers/cpu_kernels/conv2_kernels.simd.hpp | 35 +- modules/dnn/src/layers/maxpool_layer.cpp | 11 +- modules/dnn/src/net_impl.cpp | 11 +- modules/dnn/src/net_impl2.cpp | 11 - 7 files changed, 508 insertions(+), 500 deletions(-) diff --git a/modules/core/include/opencv2/core/hal/intrin_rvv_scalable.hpp b/modules/core/include/opencv2/core/hal/intrin_rvv_scalable.hpp index 2afc277ee1..d4495b88b2 100644 --- a/modules/core/include/opencv2/core/hal/intrin_rvv_scalable.hpp +++ b/modules/core/include/opencv2/core/hal/intrin_rvv_scalable.hpp @@ -39,21 +39,21 @@ CV_CPU_OPTIMIZATION_HAL_NAMESPACE_BEGIN #endif -using v_uint8 = vuint8m2_t; -using v_int8 = vint8m2_t; -using v_uint16 = vuint16m2_t; -using v_int16 = vint16m2_t; -using v_uint32 = vuint32m2_t; -using v_int32 = vint32m2_t; -using v_uint64 = vuint64m2_t; -using v_int64 = vint64m2_t; +using v_uint8 = vuint8m1_t; +using v_int8 = vint8m1_t; +using v_uint16 = vuint16m1_t; +using v_int16 = vint16m1_t; +using v_uint32 = vuint32m1_t; +using v_int32 = vint32m1_t; +using v_uint64 = vuint64m1_t; +using v_int64 = vint64m1_t; #if CV_SIMD_SCALABLE_FP16 -using v_float16 = vfloat16m2_t; +using v_float16 = vfloat16m1_t; #endif -using v_float32 = vfloat32m2_t; +using v_float32 = vfloat32m1_t; #if CV_SIMD_SCALABLE_64F -using v_float64 = vfloat64m2_t; +using v_float64 = vfloat64m1_t; #endif using uchar = unsigned char; @@ -73,9 +73,9 @@ struct VTraits \ { \ static inline int vlanes() { return __riscv_vsetvlmax_##SUF(); } \ using lane_type = TYP; \ - static const int max_nlanes = CV_RVV_MAX_VLEN/SZ*2; \ + static const int max_nlanes = CV_RVV_MAX_VLEN/SZ; \ }; -// `max_nlanes` is multiplied by 2 because of using LMUL=2 (m2) +// `max_nlanes` uses LMUL=1 (m1): one universal-intrinsic vector == one vector register OPENCV_HAL_IMPL_RVV_TRAITS(vint8m1_t, int8_t, e8m1, 8) OPENCV_HAL_IMPL_RVV_TRAITS(vint8m2_t, int8_t, e8m2, 8) @@ -180,11 +180,11 @@ inline double v_get0(const v_float64& v) \ #define OPENCV_HAL_IMPL_RVV_INIT_INTEGER(_Tpvec, _Tp, suffix1, suffix2, vl) \ inline v_##_Tpvec v_setzero_##suffix1() \ { \ - return __riscv_vmv_v_x_##suffix2##m2(0, vl); \ + return __riscv_vmv_v_x_##suffix2##m1(0, vl); \ } \ inline v_##_Tpvec v_setall_##suffix1(_Tp v) \ { \ - return __riscv_vmv_v_x_##suffix2##m2(v, vl); \ + return __riscv_vmv_v_x_##suffix2##m1(v, vl); \ } \ template <> inline v_##_Tpvec v_setzero_() \ { \ @@ -207,11 +207,11 @@ OPENCV_HAL_IMPL_RVV_INIT_INTEGER(int64, int64, s64, i64, VTraits::vlane #define OPENCV_HAL_IMPL_RVV_INIT_FP(_Tpv, _Tp, suffix, vl) \ inline v_##_Tpv v_setzero_##suffix() \ { \ - return __riscv_vfmv_v_f_##suffix##m2(0, vl); \ + return __riscv_vfmv_v_f_##suffix##m1(0, vl); \ } \ inline v_##_Tpv v_setall_##suffix(_Tp v) \ { \ - return __riscv_vfmv_v_f_##suffix##m2(v, vl); \ + return __riscv_vfmv_v_f_##suffix##m1(v, vl); \ } \ template <> inline v_##_Tpv v_setzero_() \ { \ @@ -225,23 +225,15 @@ template <> inline v_##_Tpv v_setall_(_Tp v) \ #if CV_SIMD_SCALABLE_FP16 inline v_float16 v_setzero_f16() { - return __riscv_vfmv_v_f_f16m2(0, VTraits::vlanes()); + return __riscv_vfmv_v_f_f16m1(0, VTraits::vlanes()); } inline v_float16 v_setall_f16(float v) // In some cases we may use v_setall_f16(1.0f) { - return __riscv_vfmv_v_f_f16m2((_Float16)v, VTraits::vlanes()); + return __riscv_vfmv_v_f_f16m1((_Float16)v, VTraits::vlanes()); } inline v_float16 v_setall_f16(hfloat v) { - return __riscv_vfmv_v_f_f16m2((_Float16)v, VTraits::vlanes()); -} -template <> inline v_float16 v_setzero_() \ -{ \ - return v_setzero_f16(); \ -} \ -template <> inline v_float16 v_setall_(hfloat v) \ -{ \ - return v_setall_f16(v); \ + return __riscv_vfmv_v_f_f16m1((_Float16)v, VTraits::vlanes()); } #endif OPENCV_HAL_IMPL_RVV_INIT_FP(float32, float, f32, VTraits::vlanes()) @@ -274,11 +266,11 @@ OPENCV_HAL_IMPL_RVV_NOTHING_REINTERPRET(float64, f64) #define OPENCV_HAL_IMPL_RVV_NATIVE_REINTERPRET(_Tpvec1, _Tpvec2, suffix1, suffix2, nsuffix1, nsuffix2) \ inline v_##_Tpvec1 v_reinterpret_as_##suffix1(const v_##_Tpvec2& v) \ { \ - return v_##_Tpvec1(__riscv_vreinterpret_v_##nsuffix2##m2_##nsuffix1##m2(v));\ + return v_##_Tpvec1(__riscv_vreinterpret_v_##nsuffix2##m1_##nsuffix1##m1(v));\ } \ inline v_##_Tpvec2 v_reinterpret_as_##suffix2(const v_##_Tpvec1& v) \ { \ - return v_##_Tpvec2(__riscv_vreinterpret_v_##nsuffix1##m2_##nsuffix2##m2(v));\ + return v_##_Tpvec2(__riscv_vreinterpret_v_##nsuffix1##m1_##nsuffix2##m1(v));\ } OPENCV_HAL_IMPL_RVV_NATIVE_REINTERPRET(uint8, int8, u8, s8, u8, i8) @@ -312,11 +304,11 @@ OPENCV_HAL_IMPL_RVV_NATIVE_REINTERPRET(int32, int64, s32, s64, i32, i64) #define OPENCV_HAL_IMPL_RVV_TWO_TIMES_REINTERPRET(_Tpvec1, _Tpvec2, suffix1, suffix2, nsuffix1, nsuffix2, width1, width2) \ inline v_##_Tpvec1 v_reinterpret_as_##suffix1(const v_##_Tpvec2& v) \ { \ - return __riscv_vreinterpret_v_##nsuffix1##width2##m2_##nsuffix1##width1##m2(__riscv_vreinterpret_v_##nsuffix2##width2##m2_##nsuffix1##width2##m2(v));\ + return __riscv_vreinterpret_v_##nsuffix1##width2##m1_##nsuffix1##width1##m1(__riscv_vreinterpret_v_##nsuffix2##width2##m1_##nsuffix1##width2##m1(v));\ } \ inline v_##_Tpvec2 v_reinterpret_as_##suffix2(const v_##_Tpvec1& v) \ { \ - return __riscv_vreinterpret_v_##nsuffix1##width2##m2_##nsuffix2##width2##m2(__riscv_vreinterpret_v_##nsuffix1##width1##m2_##nsuffix1##width2##m2(v));\ + return __riscv_vreinterpret_v_##nsuffix1##width2##m1_##nsuffix2##width2##m1(__riscv_vreinterpret_v_##nsuffix1##width1##m1_##nsuffix1##width2##m1(v));\ } OPENCV_HAL_IMPL_RVV_TWO_TIMES_REINTERPRET(uint8, int16, u8, s16, u, i, 8, 16) @@ -356,22 +348,22 @@ OPENCV_HAL_IMPL_RVV_TWO_TIMES_REINTERPRET(int32, float64, s32, f64, i, f, 32, 64 #if CV_SIMD_SCALABLE_FP16 inline v_float16 v_reinterpret_as_f16(const v_float64& v) \ { \ - return __riscv_vreinterpret_v_u16m2_f16m2(__riscv_vreinterpret_v_u64m2_u16m2(__riscv_vreinterpret_v_f64m2_u64m2(v)));\ + return __riscv_vreinterpret_v_u16m1_f16m1(__riscv_vreinterpret_v_u64m1_u16m1(__riscv_vreinterpret_v_f64m1_u64m1(v)));\ } inline v_float64 v_reinterpret_as_f64(const v_float16& v) \ { \ - return __riscv_vreinterpret_v_u64m2_f64m2(__riscv_vreinterpret_v_u16m2_u64m2(__riscv_vreinterpret_v_f16m2_u16m2(v)));\ + return __riscv_vreinterpret_v_u64m1_f64m1(__riscv_vreinterpret_v_u16m1_u64m1(__riscv_vreinterpret_v_f16m1_u16m1(v)));\ } #endif inline v_float32 v_reinterpret_as_f32(const v_float64& v) \ { \ - return __riscv_vreinterpret_v_u32m2_f32m2(__riscv_vreinterpret_v_u64m2_u32m2(__riscv_vreinterpret_v_f64m2_u64m2(v)));\ + return __riscv_vreinterpret_v_u32m1_f32m1(__riscv_vreinterpret_v_u64m1_u32m1(__riscv_vreinterpret_v_f64m1_u64m1(v)));\ } inline v_float64 v_reinterpret_as_f64(const v_float32& v) \ { \ - return __riscv_vreinterpret_v_u64m2_f64m2(__riscv_vreinterpret_v_u32m2_u64m2(__riscv_vreinterpret_v_f32m2_u32m2(v)));\ + return __riscv_vreinterpret_v_u64m1_f64m1(__riscv_vreinterpret_v_u32m1_u64m1(__riscv_vreinterpret_v_f32m1_u32m1(v)));\ } #endif @@ -443,23 +435,23 @@ OPENCV_HAL_IMPL_RVV_EXTRACT(v_float64, double, VTraits::vlanes()) #define OPENCV_HAL_IMPL_RVV_LOADSTORE_OP(_Tpvec, _nTpvec, _Tp, hvl, vl, width, suffix) \ inline _Tpvec v_load(const _Tp* ptr) \ { \ - return __riscv_vle##width##_v_##suffix##m2(ptr, vl); \ + return __riscv_vle##width##_v_##suffix##m1(ptr, vl); \ } \ inline _Tpvec v_load_aligned(const _Tp* ptr) \ { \ - return __riscv_vle##width##_v_##suffix##m2(ptr, vl); \ + return __riscv_vle##width##_v_##suffix##m1(ptr, vl); \ } \ inline void v_store(_Tp* ptr, const _Tpvec& a, hal::StoreMode /*mode*/) \ { \ - __riscv_vse##width##_v_##suffix##m2(ptr, a, vl); \ + __riscv_vse##width##_v_##suffix##m1(ptr, a, vl); \ } \ inline _Tpvec v_load_low(const _Tp* ptr) \ { \ - return __riscv_vle##width##_v_##suffix##m2(ptr, hvl); \ + return __riscv_vle##width##_v_##suffix##m1(ptr, hvl); \ } \ inline _Tpvec v_load_halves(const _Tp* ptr0, const _Tp* ptr1) \ { \ - return __riscv_vslideup(__riscv_vle##width##_v_##suffix##m2(ptr0, hvl), __riscv_vle##width##_v_##suffix##m2(ptr1, hvl), hvl, vl); \ + return __riscv_vslideup(__riscv_vle##width##_v_##suffix##m1(ptr0, hvl), __riscv_vle##width##_v_##suffix##m1(ptr1, hvl), hvl, vl); \ } \ inline void v_store(_Tp* ptr, const _Tpvec& a) \ { \ @@ -479,7 +471,7 @@ inline void v_store_low(_Tp* ptr, const _Tpvec& a) \ } \ inline void v_store_high(_Tp* ptr, const _Tpvec& a) \ { \ - __riscv_vse##width(ptr, __riscv_vslidedown_vx_##suffix##m2(a, hvl, vl), hvl); \ + __riscv_vse##width(ptr, __riscv_vslidedown_vx_##suffix##m1(a, hvl, vl), hvl); \ } \ template \ _Tpvec v_load_##suffix(Targs... nScalars) \ @@ -490,23 +482,23 @@ _Tpvec v_load_##suffix(Targs... nScalars) \ #define OPENCV_HAL_IMPL_RVV_LOADSTORE_OP_FP16(_Tpvec, _nTpvec, _Tp, hvl, vl, width, suffix) \ inline _Tpvec v_load(const _Tp* ptr) \ { \ - return __riscv_vle##width##_v_##suffix##m2((_Float16*)ptr, vl); \ + return __riscv_vle##width##_v_##suffix##m1((_Float16*)ptr, vl); \ } \ inline _Tpvec v_load_aligned(const _Tp* ptr) \ { \ - return __riscv_vle##width##_v_##suffix##m2((_Float16*)ptr, vl); \ + return __riscv_vle##width##_v_##suffix##m1((_Float16*)ptr, vl); \ } \ inline void v_store(_Tp* ptr, const _Tpvec& a, hal::StoreMode /*mode*/) \ { \ - __riscv_vse##width##_v_##suffix##m2((_Float16*)ptr, a, vl); \ + __riscv_vse##width##_v_##suffix##m1((_Float16*)ptr, a, vl); \ } \ inline _Tpvec v_load_low(const _Tp* ptr) \ { \ - return __riscv_vle##width##_v_##suffix##m2((_Float16*)ptr, hvl); \ + return __riscv_vle##width##_v_##suffix##m1((_Float16*)ptr, hvl); \ } \ inline _Tpvec v_load_halves(const _Tp* ptr0, const _Tp* ptr1) \ { \ - return __riscv_vslideup(__riscv_vle##width##_v_##suffix##m2((_Float16*)ptr0, hvl), __riscv_vle##width##_v_##suffix##m2((_Float16*)ptr1, hvl), hvl, vl); \ + return __riscv_vslideup(__riscv_vle##width##_v_##suffix##m1((_Float16*)ptr0, hvl), __riscv_vle##width##_v_##suffix##m1((_Float16*)ptr1, hvl), hvl, vl); \ } \ inline void v_store(_Tp* ptr, const _Tpvec& a) \ { \ @@ -526,24 +518,24 @@ inline void v_store_low(_Tp* ptr, const _Tpvec& a) \ } \ inline void v_store_high(_Tp* ptr, const _Tpvec& a) \ { \ - __riscv_vse##width((_Float16*)ptr, __riscv_vslidedown_vx_##suffix##m2(a, hvl, vl), hvl); \ + __riscv_vse##width((_Float16*)ptr, __riscv_vslidedown_vx_##suffix##m1(a, hvl, vl), hvl); \ } -OPENCV_HAL_IMPL_RVV_LOADSTORE_OP(v_uint8, vuint8m2_t, uchar, VTraits::vlanes() / 2, VTraits::vlanes(), 8, u8) -OPENCV_HAL_IMPL_RVV_LOADSTORE_OP(v_int8, vint8m2_t, schar, VTraits::vlanes() / 2, VTraits::vlanes(), 8, i8) -OPENCV_HAL_IMPL_RVV_LOADSTORE_OP(v_uint16, vuint16m2_t, ushort, VTraits::vlanes() / 2, VTraits::vlanes(), 16, u16) -OPENCV_HAL_IMPL_RVV_LOADSTORE_OP(v_int16, vint16m2_t, short, VTraits::vlanes() / 2, VTraits::vlanes(), 16, i16) -OPENCV_HAL_IMPL_RVV_LOADSTORE_OP(v_uint32, vuint32m2_t, unsigned int, VTraits::vlanes() / 2, VTraits::vlanes(), 32, u32) -OPENCV_HAL_IMPL_RVV_LOADSTORE_OP(v_int32, vint32m2_t, int, VTraits::vlanes() / 2, VTraits::vlanes(), 32, i32) -OPENCV_HAL_IMPL_RVV_LOADSTORE_OP(v_uint64, vuint64m2_t, uint64, VTraits::vlanes() / 2, VTraits::vlanes(), 64, u64) -OPENCV_HAL_IMPL_RVV_LOADSTORE_OP(v_int64, vint64m2_t, int64, VTraits::vlanes() / 2, VTraits::vlanes(), 64, i64) +OPENCV_HAL_IMPL_RVV_LOADSTORE_OP(v_uint8, vuint8m1_t, uchar, VTraits::vlanes() / 2, VTraits::vlanes(), 8, u8) +OPENCV_HAL_IMPL_RVV_LOADSTORE_OP(v_int8, vint8m1_t, schar, VTraits::vlanes() / 2, VTraits::vlanes(), 8, i8) +OPENCV_HAL_IMPL_RVV_LOADSTORE_OP(v_uint16, vuint16m1_t, ushort, VTraits::vlanes() / 2, VTraits::vlanes(), 16, u16) +OPENCV_HAL_IMPL_RVV_LOADSTORE_OP(v_int16, vint16m1_t, short, VTraits::vlanes() / 2, VTraits::vlanes(), 16, i16) +OPENCV_HAL_IMPL_RVV_LOADSTORE_OP(v_uint32, vuint32m1_t, unsigned int, VTraits::vlanes() / 2, VTraits::vlanes(), 32, u32) +OPENCV_HAL_IMPL_RVV_LOADSTORE_OP(v_int32, vint32m1_t, int, VTraits::vlanes() / 2, VTraits::vlanes(), 32, i32) +OPENCV_HAL_IMPL_RVV_LOADSTORE_OP(v_uint64, vuint64m1_t, uint64, VTraits::vlanes() / 2, VTraits::vlanes(), 64, u64) +OPENCV_HAL_IMPL_RVV_LOADSTORE_OP(v_int64, vint64m1_t, int64, VTraits::vlanes() / 2, VTraits::vlanes(), 64, i64) #if CV_SIMD_SCALABLE_FP16 -OPENCV_HAL_IMPL_RVV_LOADSTORE_OP_FP16(v_float16, vfloat16m2_t, hfloat, VTraits::vlanes() /2 , VTraits::vlanes(), 16, f16) +OPENCV_HAL_IMPL_RVV_LOADSTORE_OP_FP16(v_float16, vfloat16m1_t, hfloat, VTraits::vlanes() /2 , VTraits::vlanes(), 16, f16) #endif -OPENCV_HAL_IMPL_RVV_LOADSTORE_OP(v_float32, vfloat32m2_t, float, VTraits::vlanes() /2 , VTraits::vlanes(), 32, f32) +OPENCV_HAL_IMPL_RVV_LOADSTORE_OP(v_float32, vfloat32m1_t, float, VTraits::vlanes() /2 , VTraits::vlanes(), 32, f32) #if CV_SIMD_SCALABLE_64F -OPENCV_HAL_IMPL_RVV_LOADSTORE_OP(v_float64, vfloat64m2_t, double, VTraits::vlanes() / 2, VTraits::vlanes(), 64, f64) +OPENCV_HAL_IMPL_RVV_LOADSTORE_OP(v_float64, vfloat64m1_t, double, VTraits::vlanes() / 2, VTraits::vlanes(), 64, f64) #endif template ::max_nlanes> @@ -587,22 +579,22 @@ inline _Tpvec v_lut(const _Tp* tab, const int* idx) \ auto vidx = __riscv_vmul(__riscv_vreinterpret_u32##suffix(__riscv_vle32_v_i32##suffix(idx, VTraits<_Tpvec>::vlanes())), sizeof(_Tp), VTraits<_Tpvec>::vlanes()); \ return __riscv_vloxei32(tab, vidx, VTraits<_Tpvec>::vlanes()); \ } -OPENCV_HAL_IMPL_RVV_LUT(v_int8, schar, m8) #define OPENCV_HAL_IMPL_RVV_LUT_FP16(_Tpvec, _Tp, suffix) \ inline _Tpvec v_lut(const _Tp* tab, const int* idx) \ { \ auto vidx = __riscv_vmul(__riscv_vreinterpret_u32##suffix(__riscv_vle32_v_i32##suffix(idx, VTraits<_Tpvec>::vlanes())), sizeof(_Tp), VTraits<_Tpvec>::vlanes()); \ return __riscv_vloxei32((_Float16*)tab, vidx, VTraits<_Tpvec>::vlanes()); \ } -OPENCV_HAL_IMPL_RVV_LUT(v_int16, short, m4) -OPENCV_HAL_IMPL_RVV_LUT(v_int32, int, m2) -OPENCV_HAL_IMPL_RVV_LUT(v_int64, int64_t, m1) +OPENCV_HAL_IMPL_RVV_LUT(v_int8, schar, m4) +OPENCV_HAL_IMPL_RVV_LUT(v_int16, short, m2) +OPENCV_HAL_IMPL_RVV_LUT(v_int32, int, m1) +OPENCV_HAL_IMPL_RVV_LUT(v_int64, int64_t, mf2) #if CV_SIMD_SCALABLE_FP16 -OPENCV_HAL_IMPL_RVV_LUT_FP16(v_float16, hfloat, m4) +OPENCV_HAL_IMPL_RVV_LUT_FP16(v_float16, hfloat, m2) #endif -OPENCV_HAL_IMPL_RVV_LUT(v_float32, float, m2) +OPENCV_HAL_IMPL_RVV_LUT(v_float32, float, m1) #if CV_SIMD_SCALABLE_64F -OPENCV_HAL_IMPL_RVV_LUT(v_float64, double, m1) +OPENCV_HAL_IMPL_RVV_LUT(v_float64, double, mf2) #endif #define OPENCV_HAL_IMPL_RVV_LUT_PAIRS(_Tpvec, _Tp, _TpCast, suffix1, suffix2, v_trunc) \ @@ -617,16 +609,16 @@ inline _Tpvec v_lut_pairs(const _Tp* tab, const int* idx) \ auto vidx = __riscv_vmul(vid, sizeof(_Tp), VTraits<_Tpvec>::vlanes()); \ return __riscv_vloxei32((_TpCast *)tab, vidx, VTraits<_Tpvec>::vlanes()); \ } -OPENCV_HAL_IMPL_RVV_LUT_PAIRS(v_int8, schar, schar, m4, m8, OPENCV_HAL_NOP) -OPENCV_HAL_IMPL_RVV_LUT_PAIRS(v_int16, short, short, m2, m4, OPENCV_HAL_NOP) +OPENCV_HAL_IMPL_RVV_LUT_PAIRS(v_int8, schar, schar, m2, m4, OPENCV_HAL_NOP) +OPENCV_HAL_IMPL_RVV_LUT_PAIRS(v_int16, short, short, m1, m2, OPENCV_HAL_NOP) #if CV_SIMD_SCALABLE_FP16 -OPENCV_HAL_IMPL_RVV_LUT_PAIRS(v_float16, hfloat, _Float16, m2, m4, OPENCV_HAL_NOP) +OPENCV_HAL_IMPL_RVV_LUT_PAIRS(v_float16, hfloat, _Float16, m1, m2, OPENCV_HAL_NOP) #endif -OPENCV_HAL_IMPL_RVV_LUT_PAIRS(v_int32, int, int, m1, m2, OPENCV_HAL_NOP) -OPENCV_HAL_IMPL_RVV_LUT_PAIRS(v_float32, float, float, m1, m2, OPENCV_HAL_NOP) -OPENCV_HAL_IMPL_RVV_LUT_PAIRS(v_int64, int64_t, int64_t, m1, m2, __riscv_vlmul_trunc_u32m1) +OPENCV_HAL_IMPL_RVV_LUT_PAIRS(v_int32, int, int, mf2, m1, OPENCV_HAL_NOP) +OPENCV_HAL_IMPL_RVV_LUT_PAIRS(v_float32, float, float, mf2, m1, OPENCV_HAL_NOP) +OPENCV_HAL_IMPL_RVV_LUT_PAIRS(v_int64, int64_t, int64_t, mf2, m1, __riscv_vlmul_trunc_u32mf2) #if CV_SIMD_SCALABLE_64F -OPENCV_HAL_IMPL_RVV_LUT_PAIRS(v_float64, double, double, m1, m2, __riscv_vlmul_trunc_u32m1) +OPENCV_HAL_IMPL_RVV_LUT_PAIRS(v_float64, double, double, mf2, m1, __riscv_vlmul_trunc_u32mf2) #endif @@ -652,18 +644,18 @@ inline _Tpvec v_lut_quads(const _Tp* tab, const int* idx) \ auto vidx = __riscv_vmul(vid, sizeof(_Tp), VTraits<_Tpvec>::vlanes()); \ return __riscv_vloxei32((_TpCast *)tab, vidx, VTraits<_Tpvec>::vlanes()); \ } -OPENCV_HAL_IMPL_RVV_LUT_QUADS(v_int8, schar, schar, m2, m4, m8, OPENCV_HAL_NOP) -OPENCV_HAL_IMPL_RVV_LUT_QUADS(v_int16, short, short, m1 , m2, m4, OPENCV_HAL_NOP) -OPENCV_HAL_IMPL_RVV_LUT_QUADS(v_int32, int, int, m1, m2, m2, __riscv_vlmul_trunc_u32m1) +OPENCV_HAL_IMPL_RVV_LUT_QUADS(v_int8, schar, schar, m1, m2, m4, OPENCV_HAL_NOP) +OPENCV_HAL_IMPL_RVV_LUT_QUADS(v_int16, short, short, mf2 , m1, m2, OPENCV_HAL_NOP) +OPENCV_HAL_IMPL_RVV_LUT_QUADS(v_int32, int, int, mf2, m1, m1, __riscv_vlmul_trunc_u32mf2) #if CV_SIMD_SCALABLE_FP16 -OPENCV_HAL_IMPL_RVV_LUT_QUADS(v_float16, hfloat, _Float16, m1 , m2, m4, OPENCV_HAL_NOP) +OPENCV_HAL_IMPL_RVV_LUT_QUADS(v_float16, hfloat, _Float16, mf2 , m1, m2, OPENCV_HAL_NOP) #endif -OPENCV_HAL_IMPL_RVV_LUT_QUADS(v_float32, float, float, m1, m2, m2, __riscv_vlmul_trunc_u32m1) +OPENCV_HAL_IMPL_RVV_LUT_QUADS(v_float32, float, float, mf2, m1, m1, __riscv_vlmul_trunc_u32mf2) #define OPENCV_HAL_IMPL_RVV_LUT_VEC(_Tpvec, _Tp) \ inline _Tpvec v_lut(const _Tp* tab, const v_int32& vidx) \ { \ - v_uint32 vidx_ = __riscv_vmul(__riscv_vreinterpret_u32m2(vidx), sizeof(_Tp), VTraits::vlanes()); \ + v_uint32 vidx_ = __riscv_vmul(__riscv_vreinterpret_u32m1(vidx), sizeof(_Tp), VTraits::vlanes()); \ return __riscv_vloxei32(tab, vidx_, VTraits<_Tpvec>::vlanes()); \ } OPENCV_HAL_IMPL_RVV_LUT_VEC(v_float32, float) @@ -673,7 +665,7 @@ OPENCV_HAL_IMPL_RVV_LUT_VEC(v_uint32, unsigned) #if CV_SIMD_SCALABLE_64F inline v_float64 v_lut(const double* tab, const v_int32& vidx) \ { \ - vuint32m1_t vidx_ = __riscv_vmul(__riscv_vlmul_trunc_u32m1(__riscv_vreinterpret_u32m2(vidx)), sizeof(double), VTraits::vlanes()); \ + vuint32mf2_t vidx_ = __riscv_vmul(__riscv_vlmul_trunc_u32mf2(__riscv_vreinterpret_u32m1(vidx)), sizeof(double), VTraits::vlanes()); \ return __riscv_vloxei32(tab, vidx_, VTraits::vlanes()); \ } #endif @@ -683,7 +675,7 @@ inline v_float64 v_lut(const double* tab, const v_int32& vidx) \ #define OPENCV_HAL_IMPL_RVV_LUT_DEINTERLEAVE(_Tpvec, _Tp, suffix) \ inline void v_lut_deinterleave(const _Tp* tab, const v_int32& vidx, _Tpvec& vx, _Tpvec& vy) \ { \ - v_uint32 vidx_ = __riscv_vmul(__riscv_vreinterpret_u32m2(vidx), sizeof(_Tp), VTraits::vlanes()); \ + v_uint32 vidx_ = __riscv_vmul(__riscv_vreinterpret_u32m1(vidx), sizeof(_Tp), VTraits::vlanes()); \ vx = __riscv_vluxei32(tab, vidx_, VTraits<_Tpvec>::vlanes()); \ vy = __riscv_vluxei32(tab, __riscv_vadd(vidx_, sizeof(_Tp), VTraits::vlanes()), VTraits<_Tpvec>::vlanes()); \ } @@ -694,7 +686,7 @@ OPENCV_HAL_IMPL_RVV_LUT_DEINTERLEAVE(v_uint32, unsigned, u32) #if CV_SIMD_SCALABLE_64F inline void v_lut_deinterleave(const double* tab, const v_int32& vidx, v_float64& vx, v_float64& vy) \ { \ - vuint32m1_t vidx_ = __riscv_vmul(__riscv_vlmul_trunc_u32m1(__riscv_vreinterpret_u32m2(vidx)), sizeof(double), VTraits::vlanes()); \ + vuint32mf2_t vidx_ = __riscv_vmul(__riscv_vlmul_trunc_u32mf2(__riscv_vreinterpret_u32m1(vidx)), sizeof(double), VTraits::vlanes()); \ vx = __riscv_vluxei32(tab, vidx_, VTraits::vlanes()); \ vy = __riscv_vluxei32(tab, __riscv_vadd(vidx_, sizeof(double), VTraits::vlanes()), VTraits::vlanes()); \ } @@ -707,7 +699,7 @@ inline v_uint8 v_lut_quads(const uchar* tab, const int* idx) { return v_reinterp // Byte-indexed LUT: vector byte indices -> looked-up bytes (uses RVV indexed load) inline v_uint8 v_lut(const uchar* tab, const v_uint8& idx) -{ return __riscv_vluxei8_v_u8m2(tab, idx, VTraits::vlanes()); } +{ return __riscv_vluxei8_v_u8m1(tab, idx, VTraits::vlanes()); } inline v_int8 v_lut(const schar* tab, const v_uint8& idx) { return v_reinterpret_as_s8(v_lut((const uchar*)tab, idx)); } @@ -723,24 +715,24 @@ inline v_uint64 v_lut_pairs(const uint64* tab, const int* idx) { return v_reinte ////////////// Pack boolean //////////////////// inline v_uint8 v_pack_b(const v_uint16& a, const v_uint16& b) { - return __riscv_vnsrl(__riscv_vset(__riscv_vlmul_ext_v_u16m2_u16m4(a),1,b), 0, VTraits::vlanes()); + return __riscv_vnsrl(__riscv_vset(__riscv_vlmul_ext_v_u16m1_u16m2(a),1,b), 0, VTraits::vlanes()); } inline v_uint8 v_pack_b(const v_uint32& a, const v_uint32& b, const v_uint32& c, const v_uint32& d) { - return __riscv_vnsrl(__riscv_vnsrl(__riscv_vset(__riscv_vset(__riscv_vset(__riscv_vlmul_ext_u32m8(a),1,b),2,c),3,d), 0, VTraits::vlanes()), 0, VTraits::vlanes()); + return __riscv_vnsrl(__riscv_vnsrl(__riscv_vset(__riscv_vset(__riscv_vset(__riscv_vlmul_ext_u32m4(a),1,b),2,c),3,d), 0, VTraits::vlanes()), 0, VTraits::vlanes()); } inline v_uint8 v_pack_b(const v_uint64& a, const v_uint64& b, const v_uint64& c, const v_uint64& d, const v_uint64& e, const v_uint64& f, const v_uint64& g, const v_uint64& h) { - vuint8m1_t t0 = __riscv_vnsrl(__riscv_vnsrl(__riscv_vnsrl(__riscv_vset(__riscv_vset(__riscv_vset(__riscv_vlmul_ext_u64m8(a),1,b),2,c),3,d), 0, VTraits::vlanes()), 0, VTraits::vlanes()), 0, VTraits::vlanes()); - vuint8m1_t t1 = __riscv_vnsrl(__riscv_vnsrl(__riscv_vnsrl(__riscv_vset(__riscv_vset(__riscv_vset(__riscv_vlmul_ext_u64m8(e),1,f),2,g),3,h), 0, VTraits::vlanes()), 0, VTraits::vlanes()), 0, VTraits::vlanes()); - - return __riscv_vset(__riscv_vlmul_ext_u8m2(t0), 1, t1); + return __riscv_vnsrl(__riscv_vnsrl(__riscv_vnsrl( + __riscv_vset(__riscv_vset(__riscv_vset(__riscv_vset(__riscv_vset(__riscv_vset(__riscv_vset(__riscv_vlmul_ext_u64m8(a), + 1,b),2,c),3,d),4,e),5,f),6,g),7,h), + 0, VTraits::vlanes()), 0, VTraits::vlanes()), 0, VTraits::vlanes()); } ////////////// Arithmetics ////////////// @@ -822,15 +814,15 @@ OPENCV_HAL_IMPL_RVV_BIN_MMUL(v_float64, __riscv_vfmul) inline void v_mul_expand(const _Tpvec& a, const _Tpvec& b, _Tpwvec& c, _Tpwvec& d) \ { \ _TpwvecM2 temp = wmul(a, b, VTraits<_Tpvec>::vlanes()); \ - c = __riscv_vget_##suffix##m2(temp, 0); \ - d = __riscv_vget_##suffix##m2(temp, 1); \ + c = __riscv_vget_##suffix##m1(temp, 0); \ + d = __riscv_vget_##suffix##m1(temp, 1); \ } -OPENCV_HAL_IMPL_RVV_MUL_EXPAND(v_uint8, v_uint16, vuint16m4_t, u16, __riscv_vwmulu) -OPENCV_HAL_IMPL_RVV_MUL_EXPAND(v_int8, v_int16, vint16m4_t, i16, __riscv_vwmul) -OPENCV_HAL_IMPL_RVV_MUL_EXPAND(v_uint16, v_uint32, vuint32m4_t, u32, __riscv_vwmulu) -OPENCV_HAL_IMPL_RVV_MUL_EXPAND(v_int16, v_int32, vint32m4_t, i32, __riscv_vwmul) -OPENCV_HAL_IMPL_RVV_MUL_EXPAND(v_uint32, v_uint64, vuint64m4_t, u64, __riscv_vwmulu) +OPENCV_HAL_IMPL_RVV_MUL_EXPAND(v_uint8, v_uint16, vuint16m2_t, u16, __riscv_vwmulu) +OPENCV_HAL_IMPL_RVV_MUL_EXPAND(v_int8, v_int16, vint16m2_t, i16, __riscv_vwmul) +OPENCV_HAL_IMPL_RVV_MUL_EXPAND(v_uint16, v_uint32, vuint32m2_t, u32, __riscv_vwmulu) +OPENCV_HAL_IMPL_RVV_MUL_EXPAND(v_int16, v_int32, vint32m2_t, i32, __riscv_vwmul) +OPENCV_HAL_IMPL_RVV_MUL_EXPAND(v_uint32, v_uint64, vuint64m2_t, u64, __riscv_vwmulu) inline v_int16 v_mul_hi(const v_int16& a, const v_int16& b) { @@ -904,7 +896,7 @@ OPENCV_HAL_IMPL_RVV_LOGIC_OP(v_int64, VTraits::vlanes()) #define OPENCV_HAL_IMPL_RVV_FLT16_BIT_OP(intrin) \ inline v_float16 intrin (const v_float16& a, const v_float16& b) \ { \ - return __riscv_vreinterpret_f16m2(intrin(__riscv_vreinterpret_i16m2(a), __riscv_vreinterpret_i16m2(b))); \ + return __riscv_vreinterpret_f16m1(intrin(__riscv_vreinterpret_i16m1(a), __riscv_vreinterpret_i16m1(b))); \ } OPENCV_HAL_IMPL_RVV_FLT16_BIT_OP(v_and) OPENCV_HAL_IMPL_RVV_FLT16_BIT_OP(v_or) @@ -912,14 +904,14 @@ OPENCV_HAL_IMPL_RVV_FLT16_BIT_OP(v_xor) inline v_float16 v_not (const v_float16& a) \ { \ - return __riscv_vreinterpret_f16m2(v_not(__riscv_vreinterpret_i16m2(a))); \ + return __riscv_vreinterpret_f16m1(v_not(__riscv_vreinterpret_i16m1(a))); \ } #endif #define OPENCV_HAL_IMPL_RVV_FLT32_BIT_OP(intrin) \ inline v_float32 intrin (const v_float32& a, const v_float32& b) \ { \ - return __riscv_vreinterpret_f32m2(intrin(__riscv_vreinterpret_i32m2(a), __riscv_vreinterpret_i32m2(b))); \ + return __riscv_vreinterpret_f32m1(intrin(__riscv_vreinterpret_i32m1(a), __riscv_vreinterpret_i32m1(b))); \ } OPENCV_HAL_IMPL_RVV_FLT32_BIT_OP(v_and) OPENCV_HAL_IMPL_RVV_FLT32_BIT_OP(v_or) @@ -927,14 +919,14 @@ OPENCV_HAL_IMPL_RVV_FLT32_BIT_OP(v_xor) inline v_float32 v_not (const v_float32& a) \ { \ - return __riscv_vreinterpret_f32m2(v_not(__riscv_vreinterpret_i32m2(a))); \ + return __riscv_vreinterpret_f32m1(v_not(__riscv_vreinterpret_i32m1(a))); \ } #if CV_SIMD_SCALABLE_64F #define OPENCV_HAL_IMPL_RVV_FLT64_BIT_OP(intrin) \ inline v_float64 intrin (const v_float64& a, const v_float64& b) \ { \ - return __riscv_vreinterpret_f64m2(intrin(__riscv_vreinterpret_i64m2(a), __riscv_vreinterpret_i64m2(b))); \ + return __riscv_vreinterpret_f64m1(intrin(__riscv_vreinterpret_i64m1(a), __riscv_vreinterpret_i64m1(b))); \ } OPENCV_HAL_IMPL_RVV_FLT64_BIT_OP(v_and) OPENCV_HAL_IMPL_RVV_FLT64_BIT_OP(v_or) @@ -942,7 +934,7 @@ OPENCV_HAL_IMPL_RVV_FLT64_BIT_OP(v_xor) inline v_float64 v_not (const v_float64& a) \ { \ - return __riscv_vreinterpret_f64m2(v_not(__riscv_vreinterpret_i64m2(a))); \ + return __riscv_vreinterpret_f64m1(v_not(__riscv_vreinterpret_i64m1(a))); \ } #endif @@ -986,7 +978,7 @@ inline _Tpvec v_##op(const _Tpvec& a, const _Tpvec& b) \ { \ size_t VLEN = VTraits<_Tpvec>::vlanes(); \ uint64_t ones = -1; \ - return __riscv_vmerge(__riscv_vmv_v_x_##suffix##m2(0, VLEN), ones, intrin(a, b, VLEN), VLEN); \ + return __riscv_vmerge(__riscv_vmv_v_x_##suffix##m1(0, VLEN), ones, intrin(a, b, VLEN), VLEN); \ } #define OPENCV_HAL_IMPL_RVV_FLOAT_CMP_OP(_Tpvec, op, intrin, suffix) \ @@ -996,7 +988,7 @@ inline _Tpvec v_##op (const _Tpvec& a, const _Tpvec& b) \ union { uint64_t u; VTraits<_Tpvec>::lane_type d; } ones; \ ones.u = -1; \ auto diff = intrin(a, b, VLEN); \ - auto z = __riscv_vfmv_v_f_##suffix##m2(0, VLEN); \ + auto z = __riscv_vfmv_v_f_##suffix##m1(0, VLEN); \ auto res = __riscv_vfmerge(z, ones.d, diff, VLEN); \ return _Tpvec(res); \ } //TODO @@ -1008,7 +1000,7 @@ inline _Tpvec v_##op (const _Tpvec& a, const _Tpvec& b) \ union { uint64_t u; _Float16 d; } ones; \ ones.u = -1; \ auto diff = intrin(a, b, VLEN); \ - auto z = __riscv_vfmv_v_f_##suffix##m2(0, VLEN); \ + auto z = __riscv_vfmv_v_f_##suffix##m1(0, VLEN); \ auto res = __riscv_vfmerge(z, ones.d, diff, VLEN); \ return _Tpvec(res); \ } //TODO @@ -1109,18 +1101,77 @@ OPENCV_HAL_IMPL_RVV_BIN_FUNC(v_float64, v_max, __riscv_vfmax, VTraits #define OPENCV_HAL_IMPL_RVV_ZIP4(_Tpvec, _wTpvec, suffix, convert2u, convert) \ inline void v_zip4(const _Tpvec& a0, const _Tpvec& a1, _Tpvec& b0, _Tpvec& b1) { \ int vl = 4; \ - _wTpvec temp = __riscv_vreinterpret_##suffix##m4(convert2u( \ + _wTpvec temp = __riscv_vreinterpret_##suffix##m2(convert2u( \ __riscv_vor(__riscv_vzext_vf2(convert(a0), vl), \ - __riscv_vreinterpret_u64m4(__riscv_vslide1up(__riscv_vreinterpret_u32m4(__riscv_vzext_vf2(convert(a1), vl)), 0, vl*2)), \ + __riscv_vreinterpret_u64m2(__riscv_vslide1up(__riscv_vreinterpret_u32m2(__riscv_vzext_vf2(convert(a1), vl)), 0, vl*2)), \ vl))); \ - b0 = __riscv_vget_##suffix##m2(temp, 0); \ - b1 = __riscv_vget_##suffix##m2(__riscv_vrgather(temp, __riscv_vadd(__riscv_vid_v_u32m4(vl), 4, vl)/*{4,5,6,7} */, vl) ,0); \ + b0 = __riscv_vget_##suffix##m1(temp, 0); \ + b1 = __riscv_vget_##suffix##m1(__riscv_vrgather(temp, __riscv_vadd(__riscv_vid_v_u32m2(vl), 4, vl)/*{4,5,6,7} */, vl) ,0); \ } -OPENCV_HAL_IMPL_RVV_ZIP4(v_uint32, vuint32m4_t, u32, OPENCV_HAL_NOP, OPENCV_HAL_NOP) -OPENCV_HAL_IMPL_RVV_ZIP4(v_int32, vint32m4_t, i32, __riscv_vreinterpret_u32m4, __riscv_vreinterpret_u32m2) -OPENCV_HAL_IMPL_RVV_ZIP4(v_float32, vfloat32m4_t, f32, __riscv_vreinterpret_u32m4, __riscv_vreinterpret_u32m2) +OPENCV_HAL_IMPL_RVV_ZIP4(v_uint32, vuint32m2_t, u32, OPENCV_HAL_NOP, OPENCV_HAL_NOP) +OPENCV_HAL_IMPL_RVV_ZIP4(v_int32, vint32m2_t, i32, __riscv_vreinterpret_u32m2, __riscv_vreinterpret_u32m1) +OPENCV_HAL_IMPL_RVV_ZIP4(v_float32, vfloat32m2_t, f32, __riscv_vreinterpret_u32m2, __riscv_vreinterpret_u32m1) +#if 0 +// this is v_zip4 and v_tranpose4x4 for scalable VLEN, costs more instruction than current 128-bit only version. +inline void v_zip4(const v_float32& a0, const v_float32& a1, v_float32& b0, v_float32& b1) { + vuint64m1_t vid1 = __riscv_vid_v_u64m1(VTraits::vlanes()); + vuint16m1_t t1 = __riscv_vreinterpret_u16m1(vid1); + vuint16m1_t t2 = __riscv_vslide1up(t1, 0, VTraits::vlanes()); + vuint16m1_t t3 = __riscv_vslide1up(t2, 0, VTraits::vlanes()); + vuint16m1_t t4 = __riscv_vslide1up(t3, 0, VTraits::vlanes()); + t1 = __riscv_vor( + __riscv_vor(t1, t2, VTraits::vlanes()), + __riscv_vor(t3, t4, VTraits::vlanes()), + VTraits::vlanes() + ); + vuint32m2_t vidx0 = __riscv_vwmulu(t1, 4, VTraits::vlanes()); + vidx0 = __riscv_vadd(vidx0, __riscv_vid_v_u32m2(VTraits::vlanes()), VTraits::vlanes()); + vuint32m2_t vidx1 = __riscv_vadd(vidx0, 4, VTraits::vlanes()); + vfloat32m2_t temp = __riscv_vreinterpret_f32m2(__riscv_vreinterpret_u32m2( + __riscv_vor(__riscv_vzext_vf2(__riscv_vreinterpret_u32m1(a0), VTraits::vlanes()), + __riscv_vreinterpret_u64m2(__riscv_vslide1up(__riscv_vreinterpret_u32m2(__riscv_vzext_vf2(__riscv_vreinterpret_u32m1(a1), VTraits::vlanes())), 0, VTraits::vlanes()*2)), + VTraits::vlanes()))); + b0 = __riscv_vlmul_trunc_f32m1(__riscv_vrgather(temp, vidx0, VTraits::vlanes())); + b1 = __riscv_vlmul_trunc_f32m1(__riscv_vrgather(temp, vidx1, VTraits::vlanes())); +} + +inline void v_transpose4x4(const v_float32& a0, const v_float32& a1, const v_float32& a2, const v_float32& a3,\ + v_float32& b0, v_float32& b1, v_float32& b2, v_float32& b3) { \ + vuint64m2_t vid1 = __riscv_vid_v_u64m2(VTraits::vlanes()); + vuint16m2_t t1 = __riscv_vreinterpret_u16m2(vid1); + vuint16m2_t t2 = __riscv_vslide1up(t1, 0, VTraits::vlanes()); + vuint16m2_t t3 = __riscv_vslide1up(t2, 0, VTraits::vlanes()); + vuint16m2_t t4 = __riscv_vslide1up(t3, 0, VTraits::vlanes()); + t1 = __riscv_vor( + __riscv_vor(t1, t2, VTraits::vlanes()), + __riscv_vor(t3, t4, VTraits::vlanes()), + VTraits::vlanes() + ); + vuint16m2_t vidx0 = __riscv_vmul(t1, 12, VTraits::vlanes()); + vidx0 = __riscv_vadd(vidx0, __riscv_vid_v_u16m2(VTraits::vlanes()), VTraits::vlanes()); + vuint16m2_t vidx1 = __riscv_vadd(vidx0, 4, VTraits::vlanes()); + vuint16m2_t vidx2 = __riscv_vadd(vidx0, 8, VTraits::vlanes()); + vuint16m2_t vidx3 = __riscv_vadd(vidx0, 12, VTraits::vlanes()); + vuint32m2_t tempA = __riscv_vreinterpret_u32m2( \ + __riscv_vor(__riscv_vzext_vf2(__riscv_vreinterpret_u32m1(a0), VTraits::vlanes()), \ + __riscv_vreinterpret_u64m2(__riscv_vslide1up(__riscv_vreinterpret_u32m2(__riscv_vzext_vf2(__riscv_vreinterpret_u32m1(a2), VTraits::vlanes())), 0, VTraits::vlanes())), \ + VTraits::vlanes())); \ + vuint32m2_t tempB = __riscv_vreinterpret_u32m2( \ + __riscv_vor(__riscv_vzext_vf2(__riscv_vreinterpret_u32m1(a1), VTraits::vlanes()), \ + __riscv_vreinterpret_u64m2(__riscv_vslide1up(__riscv_vreinterpret_u32m2(__riscv_vzext_vf2(__riscv_vreinterpret_u32m1(a3), VTraits::vlanes())), 0, VTraits::vlanes())), \ + VTraits::vlanes())); \ + vfloat32m4_t temp = __riscv_vreinterpret_f32m4(__riscv_vreinterpret_u32m4( \ + __riscv_vor(__riscv_vzext_vf2(tempA, VTraits::vlanes()), \ + __riscv_vreinterpret_u64m4(__riscv_vslide1up(__riscv_vreinterpret_u32m4(__riscv_vzext_vf2(tempB, VTraits::vlanes())), 0, VTraits::vlanes())), \ + VTraits::vlanes()))); \ + b0 = __riscv_vlmul_trunc_f32m1(__riscv_vrgatherei16(temp, vidx0, VTraits::vlanes())); + b1 = __riscv_vlmul_trunc_f32m1(__riscv_vrgatherei16(temp, vidx1, VTraits::vlanes())); + b2 = __riscv_vlmul_trunc_f32m1(__riscv_vrgatherei16(temp, vidx2, VTraits::vlanes())); + b3 = __riscv_vlmul_trunc_f32m1(__riscv_vrgatherei16(temp, vidx3, VTraits::vlanes())); +} +#endif #define OPENCV_HAL_IMPL_RVV_TRANSPOSE4x4(_Tpvec, suffix) \ inline void v_transpose4x4(const _Tpvec& a0, const _Tpvec& a1, const _Tpvec& a2, const _Tpvec& a3, _Tpvec& b0, _Tpvec& b1, _Tpvec& b2, _Tpvec& b3) { \ @@ -1143,7 +1194,7 @@ inline scalartype v_reduce_sum(const _Tpvec& a) \ _nwTpvec zero = __riscv_vmv_v_x_##wsuffix##m1(0, vl); \ _nwTpvec res = __riscv_vmv_v_x_##wsuffix##m1(0, vl); \ res = __riscv_v##red(a, zero, vl); \ - return (scalartype)__riscv_vmv_x(res); \ + return (scalartype)v_get0(res); \ } OPENCV_HAL_IMPL_RVV_REDUCE_SUM(v_uint8, v_uint16, vuint16m1_t, unsigned, u16, VTraits::vlanes(), wredsumu) OPENCV_HAL_IMPL_RVV_REDUCE_SUM(v_int8, v_int16, vint16m1_t, int, i16, VTraits::vlanes(), wredsum) @@ -1161,93 +1212,86 @@ inline scalartype v_reduce_sum(const _Tpvec& a) \ _nwTpvec zero = __riscv_vfmv_v_f_##wsuffix##m1(0, vl); \ _nwTpvec res = __riscv_vfmv_v_f_##wsuffix##m1(0, vl); \ res = __riscv_vfredusum(a, zero, vl); \ - return (scalartype)__riscv_vfmv_f(res); \ + return (scalartype)v_get0(res); \ } OPENCV_HAL_IMPL_RVV_REDUCE_SUM_FP(v_float32, v_float32, vfloat32m1_t, float, f32, VTraits::vlanes()) #if CV_SIMD_SCALABLE_64F OPENCV_HAL_IMPL_RVV_REDUCE_SUM_FP(v_float64, v_float64, vfloat64m1_t, float, f64, VTraits::vlanes()) #endif -#define OPENCV_HAL_IMPL_RVV_REDUCE(_Tpvec, _nTpvec, func, scalartype, suffix, vl, red) \ +#define OPENCV_HAL_IMPL_RVV_REDUCE(_Tpvec, func, scalartype, suffix, vl, red) \ inline scalartype v_reduce_##func(const _Tpvec& a) \ { \ - _nTpvec narrowM1 = __riscv_vlmul_trunc_##suffix##m1(a); \ - return (scalartype)__riscv_vmv_x(__riscv_v##red(a, narrowM1, vl)); \ + _Tpvec res = _Tpvec(__riscv_v##red(a, a, vl)); \ + return (scalartype)v_get0(res); \ } -#define OPENCV_HAL_IMPL_RVV_REDUCE_FP(_Tpvec, _nTpvec, func, scalartype, suffix, vl, red) \ -inline scalartype v_reduce_##func(const _Tpvec& a) \ -{ \ - _nTpvec narrowM1 = __riscv_vlmul_trunc_##suffix##m1(a); \ - return (scalartype)__riscv_vfmv_f(__riscv_v##red(a, narrowM1, vl)); \ -} - -OPENCV_HAL_IMPL_RVV_REDUCE(v_uint8, vuint8m1_t, min, uchar, u8, VTraits::vlanes(), redminu) -OPENCV_HAL_IMPL_RVV_REDUCE(v_int8, vint8m1_t, min, schar, i8, VTraits::vlanes(), redmin) -OPENCV_HAL_IMPL_RVV_REDUCE(v_uint16, vuint16m1_t, min, ushort, u16, VTraits::vlanes(), redminu) -OPENCV_HAL_IMPL_RVV_REDUCE(v_int16, vint16m1_t, min, short, i16, VTraits::vlanes(), redmin) -OPENCV_HAL_IMPL_RVV_REDUCE(v_uint32, vuint32m1_t, min, unsigned, u32, VTraits::vlanes(), redminu) -OPENCV_HAL_IMPL_RVV_REDUCE(v_int32, vint32m1_t, min, int, i32, VTraits::vlanes(), redmin) -OPENCV_HAL_IMPL_RVV_REDUCE_FP(v_float32, vfloat32m1_t, min, float, f32, VTraits::vlanes(), fredmin) -OPENCV_HAL_IMPL_RVV_REDUCE(v_uint8, vuint8m1_t, max, uchar, u8, VTraits::vlanes(), redmaxu) -OPENCV_HAL_IMPL_RVV_REDUCE(v_int8, vint8m1_t, max, schar, i8, VTraits::vlanes(), redmax) -OPENCV_HAL_IMPL_RVV_REDUCE(v_uint16, vuint16m1_t, max, ushort, u16, VTraits::vlanes(), redmaxu) -OPENCV_HAL_IMPL_RVV_REDUCE(v_int16, vint16m1_t, max, short, i16, VTraits::vlanes(), redmax) -OPENCV_HAL_IMPL_RVV_REDUCE(v_uint32, vuint32m1_t, max, unsigned, u32, VTraits::vlanes(), redmaxu) -OPENCV_HAL_IMPL_RVV_REDUCE(v_int32, vint32m1_t, max, int, i32, VTraits::vlanes(), redmax) -OPENCV_HAL_IMPL_RVV_REDUCE_FP(v_float32, vfloat32m1_t, max, float, f32, VTraits::vlanes(), fredmax) +OPENCV_HAL_IMPL_RVV_REDUCE(v_uint8, min, uchar, u8, VTraits::vlanes(), redminu) +OPENCV_HAL_IMPL_RVV_REDUCE(v_int8, min, schar, i8, VTraits::vlanes(), redmin) +OPENCV_HAL_IMPL_RVV_REDUCE(v_uint16, min, ushort, u16, VTraits::vlanes(), redminu) +OPENCV_HAL_IMPL_RVV_REDUCE(v_int16, min, short, i16, VTraits::vlanes(), redmin) +OPENCV_HAL_IMPL_RVV_REDUCE(v_uint32, min, unsigned, u32, VTraits::vlanes(), redminu) +OPENCV_HAL_IMPL_RVV_REDUCE(v_int32, min, int, i32, VTraits::vlanes(), redmin) +OPENCV_HAL_IMPL_RVV_REDUCE(v_float32, min, float, f32, VTraits::vlanes(), fredmin) +OPENCV_HAL_IMPL_RVV_REDUCE(v_uint8, max, uchar, u8, VTraits::vlanes(), redmaxu) +OPENCV_HAL_IMPL_RVV_REDUCE(v_int8, max, schar, i8, VTraits::vlanes(), redmax) +OPENCV_HAL_IMPL_RVV_REDUCE(v_uint16, max, ushort, u16, VTraits::vlanes(), redmaxu) +OPENCV_HAL_IMPL_RVV_REDUCE(v_int16, max, short, i16, VTraits::vlanes(), redmax) +OPENCV_HAL_IMPL_RVV_REDUCE(v_uint32, max, unsigned, u32, VTraits::vlanes(), redmaxu) +OPENCV_HAL_IMPL_RVV_REDUCE(v_int32, max, int, i32, VTraits::vlanes(), redmax) +OPENCV_HAL_IMPL_RVV_REDUCE(v_float32, max, float, f32, VTraits::vlanes(), fredmax) #if CV_SIMD_SCALABLE_FP16 -OPENCV_HAL_IMPL_RVV_REDUCE_FP(v_float16, vfloat16m1_t, max, hfloat, f16, VTraits::vlanes(), fredmax) -OPENCV_HAL_IMPL_RVV_REDUCE_FP(v_float16, vfloat16m1_t, min, hfloat, f16, VTraits::vlanes(), fredmin) +OPENCV_HAL_IMPL_RVV_REDUCE(v_float16, max, hfloat, f16, VTraits::vlanes(), fredmax) +OPENCV_HAL_IMPL_RVV_REDUCE(v_float16, min, hfloat, f16, VTraits::vlanes(), fredmin) #endif inline v_float32 v_reduce_sum4(const v_float32& a, const v_float32& b, const v_float32& c, const v_float32& d) { // 0000 1111 2222 3333 .... - vuint64m4_t vid1 = __riscv_vid_v_u64m4(VTraits::vlanes()); - vuint16m4_t t1 = __riscv_vreinterpret_u16m4(vid1); - vuint16m4_t t2 = __riscv_vslide1up(t1, 0, VTraits::vlanes()); - vuint16m4_t t3 = __riscv_vslide1up(t2, 0, VTraits::vlanes()); - vuint16m4_t t4 = __riscv_vslide1up(t3, 0, VTraits::vlanes()); + vuint64m2_t vid1 = __riscv_vid_v_u64m2(VTraits::vlanes()); + vuint16m2_t t1 = __riscv_vreinterpret_u16m2(vid1); + vuint16m2_t t2 = __riscv_vslide1up(t1, 0, VTraits::vlanes()); + vuint16m2_t t3 = __riscv_vslide1up(t2, 0, VTraits::vlanes()); + vuint16m2_t t4 = __riscv_vslide1up(t3, 0, VTraits::vlanes()); t1 = __riscv_vor( - __riscv_vor(t1, t2, VTraits::vlanes()), - __riscv_vor(t3, t4, VTraits::vlanes()), - VTraits::vlanes() + __riscv_vor(t1, t2, VTraits::vlanes()), + __riscv_vor(t3, t4, VTraits::vlanes()), + VTraits::vlanes() ); // index for transpose4X4 - vuint16m4_t vidx0 = __riscv_vmul(t1, 12, VTraits::vlanes()); - vidx0 = __riscv_vadd(vidx0, __riscv_vid_v_u16m4(VTraits::vlanes()), VTraits::vlanes()); - vuint16m4_t vidx1 = __riscv_vadd(vidx0, 4, VTraits::vlanes()); - vuint16m4_t vidx2 = __riscv_vadd(vidx0, 8, VTraits::vlanes()); - vuint16m4_t vidx3 = __riscv_vadd(vidx0, 12, VTraits::vlanes()); + vuint16m2_t vidx0 = __riscv_vmul(t1, 12, VTraits::vlanes()); + vidx0 = __riscv_vadd(vidx0, __riscv_vid_v_u16m2(VTraits::vlanes()), VTraits::vlanes()); + vuint16m2_t vidx1 = __riscv_vadd(vidx0, 4, VTraits::vlanes()); + vuint16m2_t vidx2 = __riscv_vadd(vidx0, 8, VTraits::vlanes()); + vuint16m2_t vidx3 = __riscv_vadd(vidx0, 12, VTraits::vlanes()); // zip - vuint32m4_t tempA = __riscv_vreinterpret_u32m4( \ - __riscv_vor(__riscv_vzext_vf2(__riscv_vreinterpret_u32m2(a), VTraits::vlanes()), \ - __riscv_vreinterpret_u64m4(__riscv_vslide1up(__riscv_vreinterpret_u32m4(__riscv_vzext_vf2(__riscv_vreinterpret_u32m2(c), VTraits::vlanes())), 0, VTraits::vlanes())), \ - VTraits::vlanes())); \ - vuint32m4_t tempB = __riscv_vreinterpret_u32m4( \ - __riscv_vor(__riscv_vzext_vf2(__riscv_vreinterpret_u32m2(b), VTraits::vlanes()), \ - __riscv_vreinterpret_u64m4(__riscv_vslide1up(__riscv_vreinterpret_u32m4(__riscv_vzext_vf2(__riscv_vreinterpret_u32m2(d), VTraits::vlanes())), 0, VTraits::vlanes())), \ - VTraits::vlanes())); \ - vfloat32m8_t temp = __riscv_vreinterpret_f32m8(__riscv_vreinterpret_u32m8( \ - __riscv_vor(__riscv_vzext_vf2(tempA, VTraits::vlanes()), \ - __riscv_vreinterpret_u64m8(__riscv_vslide1up(__riscv_vreinterpret_u32m8(__riscv_vzext_vf2(tempB, VTraits::vlanes())), 0, VTraits::vlanes())), \ - VTraits::vlanes()))); + vuint32m2_t tempA = __riscv_vreinterpret_u32m2( \ + __riscv_vor(__riscv_vzext_vf2(__riscv_vreinterpret_u32m1(a), VTraits::vlanes()), \ + __riscv_vreinterpret_u64m2(__riscv_vslide1up(__riscv_vreinterpret_u32m2(__riscv_vzext_vf2(__riscv_vreinterpret_u32m1(c), VTraits::vlanes())), 0, VTraits::vlanes())), \ + VTraits::vlanes())); \ + vuint32m2_t tempB = __riscv_vreinterpret_u32m2( \ + __riscv_vor(__riscv_vzext_vf2(__riscv_vreinterpret_u32m1(b), VTraits::vlanes()), \ + __riscv_vreinterpret_u64m2(__riscv_vslide1up(__riscv_vreinterpret_u32m2(__riscv_vzext_vf2(__riscv_vreinterpret_u32m1(d), VTraits::vlanes())), 0, VTraits::vlanes())), \ + VTraits::vlanes())); \ + vfloat32m4_t temp = __riscv_vreinterpret_f32m4(__riscv_vreinterpret_u32m4( \ + __riscv_vor(__riscv_vzext_vf2(tempA, VTraits::vlanes()), \ + __riscv_vreinterpret_u64m4(__riscv_vslide1up(__riscv_vreinterpret_u32m4(__riscv_vzext_vf2(tempB, VTraits::vlanes())), 0, VTraits::vlanes())), \ + VTraits::vlanes()))); // transpose - vfloat32m2_t b0 = __riscv_vlmul_trunc_f32m2(__riscv_vrgatherei16(temp, vidx0, VTraits::vlanes())); - vfloat32m2_t b1 = __riscv_vlmul_trunc_f32m2(__riscv_vrgatherei16(temp, vidx1, VTraits::vlanes())); - vfloat32m2_t b2 = __riscv_vlmul_trunc_f32m2(__riscv_vrgatherei16(temp, vidx2, VTraits::vlanes())); - vfloat32m2_t b3 = __riscv_vlmul_trunc_f32m2(__riscv_vrgatherei16(temp, vidx3, VTraits::vlanes())); + vfloat32m1_t b0 = __riscv_vlmul_trunc_f32m1(__riscv_vrgatherei16(temp, vidx0, VTraits::vlanes())); + vfloat32m1_t b1 = __riscv_vlmul_trunc_f32m1(__riscv_vrgatherei16(temp, vidx1, VTraits::vlanes())); + vfloat32m1_t b2 = __riscv_vlmul_trunc_f32m1(__riscv_vrgatherei16(temp, vidx2, VTraits::vlanes())); + vfloat32m1_t b3 = __riscv_vlmul_trunc_f32m1(__riscv_vrgatherei16(temp, vidx3, VTraits::vlanes())); // vector add v_float32 res = __riscv_vfadd( - __riscv_vfadd(b0, b1, VTraits::vlanes()), - __riscv_vfadd(b2, b3, VTraits::vlanes()), - VTraits::vlanes() + __riscv_vfadd(b0, b1, VTraits::vlanes()), + __riscv_vfadd(b2, b3, VTraits::vlanes()), + VTraits::vlanes() ); return res; } @@ -1316,7 +1360,7 @@ inline v_float16 v_muladd(const v_float16& a, const v_float16& b, const v_float1 #if CV_SIMD_SCALABLE_64F inline v_float64 v_fma(const v_float64& a, const v_float64& b, const v_float64& c) { - return __riscv_vfmacc_vv_f64m2(c, a, b, VTraits::vlanes()); + return __riscv_vfmacc_vv_f64m1(c, a, b, VTraits::vlanes()); } inline v_float64 v_muladd(const v_float64& a, const v_float64& b, const v_float64& c) @@ -1406,7 +1450,7 @@ OPENCV_HAL_IMPL_RVV_ABSDIFF(v_int16, absdiffs) #define OPENCV_HAL_IMPL_RVV_ABSDIFF_S(_Tpvec, _rTpvec, width) \ inline _rTpvec v_absdiff(const _Tpvec& a, const _Tpvec& b) \ { \ - return __riscv_vnclipu(__riscv_vreinterpret_u##width##m4(__riscv_vwsub_vv(v_max(a, b), v_min(a, b), VTraits<_Tpvec>::vlanes())), 0, 0, VTraits<_Tpvec>::vlanes()); \ + return __riscv_vnclipu(__riscv_vreinterpret_u##width##m2(__riscv_vwsub_vv(v_max(a, b), v_min(a, b), VTraits<_Tpvec>::vlanes())), 0, 0, VTraits<_Tpvec>::vlanes()); \ } OPENCV_HAL_IMPL_RVV_ABSDIFF_S(v_int8, v_uint8, 16) @@ -1487,7 +1531,7 @@ template inline _Tpvec v_rotate_right(const _Tpvec& a) \ } \ template inline _Tpvec v_rotate_left(const _Tpvec& a) \ { \ - return __riscv_vslideup(__riscv_vmv_v_x_##suffix##m2(0, vl), a, n, vl); \ + return __riscv_vslideup(__riscv_vmv_v_x_##suffix##m1(0, vl), a, n, vl); \ } \ template<> inline _Tpvec v_rotate_left<0>(const _Tpvec& a) \ { return a; } \ @@ -1518,7 +1562,7 @@ template inline _Tpvec v_rotate_right(const _Tpvec& a) \ } \ template inline _Tpvec v_rotate_left(const _Tpvec& a) \ { \ - return __riscv_vslideup(__riscv_vfmv_v_f_##suffix##m2(0, vl), a, n, vl); \ + return __riscv_vslideup(__riscv_vfmv_v_f_##suffix##m1(0, vl), a, n, vl); \ } \ template<> inline _Tpvec v_rotate_left<0>(const _Tpvec& a) \ { return a; } \ @@ -1546,11 +1590,11 @@ OPENCV_HAL_IMPL_RVV_ROTATE_FP(v_float64, f64, VTraits::vlanes()) #if CV_SIMD_SCALABLE_FP16 inline v_float16 v_cvt_f16(const v_float32 &a) { - return __riscv_vfncvt_f(__riscv_vlmul_ext_f32m4(a), VTraits::vlanes()); + return __riscv_vfncvt_f(__riscv_vlmul_ext_f32m2(a), VTraits::vlanes()); } inline v_float16 v_cvt_f16(const v_float32 &a, const v_float32 &b) { - return __riscv_vfncvt_f(__riscv_vset(__riscv_vlmul_ext_f32m4(a),1,b), VTraits::vlanes()); + return __riscv_vfncvt_f(__riscv_vset(__riscv_vlmul_ext_f32m2(a),1,b), VTraits::vlanes()); } inline v_float16 v_cvt_f16(const v_int16 &a) { @@ -1558,48 +1602,48 @@ inline v_float16 v_cvt_f16(const v_int16 &a) } inline v_float32 v_cvt_f32(const v_float16 &a) { - return __riscv_vget_f32m2(__riscv_vfwcvt_f(a, VTraits::vlanes()), 0); + return __riscv_vget_f32m1(__riscv_vfwcvt_f(a, VTraits::vlanes()), 0); } inline v_float32 v_cvt_f32_high(const v_float16 &a) { - return __riscv_vget_f32m2(__riscv_vfwcvt_f(a, VTraits::vlanes()), 1); + return __riscv_vget_f32m1(__riscv_vfwcvt_f(a, VTraits::vlanes()), 1); } #endif inline v_float32 v_cvt_f32(const v_int32& a) { - return __riscv_vfcvt_f_x_v_f32m2(a, VTraits::vlanes()); + return __riscv_vfcvt_f_x_v_f32m1(a, VTraits::vlanes()); } #if CV_SIMD_SCALABLE_64F inline v_float32 v_cvt_f32(const v_float64& a) { - return __riscv_vfncvt_f(__riscv_vlmul_ext_f64m4(a), VTraits::vlanes()); + return __riscv_vfncvt_f(__riscv_vlmul_ext_f64m2(a), VTraits::vlanes()); } inline v_float32 v_cvt_f32(const v_float64& a, const v_float64& b) { - return __riscv_vfncvt_f(__riscv_vset(__riscv_vlmul_ext_f64m4(a),1,b), VTraits::vlanes()); + return __riscv_vfncvt_f(__riscv_vset(__riscv_vlmul_ext_f64m2(a),1,b), VTraits::vlanes()); } inline v_float64 v_cvt_f64(const v_int32& a) { - return __riscv_vget_f64m2(__riscv_vfwcvt_f(a, VTraits::vlanes()), 0); + return __riscv_vget_f64m1(__riscv_vfwcvt_f(a, VTraits::vlanes()), 0); } inline v_float64 v_cvt_f64_high(const v_int32& a) { - return __riscv_vget_f64m2(__riscv_vfwcvt_f(a, VTraits::vlanes()), 1); + return __riscv_vget_f64m1(__riscv_vfwcvt_f(a, VTraits::vlanes()), 1); } inline v_float64 v_cvt_f64(const v_float32& a) { - return __riscv_vget_f64m2(__riscv_vfwcvt_f(a, VTraits::vlanes()), 0); + return __riscv_vget_f64m1(__riscv_vfwcvt_f(a, VTraits::vlanes()), 0); } inline v_float64 v_cvt_f64_high(const v_float32& a) { - return __riscv_vget_f64m2(__riscv_vfwcvt_f(a, VTraits::vlanes()), 1); + return __riscv_vget_f64m1(__riscv_vfwcvt_f(a, VTraits::vlanes()), 1); } inline v_float64 v_cvt_f64(const v_int64& a) @@ -1632,7 +1676,7 @@ OPENCV_HAL_IMPL_RVV_BROADCAST(v_float32, f32) #define OPENCV_HAL_IMPL_RVV_REVERSE(_Tpvec, width) \ inline _Tpvec v_reverse(const _Tpvec& a) \ { \ - vuint##width##m2_t vidx = __riscv_vrsub(__riscv_vid_v_u##width##m2(VTraits<_Tpvec>::vlanes()), VTraits<_Tpvec>::vlanes()-1, VTraits<_Tpvec>::vlanes()); \ + vuint##width##m1_t vidx = __riscv_vrsub(__riscv_vid_v_u##width##m1(VTraits<_Tpvec>::vlanes()), VTraits<_Tpvec>::vlanes()-1, VTraits<_Tpvec>::vlanes()); \ return __riscv_vrgather(a, vidx, VTraits<_Tpvec>::vlanes()); \ } OPENCV_HAL_IMPL_RVV_REVERSE(v_uint8, 8) @@ -1657,30 +1701,30 @@ OPENCV_HAL_IMPL_RVV_REVERSE(v_float64, 64) inline void v_expand(const _Tpvec& a, _Tpwvec& b0, _Tpwvec& b1) \ { \ _Tpwvec_m2 temp = cvt(a, VTraits<_Tpvec>::vlanes()); \ - b0 = __riscv_vget_##suffix##m2(temp, 0); \ - b1 = __riscv_vget_##suffix##m2(temp, 1); \ + b0 = __riscv_vget_##suffix##m1(temp, 0); \ + b1 = __riscv_vget_##suffix##m1(temp, 1); \ } \ inline _Tpwvec v_expand_low(const _Tpvec& a) \ { \ _Tpwvec_m2 temp = cvt(a, VTraits<_Tpvec>::vlanes()); \ - return __riscv_vget_##suffix##m2(temp, 0); \ + return __riscv_vget_##suffix##m1(temp, 0); \ } \ inline _Tpwvec v_expand_high(const _Tpvec& a) \ { \ _Tpwvec_m2 temp = cvt(a, VTraits<_Tpvec>::vlanes()); \ - return __riscv_vget_##suffix##m2(temp, 1); \ + return __riscv_vget_##suffix##m1(temp, 1); \ } \ inline _Tpwvec v_load_expand(const _Tp* ptr) \ { \ - return cvt(__riscv_vle##width##_v_##suffix2##m1(ptr, VTraits<_Tpvec>::vlanes()), VTraits<_Tpvec>::vlanes()); \ + return cvt(__riscv_vle##width##_v_##suffix2##mf2(ptr, VTraits<_Tpvec>::vlanes()), VTraits<_Tpvec>::vlanes()); \ } -OPENCV_HAL_IMPL_RVV_EXPAND(uchar, v_uint16, vuint16m4_t, v_uint8, 8, u16, u8, __riscv_vwcvtu_x) -OPENCV_HAL_IMPL_RVV_EXPAND(schar, v_int16, vint16m4_t, v_int8, 8, i16, i8, __riscv_vwcvt_x) -OPENCV_HAL_IMPL_RVV_EXPAND(ushort, v_uint32, vuint32m4_t, v_uint16, 16, u32, u16, __riscv_vwcvtu_x) -OPENCV_HAL_IMPL_RVV_EXPAND(short, v_int32, vint32m4_t, v_int16, 16, i32, i16, __riscv_vwcvt_x) -OPENCV_HAL_IMPL_RVV_EXPAND(uint, v_uint64, vuint64m4_t, v_uint32, 32, u64, u32, __riscv_vwcvtu_x) -OPENCV_HAL_IMPL_RVV_EXPAND(int, v_int64, vint64m4_t, v_int32, 32, i64, i32, __riscv_vwcvt_x) +OPENCV_HAL_IMPL_RVV_EXPAND(uchar, v_uint16, vuint16m2_t, v_uint8, 8, u16, u8, __riscv_vwcvtu_x) +OPENCV_HAL_IMPL_RVV_EXPAND(schar, v_int16, vint16m2_t, v_int8, 8, i16, i8, __riscv_vwcvt_x) +OPENCV_HAL_IMPL_RVV_EXPAND(ushort, v_uint32, vuint32m2_t, v_uint16, 16, u32, u16, __riscv_vwcvtu_x) +OPENCV_HAL_IMPL_RVV_EXPAND(short, v_int32, vint32m2_t, v_int16, 16, i32, i16, __riscv_vwcvt_x) +OPENCV_HAL_IMPL_RVV_EXPAND(uint, v_uint64, vuint64m2_t, v_uint32, 32, u64, u32, __riscv_vwcvtu_x) +OPENCV_HAL_IMPL_RVV_EXPAND(int, v_int64, vint64m2_t, v_int32, 32, i64, i32, __riscv_vwcvt_x) template ::max_nlanes> inline v_float32 v_load(const float* ptr) @@ -1720,12 +1764,12 @@ template <> inline v_uint32 v_load_expand<4>(const ushort* ptr) inline v_uint32 v_load_expand_q(const uchar* ptr) { - return __riscv_vwcvtu_x(__riscv_vwcvtu_x(__riscv_vle8_v_u8mf2(ptr, VTraits::vlanes()), VTraits::vlanes()), VTraits::vlanes()); + return __riscv_vwcvtu_x(__riscv_vwcvtu_x(__riscv_vle8_v_u8mf4(ptr, VTraits::vlanes()), VTraits::vlanes()), VTraits::vlanes()); } inline v_int32 v_load_expand_q(const schar* ptr) { - return __riscv_vwcvt_x(__riscv_vwcvt_x(__riscv_vle8_v_i8mf2(ptr, VTraits::vlanes()), VTraits::vlanes()), VTraits::vlanes()); + return __riscv_vwcvt_x(__riscv_vwcvt_x(__riscv_vle8_v_i8mf4(ptr, VTraits::vlanes()), VTraits::vlanes()), VTraits::vlanes()); } template ::max_nlanes> @@ -1749,41 +1793,41 @@ template <> inline v_uint32 v_load_expand_q<4>(const uchar* ptr) #define OPENCV_HAL_IMPL_RVV_PACK(_Tpvec, _Tp, _wTpvec, hwidth, hsuffix, suffix, rshr, shr) \ inline _Tpvec v_pack(const _wTpvec& a, const _wTpvec& b) \ { \ - return shr(__riscv_vset(__riscv_vlmul_ext_##suffix##m4(a), 1, b), 0, 0, VTraits<_Tpvec>::vlanes()); \ + return shr(__riscv_vset(__riscv_vlmul_ext_##suffix##m2(a), 1, b), 0, 0, VTraits<_Tpvec>::vlanes()); \ } \ inline void v_pack_store(_Tp* ptr, const _wTpvec& a) \ { \ - __riscv_vse##hwidth##_v_##hsuffix##m1(ptr, shr(a, 0, 0, VTraits<_Tpvec>::vlanes()), VTraits<_wTpvec>::vlanes()); \ + __riscv_vse##hwidth##_v_##hsuffix##mf2(ptr, shr(a, 0, 0, VTraits<_Tpvec>::vlanes()), VTraits<_wTpvec>::vlanes()); \ } \ template inline \ _Tpvec v_rshr_pack(const _wTpvec& a, const _wTpvec& b, int N = n) \ { \ - return rshr(__riscv_vset(__riscv_vlmul_ext_##suffix##m4(a), 1, b), N, 0, VTraits<_Tpvec>::vlanes()); \ + return rshr(__riscv_vset(__riscv_vlmul_ext_##suffix##m2(a), 1, b), N, 0, VTraits<_Tpvec>::vlanes()); \ } \ template inline \ void v_rshr_pack_store(_Tp* ptr, const _wTpvec& a, int N = n) \ { \ - __riscv_vse##hwidth##_v_##hsuffix##m1(ptr, rshr(a, N, 0, VTraits<_Tpvec>::vlanes()), VTraits<_wTpvec>::vlanes()); \ + __riscv_vse##hwidth##_v_##hsuffix##mf2(ptr, rshr(a, N, 0, VTraits<_Tpvec>::vlanes()), VTraits<_wTpvec>::vlanes()); \ } #define OPENCV_HAL_IMPL_RVV_PACK_32(_Tpvec, _Tp, _wTpvec, hwidth, hsuffix, suffix, rshr, shr) \ inline _Tpvec v_pack(const _wTpvec& a, const _wTpvec& b) \ { \ - return shr(__riscv_vset(__riscv_vlmul_ext_##suffix##m4(a), 1, b), 0, VTraits<_Tpvec>::vlanes()); \ + return shr(__riscv_vset(__riscv_vlmul_ext_##suffix##m2(a), 1, b), 0, VTraits<_Tpvec>::vlanes()); \ } \ inline void v_pack_store(_Tp* ptr, const _wTpvec& a) \ { \ - __riscv_vse##hwidth##_v_##hsuffix##m1(ptr, shr(a, 0, VTraits<_Tpvec>::vlanes()), VTraits<_wTpvec>::vlanes()); \ + __riscv_vse##hwidth##_v_##hsuffix##mf2(ptr, shr(a, 0, VTraits<_Tpvec>::vlanes()), VTraits<_wTpvec>::vlanes()); \ } \ template inline \ _Tpvec v_rshr_pack(const _wTpvec& a, const _wTpvec& b, int N = n) \ { \ - return rshr(__riscv_vset(__riscv_vlmul_ext_##suffix##m4(a), 1, b), N, 0, VTraits<_Tpvec>::vlanes()); \ + return rshr(__riscv_vset(__riscv_vlmul_ext_##suffix##m2(a), 1, b), N, 0, VTraits<_Tpvec>::vlanes()); \ } \ template inline \ void v_rshr_pack_store(_Tp* ptr, const _wTpvec& a, int N = n) \ { \ - __riscv_vse##hwidth##_v_##hsuffix##m1(ptr, rshr(a, N, 0, VTraits<_Tpvec>::vlanes()), VTraits<_wTpvec>::vlanes()); \ + __riscv_vse##hwidth##_v_##hsuffix##mf2(ptr, rshr(a, N, 0, VTraits<_Tpvec>::vlanes()), VTraits<_wTpvec>::vlanes()); \ } OPENCV_HAL_IMPL_RVV_PACK(v_uint8, uchar, v_uint16, 8, u8, u16, __riscv_vnclipu, __riscv_vnclipu) @@ -1825,25 +1869,25 @@ template <> inline v_uint16 v_pack<4>(const v_uint32& a, const v_uint32& b) #define OPENCV_HAL_IMPL_RVV_PACK_U(_Tpvec, _Tp, _wTpvec, _wTp, hwidth, width, hsuffix, suffix, cast, hvl, vl) \ inline _Tpvec v_pack_u(const _wTpvec& a, const _wTpvec& b) \ { \ - return __riscv_vnclipu(cast(__riscv_vmax(__riscv_vset(__riscv_vlmul_ext_##suffix##m4(a), 1, b), 0, vl)), 0, 0, vl); \ + return __riscv_vnclipu(cast(__riscv_vmax(__riscv_vset(__riscv_vlmul_ext_##suffix##m2(a), 1, b), 0, vl)), 0, 0, vl); \ } \ inline void v_pack_u_store(_Tp* ptr, const _wTpvec& a) \ { \ - __riscv_vse##hwidth##_v_##hsuffix##m1(ptr, __riscv_vnclipu(__riscv_vreinterpret_u##width##m2(__riscv_vmax(a, 0, vl)), 0, 0, vl), hvl); \ + __riscv_vse##hwidth##_v_##hsuffix##mf2(ptr, __riscv_vnclipu(__riscv_vreinterpret_u##width##m1(__riscv_vmax(a, 0, vl)), 0, 0, vl), hvl); \ } \ template inline \ _Tpvec v_rshr_pack_u(const _wTpvec& a, const _wTpvec& b, int n = N) \ { \ - return __riscv_vnclipu(cast(__riscv_vmax(__riscv_vset(__riscv_vlmul_ext_##suffix##m4(a), 1, b), 0, vl)), n, 0, vl); \ + return __riscv_vnclipu(cast(__riscv_vmax(__riscv_vset(__riscv_vlmul_ext_##suffix##m2(a), 1, b), 0, vl)), n, 0, vl); \ } \ template inline \ void v_rshr_pack_u_store(_Tp* ptr, const _wTpvec& a, int n = N) \ { \ - __riscv_vse##hwidth##_v_##hsuffix##m1(ptr, __riscv_vnclipu(__riscv_vreinterpret_u##width##m2(__riscv_vmax(a, 0, vl)), n, 0, vl), hvl); \ + __riscv_vse##hwidth##_v_##hsuffix##mf2(ptr, __riscv_vnclipu(__riscv_vreinterpret_u##width##m1(__riscv_vmax(a, 0, vl)), n, 0, vl), hvl); \ } -OPENCV_HAL_IMPL_RVV_PACK_U(v_uint8, uchar, v_int16, short, 8, 16, u8, i16, __riscv_vreinterpret_v_i16m4_u16m4, VTraits::vlanes(), VTraits::vlanes()) -OPENCV_HAL_IMPL_RVV_PACK_U(v_uint16, ushort, v_int32, int, 16, 32, u16, i32, __riscv_vreinterpret_v_i32m4_u32m4, VTraits::vlanes(), VTraits::vlanes()) +OPENCV_HAL_IMPL_RVV_PACK_U(v_uint8, uchar, v_int16, short, 8, 16, u8, i16, __riscv_vreinterpret_v_i16m2_u16m2, VTraits::vlanes(), VTraits::vlanes()) +OPENCV_HAL_IMPL_RVV_PACK_U(v_uint16, ushort, v_int32, int, 16, 32, u16, i32, __riscv_vreinterpret_v_i32m2_u32m2, VTraits::vlanes(), VTraits::vlanes()) template ::max_nlanes> inline v_uint16 v_pack_u(const v_int32& a, const v_int32& b) @@ -1901,41 +1945,41 @@ template <> inline void v_pack_store<8>(uchar* ptr, const v_uint16& a) #define OPENCV_HAL_IMPL_RVV_ZIP(_Tpvec, _wTpvec, suffix, width, width2, convert2um2, convert2um1) \ inline void v_zip(const _Tpvec& a0, const _Tpvec& a1, _Tpvec& b0, _Tpvec& b1) { \ - _wTpvec temp = __riscv_vreinterpret_##suffix##m4(convert2um2( \ + _wTpvec temp = __riscv_vreinterpret_##suffix##m2(convert2um2( \ __riscv_vor(__riscv_vzext_vf2(convert2um1(a0), VTraits<_Tpvec>::vlanes()*2), \ - __riscv_vreinterpret_u##width2##m4(__riscv_vslide1up(__riscv_vreinterpret_u##width##m4(__riscv_vzext_vf2(convert2um1(a1), VTraits<_Tpvec>::vlanes()*2)), 0, VTraits<_Tpvec>::vlanes()*2)), \ + __riscv_vreinterpret_u##width2##m2(__riscv_vslide1up(__riscv_vreinterpret_u##width##m2(__riscv_vzext_vf2(convert2um1(a1), VTraits<_Tpvec>::vlanes()*2)), 0, VTraits<_Tpvec>::vlanes()*2)), \ VTraits<_Tpvec>::vlanes()))); \ - b0 = __riscv_vget_##suffix##m2(temp, 0); \ - b1 = __riscv_vget_##suffix##m2(temp, 1); \ + b0 = __riscv_vget_##suffix##m1(temp, 0); \ + b1 = __riscv_vget_##suffix##m1(temp, 1); \ } -OPENCV_HAL_IMPL_RVV_ZIP(v_uint8, vuint8m4_t, u8, 8, 16, OPENCV_HAL_NOP, OPENCV_HAL_NOP) -OPENCV_HAL_IMPL_RVV_ZIP(v_int8, vint8m4_t, i8, 8, 16, __riscv_vreinterpret_u8m4, __riscv_vreinterpret_u8m2) -OPENCV_HAL_IMPL_RVV_ZIP(v_uint16, vuint16m4_t, u16, 16, 32, OPENCV_HAL_NOP, OPENCV_HAL_NOP) -OPENCV_HAL_IMPL_RVV_ZIP(v_int16, vint16m4_t, i16, 16, 32, __riscv_vreinterpret_u16m4, __riscv_vreinterpret_u16m2) +OPENCV_HAL_IMPL_RVV_ZIP(v_uint8, vuint8m2_t, u8, 8, 16, OPENCV_HAL_NOP, OPENCV_HAL_NOP) +OPENCV_HAL_IMPL_RVV_ZIP(v_int8, vint8m2_t, i8, 8, 16, __riscv_vreinterpret_u8m2, __riscv_vreinterpret_u8m1) +OPENCV_HAL_IMPL_RVV_ZIP(v_uint16, vuint16m2_t, u16, 16, 32, OPENCV_HAL_NOP, OPENCV_HAL_NOP) +OPENCV_HAL_IMPL_RVV_ZIP(v_int16, vint16m2_t, i16, 16, 32, __riscv_vreinterpret_u16m2, __riscv_vreinterpret_u16m1) #if CV_SIMD_SCALABLE_FP16 -OPENCV_HAL_IMPL_RVV_ZIP(v_float16, vfloat16m4_t, f16, 16, 32, __riscv_vreinterpret_u16m4, __riscv_vreinterpret_u16m2) +OPENCV_HAL_IMPL_RVV_ZIP(v_float16, vfloat16m2_t, f16, 16, 32, __riscv_vreinterpret_u16m2, __riscv_vreinterpret_u16m1) #endif -OPENCV_HAL_IMPL_RVV_ZIP(v_uint32, vuint32m4_t, u32, 32, 64, OPENCV_HAL_NOP, OPENCV_HAL_NOP) -OPENCV_HAL_IMPL_RVV_ZIP(v_int32, vint32m4_t, i32, 32, 64, __riscv_vreinterpret_u32m4, __riscv_vreinterpret_u32m2) -OPENCV_HAL_IMPL_RVV_ZIP(v_float32, vfloat32m4_t, f32, 32, 64, __riscv_vreinterpret_u32m4, __riscv_vreinterpret_u32m2) +OPENCV_HAL_IMPL_RVV_ZIP(v_uint32, vuint32m2_t, u32, 32, 64, OPENCV_HAL_NOP, OPENCV_HAL_NOP) +OPENCV_HAL_IMPL_RVV_ZIP(v_int32, vint32m2_t, i32, 32, 64, __riscv_vreinterpret_u32m2, __riscv_vreinterpret_u32m1) +OPENCV_HAL_IMPL_RVV_ZIP(v_float32, vfloat32m2_t, f32, 32, 64, __riscv_vreinterpret_u32m2, __riscv_vreinterpret_u32m1) #if CV_SIMD_SCALABLE_64F inline void v_zip(const v_float64& a0, const v_float64& a1, v_float64& b0, v_float64& b1) { \ - vuint16mf2_t idx0 = __riscv_vid_v_u16mf2(VTraits::vlanes()); - vuint16mf2_t idx1 = __riscv_vadd(idx0, VTraits::vlanes(), VTraits::vlanes()); - vuint16m1_t idx = __riscv_vreinterpret_u16m1(( \ + vuint16mf4_t idx0 = __riscv_vid_v_u16mf4(VTraits::vlanes()); + vuint16mf4_t idx1 = __riscv_vadd(idx0, VTraits::vlanes(), VTraits::vlanes()); + vuint16mf2_t idx = __riscv_vreinterpret_u16mf2(( \ __riscv_vor(__riscv_vzext_vf2(idx0, VTraits::vlanes()), \ - __riscv_vreinterpret_u32m1(__riscv_vslide1up(__riscv_vreinterpret_u16m1(__riscv_vzext_vf2(idx1, VTraits::vlanes())), 0, VTraits::vlanes())), \ + __riscv_vreinterpret_u32mf2(__riscv_vslide1up(__riscv_vreinterpret_u16mf2(__riscv_vzext_vf2(idx1, VTraits::vlanes())), 0, VTraits::vlanes())), \ VTraits::vlanes()))); #if 0 - vfloat64m4_t temp = __riscv_vcreate_v_f64m2_f64m4(a0, a1); + vfloat64m2_t temp = __riscv_vcreate_v_f64m1_f64m2(a0, a1); #else // TODO: clean up when RVV Intrinsic is frozen. - vfloat64m4_t temp = __riscv_vlmul_ext_f64m4(a0); + vfloat64m2_t temp = __riscv_vlmul_ext_f64m2(a0); temp = __riscv_vset(temp, 1, a1); #endif temp = __riscv_vrgatherei16(temp, idx, VTraits::vlanes()*2); - b0 = __riscv_vget_f64m2(temp, 0); \ - b1 = __riscv_vget_f64m2(temp, 1); \ + b0 = __riscv_vget_f64m1(temp, 0); \ + b1 = __riscv_vget_f64m1(temp, 1); \ } #endif @@ -1975,23 +2019,23 @@ OPENCV_HAL_IMPL_RVV_UNPACKS(v_float64, 64) #define OPENCV_HAL_IMPL_RVV_INTERLEAVED(_Tpvec, _Tp, _TpCast, suffix, width, hwidth, vl) \ inline void v_load_deinterleave(const _Tp* ptr, v_##_Tpvec& a, v_##_Tpvec& b) \ { \ - a = __riscv_vlse##width##_v_##suffix##m2((_TpCast *)ptr , sizeof(_Tp)*2, VTraits::vlanes()); \ - b = __riscv_vlse##width##_v_##suffix##m2((_TpCast *)(ptr+1), sizeof(_Tp)*2, VTraits::vlanes()); \ + a = __riscv_vlse##width##_v_##suffix##m1((_TpCast *)ptr , sizeof(_Tp)*2, VTraits::vlanes()); \ + b = __riscv_vlse##width##_v_##suffix##m1((_TpCast *)(ptr+1), sizeof(_Tp)*2, VTraits::vlanes()); \ }\ inline void v_load_deinterleave(const _Tp* ptr, v_##_Tpvec& a, v_##_Tpvec& b, v_##_Tpvec& c) \ { \ - a = __riscv_vlse##width##_v_##suffix##m2((_TpCast *)ptr , sizeof(_Tp)*3, VTraits::vlanes()); \ - b = __riscv_vlse##width##_v_##suffix##m2((_TpCast *)(ptr+1), sizeof(_Tp)*3, VTraits::vlanes()); \ - c = __riscv_vlse##width##_v_##suffix##m2((_TpCast *)(ptr+2), sizeof(_Tp)*3, VTraits::vlanes()); \ + a = __riscv_vlse##width##_v_##suffix##m1((_TpCast *)ptr , sizeof(_Tp)*3, VTraits::vlanes()); \ + b = __riscv_vlse##width##_v_##suffix##m1((_TpCast *)(ptr+1), sizeof(_Tp)*3, VTraits::vlanes()); \ + c = __riscv_vlse##width##_v_##suffix##m1((_TpCast *)(ptr+2), sizeof(_Tp)*3, VTraits::vlanes()); \ } \ inline void v_load_deinterleave(const _Tp* ptr, v_##_Tpvec& a, v_##_Tpvec& b, \ v_##_Tpvec& c, v_##_Tpvec& d) \ { \ \ - a = __riscv_vlse##width##_v_##suffix##m2((_TpCast *)ptr , sizeof(_Tp)*4, VTraits::vlanes()); \ - b = __riscv_vlse##width##_v_##suffix##m2((_TpCast *)(ptr+1), sizeof(_Tp)*4, VTraits::vlanes()); \ - c = __riscv_vlse##width##_v_##suffix##m2((_TpCast *)(ptr+2), sizeof(_Tp)*4, VTraits::vlanes()); \ - d = __riscv_vlse##width##_v_##suffix##m2((_TpCast *)(ptr+3), sizeof(_Tp)*4, VTraits::vlanes()); \ + a = __riscv_vlse##width##_v_##suffix##m1((_TpCast *)ptr , sizeof(_Tp)*4, VTraits::vlanes()); \ + b = __riscv_vlse##width##_v_##suffix##m1((_TpCast *)(ptr+1), sizeof(_Tp)*4, VTraits::vlanes()); \ + c = __riscv_vlse##width##_v_##suffix##m1((_TpCast *)(ptr+2), sizeof(_Tp)*4, VTraits::vlanes()); \ + d = __riscv_vlse##width##_v_##suffix##m1((_TpCast *)(ptr+3), sizeof(_Tp)*4, VTraits::vlanes()); \ } \ inline void v_store_interleave( _Tp* ptr, const v_##_Tpvec& a, const v_##_Tpvec& b, \ hal::StoreMode /*mode*/=hal::STORE_UNALIGNED) \ @@ -2055,8 +2099,8 @@ static std::array idx_interleave_quads = { \ #define OPENCV_HAL_IMPL_RVV_INTERLEAVED_PQ_NOEXPEND(_Tpvec, func) \ inline _Tpvec v_interleave_##func(const _Tpvec& vec) { \ CV_CheckLE(VTraits<_Tpvec>::vlanes(), VTraits<_Tpvec>::max_nlanes, "RVV implementation only supports VLEN in the range [128, 1024]"); \ - vuint8m2_t vidx = __riscv_vundefined_u8m2();\ - vidx = __riscv_vreinterpret_u8m2(__riscv_vle64_v_u64m2(idx_interleave_##func.data(), idx_interleave_##func.size())); \ + vuint8m1_t vidx = __riscv_vundefined_u8m1();\ + vidx = __riscv_vreinterpret_u8m1(__riscv_vle64_v_u64m1(idx_interleave_##func.data(), idx_interleave_##func.size())); \ return __riscv_vrgather(vec, vidx, VTraits::vlanes()); \ } OPENCV_HAL_IMPL_RVV_INTERLEAVED_PQ_NOEXPEND(v_uint8, pairs) @@ -2067,8 +2111,8 @@ OPENCV_HAL_IMPL_RVV_INTERLEAVED_PQ_NOEXPEND(v_int8, quads) #define OPENCV_HAL_IMPL_RVV_INTERLEAVED_PQ(_Tpvec, width, vzext_vfx, func) \ inline _Tpvec v_interleave_##func(const _Tpvec& vec) { \ CV_CheckLE(VTraits<_Tpvec>::vlanes(), VTraits<_Tpvec>::max_nlanes, "RVV implementation only supports VLEN in the range [128, 1024]"); \ - vuint##width##m2_t vidx = __riscv_vundefined_u##width##m2();\ - vidx = __riscv_vget_u##width##m2(vzext_vfx(__riscv_vreinterpret_u8m2(__riscv_vle64_v_u64m2(idx_interleave_##func.data(), idx_interleave_##func.size())), VTraits::vlanes()), 0); \ + vuint##width##m1_t vidx = __riscv_vundefined_u##width##m1();\ + vidx = __riscv_vget_u##width##m1(vzext_vfx(__riscv_vreinterpret_u8m1(__riscv_vle64_v_u64m1(idx_interleave_##func.data(), idx_interleave_##func.size())), VTraits::vlanes()), 0); \ return __riscv_vrgather(vec, vidx, VTraits<_Tpvec>::vlanes()); \ } @@ -2106,20 +2150,20 @@ static const unsigned char popCountTable[256] = }; #define OPENCV_HAL_IMPL_RVV_HADD(_Tpvec, _Tpvec2, _Tm2, width, width2, suffix, add) \ static inline _Tpvec2 v_hadd(_Tpvec a) { \ - vuint##width2##m2_t oneX2 = __riscv_vmv_v_x_u##width2##m2(1, VTraits::vlanes()); \ - vuint##width##m2_t one = __riscv_vreinterpret_u##width##m2(oneX2); \ + vuint##width2##m1_t oneX2 = __riscv_vmv_v_x_u##width2##m1(1, VTraits::vlanes()); \ + vuint##width##m1_t one = __riscv_vreinterpret_u##width##m1(oneX2); \ _Tm2 res = add(a, __riscv_vslide1down(a, 0, VTraits::vlanes()), VTraits::vlanes()); \ - return __riscv_vget_##suffix##m2(__riscv_vcompress(res, __riscv_vmseq(one, 1, VTraits::vlanes()), VTraits::vlanes()), 0); \ + return __riscv_vget_##suffix##m1(__riscv_vcompress(res, __riscv_vmseq(one, 1, VTraits::vlanes()), VTraits::vlanes()), 0); \ } -OPENCV_HAL_IMPL_RVV_HADD(v_uint8, v_uint16, vuint16m4_t, 8, 16, u16, __riscv_vwaddu_vv) -OPENCV_HAL_IMPL_RVV_HADD(v_uint16, v_uint32, vuint32m4_t, 16, 32, u32, __riscv_vwaddu_vv) -OPENCV_HAL_IMPL_RVV_HADD(v_uint32, v_uint64, vuint64m4_t, 32, 64, u64, __riscv_vwaddu_vv) -OPENCV_HAL_IMPL_RVV_HADD(v_int8, v_int16, vint16m4_t, 8, 16, i16, __riscv_vwadd_vv) -OPENCV_HAL_IMPL_RVV_HADD(v_int16, v_int32, vint32m4_t, 16, 32, i32, __riscv_vwadd_vv) -OPENCV_HAL_IMPL_RVV_HADD(v_int32, v_int64, vint64m4_t, 32, 64, i64, __riscv_vwadd_vv) +OPENCV_HAL_IMPL_RVV_HADD(v_uint8, v_uint16, vuint16m2_t, 8, 16, u16, __riscv_vwaddu_vv) +OPENCV_HAL_IMPL_RVV_HADD(v_uint16, v_uint32, vuint32m2_t, 16, 32, u32, __riscv_vwaddu_vv) +OPENCV_HAL_IMPL_RVV_HADD(v_uint32, v_uint64, vuint64m2_t, 32, 64, u64, __riscv_vwaddu_vv) +OPENCV_HAL_IMPL_RVV_HADD(v_int8, v_int16, vint16m2_t, 8, 16, i16, __riscv_vwadd_vv) +OPENCV_HAL_IMPL_RVV_HADD(v_int16, v_int32, vint32m2_t, 16, 32, i32, __riscv_vwadd_vv) +OPENCV_HAL_IMPL_RVV_HADD(v_int32, v_int64, vint64m2_t, 32, 64, i64, __riscv_vwadd_vv) -OPENCV_HAL_IMPL_RVV_HADD(vint32m4_t, v_int32, vint32m4_t, 16, 32, i32, __riscv_vadd) -OPENCV_HAL_IMPL_RVV_HADD(vint64m4_t, v_int64, vint64m4_t, 32, 64, i64, __riscv_vadd) +OPENCV_HAL_IMPL_RVV_HADD(vint32m2_t, v_int32, vint32m2_t, 16, 32, i32, __riscv_vadd) +OPENCV_HAL_IMPL_RVV_HADD(vint64m2_t, v_int64, vint64m2_t, 32, 64, i64, __riscv_vadd) inline v_uint8 v_popcount(const v_uint8& a) { @@ -2127,32 +2171,32 @@ inline v_uint8 v_popcount(const v_uint8& a) } inline v_uint16 v_popcount(const v_uint16& a) { - return v_hadd(v_popcount(__riscv_vreinterpret_u8m2(a))); + return v_hadd(v_popcount(__riscv_vreinterpret_u8m1(a))); } inline v_uint32 v_popcount(const v_uint32& a) { - return v_hadd(v_hadd(v_popcount(__riscv_vreinterpret_u8m2(a)))); + return v_hadd(v_hadd(v_popcount(__riscv_vreinterpret_u8m1(a)))); } inline v_uint64 v_popcount(const v_uint64& a) { - return v_hadd(v_hadd(v_hadd(v_popcount(__riscv_vreinterpret_u8m2(a))))); + return v_hadd(v_hadd(v_hadd(v_popcount(__riscv_vreinterpret_u8m1(a))))); } inline v_uint8 v_popcount(const v_int8& a) { - return v_popcount(__riscv_vreinterpret_u8m2(a));\ + return v_popcount(__riscv_vreinterpret_u8m1(a));\ } inline v_uint16 v_popcount(const v_int16& a) { - return v_popcount(__riscv_vreinterpret_u16m2(a));\ + return v_popcount(__riscv_vreinterpret_u16m1(a));\ } inline v_uint32 v_popcount(const v_int32& a) { - return v_popcount(__riscv_vreinterpret_u32m2(a));\ + return v_popcount(__riscv_vreinterpret_u32m1(a));\ } inline v_uint64 v_popcount(const v_int64& a) { - return v_popcount(__riscv_vreinterpret_u64m2(a)); + return v_popcount(__riscv_vreinterpret_u64m1(a)); } @@ -2215,23 +2259,23 @@ inline int v_scan_forward(const v_float64& a) #define OPENCV_HAL_IMPL_RVV_PACK_TRIPLETS(_Tpvec, v_trunc) \ inline _Tpvec v_pack_triplets(const _Tpvec& vec) { \ size_t vl = VTraits::vlanes(); \ - vuint32m2_t one = __riscv_vmv_v_x_u32m2(1, VTraits::vlanes()); \ - vuint8m2_t zero = __riscv_vmv_v_x_u8m2(0, vl); \ - vuint8m2_t mask = __riscv_vreinterpret_u8m2(one); \ + vuint32m1_t one = __riscv_vmv_v_x_u32m1(1, VTraits::vlanes()); \ + vuint8m1_t zero = __riscv_vmv_v_x_u8m1(0, vl); \ + vuint8m1_t mask = __riscv_vreinterpret_u8m1(one); \ return __riscv_vcompress(vec, __riscv_vmseq(v_trunc(__riscv_vslideup(zero, mask, 3, vl)), 0, vl), VTraits<_Tpvec>::vlanes()); \ } OPENCV_HAL_IMPL_RVV_PACK_TRIPLETS(v_uint8, OPENCV_HAL_NOP) OPENCV_HAL_IMPL_RVV_PACK_TRIPLETS(v_int8, OPENCV_HAL_NOP) -OPENCV_HAL_IMPL_RVV_PACK_TRIPLETS(v_uint16, __riscv_vlmul_trunc_u8m1) -OPENCV_HAL_IMPL_RVV_PACK_TRIPLETS(v_int16, __riscv_vlmul_trunc_u8m1) -OPENCV_HAL_IMPL_RVV_PACK_TRIPLETS(v_uint32, __riscv_vlmul_trunc_u8mf2) -OPENCV_HAL_IMPL_RVV_PACK_TRIPLETS(v_int32, __riscv_vlmul_trunc_u8mf2) -OPENCV_HAL_IMPL_RVV_PACK_TRIPLETS(v_float32, __riscv_vlmul_trunc_u8mf2) -OPENCV_HAL_IMPL_RVV_PACK_TRIPLETS(v_uint64, __riscv_vlmul_trunc_u8mf4) -OPENCV_HAL_IMPL_RVV_PACK_TRIPLETS(v_int64, __riscv_vlmul_trunc_u8mf4) +OPENCV_HAL_IMPL_RVV_PACK_TRIPLETS(v_uint16, __riscv_vlmul_trunc_u8mf2) +OPENCV_HAL_IMPL_RVV_PACK_TRIPLETS(v_int16, __riscv_vlmul_trunc_u8mf2) +OPENCV_HAL_IMPL_RVV_PACK_TRIPLETS(v_uint32, __riscv_vlmul_trunc_u8mf4) +OPENCV_HAL_IMPL_RVV_PACK_TRIPLETS(v_int32, __riscv_vlmul_trunc_u8mf4) +OPENCV_HAL_IMPL_RVV_PACK_TRIPLETS(v_float32, __riscv_vlmul_trunc_u8mf4) +OPENCV_HAL_IMPL_RVV_PACK_TRIPLETS(v_uint64, __riscv_vlmul_trunc_u8mf8) +OPENCV_HAL_IMPL_RVV_PACK_TRIPLETS(v_int64, __riscv_vlmul_trunc_u8mf8) #if CV_SIMD_SCALABLE_64F -OPENCV_HAL_IMPL_RVV_PACK_TRIPLETS(v_float64, __riscv_vlmul_trunc_u8mf4) +OPENCV_HAL_IMPL_RVV_PACK_TRIPLETS(v_float64, __riscv_vlmul_trunc_u8mf8) #endif @@ -2240,12 +2284,12 @@ OPENCV_HAL_IMPL_RVV_PACK_TRIPLETS(v_float64, __riscv_vlmul_trunc_u8mf4) #if defined(__riscv_zfh) && __riscv_zfh inline v_float32 v_load_expand(const hfloat* ptr) { - return __riscv_vfwcvt_f(__riscv_vle16_v_f16m1((_Float16*)ptr, VTraits::vlanes()) ,VTraits::vlanes());; + return __riscv_vfwcvt_f(__riscv_vle16_v_f16mf2((_Float16*)ptr, VTraits::vlanes()) ,VTraits::vlanes());; } inline void v_pack_store(hfloat* ptr, const v_float32& v) { - __riscv_vse16_v_f16m1((_Float16*)ptr, __riscv_vfncvt_f_f_w_f16m1(v, VTraits::vlanes()), VTraits::vlanes()); + __riscv_vse16_v_f16mf2((_Float16*)ptr, __riscv_vfncvt_f_f_w_f16mf2(v, VTraits::vlanes()), VTraits::vlanes()); } #else inline v_float32 v_load_expand(const hfloat* ptr) @@ -2271,12 +2315,20 @@ inline v_int16 v_round(const v_float16& a) inline v_int16 v_floor(const v_float16& a) { - return __riscv_vfcvt_x_f_v_i16m2_rm(a, 1 /*RNE, round-to-nearest-even*/, VTraits::vlanes()); +#if defined(__riscv_v_intrinsic) && __riscv_v_intrinsic>11999 + return __riscv_vfcvt_x_f_v_i16m1_rm(a, 1 /*RNE, round-to-nearest-even*/, VTraits::vlanes()); +#else + return __riscv_vfcvt_x(vfsub(a, 0.5f - 1e-5, VTraits::vlanes()), VTraits::vlanes()); +#endif } inline v_int16 v_ceil(const v_float16& a) { - return __riscv_vfcvt_x_f_v_i16m2_rm(a, 3 /*ROD, round-to-odd*/, VTraits::vlanes()); +#if defined(__riscv_v_intrinsic) && __riscv_v_intrinsic>11999 + return __riscv_vfcvt_x_f_v_i16m1_rm(a, 3 /*ROD, round-to-odd*/, VTraits::vlanes()); +#else + return __riscv_vfcvt_x(vfadd(a, 0.5f - 1e-5, VTraits::vlanes()), VTraits::vlanes()); +#endif } inline v_int16 v_trunc(const v_float16& a) @@ -2308,29 +2360,29 @@ inline v_int32 v_trunc(const v_float32& a) #if CV_SIMD_SCALABLE_64F inline v_int32 v_round(const v_float64& a) { - return __riscv_vfncvt_x(__riscv_vlmul_ext_f64m4(a), VTraits::vlanes()); + return __riscv_vfncvt_x(__riscv_vlmul_ext_f64m2(a), VTraits::vlanes()); } inline v_int32 v_round(const v_float64& a, const v_float64& b) { // return vfncvt_x(vset(vlmul_ext_f64m2(vfadd(a, 1e-6, VTraits::vlanes())), 1, b), VTraits::vlanes()); // Fix https://github.com/opencv/opencv/issues/24746 - return __riscv_vfncvt_x(__riscv_vset(__riscv_vlmul_ext_f64m4(a), 1, b), VTraits::vlanes()); + return __riscv_vfncvt_x(__riscv_vset(__riscv_vlmul_ext_f64m2(a), 1, b), VTraits::vlanes()); } inline v_int32 v_floor(const v_float64& a) { - return __riscv_vfncvt_x(__riscv_vlmul_ext_f64m4(__riscv_vfsub(a, 0.5f - 1e-6, VTraits::vlanes())), VTraits::vlanes()); + return __riscv_vfncvt_x(__riscv_vlmul_ext_f64m2(__riscv_vfsub(a, 0.5f - 1e-6, VTraits::vlanes())), VTraits::vlanes()); } inline v_int32 v_ceil(const v_float64& a) { - return __riscv_vfncvt_x(__riscv_vlmul_ext_f64m4(__riscv_vfadd(a, 0.5f - 1e-6, VTraits::vlanes())), VTraits::vlanes()); + return __riscv_vfncvt_x(__riscv_vlmul_ext_f64m2(__riscv_vfadd(a, 0.5f - 1e-6, VTraits::vlanes())), VTraits::vlanes()); } inline v_int32 v_trunc(const v_float64& a) { - return __riscv_vfncvt_rtz_x(__riscv_vlmul_ext_f64m4(a), VTraits::vlanes()); + return __riscv_vfncvt_rtz_x(__riscv_vlmul_ext_f64m2(a), VTraits::vlanes()); } #endif @@ -2339,154 +2391,154 @@ inline v_int32 v_trunc(const v_float64& a) // 16 >> 32 inline v_int32 v_dotprod(const v_int16& a, const v_int16& b) { - vint32m4_t temp1 = __riscv_vwmul(a, b, VTraits::vlanes()); + vint32m2_t temp1 = __riscv_vwmul(a, b, VTraits::vlanes()); return v_hadd(temp1); } inline v_int32 v_dotprod(const v_int16& a, const v_int16& b, const v_int32& c) { - vint32m4_t temp1 = __riscv_vwmul(a, b, VTraits::vlanes()); + vint32m2_t temp1 = __riscv_vwmul(a, b, VTraits::vlanes()); return __riscv_vadd(v_hadd(temp1), c, VTraits::vlanes()); } // 32 >> 64 inline v_int64 v_dotprod(const v_int32& a, const v_int32& b) { - vuint64m2_t one64 = __riscv_vmv_v_x_u64m2(1, VTraits::vlanes()); \ - vuint32m2_t one32 = __riscv_vreinterpret_u32m2(one64); \ - vbool16_t mask = __riscv_vmseq(one32, 1, VTraits::vlanes()); \ - vint64m4_t temp1 = __riscv_vwmul(a, b, VTraits::vlanes()); \ - vint64m4_t temp2 = __riscv_vslide1down(temp1, 0, VTraits::vlanes()); - vint64m4_t res = __riscv_vadd(temp1, temp2, VTraits::vlanes()); + vuint64m1_t one64 = __riscv_vmv_v_x_u64m1(1, VTraits::vlanes()); \ + vuint32m1_t one32 = __riscv_vreinterpret_u32m1(one64); \ + vbool32_t mask = __riscv_vmseq(one32, 1, VTraits::vlanes()); \ + vint64m2_t temp1 = __riscv_vwmul(a, b, VTraits::vlanes()); \ + vint64m2_t temp2 = __riscv_vslide1down(temp1, 0, VTraits::vlanes()); + vint64m2_t res = __riscv_vadd(temp1, temp2, VTraits::vlanes()); res = __riscv_vcompress(res, mask, VTraits::vlanes()); \ - return __riscv_vlmul_trunc_i64m2(res); \ + return __riscv_vlmul_trunc_i64m1(res); \ } inline v_int64 v_dotprod(const v_int32& a, const v_int32& b, const v_int64& c) { - vuint64m2_t one64 = __riscv_vmv_v_x_u64m2(1, VTraits::vlanes()); \ - vuint32m2_t one32 = __riscv_vreinterpret_u32m2(one64); \ - vbool16_t mask = __riscv_vmseq(one32, 1, VTraits::vlanes()); \ - vint64m4_t temp1 = __riscv_vwmul(a, b, VTraits::vlanes()); \ - vint64m4_t temp2 = __riscv_vslide1down(temp1, 0, VTraits::vlanes()); - vint64m4_t res = __riscv_vadd(temp1, temp2, VTraits::vlanes()); + vuint64m1_t one64 = __riscv_vmv_v_x_u64m1(1, VTraits::vlanes()); \ + vuint32m1_t one32 = __riscv_vreinterpret_u32m1(one64); \ + vbool32_t mask = __riscv_vmseq(one32, 1, VTraits::vlanes()); \ + vint64m2_t temp1 = __riscv_vwmul(a, b, VTraits::vlanes()); \ + vint64m2_t temp2 = __riscv_vslide1down(temp1, 0, VTraits::vlanes()); + vint64m2_t res = __riscv_vadd(temp1, temp2, VTraits::vlanes()); res = __riscv_vcompress(res, mask, VTraits::vlanes()); \ - return __riscv_vadd(__riscv_vlmul_trunc_i64m2(res), c, VTraits::vlanes()); \ + return __riscv_vadd(__riscv_vlmul_trunc_i64m1(res), c, VTraits::vlanes()); \ } // 8 >> 32 inline v_uint32 v_dotprod_expand(const v_uint8& a, const v_uint8& b) { - vuint32m2_t one32 = __riscv_vmv_v_x_u32m2(1, VTraits::vlanes()); \ - vuint8m2_t one8 = __riscv_vreinterpret_u8m2(one32); \ - vbool4_t mask = __riscv_vmseq(one8, 1, VTraits::vlanes()); \ - vuint16m4_t t0 = __riscv_vwmulu(a, b, VTraits::vlanes()); \ - vuint16m4_t t1= __riscv_vslide1down(t0, 0, VTraits::vlanes()); - vuint16m4_t t2= __riscv_vslide1down(t1, 0, VTraits::vlanes()); - vuint16m4_t t3= __riscv_vslide1down(t2, 0, VTraits::vlanes()); - vuint32m8_t res = __riscv_vadd(__riscv_vwaddu_vv(t2, t3, VTraits::vlanes()), __riscv_vwaddu_vv(t0, t1, VTraits::vlanes()), VTraits::vlanes()); + vuint32m1_t one32 = __riscv_vmv_v_x_u32m1(1, VTraits::vlanes()); \ + vuint8m1_t one8 = __riscv_vreinterpret_u8m1(one32); \ + vbool8_t mask = __riscv_vmseq(one8, 1, VTraits::vlanes()); \ + vuint16m2_t t0 = __riscv_vwmulu(a, b, VTraits::vlanes()); \ + vuint16m2_t t1= __riscv_vslide1down(t0, 0, VTraits::vlanes()); + vuint16m2_t t2= __riscv_vslide1down(t1, 0, VTraits::vlanes()); + vuint16m2_t t3= __riscv_vslide1down(t2, 0, VTraits::vlanes()); + vuint32m4_t res = __riscv_vadd(__riscv_vwaddu_vv(t2, t3, VTraits::vlanes()), __riscv_vwaddu_vv(t0, t1, VTraits::vlanes()), VTraits::vlanes()); res = __riscv_vcompress(res, mask, VTraits::vlanes()); \ - return __riscv_vlmul_trunc_u32m2(res); + return __riscv_vlmul_trunc_u32m1(res); } inline v_uint32 v_dotprod_expand(const v_uint8& a, const v_uint8& b, const v_uint32& c) { - vuint32m2_t one32 = __riscv_vmv_v_x_u32m2(1, VTraits::vlanes()); \ - vuint8m2_t one8 = __riscv_vreinterpret_u8m2(one32); \ - vbool4_t mask = __riscv_vmseq(one8, 1, VTraits::vlanes()); \ - vuint16m4_t t0 = __riscv_vwmulu(a, b, VTraits::vlanes()); \ - vuint16m4_t t1= __riscv_vslide1down(t0, 0, VTraits::vlanes()); - vuint16m4_t t2= __riscv_vslide1down(t1, 0, VTraits::vlanes()); - vuint16m4_t t3= __riscv_vslide1down(t2, 0, VTraits::vlanes()); - vuint32m8_t res = __riscv_vadd(__riscv_vwaddu_vv(t2, t3, VTraits::vlanes()), __riscv_vwaddu_vv(t0, t1, VTraits::vlanes()), VTraits::vlanes()); + vuint32m1_t one32 = __riscv_vmv_v_x_u32m1(1, VTraits::vlanes()); \ + vuint8m1_t one8 = __riscv_vreinterpret_u8m1(one32); \ + vbool8_t mask = __riscv_vmseq(one8, 1, VTraits::vlanes()); \ + vuint16m2_t t0 = __riscv_vwmulu(a, b, VTraits::vlanes()); \ + vuint16m2_t t1= __riscv_vslide1down(t0, 0, VTraits::vlanes()); + vuint16m2_t t2= __riscv_vslide1down(t1, 0, VTraits::vlanes()); + vuint16m2_t t3= __riscv_vslide1down(t2, 0, VTraits::vlanes()); + vuint32m4_t res = __riscv_vadd(__riscv_vwaddu_vv(t2, t3, VTraits::vlanes()), __riscv_vwaddu_vv(t0, t1, VTraits::vlanes()), VTraits::vlanes()); res = __riscv_vcompress(res, mask, VTraits::vlanes()); \ - return __riscv_vadd(__riscv_vlmul_trunc_u32m2(res), c, VTraits::vlanes()); + return __riscv_vadd(__riscv_vlmul_trunc_u32m1(res), c, VTraits::vlanes()); } inline v_int32 v_dotprod_expand(const v_int8& a, const v_int8& b) { - vuint32m2_t one32 = __riscv_vmv_v_x_u32m2(1, VTraits::vlanes()); \ - vuint8m2_t one8 = __riscv_vreinterpret_u8m2(one32); \ - vbool4_t mask = __riscv_vmseq(one8, 1, VTraits::vlanes()); \ - vint16m4_t t0 = __riscv_vwmul(a, b, VTraits::vlanes()); \ - vint16m4_t t1= __riscv_vslide1down(t0, 0, VTraits::vlanes()); - vint16m4_t t2= __riscv_vslide1down(t1, 0, VTraits::vlanes()); - vint16m4_t t3= __riscv_vslide1down(t2, 0, VTraits::vlanes()); - vint32m8_t res = __riscv_vadd(__riscv_vwadd_vv(t2, t3, VTraits::vlanes()), __riscv_vwadd_vv(t0, t1, VTraits::vlanes()), VTraits::vlanes()); + vuint32m1_t one32 = __riscv_vmv_v_x_u32m1(1, VTraits::vlanes()); \ + vuint8m1_t one8 = __riscv_vreinterpret_u8m1(one32); \ + vbool8_t mask = __riscv_vmseq(one8, 1, VTraits::vlanes()); \ + vint16m2_t t0 = __riscv_vwmul(a, b, VTraits::vlanes()); \ + vint16m2_t t1= __riscv_vslide1down(t0, 0, VTraits::vlanes()); + vint16m2_t t2= __riscv_vslide1down(t1, 0, VTraits::vlanes()); + vint16m2_t t3= __riscv_vslide1down(t2, 0, VTraits::vlanes()); + vint32m4_t res = __riscv_vadd(__riscv_vwadd_vv(t2, t3, VTraits::vlanes()), __riscv_vwadd_vv(t0, t1, VTraits::vlanes()), VTraits::vlanes()); res = __riscv_vcompress(res, mask, VTraits::vlanes()); \ - return __riscv_vlmul_trunc_i32m2(res); + return __riscv_vlmul_trunc_i32m1(res); } inline v_int32 v_dotprod_expand(const v_int8& a, const v_int8& b, const v_int32& c) { - vuint32m2_t one32 = __riscv_vmv_v_x_u32m2(1, VTraits::vlanes()); \ - vuint8m2_t one8 = __riscv_vreinterpret_u8m2(one32); \ - vbool4_t mask = __riscv_vmseq(one8, 1, VTraits::vlanes()); \ - vint16m4_t t0 = __riscv_vwmul(a, b, VTraits::vlanes()); \ - vint16m4_t t1= __riscv_vslide1down(t0, 0, VTraits::vlanes()); - vint16m4_t t2= __riscv_vslide1down(t1, 0, VTraits::vlanes()); - vint16m4_t t3= __riscv_vslide1down(t2, 0, VTraits::vlanes()); - vint32m8_t res = __riscv_vadd(__riscv_vwadd_vv(t2, t3, VTraits::vlanes()), __riscv_vwadd_vv(t0, t1, VTraits::vlanes()), VTraits::vlanes()); + vuint32m1_t one32 = __riscv_vmv_v_x_u32m1(1, VTraits::vlanes()); \ + vuint8m1_t one8 = __riscv_vreinterpret_u8m1(one32); \ + vbool8_t mask = __riscv_vmseq(one8, 1, VTraits::vlanes()); \ + vint16m2_t t0 = __riscv_vwmul(a, b, VTraits::vlanes()); \ + vint16m2_t t1= __riscv_vslide1down(t0, 0, VTraits::vlanes()); + vint16m2_t t2= __riscv_vslide1down(t1, 0, VTraits::vlanes()); + vint16m2_t t3= __riscv_vslide1down(t2, 0, VTraits::vlanes()); + vint32m4_t res = __riscv_vadd(__riscv_vwadd_vv(t2, t3, VTraits::vlanes()), __riscv_vwadd_vv(t0, t1, VTraits::vlanes()), VTraits::vlanes()); res = __riscv_vcompress(res, mask, VTraits::vlanes()); \ - return __riscv_vadd(__riscv_vlmul_trunc_i32m2(res), c, VTraits::vlanes()); + return __riscv_vadd(__riscv_vlmul_trunc_i32m1(res), c, VTraits::vlanes()); } // // 16 >> 64 inline v_uint64 v_dotprod_expand(const v_uint16& a, const v_uint16& b) { - vuint64m2_t one64 = __riscv_vmv_v_x_u64m2(1, VTraits::vlanes()); \ - vuint16m2_t one16 = __riscv_vreinterpret_u16m2(one64); \ - vbool8_t mask = __riscv_vmseq(one16, 1, VTraits::vlanes()); \ - vuint32m4_t t0 = __riscv_vwmulu(a, b, VTraits::vlanes()); \ - vuint32m4_t t1= __riscv_vslide1down(t0, 0, VTraits::vlanes()); - vuint32m4_t t2= __riscv_vslide1down(t1, 0, VTraits::vlanes()); - vuint32m4_t t3= __riscv_vslide1down(t2, 0, VTraits::vlanes()); - vuint64m8_t res = __riscv_vadd(__riscv_vwaddu_vv(t2, t3, VTraits::vlanes()), __riscv_vwaddu_vv(t0, t1, VTraits::vlanes()), VTraits::vlanes()); + vuint64m1_t one64 = __riscv_vmv_v_x_u64m1(1, VTraits::vlanes()); \ + vuint16m1_t one16 = __riscv_vreinterpret_u16m1(one64); \ + vbool16_t mask = __riscv_vmseq(one16, 1, VTraits::vlanes()); \ + vuint32m2_t t0 = __riscv_vwmulu(a, b, VTraits::vlanes()); \ + vuint32m2_t t1= __riscv_vslide1down(t0, 0, VTraits::vlanes()); + vuint32m2_t t2= __riscv_vslide1down(t1, 0, VTraits::vlanes()); + vuint32m2_t t3= __riscv_vslide1down(t2, 0, VTraits::vlanes()); + vuint64m4_t res = __riscv_vadd(__riscv_vwaddu_vv(t2, t3, VTraits::vlanes()), __riscv_vwaddu_vv(t0, t1, VTraits::vlanes()), VTraits::vlanes()); res = __riscv_vcompress(res, mask, VTraits::vlanes()); \ - return __riscv_vlmul_trunc_u64m2(res); + return __riscv_vlmul_trunc_u64m1(res); } inline v_uint64 v_dotprod_expand(const v_uint16& a, const v_uint16& b, const v_uint64& c) { - vuint64m2_t one64 = __riscv_vmv_v_x_u64m2(1, VTraits::vlanes()); \ - vuint16m2_t one16 = __riscv_vreinterpret_u16m2(one64); \ - vbool8_t mask = __riscv_vmseq(one16, 1, VTraits::vlanes()); \ - vuint32m4_t t0 = __riscv_vwmulu(a, b, VTraits::vlanes()); \ - vuint32m4_t t1= __riscv_vslide1down(t0, 0, VTraits::vlanes()); - vuint32m4_t t2= __riscv_vslide1down(t1, 0, VTraits::vlanes()); - vuint32m4_t t3= __riscv_vslide1down(t2, 0, VTraits::vlanes()); - vuint64m8_t res = __riscv_vadd(__riscv_vwaddu_vv(t2, t3, VTraits::vlanes()), __riscv_vwaddu_vv(t0, t1, VTraits::vlanes()), VTraits::vlanes()); + vuint64m1_t one64 = __riscv_vmv_v_x_u64m1(1, VTraits::vlanes()); \ + vuint16m1_t one16 = __riscv_vreinterpret_u16m1(one64); \ + vbool16_t mask = __riscv_vmseq(one16, 1, VTraits::vlanes()); \ + vuint32m2_t t0 = __riscv_vwmulu(a, b, VTraits::vlanes()); \ + vuint32m2_t t1= __riscv_vslide1down(t0, 0, VTraits::vlanes()); + vuint32m2_t t2= __riscv_vslide1down(t1, 0, VTraits::vlanes()); + vuint32m2_t t3= __riscv_vslide1down(t2, 0, VTraits::vlanes()); + vuint64m4_t res = __riscv_vadd(__riscv_vwaddu_vv(t2, t3, VTraits::vlanes()), __riscv_vwaddu_vv(t0, t1, VTraits::vlanes()), VTraits::vlanes()); res = __riscv_vcompress(res, mask, VTraits::vlanes()); \ - return __riscv_vadd(__riscv_vlmul_trunc_u64m2(res), c, VTraits::vlanes()); + return __riscv_vadd(__riscv_vlmul_trunc_u64m1(res), c, VTraits::vlanes()); } inline v_int64 v_dotprod_expand(const v_int16& a, const v_int16& b) { - vuint64m2_t one64 = __riscv_vmv_v_x_u64m2(1, VTraits::vlanes()); \ - vuint16m2_t one16 = __riscv_vreinterpret_u16m2(one64); \ - vbool8_t mask = __riscv_vmseq(one16, 1, VTraits::vlanes()); \ - vint32m4_t t0 = __riscv_vwmul(a, b, VTraits::vlanes()); \ - vint32m4_t t1= __riscv_vslide1down(t0, 0, VTraits::vlanes()); - vint32m4_t t2= __riscv_vslide1down(t1, 0, VTraits::vlanes()); - vint32m4_t t3= __riscv_vslide1down(t2, 0, VTraits::vlanes()); - vint64m8_t res = __riscv_vadd(__riscv_vwadd_vv(t2, t3, VTraits::vlanes()), __riscv_vwadd_vv(t0, t1, VTraits::vlanes()), VTraits::vlanes()); + vuint64m1_t one64 = __riscv_vmv_v_x_u64m1(1, VTraits::vlanes()); \ + vuint16m1_t one16 = __riscv_vreinterpret_u16m1(one64); \ + vbool16_t mask = __riscv_vmseq(one16, 1, VTraits::vlanes()); \ + vint32m2_t t0 = __riscv_vwmul(a, b, VTraits::vlanes()); \ + vint32m2_t t1= __riscv_vslide1down(t0, 0, VTraits::vlanes()); + vint32m2_t t2= __riscv_vslide1down(t1, 0, VTraits::vlanes()); + vint32m2_t t3= __riscv_vslide1down(t2, 0, VTraits::vlanes()); + vint64m4_t res = __riscv_vadd(__riscv_vwadd_vv(t2, t3, VTraits::vlanes()), __riscv_vwadd_vv(t0, t1, VTraits::vlanes()), VTraits::vlanes()); res = __riscv_vcompress(res, mask, VTraits::vlanes()); \ - return __riscv_vlmul_trunc_i64m2(res); + return __riscv_vlmul_trunc_i64m1(res); } inline v_int64 v_dotprod_expand(const v_int16& a, const v_int16& b, const v_int64& c) { - vuint64m2_t one64 = __riscv_vmv_v_x_u64m2(1, VTraits::vlanes()); \ - vuint16m2_t one16 = __riscv_vreinterpret_u16m2(one64); \ - vbool8_t mask = __riscv_vmseq(one16, 1, VTraits::vlanes()); \ - vint32m4_t t0 = __riscv_vwmul(a, b, VTraits::vlanes()); \ - vint32m4_t t1= __riscv_vslide1down(t0, 0, VTraits::vlanes()); - vint32m4_t t2= __riscv_vslide1down(t1, 0, VTraits::vlanes()); - vint32m4_t t3= __riscv_vslide1down(t2, 0, VTraits::vlanes()); - vint64m8_t res = __riscv_vadd(__riscv_vwadd_vv(t2, t3, VTraits::vlanes()), __riscv_vwadd_vv(t0, t1, VTraits::vlanes()), VTraits::vlanes()); + vuint64m1_t one64 = __riscv_vmv_v_x_u64m1(1, VTraits::vlanes()); \ + vuint16m1_t one16 = __riscv_vreinterpret_u16m1(one64); \ + vbool16_t mask = __riscv_vmseq(one16, 1, VTraits::vlanes()); \ + vint32m2_t t0 = __riscv_vwmul(a, b, VTraits::vlanes()); \ + vint32m2_t t1= __riscv_vslide1down(t0, 0, VTraits::vlanes()); + vint32m2_t t2= __riscv_vslide1down(t1, 0, VTraits::vlanes()); + vint32m2_t t3= __riscv_vslide1down(t2, 0, VTraits::vlanes()); + vint64m4_t res = __riscv_vadd(__riscv_vwadd_vv(t2, t3, VTraits::vlanes()), __riscv_vwadd_vv(t0, t1, VTraits::vlanes()), VTraits::vlanes()); res = __riscv_vcompress(res, mask, VTraits::vlanes()); \ - return __riscv_vadd(__riscv_vlmul_trunc_i64m2(res), c, VTraits::vlanes()); + return __riscv_vadd(__riscv_vlmul_trunc_i64m1(res), c, VTraits::vlanes()); } // // 32 >> 64f @@ -2502,72 +2554,70 @@ inline v_float64 v_dotprod_expand(const v_int32& a, const v_int32& b, // 16 >> 32 inline v_int32 v_dotprod_fast(const v_int16& a, const v_int16& b) { - vint32m1_t zero = __riscv_vmv_v_x_i32m1(0, VTraits::vlanes()); - return __riscv_vset(__riscv_vmv_v_x_i32m2(0, VTraits::vlanes()), 0, __riscv_vredsum_tu(zero, __riscv_vwmul(a, b, VTraits::vlanes()), zero, VTraits::vlanes())); + v_int32 zero = v_setzero_s32(); + return __riscv_vredsum_tu(zero, __riscv_vwmul(a, b, VTraits::vlanes()), zero, VTraits::vlanes()); } inline v_int32 v_dotprod_fast(const v_int16& a, const v_int16& b, const v_int32& c) { - vint32m1_t zero = __riscv_vmv_v_x_i32m1(0, VTraits::vlanes()); - return __riscv_vadd(c, __riscv_vset(__riscv_vmv_v_x_i32m2(0, VTraits::vlanes()), 0, __riscv_vredsum_tu(zero, __riscv_vwmul(a, b, VTraits::vlanes()), zero, VTraits::vlanes())), VTraits::vlanes()); + v_int32 zero = v_setzero_s32(); + return __riscv_vredsum_tu(zero, __riscv_vwmul(a, b, VTraits::vlanes()), __riscv_vredsum_tu(zero,c, zero, VTraits::vlanes()), VTraits::vlanes()); } // 32 >> 64 inline v_int64 v_dotprod_fast(const v_int32& a, const v_int32& b) { - vint64m1_t zero = __riscv_vmv_v_x_i64m1(0, VTraits::vlanes()); - return __riscv_vset(__riscv_vmv_v_x_i64m2(0, VTraits::vlanes()), 0, __riscv_vredsum_tu(zero, __riscv_vwmul(a, b, VTraits::vlanes()), zero, VTraits::vlanes())); + v_int64 zero = v_setzero_s64(); + return __riscv_vredsum_tu(zero, __riscv_vwmul(a, b, VTraits::vlanes()), zero, VTraits::vlanes()); } inline v_int64 v_dotprod_fast(const v_int32& a, const v_int32& b, const v_int64& c) { - vint64m1_t zero = __riscv_vmv_v_x_i64m1(0, VTraits::vlanes()); - return __riscv_vadd(c, __riscv_vset(__riscv_vmv_v_x_i64m2(0, VTraits::vlanes()), 0, __riscv_vredsum_tu(zero, __riscv_vwmul(a, b, VTraits::vlanes()), zero, VTraits::vlanes())), VTraits::vlanes()); + v_int64 zero = v_setzero_s64(); + return __riscv_vadd(__riscv_vredsum_tu(zero,__riscv_vwmul(a, b, VTraits::vlanes()), zero, VTraits::vlanes()) , __riscv_vredsum_tu(zero,c, zero, VTraits::vlanes()), VTraits::vlanes()); } // 8 >> 32 inline v_uint32 v_dotprod_expand_fast(const v_uint8& a, const v_uint8& b) { - vuint32m1_t zero = __riscv_vmv_v_x_u32m1(0, VTraits::vlanes()); - auto res = __riscv_vwredsumu_tu(zero, __riscv_vwmulu(a, b, VTraits::vlanes()), zero, VTraits::vlanes()); - return __riscv_vset(__riscv_vmv_v_x_u32m2(0, VTraits::vlanes()), 0, res); + v_uint32 zero = v_setzero_u32(); + return __riscv_vwredsumu_tu(zero, __riscv_vwmulu(a, b, VTraits::vlanes()), zero, VTraits::vlanes()); } inline v_uint32 v_dotprod_expand_fast(const v_uint8& a, const v_uint8& b, const v_uint32& c) { - vuint32m1_t zero = __riscv_vmv_v_x_u32m1(0, VTraits::vlanes()); - auto res = __riscv_vwredsumu_tu(zero, __riscv_vwmulu(a, b, VTraits::vlanes()), zero, VTraits::vlanes()); - return __riscv_vadd(c, __riscv_vset(__riscv_vmv_v_x_u32m2(0, VTraits::vlanes()), 0, res), VTraits::vlanes()); + v_uint32 zero = v_setzero_u32(); + return __riscv_vadd(__riscv_vwredsumu_tu(zero, __riscv_vwmulu(a, b, VTraits::vlanes()), zero, VTraits::vlanes()) , __riscv_vredsum_tu(zero, c, zero, VTraits::vlanes()), VTraits::vlanes()); } inline v_int32 v_dotprod_expand_fast(const v_int8& a, const v_int8& b) { - vint32m1_t zero = __riscv_vmv_v_x_i32m1(0, VTraits::vlanes()); - return __riscv_vset(__riscv_vmv_v_x_i32m2(0, VTraits::vlanes()), 0, __riscv_vwredsum_tu(zero, __riscv_vwmul(a, b, VTraits::vlanes()), zero, VTraits::vlanes())); + v_int32 zero = v_setzero_s32(); + return __riscv_vwredsum_tu(zero, __riscv_vwmul(a, b, VTraits::vlanes()), zero, VTraits::vlanes()); } inline v_int32 v_dotprod_expand_fast(const v_int8& a, const v_int8& b, const v_int32& c) { - vint32m1_t zero = __riscv_vmv_v_x_i32m1(0, VTraits::vlanes()); - return __riscv_vadd(c, __riscv_vset(__riscv_vmv_v_x_i32m2(0, VTraits::vlanes()), 0, __riscv_vwredsum_tu(zero, __riscv_vwmul(a, b, VTraits::vlanes()), zero, VTraits::vlanes())), VTraits::vlanes()); + v_int32 zero = v_setzero_s32(); + return __riscv_vadd(__riscv_vwredsum_tu(zero, __riscv_vwmul(a, b, VTraits::vlanes()), zero, VTraits::vlanes()) , __riscv_vredsum_tu(zero,c, zero, VTraits::vlanes()), VTraits::vlanes()); } // 16 >> 64 inline v_uint64 v_dotprod_expand_fast(const v_uint16& a, const v_uint16& b) { - vuint64m1_t zero = __riscv_vmv_v_x_u64m1(0, VTraits::vlanes()); - return __riscv_vset(__riscv_vmv_v_x_u64m2(0, VTraits::vlanes()), 0, __riscv_vwredsumu_tu(zero, __riscv_vwmulu(a, b, VTraits::vlanes()), zero, VTraits::vlanes())); + v_uint64 zero = v_setzero_u64(); + return __riscv_vwredsumu_tu(zero, __riscv_vwmulu(a, b, VTraits::vlanes()), zero, VTraits::vlanes()); } inline v_uint64 v_dotprod_expand_fast(const v_uint16& a, const v_uint16& b, const v_uint64& c) { - vuint64m1_t zero = __riscv_vmv_v_x_u64m1(0, VTraits::vlanes()); - return __riscv_vadd(c, __riscv_vset(__riscv_vmv_v_x_u64m2(0, VTraits::vlanes()), 0, __riscv_vwredsumu_tu(zero, __riscv_vwmulu(a, b, VTraits::vlanes()), zero, VTraits::vlanes())), VTraits::vlanes()); + v_uint64 zero = v_setzero_u64(); + return __riscv_vadd(__riscv_vwredsumu_tu(zero, __riscv_vwmulu(a, b, VTraits::vlanes()), zero, VTraits::vlanes()), __riscv_vredsum_tu(zero,c, zero, VTraits::vlanes()), VTraits::vlanes()); } inline v_int64 v_dotprod_expand_fast(const v_int16& a, const v_int16& b) { - vint64m1_t zero = __riscv_vmv_v_x_i64m1(0, VTraits::vlanes()); - return __riscv_vset(__riscv_vmv_v_x_i64m2(0, VTraits::vlanes()), 0, __riscv_vwredsum_tu(zero, __riscv_vwmul(a, b, VTraits::vlanes()), zero, VTraits::vlanes())); + v_int64 zero = v_setzero_s64(); + return __riscv_vwredsum_tu(zero, __riscv_vwmul(a, b, VTraits::vlanes()), zero, VTraits::vlanes()); } inline v_int64 v_dotprod_expand_fast(const v_int16& a, const v_int16& b, const v_int64& c) { - vint64m1_t zero = __riscv_vmv_v_x_i64m1(0, VTraits::vlanes()); - return __riscv_vadd(c, __riscv_vset(__riscv_vmv_v_x_i64m2(0, VTraits::vlanes()), 0, __riscv_vwredsum_tu(zero, __riscv_vwmul(a, b, VTraits::vlanes()), zero, VTraits::vlanes())), VTraits::vlanes()); + v_int64 zero = v_setzero_s64(); + return __riscv_vadd(__riscv_vwredsum_tu(zero, __riscv_vwmul(a, b, VTraits::vlanes()), zero, VTraits::vlanes()), __riscv_vredsum_tu(zero, c, zero, VTraits::vlanes()), VTraits::vlanes()); } // 32 >> 64f @@ -2578,11 +2628,8 @@ inline v_float64 v_dotprod_expand_fast(const v_int32& a, const v_int32& b) auto prod_i64 = __riscv_vwmul(a, b, VTraits::vlanes()); // Convert to f64 before reduction to avoid overflow: #27003 auto prod_f64 = __riscv_vfcvt_f(prod_i64, VTraits::vlanes()); - return __riscv_vset( // Needs v_float64 (vfloat64m2_t) here. - v_setall_f64(0.0f), // zero_f64m2 - 0, - __riscv_vfredusum_tu(zero, prod_f64, zero, VTraits::vlanes()) - ); + // At LMUL=1 the reduction result (f64m1) is already a full v_float64. + return __riscv_vfredusum_tu(zero, prod_f64, zero, VTraits::vlanes()); } inline v_float64 v_dotprod_expand_fast(const v_int32& a, const v_int32& b, const v_float64& c) { return v_add(v_dotprod_expand_fast(a, b) , c); } @@ -2595,15 +2642,15 @@ inline v_float16 v_matmul( const v_float16 &v, const v_float16 &m2, const v_float16 &m3, const v_float16 &m4, const v_float16 &m5, const v_float16 &m6, const v_float16 &m7) { - vfloat16m2_t res; - res = __riscv_vfmul_vf_f16m2(m0, (_Float16)v_extract_n(v, 0), VTraits::vlanes()); - res = __riscv_vfmacc_vf_f16m2(res, (_Float16)v_extract_n(v, 1), m1, VTraits::vlanes()); - res = __riscv_vfmacc_vf_f16m2(res, (_Float16)v_extract_n(v, 2), m2, VTraits::vlanes()); - res = __riscv_vfmacc_vf_f16m2(res, (_Float16)v_extract_n(v, 3), m3, VTraits::vlanes()); - res = __riscv_vfmacc_vf_f16m2(res, (_Float16)v_extract_n(v, 4), m4, VTraits::vlanes()); - res = __riscv_vfmacc_vf_f16m2(res, (_Float16)v_extract_n(v, 5), m5, VTraits::vlanes()); - res = __riscv_vfmacc_vf_f16m2(res, (_Float16)v_extract_n(v, 6), m6, VTraits::vlanes()); - res = __riscv_vfmacc_vf_f16m2(res, (_Float16)v_extract_n(v, 7), m7, VTraits::vlanes()); + vfloat16m1_t res; + res = __riscv_vfmul_vf_f16m1(m0, (_Float16)v_extract_n(v, 0), VTraits::vlanes()); + res = __riscv_vfmacc_vf_f16m1(res, (_Float16)v_extract_n(v, 1), m1, VTraits::vlanes()); + res = __riscv_vfmacc_vf_f16m1(res, (_Float16)v_extract_n(v, 2), m2, VTraits::vlanes()); + res = __riscv_vfmacc_vf_f16m1(res, (_Float16)v_extract_n(v, 3), m3, VTraits::vlanes()); + res = __riscv_vfmacc_vf_f16m1(res, (_Float16)v_extract_n(v, 4), m4, VTraits::vlanes()); + res = __riscv_vfmacc_vf_f16m1(res, (_Float16)v_extract_n(v, 5), m5, VTraits::vlanes()); + res = __riscv_vfmacc_vf_f16m1(res, (_Float16)v_extract_n(v, 6), m6, VTraits::vlanes()); + res = __riscv_vfmacc_vf_f16m1(res, (_Float16)v_extract_n(v, 7), m7, VTraits::vlanes()); return res; } inline v_float16 v_matmuladd( const v_float16 &v, @@ -2612,14 +2659,14 @@ inline v_float16 v_matmuladd( const v_float16 &v, const v_float16 &m4, const v_float16 &m5, const v_float16 &m6, const v_float16 &a) { - vfloat16m2_t res; - res = __riscv_vfmul_vf_f16m2(m0, (_Float16)v_extract_n(v, 0), VTraits::vlanes()); - res = __riscv_vfmacc_vf_f16m2(res, (_Float16)v_extract_n(v, 1), m1, VTraits::vlanes()); - res = __riscv_vfmacc_vf_f16m2(res, (_Float16)v_extract_n(v, 2), m2, VTraits::vlanes()); - res = __riscv_vfmacc_vf_f16m2(res, (_Float16)v_extract_n(v, 3), m3, VTraits::vlanes()); - res = __riscv_vfmacc_vf_f16m2(res, (_Float16)v_extract_n(v, 4), m4, VTraits::vlanes()); - res = __riscv_vfmacc_vf_f16m2(res, (_Float16)v_extract_n(v, 5), m5, VTraits::vlanes()); - res = __riscv_vfmacc_vf_f16m2(res, (_Float16)v_extract_n(v, 6), m6, VTraits::vlanes()); + vfloat16m1_t res; + res = __riscv_vfmul_vf_f16m1(m0, (_Float16)v_extract_n(v, 0), VTraits::vlanes()); + res = __riscv_vfmacc_vf_f16m1(res, (_Float16)v_extract_n(v, 1), m1, VTraits::vlanes()); + res = __riscv_vfmacc_vf_f16m1(res, (_Float16)v_extract_n(v, 2), m2, VTraits::vlanes()); + res = __riscv_vfmacc_vf_f16m1(res, (_Float16)v_extract_n(v, 3), m3, VTraits::vlanes()); + res = __riscv_vfmacc_vf_f16m1(res, (_Float16)v_extract_n(v, 4), m4, VTraits::vlanes()); + res = __riscv_vfmacc_vf_f16m1(res, (_Float16)v_extract_n(v, 5), m5, VTraits::vlanes()); + res = __riscv_vfmacc_vf_f16m1(res, (_Float16)v_extract_n(v, 6), m6, VTraits::vlanes()); return __riscv_vfadd(res, a, VTraits::vlanes()); } #endif @@ -2629,7 +2676,7 @@ inline v_float32 v_matmul(const v_float32& v, const v_float32& mat0, const v_float32& mat3) { const int vl = VTraits::vlanes(); - vuint32m2_t idx = __riscv_vand(__riscv_vid_v_u32m2(vl), 0xfffffffc, vl); + vuint32m1_t idx = __riscv_vand(__riscv_vid_v_u32m1(vl), 0xfffffffc, vl); v_float32 v0 = __riscv_vrgather(v, idx, vl); v_float32 v1 = __riscv_vrgather(v, __riscv_vadd(idx, 1, vl), vl); v_float32 v2 = __riscv_vrgather(v, __riscv_vadd(idx, 2, vl), vl); @@ -2642,7 +2689,7 @@ inline v_float32 v_matmuladd(const v_float32& v, const v_float32& mat0, const v_float32& a) { const int vl = VTraits::vlanes(); - vuint32m2_t idx = __riscv_vand(__riscv_vid_v_u32m2(vl), 0xfffffffc, vl); + vuint32m1_t idx = __riscv_vand(__riscv_vid_v_u32m1(vl), 0xfffffffc, vl); v_float32 v0 = __riscv_vrgather(v, idx, vl); v_float32 v1 = __riscv_vrgather(v, __riscv_vadd(idx, 1, vl), vl); v_float32 v2 = __riscv_vrgather(v, __riscv_vadd(idx, 2, vl), vl); @@ -2652,13 +2699,6 @@ inline v_float32 v_matmuladd(const v_float32& v, const v_float32& mat0, inline void v_cleanup() {} #include "intrin_math.hpp" -#if CV_SIMD_SCALABLE_FP16 -inline v_float16 v_exp(const v_float16& x) { return v_exp_default_16f(x); } -inline v_float16 v_log(const v_float16& x) { return v_log_default_16f(x); } -inline void v_sincos(const v_float16& x, v_float16& s, v_float16& c) { v_sincos_default_16f(x, s, c); } -inline v_float16 v_sin(const v_float16& x) { return v_sin_default_16f(x); } -inline v_float16 v_cos(const v_float16& x) { return v_cos_default_16f(x); } -#endif inline v_float32 v_exp(const v_float32& x) { return v_exp_default_32f(x); } inline v_float32 v_log(const v_float32& x) { return v_log_default_32f(x); } inline void v_sincos(const v_float32& x, v_float32& s, v_float32& c) { v_sincos_default_32f(x, s, c); } diff --git a/modules/dnn/src/layers/avgpool_layer.cpp b/modules/dnn/src/layers/avgpool_layer.cpp index 861baec27b..ef5b5f00e3 100644 --- a/modules/dnn/src/layers/avgpool_layer.cpp +++ b/modules/dnn/src/layers/avgpool_layer.cpp @@ -53,8 +53,11 @@ static void avgPool32f(const void* inp_, void* out_, float* out = (float*)out_ + nc0*planesize; float iksize = 1.f/ksize; -#if CV_SIMD || CV_SIMD_SCALABLE +#if CV_SIMD int nlanes = VTraits::vlanes(); + // RVV (CV_SIMD_SCALABLE) disabled for the m1 switch: with the fixed C0=8 the + // block is narrower than the register at VLEN>=512, tripping this assert. Runs + // scalar on RVV; re-enable via v_setvlmax(C0) (#29493, cf #29180). CV_Assert(C0 == nlanes || C0 == nlanes*2 || C0 % (nlanes*4) == 0); v_float32 z = vx_setzero_f32(); v_float32 vscale0 = vx_setall_f32(iksize); @@ -69,12 +72,12 @@ static void avgPool32f(const void* inp_, void* out_, y0 >= inner_y0 && y0 < inner_y1 ? inner_x0 : W; int yi_ = y0*SY - padY0; - #if !(CV_SIMD || CV_SIMD_SCALABLE) + #if !(CV_SIMD) memset(out, 0, W*C0*sizeof(out[0])); #endif for(;;) { - #if CV_SIMD || CV_SIMD_SCALABLE + #if CV_SIMD if (nlanes == C0) { for (; x0 < x1; x0++) { int xi_ = x0*SX - padX0; @@ -162,7 +165,7 @@ static void avgPool32f(const void* inp_, void* out_, break; x1 = inner_x1; - #if CV_SIMD || CV_SIMD_SCALABLE + #if CV_SIMD if (nlanes == C0) { for (; x0 < x1; x0++) { int xi_ = x0*SX - padX0; diff --git a/modules/dnn/src/layers/cpu_kernels/conv2_depthwise.simd.hpp b/modules/dnn/src/layers/cpu_kernels/conv2_depthwise.simd.hpp index 642695dfbe..d43d5a972f 100644 --- a/modules/dnn/src/layers/cpu_kernels/conv2_depthwise.simd.hpp +++ b/modules/dnn/src/layers/cpu_kernels/conv2_depthwise.simd.hpp @@ -38,14 +38,7 @@ static void depthwiseConv32f(const void* inp__, const void* residual__, parallel_for_(Range(0, NC1), [&](const Range& range) { constexpr int MAX_CONV_DIMS = ConvState::MAX_CONV_DIMS; -#if CV_SIMD_SCALABLE - // RVV: universal intrinsics use LMUL=2, so the float vector width tracks - // the hardware VLEN at runtime. The block size C0 == defaultC0 == vlanes() - // (e.g. 16 at VLEN=256), so read it at runtime instead of fixing it to 8 (#28852). - const int C0 = (int)cs.inpshape.back(); -#else constexpr int C0 = 8; -#endif CV_Assert(cs.nspatialdims <= MAX_CONV_DIMS && MAX_CONV_DIMS == 3); CV_Assert(C0 == cs.inpshape.back()); @@ -80,14 +73,7 @@ static void depthwiseConv32f(const void* inp__, const void* residual__, const float* activParams = cs.activParams.data(); ActivationFunc activation = cs.activation; float maxval = FLT_MAX, defaultAlpha = 0.f; -#if CV_SIMD_SCALABLE - // C0 is runtime on RVV; size scratch by the compile-time upper bound. - float scalebuf[VTraits::max_nlanes]; - float biasbuf[VTraits::max_nlanes]; - float alphabuf[VTraits::max_nlanes]; -#else float scalebuf[C0], biasbuf[C0], alphabuf[C0]; -#endif if (fastActivation == FAST_ACTIV_CLIP) { CV_Assert(cs.activParams.size() == 2u); maxval = activParams[1]; @@ -103,10 +89,13 @@ static void depthwiseConv32f(const void* inp__, const void* residual__, defaultAlpha = 1.f; } - #if CV_SIMD || CV_SIMD_SCALABLE + #if CV_SIMD v_float32 v_maxval = vx_setall_f32(maxval); v_float32 z = vx_setzero_f32(); const int nlanes = VTraits::vlanes(); + // RVV (CV_SIMD_SCALABLE) disabled for the m1 switch: with the fixed C0=8 the + // block is narrower than the register at VLEN>=512, tripping this assert. Runs + // scalar on RVV; re-enable via v_setvlmax(C0) (#29493, cf #29180). CV_Assert(C0 == nlanes || C0 == nlanes*2 || C0 % (nlanes*4) == 0); #endif @@ -140,12 +129,12 @@ static void depthwiseConv32f(const void* inp__, const void* residual__, y0 >= inner_y0 && y0 < inner_y1 ? inner_x0 : W; int yi_ = y0*SY - padY0; - #if !(CV_SIMD || CV_SIMD_SCALABLE) + #if !(CV_SIMD) memset(out, 0, W*C0*sizeof(out[0])); #endif for(;;) { - #if CV_SIMD || CV_SIMD_SCALABLE + #if CV_SIMD if (nlanes == C0) { v_float32 sc0 = vx_load(scalebuf), b0 = vx_load(biasbuf); v_float32 alpha0 = vx_load(alphabuf); @@ -232,7 +221,7 @@ static void depthwiseConv32f(const void* inp__, const void* residual__, break; x1 = inner_x1; - #if CV_SIMD || CV_SIMD_SCALABLE + #if CV_SIMD if (nlanes == C0) { v_float32 sc0 = vx_load(scalebuf), b0 = vx_load(biasbuf), alpha0 = vx_load(alphabuf); for (; x0 < x1; x0++) { @@ -349,7 +338,7 @@ static void depthwiseConv32f(const void* inp__, const void* residual__, x1 = W; } - #if !(CV_SIMD || CV_SIMD_SCALABLE) + #if !(CV_SIMD) if (residual) { for (int x = 0; x < W*C0; x += C0) { for (int c = 0; c < C0; c++) { diff --git a/modules/dnn/src/layers/cpu_kernels/conv2_kernels.simd.hpp b/modules/dnn/src/layers/cpu_kernels/conv2_kernels.simd.hpp index c49f02b24d..b36d38792e 100644 --- a/modules/dnn/src/layers/cpu_kernels/conv2_kernels.simd.hpp +++ b/modules/dnn/src/layers/cpu_kernels/conv2_kernels.simd.hpp @@ -386,7 +386,13 @@ CV_CPU_OPTIMIZATION_NAMESPACE_BEGIN CONV_FINALIZE_OUT2(8, 9, CONV_ADD_NO_RESIDUAL2); \ } -#elif CV_SIMD_SCALABLE +// TODO(#29493): RVV conv is temporarily scalar. This universal path assumed +// K0 == vlanes(), which under m1 holds only at VLEN=256. Disabled as in #29180. +// Re-enable in a follow-up with a portable v_setvlmax(K0) universal +// intrinsic (no-op on fixed-width ISAs, sets the vl/mask on RVV/SVE) so one vector +// spans exactly the K0=8 block at any VLEN; the optimized multi-pixel case is a +// later RVV-HAL step. See PR #29493 discussion. +#elif 0 // CV_SIMD_SCALABLE: RVV temporarily disabled for the m1 switch (#29493) /////////////////////////// scalable (RVV) implementation ///////////////////////////// // K0 == vlanes(), so each of the 10 spatial positions needs exactly one vector @@ -603,8 +609,7 @@ static void scatterScalarOut(bool aligned_k, int k_base, int k_count, int K0shif #define CONV_INIT_SCALAR_SUMS() \ v_float32x4 zz = v_setzero_f32(); \ v_float32x4 s0 = zz, s1 = zz -#elif CV_SIMD_SCALABLE -// RVV: K0 == vlanes(), so a single scalable vector spans the whole output block. +#elif 0 // CV_SIMD_SCALABLE: RVV temporarily disabled for the m1 switch (#29493) #define CONV_INIT_SCALAR_SUMS() \ v_float32 zz = vx_setzero_f32(); \ v_float32 s0 = zz @@ -642,7 +647,7 @@ static void scatterScalarOut(bool aligned_k, int k_base, int k_count, int K0shif s0 = v_min(s0, _vmx); s1 = v_min(s1, _vmx); \ v_store((outbuf), s0); v_store((outbuf) + 4, s1); \ } -#elif CV_SIMD_SCALABLE +#elif 0 // CV_SIMD_SCALABLE: RVV temporarily disabled for the m1 switch (#29493) #define CONV_FINALIZE_SCALAR_OUT(outbuf) \ { \ v_float32 _vsc = vx_load(scalebuf); \ @@ -2202,17 +2207,11 @@ static void conv32fC8(const void* inp__, const void* residual__, void* out__, parallel_for_(Range(0, total_tasks_gen), [&](const Range& range) { constexpr int SPAT_BLOCK_SIZE = 10; -#if CV_SIMD_SCALABLE - // RVV: block size follows the runtime vector width (defaultC0 = vlanes(), LMUL=2). - const int C0 = (int)inpshape.back(); - int C0shift = 0; while ((1 << C0shift) < C0) C0shift++; - const int K0 = C0, K0shift = C0shift; - constexpr int C0BUF = VTraits::max_nlanes; // compile-time scratch bound -#else + // The block size is a fixed property of the blocked layout on every platform, + // so C0/K0 are compile-time constants here (#29493). constexpr int C0shift = 3, K0shift = C0shift; constexpr int C0 = 1 << C0shift, K0 = C0; constexpr int C0BUF = K0; -#endif CV_Assert_N(inpshape.back() == C0, outshape.back() == K0); @@ -2523,10 +2522,7 @@ static void conv32fC8(const void* inp__, const void* residual__, void* out__, CONV_UPDATE_BLOCK1(5); CONV_UPDATE_BLOCK1(6); CONV_UPDATE_BLOCK1(7); - #elif CV_SIMD_SCALABLE - // RVV: K0 == vlanes(); weights are contiguous over kk for a fixed c0, - // so one vx_load gives the whole K0-wide weight vector. Broadcast the - // input lane and accumulate into the persistent block accumulator. + #elif 0 // CV_SIMD_SCALABLE: RVV temporarily disabled for the m1 switch (#29493) for (int c0 = 0; c0 < C0; ++c0) { v_float32 w = vx_load(wptr + c0*K0); v_float32 x = vx_setall_f32(inptr[c0]); @@ -2557,16 +2553,9 @@ static void conv32fC8(const void* inp__, const void* residual__, void* out__, cv::dnn::ConvFunc getConvFunc_(int depth, int C0) { ConvFunc func = nullptr; -#if CV_SIMD_SCALABLE - // RVV: block size follows the runtime vector width; accept the supported pow2 widths. - if (depth == CV_32F && (C0 == 8 || C0 == 16 || C0 == 32 || C0 == 64)) { - func = conv32fC8; - } -#else if (depth == CV_32F && C0 == 8) { func = conv32fC8; } -#endif return func; } diff --git a/modules/dnn/src/layers/maxpool_layer.cpp b/modules/dnn/src/layers/maxpool_layer.cpp index 0cc6c5ac4c..bb156d1aba 100644 --- a/modules/dnn/src/layers/maxpool_layer.cpp +++ b/modules/dnn/src/layers/maxpool_layer.cpp @@ -50,9 +50,12 @@ static void maxPool32f(const void* inp_, void* out_, const ConvState& cs) float* out = (float*)out_ + nc0*planesize; const float INITVAL = -FLT_MAX; - #if CV_SIMD || CV_SIMD_SCALABLE + #if CV_SIMD int nlanes = VTraits::vlanes(); v_float32 s_min = vx_setall_f32(INITVAL); + // RVV (CV_SIMD_SCALABLE) disabled for the m1 switch: with the fixed C0=8 the + // block is narrower than the register at VLEN>=512, tripping this assert. Runs + // scalar on RVV; re-enable via v_setvlmax(C0) (#29493, cf #29180). CV_Assert(C0 == nlanes || C0 == nlanes*2 || C0 % (nlanes*4) == 0); #endif @@ -65,13 +68,13 @@ static void maxPool32f(const void* inp_, void* out_, const ConvState& cs) y0 >= inner_y0 && y0 < inner_y1 ? inner_x0 : W; int yi_ = y0*SY - padY0; - #if !(CV_SIMD || CV_SIMD_SCALABLE) + #if !(CV_SIMD) for (int c = 0; c < C0*W; c++) out[c] = INITVAL; #endif for(;;) { - #if CV_SIMD || CV_SIMD_SCALABLE + #if CV_SIMD if (nlanes == C0) { for (; x0 < x1; x0++) { int xi_ = x0*SX - padX0; @@ -136,7 +139,7 @@ static void maxPool32f(const void* inp_, void* out_, const ConvState& cs) break; x1 = inner_x1; - #if CV_SIMD || CV_SIMD_SCALABLE + #if CV_SIMD if (nlanes == C0) { for (; x0 < x1; x0++) { int xi_ = x0*SX - padX0; diff --git a/modules/dnn/src/net_impl.cpp b/modules/dnn/src/net_impl.cpp index 3cc13e497e..dc2444e994 100644 --- a/modules/dnn/src/net_impl.cpp +++ b/modules/dnn/src/net_impl.cpp @@ -5,7 +5,6 @@ #include "precomp.hpp" #include "net_impl.hpp" -#include "opencv2/core/hal/intrin.hpp" #ifdef HAVE_ONNXRUNTIME #include @@ -74,14 +73,10 @@ Net::Impl::Impl() // onnx_opset = 0; accuracy = CV_32F; + // The block size is a network-wide memory-layout contract, so it must not depend + // on the SIMD width of the host: C0 == 8 is the mainstream, well-tested setting. + // (Deriving it from vlanes() made the layout hardware-dependent; see the #29493 discussion.) defaultC0 = DEFAULT_C0; -#if CV_SIMD_SCALABLE - // RVV: the universal intrinsics use LMUL=2, so the float vector width is - // (VLEN/32)*2 (16 at VLEN=256). The blocked-layout kernels fill a full - // vector per channel block, so the net-wide block size must match that - // width rather than the fixed default of 8 (#28852). - defaultC0 = std::max((int)DEFAULT_C0, VTraits::vlanes()); -#endif enableFP16 = haveFP16 = false; // FP16 is not ready yet in the new DNN engine // Ticket: https://github.com/opencv/opencv/issues/26196 diff --git a/modules/dnn/src/net_impl2.cpp b/modules/dnn/src/net_impl2.cpp index 597695b8ef..54915da9a5 100644 --- a/modules/dnn/src/net_impl2.cpp +++ b/modules/dnn/src/net_impl2.cpp @@ -5,7 +5,6 @@ #include "precomp.hpp" #include "net_impl.hpp" -#include "opencv2/core/hal/intrin.hpp" #include @@ -546,16 +545,6 @@ void Net::Impl::prepareForInference() #endif if (!prepared) { -#if CV_SIMD_SCALABLE - // RVV (#28852): keep quantized graphs at C0=8. The int8 kernels are hardwired to - // C0=8 (VNNI/NEON weight packing + per-channel quantization) and run scalar on RVV, - // so a wider block gives no benefit and breaks them. Only fp32 graphs use the wider - // vlanes()-based defaultC0. Signed-int8 (CV_8S) args are the quantization signature - // (uint8 image inputs are CV_8U, so they don't trigger this). - for (const ArgData& a : args) { - if (a.type == CV_8S) { defaultC0 = 8; break; } - } -#endif fuseQDQ(); constFold(); fuseBN();