mirror of
https://github.com/opencv/opencv.git
synced 2026-07-29 15:23:05 +04:00
RVV: switch scalable universal intrinsics from LMUL=2 to LMUL=1 (#29493)
* dnn:rvv: restore the fixed block size C0 == 8
#29304 made the net-wide block size follow the RVV vector width
(defaultC0 = max(8, vlanes())), so the blocked memory layout of every tensor
depended on the host's VLEN and on the LMUL of the universal intrinsics in core.
The same model gets a different layout on a 128-, 256- or 1024-bit machine.
Restore the fixed default of 8, which is the mainstream, well-tested setting on
every platform. C0/K0 become compile-time constants again on RVV as they already
were elsewhere, so the runtime block-size plumbing and the vlanes()-based scratch
sizing are dropped, and getConvFunc_ no longer advertises block sizes the kernels
cannot handle.
The blocked kernels currently require C0 == vlanes(), which this alone does not
satisfy; the next commit removes that requirement.
* dnn:rvv: pin the vector length to the channel block
The blocked kernels assumed one channel block is exactly one vector
(C0 == vlanes()). That is not a property of the layout, it is a property of the
hardware, so the kernels only worked where the two happened to coincide: with
C0 fixed at 8 they break wherever vlanes() != 8 -- under-filling the block when
the vector is narrower (silently leaving output channels unwritten) and
overrunning it when the vector is wider (CV_Assert).
The universal intrinsics cannot express this: v_float32 is always vlanes() wide
and there is no way to make it span exactly C0 elements. So the RVV paths drop to
native intrinsics and set the vector length explicitly:
const size_t _vl = __riscv_vsetvl_e32m2(C0); // == 8 on every VLEN >= 128
LMUL=2 is required, not incidental: vlmax(e32,m2) = VLEN/32*2 >= 8 even at
VLEN=128, the narrowest RVV configuration, so _vl == C0 on every RVV 1.0 core.
One vector then spans precisely one channel block regardless of VLEN, and the
C0-vs-vlanes asserts and VLEN-dependent branches are removed rather than extended.
The existing int8 kernels already choose LMUL=2 for the same reason
(conv2_int8_kernels.simd.hpp).
This is a native per-kernel choice and is independent of the LMUL the universal
intrinsics are built with; dnn no longer reads the vector width of
intrin_rvv_scalable.hpp at all.
conv, deconv, maxpool, avgpool and depthwise use native intrinsics. batch_norm
stays portable and instead replicates the C0-wide coefficient pattern across the
vector, which keeps full lane utilisation where a fixed vl would idle most of it.
The x86/ARM paths are restored to their pre-#29304 form and are unchanged.
Tested on SpaceMIT K3 (X100 VLEN=256, A100 VLEN=1024), opencv_test_dnn: no
regressions against upstream on either core.
* core:rvv: switch scalable universal intrinsics from LMUL=2 to LMUL=1
Move the RVV universal-intrinsic types (v_uint8..v_float64) from m2 (paired
registers, 16 usable) to m1 (individual registers, 32 usable) per #29454, to
relieve register pressure in the complex kernels that rely on universal
intrinsics. Simple kernels that regress can take native RVV branches; the complex
ones cannot, so the default should suit them.
Only intrin_rvv_scalable.hpp changes. The widening/narrowing cascades and
max_nlanes are adjusted, and helpers that spell an LMUL into the intrinsic name
shift down one step (m2->m1, m1->mf2): v_popcount, v_lut (vector and f64 paths),
the f64 dot-product reduction (#27003), v_matmul, and the byte-indexed v_lut
added by #29250, whose __riscv_vluxei8_v_u8m2 no longer matches the m1 base type.
VLEN=1024 needs no change: under m1, max_nlanes = CV_RVV_MAX_VLEN/SZ equals
vlanes() at 1024, and CV_RVV_MAX_VLEN already defaults to 1024. Verified on a
1024-bit core.
Tested on SpaceMIT K3, which exposes both a 256-bit (X100) and a 1024-bit (A100)
core. opencv_test_core: 5605/5612, the 7 failures (filestorage_base64, globbing)
non-SIMD and failing identically on the m2 baseline. Perf (x-factor = m2/m1,
>1 = m1 faster), median >= 300us: core 0.966 -> 0.996 and imgproc 0.950 -> 1.038
as VLEN goes 256 -> 1024, i.e. m2's width advantage disappears once both are wide
while m1's register headroom persists. m1 wins resize (1.50/1.41), Exp (1.28),
gaussianBlur (1.23), norm (1.22), scharr (1.20); m2 still wins the light
memory-streaming kernels addWeighted (0.76), compare (0.82) and dot (0.87), which
are the natural candidates for native RVV branches.
* Revert "dnn:rvv: pin the vector length to the channel block"
This reverts commit 40587c6dd9.
* dnn:rvv: disable CV_SIMD_SCALABLE in the new-engine conv kernel
Under the m1 switch the RVV conv kernel breaks at VLEN=128: it assumes
K0 == vlanes(), but m1 makes vlanes()=4 while the block stays C0=K0=8, so it
computes only half of each output block. Temporarily disable the RVV
(CV_SIMD_SCALABLE) branches so conv runs scalar on RVV, exactly as #29180 did for
the other new-engine layers. The vectorized path will be restored in a follow-up.
Only conv is affected: maxpool/avgpool/depthwise/deconv/batch_norm already handle
C0 == 2*vlanes (the SSE/NEON case) and stay on their universal-intrinsic paths.
* dnn:rvv: document the supported VLEN range and the v_setvlmax follow-up
With the fixed C0=8 the RVV blocked kernels cover VLEN=128 and 256 (C0 >= vlanes);
VLEN>=512 (C0 < vlanes) is not yet handled. Add TODO notes recording this and
pointing at the planned fix: a portable v_setvlmax<VT>(n) universal intrinsic that
caps the vector to the block (no-op on fixed-width ISAs, sets the vl/mask on
RVV/SVE), with an optimized multi-pixel variant as a later step, possibly an RVV
HAL for the new dnn engine. See the #29493 discussion.
* dnn:rvv: disable CV_SIMD_SCALABLE in max/avg pooling and depthwise
These three blocked kernels assert (C0 == nlanes || ...) at VLEN>=512, where the
fixed C0=8 is narrower than the vector register — breaking whole networks
(AlexNet/ResNet/YOLO) on wide RVV cores, though CI (VLEN=128) is unaffected.
Disable their RVV (CV_SIMD_SCALABLE) paths so they run scalar on RVV at every
VLEN, matching #29180's scope. Verified on a SpaceMIT K3: this removes the 92
VLEN=1024 assert failures, with no change at VLEN=128/256.
deconv and batch_norm are left enabled: they degrade to scalar at VLEN>=512
(deconv's C0 % vlanes guard, batch_norm's vector-loop bound) without asserting,
so they stay correct and keep their 128/256 vectorization. All of these
re-vectorize uniformly via the v_setvlmax<VT>(n) follow-up.
This commit is contained in:
File diff suppressed because it is too large
Load Diff
@@ -53,8 +53,11 @@ static void avgPool32f(const void* inp_, void* out_,
|
||||
float* out = (float*)out_ + nc0*planesize;
|
||||
float iksize = 1.f/ksize;
|
||||
|
||||
#if CV_SIMD || CV_SIMD_SCALABLE
|
||||
#if CV_SIMD
|
||||
int nlanes = VTraits<v_float32>::vlanes();
|
||||
// RVV (CV_SIMD_SCALABLE) disabled for the m1 switch: with the fixed C0=8 the
|
||||
// block is narrower than the register at VLEN>=512, tripping this assert. Runs
|
||||
// scalar on RVV; re-enable via v_setvlmax<v_float32>(C0) (#29493, cf #29180).
|
||||
CV_Assert(C0 == nlanes || C0 == nlanes*2 || C0 % (nlanes*4) == 0);
|
||||
v_float32 z = vx_setzero_f32();
|
||||
v_float32 vscale0 = vx_setall_f32(iksize);
|
||||
@@ -69,12 +72,12 @@ static void avgPool32f(const void* inp_, void* out_,
|
||||
y0 >= inner_y0 && y0 < inner_y1 ? inner_x0 : W;
|
||||
int yi_ = y0*SY - padY0;
|
||||
|
||||
#if !(CV_SIMD || CV_SIMD_SCALABLE)
|
||||
#if !(CV_SIMD)
|
||||
memset(out, 0, W*C0*sizeof(out[0]));
|
||||
#endif
|
||||
|
||||
for(;;) {
|
||||
#if CV_SIMD || CV_SIMD_SCALABLE
|
||||
#if CV_SIMD
|
||||
if (nlanes == C0) {
|
||||
for (; x0 < x1; x0++) {
|
||||
int xi_ = x0*SX - padX0;
|
||||
@@ -162,7 +165,7 @@ static void avgPool32f(const void* inp_, void* out_,
|
||||
break;
|
||||
x1 = inner_x1;
|
||||
|
||||
#if CV_SIMD || CV_SIMD_SCALABLE
|
||||
#if CV_SIMD
|
||||
if (nlanes == C0) {
|
||||
for (; x0 < x1; x0++) {
|
||||
int xi_ = x0*SX - padX0;
|
||||
|
||||
@@ -38,14 +38,7 @@ static void depthwiseConv32f(const void* inp__, const void* residual__,
|
||||
parallel_for_(Range(0, NC1), [&](const Range& range)
|
||||
{
|
||||
constexpr int MAX_CONV_DIMS = ConvState::MAX_CONV_DIMS;
|
||||
#if CV_SIMD_SCALABLE
|
||||
// RVV: universal intrinsics use LMUL=2, so the float vector width tracks
|
||||
// the hardware VLEN at runtime. The block size C0 == defaultC0 == vlanes()
|
||||
// (e.g. 16 at VLEN=256), so read it at runtime instead of fixing it to 8 (#28852).
|
||||
const int C0 = (int)cs.inpshape.back();
|
||||
#else
|
||||
constexpr int C0 = 8;
|
||||
#endif
|
||||
|
||||
CV_Assert(cs.nspatialdims <= MAX_CONV_DIMS && MAX_CONV_DIMS == 3);
|
||||
CV_Assert(C0 == cs.inpshape.back());
|
||||
@@ -80,14 +73,7 @@ static void depthwiseConv32f(const void* inp__, const void* residual__,
|
||||
const float* activParams = cs.activParams.data();
|
||||
ActivationFunc activation = cs.activation;
|
||||
float maxval = FLT_MAX, defaultAlpha = 0.f;
|
||||
#if CV_SIMD_SCALABLE
|
||||
// C0 is runtime on RVV; size scratch by the compile-time upper bound.
|
||||
float scalebuf[VTraits<v_float32>::max_nlanes];
|
||||
float biasbuf[VTraits<v_float32>::max_nlanes];
|
||||
float alphabuf[VTraits<v_float32>::max_nlanes];
|
||||
#else
|
||||
float scalebuf[C0], biasbuf[C0], alphabuf[C0];
|
||||
#endif
|
||||
if (fastActivation == FAST_ACTIV_CLIP) {
|
||||
CV_Assert(cs.activParams.size() == 2u);
|
||||
maxval = activParams[1];
|
||||
@@ -103,10 +89,13 @@ static void depthwiseConv32f(const void* inp__, const void* residual__,
|
||||
defaultAlpha = 1.f;
|
||||
}
|
||||
|
||||
#if CV_SIMD || CV_SIMD_SCALABLE
|
||||
#if CV_SIMD
|
||||
v_float32 v_maxval = vx_setall_f32(maxval);
|
||||
v_float32 z = vx_setzero_f32();
|
||||
const int nlanes = VTraits<v_float32>::vlanes();
|
||||
// RVV (CV_SIMD_SCALABLE) disabled for the m1 switch: with the fixed C0=8 the
|
||||
// block is narrower than the register at VLEN>=512, tripping this assert. Runs
|
||||
// scalar on RVV; re-enable via v_setvlmax<v_float32>(C0) (#29493, cf #29180).
|
||||
CV_Assert(C0 == nlanes || C0 == nlanes*2 || C0 % (nlanes*4) == 0);
|
||||
#endif
|
||||
|
||||
@@ -140,12 +129,12 @@ static void depthwiseConv32f(const void* inp__, const void* residual__,
|
||||
y0 >= inner_y0 && y0 < inner_y1 ? inner_x0 : W;
|
||||
int yi_ = y0*SY - padY0;
|
||||
|
||||
#if !(CV_SIMD || CV_SIMD_SCALABLE)
|
||||
#if !(CV_SIMD)
|
||||
memset(out, 0, W*C0*sizeof(out[0]));
|
||||
#endif
|
||||
|
||||
for(;;) {
|
||||
#if CV_SIMD || CV_SIMD_SCALABLE
|
||||
#if CV_SIMD
|
||||
if (nlanes == C0) {
|
||||
v_float32 sc0 = vx_load(scalebuf), b0 = vx_load(biasbuf);
|
||||
v_float32 alpha0 = vx_load(alphabuf);
|
||||
@@ -232,7 +221,7 @@ static void depthwiseConv32f(const void* inp__, const void* residual__,
|
||||
break;
|
||||
x1 = inner_x1;
|
||||
|
||||
#if CV_SIMD || CV_SIMD_SCALABLE
|
||||
#if CV_SIMD
|
||||
if (nlanes == C0) {
|
||||
v_float32 sc0 = vx_load(scalebuf), b0 = vx_load(biasbuf), alpha0 = vx_load(alphabuf);
|
||||
for (; x0 < x1; x0++) {
|
||||
@@ -349,7 +338,7 @@ static void depthwiseConv32f(const void* inp__, const void* residual__,
|
||||
x1 = W;
|
||||
}
|
||||
|
||||
#if !(CV_SIMD || CV_SIMD_SCALABLE)
|
||||
#if !(CV_SIMD)
|
||||
if (residual) {
|
||||
for (int x = 0; x < W*C0; x += C0) {
|
||||
for (int c = 0; c < C0; c++) {
|
||||
|
||||
@@ -386,7 +386,13 @@ CV_CPU_OPTIMIZATION_NAMESPACE_BEGIN
|
||||
CONV_FINALIZE_OUT2(8, 9, CONV_ADD_NO_RESIDUAL2); \
|
||||
}
|
||||
|
||||
#elif CV_SIMD_SCALABLE
|
||||
// TODO(#29493): RVV conv is temporarily scalar. This universal path assumed
|
||||
// K0 == vlanes(), which under m1 holds only at VLEN=256. Disabled as in #29180.
|
||||
// Re-enable in a follow-up with a portable v_setvlmax<v_float32>(K0) universal
|
||||
// intrinsic (no-op on fixed-width ISAs, sets the vl/mask on RVV/SVE) so one vector
|
||||
// spans exactly the K0=8 block at any VLEN; the optimized multi-pixel case is a
|
||||
// later RVV-HAL step. See PR #29493 discussion.
|
||||
#elif 0 // CV_SIMD_SCALABLE: RVV temporarily disabled for the m1 switch (#29493)
|
||||
|
||||
/////////////////////////// scalable (RVV) implementation /////////////////////////////
|
||||
// K0 == vlanes(), so each of the 10 spatial positions needs exactly one vector
|
||||
@@ -603,8 +609,7 @@ static void scatterScalarOut(bool aligned_k, int k_base, int k_count, int K0shif
|
||||
#define CONV_INIT_SCALAR_SUMS() \
|
||||
v_float32x4 zz = v_setzero_f32(); \
|
||||
v_float32x4 s0 = zz, s1 = zz
|
||||
#elif CV_SIMD_SCALABLE
|
||||
// RVV: K0 == vlanes(), so a single scalable vector spans the whole output block.
|
||||
#elif 0 // CV_SIMD_SCALABLE: RVV temporarily disabled for the m1 switch (#29493)
|
||||
#define CONV_INIT_SCALAR_SUMS() \
|
||||
v_float32 zz = vx_setzero_f32(); \
|
||||
v_float32 s0 = zz
|
||||
@@ -642,7 +647,7 @@ static void scatterScalarOut(bool aligned_k, int k_base, int k_count, int K0shif
|
||||
s0 = v_min(s0, _vmx); s1 = v_min(s1, _vmx); \
|
||||
v_store((outbuf), s0); v_store((outbuf) + 4, s1); \
|
||||
}
|
||||
#elif CV_SIMD_SCALABLE
|
||||
#elif 0 // CV_SIMD_SCALABLE: RVV temporarily disabled for the m1 switch (#29493)
|
||||
#define CONV_FINALIZE_SCALAR_OUT(outbuf) \
|
||||
{ \
|
||||
v_float32 _vsc = vx_load(scalebuf); \
|
||||
@@ -2202,17 +2207,11 @@ static void conv32fC8(const void* inp__, const void* residual__, void* out__,
|
||||
|
||||
parallel_for_(Range(0, total_tasks_gen), [&](const Range& range) {
|
||||
constexpr int SPAT_BLOCK_SIZE = 10;
|
||||
#if CV_SIMD_SCALABLE
|
||||
// RVV: block size follows the runtime vector width (defaultC0 = vlanes(), LMUL=2).
|
||||
const int C0 = (int)inpshape.back();
|
||||
int C0shift = 0; while ((1 << C0shift) < C0) C0shift++;
|
||||
const int K0 = C0, K0shift = C0shift;
|
||||
constexpr int C0BUF = VTraits<v_float32>::max_nlanes; // compile-time scratch bound
|
||||
#else
|
||||
// The block size is a fixed property of the blocked layout on every platform,
|
||||
// so C0/K0 are compile-time constants here (#29493).
|
||||
constexpr int C0shift = 3, K0shift = C0shift;
|
||||
constexpr int C0 = 1 << C0shift, K0 = C0;
|
||||
constexpr int C0BUF = K0;
|
||||
#endif
|
||||
|
||||
CV_Assert_N(inpshape.back() == C0, outshape.back() == K0);
|
||||
|
||||
@@ -2523,10 +2522,7 @@ static void conv32fC8(const void* inp__, const void* residual__, void* out__,
|
||||
CONV_UPDATE_BLOCK1(5);
|
||||
CONV_UPDATE_BLOCK1(6);
|
||||
CONV_UPDATE_BLOCK1(7);
|
||||
#elif CV_SIMD_SCALABLE
|
||||
// RVV: K0 == vlanes(); weights are contiguous over kk for a fixed c0,
|
||||
// so one vx_load gives the whole K0-wide weight vector. Broadcast the
|
||||
// input lane and accumulate into the persistent block accumulator.
|
||||
#elif 0 // CV_SIMD_SCALABLE: RVV temporarily disabled for the m1 switch (#29493)
|
||||
for (int c0 = 0; c0 < C0; ++c0) {
|
||||
v_float32 w = vx_load(wptr + c0*K0);
|
||||
v_float32 x = vx_setall_f32(inptr[c0]);
|
||||
@@ -2557,16 +2553,9 @@ static void conv32fC8(const void* inp__, const void* residual__, void* out__,
|
||||
cv::dnn::ConvFunc getConvFunc_(int depth, int C0)
|
||||
{
|
||||
ConvFunc func = nullptr;
|
||||
#if CV_SIMD_SCALABLE
|
||||
// RVV: block size follows the runtime vector width; accept the supported pow2 widths.
|
||||
if (depth == CV_32F && (C0 == 8 || C0 == 16 || C0 == 32 || C0 == 64)) {
|
||||
func = conv32fC8;
|
||||
}
|
||||
#else
|
||||
if (depth == CV_32F && C0 == 8) {
|
||||
func = conv32fC8;
|
||||
}
|
||||
#endif
|
||||
return func;
|
||||
}
|
||||
|
||||
|
||||
@@ -50,9 +50,12 @@ static void maxPool32f(const void* inp_, void* out_, const ConvState& cs)
|
||||
float* out = (float*)out_ + nc0*planesize;
|
||||
const float INITVAL = -FLT_MAX;
|
||||
|
||||
#if CV_SIMD || CV_SIMD_SCALABLE
|
||||
#if CV_SIMD
|
||||
int nlanes = VTraits<v_float32>::vlanes();
|
||||
v_float32 s_min = vx_setall_f32(INITVAL);
|
||||
// RVV (CV_SIMD_SCALABLE) disabled for the m1 switch: with the fixed C0=8 the
|
||||
// block is narrower than the register at VLEN>=512, tripping this assert. Runs
|
||||
// scalar on RVV; re-enable via v_setvlmax<v_float32>(C0) (#29493, cf #29180).
|
||||
CV_Assert(C0 == nlanes || C0 == nlanes*2 || C0 % (nlanes*4) == 0);
|
||||
#endif
|
||||
|
||||
@@ -65,13 +68,13 @@ static void maxPool32f(const void* inp_, void* out_, const ConvState& cs)
|
||||
y0 >= inner_y0 && y0 < inner_y1 ? inner_x0 : W;
|
||||
int yi_ = y0*SY - padY0;
|
||||
|
||||
#if !(CV_SIMD || CV_SIMD_SCALABLE)
|
||||
#if !(CV_SIMD)
|
||||
for (int c = 0; c < C0*W; c++)
|
||||
out[c] = INITVAL;
|
||||
#endif
|
||||
|
||||
for(;;) {
|
||||
#if CV_SIMD || CV_SIMD_SCALABLE
|
||||
#if CV_SIMD
|
||||
if (nlanes == C0) {
|
||||
for (; x0 < x1; x0++) {
|
||||
int xi_ = x0*SX - padX0;
|
||||
@@ -136,7 +139,7 @@ static void maxPool32f(const void* inp_, void* out_, const ConvState& cs)
|
||||
break;
|
||||
x1 = inner_x1;
|
||||
|
||||
#if CV_SIMD || CV_SIMD_SCALABLE
|
||||
#if CV_SIMD
|
||||
if (nlanes == C0) {
|
||||
for (; x0 < x1; x0++) {
|
||||
int xi_ = x0*SX - padX0;
|
||||
|
||||
@@ -5,7 +5,6 @@
|
||||
#include "precomp.hpp"
|
||||
|
||||
#include "net_impl.hpp"
|
||||
#include "opencv2/core/hal/intrin.hpp"
|
||||
|
||||
#ifdef HAVE_ONNXRUNTIME
|
||||
#include <onnxruntime_cxx_api.h>
|
||||
@@ -74,14 +73,10 @@ Net::Impl::Impl()
|
||||
// onnx_opset = 0;
|
||||
|
||||
accuracy = CV_32F;
|
||||
// The block size is a network-wide memory-layout contract, so it must not depend
|
||||
// on the SIMD width of the host: C0 == 8 is the mainstream, well-tested setting.
|
||||
// (Deriving it from vlanes() made the layout hardware-dependent; see the #29493 discussion.)
|
||||
defaultC0 = DEFAULT_C0;
|
||||
#if CV_SIMD_SCALABLE
|
||||
// RVV: the universal intrinsics use LMUL=2, so the float vector width is
|
||||
// (VLEN/32)*2 (16 at VLEN=256). The blocked-layout kernels fill a full
|
||||
// vector per channel block, so the net-wide block size must match that
|
||||
// width rather than the fixed default of 8 (#28852).
|
||||
defaultC0 = std::max((int)DEFAULT_C0, VTraits<v_float32>::vlanes());
|
||||
#endif
|
||||
enableFP16 = haveFP16 = false;
|
||||
// FP16 is not ready yet in the new DNN engine
|
||||
// Ticket: https://github.com/opencv/opencv/issues/26196
|
||||
|
||||
@@ -5,7 +5,6 @@
|
||||
#include "precomp.hpp"
|
||||
|
||||
#include "net_impl.hpp"
|
||||
#include "opencv2/core/hal/intrin.hpp"
|
||||
|
||||
#include <limits>
|
||||
|
||||
@@ -546,16 +545,6 @@ void Net::Impl::prepareForInference()
|
||||
#endif
|
||||
|
||||
if (!prepared) {
|
||||
#if CV_SIMD_SCALABLE
|
||||
// RVV (#28852): keep quantized graphs at C0=8. The int8 kernels are hardwired to
|
||||
// C0=8 (VNNI/NEON weight packing + per-channel quantization) and run scalar on RVV,
|
||||
// so a wider block gives no benefit and breaks them. Only fp32 graphs use the wider
|
||||
// vlanes()-based defaultC0. Signed-int8 (CV_8S) args are the quantization signature
|
||||
// (uint8 image inputs are CV_8U, so they don't trigger this).
|
||||
for (const ArgData& a : args) {
|
||||
if (a.type == CV_8S) { defaultC0 = 8; break; }
|
||||
}
|
||||
#endif
|
||||
fuseQDQ();
|
||||
constFold();
|
||||
fuseBN();
|
||||
|
||||
Reference in New Issue
Block a user