Fix frame seeking with negative DTS values in FFMPEG backend #27878
**Merge with extra**: https://github.com/opencv/opencv_extra/pull/1289
Fixes https://github.com/opencv/opencv/issues/27819
Fixes https://github.com/opencv/opencv/issues/23472
Accompanied by PR on https://github.com/opencv/opencv_extra/pull/1289
The FFmpeg backend fails to correctly seek in H.264 videos that contain negative DTS values in their initial frames. This is a valid encoding practice used by modern video encoders (such as DaVinci Resolve's current export) where B-frame reordering causes the first few frames to have negative DTS values.
When picture_pts is unavailable (AV_NOPTS_VALUE), the code falls back to using pkt_dts:
`picture_pts = packet_raw.pts != AV_NOPTS_VALUE_ ? packet_raw.pts : packet_raw.dts;`
If this DTS value is negative (which is legal per H.264 spec), it propagates through the frame number calculation:
`frame_number = dts_to_frame_number(picture_pts) - first_frame_number;`
This results in negative frame numbers, messing up seeking operations.
Solution implemented in this branch is a timestamp normalization similar to FFmpegs -avoid_negative_ts_make_zero flag:
- Calculate a global offset once on the first decoded frame by getting the minimum timestamp in either:
- Container start_time
- Stream start_time
- First observed timestamp (PTS, then DTS).
- Apply the offset consistently to all timestamps, shifting negative values to begin at 0 while keeping relative timing.
- Simplify timestamp converters to remove `start_time` subtractions since timestamps are pre-normalized.
This also includes a new test `videoio_ffmpeg.seek_with_negative_dts`
This test verifies that seeking behavior performs as expected on a file which has negative DTS values in the first frames.
A PR on opencv_extra accompanies this one with that testing file: https://github.com/opencv/opencv_extra/pull/1279
```
opencv_extra=ffmpeg-videoio-negative-dts-test-data
```
<cut/>
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [ ] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
FIX: CvCapture_FFMPEG::setProperty(CAP_PROP_POS_*) followed by getProperty #27523
Partially fixes#23088 and #23472. This PR fixes `get(CAP_PROP_POS_MSEC)` calls after a `set(CAP_PROP_POS_*)` without calling `read` first.
Since `seek` calls `grabFrame` which already sets `picture_pts`, manually setting `picture_pts` anywhere else in the call stack should not be necessary (except for the special case of seeking to frame 0).
Minimal example from #23088
```cpp
for(int i = 0; i < 3; i++) cap.read(img);
printf("at: %f frames, %f msec\n", cap.get(CAP_PROP_POS_FRAMES), cap.get(CAP_PROP_POS_MSEC));
cap.set(CAP_PROP_POS_FRAMES, 3);
printf("at: %f frames, %f msec\n", cap.get(CAP_PROP_POS_FRAMES), cap.get(CAP_PROP_POS_MSEC));
```
Current
```txt
at: 3.000000 frames, 80.000000 msec
at: 3.000000 frames, 0.234375 msec
```
PR
```txt
at: 3.000000 frames, 80.000000 msec
at: 3.000000 frames, 80.000000 msec
```
It similarly helps with `CAP_PROP_POS_MSEC`:
Current
```txt
at: 3.000000 frames, 80.000000 msec
at: 2.000000 frames, 6.250000 msec
```
PR
```txt
at: 3.000000 frames, 80.000000 msec
at: 2.000000 frames, 40.000000 msec
```
Note the seek operation is still inconsistent between the `CAP_PROP_POS_*` options as mentioned by #23088, and VFR video seeking has issues discussed in #9053. For fixed-frame rate video, we could change 0.5 to 1 in `void CvCapture_FFMPEG::seek(double sec);` to align `CAP_PROP_POS_MSEC` with `CAP_PROP_POS_FRAMES`, but `CAP_POS_AVI_RATIO` and VFR video would still be broken.
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
fix#26276: cv::VideoWriter fails writing colorless images #27153
When writing grayscale images (`isColor=false`), `cv::VideoWriter` failed as FFmpeg backend requires input frames to be in a grayscale format. Unlike other backends, FFmpeg doesn't perform this conversion internally and expects the input to be pre-converted. To fix this, I inserted a check in the `CvVideoWriter_FFMPEG_proxy::write` to convert input frames to grayscale when the input has more than 1 channel. If this is true, then the input is converted to a gray image using `cv::cvtColor` (with `cv::COLOR_BGR2GRAY`).
Additionally, as suggested in the issue comments, I have correctly propagated the return value of `CvVideoWriter_FFMPEG::writeFrame` back to the `CvVideoWriter_FFMPEG_proxy::write`. This return value wasn't being used, and `writeFrame` was always returning false since the FFmeg's return code `AVERROR(EAGAIN)` was mistakenly being treated as an error. Now it's handled as a signal for additional input (current input was successfully written, and a new frame should be sent. [See FFmpeg documentation for `avcodec_receive_packet`](https://ffmpeg.org/doxygen/6.1/group__lavc__decoding.html)). A warning is displayed if `CvVideoWriter_FFMPEG::writeFrame` returns false. Alternatively, this could be propagated back up to the user, making cv::VideoWriter::write a boolean.
Finally, I added a test case to verify if the grayscale conversion is being done correctly.
Fixes#26276
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [ ] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [ ] The feature is well documented and sample code can be built with the project CMake
videoio: print test params instead of indexes #26948
_videoio_ test names changed - use string instead of index.
E.g. `videoio_read.threads/0` is now `videoio_read.threads/h264_0_RAW`.
It allows to filter tests independently of the platform.
**Notes:**
- not all tests has been updated - only simpler ones and those which have varying parameters depending on platform
videoio: fix cv::VideoWriter with FFmpeg encapsulation timestamps #25874
Fix https://github.com/opencv/opencv/issues/25873 by modifying `cv::VideoWriter` to use provided presentation indices (pts).
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
videoio: Add raw encoded video stream muxing to cv::VideoWriter with CAP_FFMPEG #24363
Allow raw encoded video streams (e.g. h264[5]) to be encapsulated by `cv::VideoWriter` to video containers (e.g. mp4/mkv).
Operates in a similar way to https://github.com/opencv/opencv/pull/15290 where encapsulation is enabled by setting the `VideoWriterProperties::VIDEOWRITER_PROP_RAW_VIDEO` flag when constructing `cv::VideoWriter` e.g.
```
VideoWriter container(fileNameOut, api, fourcc, fps, { width, height }, { VideoWriterProperties::VIDEOWRITER_PROP_RAW_VIDEO, 1 });
```
and each raw encoded frame is passed as single row of a `CV_8U` `cv::Mat`.
The main reason for this PR is to allow `cudacodec::VideoWriter` to output its encoded streams to a suitable container, see https://github.com/opencv/opencv_contrib/pull/3569.
### Pull Request Readiness Checklist
See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request
- [x] I agree to contribute to the project under Apache 2 License.
- [x] To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
- [x] The PR is proposed to the proper branch
- [x] There is a reference to the original bug report and related work
- [x] There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
- [x] The feature is well documented and sample code can be built with the project CMake
* Allow the number of threads FFMpeg uses to be selected during VideoCapture::open().
Reset interupt timer in grab if
err = avformat_find_stream_info(ic, NULL);
is interupted but open is successful.
* Correct the returned number of threads and amend test cases.
* Update container test case.
* Reverse changes added to existing videoio_container test case and include test combining thread change and raw read in the newly added videoio_read test case.
Add capacity to Videocapture to return the extraData from FFmpeg when required
* Update rawMode to append any extra data recieved during the initial negotiation of an RTSP stream or during the parsing of an MPEG4 file header.
For h264[5] RTSP streams this ensures the parameter sets if available are always returned on the first call to grab()/read() and has two purposes:
1) To ensure the parameter sets are available even if they are not transmitted in band. This is common for axis ip camera's.
2) To allow callers of VideoCapture::grab()[read()] to write to split the raw stream over multiple files by appending the parameter sets to the begining of any new files.
For (1) there is no alternative, for (2) if the parameter sets were provided in band it would be possible to parse the raw bit stream and search for the parameter sets however that would be a lot of work when that information is already provided by FFMPEG.
For MPEG4 files this information is only suplied in the header and is required for decoding.
Two properties are also required to enable the raw encoded bitstream to be written to multiple files, these are;
1) an indicator as to whether the last frame was a key frame or not - each new file needs to start at a key frame to avoid storing unusable frame diffs,
2) the length in bytes of the paramater sets contained in the last frame - required to split the paramater sets from the frame without having to parse the stream. Any call to VideoCapture::get(CAP_PROP_LF_PARAM_SET_LEN) returning a number greater than zero indicates the presense of a parameter set at the begining of the raw bitstream.
* Adjust test data to account for extraData
* Address warning.
* Change added property names and remove paramater set start code check.
* Output extra data on calls to retrieve instead of appending to the first packet.
* Reverted old test case and added new one to evaluate new functionality.
* Add missing definition.
* Remove flag from legacy api.
Add property to determine if returning extra data is supported.
Always allow extra data to be returned on calls to cap.retrieve()
Update test case.
* Update condition which indicates CAP_PROP_CODEC_EXTRADATA_INDEX is not supported in test case.
* Include compatibility for windows dll if not updated.
Enforce existing return status convention.
* Fix return error and missing test constraints.
add video capture parameters
* add parameters
* videoio: revert unnecessary massive changes
* videoio: support capture parameters in backends API
- add tests
- FFmpeg backend sample code
- StaticBackend API is done
- support through PluginBackend API will be added later
Co-authored-by: Milashchenko <maksim.milashchenko@intel.com>
Co-authored-by: Alexander Alekhin <alexander.a.alekhin@gmail.com>
- Added test for automated rotation for MP4 videos with metadata
- Fix 180 degrees rotation bug
- Moved rotation logic to cv::VideoCapture implementation for FFmpeg and restore binary compatibility with FFmpeg wrapper.
* feature: Add video capture bitrate read-only property for FFMPEG backend
* test: For WIN32 property should be either expected or 0.
Added `IsOneOf` helper function, enabled only for _WIN32.
Add retrieve encoded frame to VideoCapture
* Add capacity to retrieve the encoded frame from a VideoCapture object.
* Correct raw codec and pixle format output from ffmpeg capture.
* Remove warnings from build.
* Added VideoCaptureRaw subclass.
* Include abstract base class VideoCaptureBase and rename new subclass VideoContainer as suggested by mshabunin.
* Remove using.
* Change base class name for compatibility with jave bindings generator.
* Move grab and retrieve and add override specifier
* Add setRaw and readRaw to IVideoCapture interface
-setRaw to disable video decoding and enable bitstream filters from mp4 to h254 and h265.
-readRaw to return the raw undecoded/filtered bitstream.
Add createRawCapture to initiate a backend with setRaw enabled.
Remove inheritance and use an independant VideoContainer subclass with IVideoCapture member.
* Address unused parameter warings.
Remove VideoContainer from python bindings as it no longer returns a Mat.
Use opencv type uchar instead of unsigned char.
Add missing destructor to VideoContainer class.
* Address build warnings and include all params in documentation.
* Include deprecated bitstream filtering API.
* Update codec_id query to work with older ffmpeg api's.
Change api version defines to be consistent - most recent api version first.
* Fix typo.
* Update test to work with naming of new files in the extra repo
* Investigate test failure
* Check bytes read by ffmpeg
* Removed mp4 video container test
* Applied suggested changes.
* videoio: rework API for extraction of RAW video streams
- FFmpeg only
* address review comments
Alexander Smorkalov
Andrew Yooeun Chun
Alex
Sachin Shah
Francisco Mónica
Maksim Shabunin
Alexander Alekhin
cudawarped
Alexander Smorkalov
Stefan Dragnev
Alexander Smorkalov
Suleyman TURKMEN
cudawarped
MaximMilashchenko
Alexander Smorkalov
Vadim Levin
Alexander Alekhin
Pierre Letessier
cudawarped