Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Switch to abseil (https://github.com/abseil/abseil-cpp) as base library #346

Closed
kqyang opened this issue Mar 7, 2018 · 3 comments
Closed

Switch to abseil (https://github.com/abseil/abseil-cpp) as base library #346

kqyang opened this issue Mar 7, 2018 · 3 comments
Assignees
@joeyparrish
Labels
priority: P1 Big impact or workaround impractical; resolve before feature release status: archived Archived and locked; will not be updated type: enhancement New feature or request
Milestone
v3.0

Comments

@kqyang
Copy link
Contributor

kqyang commented Mar 7, 2018

Right now we use base library from Chromium and use GYP as the meta build system. There are a few problems:

  • Chromium base is not really designed to be a common base library. There are a few things enabled by default, which we do not really need.
  • GYP is already deprecated in Chromium two years ago. Because of this, we cannot update our base library without switching to Chromium's new meta build system: GN.

This issue tracks the effort to switch to abseil as base library and bazel as the meta build system for Shaka Packager. It will also enable us to add a CMAKE build.
@kqyang kqyang added the type: enhancement New feature or request label Mar 7, 2018
@kqyang kqyang added this to the v2.2 milestone Mar 7, 2018
@kqyang kqyang modified the milestones: v2.2, v2.3 Jun 29, 2018
shaka-bot pushed a commit that referenced this issue Jul 23, 2018
@yohanncon
Removing dependence on base file atomicops.h
ec63c53 
This part of issue #346.

Change-Id: Iab685145d10d6e6c99ff666e08bca3c36b6d5a66
shaka-bot pushed a commit that referenced this issue Jul 24, 2018
@yohanncon
Removing dependence on base file atomic_sequence_num.h
f331f18 
This part of issue #346.

Change-Id: If03736b688a0bf95aaf09ed3cfd5d5a0134e1c44
@kqyang kqyang mentioned this issue Aug 28, 2018
Closed
@kqyang kqyang modified the milestones: v2.3, v2.4 Oct 24, 2018
@kqyang kqyang modified the milestones: v2.4, v2.5 Feb 19, 2019
@kqyang kqyang modified the milestones: v2.5, Backlog Aug 20, 2020
@joeyparrish joeyparrish self-assigned this Jun 24, 2022
@joeyparrish joeyparrish modified the milestones: Backlog, v2.7 Jun 24, 2022
@joeyparrish joeyparrish added the priority: P1 Big impact or workaround impractical; resolve before feature release label Jun 24, 2022
@joeyparrish
Copy link
Member

joeyparrish commented Jun 24, 2022 •
edited
Loading

As part of this work, C++14 C++17 will now be required to build Shaka Packager. This will reduce our dependencies, and we will rely more on the standard library.

@joeyparrish joeyparrish mentioned this issue Jul 3, 2022
Merged
joeyparrish added a commit that referenced this issue Aug 16, 2022
@joeyparrish
feat: First phase of CMake build system implementation (#1072)
e6b57f7 
There are a lot of changes in this first phase, because there was a
lot of infrastructure required to get some meaningful amount of
porting done.  Future PRs should be simpler.

<b>Summary of changes:</b><details>

 - Remove old deps:
   - boringssl (replaced with mbedtls, lighter, easier to build)
   - gflags (replaced with absl::flags)
   - Chromium build tools
 - New deps to replace parts of Chromium base:
   - abseil-cpp
   - glog
   - nlohmann::json (for tests only)
 - Submodules, updates, and CMake build rules for third-party
   libraries:
   - curl
   - gmock/gtest
 - Ported internal libraries and their tests by removing Chromium deps
   and adding CMake build rules:
   - file (now using C++17 filesystem APIs)
   - license_notice
   - status
   - version
 - Test improvements
   - Removed file tests that can never be re-enabled
   - Re-enabled all other disabled file tests
   - Debug JSON values when HTTP tests fail
   - Fixed chunked-encoding issues in HTTP tests
 - Updated and refactored Dockerfiles testing
   - All docker files working, with OS versions updated to meet the
     new tool requirements
   - Local docker builds no longer write files to your working
     directory as root
   - Local docker builds can now be run in parallel without clobbering
     each others' build outputs
   - DEBUG=1 can drop you into an interactive shell when a docker
     build fails
 - Updated and heavily refactored workflows and Dockerfiles
   - All docker files now tested in parallel on GitHub, speeding up CI
   - All common workflow components broken out and using workflow_call
     instead of custom actions
   - Self-hosted runners now optional, to make testing easier on forks
   - CMake porting works-in-process can now be fully tested on GitHub
   - Building ported libraries and passing ported tests on all three
     platforms!
 - CI hacks for macOS removed, now testing on macos-latest!
 - Python2 no longer required!  (Only Python3)
 - Using strict build flags, treating all warnings as errors.

</details>

<b>Required to build:</b>

 - CMake >= 3.16
 - Python 3
 - A compiler supporting C++ >= 17
   - g++ >= 9 if using GCC (Clang also fine)
   - MSVC for Windows

<b>Still needs work:</b><details>

 - Moving other dependencies into submodules (if we keep them):
   - apple_apsl
   - icu
   - libevent
   - libpng
   - libwebm
   - libxml
   - modp_b64
   - protobuf
   - zlib
 - Port remaining internal libraries:
   - app
   - hls
   - media/base
   - media/chunking
   - media/codecs
   - media/crypto
   - media/demuxer
   - media/event
   - media/formats/dvb
   - media/formats/mp2t
   - media/formats/mp4
   - media/formats/packed_audio
   - media/formats/ttml
   - media/formats/webm
   - media/formats/webvtt
   - media/formats/wvm
   - media/origin
   - media/public
   - media/replicator
   - media/trick_play
   - mpd
 - Port main application
   - Add logging flags in absl and connect them to glog (which expects
     gflags)
 - Port pssh-box.py
 - Port main test targets (packager_test.py and packager_app.py)
 - Updating all requirement and build documentation
 - Remove any remaining refs to gclient, depot_tools, ninja
 - Update and complete release workflows using release-please
</details>

Issue #346 (Switch to abseil)
Issue #1047 (New build system)
@joeyparrish
Copy link
Member

If you would like to help with this effort, please see #1047 (comment) to volunteer. Thanks!

@joeyparrish joeyparrish mentioned this issue Oct 24, 2022
Merged
joeyparrish added a commit that referenced this issue Oct 25, 2022
@joeyparrish
feat: port media/test (#1116)
a9a4f0d 
Rewrite test_data_util.cc to locate files relative to the source file
itself, rather than using a service from chromium `base::`.

Issue #1047 (CMake porting)
Issue #346 (absl porting)
@joeyparrish joeyparrish mentioned this issue Oct 25, 2022
Merged
joeyparrish added a commit that referenced this issue Nov 2, 2022
@joeyparrish
feat: Port AES and RSA crypto to mbedtls (#1119)
7b33f20 
mbedtls works very differently from BoringSSL, and many changes had to
be made in the details of AES decryption to accomodate this.

Beyond the basic changes required for mbedtls, part of the CTS padding
implementation had to be rewritten. I believe this is because of an
assumption that held for BoringSSL, but not for mbedtls. I was unable to
determine what it was, so I rewrote the CTS decryption using reference
materials. After this, tests passed.

The deterministc PRNG I used with mbedtls in the RSA tests differs
somewhat from the old one, so the expected vectors had to be
regenerated. The old determinstic tests were also disabled, and are now
re-enabled.

Since cryptography is sensitive code, and because there were far more
changes needed here than just updating some headers and utility function
calls, this has been split into its own PR for separate review from the
rest of the media/base porting work.

Issue #1047 (CMake porting)
Issue #346 (absl porting)
@joeyparrish joeyparrish mentioned this issue Nov 2, 2022
Merged
joeyparrish added a commit that referenced this issue Nov 4, 2022
@joeyparrish
feat: port media/base (#1127)
b1095f6 
This removes all chromium dependencies from media/base/ and completes
the build system in CMake.

The ClosureThread class and its classes were removed, as they were
specific to chromium base. ClosureThread has been replaced by
std::thread.

The byte-swapping utilities in network_util.cc have been removed and
replaced with absl.

generate_unique_temp_path() was split out of file_unittest.cc into
file_test_util.cc, where other test suites could make use of it.

WARN_UNUSED_RESULT was replaced with the C++ standard attribute
[[nodiscard]].

The base::Clock interface was replaced with a typedef for a function
pointer that returns the current time.

This re-enables the tests in http_key_fetcher_unittest.cc by using
httpbin.org.

Issue #1047 (CMake porting)
Issue #346 (absl porting)
@joeyparrish joeyparrish mentioned this issue Dec 6, 2022
Closed
joeyparrish pushed a commit that referenced this issue Dec 16, 2022
@cadubentzen
feat: port media/codecs to CMake (#1143)
e9bf0c6 
Issue #1047 (CMake port)
Issue #346 (absl port)
joeyparrish added a commit to joeyparrish/shaka-packager that referenced this issue Jul 20, 2023
@joeyparrish
feat: Port app/job_manager to cmake and absl
0430490 
Issue shaka-project#1047 (cmake)
Issue shaka-project#346 (absl)
@joeyparrish joeyparrish mentioned this issue Jul 20, 2023
Merged
@joeyparrish joeyparrish pinned this issue Jul 20, 2023
cosmin pushed a commit that referenced this issue Jul 21, 2023
@joeyparrish
feat: Port app/job_manager to cmake and absl (#1249)
ba51270 
Issue #1047 (cmake)
Issue #346 (absl)
@joeyparrish
Copy link
Member

There are no longer any references to chromium base in the cmake branch. Once we clean up documentation, logging, and some automated workflows, we can merge the cmake branch into main. But since the absl porting away from base is complete, I'm closing this issue.

@github-actions github-actions bot added the status: archived Archived and locked; will not be updated label Nov 12, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Nov 12, 2023
@joeyparrish joeyparrish unpinned this issue Feb 20, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@joeyparrish joeyparrish

Labels
priority: P1 Big impact or workaround impractical; resolve before feature release status: archived Archived and locked; will not be updated type: enhancement New feature or request
Projects
None yet
Milestone
v3.0
Development

No branches or pull requests
2 participants
@joeyparrish @kqyang